From 320d640d13670f3c9a5d241ad3d46f848a77a157 Mon Sep 17 00:00:00 2001 From: jordan Date: Mon, 31 Aug 2026 01:57:27 -0600 Subject: [PATCH] ci: move the in-process suites to the nightly; the gate keeps its measured budget MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pipelines #8 and #9 both failed, and the second one told me exactly why once the nested build's stderr was no longer discarded: cluster_lifecycle.rs:362 timed out: WAL relay alone must reconverge all three nodes to 1e-6 after the rolling upgrade That is verbatim the budget sensitivity this file already documents at the gate: "at the default budget this test times out on 'WAL relay alone must reconverge all three nodes to 1e-6' on a loaded machine, and passes in 21s with the raised budget — the convergence itself is fast, the default just leaves no slack." So it was not a code regression and not disk. `fast-suites` burned ~872s of CPU-saturating rustc immediately before the gate, and the gate's 300s/180s budgets are calibrated for a node that is NOT fresh off fifteen minutes of parallel compilation. I invalidated the calibration by adding load in front of it. Raising the budget would be loosening a measured threshold to hide load I introduced — the exact move this project forbids. Blocking the image build is the gate's job and it outranks fifteen-minutes-faster feedback on in-process suites, so the eight suites moved into `nightly-security-ops` (already the light in-process nightly step) and the gate kept its calibration and its proven shape from pipelines #5 and #6. Coverage still goes from NEVER to nightly for all fourteen previously-unscheduled suites; every one of the 23 now has a runner. CARGO_INCREMENTAL=0 stays: it is correct in CI regardless, since each workflow gets a fresh 10Gi workspace and incremental artifacts measured 11G of a 27G target tree. Also documents an anchoring cost discovered the hard way: the resource shapes are declared on their first consuming step (to avoid a schema-risky top-level key), and moving the step that held `&resources-light` broke its aliases. The note now says to check for anchor definitions before removing a step. --- .woodpecker.yaml | 91 ++++++++++++++++++++++++------------------------ 1 file changed, 46 insertions(+), 45 deletions(-) diff --git a/.woodpecker.yaml b/.woodpecker.yaml index 5edfcce..201ec81 100644 --- a/.woodpecker.yaml +++ b/.woodpecker.yaml @@ -45,49 +45,6 @@ when: # said nothing (now fixed in tests/support/multiproc.rs). steps: - # ── Fast in-process suites (push) ─────────────────────────────────────────── - # ADDED 2026-08-31 after a survey found that 14 of 23 integration suites were - # run by NO pipeline at all. That is not a theoretical gap: `cluster_routes` - # still asserted a wire fabrication that had been removed hours earlier - # (`applied_events == 0` with a lag derived from it), and nothing caught it - # because nothing ran it. `cluster_sharding` and `vector_search` — which hold - # the dense-rank, /sharded/* opt-in and vector distance-contract gates — were - # in the same position, so the guards written for those fixes would have rotted - # the same way. - # - # These eight need neither the `cluster-e2e` feature nor spawned processes, so - # they take the LIGHT shape. Measured locally: 71s wall for all eight, 51 tests. - # Against a 6.5-minute gate that is free, and it runs FIRST so a cheap failure - # is reported before the expensive one starts. - fast-suites: - image: rust:1-bookworm - when: - event: push - backend_options: &resources-light - kubernetes: - resources: - requests: { cpu: "500m", memory: 1Gi } - limits: { cpu: "2", memory: 4Gi } - # Declared ONCE here (first consumer) and aliased by every later Rust step — - # same technique as the resource shapes, and for the same reason: a custom - # top-level key would be a schema risk, and because `when: branch: main` means - # only main triggers a pipeline, a parse error could not be caught on a - # throwaway branch and would break every push until reverted. - environment: &cargo_env - CARGO_INCREMENTAL: "0" - CARGO_TERM_COLOR: never - commands: - - apt-get update && apt-get install -y --no-install-recommends protobuf-compiler cmake clang - - cargo test -p tidaldb --lib - - cargo test -p tidal-server --test middleware - - cargo test -p tidal-server --test standalone - - cargo test -p tidal-server --test standalone_offload - - cargo test -p tidal-server --test vector_search - - cargo test -p tidal-server --test cluster_routes - - cargo test -p tidal-server --test cluster_region - - cargo test -p tidal-server --test cluster_grpc - - cargo test -p tidal-server --test reseed_install - # ── Release gate (push) ───────────────────────────────────────────────────── # m11p8 release gate: prove a rolling upgrade under load loses no acknowledged # write and never stalls (mp_rolling_upgrade_no_loss_no_stall — a tier-3 test @@ -138,7 +95,11 @@ steps: # This was the ONLY push-path step without headroom, while every nightly step # already sets it "for a shared CI runner" — and this is the step that BLOCKS # the image build, so its flake cost is the highest of any step in the file. - environment: + environment: &cargo_env + # Declared ONCE here (first consumer) and aliased by every later Rust step. + # Incremental artifacts are pure waste in CI — each workflow gets a fresh + # 10Gi workspace PVC so nothing is ever reused, and they measured 11G of a + # 27G target tree locally. CARGO_INCREMENTAL: "0" CARGO_TERM_COLOR: never TIDAL_TEST_BOOT_BUDGET_SECS: "300" @@ -156,7 +117,21 @@ steps: # Kaniko build: single process, but it OOMKilled at the 2Gi namespace default. # A build does not need the heavy shape's CPU floor, so it is sized separately # to leave headroom for anything co-scheduled on the same node. - backend_options: *resources-light + # ── Resource shape: LIGHT (declared once here, aliased by later steps) ───── + # Kaniko build: single process, but it OOMKilled at the 2Gi namespace default. + # + # NOTE on anchoring: both shapes are declared on their FIRST CONSUMING STEP + # rather than a top-level `variables:` key, because the file is schema-validated + # and `when: branch: main` means a rejected key could not be caught on a + # throwaway branch — it would break every push. The cost of that choice is that + # deleting a defining step breaks its alias, which happened once on 2026-08-31 + # when the step holding this anchor moved to the nightly. If you remove a step, + # check whether it defines an anchor. + backend_options: &resources-light + kubernetes: + resources: + requests: { cpu: "500m", memory: 1Gi } + limits: { cpu: "2", memory: 4Gi } settings: repo: tidal/server dockerfile: docker/standalone/Dockerfile @@ -252,3 +227,29 @@ steps: - cargo test -p tidal-server --lib - cargo test -p tidalctl - cargo test -p tidaldb --lib wal::compaction + # ADDED 2026-08-31. These eight were run by NO pipeline before today, which + # is how `cluster_routes` kept asserting a wire fabrication removed hours + # earlier, and how the dense-rank (`cluster_sharding`) and vector + # distance-contract (`vector_search`) guards came to sit unexecuted. + # + # They were briefly on the PUSH path and are deliberately here instead. + # Pipelines #8 and #9 proved why: running them first burns ~872s of + # CPU-saturating rustc, and the release gate immediately after then failed on + # "WAL relay alone must reconverge all three nodes to 1e-6" — the exact + # budget sensitivity this file already documents at the gate. The gate's + # 300s/180s budgets are calibrated for a node that is NOT fresh off fifteen + # minutes of parallel compilation. + # + # Raising the gate's budget to accommodate them would be loosening a measured + # threshold to hide load I introduced. Blocking the image build is the gate's + # job and it matters more than fifteen-minutes-faster feedback on in-process + # suites, so the suites moved and the gate kept its calibration. Coverage + # still went from never to nightly. + - cargo test -p tidal-server --test middleware + - cargo test -p tidal-server --test standalone + - cargo test -p tidal-server --test standalone_offload + - cargo test -p tidal-server --test vector_search + - cargo test -p tidal-server --test cluster_routes + - cargo test -p tidal-server --test cluster_region + - cargo test -p tidal-server --test cluster_grpc + - cargo test -p tidal-server --test reseed_install