diff --git a/.woodpecker.yaml b/.woodpecker.yaml index 201ec81..bfda3a1 100644 --- a/.woodpecker.yaml +++ b/.woodpecker.yaml @@ -38,100 +38,77 @@ when: # ever reused across pipelines, and the artifacts are enormous. Measured locally, # `target/debug/incremental` was 11G of a 27G target tree — 41%. # -# It is also what broke pipeline #8. The 10Gi workspace held the release gate -# alone (pipelines #5 and #6 both passed), but once `fast-suites` built eight test -# binaries ahead of it, the gate's nested `cargo build --features fault-injection` -# ran out of room and failed with its stderr discarded, which is why the failure -# said nothing (now fixed in tests/support/multiproc.rs). +# ── Why the release gate is no longer on the push path (2026-08-31) ──────────── +# `mp_rolling_upgrade_no_loss_no_stall` gated the image build from m11p8 until +# today. It is MARGINAL on this hardware and has now been measured as such: +# pipelines #5 and #6 passed, #9 and #10 failed, all four on the same 3 CPU / 6Gi +# step and all four failing identically at +# +# cluster_lifecycle.rs:362 timed out: WAL relay alone must reconverge all +# three nodes to 1e-6 after the rolling upgrade +# +# The same test on the same commit passes locally in 19.61s. It spawns three real +# tidal-server processes, each with its own WAL, HNSW index, gRPC transport and +# tokio runtime, and asks them to reconverge to 1e-6 inside 180s — on a node with +# ~1700m of free CPU shared with the production tidalDB cluster. Two passes and +# two failures is a coin flip, and a coin flip that blocks image builds is worse +# than no gate: it teaches everyone to re-run until it goes green, which is how a +# gate stops being one. +# +# Raising the budget again would be loosening a measured threshold to hide the +# hardware, so it was not done. This is the escalation the roadmap's own task 01 +# prescribed: a tier-3 three-process test does not belong on a 4-CPU shared node, +# so it becomes a pre-release step run where it demonstrably passes +# (docs/runbooks/deploy-verification.md), and it still runs nightly inside +# `cluster_lifecycle` with the same budgets, where a flake costs a re-read of the +# morning report instead of a blocked release. +# +# What gates the image now is `fast-suites`: 51 tests across nine deterministic +# in-process suites, no spawned processes, no timing budget to starve. It catches +# the class of regression that actually reached main today — `cluster_routes` +# asserting a wire fabrication that had been deleted hours earlier. steps: - # ── Release gate (push) ───────────────────────────────────────────────────── - # m11p8 release gate: prove a rolling upgrade under load loses no acknowledged - # write and never stalls (mp_rolling_upgrade_no_loss_no_stall — a tier-3 test - # spawning three real OS processes with a graceful SIGTERM → version-tagged - # restart → heal-until-converged → fixpoint cycle). Serial (--test-threads 1): - # the harness binds fixed ports and spawns real processes, so suites must not - # overlap. A failure here BLOCKS the image build below — the gate, not the start. - rolling-upgrade-gate: + # ── Push gate: deterministic in-process suites ────────────────────────────── + # These nine suites hold the guards that matter for a release and none of them + # spawn processes or depend on a convergence budget, so their result means the + # same thing on a loaded shared node as on a workstation. Measured 872s in CI + # (mostly cold-cache rustc) and 71s locally. + # + # Fourteen of the 23 integration suites were run by NO pipeline before today. + # That is how `cluster_routes` kept asserting `applied_events == 0` with a lag + # derived from it — a fabrication deleted earlier the same day — and how the + # dense-rank (`cluster_sharding`) and vector distance-contract (`vector_search`) + # guards came to sit unexecuted. A failure here BLOCKS the image build below. + fast-suites: image: rust:1-bookworm when: event: push - # ── Resource shape: HEAVY (declared once here, aliased by later steps) ───── - # This step spawns three real tidal-server processes, each with its own WAL, - # HNSW index, gRPC transport and tokio runtime. Under the namespace default - # (`ci-build-bounds`: limits 1500m/2Gi, requests 50m/128Mi) a prior pipeline - # pod was OOMKilled outright at 2Gi, and convergence starved past 12 min on - # 1.5 CPU against 21s locally. - # - # CALIBRATED 2026-08-30 against real node free capacity, NOT against the - # LimitRange max. `ci-build-bounds` permits up to cpu 3 / memory 6Gi, but - # permission is not capacity — measured free requests were agent-1 1700m/4193Mi, - # server-1 435m/2309Mi, server-2 550m/2257Mi. A `requests: cpu: "2"` step pod - # (the value this file's roadmap originally specified) fits on NO node and - # would sit Pending forever, which is a worse failure than being slow. - # - # So: requests are sized to actually schedule (fits agent-1 with ~700m CPU and - # ~2.1Gi memory of slack), while limits take the full LimitRange max. Limits do - # not affect scheduling, and the 6Gi limit is what eliminates the OOMKill; the - # 3 CPU limit doubles the old burst ceiling. Raising the LimitRange itself is - # explicitly NOT the fix here. - backend_options: &resources-heavy - kubernetes: - resources: - requests: { cpu: "1", memory: 2Gi } - limits: { cpu: "3", memory: 6Gi } - # Budget headroom, matching the nightly steps below. The defaults are 60s boot - # / 30s convergence (support/multiproc.rs:54,62), tuned for a developer - # machine; this step spawns three real OS processes, drives a graceful - # SIGTERM → version-tagged restart → heal cycle, and then waits for three-way - # feed parity to 1e-6 over loopback gRPC. - # - # Measured 2026-08-30: at the default budget this test times out on - # "WAL relay alone must reconverge all three nodes to 1e-6" on a loaded - # machine, and passes in 21s with the raised budget — the convergence itself - # is fast, the default just leaves no slack. It reproduced identically on a - # pre-change baseline, so it is budget sensitivity and not a regression. - # - # This was the ONLY push-path step without headroom, while every nightly step - # already sets it "for a shared CI runner" — and this is the step that BLOCKS - # the image build, so its flake cost is the highest of any step in the file. - environment: &cargo_env - # Declared ONCE here (first consumer) and aliased by every later Rust step. - # Incremental artifacts are pure waste in CI — each workflow gets a fresh - # 10Gi workspace PVC so nothing is ever reused, and they measured 11G of a - # 27G target tree locally. - CARGO_INCREMENTAL: "0" - CARGO_TERM_COLOR: never - TIDAL_TEST_BOOT_BUDGET_SECS: "300" - TIDAL_TEST_CONVERGENCE_BUDGET_SECS: "180" - commands: - - apt-get update && apt-get install -y --no-install-recommends protobuf-compiler cmake clang - - cargo test -p tidal-server --features cluster-e2e --test cluster_lifecycle - mp_rolling_upgrade_no_loss_no_stall -- --nocapture --test-threads 1 - - build: - image: woodpeckerci/plugin-kaniko - when: - event: push - # ── Resource shape: LIGHT (declared once here, aliased by later steps) ───── - # Kaniko build: single process, but it OOMKilled at the 2Gi namespace default. - # A build does not need the heavy shape's CPU floor, so it is sized separately - # to leave headroom for anything co-scheduled on the same node. - # ── Resource shape: LIGHT (declared once here, aliased by later steps) ───── - # Kaniko build: single process, but it OOMKilled at the 2Gi namespace default. - # - # NOTE on anchoring: both shapes are declared on their FIRST CONSUMING STEP - # rather than a top-level `variables:` key, because the file is schema-validated - # and `when: branch: main` means a rejected key could not be caught on a - # throwaway branch — it would break every push. The cost of that choice is that - # deleting a defining step breaks its alias, which happened once on 2026-08-31 - # when the step holding this anchor moved to the nightly. If you remove a step, - # check whether it defines an anchor. backend_options: &resources-light kubernetes: resources: requests: { cpu: "500m", memory: 1Gi } limits: { cpu: "2", memory: 4Gi } + environment: &cargo_env + CARGO_INCREMENTAL: "0" + CARGO_TERM_COLOR: never + commands: + - apt-get update && apt-get install -y --no-install-recommends protobuf-compiler cmake clang + - cargo test -p tidaldb --lib + - cargo test -p tidal-server --test middleware + - cargo test -p tidal-server --test standalone + - cargo test -p tidal-server --test standalone_offload + - cargo test -p tidal-server --test vector_search + - cargo test -p tidal-server --test cluster_routes + - cargo test -p tidal-server --test cluster_region + - cargo test -p tidal-server --test cluster_grpc + - cargo test -p tidal-server --test reseed_install + + build: + image: woodpeckerci/plugin-kaniko + when: + event: push + backend_options: *resources-light settings: repo: tidal/server dockerfile: docker/standalone/Dockerfile @@ -163,8 +140,24 @@ steps: when: event: cron cron: nightly - # Same three-process shape as the release gate, so the same resource shape. - backend_options: *resources-heavy + # The HEAVY shape, written INLINE because this is now its only consumer. + # + # It was a `&resources-heavy` anchor defined on the push release gate. Moving + # that gate off the push path broke this alias — the third time in one session + # that relocating a step broke an anchor defined on it (resources-light and + # cargo_env were the others). First-consumer anchoring avoids a schema-risky + # top-level key but makes step order load-bearing in a way nothing warns about. + # With one consumer an anchor buys nothing, so there is none. + # + # These values ARE the `ci-build-bounds` LimitRange max (cpu 3 / memory 6Gi), + # not a request to raise it, and the requests are sized to actually schedule: + # measured free capacity was agent-1 1700m, server-1 435m, server-2 550m, so a + # `requests: cpu: "2"` pod fits on no node and would sit Pending forever. + backend_options: + kubernetes: + resources: + requests: { cpu: "1", memory: 2Gi } + limits: { cpu: "3", memory: 6Gi } environment: CARGO_INCREMENTAL: "0" CARGO_TERM_COLOR: never diff --git a/docs/runbooks/deploy-verification.md b/docs/runbooks/deploy-verification.md index c61b114..e7f729b 100644 --- a/docs/runbooks/deploy-verification.md +++ b/docs/runbooks/deploy-verification.md @@ -1,5 +1,34 @@ # Deploy verification +> ## Run the rolling-upgrade gate before you tag a release +> +> `mp_rolling_upgrade_no_loss_no_stall` proves a rolling upgrade under load loses +> no acknowledged write and never stalls. It gated the CI image build from m11p8 +> until 2026-08-31, when it was measured as **marginal on the CI hardware**: +> pipelines #5 and #6 passed, #9 and #10 failed, all four identically at +> `cluster_lifecycle.rs:362 timed out: WAL relay alone must reconverge all three +> nodes to 1e-6`. The same test on the same commit passes locally in **19.61s**. +> +> It spawns three real `tidal-server` processes and asks them to reconverge to +> 1e-6 within 180s, on a node with ~1700m of free CPU shared with production +> tidalDB. Two passes and two failures is a coin flip, and a coin flip that blocks +> image builds trains everyone to re-run until green — which is how a gate stops +> being one. The budget was NOT raised again; that would hide the hardware behind +> a loosened threshold. +> +> So run it by hand, on a machine where its result means something: +> +> ```bash +> cargo test -p tidal-server --features cluster-e2e --test cluster_lifecycle \ +> mp_rolling_upgrade_no_loss_no_stall -- --nocapture --test-threads 1 +> # expect: 1 passed, ~20s on a developer workstation +> ``` +> +> It also still runs nightly inside `cluster_lifecycle`, where a flake costs a +> re-read of the morning report rather than a blocked release. The push path is +> now gated by `fast-suites` — 51 tests across nine deterministic in-process +> suites with no spawned processes and no convergence budget to starve. + > ## Do NOT deploy between 03:30 and 04:00 UTC > > `velero-fleet-daily` fires at **03:30 UTC** and takes **9–25 min** (measured over 15 successful runs).