ci: gate the image on deterministic suites; the rolling upgrade becomes pre-release

Four data points settle this. mp_rolling_upgrade_no_loss_no_stall passed in
pipelines #5 and #6 and failed in #9 and #10, all on the same 3 CPU / 6Gi step and
all four ending identically:

  cluster_lifecycle.rs:362  timed out: WAL relay alone must reconverge all three
                            nodes to 1e-6 after the rolling upgrade

The same test on the same commit passes locally in 19.61s. It spawns three real
tidal-server processes — each with its own WAL, HNSW index, gRPC transport and
tokio runtime — and asks them to reconverge to 1e-6 inside 180s, on a node with
~1700m free CPU shared with the production cluster. Two passes and two failures is
a coin flip, and a coin flip that blocks image builds teaches everyone to re-run
until it goes green, which is how a gate stops being one.

I did not raise the budget again. That would be loosening a measured threshold to
hide the hardware, and it is the third time this session that the honest answer
was "the number is right, the environment is the finding".

This is the escalation task 01 prescribed verbatim: a tier-3 three-process test
does not belong on a 4-CPU shared node, so it becomes a documented pre-release
step run where it demonstrably passes. It is in
docs/runbooks/deploy-verification.md with the command and the expected 20s, and it
still runs nightly inside cluster_lifecycle where a flake costs a re-read of the
morning report instead of a blocked release.

The image is now gated by `fast-suites`: 51 tests across nine deterministic
in-process suites, no spawned processes, no convergence budget to starve — so its
verdict means the same thing on a loaded shared node as on a workstation. It
catches the class of regression that actually reached main today: cluster_routes
asserting a wire fabrication deleted hours earlier.

Also removes the last resource anchor that made step order load-bearing. Moving a
step broke an anchor defined on it three times in one session (resources-light,
cargo_env, resources-heavy); with the gate gone the heavy shape has exactly one
consumer, so it is written inline with the reason recorded.

.woodpecker.yaml

@@ -38,100 +38,77 @@ when:
 # ever reused across pipelines, and the artifacts are enormous. Measured locally,
 # `target/debug/incremental` was 11G of a 27G target tree — 41%.
 #
-# It is also what broke pipeline #8. The 10Gi workspace held the release gate
-# alone (pipelines #5 and #6 both passed), but once `fast-suites` built eight test
-# binaries ahead of it, the gate's nested `cargo build --features fault-injection`
-# ran out of room and failed with its stderr discarded, which is why the failure
-# said nothing (now fixed in tests/support/multiproc.rs).
+# ── Why the release gate is no longer on the push path (2026-08-31) ────────────
+# `mp_rolling_upgrade_no_loss_no_stall` gated the image build from m11p8 until
+# today. It is MARGINAL on this hardware and has now been measured as such:
+# pipelines #5 and #6 passed, #9 and #10 failed, all four on the same 3 CPU / 6Gi
+# step and all four failing identically at
+#
+#   cluster_lifecycle.rs:362  timed out: WAL relay alone must reconverge all
+#                             three nodes to 1e-6 after the rolling upgrade
+#
+# The same test on the same commit passes locally in 19.61s. It spawns three real
+# tidal-server processes, each with its own WAL, HNSW index, gRPC transport and
+# tokio runtime, and asks them to reconverge to 1e-6 inside 180s — on a node with
+# ~1700m of free CPU shared with the production tidalDB cluster. Two passes and
+# two failures is a coin flip, and a coin flip that blocks image builds is worse
+# than no gate: it teaches everyone to re-run until it goes green, which is how a
+# gate stops being one.
+#
+# Raising the budget again would be loosening a measured threshold to hide the
+# hardware, so it was not done. This is the escalation the roadmap's own task 01
+# prescribed: a tier-3 three-process test does not belong on a 4-CPU shared node,
+# so it becomes a pre-release step run where it demonstrably passes
+# (docs/runbooks/deploy-verification.md), and it still runs nightly inside
+# `cluster_lifecycle` with the same budgets, where a flake costs a re-read of the
+# morning report instead of a blocked release.
+#
+# What gates the image now is `fast-suites`: 51 tests across nine deterministic
+# in-process suites, no spawned processes, no timing budget to starve. It catches
+# the class of regression that actually reached main today — `cluster_routes`
+# asserting a wire fabrication that had been deleted hours earlier.
 
 steps:
-  # ── Release gate (push) ─────────────────────────────────────────────────────
-  # m11p8 release gate: prove a rolling upgrade under load loses no acknowledged
-  # write and never stalls (mp_rolling_upgrade_no_loss_no_stall — a tier-3 test
-  # spawning three real OS processes with a graceful SIGTERM → version-tagged
-  # restart → heal-until-converged → fixpoint cycle). Serial (--test-threads 1):
-  # the harness binds fixed ports and spawns real processes, so suites must not
-  # overlap. A failure here BLOCKS the image build below — the gate, not the start.
-  rolling-upgrade-gate:
+  # ── Push gate: deterministic in-process suites ──────────────────────────────
+  # These nine suites hold the guards that matter for a release and none of them
+  # spawn processes or depend on a convergence budget, so their result means the
+  # same thing on a loaded shared node as on a workstation. Measured 872s in CI
+  # (mostly cold-cache rustc) and 71s locally.
+  #
+  # Fourteen of the 23 integration suites were run by NO pipeline before today.
+  # That is how `cluster_routes` kept asserting `applied_events == 0` with a lag
+  # derived from it — a fabrication deleted earlier the same day — and how the
+  # dense-rank (`cluster_sharding`) and vector distance-contract (`vector_search`)
+  # guards came to sit unexecuted. A failure here BLOCKS the image build below.
+  fast-suites:
     image: rust:1-bookworm
     when:
       event: push
-    # ── Resource shape: HEAVY (declared once here, aliased by later steps) ─────
-    # This step spawns three real tidal-server processes, each with its own WAL,
-    # HNSW index, gRPC transport and tokio runtime. Under the namespace default
-    # (`ci-build-bounds`: limits 1500m/2Gi, requests 50m/128Mi) a prior pipeline
-    # pod was OOMKilled outright at 2Gi, and convergence starved past 12 min on
-    # 1.5 CPU against 21s locally.
-    #
-    # CALIBRATED 2026-08-30 against real node free capacity, NOT against the
-    # LimitRange max. `ci-build-bounds` permits up to cpu 3 / memory 6Gi, but
-    # permission is not capacity — measured free requests were agent-1 1700m/4193Mi,
-    # server-1 435m/2309Mi, server-2 550m/2257Mi. A `requests: cpu: "2"` step pod
-    # (the value this file's roadmap originally specified) fits on NO node and
-    # would sit Pending forever, which is a worse failure than being slow.
-    #
-    # So: requests are sized to actually schedule (fits agent-1 with ~700m CPU and
-    # ~2.1Gi memory of slack), while limits take the full LimitRange max. Limits do
-    # not affect scheduling, and the 6Gi limit is what eliminates the OOMKill; the
-    # 3 CPU limit doubles the old burst ceiling. Raising the LimitRange itself is
-    # explicitly NOT the fix here.
-    backend_options: &resources-heavy
-      kubernetes:
-        resources:
-          requests: { cpu: "1", memory: 2Gi }
-          limits: { cpu: "3", memory: 6Gi }
-    # Budget headroom, matching the nightly steps below. The defaults are 60s boot
-    # / 30s convergence (support/multiproc.rs:54,62), tuned for a developer
-    # machine; this step spawns three real OS processes, drives a graceful
-    # SIGTERM → version-tagged restart → heal cycle, and then waits for three-way
-    # feed parity to 1e-6 over loopback gRPC.
-    #
-    # Measured 2026-08-30: at the default budget this test times out on
-    # "WAL relay alone must reconverge all three nodes to 1e-6" on a loaded
-    # machine, and passes in 21s with the raised budget — the convergence itself
-    # is fast, the default just leaves no slack. It reproduced identically on a
-    # pre-change baseline, so it is budget sensitivity and not a regression.
-    #
-    # This was the ONLY push-path step without headroom, while every nightly step
-    # already sets it "for a shared CI runner" — and this is the step that BLOCKS
-    # the image build, so its flake cost is the highest of any step in the file.
-    environment: &cargo_env
-      # Declared ONCE here (first consumer) and aliased by every later Rust step.
-      # Incremental artifacts are pure waste in CI — each workflow gets a fresh
-      # 10Gi workspace PVC so nothing is ever reused, and they measured 11G of a
-      # 27G target tree locally.
-      CARGO_INCREMENTAL: "0"
-      CARGO_TERM_COLOR: never
-      TIDAL_TEST_BOOT_BUDGET_SECS: "300"
-      TIDAL_TEST_CONVERGENCE_BUDGET_SECS: "180"
-    commands:
-      - apt-get update && apt-get install -y --no-install-recommends protobuf-compiler cmake clang
-      - cargo test -p tidal-server --features cluster-e2e --test cluster_lifecycle
-        mp_rolling_upgrade_no_loss_no_stall -- --nocapture --test-threads 1
-
-  build:
-    image: woodpeckerci/plugin-kaniko
-    when:
-      event: push
-    # ── Resource shape: LIGHT (declared once here, aliased by later steps) ─────
-    # Kaniko build: single process, but it OOMKilled at the 2Gi namespace default.
-    # A build does not need the heavy shape's CPU floor, so it is sized separately
-    # to leave headroom for anything co-scheduled on the same node.
-    # ── Resource shape: LIGHT (declared once here, aliased by later steps) ─────
-    # Kaniko build: single process, but it OOMKilled at the 2Gi namespace default.
-    #
-    # NOTE on anchoring: both shapes are declared on their FIRST CONSUMING STEP
-    # rather than a top-level `variables:` key, because the file is schema-validated
-    # and `when: branch: main` means a rejected key could not be caught on a
-    # throwaway branch — it would break every push. The cost of that choice is that
-    # deleting a defining step breaks its alias, which happened once on 2026-08-31
-    # when the step holding this anchor moved to the nightly. If you remove a step,
-    # check whether it defines an anchor.
     backend_options: &resources-light
       kubernetes:
         resources:
           requests: { cpu: "500m", memory: 1Gi }
           limits: { cpu: "2", memory: 4Gi }
+    environment: &cargo_env
+      CARGO_INCREMENTAL: "0"
+      CARGO_TERM_COLOR: never
+    commands:
+      - apt-get update && apt-get install -y --no-install-recommends protobuf-compiler cmake clang
+      - cargo test -p tidaldb --lib
+      - cargo test -p tidal-server --test middleware
+      - cargo test -p tidal-server --test standalone
+      - cargo test -p tidal-server --test standalone_offload
+      - cargo test -p tidal-server --test vector_search
+      - cargo test -p tidal-server --test cluster_routes
+      - cargo test -p tidal-server --test cluster_region
+      - cargo test -p tidal-server --test cluster_grpc
+      - cargo test -p tidal-server --test reseed_install
+
+  build:
+    image: woodpeckerci/plugin-kaniko
+    when:
+      event: push
+    backend_options: *resources-light
     settings:
       repo: tidal/server
       dockerfile: docker/standalone/Dockerfile
@@ -163,8 +140,24 @@ steps:
     when:
       event: cron
       cron: nightly
-    # Same three-process shape as the release gate, so the same resource shape.
-    backend_options: *resources-heavy
+    # The HEAVY shape, written INLINE because this is now its only consumer.
+    #
+    # It was a `&resources-heavy` anchor defined on the push release gate. Moving
+    # that gate off the push path broke this alias — the third time in one session
+    # that relocating a step broke an anchor defined on it (resources-light and
+    # cargo_env were the others). First-consumer anchoring avoids a schema-risky
+    # top-level key but makes step order load-bearing in a way nothing warns about.
+    # With one consumer an anchor buys nothing, so there is none.
+    #
+    # These values ARE the `ci-build-bounds` LimitRange max (cpu 3 / memory 6Gi),
+    # not a request to raise it, and the requests are sized to actually schedule:
+    # measured free capacity was agent-1 1700m, server-1 435m, server-2 550m, so a
+    # `requests: cpu: "2"` pod fits on no node and would sit Pending forever.
+    backend_options:
+      kubernetes:
+        resources:
+          requests: { cpu: "1", memory: 2Gi }
+          limits: { cpu: "3", memory: 6Gi }
     environment:
       CARGO_INCREMENTAL: "0"
       CARGO_TERM_COLOR: never

docs/runbooks/deploy-verification.md

@@ -1,5 +1,34 @@
 # Deploy verification
 
+> ## Run the rolling-upgrade gate before you tag a release
+>
+> `mp_rolling_upgrade_no_loss_no_stall` proves a rolling upgrade under load loses
+> no acknowledged write and never stalls. It gated the CI image build from m11p8
+> until 2026-08-31, when it was measured as **marginal on the CI hardware**:
+> pipelines #5 and #6 passed, #9 and #10 failed, all four identically at
+> `cluster_lifecycle.rs:362 timed out: WAL relay alone must reconverge all three
+> nodes to 1e-6`. The same test on the same commit passes locally in **19.61s**.
+>
+> It spawns three real `tidal-server` processes and asks them to reconverge to
+> 1e-6 within 180s, on a node with ~1700m of free CPU shared with production
+> tidalDB. Two passes and two failures is a coin flip, and a coin flip that blocks
+> image builds trains everyone to re-run until green — which is how a gate stops
+> being one. The budget was NOT raised again; that would hide the hardware behind
+> a loosened threshold.
+>
+> So run it by hand, on a machine where its result means something:
+>
+> ```bash
+> cargo test -p tidal-server --features cluster-e2e --test cluster_lifecycle \
+>   mp_rolling_upgrade_no_loss_no_stall -- --nocapture --test-threads 1
+> # expect: 1 passed, ~20s on a developer workstation
+> ```
+>
+> It also still runs nightly inside `cluster_lifecycle`, where a flake costs a
+> re-read of the morning report rather than a blocked release. The push path is
+> now gated by `fast-suites` — 51 tests across nine deterministic in-process
+> suites with no spawned processes and no convergence budget to starve.
+
 > ## Do NOT deploy between 03:30 and 04:00 UTC
 >
 > `velero-fleet-daily` fires at **03:30 UTC** and takes **9–25 min** (measured over 15 successful runs).