ci: move the in-process suites to the nightly; the gate keeps its measured budget
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed

Pipelines #8 and #9 both failed, and the second one told me exactly why once the
nested build's stderr was no longer discarded:

  cluster_lifecycle.rs:362  timed out: WAL relay alone must reconverge all three
                            nodes to 1e-6 after the rolling upgrade

That is verbatim the budget sensitivity this file already documents at the gate:
"at the default budget this test times out on 'WAL relay alone must reconverge all
three nodes to 1e-6' on a loaded machine, and passes in 21s with the raised
budget — the convergence itself is fast, the default just leaves no slack."

So it was not a code regression and not disk. `fast-suites` burned ~872s of
CPU-saturating rustc immediately before the gate, and the gate's 300s/180s budgets
are calibrated for a node that is NOT fresh off fifteen minutes of parallel
compilation. I invalidated the calibration by adding load in front of it.

Raising the budget would be loosening a measured threshold to hide load I
introduced — the exact move this project forbids. Blocking the image build is the
gate's job and it outranks fifteen-minutes-faster feedback on in-process suites,
so the eight suites moved into `nightly-security-ops` (already the light
in-process nightly step) and the gate kept its calibration and its proven shape
from pipelines #5 and #6.

Coverage still goes from NEVER to nightly for all fourteen previously-unscheduled
suites; every one of the 23 now has a runner. CARGO_INCREMENTAL=0 stays: it is
correct in CI regardless, since each workflow gets a fresh 10Gi workspace and
incremental artifacts measured 11G of a 27G target tree.

Also documents an anchoring cost discovered the hard way: the resource shapes are
declared on their first consuming step (to avoid a schema-risky top-level key),
and moving the step that held `&resources-light` broke its aliases. The note now
says to check for anchor definitions before removing a step.
This commit is contained in:
jordan 2026-08-31 01:57:27 -06:00
parent 25361bb660
commit 320d640d13

View File

@ -45,49 +45,6 @@ when:
# said nothing (now fixed in tests/support/multiproc.rs).
steps:
# ── Fast in-process suites (push) ───────────────────────────────────────────
# ADDED 2026-08-31 after a survey found that 14 of 23 integration suites were
# run by NO pipeline at all. That is not a theoretical gap: `cluster_routes`
# still asserted a wire fabrication that had been removed hours earlier
# (`applied_events == 0` with a lag derived from it), and nothing caught it
# because nothing ran it. `cluster_sharding` and `vector_search` — which hold
# the dense-rank, /sharded/* opt-in and vector distance-contract gates — were
# in the same position, so the guards written for those fixes would have rotted
# the same way.
#
# These eight need neither the `cluster-e2e` feature nor spawned processes, so
# they take the LIGHT shape. Measured locally: 71s wall for all eight, 51 tests.
# Against a 6.5-minute gate that is free, and it runs FIRST so a cheap failure
# is reported before the expensive one starts.
fast-suites:
image: rust:1-bookworm
when:
event: push
backend_options: &resources-light
kubernetes:
resources:
requests: { cpu: "500m", memory: 1Gi }
limits: { cpu: "2", memory: 4Gi }
# Declared ONCE here (first consumer) and aliased by every later Rust step —
# same technique as the resource shapes, and for the same reason: a custom
# top-level key would be a schema risk, and because `when: branch: main` means
# only main triggers a pipeline, a parse error could not be caught on a
# throwaway branch and would break every push until reverted.
environment: &cargo_env
CARGO_INCREMENTAL: "0"
CARGO_TERM_COLOR: never
commands:
- apt-get update && apt-get install -y --no-install-recommends protobuf-compiler cmake clang
- cargo test -p tidaldb --lib
- cargo test -p tidal-server --test middleware
- cargo test -p tidal-server --test standalone
- cargo test -p tidal-server --test standalone_offload
- cargo test -p tidal-server --test vector_search
- cargo test -p tidal-server --test cluster_routes
- cargo test -p tidal-server --test cluster_region
- cargo test -p tidal-server --test cluster_grpc
- cargo test -p tidal-server --test reseed_install
# ── Release gate (push) ─────────────────────────────────────────────────────
# m11p8 release gate: prove a rolling upgrade under load loses no acknowledged
# write and never stalls (mp_rolling_upgrade_no_loss_no_stall — a tier-3 test
@ -138,7 +95,11 @@ steps:
# This was the ONLY push-path step without headroom, while every nightly step
# already sets it "for a shared CI runner" — and this is the step that BLOCKS
# the image build, so its flake cost is the highest of any step in the file.
environment:
environment: &cargo_env
# Declared ONCE here (first consumer) and aliased by every later Rust step.
# Incremental artifacts are pure waste in CI — each workflow gets a fresh
# 10Gi workspace PVC so nothing is ever reused, and they measured 11G of a
# 27G target tree locally.
CARGO_INCREMENTAL: "0"
CARGO_TERM_COLOR: never
TIDAL_TEST_BOOT_BUDGET_SECS: "300"
@ -156,7 +117,21 @@ steps:
# Kaniko build: single process, but it OOMKilled at the 2Gi namespace default.
# A build does not need the heavy shape's CPU floor, so it is sized separately
# to leave headroom for anything co-scheduled on the same node.
backend_options: *resources-light
# ── Resource shape: LIGHT (declared once here, aliased by later steps) ─────
# Kaniko build: single process, but it OOMKilled at the 2Gi namespace default.
#
# NOTE on anchoring: both shapes are declared on their FIRST CONSUMING STEP
# rather than a top-level `variables:` key, because the file is schema-validated
# and `when: branch: main` means a rejected key could not be caught on a
# throwaway branch — it would break every push. The cost of that choice is that
# deleting a defining step breaks its alias, which happened once on 2026-08-31
# when the step holding this anchor moved to the nightly. If you remove a step,
# check whether it defines an anchor.
backend_options: &resources-light
kubernetes:
resources:
requests: { cpu: "500m", memory: 1Gi }
limits: { cpu: "2", memory: 4Gi }
settings:
repo: tidal/server
dockerfile: docker/standalone/Dockerfile
@ -252,3 +227,29 @@ steps:
- cargo test -p tidal-server --lib
- cargo test -p tidalctl
- cargo test -p tidaldb --lib wal::compaction
# ADDED 2026-08-31. These eight were run by NO pipeline before today, which
# is how `cluster_routes` kept asserting a wire fabrication removed hours
# earlier, and how the dense-rank (`cluster_sharding`) and vector
# distance-contract (`vector_search`) guards came to sit unexecuted.
#
# They were briefly on the PUSH path and are deliberately here instead.
# Pipelines #8 and #9 proved why: running them first burns ~872s of
# CPU-saturating rustc, and the release gate immediately after then failed on
# "WAL relay alone must reconverge all three nodes to 1e-6" — the exact
# budget sensitivity this file already documents at the gate. The gate's
# 300s/180s budgets are calibrated for a node that is NOT fresh off fifteen
# minutes of parallel compilation.
#
# Raising the gate's budget to accommodate them would be loosening a measured
# threshold to hide load I introduced. Blocking the image build is the gate's
# job and it matters more than fifteen-minutes-faster feedback on in-process
# suites, so the suites moved and the gate kept its calibration. Coverage
# still went from never to nightly.
- cargo test -p tidal-server --test middleware
- cargo test -p tidal-server --test standalone
- cargo test -p tidal-server --test standalone_offload
- cargo test -p tidal-server --test vector_search
- cargo test -p tidal-server --test cluster_routes
- cargo test -p tidal-server --test cluster_region
- cargo test -p tidal-server --test cluster_grpc
- cargo test -p tidal-server --test reseed_install