ci: give the release gate the budget headroom every nightly step already has
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed

The push-path release gate (mp_rolling_upgrade_no_loss_no_stall) ran with the
compiled-in defaults of 60s boot / 30s convergence
(tidal-server/tests/support/multiproc.rs:54,62), which are tuned for a developer
machine. It spawns three real OS processes, drives a graceful SIGTERM ->
version-tagged restart -> heal cycle, then waits for three-way feed parity to 1e-6
over loopback gRPC.

Measured today: at the default budget it times out on "WAL relay alone must
reconverge all three nodes to 1e-6 after the rolling upgrade". With
TIDAL_TEST_BOOT_BUDGET_SECS=300 / TIDAL_TEST_CONVERGENCE_BUDGET_SECS=180 it passes
in 21s. So convergence is fast; the default simply leaves no slack. Reproduced
identically on a pre-change baseline (53c345e) in a separate worktree, so this is
budget sensitivity, not a regression from the vector-search or e2e work.

This was the only push-path step without headroom, while every nightly step already
sets it with the comment "Boot / convergence budgets are raised for a shared CI
runner" - and this is the step whose failure BLOCKS the Kaniko image build, so its
flake cost is the highest in the file.

Budgets are overrides, not weakened assertions: the test still demands exact
three-way parity to 1e-6 with no reconcile, and still fails if convergence stalls.
This commit is contained in:
jordan 2026-08-30 15:49:44 -06:00
parent 71e80ef655
commit 488aa515c5

View File

@ -28,6 +28,24 @@ steps:
image: rust:1-bookworm image: rust:1-bookworm
when: when:
event: push event: push
# Budget headroom, matching the nightly steps below. The defaults are 60s boot
# / 30s convergence (support/multiproc.rs:54,62), tuned for a developer
# machine; this step spawns three real OS processes, drives a graceful
# SIGTERM → version-tagged restart → heal cycle, and then waits for three-way
# feed parity to 1e-6 over loopback gRPC.
#
# Measured 2026-08-30: at the default budget this test times out on
# "WAL relay alone must reconverge all three nodes to 1e-6" on a loaded
# machine, and passes in 21s with the raised budget — the convergence itself
# is fast, the default just leaves no slack. It reproduced identically on a
# pre-change baseline, so it is budget sensitivity and not a regression.
#
# This was the ONLY push-path step without headroom, while every nightly step
# already sets it "for a shared CI runner" — and this is the step that BLOCKS
# the image build, so its flake cost is the highest of any step in the file.
environment:
TIDAL_TEST_BOOT_BUDGET_SECS: "300"
TIDAL_TEST_CONVERGENCE_BUDGET_SECS: "180"
commands: commands:
- apt-get update && apt-get install -y --no-install-recommends protobuf-compiler cmake clang - apt-get update && apt-get install -y --no-install-recommends protobuf-compiler cmake clang
- cargo test -p tidal-server --features cluster-e2e --test cluster_lifecycle - cargo test -p tidal-server --features cluster-e2e --test cluster_lifecycle