From 488aa515c58a0bccca8725472baef52aaa121482 Mon Sep 17 00:00:00 2001 From: jordan Date: Sun, 30 Aug 2026 15:49:44 -0600 Subject: [PATCH] ci: give the release gate the budget headroom every nightly step already has The push-path release gate (mp_rolling_upgrade_no_loss_no_stall) ran with the compiled-in defaults of 60s boot / 30s convergence (tidal-server/tests/support/multiproc.rs:54,62), which are tuned for a developer machine. It spawns three real OS processes, drives a graceful SIGTERM -> version-tagged restart -> heal cycle, then waits for three-way feed parity to 1e-6 over loopback gRPC. Measured today: at the default budget it times out on "WAL relay alone must reconverge all three nodes to 1e-6 after the rolling upgrade". With TIDAL_TEST_BOOT_BUDGET_SECS=300 / TIDAL_TEST_CONVERGENCE_BUDGET_SECS=180 it passes in 21s. So convergence is fast; the default simply leaves no slack. Reproduced identically on a pre-change baseline (53c345e) in a separate worktree, so this is budget sensitivity, not a regression from the vector-search or e2e work. This was the only push-path step without headroom, while every nightly step already sets it with the comment "Boot / convergence budgets are raised for a shared CI runner" - and this is the step whose failure BLOCKS the Kaniko image build, so its flake cost is the highest in the file. Budgets are overrides, not weakened assertions: the test still demands exact three-way parity to 1e-6 with no reconcile, and still fails if convergence stalls. --- .woodpecker.yaml | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/.woodpecker.yaml b/.woodpecker.yaml index 75c66c5..8be40ed 100644 --- a/.woodpecker.yaml +++ b/.woodpecker.yaml @@ -28,6 +28,24 @@ steps: image: rust:1-bookworm when: event: push + # Budget headroom, matching the nightly steps below. The defaults are 60s boot + # / 30s convergence (support/multiproc.rs:54,62), tuned for a developer + # machine; this step spawns three real OS processes, drives a graceful + # SIGTERM → version-tagged restart → heal cycle, and then waits for three-way + # feed parity to 1e-6 over loopback gRPC. + # + # Measured 2026-08-30: at the default budget this test times out on + # "WAL relay alone must reconverge all three nodes to 1e-6" on a loaded + # machine, and passes in 21s with the raised budget — the convergence itself + # is fast, the default just leaves no slack. It reproduced identically on a + # pre-change baseline, so it is budget sensitivity and not a regression. + # + # This was the ONLY push-path step without headroom, while every nightly step + # already sets it "for a shared CI runner" — and this is the step that BLOCKS + # the image build, so its flake cost is the highest of any step in the file. + environment: + TIDAL_TEST_BOOT_BUDGET_SECS: "300" + TIDAL_TEST_CONVERGENCE_BUDGET_SECS: "180" commands: - apt-get update && apt-get install -y --no-install-recommends protobuf-compiler cmake clang - cargo test -p tidal-server --features cluster-e2e --test cluster_lifecycle