ci: give the release gate the budget headroom every nightly step already has
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
The push-path release gate (mp_rolling_upgrade_no_loss_no_stall) ran with the
compiled-in defaults of 60s boot / 30s convergence
(tidal-server/tests/support/multiproc.rs:54,62), which are tuned for a developer
machine. It spawns three real OS processes, drives a graceful SIGTERM ->
version-tagged restart -> heal cycle, then waits for three-way feed parity to 1e-6
over loopback gRPC.
Measured today: at the default budget it times out on "WAL relay alone must
reconverge all three nodes to 1e-6 after the rolling upgrade". With
TIDAL_TEST_BOOT_BUDGET_SECS=300 / TIDAL_TEST_CONVERGENCE_BUDGET_SECS=180 it passes
in 21s. So convergence is fast; the default simply leaves no slack. Reproduced
identically on a pre-change baseline (53c345e) in a separate worktree, so this is
budget sensitivity, not a regression from the vector-search or e2e work.
This was the only push-path step without headroom, while every nightly step already
sets it with the comment "Boot / convergence budgets are raised for a shared CI
runner" - and this is the step whose failure BLOCKS the Kaniko image build, so its
flake cost is the highest in the file.
Budgets are overrides, not weakened assertions: the test still demands exact
three-way parity to 1e-6 with no reconcile, and still fails if convergence stalls.
This commit is contained in:
parent
71e80ef655
commit
488aa515c5
@ -28,6 +28,24 @@ steps:
|
||||
image: rust:1-bookworm
|
||||
when:
|
||||
event: push
|
||||
# Budget headroom, matching the nightly steps below. The defaults are 60s boot
|
||||
# / 30s convergence (support/multiproc.rs:54,62), tuned for a developer
|
||||
# machine; this step spawns three real OS processes, drives a graceful
|
||||
# SIGTERM → version-tagged restart → heal cycle, and then waits for three-way
|
||||
# feed parity to 1e-6 over loopback gRPC.
|
||||
#
|
||||
# Measured 2026-08-30: at the default budget this test times out on
|
||||
# "WAL relay alone must reconverge all three nodes to 1e-6" on a loaded
|
||||
# machine, and passes in 21s with the raised budget — the convergence itself
|
||||
# is fast, the default just leaves no slack. It reproduced identically on a
|
||||
# pre-change baseline, so it is budget sensitivity and not a regression.
|
||||
#
|
||||
# This was the ONLY push-path step without headroom, while every nightly step
|
||||
# already sets it "for a shared CI runner" — and this is the step that BLOCKS
|
||||
# the image build, so its flake cost is the highest of any step in the file.
|
||||
environment:
|
||||
TIDAL_TEST_BOOT_BUDGET_SECS: "300"
|
||||
TIDAL_TEST_CONVERGENCE_BUDGET_SECS: "180"
|
||||
commands:
|
||||
- apt-get update && apt-get install -y --no-install-recommends protobuf-compiler cmake clang
|
||||
- cargo test -p tidal-server --features cluster-e2e --test cluster_lifecycle
|
||||
|
||||
Loading…
Reference in New Issue
Block a user