580142df49
Durable `leader_acked` frontier in `ShardReplica` tracks the highest seqno acked under `ack=leader` (journal-only, un-replicated); `decide_join` now quarantines on THIS node's own frontier rather than comparing stream numbers across stream boundaries — eliminates false-quarantine churn on rolling restarts. `SHUTDOWN_HANDOFF_WAIT` (3s) drains the leader's tail to quorum before step-down so the next leader inherits a clean prefix. New `load_leader_acked`/`persist_leader_acked` helpers; `cluster_reseed.rs` gains the divergence-fix regression suite; `replication_ops.rs` threads the signal. Soak-eval: `tidal_stress::soak_eval` + `soak-eval` binary implement the 30-night streak (ledger.tsv × restarts.tsv → streak.tsv); monitor and nightly CronJob k8s YAMLs updated; phase-9 doc clarifies the dual-stream streak definition (ledger PASS AND zero pod restarts in window). `run-reliability.sh` gates the election-divergence suite before any k8s push. Release tooling: `docker/release/` multi-stage Dockerfile + DR image; `scripts/build-release.sh` single repeatable cross-compile+buildx path.
56 lines
2.2 KiB
Bash
Executable File
56 lines
2.2 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# run-reliability.sh — the election-divergence reliability gate.
|
|
#
|
|
# Runs the cluster-mode e2e divergence suite N times SERIALLY and requires every
|
|
# iteration to be all-green. The rolling-restart-under-load false-quarantine was
|
|
# ~50% flaky, so a single green is meaningless — the bar is N consecutive passes
|
|
# (default 10). Any single failure stops the loop and prints the iteration so the
|
|
# fix can be root-caused, never masked with a wider budget.
|
|
#
|
|
# This is NOT a CI pipeline (the project bans CI/CD — see CLAUDE.md). It is a
|
|
# manual operator loop. Run it from the tidaldb workspace root.
|
|
#
|
|
# Usage:
|
|
# scripts/run-reliability.sh # 10 iterations (the gate)
|
|
# scripts/run-reliability.sh 3 # quick 3-iteration smoke
|
|
# N=20 scripts/run-reliability.sh # via env
|
|
set -euo pipefail
|
|
|
|
ITERS="${1:-${N:-10}}"
|
|
TESTS=(cluster_reseed cluster_membership)
|
|
|
|
cd "$(dirname "$0")/.."
|
|
|
|
echo "── election-divergence reliability gate: ${ITERS} consecutive serial runs ──"
|
|
echo " suite: ${TESTS[*]} (--features cluster-e2e, --test-threads=1)"
|
|
|
|
# Build once up front so per-iteration timing reflects the run, not the compile.
|
|
echo "[build] compiling the e2e test binaries…"
|
|
cargo test -p tidal-server --features cluster-e2e \
|
|
$(printf -- '--test %s ' "${TESTS[@]}") --no-run
|
|
|
|
pass=0
|
|
for i in $(seq 1 "${ITERS}"); do
|
|
start=$(date +%s)
|
|
if cargo test -p tidal-server --features cluster-e2e \
|
|
$(printf -- '--test %s ' "${TESTS[@]}") -- --test-threads=1 \
|
|
>"/tmp/reliability-iter-${i}.log" 2>&1; then
|
|
elapsed=$(( $(date +%s) - start ))
|
|
pass=$(( pass + 1 ))
|
|
echo "[iter ${i}/${ITERS}] PASS (${elapsed}s)"
|
|
else
|
|
elapsed=$(( $(date +%s) - start ))
|
|
echo "[iter ${i}/${ITERS}] FAIL (${elapsed}s) — log: /tmp/reliability-iter-${i}.log"
|
|
echo
|
|
echo "── result lines ──"
|
|
grep -E "test result|FAILED|panicked|reseed|quarantin" "/tmp/reliability-iter-${i}.log" | tail -30 || true
|
|
echo
|
|
echo "GATE FAILED after ${pass}/${ITERS} green. A single flake means the fix is"
|
|
echo "incomplete — root-cause it back into the divergence seam; do NOT widen budgets."
|
|
exit 1
|
|
fi
|
|
done
|
|
|
|
echo
|
|
echo "GATE PASSED: ${pass}/${ITERS} consecutive serial runs all-green."
|