# Two pipelines in one file, split by event (Woodpecker, NEVER GitHub Actions): # # • event: push → the m11p8 RELEASE GATE (rolling-upgrade tier-3 test) then a # Kaniko image build. Deployment is MANUAL (kustomize, orchard9-k3sf ops repo). # • event: cron → the m11p9 NIGHTLY CONTINUOUS-CORRECTNESS run: the tier-3 # chaos suites (incl. the disk-full / slow-fsync / asymmetric-partition fault # classes) with elevated kill-points, plus the security/ops owner-tests (mTLS, # backup/restore round-trip, gap-free WAL archival). A nightly failure flags a # correctness regression for that day. # # Per-step `when:` routes each step to its event; the workflow-level `when` admits # both. The cron pipeline requires a cron named "nightly" in the Woodpecker repo # settings (the same `tidal-server` binary serves standalone AND multi-process # `cluster --region`, so one image covers both deployments). # # ── Honesty note, 2026-08-30 ──────────────────────────────────────────────────── # For 216 days this file described a nightly correctness programme that had NEVER # RUN: the "nightly" cron was never created, so four pipelines existed in total # and all were push events. Tier-3 chaos, the fault classes, the soak's p99 and # error-rate gates, mTLS and the PITR test produced exactly ZERO signal, while # reading to anyone opening this file like rigorous standing coverage. A gate # nobody runs is worse than no gate, because it makes the project look covered. # # The nightly SOAK step was REMOVED rather than scheduled. It is not a capacity # quibble — it is unfalsifiable here. Measured free capacity on the best node is # 1700m CPU (agent-1; server-1/2 have 435m/550m), and the step drove 1000 rps for # 600s while gating on p99 ≤ 250ms. On that hardware the gate fails from CPU # starvation, not from a regression, so scheduling it would manufacture a nightly # false alarm — the same fake-coverage defect inverted. Its command list now lives # in docs/runbooks/nightly-soak.md as a pre-release step, run where the capacity # to make its numbers mean something actually exists. when: branch: main event: [push, cron] # CARGO_INCREMENTAL=0 for every Rust step. Incremental compilation is pure waste # here: Woodpecker gives each workflow a FRESH 10Gi workspace PVC, so nothing is # ever reused across pipelines, and the artifacts are enormous. Measured locally, # `target/debug/incremental` was 11G of a 27G target tree — 41%. # # It is also what broke pipeline #8. The 10Gi workspace held the release gate # alone (pipelines #5 and #6 both passed), but once `fast-suites` built eight test # binaries ahead of it, the gate's nested `cargo build --features fault-injection` # ran out of room and failed with its stderr discarded, which is why the failure # said nothing (now fixed in tests/support/multiproc.rs). steps: # ── Release gate (push) ───────────────────────────────────────────────────── # m11p8 release gate: prove a rolling upgrade under load loses no acknowledged # write and never stalls (mp_rolling_upgrade_no_loss_no_stall — a tier-3 test # spawning three real OS processes with a graceful SIGTERM → version-tagged # restart → heal-until-converged → fixpoint cycle). Serial (--test-threads 1): # the harness binds fixed ports and spawns real processes, so suites must not # overlap. A failure here BLOCKS the image build below — the gate, not the start. rolling-upgrade-gate: image: rust:1-bookworm when: event: push # ── Resource shape: HEAVY (declared once here, aliased by later steps) ───── # This step spawns three real tidal-server processes, each with its own WAL, # HNSW index, gRPC transport and tokio runtime. Under the namespace default # (`ci-build-bounds`: limits 1500m/2Gi, requests 50m/128Mi) a prior pipeline # pod was OOMKilled outright at 2Gi, and convergence starved past 12 min on # 1.5 CPU against 21s locally. # # CALIBRATED 2026-08-30 against real node free capacity, NOT against the # LimitRange max. `ci-build-bounds` permits up to cpu 3 / memory 6Gi, but # permission is not capacity — measured free requests were agent-1 1700m/4193Mi, # server-1 435m/2309Mi, server-2 550m/2257Mi. A `requests: cpu: "2"` step pod # (the value this file's roadmap originally specified) fits on NO node and # would sit Pending forever, which is a worse failure than being slow. # # So: requests are sized to actually schedule (fits agent-1 with ~700m CPU and # ~2.1Gi memory of slack), while limits take the full LimitRange max. Limits do # not affect scheduling, and the 6Gi limit is what eliminates the OOMKill; the # 3 CPU limit doubles the old burst ceiling. Raising the LimitRange itself is # explicitly NOT the fix here. backend_options: &resources-heavy kubernetes: resources: requests: { cpu: "1", memory: 2Gi } limits: { cpu: "3", memory: 6Gi } # Budget headroom, matching the nightly steps below. The defaults are 60s boot # / 30s convergence (support/multiproc.rs:54,62), tuned for a developer # machine; this step spawns three real OS processes, drives a graceful # SIGTERM → version-tagged restart → heal cycle, and then waits for three-way # feed parity to 1e-6 over loopback gRPC. # # Measured 2026-08-30: at the default budget this test times out on # "WAL relay alone must reconverge all three nodes to 1e-6" on a loaded # machine, and passes in 21s with the raised budget — the convergence itself # is fast, the default just leaves no slack. It reproduced identically on a # pre-change baseline, so it is budget sensitivity and not a regression. # # This was the ONLY push-path step without headroom, while every nightly step # already sets it "for a shared CI runner" — and this is the step that BLOCKS # the image build, so its flake cost is the highest of any step in the file. environment: &cargo_env # Declared ONCE here (first consumer) and aliased by every later Rust step. # Incremental artifacts are pure waste in CI — each workflow gets a fresh # 10Gi workspace PVC so nothing is ever reused, and they measured 11G of a # 27G target tree locally. CARGO_INCREMENTAL: "0" CARGO_TERM_COLOR: never TIDAL_TEST_BOOT_BUDGET_SECS: "300" TIDAL_TEST_CONVERGENCE_BUDGET_SECS: "180" commands: - apt-get update && apt-get install -y --no-install-recommends protobuf-compiler cmake clang - cargo test -p tidal-server --features cluster-e2e --test cluster_lifecycle mp_rolling_upgrade_no_loss_no_stall -- --nocapture --test-threads 1 build: image: woodpeckerci/plugin-kaniko when: event: push # ── Resource shape: LIGHT (declared once here, aliased by later steps) ───── # Kaniko build: single process, but it OOMKilled at the 2Gi namespace default. # A build does not need the heavy shape's CPU floor, so it is sized separately # to leave headroom for anything co-scheduled on the same node. # ── Resource shape: LIGHT (declared once here, aliased by later steps) ───── # Kaniko build: single process, but it OOMKilled at the 2Gi namespace default. # # NOTE on anchoring: both shapes are declared on their FIRST CONSUMING STEP # rather than a top-level `variables:` key, because the file is schema-validated # and `when: branch: main` means a rejected key could not be caught on a # throwaway branch — it would break every push. The cost of that choice is that # deleting a defining step breaks its alias, which happened once on 2026-08-31 # when the step holding this anchor moved to the nightly. If you remove a step, # check whether it defines an anchor. backend_options: &resources-light kubernetes: resources: requests: { cpu: "500m", memory: 1Gi } limits: { cpu: "2", memory: 4Gi } settings: repo: tidal/server dockerfile: docker/standalone/Dockerfile context: . # Tag by immutable identity only: `latest` plus the commit SHA (the durable # handle). A per-milestone literal tag drifts every milestone and lies about # the image's vintage (a hand-edited `m8p10` once tagged m11 code) — the SHA # never goes stale and the version-skew machinery keys off the binary's # CARGO_PKG_VERSION/BUILD_HASH, not the image tag. tags: - latest - ${CI_COMMIT_SHA} registry: registry.threesix.ai build_args: - TARGETPLATFORM=linux/amd64 extra_args: --customPlatform=linux/amd64 # ── Nightly chaos (cron) ──────────────────────────────────────────────────── # The tier-3 chaos/correctness suites over REAL OS processes, run serially # (fixed ports + spawned processes must not overlap). Kill-points are elevated # above the per-PR defaults (8/5) but kept below the 100-point exit-gate run so # the nightly stays bounded; widen TIDAL_*_KILLPOINTS for a deeper sweep. The # `fault-injection` feature compiles in the WAL slow-fsync / disk-full hooks for # cluster_faults (inert in every other suite — see tidal/src/fault.rs). Boot / # convergence budgets are raised for a shared CI runner. A failure = a # correctness regression for the night. nightly-chaos: image: rust:1-bookworm when: event: cron cron: nightly # Same three-process shape as the release gate, so the same resource shape. backend_options: *resources-heavy environment: CARGO_INCREMENTAL: "0" CARGO_TERM_COLOR: never TIDAL_QUORUM_KILLPOINTS: "25" TIDAL_ELECTION_KILLPOINTS: "15" # Raised from 120/90 to match the release gate. These budgets were TIGHTER # than the gate's on the same constrained runner while running the same # three-process suites, so the step would have failed nightly for the # identical budget-sensitivity reason measured on 2026-08-30 — not for a # correctness regression, which is the only thing this step should ever # report. TIDAL_TEST_BOOT_BUDGET_SECS: "300" TIDAL_TEST_CONVERGENCE_BUDGET_SECS: "180" commands: - apt-get update && apt-get install -y --no-install-recommends protobuf-compiler cmake clang # New m11p9 fault classes first (disk-full, slow-fsync, asymmetric partition). - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_faults -- --nocapture --test-threads 1 # The standing chaos / durability / availability / elasticity gates. - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_chaos -- --nocapture --test-threads 1 - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_quorum -- --nocapture --test-threads 1 - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_election -- --nocapture --test-threads 1 - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_membership -- --nocapture --test-threads 1 - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_reseed -- --nocapture --test-threads 1 - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_lifecycle -- --nocapture --test-threads 1 - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_runbook -- --nocapture --test-threads 1 # ADDED 2026-08-31. These six spawn real 3-process clusters and were run by # NO pipeline before today — including `cluster_sharding`, which holds the # dense-rank and /sharded/* opt-in gates, and `cluster_poison_embedding`, # the regression gate for the malformed-embedding outage. Guards nothing # runs are guards that rot: `cluster_routes` proved it by still asserting a # wire fabrication that had been removed hours earlier. # # They live in the nightly rather than the push path because each boots # three OS processes; the eight in-process suites moved to `fast-suites` # above, which costs 71s and runs on every push. - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_sharding -- --nocapture --test-threads 1 - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_poison_embedding -- --nocapture --test-threads 1 - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_cross_shard_reads -- --nocapture --test-threads 1 - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_graph_persistence -- --nocapture --test-threads 1 - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_multiproc -- --nocapture --test-threads 1 - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_e2e -- --nocapture --test-threads 1 # ── Nightly security + ops correctness (cron) ─────────────────────────────── # The G-Sec and G-Op owner-tests run nightly too (not just on-demand): mTLS / # foreign-pod rejection / cert rotation (tidal-net mtls + cluster_security), the # tidalctl backup/restore round-trip, and the gap-free WAL-archival (PITR) unit # test (in the engine lib). In-process + fast; serial for the TLS port binds. nightly-security-ops: image: rust:1-bookworm when: event: cron cron: nightly # In-process and fast; the light shape is sufficient. backend_options: *resources-light environment: *cargo_env commands: - apt-get update && apt-get install -y --no-install-recommends protobuf-compiler cmake clang - cargo test -p tidal-net --test mtls -- --test-threads 1 - cargo test -p tidal-server --test cluster_security -- --test-threads 1 - cargo test -p tidal-server --lib - cargo test -p tidalctl - cargo test -p tidaldb --lib wal::compaction # ADDED 2026-08-31. These eight were run by NO pipeline before today, which # is how `cluster_routes` kept asserting a wire fabrication removed hours # earlier, and how the dense-rank (`cluster_sharding`) and vector # distance-contract (`vector_search`) guards came to sit unexecuted. # # They were briefly on the PUSH path and are deliberately here instead. # Pipelines #8 and #9 proved why: running them first burns ~872s of # CPU-saturating rustc, and the release gate immediately after then failed on # "WAL relay alone must reconverge all three nodes to 1e-6" — the exact # budget sensitivity this file already documents at the gate. The gate's # 300s/180s budgets are calibrated for a node that is NOT fresh off fifteen # minutes of parallel compilation. # # Raising the gate's budget to accommodate them would be loosening a measured # threshold to hide load I introduced. Blocking the image build is the gate's # job and it matters more than fifteen-minutes-faster feedback on in-process # suites, so the suites moved and the gate kept its calibration. Coverage # still went from never to nightly. - cargo test -p tidal-server --test middleware - cargo test -p tidal-server --test standalone - cargo test -p tidal-server --test standalone_offload - cargo test -p tidal-server --test vector_search - cargo test -p tidal-server --test cluster_routes - cargo test -p tidal-server --test cluster_region - cargo test -p tidal-server --test cluster_grpc - cargo test -p tidal-server --test reseed_install