# m11p9 — Continuous Correctness (COMPLETE — 2026-06-13)

Phase spec and exit gate: [docs/roadmap-to-cluster.md §4/m11p9](../../roadmap-to-cluster.md).
Closes the roadmap's **G-C (Continuous correctness)** — trust is a pipeline, not
a milestone — and wires the named owner-test for every other §2 guarantee
([guarantee-traceability.md](guarantee-traceability.md)).
Predecessors: every prior m11 phase (this phase industrializes their tier-3
suites and adds the fault classes they lacked).

**Goal:** make correctness continuous — new fault classes (disk-full, slow-fsync,
asymmetric partition) as REAL faults; the invariant checks the durability and
election gates carried inline promoted to first-class reusable checkers; a
nightly chaos + soak pipeline that fails on a regression; and a guarantee→test
matrix so the GA bar is auditable.

## Design (as adopted)

### 1. Fault injection compiled out of production (`tidal/src/fault.rs`)

Two new fault classes need REAL faults at the storage boundary, not engine flags:

- **`TIDAL_FAULT_FSYNC_DELAY_MS`** — sleep before every durable WAL fsync
  (`wal::sync_file_durable`). Models a slow disk: data still reaches stable
  storage, each group-commit fsync costs `delay + real`.
- **`TIDAL_FAULT_DISK_FULL_AFTER_BYTES`** — after N cumulative segment bytes this
  process lifetime, every segment write returns a real `ENOSPC`
  (`std::io::Error::from_raw_os_error(28)` — the errno on Linux AND macOS,
  wrapped in the same `WalError::Io` a genuine full disk produces) at
  `segment::write_batch_bytes`.

Both are behind the **non-default `fault-injection` cargo feature** (tidaldb +
a tidal-server passthrough). Production builds (`docker/standalone/Dockerfile`:
`cargo build -p tidal-server --release --locked`) never pass it, so the hooks
are **compiled out entirely** — a disk-write or fsync fault a stray env var could
trip in production is exactly the 3am footgun we refuse to ship, so the safety is
structural, not a default. Even compiled in, faults are inert until an env var
arms them (the posture of `TIDAL_HLC_SKEW_MS` / the kill-point knobs). The
`MultiProcCluster` harness's `tidal_server_bin()` builds the spawned binary with
the feature; every non-fault suite spawns the same binary and is unaffected.

The injection returns `ENOSPC` *before* the partial `write_all` (the standard
clean-fault model). The follower apply path already handles a WAL write error
correctly (proven by the suite below): the receiver halts (degraded, alive),
NEVER advances its applied frontier past the fault (no gap-swallowing), and a
restart re-pulls the suffix via the catch-up stream.

### 2. First-class invariant checkers (`tidal-server/tests/support/invariants.rs`)

The roadmap's named invariants were inline copies in three suites; m11p9 promotes
them to one audited module every suite consumes:

- **`AckLedger`** — no acknowledged-write loss: records every write the client
  saw a 2xx + `x-tidal-seq` for, then proves (A) no acked seqno exceeds the
  max-applied survivor's contiguous frontier, and (B) every acked item is present
  on the promoted/recovered leader via `/search`. Extracted verbatim from the
  m11p3 ledger gate, which now CONSUMES it (re-verified: 3/3 kill-points zero
  acked loss after the refactor).
- **`assert_feed_parity` / `feed_pairs`** — cross-replica decay/score parity to
  `1e-6` between continuously-running replicas.
- **`feed_item_ids`** — cross-replica DATA parity (the materialized item set),
  robust across a node RESTART where a velocity/rate score legitimately differs
  (its time-bucketed windowed counts are rebuilt from a burst WAL-replay rather
  than continuous accumulation, even with identical durable data — see the
  disk-full test).
- **`assert_single_leader_per_term`** — membership safety: at most one leader per
  `(shard, term)` across a set of status snapshots (reads the m11p6 `shards[]`
  rows or the flat fields). A second leader at the same term is a split brain.
- **`MonotonicCounters`** — per-entity monotonic counters: a node's applied
  frontier and the leader's commit index never move backward across observations
  (legitimate epoch resets — a reseeding node, a leader change — are forgotten,
  not flagged), so a regression means durably-acked state was lost.

### 3. New fault-class suite (`tidal-server/tests/cluster_faults.rs`)

Four tier-3 tests over real OS processes, asserting through the checkers:

- `mp_disk_full_follower_degrades_no_acked_loss` — a follower hits ENOSPC
  mid-replication, its receiver halts (degraded, alive), the healthy majority
  keeps acking quorum writes, NO acked write is lost, and a restart with space
  recovers it to full content + item-set parity.
- `mp_slow_fsync_follower_lags_but_quorum_holds` — a slow disk lags one follower;
  the fast follower supplies quorum so every write commits; the slow node
  converges once the burst ends; no loss.
- `mp_slow_fsync_both_followers_force_honest_quorum_timeout` — neither follower
  can confirm inside the budget ⇒ `ack=quorum` returns a retryable 503 naming the
  laggards (never a false 2xx) while `ack=leader` is unaffected; the followers
  recover.
- `mp_asymmetric_partition_no_split_brain_no_loss` — a follower that loses INBOUND
  links (can still send) cannot disrupt the cluster: pre-vote + check-quorum keep
  the standing leader, there is never a second leader at the same term, no acked
  write is lost, and it rejoins cleanly on heal.

The asymmetric partition reuses the harness's directed-edge proxies
(`proxied_rewrite(&["ap-south"])` + `region("ap-south").sever_all()` severs only
inbound to ap-south; its outbound dials are unproxied, so it keeps sending the
disruptive RequestVotes pre-vote neutralizes) — no new mechanism, the existing
`PartitionProxy` was already directional.

### 4. Soak + regression gates (`tidal-stress`)

`tidal-stress` gained, for the nightly soak:

- **`--json-summary <path>`** — a flat, machine-readable per-stage roll-up
  (p99 / throughput / error-rate / client-shed + the gate verdict), hand-rolled
  (no serializer dep, matching the metrics module's posture) for trend-line
  archival.
- **`--max-p99-ms`, `--max-error-pct`, `--fail-on-knee`** — PASS/FAIL regression
  gates. A breach returns `StressError::Gate` → non-zero exit (the tool always
  exited 0 before, so a regression scrolled past in green). Unarmed = the
  original informational behavior, byte-for-byte.

### 5. Nightly pipeline (`.woodpecker.yaml`) — Woodpecker, never GitHub Actions

The pipeline now carries two event-routed flows (workflow-level `when:
event: [push, cron]`, per-step `when`):

- **push** — the m11p8 rolling-upgrade release gate, then the Kaniko image build
  (unchanged).
- **cron `nightly`** — `nightly-chaos` runs every tier-3 suite serially with
  elevated kill-points (`TIDAL_QUORUM_KILLPOINTS=25`, `TIDAL_ELECTION_KILLPOINTS=15`)
  and the `fault-injection` feature (so `cluster_faults` runs); `nightly-soak`
  boots a target and runs `tidal-stress` with the regression gates, archiving the
  JSON summary. Point `TIDAL_SOAK_TARGET` at the live Ref-A cluster and raise
  `TIDAL_SOAK_SECS=3600` for the GA-bar 1-hour 100k-DAU soak.

### 6. Guarantee traceability

[guarantee-traceability.md](guarantee-traceability.md) maps every §2 guarantee
(G-D, G-A, G-S, G-E, G-Sec, G-O, G-Op, G-C) to its named automated test(s), the
fault each injects, and the checker each asserts through.

## Exit gate (from the roadmap)

- Every guarantee in §2 maps to a named automated test.
- Nightly suite green 30 consecutive days before GA.

## Exit-gate evidence (local; debug builds, real OS processes)

| Gate | Target | Measured |
|------|--------|----------|
| New fault classes pass | disk-full / slow-fsync / asymmetric | **`cluster_faults` 4/4 green** (73s): disk-full follower froze at applied 71 while node1 reached 240, recovered after restart to 120-item parity, zero acked loss; slow follower lagged 15 vs 0 then converged; both-slow → honest 503 naming `["eu-west","ap-south"]` while `ack=leader` 204'd; asymmetric → leader held us-east, single-leader-per-term, ap-south term bounded (0→0), rejoined. |
| First-class checkers faithful | extraction = no regression | `cluster_quorum::mp_quorum_ledger_zero_acked_loss_across_killpoints` refactored onto `AckLedger`: **3/3 kill-points zero acked loss** (29s) — extraction verified against the live gate. |
| Soak regression gate works | PASS exits 0, breach exits ≠0 | Against a real standalone server: a within-threshold run **exited 0** with a valid JSON summary (p99 39.7ms, 0% error); an impossible `--max-p99-ms 0.01` run **exited 1** with the breach in stderr and `"passed": false` in the JSON. |
| Every guarantee → named test | §2 coverage | [guarantee-traceability.md](guarantee-traceability.md): G-D/G-A/G-E/G-Sec/G-Op each map to a green named test; G-S to the m11p6 owner-test (in progress); G-O to the m11p8 metric/dashboard/alert artifacts; G-C to this apparatus. |
| Nightly green 30 days | calendar | **Pipeline established, not yet accrued** — this is a wall-clock criterion the cron pipeline produces over 30 nights; it cannot be completed in one session. Honestly out-of-session (like the Ref-A k3s runs). |

## Status

- [x] `fault-injection` feature (slow-fsync + disk-full WAL hooks), compiled out of production
- [x] First-class invariant checkers (`support/invariants.rs`): ledger, decay parity, single-leader-per-term, monotonic counters
- [x] `cluster_quorum` refactored onto `AckLedger` (extraction re-verified)
- [x] `cluster_faults.rs`: disk-full, slow-fsync (×2), asymmetric partition — 4/4 green
- [x] `tidal-stress` `--json-summary` + `--max-p99-ms` / `--max-error-pct` / `--fail-on-knee` + non-zero exit (verified vs a real server)
- [x] Woodpecker nightly cron pipeline (chaos suites + gated soak) beside the push release gate
- [x] Guarantee → automated-test traceability matrix
- [x] Docs (this phase, runbook chaos/soak section, monitoring soak note) + CHANGELOG + ROADMAP + roadmap-to-cluster as-built
- [ ] **Calendar:** nightly green for 30 consecutive days before GA (the pipeline accrues this; not an in-session deliverable)

## Adversarial review (6-dimension, read-only)

A read-only review fanned out over six dimensions (production-safety, checkers,
fault-tests, soak-gates, Woodpecker, doc-honesty), each finding adversarially
verified before adjudication. **10 confirmed findings, all fixed:**

- **Woodpecker (blocker):** the soak step referenced `environment:`-block vars as
  `${VAR}`, which Woodpecker's preprocessor blanks BEFORE the shell — the gate
  would have run with an empty ramp/thresholds (never executing). Fixed with the
  `$${VAR}` escape so the container shell expands them at runtime.
- **Gate threshold (blocker):** a `--max-p99-ms nan`/`inf` reported the gate ARMED
  while every `> NaN` comparison is false — silently passing a regression. Now
  rejected (finite, non-negative) at the CLI boundary.
- **JSON validity (blocker):** a non-finite `target_rps` (`--ramp "inf:30"`) emitted
  bare `inf`/`NaN`, invalid JSON. Fixed at the parse boundary + a `to_json`
  null-sanitizer (defense in depth).
- **Checkers (warnings):** `assert_single_leader_per_term` false-failed on a
  duplicate same-region snapshot (`assert_ne!` → gated `if prev != region`);
  `MonotonicCounters` now re-baselines a region's role-relative applied frontier
  on a leader↔follower flip (not just reseed); `assert_items_present` distinguishes
  a transport/non-2xx probe failure from true absence (no false ACKNOWLEDGED LOSS).
- **Coverage (warning):** added a `summary.rs` test module (gate breach/pass,
  fail-on-knee, JSON round-trip through a real parser, null-sanitize) — 6 tests.
- **Doc honesty (warnings):** the G-Sec/G-Op owner-tests (mTLS, `cluster_security`,
  `tidalctl` backup/restore, WAL-archival) were NOT in the nightly — ADDED a
  `nightly-security-ops` cron step so the "runs nightly" claim is true; the G-S
  row is marked "(NOT YET WRITTEN — m11p6 L4)" so it doesn't read as delivered.
- **Cron name (nit, external):** the nightly is silently-green until a cron named
  `nightly` is created in Woodpecker repo settings — documented in the pipeline
  header; a one-time repo-settings step.

After the fixes the fault suite re-ran **4/4 green** and the ledger gate **3/3**
(the checker changes are faithful); the slow-fsync and asymmetric tests moved to
item-SET parity (a burst catch-up reconstructs velocity differently — the same
restart-robustness rationale as disk-full).

### Formal seven-dimension review (`/review-code` → `fix-all`)

A second pass ran the `code-reviewer` seven-dimension protocol (each dimension a
distinct frame, each finding adversarially verified): **0 blockers, 0 critical, 7
warnings, 12 suggestions** (APPROVE-with-fixes). All 7 warnings + 10 of 12
suggestions fixed; final score 90/100. Notable fixes: a real diagnostic bug
(`assert_items_present`'s two guards were swapped — a true acknowledged-loss
printed "PROBE UNREACHABLE" and vice-versa); `StressError::Gate` was overloaded
across breach / invalid-threshold / IO-write and split into `Gate`/`BadGate`/
`Summary` so the operator-facing message names the real cause; the soak port
moved off iknowyou's 59521 to a `TIDAL_SOAK_PORT` (59526); the soak step now
fails fast on a boot failure and reaps its background server; doc-truthing the
"every suite consumes the shared checkers" claim (only `cluster_quorum` +
`cluster_faults` do — the legacy suites' migration is a tracked follow-up);
S=1-scope notes on the flat-field checkers; a real `from_stage` unit test; and
the `observe_cluster_frontiers` double-status-fetch removed. Two suggestions
deferred with justification: consolidating the `role`-vs-`is_leader` leadership
read into one helper (steady-state-equal, >5min, regression risk on green gates)
and Woodpecker YAML anchors for the repeated apt-get/cargo lines (untestable CI
refactor, benign duplication).

## Verification status

- Workspace `cargo fmt --all -- --check`: clean.
- `cargo clippy` `-D warnings`: clean on tidaldb (`fault-injection` on + off), tidal-server
  (`cluster-e2e fault-injection`), and tidal-stress (all targets).
- Production safety: `cargo check -p tidaldb` with the feature OFF compiles the
  fault module and both WAL hooks out entirely (the shipped binary is unchanged).
- The tree is UNCOMMITTED (continues the m11p1–p8 uncommitted tree; the user commits).