c97aaa8e5b
`cargo test --workspace` could not run at all: dependency resolution failed with "aws-types@1.3.16 requires rustc 1.91.1" on the 1.91.0 default toolchain, so the gate the project documents was dead. Making it run exposed a compile break and two wrong tests that had been invisible for months. Now green end to end: 143 suites, 3155 tests, exit 0. Toolchain - rust-toolchain.toml pins the DEV toolchain to 1.91.1. The published MSRV stays `rust-version = "1.91"` (the engine builds on 1.91.0); only tidalctl's AWS SDK chain needs the patch release, and it now declares that itself. Consumer crates migrated to the current engine API (clean cutover) - iknowyou-engine: `AgentPolicy` gained five m10 read/profile-override fields; the literal now spreads `..AgentPolicy::default()` as the engine's own doc example does, so future fields do not break it again. - forage-engine: `RetrieveResult` gained p1 `reasons`. The app builds its own candidate pool, so it now tags what it knows: PreferenceMatch for the preference-vector blend, SemanticMatch (with the seed item) for similar-to-saved, ExplorationBudget for pinned discoveries. - forage-engine: `url_to_item_id` folded into the u32 item universe. The engine narrows item IDs to a u32 slot in durable per-user state and rejects anything above u32::MAX rather than alias two items forever, so every add_item with a 64-bit FNV hash failed. 9 of 28 smoke tests were failing on this alone. - forage-engine: bridge items read the top-2 preference CLUSTERS via `query_vectors`, not the single centroid from `preference_vectors().get()`. Since m12 that accessor returns only the strongest cluster, so a tech+jazz user whose interests split into two clusters looked single-interest and never bridged. Falls back to top-2 dimensions when a user has one cluster. Reconcile tests corrected to the shipped contract - tidal/tests/m8p3_reconcile_production.rs asserted `3 + 5 == 8` for a windowed count after heal. `take_crdt_snapshot` deliberately keys signal contributions to ONE canonical contributor (ShardId::SINGLE) because signals are relayed from a single writer, so per-node attribution double-counted every replicated event on every reconcile. Merge is therefore LWW on (last_update_ns, score) plus PN-counter per-node max: nodes converge on the more complete accumulator. The old expectation was asserting the bug that fix removed. - Rewrote to assert convergence, count survival (not 0), and no inflation, and added `repeated_reconcile_of_converged_nodes_does_not_creep` - the regression guard for the creep itself, which nothing covered. Pre-commit hook unified - hooks/pre-commit dropped `-D warnings`: each crate's `[lints]` table is the source of truth (`clippy::all`/`unwrap_used` deny, `pedantic` warn), and the flag promoted ~58 deliberate pedantic warnings in integration tests to errors, making every Rust commit impossible. - It now lints all five tidal crates instead of path-matching `tidal/`, which silently skipped tidal-server, tidal-net, tidal-stress, tidalctl and applications/ - the rot above lived in exactly those crates. Ported the CODING_GUIDELINES file-length, println, and unsafe-SAFETY checks from the divergent untracked copy that this replaces. - CONTRIBUTING.md now documents the real commands and the toolchain/MSRV split. Fleet recovery and soak - scripts/restore-fleet.sh: the fail-closed selective restore, promoted out of an ignored tmp/ directory into the repository. Preflights retained storage, digest-pinned images, parked state, and aggregate plus per-PV-node scheduler headroom before the first scale; writes a durable transcript under tmp/restore-logs/ with structured start/error/rollback/complete events. - k8s manifests park the standalone store, the RF3 cluster, and the soak monitor at zero replicas with restore-fleet.sh as the only supported scale-up path. - soak-eval/soak-watch and the nightly CronJob fail closed on stale or missing restart evidence instead of silently skipping the restart-aware half of the gate. - docs/ops/capacity-planning.md corrects the RAM envelope to the real hot-tier formula and separates analytic totals from the measured process envelope.
257 lines
16 KiB
Markdown
257 lines
16 KiB
Markdown
# m11p9 — Continuous Correctness (COMPLETE — 2026-06-13)
|
||
|
||
Phase spec and exit gate: [docs/roadmap-to-cluster.md §4/m11p9](../../roadmap-to-cluster.md).
|
||
Closes the roadmap's **G-C (Continuous correctness)** — trust is a pipeline, not
|
||
a milestone — and wires the named owner-test for every other §2 guarantee
|
||
([guarantee-traceability.md](guarantee-traceability.md)).
|
||
Predecessors: every prior m11 phase (this phase industrializes their tier-3
|
||
suites and adds the fault classes they lacked).
|
||
|
||
**Goal:** make correctness continuous — new fault classes (disk-full, slow-fsync,
|
||
asymmetric partition) as REAL faults; the invariant checks the durability and
|
||
election gates carried inline promoted to first-class reusable checkers; a
|
||
nightly chaos + soak pipeline that fails on a regression; and a guarantee→test
|
||
matrix so the GA bar is auditable.
|
||
|
||
## Design (as adopted)
|
||
|
||
### 1. Fault injection compiled out of production (`tidal/src/fault.rs`)
|
||
|
||
Two new fault classes need REAL faults at the storage boundary, not engine flags:
|
||
|
||
- **`TIDAL_FAULT_FSYNC_DELAY_MS`** — sleep before every durable WAL fsync
|
||
(`wal::sync_file_durable`). Models a slow disk: data still reaches stable
|
||
storage, each group-commit fsync costs `delay + real`.
|
||
- **`TIDAL_FAULT_DISK_FULL_AFTER_BYTES`** — after N cumulative segment bytes this
|
||
process lifetime, every segment write returns a real `ENOSPC`
|
||
(`std::io::Error::from_raw_os_error(28)` — the errno on Linux AND macOS,
|
||
wrapped in the same `WalError::Io` a genuine full disk produces) at
|
||
`segment::write_batch_bytes`.
|
||
|
||
Both are behind the **non-default `fault-injection` cargo feature** (tidaldb +
|
||
a tidal-server passthrough). Production builds (`docker/standalone/Dockerfile`:
|
||
`cargo build -p tidal-server --release --locked`) never pass it, so the hooks
|
||
are **compiled out entirely** — a disk-write or fsync fault a stray env var could
|
||
trip in production is exactly the 3am footgun we refuse to ship, so the safety is
|
||
structural, not a default. Even compiled in, faults are inert until an env var
|
||
arms them (the posture of `TIDAL_HLC_SKEW_MS` / the kill-point knobs). The
|
||
`MultiProcCluster` harness's `tidal_server_bin()` builds the spawned binary with
|
||
the feature; every non-fault suite spawns the same binary and is unaffected.
|
||
|
||
The injection returns `ENOSPC` *before* the partial `write_all` (the standard
|
||
clean-fault model). The follower apply path already handles a WAL write error
|
||
correctly (proven by the suite below): the receiver halts (degraded, alive),
|
||
NEVER advances its applied frontier past the fault (no gap-swallowing), and a
|
||
restart re-pulls the suffix via the catch-up stream.
|
||
|
||
### 2. First-class invariant checkers (`tidal-server/tests/support/invariants.rs`)
|
||
|
||
The roadmap's named invariants were inline copies in three suites; m11p9 promotes
|
||
them to one audited module every suite consumes:
|
||
|
||
- **`AckLedger`** — no acknowledged-write loss: records every write the client
|
||
saw a 2xx + `x-tidal-seq` for, then proves (A) no acked seqno exceeds the
|
||
max-applied survivor's contiguous frontier, and (B) every acked item is present
|
||
on the promoted/recovered leader via `/search`. Extracted verbatim from the
|
||
m11p3 ledger gate, which now CONSUMES it (re-verified: 3/3 kill-points zero
|
||
acked loss after the refactor).
|
||
- **`assert_feed_parity` / `feed_pairs`** — cross-replica decay/score parity to
|
||
`1e-6` between continuously-running replicas.
|
||
- **`feed_item_ids`** — cross-replica DATA parity (the materialized item set),
|
||
robust across a node RESTART where a velocity/rate score legitimately differs
|
||
(its time-bucketed windowed counts are rebuilt from a burst WAL-replay rather
|
||
than continuous accumulation, even with identical durable data — see the
|
||
disk-full test).
|
||
- **`assert_single_leader_per_term`** — membership safety: at most one leader per
|
||
`(shard, term)` across a set of status snapshots (reads the m11p6 `shards[]`
|
||
rows or the flat fields). A second leader at the same term is a split brain.
|
||
- **`MonotonicCounters`** — per-entity monotonic counters: a node's applied
|
||
frontier and the leader's commit index never move backward across observations
|
||
(legitimate epoch resets — a reseeding node, a leader change — are forgotten,
|
||
not flagged), so a regression means durably-acked state was lost.
|
||
|
||
### 3. New fault-class suite (`tidal-server/tests/cluster_faults.rs`)
|
||
|
||
Four tier-3 tests over real OS processes, asserting through the checkers:
|
||
|
||
- `mp_disk_full_follower_degrades_no_acked_loss` — a follower hits ENOSPC
|
||
mid-replication, its receiver halts (degraded, alive), the healthy majority
|
||
keeps acking quorum writes, NO acked write is lost, and a restart with space
|
||
recovers it to full content + item-set parity.
|
||
- `mp_slow_fsync_follower_lags_but_quorum_holds` — a slow disk lags one follower;
|
||
the fast follower supplies quorum so every write commits; the slow node
|
||
converges once the burst ends; no loss.
|
||
- `mp_slow_fsync_both_followers_force_honest_quorum_timeout` — neither follower
|
||
can confirm inside the budget ⇒ `ack=quorum` returns a retryable 503 naming the
|
||
laggards (never a false 2xx) while `ack=leader` is unaffected; the followers
|
||
recover.
|
||
- `mp_asymmetric_partition_no_split_brain_no_loss` — a follower that loses INBOUND
|
||
links (can still send) cannot disrupt the cluster: pre-vote + check-quorum keep
|
||
the standing leader, there is never a second leader at the same term, no acked
|
||
write is lost, and it rejoins cleanly on heal.
|
||
|
||
The asymmetric partition reuses the harness's directed-edge proxies
|
||
(`proxied_rewrite(&["ap-south"])` + `region("ap-south").sever_all()` severs only
|
||
inbound to ap-south; its outbound dials are unproxied, so it keeps sending the
|
||
disruptive RequestVotes pre-vote neutralizes) — no new mechanism, the existing
|
||
`PartitionProxy` was already directional.
|
||
|
||
### 4. Soak + regression gates (`tidal-stress`)
|
||
|
||
`tidal-stress` gained, for the nightly soak:
|
||
|
||
- **`--json-summary <path>`** — a flat, machine-readable per-stage roll-up
|
||
(p99 / throughput / error-rate / client-shed + the gate verdict), hand-rolled
|
||
(no serializer dep, matching the metrics module's posture) for trend-line
|
||
archival.
|
||
- **`--max-p99-ms`, `--max-error-pct`, `--fail-on-knee`** — PASS/FAIL regression
|
||
gates. A breach returns `StressError::Gate` → non-zero exit (the tool always
|
||
exited 0 before, so a regression scrolled past in green). Unarmed = the
|
||
original informational behavior, byte-for-byte.
|
||
|
||
### 5. Nightly pipeline (`.woodpecker.yaml`) — Woodpecker, never GitHub Actions
|
||
|
||
The pipeline now carries two event-routed flows (workflow-level `when:
|
||
event: [push, cron]`, per-step `when`):
|
||
|
||
- **push** — the m11p8 rolling-upgrade release gate, then the Kaniko image build
|
||
(unchanged).
|
||
- **cron `nightly`** — `nightly-chaos` runs every tier-3 suite serially with
|
||
elevated kill-points (`TIDAL_QUORUM_KILLPOINTS=25`, `TIDAL_ELECTION_KILLPOINTS=15`)
|
||
and the `fault-injection` feature (so `cluster_faults` runs); `nightly-soak`
|
||
boots a target and runs `tidal-stress` with the regression gates, archiving the
|
||
JSON summary. Point `TIDAL_SOAK_TARGET` at the live Ref-A cluster and raise
|
||
`TIDAL_SOAK_SECS=3600` for the GA-bar 1-hour 100k-DAU soak.
|
||
|
||
### 6. Guarantee traceability
|
||
|
||
[guarantee-traceability.md](guarantee-traceability.md) maps every §2 guarantee
|
||
(G-D, G-A, G-S, G-E, G-Sec, G-O, G-Op, G-C) to its named automated test(s), the
|
||
fault each injects, and the checker each asserts through.
|
||
|
||
## Exit gate (from the roadmap)
|
||
|
||
- Every guarantee in §2 maps to a named automated test.
|
||
- Nightly suite green 30 consecutive days before GA.
|
||
|
||
### How the streak is computed (and what resets it)
|
||
|
||
The bar is NOT "30 green soak Job exits". It is **30 consecutive nights where the
|
||
soak passed its SLO gates AND no cluster pod restarted under load**. The two
|
||
halves come from two streams on the shared result PVC:
|
||
|
||
- `ledger.tsv` — one row per UTC date from the nightly CronJob: `PASS`/`FAIL`
|
||
on the armed gates (`--fail-on-knee`, `--max-p99-ms 150`,
|
||
`--max-error-pct 1`), plus exact `start_utc` / `end_utc` load bounds.
|
||
- `restarts.tsv` — the monitor's one-minute snapshot of each
|
||
`tidaldb-{0,1,2}` pod UID and cumulative `restartCount`.
|
||
|
||
`soak-eval` (the proven core in `tidal_stress::soak_eval`,
|
||
table-driven-tested; the binary runs in the monitor's evaluator container and
|
||
once per night in the CronJob) joins them into `streak.tsv`:
|
||
|
||
- a night is **GREEN** iff its ledger row is `PASS`, every expected pod has a
|
||
fresh sample immediately before and after the measured load window, and
|
||
neither its pod UID nor restart count changes between those samples;
|
||
- the **streak** is the trailing run of unique, consecutive UTC calendar dates;
|
||
- a gate breach, restart, pod replacement, missing/stale boundary sample,
|
||
duplicate date, or date gap cannot advance the streak. This closes both
|
||
fail-open paths: a green load exit with a recovery event and multiple rows
|
||
masquerading as multiple nights.
|
||
|
||
The evaluator emits exactly one alert per non-green transition (a durable
|
||
`ALERT-<date>.txt` the http surface serves, plus an optional `NOTIFY_WEBHOOK`
|
||
POST). **Dependency:** the streak is only *reachable* once the graceful-restart
|
||
reseed-on-rejoin churn is fixed — before that, every rollout/upgrade/node reboot
|
||
increments a restart count and would reset the streak forever. The production-
|
||
readiness divergence fix (tasks 01/02) is what makes a restart-in-window a real,
|
||
rare anomaly worth failing on.
|
||
|
||
## Exit-gate evidence (local; debug builds, real OS processes)
|
||
|
||
| Gate | Target | Measured |
|
||
|------|--------|----------|
|
||
| New fault classes pass | disk-full / slow-fsync / asymmetric | **`cluster_faults` 4/4 green** (73s): disk-full follower froze at applied 71 while node1 reached 240, recovered after restart to 120-item parity, zero acked loss; slow follower lagged 15 vs 0 then converged; both-slow → honest 503 naming `["eu-west","ap-south"]` while `ack=leader` 204'd; asymmetric → leader held us-east, single-leader-per-term, ap-south term bounded (0→0), rejoined. |
|
||
| First-class checkers faithful | extraction = no regression | `cluster_quorum::mp_quorum_ledger_zero_acked_loss_across_killpoints` refactored onto `AckLedger`: **3/3 kill-points zero acked loss** (29s) — extraction verified against the live gate. |
|
||
| Soak regression gate works | PASS exits 0, breach exits ≠0 | Against a real standalone server: a within-threshold run **exited 0** with a valid JSON summary (p99 39.7ms, 0% error); an impossible `--max-p99-ms 0.01` run **exited 1** with the breach in stderr and `"passed": false` in the JSON. |
|
||
| Every guarantee → named test | §2 coverage | [guarantee-traceability.md](guarantee-traceability.md): G-D/G-A/G-E/G-Sec/G-Op each map to a green named test; G-S to the m11p6 owner-test (in progress); G-O to the m11p8 metric/dashboard/alert artifacts; G-C to this apparatus. |
|
||
| Nightly green 30 days | calendar | **Pipeline established, not yet accrued** — this is a wall-clock criterion the cron pipeline produces over 30 nights; it cannot be completed in one session. Honestly out-of-session (like the Ref-A k3s runs). |
|
||
|
||
## Status
|
||
|
||
- [x] `fault-injection` feature (slow-fsync + disk-full WAL hooks), compiled out of production
|
||
- [x] First-class invariant checkers (`support/invariants.rs`): ledger, decay parity, single-leader-per-term, monotonic counters
|
||
- [x] `cluster_quorum` refactored onto `AckLedger` (extraction re-verified)
|
||
- [x] `cluster_faults.rs`: disk-full, slow-fsync (×2), asymmetric partition — 4/4 green
|
||
- [x] `tidal-stress` `--json-summary` + `--max-p99-ms` / `--max-error-pct` / `--fail-on-knee` + non-zero exit (verified vs a real server)
|
||
- [x] Woodpecker nightly cron pipeline (chaos suites + gated soak) beside the push release gate
|
||
- [x] Guarantee → automated-test traceability matrix
|
||
- [x] Docs (this phase, runbook chaos/soak section, monitoring soak note) + CHANGELOG + ROADMAP + roadmap-to-cluster as-built
|
||
- [ ] **Calendar:** nightly green for 30 consecutive days before GA (the pipeline accrues this; not an in-session deliverable)
|
||
|
||
## Adversarial review (6-dimension, read-only)
|
||
|
||
A read-only review fanned out over six dimensions (production-safety, checkers,
|
||
fault-tests, soak-gates, Woodpecker, doc-honesty), each finding adversarially
|
||
verified before adjudication. **10 confirmed findings, all fixed:**
|
||
|
||
- **Woodpecker (blocker):** the soak step referenced `environment:`-block vars as
|
||
`${VAR}`, which Woodpecker's preprocessor blanks BEFORE the shell — the gate
|
||
would have run with an empty ramp/thresholds (never executing). Fixed with the
|
||
`$${VAR}` escape so the container shell expands them at runtime.
|
||
- **Gate threshold (blocker):** a `--max-p99-ms nan`/`inf` reported the gate ARMED
|
||
while every `> NaN` comparison is false — silently passing a regression. Now
|
||
rejected (finite, non-negative) at the CLI boundary.
|
||
- **JSON validity (blocker):** a non-finite `target_rps` (`--ramp "inf:30"`) emitted
|
||
bare `inf`/`NaN`, invalid JSON. Fixed at the parse boundary + a `to_json`
|
||
null-sanitizer (defense in depth).
|
||
- **Checkers (warnings):** `assert_single_leader_per_term` false-failed on a
|
||
duplicate same-region snapshot (`assert_ne!` → gated `if prev != region`);
|
||
`MonotonicCounters` now re-baselines a region's role-relative applied frontier
|
||
on a leader↔follower flip (not just reseed); `assert_items_present` distinguishes
|
||
a transport/non-2xx probe failure from true absence (no false ACKNOWLEDGED LOSS).
|
||
- **Coverage (warning):** added a `summary.rs` test module (gate breach/pass,
|
||
fail-on-knee, JSON round-trip through a real parser, null-sanitize) — 6 tests.
|
||
- **Doc honesty (warnings):** the G-Sec/G-Op owner-tests (mTLS, `cluster_security`,
|
||
`tidalctl` backup/restore, WAL-archival) were NOT in the nightly — ADDED a
|
||
`nightly-security-ops` cron step so the "runs nightly" claim is true; the G-S
|
||
row is marked "(NOT YET WRITTEN — m11p6 L4)" so it doesn't read as delivered.
|
||
- **Cron name (nit, external):** the nightly is silently-green until a cron named
|
||
`nightly` is created in Woodpecker repo settings — documented in the pipeline
|
||
header; a one-time repo-settings step.
|
||
|
||
After the fixes the fault suite re-ran **4/4 green** and the ledger gate **3/3**
|
||
(the checker changes are faithful); the slow-fsync and asymmetric tests moved to
|
||
item-SET parity (a burst catch-up reconstructs velocity differently — the same
|
||
restart-robustness rationale as disk-full).
|
||
|
||
### Formal seven-dimension review (`/review-code` → `fix-all`)
|
||
|
||
A second pass ran the `code-reviewer` seven-dimension protocol (each dimension a
|
||
distinct frame, each finding adversarially verified): **0 blockers, 0 critical, 7
|
||
warnings, 12 suggestions** (APPROVE-with-fixes). All 7 warnings + 10 of 12
|
||
suggestions fixed; final score 90/100. Notable fixes: a real diagnostic bug
|
||
(`assert_items_present`'s two guards were swapped — a true acknowledged-loss
|
||
printed "PROBE UNREACHABLE" and vice-versa); `StressError::Gate` was overloaded
|
||
across breach / invalid-threshold / IO-write and split into `Gate`/`BadGate`/
|
||
`Summary` so the operator-facing message names the real cause; the soak port
|
||
moved off iknowyou's 59521 to a `TIDAL_SOAK_PORT` (59526); the soak step now
|
||
fails fast on a boot failure and reaps its background server; doc-truthing the
|
||
"every suite consumes the shared checkers" claim (only `cluster_quorum` +
|
||
`cluster_faults` do — the legacy suites' migration is a tracked follow-up);
|
||
S=1-scope notes on the flat-field checkers; a real `from_stage` unit test; and
|
||
the `observe_cluster_frontiers` double-status-fetch removed. Two suggestions
|
||
deferred with justification: consolidating the `role`-vs-`is_leader` leadership
|
||
read into one helper (steady-state-equal, >5min, regression risk on green gates)
|
||
and Woodpecker YAML anchors for the repeated apt-get/cargo lines (untestable CI
|
||
refactor, benign duplication).
|
||
|
||
## Verification status
|
||
|
||
- Workspace `cargo fmt --all -- --check`: clean.
|
||
- `cargo clippy` `-D warnings`: clean on tidaldb (`fault-injection` on + off), tidal-server
|
||
(`cluster-e2e fault-injection`), and tidal-stress (all targets).
|
||
- Production safety: `cargo check -p tidaldb` with the feature OFF compiles the
|
||
fault module and both WAL hooks out entirely (the shipped binary is unchanged).
|
||
- The tree is UNCOMMITTED (continues the m11p1–p8 uncommitted tree; the user commits).
|