The 2026-08-20 livelock ran 21h with no page. TidalDBClusterReseedPending is
`tidaldb_cluster_reseed_required == 1 for 10m`, written for exactly this, but the
defect cleared the marker ~200ms after each latch, so the gauge flapped 1->0
every ~30s and never held 1 for 10m. Add TidalDBClusterReseedFlapping, which
keys off `changes(...[15m]) > 2` instead of a hold duration, so a latch/clear
loop pages even when the gauge reads 0 at both ends of the window and even if a
future clear path reintroduces the spurious clear.
Also add an on-call banner for the operator intervention now in force: the
client Service selector is narrowed to keep tidaldb-0 out of the read path,
because its shard-1 frontier was cross-seeded from shard 2's snapshot artifact
and its reads are untrustworthy even at lag_events: 0. The banner carries the
footgun (the label is on the pods, not the template, so it does not survive pod
recreation) plus repair and revert commands.
`cargo test --workspace` could not run at all: dependency resolution failed with
"aws-types@1.3.16 requires rustc 1.91.1" on the 1.91.0 default toolchain, so the
gate the project documents was dead. Making it run exposed a compile break and
two wrong tests that had been invisible for months. Now green end to end:
143 suites, 3155 tests, exit 0.
Toolchain
- rust-toolchain.toml pins the DEV toolchain to 1.91.1. The published MSRV stays
`rust-version = "1.91"` (the engine builds on 1.91.0); only tidalctl's AWS SDK
chain needs the patch release, and it now declares that itself.
Consumer crates migrated to the current engine API (clean cutover)
- iknowyou-engine: `AgentPolicy` gained five m10 read/profile-override fields;
the literal now spreads `..AgentPolicy::default()` as the engine's own doc
example does, so future fields do not break it again.
- forage-engine: `RetrieveResult` gained p1 `reasons`. The app builds its own
candidate pool, so it now tags what it knows: PreferenceMatch for the
preference-vector blend, SemanticMatch (with the seed item) for
similar-to-saved, ExplorationBudget for pinned discoveries.
- forage-engine: `url_to_item_id` folded into the u32 item universe. The engine
narrows item IDs to a u32 slot in durable per-user state and rejects anything
above u32::MAX rather than alias two items forever, so every add_item with a
64-bit FNV hash failed. 9 of 28 smoke tests were failing on this alone.
- forage-engine: bridge items read the top-2 preference CLUSTERS via
`query_vectors`, not the single centroid from `preference_vectors().get()`.
Since m12 that accessor returns only the strongest cluster, so a tech+jazz user
whose interests split into two clusters looked single-interest and never
bridged. Falls back to top-2 dimensions when a user has one cluster.
Reconcile tests corrected to the shipped contract
- tidal/tests/m8p3_reconcile_production.rs asserted `3 + 5 == 8` for a windowed
count after heal. `take_crdt_snapshot` deliberately keys signal contributions
to ONE canonical contributor (ShardId::SINGLE) because signals are relayed from
a single writer, so per-node attribution double-counted every replicated event
on every reconcile. Merge is therefore LWW on (last_update_ns, score) plus
PN-counter per-node max: nodes converge on the more complete accumulator. The
old expectation was asserting the bug that fix removed.
- Rewrote to assert convergence, count survival (not 0), and no inflation, and
added `repeated_reconcile_of_converged_nodes_does_not_creep` - the regression
guard for the creep itself, which nothing covered.
Pre-commit hook unified
- hooks/pre-commit dropped `-D warnings`: each crate's `[lints]` table is the
source of truth (`clippy::all`/`unwrap_used` deny, `pedantic` warn), and the
flag promoted ~58 deliberate pedantic warnings in integration tests to errors,
making every Rust commit impossible.
- It now lints all five tidal crates instead of path-matching `tidal/`, which
silently skipped tidal-server, tidal-net, tidal-stress, tidalctl and
applications/ - the rot above lived in exactly those crates. Ported the
CODING_GUIDELINES file-length, println, and unsafe-SAFETY checks from the
divergent untracked copy that this replaces.
- CONTRIBUTING.md now documents the real commands and the toolchain/MSRV split.
Fleet recovery and soak
- scripts/restore-fleet.sh: the fail-closed selective restore, promoted out of an
ignored tmp/ directory into the repository. Preflights retained storage,
digest-pinned images, parked state, and aggregate plus per-PV-node scheduler
headroom before the first scale; writes a durable transcript under
tmp/restore-logs/ with structured start/error/rollback/complete events.
- k8s manifests park the standalone store, the RF3 cluster, and the soak monitor
at zero replicas with restore-fleet.sh as the only supported scale-up path.
- soak-eval/soak-watch and the nightly CronJob fail closed on stale or missing
restart evidence instead of silently skipping the restart-aware half of the gate.
- docs/ops/capacity-planning.md corrects the RAM envelope to the real hot-tier
formula and separates analytic totals from the measured process envelope.
- Eliminate the tidal/ self-contained doc mirror; docs now have two canonical
homes (root *.md and docs/), with planning/specs/research/reviews moved up
- Remove stale .agents/skills and .ai mirrors; canonicalize skills under .claude/
- Add pre-commit hook + scripts/check-docs.sh doc-guard + scripts/install-hooks.sh
- Implement M0-M10 seven-dimension review findings across engine, net, server,
and tidalctl (durability, replication, query, WAL, storage, CLI hardening)
