diff --git a/Cargo.lock b/Cargo.lock index 131aa3f..98a90ee 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3570,7 +3570,6 @@ dependencies = [ "prost", "rcgen", "rustls", - "rustls-pemfile", "tempfile", "thiserror 2.0.18", "tidaldb", @@ -3630,6 +3629,7 @@ dependencies = [ "proptest", "rand 0.9.2", "roaring", + "rustix 1.1.3", "serde", "serde_json", "tantivy", diff --git a/docs/reviews/M0-M10-code-review-2026-06-08-pass2.md b/docs/reviews/M0-M10-code-review-2026-06-08-pass2.md new file mode 100644 index 0000000..a863f25 --- /dev/null +++ b/docs/reviews/M0-M10-code-review-2026-06-08-pass2.md @@ -0,0 +1,1403 @@ +# tidalDB — M0–M10 Seven-Dimension Code Review (fresh pass · 2026-06-08) + +> **Scope:** the four database crates at `git HEAD` — `tidal/` (84.3k LOC), `tidal-net/` (1.9k), `tidal-server/` (5.4k), `tidalctl/` (1.4k) ≈ **93k LOC / 253 source files**, carved into **29 review zones**. +> The `applications/` consumers (`forage`, `iknowyou`) are companion products, **not** part of the M0–M10 database milestones, and were **excluded** by design. +> **Method:** the `code-reviewer` seven-dimension protocol (Completeness · Accuracy · Tech Debt · Maintainability · Extensibility · DRY · CLEAN) run as 29 parallel Opus zone reviewers, each applying all seven passes plus the Rust red-flag table and `CODING_GUIDELINES.md`, at the correctness-first "trust it at 3am" standard. **Every** BLOCKER/CRITICAL/WARNING was then handed to an independent **default-refute verifier** (Sonnet) that re-read the live source and tried to refute it; only survivors are reported. 85 agents, ~5.4M tokens. + +This is a **post-remediation re-review**. Three remediation passes already landed (see `docs/reviews/M0-M10-code-review-2026-06-07.md`, `-2026-06-08.md`, and `M0-M10-seven-dimension-review.md`). Everything below was found in the **current committed code** and survived adversarial verification; refuted items are listed for transparency. The two BLOCKERs were additionally re-confirmed by the orchestrator reading the cited source directly. + +--- + +## Remediation status: RESOLVED — all 91 actionable findings fixed (2026-06-09) + +Every actionable finding — **2 BLOCKER · 17 CRITICAL · 32 WARNING · 3 verifier-downgraded SUGGESTION · 37 standalone SUGGESTION** — was remediated at root cause. The 2 refuted findings and 58 PRAISE items needed no action. Highlights: + +- **Both BLOCKERs (session restore):** `restore_sessions()` now advances `next_session_id` past every durable snapshot/start id (no more archived-snapshot overwrite), and the durable `started_at_ns` wall clock drives both TTL-enforcement paths (sweeper + `PolicyEvaluator::check`) plus a back-dated monotonic `started_at`, so a session's age — and its TTL bound — survives a restart. Closed sessions can no longer restore as ACTIVE (storage-snapshot probe before active restore), orphaned start records are sealed into archived snapshots, and a `SessionState::new` constructor removes the dual-construction-site drift that let the bug hide. +- **CRITICAL durability/correctness:** backup now fsyncs files + dirs; the governance contributor checkpoint fsyncs per accept; USearch HNSW is wired onto the live path (size-gated, no config plumbing); cohort rescore honors the profile sort; the notification cap records only the delivered page **and** is idempotent per `(user, entity, day)`; Stage-2/creator-search id narrowing is checked; replication `finalize`/`enter_dual_write` are durable-before-observable with targeted rollback; the segment receiver and `recv_segment`/`server_terminated` surface a dead serve loop; `GrpcTransport::new` validates before building the channel; the cluster router gained timeout + concurrency limits with a bounded scatter-gather pool; and `recover()` no longer silently truncates a corrupt non-final WAL segment. +- **WARNINGs/SUGGESTIONs:** poison-recovery on the cohort cache; `is_dir`/writability probes in the builder; checked id narrowing everywhere; deterministic RRF + fusion tie-breaks; the ranking query-clock threaded through `read_*_at`; LWW merge made commutative (value tie-break); `encode_session_event` made fallible against >64 KiB fields; macOS `F_FULLFSYNC`; storage-zone tracing; and the dead `BatchConflict`/`SegmentFull`/`tenant_wal_dir`/`rustls-pemfile` removed or honestly documented. + +One genuine regression was caught and corrected during verification: the "sort replaces boosts" (spec §11.9) fix was initially applied as a blanket skip, which silently flattened the **complementary** engagement boosts on `for_you`/`following`/`related`/`notification`. It was replaced with precise per-boost **subsumption** — only a boost whose `(signal, agg, window)` the active formula sort already reads (the `trending`/`cohort_trending` velocity boosts, kept as the cohort-rescore definition) is skipped. + +**Verification (real, not wiring):** `cargo clippy -D warnings` clean across `tidaldb` + `tidal-net` + `tidal-server` + `tidalctl` (all targets, `test-utils`); **tidaldb 1997 tests / 31 binaries, tidal-server 67 / 8, tidal-net 59, tidalctl 41 — 0 failures.** ~100 regression tests were added alongside the fixes (each finding's prescribed test). Uncommitted. + +--- + +## Verdict: REQUEST_CHANGES + +The engine is in materially better shape than the prior passes — the dominant "silently-wrong-query-results" class is much reduced, durability discipline (atomic `WriteBatch`, fsync of WAL + parent dirs, recovery rebuilds) is real and consistently applied across the storage/WAL core, and the code is genuinely well-tested (1583+ lib tests, crash-property suites). But **two BLOCKERs in the session-restore path destroy or silently void durable state on a clean restart**, and the CRITICAL list is dominated by *silent* failure modes — wrong results with no error, background threads that die without a signal, and a production engine (USearch HNSW) wired into nothing. None are acceptable to ship. + +| Severity | Raw raised | After verification | +|----------|-----------:|-------------------:| +| **BLOCKER** | 2 | **2** | +| **CRITICAL** | 17 | **17** | +| **WARNING** | 32 | **32** | +| SUGGESTION (verifier-downgraded from WARNING) | — | 3 | +| Standalone SUGGESTION | 37 | 37 | +| PRAISE | 58 | 58 | +| *Refuted / false-positive* | 2 | — | + +56 non-trivial findings were raised; **54 survived** adversarial verification (2 refuted, 3 severity-downgraded to suggestion). Verifier verdicts on survivors: confirmed at reviewer severity or tightened — **0 upgrades**. + +**Confirmed findings by dimension:** Accuracy 31 · Completeness 10 · Tech Debt 5 · DRY 4 · CLEAN 2 (+ 2 cross-dimension). Accuracy-class correctness/durability defects dominate, exactly the profile expected of a database whose hardest problems are crash recovery and silent wrong-results. + +**Hotspot zones (BLOCKER+CRITICAL):** `query-executor` 3 · `tidal-net` 2 · `repl-cluster` 2 · `db-export` 2; then `db-sessions`, `db-recovery`, `session`, `governance`, `storage-vector`, `text`, `wal-recovery`, `query-search`, `repl-shipping`, `tidal-server` one each. + +--- + +## Fix-first queue + +The single highest-leverage move is to **rewrite session restore as one routine that re-derives every durable invariant** — it is the root of both BLOCKERs and two CRITICALs. + +1. **[BLOCKER]** `db/session_restore.rs:29-78` — `restore_sessions()` never advances `next_session_id`; reseeded to 1 on every open, so a clean restart re-issues id 1 and **overwrites the archived session snapshot** (silent, permanent data loss). Reviewer reproduced it: `user=1 signals=5` → `user=2 signals=0`. +2. **[BLOCKER]** `session/policy.rs:75` + `db/sweeper.rs:27` — TTL/`max_session_duration` is enforced off the **monotonic** `started_at`, which restore reseeds to `Instant::now()` (`session_restore.rs:146`). Every restart resets session age to ~0; the abuse/safety bound is silently void and expired sessions never reap. `close_session_internal` already uses the durable `started_at_ns` and documents the hazard — the two *enforcement* paths were missed. +3. **[CRITICAL]** `db/sweeper.rs:27` / `db/recovery` — same monotonic-clock root: the startup sweep that promises to reap sessions "that expired while the process was down" cannot, and a **closed session can be restored as ACTIVE** (`db-recovery`), landing the same id in both `sessions` and `closed_sessions`. +4. **[CRITICAL]** `storage/vector` — **USearch, the mandated production HNSW engine, is wired into nothing**: every deployment silently runs O(n) brute-force scans. A correctness-adjacent performance cliff hidden behind a "production" label. +5. **[CRITICAL]** `query/executor` & `query/search` — three silent wrong-results paths survive: cohort rescore discards the profile sort mode; the notification cap is recorded before pagination (over-counts deliveries the caller never sees); creator-search Category/Format filters are intersected against the *item* index and drop **all** creators. +6. **[CRITICAL]** `tidal-net` / `repl-shipping` — a dead segment-receiver thread is indistinguishable from a healthy one (the `server_terminated` probe built to detect it is never called); a follower silently stops replicating and reports success. Also `channel_capacity==0` **panics** in `GrpcTransport::new` because `validate()` runs after the channel is built. +7. **[CRITICAL]** `db/export/backup.rs` & `governance/community_ledger` — backup copy never `fsync`s files or the dest dir yet is documented "crash-consistent"; the governance contributor checkpoint (the *sole* non-WAL-rebuildable durable copy) is committed-not-fsynced while claiming a "ZERO loss window." +8. **[CRITICAL]** `wal/reader.rs:70-104` — `recover()` truncates a corrupt **non-final** segment and replays past the resulting sequence **gap** with no error surfaced. +9. **[CRITICAL]** `tidal-server/cluster.rs` — the cluster router has no concurrency limit or timeout; scatter-gather spawns **unbounded detached OS threads** per request (carried over from the prior pass). + +--- + +## Step back — structural themes (the cause under the symptoms) + +Several findings are instances of one missing abstraction. Fix the cause, not the three instances. + +- **A. Session restore does not reconstruct durable invariants.** Five findings (2 BLOCKER + 2 CRITICAL + 1 WARNING) trace to the same place: the restore path rebuilds *in-memory* session state without re-deriving what the durable record already implies — the id-allocator high-water mark, the true wall-clock age (`started_at_ns`), and closed-vs-active membership from the durable `Close` marker. The proper fix is a **single restore routine** that, after scanning all durable session sources, (a) advances `next_session_id` past every snapshot/start id, (b) back-dates `started_at` from `started_at_ns` so the existing monotonic math is correct, and (c) routes closed snapshots into `closed_sessions` *only*. Today these are three half-implemented behaviors across two functions. + +- **B. Unchecked lossy integer casts at the storage/index boundary.** A CRITICAL (`query-executor` Stage-2 intersection truncating candidate IDs with raw `as u32`) and three WARNINGs (`db-entities` `signal_with_context`, `db-recovery` `rebuild_item_indexes`, `wal-format` `encode_session_event` u16 length) are all the same bug: a 64-bit id/length narrowed with `as` instead of `try_into()`, silently aliasing or truncating once values exceed the narrow type. Centralize id-narrowing in one fallible helper that *skips with a counter* (or errors) on overflow. + +- **C. Background workers fail silently.** The replication segment-receiver and `tidal-net` `recv_segment` (CRITICAL) and the text syncer's blocking outbox under a held mutex (CRITICAL) share a gap: a worker death or stall is invisible to the foreground and to operators, violating "if you can't observe it, you can't operate it." Each long-lived worker needs a liveness latch surfaced in health/diagnostics, and a non-blocking backpressure policy. + +- **D. Documented-as-production, not on the live path.** USearch HNSW (CRITICAL), per-tenant WAL/storage isolation (`tenant_wal_dir` dead code, WARNING), and `gc_source` reclamation (WARNING) are all asserted by docs/names but not actually wired. This is doc↔impl drift that reads as "done" — the most dangerous kind, because nobody re-checks it. + +- **E. Durability claims stronger than the code.** Backup `fsync` (CRITICAL), governance contributor checkpoint (CRITICAL), `recover()` gap-on-corrupt-segment (CRITICAL), macOS `F_FULLFSYNC` and `truncate_before` dir-fsync (WARNING): the WAL/storage core fsyncs correctly, but peripheral durable writes don't, while their docs assert "crash-consistent" / "zero loss." Bring every durable-artifact path up to the WAL's own standard, and where a platform (macOS plain `fsync`) cannot meet it, scope the doc claim to where it's true. + +--- + +## Note on prior passes (regressions still open) + +The `M0-M10-code-review-2026-06-08.md` doc records "Remediation status: RESOLVED — all 142 findings remediated." This fresh pass finds that **at least three previously-reported issues are still unfixed at HEAD**, flagged as such by the reviewers: +- *Sweeper never TTL-expires restored sessions* — "raised and verified in `…2026-06-07.md` (lines 306-316) and is still unfixed at HEAD." +- *`recv_segment` cannot distinguish server death from clean shutdown* — prior CRITICAL #21. +- *Cluster router unbounded thread spawn / no timeout* — prior CRITICAL #22. + +The "RESOLVED" claim should be treated as **not fully accurate**; these belong back on the queue. (The wins are real too — the windowed-count tier-overlap double-count, the deferred-filter silent drops in RETRIEVE, and the atomic-batch durability fixes from earlier passes all hold up under this pass.) + +--- + +## How to read the findings + +Each finding below carries its verified location, the dimension it came from, the zone, the verifier's verdict, the concrete impact, and a production-quality fix. The seven-dimension **forcing-gate proof** (every zone's one-line verdict on all seven dimensions) is in the appendix. + +--- + +## BLOCKERs (2) — must fix before ship + +### [BLOCKER] Session-id allocator is not advanced past archived closed sessions on reopen — reused id silently overwrites an archived session snapshot (data loss) + +**Location:** `tidal/src/db/session_restore.rs:29-78 (restore_sessions) and tidal/src/db/mod.rs:344 / SharedDefaults seeding next_session_id = AtomicU64::new(1)` +**Dimension:** Accuracy · **Zone:** `db-sessions` · **Verifier:** CONFIRMED + +**Issue:** next_session_id is seeded to 1 on every open and is advanced ONLY by restore_session_wal_events (session_restore.rs:226-248), which advances past sessions that have a WAL Start but no Close. Cleanly-closed sessions are removed from the partition (partition_session_events removes them on Close at session_restore.rs:316-319), so they never advance the allocator. restore_sessions() (the storage scan that loads archived snapshots into closed_sessions) inserts snapshots and logs orphans but NEVER advances next_session_id. There is no durable session-id counter. Result: after a clean run that opens+closes session id 1 (snapshot durably archived at Tag::Session/b"snapshot" key for id 1) and shuts down, a reopen resets next_session_id to 1 and the next start_session re-issues id 1. close_session_internal then writes a NEW snapshot at the same key, overwriting the prior archived session, and apply_session_preference_update runs against the wrong (newly reused) session. I verified this with two real tests against a persistent data dir: (1) start/close session for user 1 (5 signals), reopen, start a new session for user 2 — the new session's id is SessionId(1), identical to the archived one; (2) after the new (user 2, 0-signal) session closes, session_snapshot(SessionId(1)) returns user_id=2, signals_written=0 — the user-1 archived snapshot is destroyed. Both assertions failed exactly as predicted (printed 'before: user=1 signals=5, after: user=2 signals=0'). + +**Why it matters:** This is silent, permanent loss of durably-archived session state — the exact 3am-incident class the durability discipline elsewhere in this zone is built to prevent. The whole point of persisting snapshots and start records (sessions.rs:87-93, the 'detect a crash during an active session at next startup' comment) is undermined: a clean restart corrupts the session archive. It also poisons cross-session preference aggregation, which keys on user_id and signaled_entities from a reused-id session. Every long-lived embedded deployment that restarts will progressively overwrite its oldest archived sessions starting from id 1. + +**Fix:** + +Make next_session_id durable across reopen by seeding it from the max id observed in ALL durable session sources, not just open WAL sessions. In restore_sessions() track a max_id while scanning: for every b"snapshot" (deserialize and read snapshot.id) AND every b"start" record (deserialize_start_record -> session_id), compute max_id = max(max_id, id). After the scan, advance the allocator with the same saturating CAS loop used in restore_session_wal_events: loop { let cur = next_session_id.load(Acquire); if max_id < cur { break; } if compare_exchange(cur, max_id.saturating_add(1), Release, Relaxed).is_ok() { break; } }. Because restore_sessions() runs in open() AFTER from_parts() (which runs restore_session_wal_events), the final allocator value must be max over both paths — the CAS-with-load-compare handles that idempotently regardless of order. Add a regression test mirroring the two I ran: assert a new session after reopen gets an id strictly greater than any previously-closed id, and that session_snapshot of the old id still returns the original user_id/signals_written. + +--- + +### [BLOCKER] Session TTL and policy-duration limits are NOT honored across a crash — restored sessions reset their age to ~zero + +**Location:** `tidal/src/session/policy.rs:75 and tidal/src/db/sweeper.rs:27 (root cause: started_at re-seeded at tidal/src/db/session_restore.rs:146)` +**Dimension:** Accuracy · **Zone:** `session` · **Verifier:** CONFIRMED + +**Issue:** Both TTL-enforcing paths measure session age from the monotonic Instant `state.started_at`: PolicyEvaluator::check computes `now.duration_since(state.started_at)` (policy.rs:75) to reject expired writes, and sweep_expired_sessions computes `now.duration_since(state.started_at)` (sweeper.rs:27) to auto-close. But on crash recovery, restore_session_wal_events re-seeds `started_at: std::time::Instant::now()` (session_restore.rs:146) because the monotonic clock is not durable — the comment there even says 'We lost the exact monotonic Instant -- approximate with now.' The durable wall-clock start, `started_at_ns`, IS restored correctly and IS used by close_session_internal (sessions.rs:163-170, which explicitly documents this exact hazard and fixes it there). So after a restart, a session that started 2 hours ago under a 1-hour max_session_duration has elapsed≈0 in BOTH the sweeper and the policy check: the sweeper will never reap it, and session_signal will keep accepting writes well past the policy limit. The team already diagnosed and fixed this monotonic-not-durable bug in close_session_internal but left the two paths that actually ENFORCE the limit on the broken clock. + +**Why it matters:** This is the literal focus item for this zone: 'TTL/lease honored across crashes.' It is not. A policy that caps an agent session at N minutes (a safety/abuse bound) is silently voided by any restart — the session's clock restarts from zero on every crash, so a long-lived or adversarial agent can evade max_session_duration indefinitely by surviving across restarts, and expired sessions accumulate in memory because the sweeper never sees them as expired. Worse, the divergence is silent: close_session reports the true ~2h age via started_at_ns, but the enforcement paths think it is brand new. There is no test covering restored-session expiry, so this regressed unnoticed. + +**Fix:** + +Make both enforcement paths measure age from the durable `started_at_ns`, exactly as close_session_internal already does. In PolicyEvaluator::check, replace `now: Instant` / `now.duration_since(state.started_at)` with a wall-clock delta: pass `now_ns: u64` (Timestamp::now().as_nanos() at the call site in session_signal) and compute `elapsed = Duration::from_nanos(now_ns.saturating_sub(state.started_at_ns))`. In sweep_expired_sessions, replace `now.duration_since(state.started_at)` with `Duration::from_nanos(Timestamp::now().as_nanos().saturating_sub(state.started_at_ns))`. Keep `started_at: Instant` only for live latency/duration metrics where monotonicity is wanted, but never for the durable TTL bound. Add a regression test (mirroring sessions.rs:636 restored_session_duration_is_age_not_time_since_restore): construct a session with started_at_ns set ~2h in the past and a fresh started_at=Instant::now(), then assert (a) sweep_expired_sessions/force_sweep closes it and (b) session_signal returns SessionExpired — proving the TTL survives a simulated restore. + +--- + + +## CRITICAL (17) — should fix before merge + +### [CRITICAL] Sweeper never TTL-expires restored sessions: age measured from a monotonic Instant that is reset to now() on restore + +**Location:** `tidal/src/db/sweeper.rs:27-41 (now.duration_since(state.started_at)); tidal/src/db/session_restore.rs:146 (started_at: Instant::now())` +**Dimension:** Accuracy · **Zone:** `db-export` · **Verifier:** CONFIRMED + +**Issue:** sweep_expired_sessions computes elapsed = now.duration_since(state.started_at). started_at is a std::time::Instant. On restart, restore_session_wal_events rebuilds each open session with started_at: Instant::now() (session_restore.rs:146, 'We lost the exact monotonic Instant -- approximate with now'), discarding the true age. The durable wall-clock anchor started_at_ns IS preserved on the restored struct (session_restore.rs:147) but is never read by the sweeper or any expiry path. Consequently a session that was already past max_session_duration at crash/shutdown time is handed a fresh full lease on every restart and is never reaped. The start_sweeper docstring (sweeper.rs:64-68, 104-106) explicitly promises the opposite: 'sessions that expired while the process was down ... are reaped promptly at startup.' That guarantee does not hold for restored sessions. This exact defect was raised and verified in docs/reviews/M0-M10-code-review-2026-06-07.md (lines 306-316) and is still unfixed at HEAD. + +**Why it matters:** GC/TTL correctness is this zone's special focus. max_session_duration is a policy bound; a crash loop or routine restart cadence lets abandoned sessions live indefinitely, leaking memory (signaled_entities, signals maps) and violating the documented session lifecycle. The bug is invisible in single-run tests and only manifests across restarts — precisely the 3am-incident failure mode. + +**Fix:** + +Make expiry use the durable wall-clock anchor. Cleanest single-path fix: in session_restore.rs back-date started_at on restore so the existing monotonic math reflects true age — let elapsed = Timestamp::now().as_nanos().saturating_sub(started_at_ns); started_at: Instant::now().checked_sub(Duration::from_nanos(elapsed)).unwrap_or_else(Instant::now). Then the unchanged sweeper computes correct elapsed for restored sessions and an over-age session is reaped on the first startup sweep. Add a test: open a session, persist, reopen with a clock/started_at_ns far in the past, force_sweep, assert the session is closed. + +--- + +### [CRITICAL] Backup copy never fsyncs files or the destination directory; a host crash can silently truncate a 'crash-consistent' backup + +**Location:** `tidal/src/db/backup.rs:57-139 (copy_dir_recursive, std::fs::copy with no sync_all) and backup.rs:273 (copy call); docstring claims at backup.rs:11-13, 144-148` +**Dimension:** Accuracy · **Zone:** `db-export` · **Verifier:** CONFIRMED + +**Issue:** copy_dir_recursive uses std::fs::copy for every file and std::fs::create_dir_all for directories, but never calls sync_all on the copied files nor fsyncs the destination directory entries. create_backup returns BackupInfo and logs 'backup completed' while the copied bytes and new dirents may still live only in the page cache. The module docstring and create_backup doc both assert the result is 'crash-consistent' and directly openable. That holds against a process crash but NOT against a host/power crash in the window after create_backup returns: files can be zero-length or missing, directory entries unlinked — a corrupt backup discovered only at restore. Note the WAL layer already fsyncs files and parent dirs everywhere (wal/segment.rs:178/281, wal/checkpoint.rs:44/54, wal/compaction.rs:150), so the backup path is the inconsistent outlier. + +**Why it matters:** A backup whose durability you cannot trust is worse than no backup — you discover it is broken at the moment you need it. Durability of backup artifacts is exactly what this zone weights heavily. The 'crash-consistent' claim in the docs is currently false for the only crash that backups exist to survive. + +**Fix:** + +After copying each file, fsync it: open the dest_path and call sync_all() (or copy then File::open(dest).sync_all()). After the whole tree is copied, fsync every directory that received entries (open the dir and sync_all on the fd, as the WAL does). Reuse the existing dir-fsync helper pattern from wal/segment.rs. Then update the docstring only if a residual gap remains. Add a fault-injection-style test or at minimum assert files are sync'd; the existing backup_captures_all_acked_signals test only proves logical completeness, not durability. + +--- + +### [CRITICAL] A closed session can be restored as ACTIVE — same session_id ends up in both self.sessions and closed_sessions (phantom state) + +**Location:** `tidal/src/db/session_restore.rs:90-251 (restore_session_wal_events) and 29-78 (restore_sessions); reachable via close ordering in tidal/src/db/sessions.rs:178-202` +**Dimension:** Accuracy · **Zone:** `db-recovery` · **Verifier:** CONFIRMED + +**Issue:** On restart there are two independent restore paths with no cross-check. restore_session_wal_events() restores every session that has a journal Start with no matching Close into self.sessions as an ACTIVE session (it never consults closed_sessions or the durable b"snapshot" row). restore_sessions() independently loads every durable b"snapshot" row into closed_sessions. close_session_internal commits the snapshot+delete-start batch FIRST (sessions.rs:186-191) and appends the journal Close SECOND (sessions.rs:197-202), explicitly tolerating a failed Close append with only a warning. The session journal (sessions.log) is append-only and NEVER compacted, so the original Start record is always still present at replay. Therefore a crash (or a failed Close append) between the snapshot batch and the Close append leaves: snapshot row present (no start row) AND Start-without-Close in the journal. Reopen then inserts the SAME session_id into closed_sessions (as ended) and into self.sessions (as active), and the active copy will accept new signals against a session the operator and the durable snapshot consider closed. CODING_GUIDELINES line 241 requires recovery to a consistent state with 'no phantom state'; this violates it. + +**Why it matters:** A cleanly-or-nearly-cleanly closed session silently resurrects as live after a crash, diverging in-memory state from the durable record of truth. Signals can be written into a session that was reported closed (wrong summaries, double-counted engagement), active_sessions() lists a session that closed_sessions also reports as ended, and there is no log or error — exactly the 3am 'why is this closed session still alive' incident. There is no test exercising the snapshot-present + Start-without-Close combination, so the divergence is currently invisible. + +**Fix:** + +Make the durable snapshot authoritative over the journal during restore. Two changes: (1) Reorder so closed snapshots are known before active restore — call restore_sessions() (or at least load the snapshot key-set) BEFORE restore_session_wal_events(), and in restore_session_wal_events skip any session_id already in closed_sessions. (2) Independently of ordering, guard the active-restore insert with a direct storage probe in restore_session_wal_events: before building SessionState, check `storage.items_engine().get(&encode_key(EntityId::new(session_id), Tag::Session, b"snapshot"))`; if a snapshot exists, the session is closed — skip it (optionally log at info) instead of inserting into self.sessions. The function already reads the b"start" record there, so the probe is one extra point-lookup on a cold path. Add a regression test: open, start+signal a session, close it, then re-append a synthetic Start-without-Close to the journal (or crash between the snapshot batch and the Close append), reopen, and assert the session is in closed_sessions and NOT in active_sessions(). + +--- + +### [CRITICAL] Per-accept contributor persist is committed-not-fsynced, but claims a 'ZERO loss window' for the SOLE non-WAL-rebuildable durable copy + +**Location:** `tidal/src/governance/community_ledger.rs:462-496 (persist_cell) and tidal/src/db/communities.rs:314-345; checkpoint at community_ledger.rs:517-565` +**Dimension:** Accuracy · **Zone:** `governance` · **Verifier:** CONFIRMED + +**Issue:** persist_cell and checkpoint commit the contributor row via storage.write_batch(batch) and return Ok with no flush()/fsync. StorageEngine::write_batch's own contract (storage/engine.rs:44-50) states 'Atomicity is NOT durability ... the bytes may still sit in an in-process memtable / OS page cache and be lost on power loss. Callers that need the batch to survive a crash MUST pair it with flush()'. The Fjall backend confirms this: write_batch ends at wb.commit() (fjall.rs:183), and only flush() issues persist(SyncAll) (fjall.rs:186-199). Yet communities.rs:314-326 asserts a 'ZERO loss window for an accepted contribution' and persist_cell's rustdoc says the cell is 'durably written ... before the caller returns Ok(true)'. The community ledger is explicitly the SOLE durable copy of per-contributor identity (the WAL community event carries none and cannot rebuild it — community_ledger.rs:504-506, confirmed: WAL replay never touches the community ledger). The fsync-durable WAL event therefore does NOT protect the attribution. On power loss / kernel panic after Ok(true) but before the next fsync barrier (only shutdown lifecycle.rs:212 or backup backup.rs:265 — the periodic maintenance loop checkpoints WITHOUT flushing, state_rebuild.rs:601-606), the contributor row is permanently lost while the WAL still reports the aggregate event. The claimed guarantee is violated. + +**Why it matters:** This is the one piece of state in the zone that cannot be rebuilt from the WAL. A power-loss crash silently drops contributor attribution — breaking M9p3 retroactive purge (the purged user's contributions can no longer be located) and M10p3 explainability/remove-by-agent, and contradicting the explicit 'zero loss' promise the API makes to callers who treated Ok(true) as durable. + +**Fix:** + +Make the per-accept durability match the claim. After the persist_cell write_batch succeeds, issue the fsync barrier on the items keyspace before returning Ok(true): follow storage.write_batch(batch)? with storage.flush()? (Fjall::flush already does rotate_only + persist(SyncAll)). To avoid an fsync per accept dominating latency, instead carry the contributor identity (user, writer_agent, epoch) in the community WAL event envelope so the ledger becomes WAL-rebuildable and the fsync-durable WAL append is the durability point — then drop persist_cell entirely. Either path closes the gap. If neither is adopted now, correct the rustdoc/comments to state the window is bounded by the next flush barrier (shutdown/backup), not zero, and add storage.flush() to the periodic maintenance loop so exposure is bounded by the checkpoint interval rather than by 'time since last clean shutdown'. + +--- + +### [CRITICAL] Cohort rescore discards profile sort mode and reported scores; silently wrong ordering for any non-boost profile + +**Location:** `tidal/src/query/executor/helpers.rs:182-267 (rescore_with_cohort), invoked from tidal/src/query/executor/mod.rs:452-468` +**Dimension:** Accuracy · **Zone:** `query-executor` · **Verifier:** CONFIRMED + +**Issue:** The main scoring path bakes the profile's Sort into ScoredCandidate.score (scoring.rs:69-143 — e.g. Sort::New scores by entity_id, Sort::MostViewed by windowed count, Sort::AlphabeticalAsc by title). rescore_with_cohort then OVERWRITES every candidate.score with a sum derived ONLY from profile.boosts, normalizes, and re-sorts strictly by descending score — never consulting profile.sort. Cohort context is wired on EVERY retrieve (query_ops.rs:104-105) and the rescore fires whenever query.cohort is set, which the public builder (types.rs:358) allows with ANY profile. For a sort-driven profile with empty boosts (e.g. the built-in `new`), cohort_score collapses to 0.0 for all candidates, normalize maps them all to 1.0, and the descending sort is a no-op — so the query returns score=1.0 for every item and the profile's New/Alphabetical/Shortest ordering is destroyed, with no error. signal_snapshot is also left reflecting the global-ledger scores, not the cohort scores reported. + +**Why it matters:** A well-formed query (cohort + any sort profile) silently returns wrong-ordered results and bogus uniform scores. This is exactly the 'cohort rescore must honor profile sort mode' failure the zone calls out — a 3am ranking-incident that looks like data corruption but is a logic gap. + +**Fix:** + +Make the cohort rescore signal-scoped, not a full re-derivation: read cohort signal values and feed them into the SAME score_by_sort / boost pipeline the main path uses (pass the cohort ledger as the signal source), so Sort modes and boosts both apply under cohort scope. At minimum, when profile.sort is metadata/ID-based or profile.boosts is empty, do not overwrite score from boosts — preserve the main-path score (or re-run score_by_sort against the cohort ledger) and keep signal_snapshot consistent. Add tests: cohort + `new` must keep descending-ID order; cohort + a boost-driven profile must rank by cohort signals. + +--- + +### [CRITICAL] Notification-cap recording runs before pagination, over-recording deliveries the caller never sees + +**Location:** `tidal/src/query/executor/pipeline.rs:430-461 (Stage 4.5) then pagination at pipeline.rs:467-479; recording in tidal/src/query/executor/helpers.rs:318-333 (apply_notification_caps notification path)` +**Dimension:** Accuracy · **Zone:** `query-executor` · **Verifier:** CONFIRMED + +**Issue:** apply_notification_caps runs on the FULL post-diversity candidate set and, for the `notification` profile, calls tracker.check_and_record (an unconditional increment of persistent per-user/per-creator/day counters — notification_tracker.rs:95-129) for every surviving candidate. Pagination (offset/limit slicing) happens AFTER, in Stage 5. So a query with limit=5 and 50 surviving candidates records 50 deliveries while returning 5 — the daily cap is consumed by 45 items the user never received on this page. Re-issuing the same query (or fetching the next page, which re-runs the whole pipeline) records them all again; check_and_record has no entity-level idempotency. + +**Why it matters:** The notification cap is persistent shared state that drives whether a user gets notified. Over-recording silently exhausts the daily budget far below the real delivery count, suppressing legitimate notifications — a correctness defect in the exact feature the cap exists to enforce, and it corrupts state that is hard to reason about after the fact. + +**Fix:** + +Record deliveries against only the items actually returned to the caller (the post-pagination page), not the full pre-slice set. Either move the recording side effect out of apply_notification_caps and apply it to the final `page` slice in Stage 5, or split check-vs-record so Stage 4.5 only filters and the record step is keyed to the delivered page. Add a test: limit smaller than surviving candidates must record exactly `limit` deliveries, and a second identical query must not double-count delivered items. + +--- + +### [CRITICAL] Stage 2 index intersection truncates candidate IDs with raw `as u32`, can wrongly retain a >u32::MAX candidate + +**Location:** `tidal/src/query/executor/pipeline.rs:291-295` +**Dimension:** Accuracy · **Zone:** `query-executor` · **Verifier:** CONFIRMED + +**Issue:** `candidates.retain(|id| bitmap.contains(id.as_u64() as u32))` performs an INCLUSION test against the FilterEvaluator's match bitmap using a raw `as u32` truncation of the candidate id. The module's own canonical helper entity_as_u32 (mod.rs:498-510) exists precisely to forbid this: per its documented polarity rule, an included id that overflows u32 must be DROPPED, because truncating `2^32 + k` to `k` aliases it onto a low slot that may be present in the match bitmap and wrongly RETAIN it. Candidates from the SignalRanked strategy are sourced straight from the ledger, which keys on full-u64 EntityId (trimmer.rs:74), so an entity id > u32::MAX is valid input that reaches this line. The SAFETY comment ('truncation is intentional and correct throughout this module') is wrong for the candidate side — every other site in this zone routes through entity_as_u32. + +**Why it matters:** A well-formed SignalRanked query with an index-backed filter (e.g. CategoryEq) can leak an item that does not satisfy the filter when its id exceeds 4.29B and aliases onto a matched low id — silently wrong results on hostile/large-id input, the same bug class the rest of the zone hardened against. + +**Fix:** + +Route the inclusion test through the checked conversion: `candidates.retain(|id| crate::query::executor::entity_as_u32(*id).is_some_and(|i| bitmap.contains(i)))`, matching the InCollection/SocialGraph arms. Correct the SAFETY comment. Add a regression test with a SignalRanked candidate at `(1u64<<32)+k` and a filter bitmap containing `k`, asserting the overflow id is dropped. + +--- + +### [CRITICAL] Creator-search CategoryEq/FormatEq filter is intersected against the item index in Stage 2, silently dropping all creators + +**Location:** `tidal/src/query/search/executor/pipeline.rs:181-182, 423-476 (apply_metadata_filter has no entity_kind guard); wired item-scoped category_index at tidal/src/db/query_ops.rs:224-229` +**Dimension:** Accuracy · **Zone:** `query-search` · **Verifier:** CONFIRMED + +**Issue:** For a SEARCH with entity_kind = Creator and a CategoryEq/FormatEq filter, execute() runs apply_metadata_filter BEFORE apply_creator_metadata_filter. apply_metadata_filter builds a FilterEvaluator over the *item-scoped* bitmap indexes (category_index/format_index, always wired from query_ops.rs regardless of entity_kind) and does candidates.retain(|id| bitmap.contains(id)). The candidates here are CREATOR ids from creator BM25/ANN retrieval, but the bitmap holds ITEM ids in that category. So creators are intersected against the wrong namespace: in a creator-only DB the item category index is empty and EVERY creator is dropped; in a mixed DB only creators whose id coincidentally aliases an item-category id survive. The Stage 2b creator-metadata post-filter (apply_creator_metadata_filter, lines 479-515) then re-checks the same filter against stored creator metadata, but it is an AND-style retain and cannot resurrect candidates Stage 2 already removed. Net effect: a documented feature (filter creators by category/format) returns zero or wrong results with no error and nothing in the logs. The m5p4 step02 test masks this — it asserts only no-panic and comments 'Results may be 0 if bitmap not populated for creators' (tidal/tests/m5p4_creator_search.rs:144-156). + +**Why it matters:** This is exactly the 3am silent-wrong-results class the review weights as CRITICAL: a caller asks for verified/jazz creators and gets an empty or arbitrary set, with no diagnostic. It also contradicts the project's own anti-pattern fix elsewhere (the W29 cross-field guard and try_eq UnknownField work were done specifically so creator metadata filters are correct) — the pre-filter stage undoes that intent for the Creator path. + +**Fix:** + +Make the Stage 2 index pre-filter item-only. In apply_metadata_filter, return early (retain all) when query.entity_kind == EntityKind::Creator, mirroring the existing guard style in apply_creator_metadata_filter (pipeline.rs:485). Thread the entity_kind into apply_metadata_filter (it already takes &Search-derived combined_filter; pass query or query.entity_kind). The creator path then relies solely on apply_creator_metadata_filter (Stage 2b) which reads the correct per-creator metadata. Add a regression test: build a DB with BOTH items (some in category 'jazz') and creators (with metadata category='news'), SEARCH creators with FilterExpr::eq('category','news'), and assert the matching creators are returned (not dropped by the item-index intersection). Note: for entity_kind = Creator, deferred variants (MinSignal/InCollection/etc.) are still no-ops in Stage 2 and handled correctly downstream, so the early-return is safe for them too. + +--- + +### [CRITICAL] finalize() swallows its checkpoint error but no later transition re-persists — a silent finalize-checkpoint failure reverts the tenant to DualWrite on crash + +**Location:** `tidal/src/replication/migration.rs:204-213 (finalize), 222-241 (gc_source), 83-97 (persist_routing)` +**Dimension:** Accuracy · **Zone:** `repl-cluster` · **Verifier:** CONFIRMED + +**Issue:** finalize() updates the router in memory (finalize_migration: clears dual-write, sets the pin) then calls self.persist_routing() — the NON-strict variant (lines 83-97) that logs and SWALLOWS a storage error. Its own comment (lines 209-211) claims this persist means 'a crash after finalization cannot revert the tenant'. That guarantee is false. If the durable write fails, disk still holds the pre-finalize {DualWrite} routing while memory holds the pin, and finalize() still returns Ok(()). The only subsequent transition, gc_source (222-241), never calls persist_routing/persist_routing_strict. So nothing re-persists. On crash + restore_migration_routing, the tenant comes back in DualWrite mode (reads route to the SOURCE shard, writes fan out to both) — the exact 'revert finalized tenant' bug this slice was built to close. The justification in persist_routing's doc ('a missed checkpoint is recoverable on the next transition') does not hold for finalize, because finalize's only next transition (gc_source) does not persist. + +**Why it matters:** finalize is the cutover that re-points every read for the tenant from source to target. A silent revert on crash means reads after recovery hit the SOURCE shard, which gc_source was about to garbage-collect — serving stale or (post-GC) empty results for that tenant with no error surfaced. This is the same class of silent mis-routing the strict-persist path in enter_dual_write was added to prevent; finalize is asymmetrically weaker on the more dangerous transition. + +**Fix:** + +Make finalize durable-before-observable like enter_dual_write: snapshot the router, apply finalize_migration in memory, then persist_routing_strict(); on Err, restore the snapshot and leave the state in DualWrite so the caller retries once storage recovers (mirror the enter_dual_write rollback at lines 164-176). If finalize must stay best-effort, then either (a) re-persist in gc_source via persist_routing_strict before flipping to Complete, or (b) correct the comment to state the revert risk explicitly. The strict-persist option is correct for a database — durability before the cutover is observable. + +--- + +### [CRITICAL] enter_dual_write rollback restores a full-router snapshot, clobbering a concurrent migration's routing for a different tenant + +**Location:** `tidal/src/replication/migration.rs:147-180 (enter_dual_write rollback), tidal/src/replication/tenant.rs:408-447 (to_checkpoint_bytes / restore_from_checkpoint_bytes)` +**Dimension:** Accuracy · **Zone:** `repl-cluster` · **Verifier:** CONFIRMED + +**Issue:** All TenantMigration instances share one Arc (db/mod.rs:288 builds a single router; each migration has only its own per-tenant Mutex). On a persist failure, enter_dual_write rolls back by calling restore_from_checkpoint_bytes(&pre_snapshot) (line 172-174), and restore_from_checkpoint_bytes (tenant.rs:433-445) does dual_write_map.clear() + shard_pin_map.clear() then reloads the WHOLE snapshot. If tenant A's enter_dual_write took its pre_snapshot, and tenant B's migration concurrently called set_dual_write / finalize_migration on the same router between A's snapshot and A's rollback, A's rollback wipes B's just-applied (and possibly already-persisted) entry. There is no global migration lock serializing distinct tenants' transitions — the only mutex is per-migration state, which does not cover the shared DashMaps. + +**Why it matters:** Multi-tenant isolation is the central promise of this zone: one tenant's migration must never corrupt another tenant's routing. Here a storage hiccup during tenant A's dual-write entry can silently revert tenant B's finalized pin or dual-write, mis-routing B's writes/reads with no error on either path. It is timing-dependent, so it passes every single-migration test and surfaces only under concurrent migrations in production — the worst kind of latent durability bug. + +**Fix:** + +Do not use a whole-router snapshot/restore for rollback. Roll back precisely the entry this migration mutated: on persist failure, call a targeted dual_write_map.remove(self.tenant_id) (and, for symmetry, restore only this tenant's prior pin if one existed) instead of clear-and-reload. Capture just this tenant's prior (dual_write, pin) entries before set_dual_write and restore exactly those. Alternatively, serialize all router-mutating migration transitions under a single coordinator-level lock so snapshot/restore is atomic with respect to other tenants — but the targeted-remove fix is cheaper and avoids the global lock on a cold path. + +--- + +### [CRITICAL] Segment-receiver thread death mid-life is silent: no health latch, indistinguishable from a still-running receiver until shutdown + +**Location:** `tidal/src/replication/receiver.rs:87-137 (spawn_receiver, halt path 117-129); tidal/src/db/diagnostics.rs:48-90 (health_check); tidal/src/db/replication_ops.rs:23-40 (start_replication)` +**Dimension:** Accuracy · **Zone:** `repl-shipping` · **Verifier:** CONFIRMED + +**Issue:** When the receiver loop hits a corrupt segment or a follower-WAL IO fault, apply_payload_with_leader_seq returns Err and the thread returns that Err and exits (receiver.rs:128). That error is ONLY retrievable via SegmentReceiverHandle::join() (receiver.rs:61), which the node calls exclusively at shutdown (db/lifecycle.rs:62-67). While the node stays open, nothing observes the death: health_check (diagnostics.rs:48-90) consults `closed`, `checkpoint_thread.is_finished()` (latching checkpoint_thread_died), `metrics.is_degraded()`, and `text_index_unhealthy()` — but never the receiver. SegmentReceiverHandle exposes no is_finished()/healthy() accessor at all (receiver.rs:49-70). Worse, the leader-HWM update lives INSIDE the same apply path that just died (receiver.rs:108-116 → apply_payload_with_leader_seq → gauge.update_leader_seqno at receiver.rs:287), so once the receiver is dead BOTH applied_seqno and leader_seqno freeze and lag_segments() (lag.rs:56-60) stays flat instead of growing. A partitioned/desynced follower whose receiver halted therefore reports zero lag and a healthy node while silently applying nothing — exactly the 'server-loop death must be distinguishable from clean shutdown' failure, invisible at 3am. recv_segment()==None (clean shutdown) and a halt-on-Err both terminate the thread; only the Err return distinguishes them, and only at join(). + +**Why it matters:** Replication exists so a follower can be promoted with the freshest state. A follower whose receiver died on one corrupt segment will be promoted believing it is caught up, silently losing every event the leader shipped after the halt. The whole point of multi-node durability is defeated by an unobservable stalled receiver. The codebase ALREADY proves the correct pattern is expected here: the checkpoint thread latches metrics.checkpoint_thread_died via a panic supervisor + JoinHandle::is_finished() and folds it into health_check (db/mod.rs:113-120, diagnostics.rs:57-70), and the text syncer latches text_index_unhealthy the same way. The receiver is the one durability-critical background loop without this. + +**Fix:** + +Give the receiver the same liveness latch as the checkpoint thread. (1) Add an Arc `replication_receiver_died` to MetricsState (or TidalDb), passed into spawn_receiver. On the Err halt path (receiver.rs:124-128) and at any unexpected thread exit, set it true before returning; on the clean recv_segment()==None path (receiver.rs:97-100) leave it false. (2) Alternatively/additionally expose SegmentReceiverHandle::is_finished(&self)->bool and have health_check inspect the receiver_handle guard with is_finished() exactly like checkpoint_thread (diagnostics.rs:60-70), latching the flag. (3) Fold the latch into health_check's degraded decision and the HTTP/metrics health surface so a dead receiver returns degraded. (4) Add a test: feed one corrupt segment, assert the receiver halts AND that health_check returns Err / the latch is set WITHOUT closing the DB (the existing receiver_thread_exits_on_corrupt_payload only asserts via join() at teardown). This makes a stalled follower loud and bounded instead of silently frozen. + +--- + +### [CRITICAL] USearch (the mandated production HNSW engine) is wired into nothing — every deployment does O(n) brute-force scans + +**Location:** `tidal/src/storage/vector/registry.rs:384 (rebuild), tidal/src/db/users.rs:144 (write auto-register), tidal/src/query/search/executor/pipeline.rs:803; UsearchIndex only constructed in tidal/src/storage/vector/usearch_index.rs tests and tidal/tests/vector_usearch.rs` +**Dimension:** Tech Debt / Extensibility · **Zone:** `storage-vector` · **Verifier:** CONFIRMED + +**Issue:** CODING_GUIDELINES §4 states 'USearch is the HNSW engine ... 126K+ QPS' and §8 sets the benchmark target 'ANN retrieval at 1M vectors <10ms p99'. But grep confirms UsearchIndex::new appears only in test code. Every production embedding-slot instantiation — the rebuild-from-store path, the write-path auto-register in users.rs, and the query pipeline — hardcodes Box::new(BruteForceIndex::new(config)). There is no schema/config selector that ever picks UsearchIndex. The entire usearch_index.rs module (save/load/view/tombstone/total_slots) is dead in production: correct, fully tested, and unreachable. + +**Why it matters:** BruteForceIndex computes l2_distance_sq against every stored vector under an RwLock read on every search. At the spec's 1M-vector / 1536D scale that is ~1.5e9 float ops plus a full O(n) collect per query — orders of magnitude past the <10ms p99 budget, and it serializes writes against searches on one RwLock. Results are exact (no correctness bug), but the database silently cannot meet its stated scale or latency contract; an operator reading the docs would believe HNSW is active. This is the kind of gap that surfaces as a production latency incident at 3am with no code-level signal that brute force is the cause. + +**Fix:** + +Add a backend selector so EmbeddingSlotState is built from UsearchIndex in persistent/production mode and BruteForceIndex only for tests/tiny slots. Concretely: introduce a `VectorBackend` enum (or a factory closure) on the registry/config, default it to USearch when a real data dir is configured, and route registry.rs:384, users.rs:144, and the query pipeline through that factory instead of the hardcoded `Box::new(BruteForceIndex::new(config))`. Gate the choice on slot size or an explicit schema flag if you want brute force for >>`) and then calls the BLOCKING `tx.send(...)` on a channel that `open_text_syncer` creates as `crossbeam::channel::bounded(WRITE_OUTBOX_CAPACITY=16_384)`. The syncer's `run()` is designed to retry a failing commit FOREVER (syncer.rs:124-128 'keeps retrying regardless'). So on a persistent commit outage (disk full, corrupt segment, fsync EIO) the syncer stops draining, the 16K buffer fills, and `send` blocks indefinitely — while the `text_tx` MutexGuard is still held. Every concurrent `write_item_with_metadata` then blocks on `self.text_tx.lock()` too. A degraded-text-search incident (which the design explicitly says must be tolerated: 'text index is derived state... best-effort') is thereby escalated into a full stall of the durable item write API. + +**Why it matters:** This inverts the core design contract that the text index is best-effort derived state that must never gate durable writes (CODING_GUIDELINES Sec 5; spec 06 axiom 3). The durable entity-store write has already completed by line 144, so blocking here adds zero durability — it only removes availability. At 3am a single full disk on the Tantivy directory would freeze all item ingestion across the process, and the cause (a held mutex around a blocking send) is non-obvious from the call site. The bounded-channel doc comment even acknowledges backpressure 'stalls' the producer but does not account for that stall happening inside a global mutex critical section. + +**Fix:** + +Do not hold the mutex across a blocking send, and do not let a derived-index outage block durable writes. Two-part fix: (1) clone the `Sender` out of the guard and drop the guard before sending — `let tx = self.text_tx.lock().ok().and_then(|g| g.clone()); if let Some(tx) = tx { ... }` (crossbeam `Sender` is cheap to clone). (2) Use `try_send` (or `send_timeout` with a short bound) instead of blocking `send`, and on `Full`/`Timeout` log a warn and drop the enqueue — the unconditional rebuild-from-store at open re-indexes anything dropped, so correctness is preserved while the write path stays non-blocking. The text_syncer.rs comment already contemplates 'or, at the call site, dropping' — implement that drop rather than relying on a blocking send under lock. + +--- + +### [CRITICAL] recv_segment cannot distinguish server death from clean shutdown; the probe built to detect it (server_terminated) is never called + +**Location:** `tidal-net/src/transport.rs:202-213 (server_terminated), :244-286 (recv_segment); tidal/src/replication/receiver.rs:96-100 (consumer)` +**Dimension:** Completeness · **Zone:** `tidal-net` · **Verifier:** CONFIRMED + +**Issue:** When the gRPC serve loop fails AFTER startup (listener dies, TLS handshake task panics, reactor torn down), the spawned server task ends, its inbound_tx sender drops, the inbound channel closes, and recv_segment returns None. The receiver thread (receiver.rs:97-100) treats None as a clean shutdown: it logs debug! 'transport closed, shutting down' and returns Ok(()). A follower whose receive side has silently died is therefore indistinguishable from one that was intentionally shut down — exactly the failure mode the zone's special-focus calls out. server_terminated() was built to surface this ('Health checks and the cluster control plane poll this so a follower whose receive side has silently died is observable, not a black hole') but grep across tidal-net, tidal-server, and tidal confirms it has ZERO callers. The serve task's terminal Err is logged at error! (server.rs:192-193), which helps a human reading logs, but no code path demotes the node or fails the receiver — at 3am the cluster control plane still believes the follower is healthy while replication has silently stopped. + +**Why it matters:** Silent replication stall on a follower whose serve loop died: the leader keeps shipping (sends may even succeed against a half-open connection until keep-alive trips), the follower's receiver thread has exited Ok, and nothing observes it. This is a durability-relevant blind spot — divergence accumulates with no signal. + +**Fix:** + +Wire server_terminated() into the liveness story so the two None causes are distinguishable. Concretely: (1) have the cluster control plane / health endpoint poll transport.server_terminated() and demote/alert the node when it flips true (it is already #[must_use] and ready); OR (2) make the distinction explicit at the receiver boundary — e.g. on recv_segment returning None, the receiver checks server_terminated() and, if the server died WITHOUT a shutdown_receivers() request, returns an Err(WalError) via SegmentReceiverHandle::join instead of Ok(()) so a dead serve loop surfaces as a failed receiver rather than a clean exit. Add an integration test that fails the serve task (e.g. drop/abort it) and asserts the follower observes server_terminated()==true (and/or the receiver surfaces an error), distinct from the shutdown_receivers() clean-exit case already covered by shutdown_unparks_recv_segment_and_thread_exits. + +--- + +### [CRITICAL] channel_capacity==0 panics in GrpcTransport::new because validate() runs after the mpsc channel is built + +**Location:** `tidal-net/src/transport.rs:138 then :142-146; tidal-net/src/config.rs:101-106 (validate); tidal-net/src/client.rs:37 (only validate() callsite)` +**Dimension:** Accuracy · **Zone:** `tidal-net` · **Verifier:** CONFIRMED + +**Issue:** GrpcTransport::new builds the inbound channel with `mpsc::channel(config.channel_capacity)` at transport.rs:138, then enters runtime.block_on where it calls start_server (:143) and PeerPool::new (:144). config.validate() — the only thing that rejects channel_capacity==0 — is invoked inside PeerPool::new (client.rs:37), i.e. AFTER the channel is already constructed. tokio's bounded channel asserts `buffer > 0` (tokio-1.49 bounded.rs:157), so GrpcTransport::new(cfg) with channel_capacity:0 panics with 'mpsc bounded channel requires buffer > 0' before validate() ever runs. The config doc at config.rs:82-85 explicitly promises the opposite: 'surface it as a typed error here instead of a panic deep in GrpcTransport::new.' The existing test rejects_invalid_config only exercises PeerPool::new directly, so it never hits the real constructor ordering and the panic is masked. + +**Why it matters:** A buggy or operator-supplied config panics the database's transport constructor instead of returning the typed Internal error the design guarantees. CODING_GUIDELINES §7: 'Panics in a database corrupt state.' The whole point of validate() (a defensive guard against a known panic) is defeated by call ordering. + +**Fix:** + +Call config.validate() at the very top of GrpcTransport::new, before mpsc::channel and before entering the runtime — e.g. add `config.validate()?;` immediately after ensure_crypto_provider() at transport.rs:129. Then either keep the validate() in PeerPool::new (idempotent, cheap) or document that the transport validates up front. Add a test that calls GrpcTransport::new (not PeerPool::new) with channel_capacity:0 and asserts Err(Internal), not a panic — guarding the real constructor path the current test misses. + +--- + +### [CRITICAL] Cluster router has no concurrency limit or timeout — scatter-gather spawns unbounded detached OS threads per request + +**Location:** `tidal-server/src/cluster.rs:578-615 (build_cluster_router) and tidal-server/src/scatter_gather.rs:309-331 (dispatch_shards)` +**Dimension:** Accuracy · **Zone:** `tidal-server` · **Verifier:** CONFIRMED + +**Issue:** The standalone router wraps protected routes in `TimeoutLayer` + `ConcurrencyLimitLayer::new(MAX_CONCURRENCY=100)` (router.rs:113-120), but `build_cluster_router` applies neither — it only adds a body limit and optional auth, then `public.merge(protected)` with no ServiceBuilder. So the cluster surface has no in-flight cap and no request timeout. Compounding this, every `/sharded/feed` and `/sharded/search` calls `dispatch_shards`, which spawns one *detached* `std::thread` per live shard per request (scatter_gather.rs:313-320). With no concurrency limit in front, a burst of sharded queries spawns shards×concurrent-requests detached OS threads with no global ceiling. The detached threads also outlive the request on a slow shard (by design), so threads accumulate faster than they retire. This is the exact 'no unbounded thread spawn' hazard called out for this zone, and it is on the un-throttled path. + +**Why it matters:** At 3am under a query storm, the cluster node spawns threads without bound and never times out a wedged request — the node exhausts OS threads/memory and falls over instead of shedding load. The standalone node degrades cleanly (429/408) under the same load; the cluster node does not. The per-shard thread spawn already has a local degrade-on-spawn-failure path, but nothing caps the aggregate. + +**Fix:** + +Mirror the standalone middleware on the cluster protected router: wrap it in `ServiceBuilder::new().layer(TimeoutLayer::with_status_code(REQUEST_TIMEOUT, ...)).layer(ConcurrencyLimitLayer::new(MAX_CONCURRENCY))` before merging public routes. Promote `REQUEST_TIMEOUT_SECS`/`MAX_CONCURRENCY`/`BODY_LIMIT_BYTES` to a shared module so both routers consume the same constants. Additionally bound scatter-gather fan-out: replace the per-request detached `std::thread::spawn` with a shared bounded worker pool (the `ClusterWritePool` pattern already in offload.rs) or at minimum a semaphore so total concurrent shard workers are capped regardless of request concurrency. + +--- + +### [CRITICAL] recover() silently truncates a corrupt non-final segment and replays past the resulting sequence gap with no error + +**Location:** `tidal/src/wal/reader.rs:70-104 (the per-segment loop) and reader.rs:287-306 (recover_segment)` +**Dimension:** Accuracy · **Zone:** `wal-recovery` · **Verifier:** CONFIRMED + +**Issue:** recover() calls recover_segment(seg_path)? for EVERY segment in seq order. recover_segment unconditionally truncates a torn/corrupt tail (last_valid_offset < total_len) on whatever segment it is handed, then recover() continues to the next segment and keeps appending its events into all_events / advancing next_seq. The doc-comment on recover_segment (lines 280-286) assumes only the active/last segment can be torn, but the loop applies it to every segment with no check that the corruption was in the final segment and no cross-segment sequence-continuity check. A sealed (already rotated+synced, per segment.rs:269-271) non-final segment that suffers silent bit-rot in the middle will: (1) have its tail truncated and discarded, (2) have all later segments replayed anyway, producing a state that is missing a contiguous run of sequences with a GAP, and (3) return Ok(RecoveryResult) from recover() with nothing surfaced — Wal::open (mod.rs:204) logs only 'recovery complete'. This is exactly the asymmetry the session journal already guards against (session_journal.rs:24-42, mod.rs:221-231 log corruption_detected), but the signal WAL — the actual source of truth for every entity/signal/relationship write — does not. + +**Why it matters:** This is silent data loss on corrupt input in the durability-critical path. At 3am, an operator who lost the tail of a sealed segment to bit-rot gets a clean 'recovery complete' log, a database that is silently missing events, and a non-contiguous sequence space — with no signal that anything was dropped. The two-phase BLAKE3 validation correctly DETECTS the corruption; the bug is that recovery's RESPONSE (truncate + continue past it, return Ok) is only correct when the corruption is the torn tail of the final segment. For any earlier segment it converts detected corruption into invisible loss. + +**Fix:** + +Make recover() track whether each segment's scan was tail-corrupt and whether it is the final segment. Replace the blanket recover_segment(seg_path)? call with logic that: (a) for the LAST segment, truncates the torn tail as today (legitimate crash-mid-append); (b) for any NON-final segment that scans tail_corrupt, do NOT silently truncate-and-continue — instead either return WalError::Corruption (mirroring the sequence-overflow handling at lines 84-89) so open fails loudly and an operator can intervene, or at minimum record a corruption flag on RecoveryResult and have Wal::open emit a tracing::error! exactly as the session-journal path does (mod.rs:221-231). Concretely: use scan_segment_summary (which already exposes tail_corrupt without truncating) inside recover() to detect the condition before deciding to truncate, and add a cross-segment check that next_seq from segment N equals header.first_seq of segment N+1's first batch; surface a gap as Corruption. Add a recovery test that writes two segments, corrupts the FIRST one's tail, and asserts recover() either errors or sets a corruption signal — not that it silently returns the second segment's events. + +--- + + +## WARNING (32) — real issues, can merge but fix soon + +### [WARNING] Cohort resolver cache mutex uses .expect() (panic) on the signal-write hot path instead of the codebase's poison-recovery pattern + +**Location:** `tidal/src/cohort/resolver.rs:77, 95, 109, 132, 145, 154` +**Dimension:** Accuracy · **Zone:** `cohort-load-misc` · **Verifier:** CONFIRMED + +**Issue:** Six production (non-test) call sites lock the LRU cache with `.lock().expect("cohort resolver cache poisoned")`. CohortResolver::resolve is reached on every cohort-attributed signal write (db/signals.rs:542). If any thread ever panics while holding this lock, the mutex is poisoned and every subsequent resolve/invalidate/cache_len call panics too — converting a one-off panic into a cascading, write-path-wide outage. This violates CODING_GUIDELINES §7 ("no expect() outside of tests and initialization; panics in a database corrupt state") and is inconsistent with the rest of the crate, which recovers from poison via unwrap_or_else(PoisonError::into_inner) (testing/faults.rs:54-55, testing/cluster.rs:308-309, and many db/ sites). The Debug impl here already does the right thing with try_lock; the hot-path methods do not. + +**Why it matters:** At 3am, a single panic in any future cohort code path (or an allocation failure under memory pressure) would poison this lock and take down all cohort-attributed signal writes cluster-wide, not just the one request that tripped it. The cache is explicitly a pure accelerator (a miss is recomputed), so a poisoned lock has zero correctness value to defend — recovering into_inner is strictly safe and is already the house pattern. + +**Fix:** + +Replace each `.lock().expect("cohort resolver cache poisoned")` with `.lock().unwrap_or_else(std::sync::PoisonError::into_inner)`, matching testing/faults.rs and the db/ modules. The cache's critical sections are short and panic-free, so resuming with the recovered guard cannot observe a torn LRU; a future panic-during-resolve then degrades to a cache miss instead of a write-path outage. Update the `# Panics` doc sections to state the methods no longer panic on poison. + +--- + +### [WARNING] write_signal reconstructs the WAL payload inline instead of calling the single_event_payload helper built to prevent exactly this drift + +**Location:** `tidal/src/testing/cluster.rs:428-434 vs tidal/src/testing/cluster_transport.rs:95-107` +**Dimension:** DRY · **Zone:** `cohort-load-misc` · **Verifier:** CONFIRMED + +**Issue:** single_event_payload (cluster_transport.rs:95) was extracted with a doc comment stating its purpose: "Both the eager ship path (SimulatedCluster::write_signal) and the recovery path (redeliver_missed) emit exactly the same payload shape ... Sharing the construction here keeps the two sites from drifting on the load-bearing event_count/leader_last_seq invariants (the receiver's monotonic advance relies on leader_last_seq == seqno)." redeliver_missed calls it; write_signal does not — it hand-builds WalSegmentPayload { id, bytes, event_count: 1, leader_last_seq: seqno } inline. The two payload constructions can now silently diverge on the very invariants the helper exists to lock together. + +**Why it matters:** This is the harness that validates crash/replication correctness; if a future edit changes the payload shape in one site (e.g. multi-event batches, a different leader_last_seq derivation) and not the other, replication convergence in tests would break or — worse — pass against a subtly wrong payload, masking a real receiver bug. The helper's own doc claims both sites share it; the code contradicts the doc. + +**Fix:** + +In write_signal, replace the inline `let payload = WalSegmentPayload { id: WalSegmentId::new(crate::replication::RegionId::SINGLE, leader_shard, seqno), bytes: bytes.clone(), event_count: 1, leader_last_seq: seqno };` with `let payload = single_event_payload(leader_shard, seqno, bytes.clone());` (importing single_event_payload alongside the other cluster_transport imports). This makes the helper's drift-prevention guarantee actually true. + +--- + +### [WARNING] validate() accepts a regular file (or any non-directory) as data_dir / wal_dir / cache_dir + +**Location:** `tidal/src/db/builder.rs:237-252` +**Dimension:** Completeness · **Zone:** `db-core` · **Verifier:** CONFIRMED + +**Issue:** The directory loop checks only `dir.exists()` and `permissions().readonly()`. If a caller passes a path that exists but is a regular file (e.g. `with_data_dir("/etc/hosts")`), validation passes; the failure surfaces much later and far less clearly when fjall/WAL try to create children under a non-directory, producing an opaque io error rather than a typed ConfigError at the open boundary. + +**Why it matters:** The builder's stated contract is to return a specific typed ConfigError per failure at the single open boundary; a non-directory path defeats that and yields a confusing downstream error for an operator misconfiguration. + +**Fix:** + +In the dirs_to_check loop, after the exists() check add: `if !dir.is_dir() { return Err(ConfigError::DirectoryNotFound { path: dir.clone() }); }` (or a new `ConfigError::NotADirectory { path }` variant for an even clearer message). This keeps all path-shape validation at the same boundary as the existing checks. + +--- + +### [WARNING] Writability validation via permissions().readonly() can pass for a directory the process cannot actually write + +**Location:** `tidal/src/db/builder.rs:241-251` +**Dimension:** Accuracy · **Zone:** `db-core` · **Verifier:** CONFIRMED + +**Issue:** `m.permissions().readonly()` on Unix reflects only the inode's write-permission bits, not whether THIS process's uid/gid can write. A directory owned by another user with mode 0755 reports readonly()==false, so validation passes, and the real EACCES only appears later when the WAL/fjall attempt to create files. The inline comment already concedes "a more robust check would attempt to create a temp file" but ships the weaker one. + +**Why it matters:** The NotWritable error exists precisely so the operator learns at open() that the directory is unusable. As written it can pass validation and then fail opaquely during component init, defeating the intent for the most common real-world cause (wrong ownership). + +**Fix:** + +Probe writability for real: attempt to create+remove a uniquely-named temp file in the directory (e.g. `tempfile::Builder::new().tempfile_in(dir)`), mapping any failure to ConfigError::NotWritable { path }. This is the same approach the lock-file open already relies on and removes the false-positive. If the temp-file probe is considered too heavy for cache_dir (reserved/no-op today), at minimum apply it to data_dir and wal_dir, the two paths the engine actually writes. + +--- + +### [WARNING] signal_with_context narrows item id with a bare `as u32`, silently aliasing over-range item IDs in hard-neg/seen/durable user-state rows + +**Location:** `tidal/src/db/signals.rs:344` +**Dimension:** Accuracy · **Zone:** `db-entities` · **Verifier:** CONFIRMED + +**Issue:** `let item_u32 = entity_id.as_u64() as u32;` truncates the item id with no warning, then feeds it to `hard_negatives.add`, `mark_seen`, `add_save_durable`/`add_like_durable`, and `hard_negatives.persist`. Every sibling path that narrows an item id to its u32 slot was hardened to be observable: `write_item_with_metadata` REJECTS an over-range id (items.rs:64-71), and the `Hide` relationship path routes through `narrow_item_slot` which emits a `tracing::warn!` on overflow (relationships.rs:53/105 -> user_state.rs:50-65). This one path is the last silent truncation, and the in-code comment at items.rs:59-63 explicitly flags it as a deferred gap. Because the truncated id is also written into the DURABLE Tag::HardNeg and Tag::UserState rows (hard_neg_key/user_state_key take the already-narrowed value), two distinct items whose ids share their low 32 bits collide permanently on disk and on restore — a wrong-results path (an item the user never rejected gets excluded, or a save/like attaches to the wrong item) with zero operator visibility. + +**Why it matters:** Silent cross-id collision in user-correctness primitives (hard-negatives, seen, saved/liked) is exactly the class of bug that is invisible until a 3am 'why is this item missing from this user's feed' incident, and the durable rows make it survive restart. Consistency with the already-hardened Hide and item-write paths is the explicit project contract. + +**Fix:** + +Route the narrowing through the existing observable helper instead of a bare cast: `let item_u32 = crate::entities::user_state::narrow_item_slot(entity_id.as_u64());` (make it `pub(crate)` reachable from db). That logs a warn on overflow, matching items.rs and the Hide path, and removes the last silent truncation. Better still, since `signal_with_context` is a public entry point and the durable rows it writes are permanent, mirror `write_item_with_metadata` and reject an over-range item id up front with `TidalError::invalid_input` so a colliding row never lands on disk; then drop the per-call narrowing comment once all item-side write paths share the one guard. + +--- + +### [WARNING] Export session-journal stream stops at the first frame that yields no events, silently dropping all later sessions + +**Location:** `tidal/src/db/export.rs:614-619 (let events = decode_session_events(&frame); if events.is_empty() { break; })` +**Dimension:** Accuracy · **Zone:** `db-export` · **Verifier:** CONFIRMED + +**Issue:** read_session_journal_signals streams one length-prefixed frame at a time and treats an empty decode result as corruption -> break. But decode_session_events returns an empty vec not only on corruption: a valid v2 frame carrying a forward-compatible unknown record type is intentionally Skipped by decode_typed_record (session.rs:424-429, RecordOutcome::Skipped, no event pushed), so a single-frame decode of such a record returns []. The whole-buffer decoder correctly skips that frame and continues; this streaming reader instead breaks and drops every subsequent session's signals from the export. Today only 3 record types exist (Start/Signal/Close), so this is latent — it bites the instant a newer writer adds a 4th type to the journal, exactly the forward-compat path the v2 framing was designed to tolerate. (Close frames are not affected — Close pushes an event.) + +**Why it matters:** Silent, partial data loss in an RLHF/training export that downstream tooling assumes is complete. It is invisible until a schema/format evolution, and then it corrupts every export from older readers with no error. It also undermines the very purpose of the v2 forward-compat skip. + +**Fix:** + +Distinguish 'frame skipped' from 'frame corrupt'. Use decode_session_events_with_diagnostics(&frame) per frame: continue on a clean skip (events empty, corruption_detected false) and only break when outcome.corruption_detected is true. Add a test that writes a journal with a Start, an unknown-but-checksum-valid v2 frame, then a Signal, and asserts the trailing Signal is still exported. + +--- + +### [WARNING] FOR SESSION missing-session handling diverges between retrieve and search + +**Location:** `tidal/src/db/query_ops.rs:111-125 (retrieve) vs 272-289 (search)` +**Dimension:** Completeness · **Zone:** `db-ops` · **Verifier:** CONFIRMED + +**Issue:** For the same user-facing clause (FOR SESSION ), retrieve hard-errors when the session is absent — returns Err(TidalError::Query(QueryError::SessionNotFound(...))) at lines 117-121 — while search logs tracing::warn! and continues without the session boost (lines 278-285). Two query types, identical clause, opposite failure modes. + +**Why it matters:** CODING_GUIDELINES §6 'Graceful degradation, never failure' states the engine must 'never return an empty result set or an error for a well-formed query.' A RETRIEVE with FOR SESSION pointing at an expired/evicted session is a well-formed query; failing it hard contradicts the guideline AND surprises a caller who got degradation from the structurally-identical SEARCH path. A client that pages a feed with FOR SESSION will see SEARCH keep working but RETRIEVE start 500-ing the moment the session is swept — a confusing, surface-specific outage. + +**Fix:** + +Unify on the graceful-degradation behavior already implemented in search: in retrieve, replace the Err(SessionNotFound) arm (lines 117-121) with the same warn-and-continue path — on session_snapshot Err, log tracing::warn!(session_id, error, "FOR SESSION: session not found; executing without session boost") and fall through to base_executor. If a strict mode is genuinely wanted, make it an explicit query option that BOTH retrieve and search honor identically, rather than an accidental divergence. + +--- + +### [WARNING] signal_for_tenant commits the local half then returns Err — a retry double-counts the local signal + +**Location:** `tidal/src/db/replication_ops.rs:306-353` +**Dimension:** Accuracy · **Zone:** `db-ops` · **Verifier:** CONFIRMED + +**Issue:** When a tenant is in dual-write and write_assignments resolves [local, remote] (source local, target remote), the loop writes the local half via self.signal(...) (line 324) and only then hits the remote assignment and returns Err::Internal (lines 341-349). The method is non-atomic: it returns an error to the caller after a durable, weight-accumulating signal write already landed. signal()/record_signal accumulates (it is not idempotent — confirmed at db/signals.rs:120-159), so a caller that treats the Err as 'nothing happened' and retries will apply the local weight twice. The test at tests/m8p5_multitenancy.rs:307-323 asserts this partial-write-then-Err is the intended contract. + +**Why it matters:** A signal ledger is durability-critical state; a partial-success-reported-as-failure is a classic source of silent inflation. The standard caller reaction to Err is retry, and every retry adds another local view/like to the decay accumulator while the operation still never succeeds (remote stays unwired). The docstring documents the fail-closed intent but does not warn the caller that the local write is non-rollback-able, so the retry-amplification hazard is undocumented at the call site. + +**Fix:** + +Make the partial-write contract explicit and safe: (a) detect a non-local assignment BEFORE writing anything — iterate assignments once to check all are local, and if any is remote return Err immediately with zero writes (fully fail-closed, no partial mutation); the local write then only happens when the whole op can succeed. This removes the retry-double-count entirely. If the 'local half still applied' behavior is deliberately wanted, change the return to a typed PartialApplied { local_written: true } variant (not a bare Internal) so callers can distinguish 'retry is safe' from 'retry double-counts', and update the test to assert the typed outcome. + +--- + +### [WARNING] rebuild_item_indexes indexes a truncated, colliding id for entity_id > u32::MAX instead of skipping it + +**Location:** `tidal/src/db/state_rebuild.rs:382-396` +**Dimension:** Tech Debt · **Zone:** `db-recovery` · **Verifier:** CONFIRMED + +**Issue:** When a scanned item row has entity_id > u32::MAX the rebuild logs a warning (lines 383-388) and then STILL truncates to u32 and inserts it into the universe bitmap and every index (lines 389-396) under a colliding lower id. The live write path takes the opposite, safer stance: it REJECTS any item id > u32::MAX BEFORE any durable write (tidal/src/db/items.rs:64-71, 'so a rejected item never lands in storage in an unindexable state'). Because the write path refuses such rows, the only way one reaches the rebuild is a corrupt or hostile durable store — and in exactly that case the rebuild silently corrupts the in-memory universe/indexes by aliasing a real lower id, producing phantom query hits, rather than skipping the bad row. + +**Why it matters:** On corrupt/hostile on-disk input the rebuild manufactures cross-id collisions in candidate generation (a query for the colliding low id returns the wrong entity's metadata), which is a silently-wrong-results defect — the kind of corruption-driven misbehavior this database must fail safe against. The comment already admits 'universe bitmap entry will collide with a lower ID' yet proceeds anyway. + +**Fix:** + +Mirror the write path's refusal: after the warning, `continue;` to skip the row entirely rather than insert a collision. Replace lines 382-396 so the over-limit branch logs and skips: +```rust +if entity_id.as_u64() > u64::from(u32::MAX) { + tracing::warn!(entity_id = entity_id.as_u64(), + "entity ID exceeds u32::MAX during rebuild; skipping (write path never persists such ids)"); + continue; +} +let id_u32 = entity_id.as_u64() as u32; +``` +This keeps the rebuild's invariant identical to the write path: an item that cannot be safely u32-indexed is never indexed, instead of being indexed under a colliding id. + +--- + +### [WARNING] Orphaned session start records are detected on every open but never deleted or sealed into a snapshot — a permanent re-warned leak when the WAL no longer carries the session + +**Location:** `tidal/src/db/session_restore.rs:60-70 (restore_sessions, b"start" arm)` +**Dimension:** Completeness · **Zone:** `db-sessions` · **Verifier:** CONFIRMED + +**Issue:** When restore_sessions() finds a b"start" record (a session active at last shutdown), it logs a warning and increments `orphaned`, but takes no remediating action. If the same session is also present as an open session in the WAL, restore_session_wal_events rehydrates it and a later normal close deletes the start key (sessions.rs:184-188). But if the WAL session events are gone — the session journal is append-only with NO rotation/compaction (confirmed: session_journal.rs has no truncate/compact, and the prior review flagged its unbounded growth) and recover() stops at the first corrupt/truncated record (session_journal.rs:104-119), dropping all later events — the storage start record has no corresponding WAL Start and is never rehydrated, never closed, and never deleted. It is then re-detected and re-warned as orphaned on every subsequent open, forever. + +**Why it matters:** Operationally this is a permanently-firing warning that trains operators to ignore session-restore warnings (alert fatigue at 3am), plus a small unbounded accumulation of dead start records in the items keyspace. With the BLOCKER above it compounds: the orphaned id is also not reflected in next_session_id, so a future session can reuse it. + +**Fix:** + +In restore_sessions(), after the full scan, for each orphaned start record whose session_id is NOT in the set restored as active from the WAL, seal it deterministically: build a frozen closed-session snapshot (duration computed from started_at_ns, zero in-memory signals — explicitly marking the lost-state condition) and write it via a WriteBatch that put()s the snapshot key and delete()s the start key atomically, exactly as close_session_internal does (sessions.rs:178-192). That converts the unrecoverable active session into an archived record once, stops the recurring warning, and reclaims the key. Advance next_session_id past its id as part of the BLOCKER fix. + +--- + +### [WARNING] join_community / rejoin_community do a non-atomic read-then-put on the membership registry, racing a concurrent leave + +**Location:** `tidal/src/db/communities.rs:42-78 (join_community) and 156-174 (rejoin_community)` +**Dimension:** Tech Debt · **Zone:** `db-sessions` · **Verifier:** CONFIRMED + +**Issue:** join_community reads the prior row via membership_registry.with_row(...) (a DashMap::get + clone of (state,epoch,joined_at_ns)), drops the guard, builds a new Membership, persists it, then calls membership_registry.put(membership) which replaces the row. Between the read and the put, a concurrent leave_community can engage the stop-forward gate and put a Left row; join's subsequent put overwrites it with a fresh Joined/Rejoined row whose stop_forward_at_ns is 0 — silently re-admitting a member who just left, with no error to either caller. leave_community itself is careful (durability-before-mutation, engages the gate on a snapshot copy then persists then publishes), but it gives no protection against a join that read the pre-leave row. The DashMap entry is not held across the persist, so this is a genuine lost-update window, not just a theoretical one. + +**Why it matters:** Community membership governs whether a user's contributions flow to a shared overlay (a privacy/consent boundary). A lost-update that re-admits a just-left member silently defeats the stop-forward guarantee the rest of communities.rs documents as '<1s, by construction'. The window is small but the failure is a privacy regression, not a transient count error. + +**Fix:** + +Serialize the read-modify-write per (user, community). Either (a) hold the DashMap entry across the whole transition by using membership_registry.entry((user,community)) / a with_row_mut that takes &mut Membership and performs persist inside the held guard (accepting persist-under-lock for this cold lifecycle path), or (b) add a per-key lifecycle mutex (a sharded Mutex keyed by (user,community)) that join/leave/rejoin all acquire before their read-modify-write. Document the single-writer-per-(user,community) invariant on MembershipRegistry either way. The signal write path (community_contribute) stays lock-free; only the rare lifecycle transitions take the lock. + +--- + +### [WARNING] PreferenceVectors::restore neutralizes NaN but leaves a non-unit vector, defeating its own cosine-poisoning guard + +**Location:** `tidal/src/entities/preference.rs:306-343 (NaN handling at 334; insert at 339) vs cosine_similarity at 191-205` +**Dimension:** Accuracy · **Zone:** `entities` · **Verifier:** CONFIRMED + +**Issue:** restore() reads each f32 and does `vec.push(if f.is_nan() { 0.0 } else { f })`, then `self.inner.insert(user_id, vec)` WITHOUT re-normalizing. The comment at line 333-334 says it neutralizes NaN 'so a torn row cannot poison cosine scoring downstream.' But cosine_similarity (191-205) computes `dot / candidate_norm` and explicitly relies on the stored preference already being unit-length ('The stored preference is already unit-length, so we only need to normalize the candidate side'). When a torn/garbled checkpoint row has one or more NaN components, zeroing them produces a vector whose L2 norm is NOT 1.0, so cosine_similarity returns a mis-scaled similarity (can exceed 1.0 or be arbitrarily wrong) — exactly the poisoning the comment claims to prevent. Even a fully-finite-but-tampered row that is not unit-length is inserted verbatim. + +**Why it matters:** This is the corrupt-input boundary the DATABASE-CRITICAL lens weights heavily. A single torn preference row silently distorts personalized ranking scores for that user with no error and no observability, and the code gives a false sense of safety via a comment that overstates what the NaN check buys. + +**Fix:** + +Re-establish the unit-length invariant at the load boundary instead of trusting the stored bytes. After building `vec` (with NaN already zeroed), call `l2_normalize(&mut vec)` before `self.inner.insert(...)` — l2_normalize already leaves an all-zero vector untouched, so a fully-corrupt row degrades to a zero pref (cosine returns 0.0) rather than a poisoned non-unit one. This matches set()'s contract (which normalizes) and makes the comment true. + +--- + +### [WARNING] User-purge and agent-purge paths are asymmetric: one re-checkpoints eagerly, the other does not + +**Location:** `tidal/src/db/communities.rs:457-486 (purge_prior_contributions) vs tidal/src/db/remove_scope.rs:60-87 (CommunityAgent)` +**Dimension:** Tech Debt · **Zone:** `governance` · **Verifier:** CONFIRMED + +**Issue:** The agent-scope purge persists its tombstone then immediately re-checkpoints the community ledger (remove_scope.rs:80-87, 'belt-and-suspenders with the tombstone replay'). The user-scope purge (communities.rs:474-481) persists the tombstone and mutates memory but never re-checkpoints, relying solely on tombstone replay. Both are crash-correct (db/mod.rs:613-618 replays user tombstones on open and purge is idempotent), so this is not a durability defect — but two paths solving the identical problem two different ways invites a future editor to 'fix' one to match the other and accidentally remove a guarantee, or to assume the user path also re-checkpoints. + +**Why it matters:** Divergent handling of the same crash window is a maintenance hazard in the most safety-critical code in the zone; the next person must re-derive that both are correct rather than reading one consistent pattern. + +**Fix:** + +Pick one policy and apply it to both. Preferred: drop the eager re-checkpoint from the agent path too (tombstone replay already covers the window for both), and add a one-line comment on each path stating 'durability is provided by tombstone replay on open; the periodic/lifecycle checkpoint GCs the stale rows'. If eager re-checkpoint is genuinely wanted, factor a shared helper that persists-tombstone + mutates + (optionally) re-checkpoints so the two callers cannot drift. + +--- + +### [WARNING] RRF fuse() tie order is non-deterministic across process runs (no entity_id tie-break) + +**Location:** `tidal/src/query/fusion.rs:92-97` +**Dimension:** Accuracy · **Zone:** `query-suggest-fusion` · **Verifier:** CONFIRMED + +**Issue:** fuse() collects scores from a HashMap (default RandomState, per-process randomized iteration order) into a Vec, then sorts with `results.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(Ordering::Equal))`. There is no secondary tie-break on entity_id, and sort_by is stable, so any two entities with an equal fused score keep whatever relative order the randomized HashMap iteration produced — which changes between process runs. Downstream the fused order flows into `candidates` (search/executor/pipeline.rs:162), and the final score sort (query/executor/helpers.rs:260) is again `partial_cmp(...).unwrap_or(Equal)`, a stable sort that PRESERVES the random tie order into the returned results. So two identical Hybrid queries over identical data — e.g. before and after a restart — can return tied results in different order. This is the exact determinism class already hardened for in the BM25 sort (pipeline.rs:346 neutralizes NaN and uses total_cmp 'so the result is deterministic'), but fusion was missed. + +**Why it matters:** A ranking database that returns different orderings for the same query on the same data breaks reproducibility, makes relevance/A-B debugging non-falsifiable, and can drop or duplicate rows under keyset pagination across restarts when adjacent rows tie on score and the unstable global tie order disagrees with the cursor's assumption of a fixed total order. RRF is documented as the robust, deterministic fusion baseline; silent run-to-run reordering violates that contract. + +**Fix:** + +Break ties on entity_id to make the order a true total order, mirroring the BM25 fix: `results.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal).then_with(|| a.0.as_u64().cmp(&b.0.as_u64())));` (RRF terms are always finite positive, so partial_cmp never sees NaN here — the only missing piece is the id tie-break). Add a test fusing inputs crafted to produce equal scores asserting ascending-by-id tie order, plus a proptest that fusing identical inputs twice yields identical output. Apply the same `.then_with(id)` tie-break to the final score sorts in query/executor/helpers.rs:260 and search/executor/pipeline.rs so determinism holds end-to-end, not just inside fuse(). + +--- + +### [WARNING] Ranking still reads wall-clock Timestamp::now() per candidate instead of the threaded query clock; scores are not reproducible + +**Location:** `tidal/src/ranking/executor/helpers.rs:54-102 (read_agg) and tidal/src/ranking/executor/scoring.rs:69-149 (score_by_sort); driven by the dead `now` param in tidal/src/ranking/executor/mod.rs:192-197, 501-509` +**Dimension:** Accuracy · **Zone:** `ranking` · **Verifier:** CONFIRMED + +**Issue:** Every time-dependent read in the scoring pipeline — SignalAgg::Value (read_windowed_count), Velocity (read_velocity), DecayScore (read_decay_score), and the ProfileDecay recency factor — goes through the ledger's wall-clock wrappers, each of which calls Timestamp::now() internally (core.rs:214, 281, and read_velocity at core.rs:325 via read_windowed_count). The `now: Timestamp` argument threaded through score()/score_with_session()/score_personalized()/compute_raw_score()/score_by_sort() is never passed to any ledger read. So within a single query each candidate is decayed/aged to a slightly different clock, and re-running the identical query against an unchanged ledger yields different scores. This is the zone's special-focus item ('decay must read the query clock, not Timestamp::now()'). The ledger already grew the explicit-clock variants read_decay_score_at (core.rs:233) and read_windowed_count_at (core.rs:296) in a prior remediation, but they were never wired into the executor, and the mod.rs:181-184 doc comment still claims this 'requires a read_decay_score_at on the ledger (out of scope)' — which is now factually false. + +**Why it matters:** For a ranking database, reproducibility of a scored result set is a correctness property: A/B replay, debugging 'why did this item rank here at 3am', and the prior review's own goal of making scoring a pure function of (ledger snapshot, now) all depend on it. The blast radius is bounded (sub-microsecond per-candidate clock skew rarely flips adjacent ranks, and ordering is otherwise correct), so this is a WARNING, not a CRITICAL — but the parameter is live, misleading dead weight and the remediation is now cheap because two of the three needed _at methods already exist. + +**Fix:** + +Thread `now` into the reads. In compute_raw_score/score_by_sort, capture now_ns = now.as_nanos() once and call read_decay_score_at / read_windowed_count_at instead of the wall-clock wrappers; add the missing read_velocity_at(entity, signal, window, now_ns) to SignalLedger (it is a one-line wrapper over read_windowed_count_at / duration_secs) and route SignalAgg::Velocity through it. Update read_agg/read_agg_for_sort/passes_gates/passes_excludes to take now_ns. Then correct the mod.rs:181-184 and scoring.rs:58-61 doc comments to state the clock is honored. Add a determinism test: score the same candidates twice against an unchanged ledger and assert byte-identical scores. + +--- + +### [WARNING] gc_source transitions to Complete but never reclaims source-shard data — the name and 'GC window' docstring overstate behavior + +**Location:** `tidal/src/replication/migration.rs:216-241 (gc_source), 327-335 (TenantRouter::finalize_migration doc: 'The old source shard can be GC'd')` +**Dimension:** Completeness · **Zone:** `repl-cluster` · **Verifier:** CONFIRMED + +**Issue:** gc_source (222-241) checks the elapsed GC window then flips MigrationState::Finalizing -> Complete. It performs no storage delete, no shipper teardown, no source-shard reclamation — it is a pure in-memory state flip. finalize_migration's doc (tenant.rs:331) asserts 'The old source shard can be GC'd', and gc_source's name implies it does the GC. After a real migration the source shard's tenant data lives forever; nothing in this zone removes it. + +**Why it matters:** An operator reading these names/docs will believe source data is reclaimed after the window, freeing disk and removing the stale copy that could otherwise be served if routing ever reverts (see the finalize finding). The unbounded retention is also a correctness hazard: a stale source copy + a routing-revert bug = serving old data. Silently doing nothing where the API name promises reclamation is the kind of gap that bites during a capacity incident. + +**Fix:** + +Either (a) implement the reclamation: on gc_source, after the window, delete the tenant's source-shard keyspace (WriteBatch delete-range + flush) and tear down any source-shard shipper/dual-write plumbing; or (b) rename to mark_complete / advance_to_complete and amend the docstrings to state explicitly that source-data reclamation is a separate, not-yet-wired step (with the spec task reference, per CODING_GUIDELINES §12). Given durability is paramount, (a) is the right end state; (b) is acceptable only as an interim with an honest doc. + +--- + +### [WARNING] tenant_wal_dir is dead code — per-tenant storage/WAL isolation does not exist; signal_for_tenant writes all tenants into one shared ledger + +**Location:** `tidal/src/replication/tenant.rs:538-549 (tenant_wal_dir, zero non-test callers), tidal/src/db/replication_ops.rs:306-353 (signal_for_tenant -> self.signal)` +**Dimension:** Completeness · **Zone:** `repl-cluster` · **Verifier:** CONFIRMED + +**Issue:** tenant_wal_dir builds per-tenant WAL paths (/data/tenants/{id}/wal) but has no production caller (grep: only its own two unit tests). signal_for_tenant enforces the per-tenant rate limiter and resolves shard assignments, then for the local shard calls self.signal(...) (replication_ops.rs:324), which writes to the single shared SignalLedger/entity store with NO tenant namespace in the key. So on a single node hosting multiple tenants, all tenants' signals land in one keyspace; tenant_id influences only rate limiting and (multi-node) routing, not at-rest data isolation. The module header (mod.rs:3-4) frames replication as a no-op single-node, but tenant.rs presents tenant_wal_dir as if storage isolation were implemented. + +**Why it matters:** 'Multi-tenant isolation' is the headline concern for this zone. Presenting a tenant_wal_dir helper while no write path uses it invites a maintainer to assume isolation holds, when a single-node multi-tenant deployment actually co-mingles every tenant's data and WAL. A noisy/abusive tenant is not storage-isolated from others on the same node, contradicting the spirit of TenantConfig's max_storage_bytes/max_entities (which are also never enforced). + +**Fix:** + +Pick one and make it true: (a) wire tenant_wal_dir and a tenant key-prefix into the write path so each tenant gets an isolated keyspace/WAL and the TenantConfig limits are enforced; or (b) delete tenant_wal_dir and the implied-isolation doc, and document in signal_for_tenant that on-node tenant isolation is logical (rate/routing) only, not physical, with the spec task that will add it. Do not leave a dead isolation helper that implies a guarantee the write path does not provide. + +--- + +### [WARNING] LWWRegister::merge is non-commutative under an identical HlcTimestamp with differing values (unstated precondition) + +**Location:** `tidal/src/replication/crdt/lww_register.rs:36-38, 102-109 (merge); property test lww_register.rs:462-472` +**Dimension:** Accuracy · **Zone:** `repl-crdt` · **Verifier:** CONFIRMED + +**Issue:** `lww_other_wins(cur, inc)` returns true only when `*inc > cur`; equality keeps the current value. The ordering key is the full HlcTimestamp (wall+logical+node_id), which excludes the payload. So if two registers carry the SAME HlcTimestamp but DIFFERENT values, merge keeps whichever is `self`: merge(a,b) and merge(b,a) diverge. I confirmed this with a standalone repro of the exact merge logic — merge(a,b).value=Some(1), merge(b,a).value=Some(2). The `merge_is_commutative` proptest asserts this law for 'all registers' with a timestamp domain of only ~1.1M points (wall 0..=1e6, logical 0..=100, node 0..=10) and independent `any::()` values, so a collision seed would falsify the asserted law; it passes only because the sampler has not hit the exact same-key/different-value pair in 256 cases. + +**Why it matters:** This is a state-based CRDT whose entire purpose is deterministic post-partition reconciliation by merging untrusted peer state over the wire. A CRDT merge that is order-dependent for any reachable input cannot guarantee replica convergence — two replicas applying the same set of registers in different orders would settle on different values, silently. For honest nodes the per-node HLC uniqueness invariant makes the collision unreachable (a node never re-emits a (wall,logical) pair; distinct nodes differ in node_id), so this is not a live data-loss bug today — but it is an unenforced, undocumented soundness precondition, and the proptest claims a stronger guarantee than the code actually provides. A corrupt/forged peer register or a future HLC change that weakens uniqueness turns this latent gap into real divergence. + +**Fix:** + +Make the tie-break total by folding the value into the decision, OR document+enforce the precondition. Cleanest: in `LWWRegister` constrain `T: Ord` and break exact-timestamp ties on the value deterministically (e.g. keep the larger value), so equal keys resolve identically regardless of merge order — mirroring how `CrdtSignalState::merge` already appends the score to its key tuple `(last_update_ns, score)` and is therefore commutative. If a total `T` ordering is undesirable, instead (a) add a debug_assert/Internal-error guard that a same-key merge with a differing value is an invariant violation, and (b) harden the proptest generator to deliberately sample colliding timestamps with differing values so the asserted commutativity law matches the code's real guarantee rather than passing by sampling luck. + +--- + +### [WARNING] Embedding-slot dimension validation enforces only != 0, not the documented [2, 4096] bound + +**Location:** `tidal/src/schema/validation/builders.rs:317-322 (and SchemaError::InvalidEmbeddingDimensions at schema/error.rs:272-273)` +**Dimension:** Completeness · **Zone:** `schema` · **Verifier:** CONFIRMED + +**Issue:** build() rejects dimensions == 0 but accepts every other usize, including 1 (the spec's EmbeddingSlot doc and 11-schema.md V-E04 require [2, 4096]) and arbitrarily huge values up to usize::MAX. The dimension flows untouched into the vector registry/index (storage/vector/registry.rs:227, brute/mod.rs:221 cast it to u64/u32) and becomes the contract every persisted vector must match. The spec claims validation is 'eager and complete -- a definition that passes validation is guaranteed to be self-consistent', which this violates. + +**Why it matters:** A dimension of 1 is a degenerate vector index; an enormous dimension is accepted at define-time and only surfaces much later as an allocation failure or a per-vector insert error, far from the schema call that caused it. The whole point of define-time validation is to fail at the schema boundary, not deep in the ANN path on first insert. It is developer-supplied (not hostile wire input), so this is quality, not a crash-on-hostile-input BLOCKER. + +**Fix:** + +In the embedding-slot validation loop, after the `== 0` check add an explicit range gate matching the documented contract: `if !(2..=4096).contains(&slot.dimensions) { return Err(SchemaError::InvalidEmbeddingDimensions { slot: slot.name.clone(), dimensions: slot.dimensions }); }`. If 4096 is genuinely too low for some model, lift the ceiling deliberately and update the spec/doc — but pick a bound and enforce it eagerly. Add reject tests for dim=1 and dim=4097 alongside the existing rejects_zero_dimension_embedding_slot. + +--- + +### [WARNING] Reserved text-field key duplicated as two independent string literals coupled only by comment + +**Location:** `tidal/src/schema/error.rs:318 (RESERVED_TEXT_FIELD_KEY = "entity_id") vs tidal/src/text/index.rs:28 (ENTITY_ID_FIELD = "entity_id")` +**Dimension:** DRY · **Zone:** `schema` · **Verifier:** CONFIRMED + +**Issue:** The schema builder rejects a user text field named `entity_id` (builders.rs:361 compares against RESERVED_TEXT_FIELD_KEY) precisely because it would collide with Tantivy's always-present internal u64 field, whose name is independently defined as ENTITY_ID_FIELD in text/index.rs:488. The two constants are kept equal only by the comment 'Kept in sync with the field name in text::index::build_tantivy_schema'. They are not the same symbol — error.rs cannot import the text-module const without a layering edge, so the literal is copied. + +**Why it matters:** If a future change renames the internal Tantivy field via ENTITY_ID_FIELD but not RESERVED_TEXT_FIELD_KEY, the define-time guard would protect against the stale name while the new name silently becomes a user-declarable text field again — reintroducing the exact `add_field` panic inside Tantivy's SchemaBuilder::build() at DB open that this guard exists to prevent. A comment is not a compile-time coupling. + +**Fix:** + +Make text::index::ENTITY_ID_FIELD the single source of truth and have the schema guard reference it: `if field.key == crate::text::index::ENTITY_ID_FIELD`. If the schema->text dependency direction is undesirable, hoist one shared `pub(crate) const ENTITY_ID_FIELD` into a neutral low-level module (e.g. storage/keys or a small constants module) that both schema and text import. Add a compile-time assertion or a test that asserts the two are equal so a future rename fails the build, not production open. + +--- + +### [WARNING] SessionState's 14-field literal is hand-duplicated across both production construction sites with no constructor + +**Location:** `tidal/src/db/sessions.rs:66-81 and tidal/src/db/session_restore.rs:140-156 (struct def tidal/src/session/state.rs:24)` +**Dimension:** DRY · **Zone:** `session` · **Verifier:** CONFIRMED + +**Issue:** SessionState has 14 fields and no constructor; start_session (sessions.rs:66) and restore_session_wal_events (session_restore.rs:140) each spell out the full literal, including the five 'always the same' defaults (signals/signaled_entities DashMap::new(), annotations Mutex::new(Vec::new()), signals_written/signals_rejected AtomicU64::new(0), audit_log Mutex::new(AuditLog::new())). The two literals must be kept in lock-step by hand: adding a field (this struct has grown across milestones) means editing every site, and a divergence in an atomic's initial value or a missing reset would be a silent state bug. The same literal is also repeated in 3 test helpers. + +**Why it matters:** This is the precise structural smell that let the started_at/started_at_ns divergence in the BLOCKER persist: the two construction sites are independently maintained, so the restore path seeding a non-durable started_at went unnoticed against the live path. Centralizing construction makes the durable-vs-monotonic field handling reviewable in one place. + +**Fix:** + +Add `SessionState::new(id, user_id, agent_id, policy_name, started_at, started_at_ns, metadata) -> Self` that fills the five empty-state fields once, and have both production sites call it. Better, give it `started_at_ns` and derive `started_at: Instant::now()` internally with a doc note that the Instant is for live metrics only and started_at_ns is the durable age source — encoding the BLOCKER's invariant structurally. Tests can use a `#[cfg(test)]` builder default. + +--- + +### [WARNING] No tracing instrumentation anywhere in the storage zone, violating CODING_GUIDELINES §11.2 + +**Location:** `tidal/src/storage/fjall.rs:120-200 (StorageEngine impl), tidal/src/storage/memory.rs:40-117` +**Dimension:** Completeness · **Zone:** `storage-engine` · **Verifier:** CONFIRMED + +**Issue:** CODING_GUIDELINES §11.2 names this exact boundary as non-negotiable: "Instrument the public API and the major internal stage boundaries: EntityStore::{get, put, scan_prefix}" and §11.1 calls #[tracing::instrument] on subsystem-crossing functions "non-negotiable — it is how query latency, signal write throughput, and WAL sync times are measured in production." `grep -rn instrument tidal/src/storage/` returns nothing. Neither StorageEngine impl carries a single span. flush()/write_batch() — the fsync and atomic-commit barriers the spec says you measure WAL sync times on — are completely dark. + +**Why it matters:** At 3am during a production incident you cannot answer "is the slow query stuck in a storage get, a scan_prefix over a hot entity, or a flush fsync?" without spans. The guideline explicitly exists so this instrumentation does not have to be retrofitted under pressure. A restart that hangs in scan_prefix(&[]) (used by every state_rebuild path) is currently unobservable from traces. + +**Fix:** + +Add #[tracing::instrument(skip(self, value), fields(key_len = key.len()))] (and analogous for get/scan_prefix/write_batch/flush) to the FjallBackend StorageEngine impl methods, or — if per-call span overhead on get is a hot-path concern — instrument at the storage_box.rs dispatch layer and emit a tracing::debug! with the operation + byte counts on flush()/write_batch() at minimum. The guideline permits instrumenting at the stage boundary; pick one and wire it. Do NOT add tracing-subscriber (the crate is a library per §11.3) — import tracing only. + +--- + +### [WARNING] rebuild_from_store aborts the whole entity kind on a single un-indexable embedding + +**Location:** `tidal/src/storage/vector/registry.rs:399 (`slot.index.insert(entity_id.as_u64(), &vector)?`) and the `?` on deserialize at line 343` +**Dimension:** Completeness / Accuracy · **Zone:** `storage-vector` · **Verifier:** CONFIRMED + +**Issue:** The rebuild loop propagates `?` on the per-embedding `index.insert` (and on `deserialize_embedding`). If even one stored EMB: value yields a vector whose length differs from the slot's first-seen dimensionality — e.g. a slot dim changed across a schema revision, a partially-written/corrupt header that still passes deserialize_embedding's self-consistent length check, or a hand-edited store — `BruteForceIndex::insert` returns DimensionMismatch and the entire loop aborts. db/mod.rs:785 then logs 'rebuild failed' and leaves ANN search for that whole entity kind empty, even though hundreds of thousands of perfectly good embeddings for the same kind were skipped. One poison vector takes down the kind. + +**Why it matters:** This is the recovery path after a crash/restart — exactly when robustness matters most. The guideline (CODING_GUIDELINES §5, mirrored in the vector docs) is 'unconditionally rebuild from the store; any entity in the store but missing is re-indexed.' A single bad row defeating the rebuild for an entire kind violates that deterministic-recovery contract and is a silent, hard-to-diagnose degradation (ANN returns nothing for all Items) with only one error line to go on. + +**Fix:** + +Make the per-embedding body of the loop fault-isolating: on a deserialize error or an index.insert error for one (entity_id, slot), `tracing::warn!` with the id/slot and `continue` rather than returning the error, accumulating a skipped-count that is logged at the end. Reserve hard `Err` returns for whole-scan failures (storage.scan_prefix entry errors), which are genuinely unrecoverable. Add a test: seed a store with two valid embeddings and one whose stored header declares a different dimension under the same slot, rebuild, and assert the two valid ones are indexed and searchable while the bad one is skipped — proving one poison row no longer bricks the kind. + +--- + +### [WARNING] Doc comments reference a non-existent module path for the rebuild-at-open primitive + +**Location:** `tidal/src/text/writer.rs:100; tidal/src/db/items.rs:196; tidal/src/db/creators.rs:31` +**Dimension:** Tech Debt · **Zone:** `text` · **Verifier:** CONFIRMED + +**Issue:** Three doc comments point readers to `db::mod::rebuild_text_indexes_at_open` (and writer.rs:100 says `db::mod::rebuild_text_indexes_at_open`). The function actually lives in `db::state_rebuild::rebuild_text_indexes_at_open` (state_rebuild.rs:761); `db/mod.rs:751` only *calls* it. The `rustdoc` intra-doc style backtick path does not resolve. + +**Why it matters:** This is the single most load-bearing recovery invariant in the zone (it is why losing un-committed Tantivy writes on crash is safe). A maintainer following the cited path during an incident lands in the wrong file and may wrongly conclude the rebuild is unwired — exactly the silent-regression failure this design was written to prevent. The writer.rs reference is inside the reviewed zone. + +**Fix:** + +Update all three comments to `crate::db::state_rebuild::rebuild_text_indexes_at_open`. Prefer a real intra-doc link (`[crate::db::state_rebuild::rebuild_text_indexes_at_open]`) so `cargo doc` / clippy `broken_intra_doc_links` keeps it from rotting again. + +--- + +### [WARNING] BM25 retrieval returns the entire match set with no top_k cap on the non-degraded path — unbounded allocation on a broad query + +**Location:** `tidal/src/text/collectors.rs:22-103 (AllScoresCollector) consumed at tidal/src/query/search/executor/pipeline.rs:335-348` +**Dimension:** Completeness · **Zone:** `text` · **Verifier:** CONFIRMED + +**Issue:** `AllScoresCollector` collects EVERY matching document (by design, so the ranker does top-K). But on the normal path nothing bounds the result: `retrieve_bm25` returns the full Vec and truncation only happens under `degradation_level.reduces_candidates()` (pipeline.rs:138-141). The spec's `top_k_text` default of 200 (06-text-retrieval.md Sec 11.2) is never enforced at steady state. A broad/common term over a 10M-doc corpus collects millions of `(EntityId, f32)` pairs (24 bytes each → tens of MB) plus a full-size HashMap at pipeline.rs:145, per query. + +**Why it matters:** A single broad or hostile query (e.g. a near-ubiquitous token) drives an allocation proportional to corpus match count, not to the requested limit — a memory-amplification / latency-spike vector under load, and a divergence from the documented candidate-sizing contract. The collector itself is correct for its stated purpose, but the contract ('caller does top-K') is only honored when degraded. + +**Fix:** + +Bound the candidate set on the normal path too: either give `AllScoresCollector` an optional cap (a bounded min-heap of size `top_k_text`, defaulting to the spec's 200/configurable per profile) so it never materializes more than K, or apply a `top_k_text` truncation in `retrieve_bm25` after the sort regardless of degradation level. A bounded collector is preferable since it also caps peak memory inside Tantivy rather than after the fact. + +--- + +### [WARNING] rustls-pemfile is a declared but unused dependency + +**Location:** `tidal-net/Cargo.toml:38 (rustls-pemfile = "2"); no usage in tidal-net/src/` +**Dimension:** Tech Debt · **Zone:** `tidal-net` · **Verifier:** CONFIRMED + +**Issue:** rustls-pemfile is listed as a direct dependency but grep finds no rustls_pemfile usage anywhere in tidal-net/src — TLS material is parsed via tonic's Identity::from_pem / Certificate::from_pem in tls.rs, which do their own PEM decoding. The crate pulls and compiles a dependency it never calls. + +**Why it matters:** CODING_GUIDELINES §10 mandates 'minimal, intentional, auditable' dependencies, each justified against 'could we write this in 200 lines?'. An unused dep is unjustifiable by definition: it grows the build, the audit surface, and the supply-chain footprint for zero benefit, and misleads a future reader into thinking PEM parsing happens here. + +**Fix:** + +Remove `rustls-pemfile = "2"` from tidal-net/Cargo.toml (and the corresponding crate_universe pin if Bazel tracks it). Run `cargo build -p tidal-net` to confirm nothing breaks. If a future raw-PEM path needs it, re-add it at that point with the actual usage. + +--- + +### [WARNING] Sharded write handlers run blocking DB writes directly on the async reactor + +**Location:** `tidal-server/src/cluster.rs:892-936 (sharded_create_item / sharded_write_embedding / sharded_write_signal)` +**Dimension:** Accuracy · **Zone:** `tidal-server` · **Verifier:** CONFIRMED + +**Issue:** The non-sharded cluster read/write handlers carefully offload blocking `TidalDb` work (reads via `offload_cluster_read`, writes via `write_pool.submit`). The three `sharded_*` write handlers instead call `crate::scatter_gather::sharded_write_item/embedding/signal` synchronously inside the async handler. Those functions do a blocking `cluster.node(shard).db.write_*` call (storage + WAL fsync) on the reactor thread. The module docs (offload.rs:5-12) explicitly state reads/writes are blocking and must be moved off the reactor; these handlers violate that. + +**Why it matters:** A reactor worker is pinned for the full duration of a storage write + fsync. Under concurrent sharded-write load this starves every other in-flight request on that worker (the precise failure offload.rs was built to prevent). It is less severe than the cluster writes (no gRPC, single shard) but still blocks the reactor — and it is on the same un-throttled cluster router as the finding above. + +**Fix:** + +Route the sharded single-shard writes through `offload_read` (they touch only the local store, no gRPC, so spawn_blocking is sufficient) — e.g. `offload_read(move || sharded_write_signal(&cluster, ...)).await`, taking an owned `cluster_arc()` clone as the other cluster handlers do. This makes all three write paths consistent with the documented offload contract. + +--- + +### [WARNING] Multi-process cluster_e2e test verifies nothing across process boundaries — every node runs all regions in-process + +**Location:** `tidal-server/tests/cluster_e2e.rs:91-115, 311-373` +**Dimension:** Completeness · **Zone:** `tidal-server` · **Verifier:** CONFIRMED + +**Issue:** `ClusterHarness::start(n)` spawns n separate `tidal-server cluster` OS processes, each given the same full topology. Each process therefore builds its own `ClusterState`/`SimulatedCluster` containing ALL n regions internally. The test only ever writes to and reads from `nodes[0]` (e.g. line 366 reads `region=region-1` from node 0, which holds region-1 internally). A write to node 0 never reaches node 1's process. The test 'passes' purely on node 0's in-process replication; it would pass identically with a single process. The comment at lines 417-419 even acknowledges G1. + +**Why it matters:** This is the headline 'multi-process E2E' harness (commit b16025b), but it does not test process isolation, cross-node routing, or real network partition — the things the cluster claims to need verification for. Per the project standard ('Wiring is not verification'), this is wiring presented as an E2E proof. An operator reading the test name trusts a guarantee the test does not provide. + +**Fix:** + +Either (a) make each spawned process own exactly ONE region and have them peer over gRPC to real sibling addresses, then assert a write on node 0 becomes visible on node 1's process — the actual multi-process contract; or (b) rename/re-scope the test honestly (e.g. `single_process_cluster_smoke`) and mark the true multi-process E2E as an explicit known gap until G1/m8p10 land. Do not leave a no-assertion-across-processes test under the `cluster_e2e` name. + +--- + +### [WARNING] scope-stats reports `complete: true` when WAL integrity could NOT be verified (corrupt checkpoint case) + +**Location:** `tidalctl/src/commands/scope_stats.rs:151-152, 174` +**Dimension:** Accuracy · **Zone:** `tidalctl` · **Verifier:** CONFIRMED + +**Issue:** `inconsistency_count = diagnose_wal(base).map_or(0, |r| r.inconsistency_count)` collapses a diagnose_wal FAILURE to 0, and line 174 then sets `complete: inconsistency_count == 0` → `true`. read_all_events (line 132) only reads segment files and never touches checkpoint.meta, so with a corrupt `checkpoint.meta` + otherwise-valid segments the tally succeeds while diagnose_wal errors out. The output then asserts `complete: true` ("tally covers every WAL batch") even though integrity was never actually proven. This is the SAME corruption condition that makes `status` and `diagnostics` exit 2 (verified by status_corrupt_wal_exits_2 / diagnostics_corrupt_wal_exits_2). The in-band integrity signal — the operator's entire go/no-go for trusting the numbers — flips to a false positive in exactly the degraded case it exists to flag. The code comment at lines 147-150 admits 'we cannot prove completeness' but the emitted field claims the opposite. + +**Why it matters:** At 3am an operator runs `scope-stats` to decide whether the per-scope share-eligible tally can be trusted (it gates what data ships). `complete: true` is a positive integrity assertion; producing it when integrity is unknown is silently-wrong reporting on a durability-adjacent surface. A consumer branching on `complete` (the field's own rustdoc invites this) will trust an unverified tally precisely when the store is corrupt. + +**Fix:** + +Model the third state honestly instead of folding diagnosis-failure into the clean path. Capture the Result and distinguish unknown from clean: change the cross-check to `let diag = diagnose_wal(base);` then `inconsistency_count = diag.as_ref().map(|r| r.inconsistency_count).unwrap_or(0);` and compute completeness as `let complete = matches!(&diag, Ok(r) if r.inconsistency_count == 0);` so a failed diagnosis yields `complete: false` (integrity not provable → do not claim whole). Optionally add an `integrity: "clean" | "torn" | "unverified"` enum field and, to match status/diagnostics, return EXIT_DEGRADED when diagnose_wal errors so a script's `scope-stats && ship` cannot treat a corrupt store as success. Add a test mirroring scope_stats_torn_tail_reports_inconsistency that corrupts checkpoint.meta and asserts `complete: false`. + +--- + +### [WARNING] encode_session_event silently truncates string length fields > 65535 bytes (u16 cast), producing an unrecoverable journal record + +**Location:** `tidal/src/wal/format/session.rs:189, 191, 209, 212` +**Dimension:** Accuracy · **Zone:** `wal-format` · **Verifier:** CONFIRMED + +**Issue:** Every variable-length string is framed as `(s.len() as u16).to_le_bytes()` followed by the full `s.as_bytes()`. When `s.len() > u16::MAX` (65535) the cast wraps (e.g. 65536 -> 0, 65540 -> 4) while the encoder still appends the full-length byte payload. The record's len prefix and BLAKE3 are computed over the real bytes, so the frame is internally self-consistent and passes the checksum, but on decode `read_str_field` reads only the truncated count, leaving the remaining string bytes to be misinterpreted as the next field/record. `agent_id` is bounded to 64 chars upstream (session/types.rs:48), but `signal_name`, `policy_name`, and especially user-supplied `annotation` (which flows unbounded from the public signal API through sessions.rs:445-451 — MAX_ANNOTATIONS caps the count, not the byte length) have no length guard before reaching this encoder. + +**Why it matters:** A single oversized annotation corrupts its own record and, because the byte stream is misframed from that point, the decoder either stops cleanly (silently dropping that write AND every later event in the journal — lost session state on recovery) or decodes a garbled string. This is a silent-data-loss path triggered by ordinary large user input, not just hostile bytes, and it violates the project rule that the WAL is the durable source of truth. Encoding must never produce a record it cannot faithfully decode. + +**Fix:** + +Make the encoder fallible (or validate at the API boundary) rather than casting. Either (a) change `encode_session_event` to return `Result, WalError>` and reject any field whose `len() > u16::MAX` with a typed error, or (b) enforce a documented max length (mirroring agent_id's 1..=64 rule) on `signal_name`/`policy_name`/`annotation` at the write API so the invariant holds before the WAL. Add a test that an annotation of length 70_000 is either rejected or round-trips losslessly — currently neither holds. Prefer (a) so the format layer is self-defending regardless of upstream. + +--- + +### [WARNING] segment.sync() rustdoc overstates durability on macOS — fsync does not flush the drive write cache without F_FULLFSYNC + +**Location:** `tidal/src/wal/segment.rs:208-235 (doc + sync_data call at 233)` +**Dimension:** CLEAN · **Zone:** `wal-write` · **Verifier:** CONFIRMED + +**Issue:** The rustdoc states sync_data() 'flushes the file's data ... a single sync() makes the freshly appended batch fully durable' and is 'genuinely sufficient'. On Linux (fdatasync) this is correct. On macOS, std's File::sync_data/sync_all issues plain fsync(2), which Apple explicitly documents as NOT flushing the storage device's volatile write cache — true durability requires fcntl(F_FULLFSYNC). No F_FULLFSYNC handling exists anywhere in the crate (grep confirms zero hits). macOS is an active platform for this project (dev and test loop run on darwin per CLAUDE.md and the working env). + +**Why it matters:** A power-loss/hard-crash on macOS can lose batches the WAL has 'fsynced' and acknowledged to callers, because the bytes are still in the drive cache. The doc tells a future maintainer the durability story is complete when on macOS it is not. The WAL is the source of truth (CODING_GUIDELINES §2) and §8 requires 'no lost events' — on macOS this is silently weaker than the doc claims. Severity is WARNING rather than CRITICAL only because the overwhelming production target is Linux and the gap is a known OS-level fsync semantic, not a logic bug. + +**Fix:** + +Either (a) on macOS, issue F_FULLFSYNC instead of fsync for segment, checkpoint, session-journal, and directory syncs (a small platform-gated helper: on target_os="macos" call libc::fcntl(fd, libc::F_FULLFSYNC); fall back to sync_data elsewhere), or (b) if macOS is explicitly dev/test-only and never a durable production target, soften the rustdoc to scope the 'fully durable' claim to Linux/fdatasync and add a one-line note that macOS needs F_FULLFSYNC for crash durability. Do not leave the doc asserting full durability while the implementation cannot provide it on a supported platform. + +--- + +### [WARNING] WalHandle::truncate_before deletes segments without the directory fsync that the compaction path performs + +**Location:** `tidal/src/wal/writer.rs:308-317 (TruncateBefore -> segment::delete_segments_before) and tidal/src/wal/segment.rs:296-306 (delete_segments_before has no dir sync), vs tidal/src/wal/compaction.rs:145-151 (compact_segments fsyncs the dir after unlinks)` +**Dimension:** Accuracy · **Zone:** `wal-write` · **Verifier:** CONFIRMED + +**Issue:** compact_segments explicitly fsyncs the WAL directory after removing segments, with a comment that without it 'a crash after deletion but before directory metadata flush could resurrect deleted segment files.' The sibling deletion path used by the TruncateBefore command — segment::delete_segments_before — performs the same remove_file calls but never fsyncs the directory. So the two deletion paths disagree on durability of the unlink. + +**Why it matters:** A resurrected old segment is benign for correctness here (it is covered by the checkpoint and replays idempotently), so this is not data loss. But it is an undocumented inconsistency between two near-identical delete loops: a reader comparing them will reasonably assume the missing fsync in one is a bug, and a future change that makes segment resurrection non-idempotent (e.g. if truncate ever advances the WAL checkpoint marker) would turn this into a real defect. truncate_before is currently only called from tests (production uses compact_wal_online), which is why this is WARNING not CRITICAL. + +**Fix:** + +Unify the two paths: route TruncateBefore through compaction::compact_segments (or have delete_segments_before fsync the directory after deleting, matching compact_segments) so there is exactly one durable-delete routine. While there, add a doc note on delete_segments_before stating whether the unlink is made durable; today the asymmetry is silent. + +--- + + +## Downgraded by verifier to SUGGESTION (3) + +- **`db-export`** — Secondary-state checkpoint block duplicated four ways with divergent error handling + `tidal/src/db/backup.rs:325-348 (checkpoint_secondary_state) vs tidal/src/db/lifecycle.rs:173-198, tidal/src/db/state_rebuild.rs:591-646, tidal/src/db/remove_scope.rs:81` — The cohort / co-engagement / community / preference-vector checkpoint sequence is copy-pasted in at least four places. backup.rs logs-and-continues (returns ()); lifecycle.rs folds each failure into first_err; the guard conditions (entry_count() > 0, !is_empty()) differ subtly between copies. The backup.rs docstring even acknowledges 'the shutdown path has the same set inline.' Divergence risk is real: a future fifth derived ledger, or a changed guard, must be edited in four spots or one path silently skips a checkpoint. + +- **`tidal-net`** — Circuit breaker module doc claims an in_flight flag the type does not have + `tidal-net/src/circuit_breaker.rs:10-12 (module doc) vs :43-45 (HalfOpen variant)` — The module doc states 'the half-open state carries an `in_flight` flag so concurrent callers cannot all probe a peer that just failed.' The actual CircuitState::HalfOpen is a fieldless variant; the single-probe invariant is enforced by the variant itself (check() returns Ok only on the Open->HalfOpen transition and Err while in HalfOpen), not by any in_flight flag. The behavior is correct and well-tested (half_open_admits_exactly_one_probe), but the doc describes a data member that does not exist. + +- **`tidal-server`** — Cluster /signals returns 204 success while follower replication is best-effort and silently dropped + `tidal-server/src/cluster.rs:782-804 (write_signal) → tidal/src/testing/cluster.rs:424-437 (SimulatedCluster::write_signal send_segment)` — The cluster `write_signal` handler returns `204 No Content` once `SimulatedCluster::write_signal` returns Ok. But that engine call applies the write to the leader and then ships to followers with `let _ = transport.send_segment(...)` — the send result is discarded (cluster.rs:436 in the engine). A follower whose gRPC ship fails is never surfaced; the client sees 204 and assumes the signal is durably replicated. There is no acknowledgement, quorum, or even a logged warning on ship failure at the engine boundary. + + +## Standalone SUGGESTIONS (37) + +- **`cohort-load-misc`** — BUILD_HASH doc comment names the wrong environment variable + `tidal/src/lib.rs:30-32` — The rustdoc says "Build hash compiled in from the `GIT_HASH` environment variable" and "Falls back to `dev` if `GIT_HASH` is unset", but the const actually reads `option_env!("TIDALDB_BUILD_HASH")`. build.rs:3-4 reads GIT_HASH at build time and re-exports it as TIDALDB_BUILD_HASH, so runtime behavior is correct, but a reader grepping for TIDALDB_BUILD_HASH (the var that actually gates the const) finds a doc that points them at the wrong name. + +- **`cohort-load-misc`** — No test for the concurrent invalidate_all / slow-path-insert staleness window in the resolver + `tidal/src/cohort/resolver.rs:74-98` — resolve() computes membership against a registry snapshot outside the cache lock (lines 83-91), then re-acquires the lock to insert (95-96). If define_cohort + invalidate_all races between the snapshot and the insert, a membership computed against the OLD cohort set can be written into the freshly-cleared cache and served until the next per-user invalidate or LRU eviction. This is benign for a pure accelerator (and documented on invalidate_all), but there is no test pinning the behavior, so a future refactor could turn the benign window into a real lost-attribution bug undetected. + +- **`db-core`** — Preference-vector dimensionality fallback duplicated across both open branches + `tidal/src/db/open.rs:113-116 and 246-249` — The block `let pref_dim = schema_def.embedding_slots().first().map_or(128, |s| s.dimensions);` appears verbatim in both the Ephemeral and Persistent arms of open_with_schema, and the magic default 128 also appears independently in from_config (mod.rs:447, PreferenceVectors::new(128)). + +- **`db-entities`** — Finite-weight validation duplicated across signal and signal_scoped instead of shared like the backpressure guard + `tidal/src/db/signals.rs:128-132 and 190-194` — `check_write_backpressure` was correctly extracted into one helper so the two admission policies cannot drift (per the W10 fix and its regression test). The immediately-adjacent `if !weight.is_finite() { return Err(...) }` block is still copy-pasted verbatim into both `signal` and `signal_scoped`. The two admission preambles are otherwise identical; a future change to the weight rule (e.g. clamping, or a max magnitude) has to be made in two places and a missed one is a silent divergence between the two entry points — the same drift risk the backpressure extraction was meant to remove. + +- **`db-ops`** — tidaldb_checkpoint_failures_total omits the partition_id label every other line carries + `tidal/src/db/metrics/mod.rs:493-503` — Every node-identity series is stamped with partition_id (uptime/health/info at 320-329, and all feature-gated gauges via the same render). The unconditional checkpoint_failures_total block emits 'tidaldb_checkpoint_failures_total {failures}' with no labels at all (line 501). The W13 cluster-scrape goal is that per-node series stay distinct when many nodes feed one Prometheus — this counter breaks that for itself. + +- **`db-recovery`** — rebuild_entity_state and rebuild_item_indexes have no direct unit test for the rebuild-equals-pre-crash-state invariant + `tidal/src/db/state_rebuild.rs:148-222 and 367-414` — The zone's two central rebuild functions are exercised only indirectly through integration crash tests. There is no focused test asserting that, after writing relationships (block/hide/follow/interaction) and items (with category/format/creator/tags/duration/created_at), a rebuild reconstructs the identical UserStateIndex sets, interaction ledger, creator_items bitmap, universe, and created_at recency order. The shared-indexer design is the safety net, but a direct test would lock the contract and catch a future change to the shared path that the integration tests might not isolate. + +- **`db-sessions`** — community_contribute mutates the in-memory aggregate before persisting the cell, inverting the durability-before-mutation discipline used elsewhere in the zone + `tidal/src/db/communities.rs:299-345 (community_contribute)` — The method records into the in-memory community_ledger (community_ledger.record at line 302) BEFORE persist_cell (line 328). On a persist_cell failure it returns Err(Durability) but the in-memory aggregate has already been incremented, so memory is ahead of disk for that contributor cell. This is the opposite order from add_to_collection (collections.rs:70-85) and leave_community (communities.rs:115-139), which both persist first and mutate memory only on success. The WAL community event IS written durably first (record_signal_scoped at line 289), so on crash the checkpoint-restore plus WAL replay reconverges and there is no permanent loss — the inconsistency is transient and lost on restart. But within a single live process after a persist_cell error, the caller sees Err while the aggregate reflects a contribution the caller believes failed. + +- **`entities`** — Relationship key hardcodes the separator and Tag::Rel byte instead of going through encode_key/Tag + `tidal/src/entities/relationship.rs:72-97 (encode_relationship_key, relationship_prefix); parse offsets at 125-137` — encode_relationship_key and relationship_prefix build keys with literal `key.push(0x00)` (separator) and `key.push(0x04)` (Tag::Rel), and parse_relationship_to hardcodes the to-id at offset 11. The canonical contract lives in storage::keys (NUL = 0x00, Tag::Rel = 0x04, encode_key layout `[id BE][0x00][tag][suffix]`). The relationship key is exactly `encode_key(from, Tag::Rel, [rel_type_byte, to_be8])`, so it could be expressed through the shared helper. If Tag::Rel's byte or the separator ever changes, storage::keys has a drift-guard test but these hand-rolled functions silently diverge, corrupting every relationship key/prefix. + +- **`governance`** — Redundant HashMap re-lookup in the sorted-fold over contributors + `tidal/src/governance/community_ledger.rs:219-226 (decay_score_at), 427-431 (explain), 658-663 (aggregate_digest)` — Each site does `let mut keys: Vec<&ContributorKey> = cell.contributors.keys().collect(); keys.sort_unstable();` then `for k in keys { if let Some(e) = cell.contributors.get(k) { ... } }`, performing a second HashMap lookup per contributor that is provably present (the key came from the same map under the same read guard). The `if let Some` also implies a fallibility that cannot occur. + +- **`query-executor`** — creator_id backfill from item_metadata is duplicated verbatim across Stage 3 and Stage 4.5 + `tidal/src/query/executor/mod.rs:480-490 and tidal/src/query/executor/pipeline.rs:442-452` — The identical block (for each candidate with creator_id None, look up meta, parse `creator_id`, set EntityId) appears twice. The comment at pipeline.rs:436-441 explains the second pass is a cheap re-backfill, but the logic is copy-pasted rather than extracted. + +- **`query-executor`** — Preference-boost map keyed by truncated u32 can collide for >u32::MAX candidate ids + `tidal/src/query/executor/personalization.rs:112 and the consumer at tidal/src/ranking/executor/mod.rs:283,296` — compute_preference_boosts inserts `entity_id.as_u64() as u32` as the key, and the scorer looks it up with the same truncation. Because both sides truncate identically, the boost lands on the right candidate in the common case, but two candidates whose ids differ only above bit 32 collide on one map slot, cross-contaminating their additive boost. + +- **`query-suggest-fusion`** — Suggest.for_user is a stored-but-never-read API knob + `tidal/src/query/suggest.rs:35,60-63 and tidal/src/db/query_ops.rs:334-337` — Suggest exposes a for_user field and a for_user(id) builder method, both documented 'reserved for future use', but SuggestionIndex::suggest takes only (prefix, limit) and the call site in query_ops.rs never forwards req.for_user. A caller that sets for_user expecting personalized suggestions silently gets the same global result. + +- **`query-suggest-fusion`** — Trending and eviction sorts (suggest) also order ties by map iteration + `tidal/src/query/suggest.rs:241 and 269` — trending_suggestions does `pairs.sort_by(|a, b| b.1.cmp(&a.1))` and evict_low_frequency does `pairs.sort_by_key(|&(_, f)| f)` over pairs collected from a DashMap iterator. Entries with equal frequency are ordered by map iteration, so the returned top-N trending terms and which terms get evicted are non-deterministic among ties. The counts are inherently approximate and ephemeral, so impact is small, but it is the same latent inconsistency as the fusion finding. + +- **`ranking`** — Trending / cohort_trending double-count velocity: the Trending sort base and the profile boosts read the same signals + `tidal/src/ranking/builtins.rs:130-154 (trending), 431-453 (cohort_trending); applied additively at tidal/src/ranking/executor/scoring.rs:205-274 (score_trending base) + tidal/src/ranking/executor/mod.rs:526-541 (boost loop)` — The trending() builtin sets sort = Sort::Trending AND boosts = [share.velocity(24h)*2, view.velocity(24h)*1]. compute_raw_score adds the Sort::Trending base (score_trending = view_vel + 2*share_vel over the same 24h window) to boost_sum (2*share_vel + view_vel), so the effective score is 2*view_vel + 4*share_vel. Spec §11.9 states the sort mode 'replaces stages 4-5 (boost and penalty application)', i.e. boosts should not also fire when a formula sort is active. Because the duplication is a uniform positive scalar over identical inputs, the resulting order is identical to the spec'd formula, so this is not a correctness bug — but it is redundant work and a latent trap: any future change that makes the boost window differ from the formula window (e.g. boost over 6h, sort over 24h) would silently produce a hybrid the spec says should not exist. + +- **`ranking`** — Shuffle sort is a fixed global permutation, not the spec'd per-user, time-windowed, quality-weighted sampling + `tidal/src/ranking/executor/formulas.rs:72-84 (shuffle_score) and tidal/src/ranking/executor/scoring.rs:80` — Spec §11.6 defines shuffle_score(item) = random(seed) * quality_weight(item) with seed = hash(user_id, timestamp_minute) and quality_weight = sqrt(quality_score). The implementation is shuffle_score(entity_id) = hash(entity_id) only — no user_id, no timestamp_minute, and no quality weighting. The result is the same ordering for every user on every request forever, and low-quality items are as likely to surface as high-quality ones. The doc comment honestly says 'deterministic hash of entity ID', so this is an under-implementation rather than a hidden bug, but the advertised 'surprise me / mood-based discovery' behavior (per-user variety that refreshes per minute, quality-biased) does not exist. + +- **`ranking`** — All-equal candidate scores normalize to 1.0, but spec §8.2 specifies 0.5 + `tidal/src/ranking/executor/helpers.rs:273-274 (range < EPSILON -> 1.0) and the asserting test tidal/src/ranking/executor/tests.rs:73-88` — Spec §8.2 says: 'If max_score == min_score (all candidates scored equally), all normalized scores are set to 0.5.' normalize() sets them to 1.0 instead, and score_normalization_all_equal asserts 1.0. Ordering is unaffected (all equal), so this is purely a reported-score semantic mismatch, but the returned `score` field is what callers surface for explainability/thresholding, and a degenerate set reporting 1.0 ('maximally confident') rather than 0.5 ('neutral / no signal') is misleading — e.g. a Scan profile with no boosts on items that all have zero signals returns every item at score 1.0. + +- **`repl-shipping`** — lag_segments() silently returns 0 for an unknown shard, masking a misconfigured gauge + `tidal/src/replication/lag.rs:56-60 and 76-78` — lag_segments() computes leader.saturating_sub(self.state.applied_seqno(self.shard_id).unwrap_or(0)). If the gauge's shard_id is not present in the shared ReplicationState (a wiring bug — e.g. a gauge built for ShardId::SINGLE against a state that only tracks other shards), applied_seqno returns None, unwrap_or(0) treats it as 'fully behind' or, combined with a 0 leader, as 'caught up'. Either way the unknown-shard case is indistinguishable from a legitimately-zero reading. applied_seqno() (lag.rs:76-78) has the same unwrap_or(0). + +- **`schema`** — Embedding-slot names bypass the identifier validation applied to signal/policy names + `tidal/src/schema/validation/builders.rs:57-69 (embedding_slot) and 310-329 (validation loop)` — Signal names and policy names must pass is_valid_signal_name (non-empty, ASCII, lowercase [a-z0-9_], leading letter). Embedding-slot names are validated only for emptiness and (name,kind) uniqueness — any UTF-8 is accepted, including control bytes. In the fingerprint, a slot name is hashed as raw bytes followed by a fixed-width [kind:1][dims:8] trailer; because the kind byte (0..=2) can appear inside an unrestricted name, a maliciously/accidentally crafted pair of slot names could in principle produce the same concatenated byte stream under a different split, unlike signal names where the 0x00 kind separator after a guaranteed-non-control name makes collisions impossible. + +- **`session`** — Session start record persists durably to storage but session restore depends only on the fire-and-forget WAL Start event + `tidal/src/db/sessions.rs:87-107 (durable start record + fire-and-forget WAL) vs tidal/src/db/session_restore.rs:60-72 and :90-105` — start_session writes a start record synchronously to items storage (sessions.rs:90, a durable put) AND fires a best-effort WAL session_start (sessions.rs:98, fire-and-forget per wal/mod.rs:329-356). On restart, active sessions are re-created ONLY from the WAL Start events (restore_session_wal_events, session_restore.rs:90); the durable storage start record is used merely to rehydrate metadata (session_restore.rs:129-138) and, when seen alone, is logged as 'orphaned' / 'in-memory signal state is lost' (restore_sessions, session_restore.rs:60-68). So if a crash lands after the durable storage put but before the fire-and-forget WAL Start fsyncs, the session was durably recorded as started yet is NOT restored as active — it only emits an orphan warning. The reverse asymmetry of two start-of-truth records is easy to misread. + +- **`signals`** — Negative signal weight is accepted and can drive a hot decay score negative + `tidal/src/db/signals.rs:128-132 (guard) and tidal/src/signals/hot.rs:147-168` — db.signal() rejects only non-finite weights (NaN/Inf); a finite negative weight is accepted. Spec §8 states 'Negative weights are rejected (negative signals use separate signal types, not negative weights)'. A negative weight folds into the decay accumulator and can make the stored score negative, at which point the debug_assert!(step.new_score >= 0.0) in on_signal (hot.rs:168) would fire in debug/test builds and the INV-SIG-3 'decay scores are non-negative' invariant is violated. The warm-tier count is unaffected (it uses fetch_add(1)), so it is not a data-corruption bug, but the score path silently diverges from the spec and from its own asserted invariant. + +- **`signals`** — warm.rs restore() does not validate last_*_rotation_ns ordering relative to bucket pointers + `tidal/src/signals/warm.rs:361-389` — restore() clamps current_minute/hour/day to valid array bounds (good, prevents OOB on next rotate), but accepts arbitrary last_minute/hour/day_rotation_ns from the snapshot with no cross-field sanity. A corrupt checkpoint with last_hour_rotation_ns > last_minute_rotation_ns (hour anchor ahead of minute anchor) makes sum_current_hour compute k via now_ns.saturating_sub(last_hour) — if now_ns < last_hour the in-progress-hour fold reads k=1 (saturates), under-counting rather than mis-indexing. So it cannot panic or corrupt, but it can silently produce a wrong windowed count from hostile input until the next real rotation re-anchors the timestamps. + +- **`storage-engine`** — scan_prefix error mid-stream is untested on the Fjall backend + `tidal/src/storage/fjall.rs:148-152` — The streaming adapter maps each guard via `guard.into_inner().map_err(map_fjall_err)?`, so a torn read / decompress failure surfaces as a `Err(StorageError)` item partway through iteration. Every test (scan_prefix, scan_empty_prefix_enumerates_all, scan_prefix_is_lazy) and most production callers use `.collect::>>()` or `.flatten()` — the latter (e.g. governance/community_ledger.rs:570, db/mod.rs:613) *silently drops* an Err item. There is no test proving a mid-stream backend error is propagated rather than swallowed, and the `.flatten()` callers would swallow it if it were. + +- **`storage-engine`** — FjallStorage::open uses a hand-rolled corruption error instead of map_fjall_err, losing error classification + `tidal/src/storage/fjall.rs:266-296` — open() and open_keyspace() map every fjall open failure to `StorageError::Corruption { message: format!("failed to open ...: {e}") }`. But map_fjall_err (lines 92-118) is the carefully-built router that distinguishes Io (ENOSPC, permissions) / Closed (Locked, Poisoned) / Corruption. A permission-denied or disk-full on open is now stringified into a *corruption* message — the exact mis-classification map_fjall_err was written to prevent (its own doc comment, lines 69-91, argues this is wrong for the get/put path). + +- **`storage-engine`** — StorageError::BatchConflict is dead — never constructed by any backend + `tidal/src/storage/error.rs:16-18` — BatchConflict is defined, Display-tested, and referenced in trait docs (engine.rs:54 "batch conflict"), but neither FjallBackend::write_batch nor InMemoryBackend::write_batch can ever produce it — fjall's OwnedWriteBatch.commit() maps through map_fjall_err (which never yields BatchConflict), and the in-memory path is infallible under the write lock. It is a phantom variant. + +- **`storage-indexes`** — total_count() and selectivity() recompute a full bitmap union on every call + `tidal/src/storage/indexes/range.rs:207-218 (total_count), 195-203 (selectivity); tidal/src/storage/indexes/bitmap.rs:196-207 (total_count)` — total_count() unions every value bitmap in the tree/map on each invocation, and RangeIndex::selectivity() materializes the entire range bitmap just to read its len(). The evaluator's selectivity() path calls range-index selectivity per predicate during AND/OR ordering, so a query with several range predicates over a large index repeatedly walks all value bitmaps. At the stated M2 scale (10K entities) this is negligible, but it is O(distinct_values * avg_bitmap_size) per call and will become a hot-path cost as the index grows. + +- **`storage-indexes`** — RangeIndex::load_from_kv_pairs accepts trailing bytes after a fixed-width value + `tidal/src/storage/indexes/range.rs:355-362` — The loader checks value_raw.len() < V::WIDTH and errors on truncation, but does not check for EXTRA bytes: from_be_key_bytes reads only raw[..WIDTH] and silently discards any remainder. A well-formed key always carries exactly WIDTH value bytes, so this never fires on data this code wrote; but on corrupt or adversarial on-disk input, two distinct keys that share the first WIDTH value bytes but differ in trailing bytes would both decode to the same value (last-write-wins overwrites the tree entry), masking corruption rather than surfacing it. The bitmap loader, by contrast, validates the full structure via parse_suffix_value. + +- **`storage-indexes`** — node_count() and eval_to_bitmap() recurse without an explicit depth bound + `tidal/src/storage/indexes/filter/expr.rs:156-164 (node_count) and tidal/src/storage/indexes/filter/evaluator.rs:67-101 (eval_to_bitmap)` — The 256-node guard in db/query_ops.rs:33-41 bounds the TOTAL node count, but it is computed by node_count(), which itself recurses the full tree before the guard can reject it — and eval_to_bitmap/evaluate_not also recurse with no separate depth cap. A pathologically deep filter (e.g. Not(Not(Not(...))) thousands deep) would recurse to compute node_count before the breadth guard ever runs. In practice FilterExpr is built programmatically by the embedder (combined_filter wraps a flat Vec in a single And; there is no text parser producing arbitrary nesting), so this is reachable only if a caller hand-builds an extreme AST, which bounds the severity. + +- **`storage-vector`** — rebuild_from_store can register a zero-dimension slot from a corrupt store and pollute search results + `tidal/src/storage/vector/registry.rs:361-399 (`let dimensions = vector.len();` then lazy register + insert)` — The write path rejects empty vectors (l2_normalize returns ZeroNormVector before anything is stored), so dim=0 is unreachable through normal operation. But rebuild trusts whatever is on disk: serialize_embedding(&[]) is a valid 4-byte payload that deserialize_embedding happily accepts (expected_len == 4), so a corrupt or hand-crafted store containing an `EMB:slot` value of exactly 4 zero bytes makes rebuild register a 0-dimensional slot and insert a 0-length vector. l2_distance_sq over empty slices returns 0.0, so that entity would tie for nearest on every query against a 0-dim slot. + +- **`tidal-net`** — Server-side payload validation rejects strictly-greater-than but the engine wire limit is inclusive at MAX_PAYLOAD_BYTES + `tidal-net/src/server.rs:46-52; tidal-net/src/transport.rs:232-237; consistency with config.rs:119` — send_segment (transport.rs:232) and ship_segment (server.rs:46) both reject `len() > max_payload_bytes`, i.e. a payload exactly equal to the configured max is accepted. config.validate (config.rs:119) caps max_payload_bytes at MAX_PAYLOAD_BYTES inclusive, and the codec limits are set to max_payload_bytes. This is internally consistent (a payload == max is accepted on both ends and within the codec), so it is not a bug. The note is only that the boundary semantics ('> max' vs '>= max') are asserted by prose comments rather than a shared test, so a future change to one guard could silently desync the two ends. + +- **`tidal-server`** — 2MB body-limit literal duplicated instead of reusing the named constant + `tidal-server/src/cluster.rs:603 vs tidal-server/src/router.rs:41` — `build_cluster_router` hardcodes `DefaultBodyLimit::max(2 * 1024 * 1024)` while the standalone router defines `const BODY_LIMIT_BYTES: usize = 2 * 1024 * 1024` and uses it. The two can silently drift (raise one, forget the other) and a reader has to compute the magic number in cluster.rs. + +- **`tidal-server`** — ScatterGatherInfo construction is copy-pasted between sharded_feed and sharded_search + `tidal-server/src/cluster.rs:1005-1011 and 1069-1075` — Both sharded handlers build an identical `ScatterGatherInfo { degraded, unavailable_shards, shards_queried, elapsed_ms, shard_deadline_ms }` from the `ScatterGatherMeta`. The five-field mapping is duplicated verbatim. + +- **`tidalctl`** — scope-stats swallows a list_segments error and reports `wal_segments: 0` while having just tallied events from those segments + `tidalctl/src/commands/scope_stats.rs:154-156` — `list_segments(&wal_dir).map(|s| s.len()).unwrap_or(0)` discards any error and reports zero segments. read_all_events (line 132) internally calls the same list_segments and would have already returned Err via `?` if it failed, so in the common path this is unreachable — but if the directory contents change between the two scans (a segment rolled/removed by a concurrent live writer, which this lock-free inspector explicitly supports), the second call can fail or return a different count, yielding the internally-inconsistent output `wal_segments: 0` alongside a non-zero `total_events`. The `0` reads as 'no segments' rather than 'segment count unknown'. + +- **`wal-format`** — Length-prefixed string write is hand-rolled 4x; no encode-side mirror of read_str_field + `tidal/src/wal/format/session.rs:189-192, 209-210, 212-213` — The decode side centralizes the `[len: u16][bytes]` framing in `read_str_field` (line 470), but the encode side hand-rolls `payload.extend_from_slice(&(x.len() as u16).to_le_bytes()); payload.extend_from_slice(x.as_bytes());` four separate times (agent_id, policy_name, signal_name, annotation). There is no single `push_str_field` owning the write-side framing. + +- **`wal-format`** — Present-but-short optional fields (seqno/idempotency_key) in a checksum-valid v2 Signal frame silently decode to None instead of stopping + `tidal/src/wal/format/session.rs:548-591` — In `decode_signal_record`, a present optional (`has_seqno != 0` / `has_key != 0`) whose value runs past `end` falls through to `None` (lines 551-555, 565-587) and the function still returns `Some(Signal{..})`, i.e. RecordOutcome::Decoded — the flag byte is consumed but the absent value is not, and the partial field is silently dropped rather than treated as Malformed. The writer always emits the full value when the flag is set, and a checksum-valid v2 frame proves the bytes are intact, so a genuine writer can never produce this; it is reachable only via a hand-crafted frame with a recomputed valid BLAKE3. No test covers it. + +- **`wal-recovery`** — Dedup window force_rotate inside record() resets the time-based rotation clock, slightly weakening the time window after a size-driven rotation + `tidal/src/wal/dedup.rs:104-110 and 152-156` — record() (called post-fsync for every kept event) calls force_rotate() when current reaches max_entries, and force_rotate() resets rotation_time = Instant::now(). Because record() does not itself drive the time window (only contains/is_duplicate call maybe_rotate), a burst that triggers size-driven rotations repeatedly pushes the time-based rotation deadline forward, so the effective time window can be longer than configured under sustained load. This is explicitly within the documented best-effort contract (dedup.rs:16-23), so it is not a correctness bug — but it is an undocumented interaction at the record() site. + +- **`wal-write`** — WalError::SegmentFull is dead code — defined and unit-tested but never constructed in production + `tidal/src/wal/error.rs:13-15 (variant) and error.rs:62-64 (display test)` — SegmentFull is documented as an 'internal signal to trigger rotation', but rotation is now driven by SegmentWriter::needs_rotation() + rotate() inside flush_batch (writer.rs:224-226); the variant is never returned anywhere in non-test code (grep confirms the only references are the definition and its Display test). + +- **`wal-write`** — No test covers a flush failure that occurs during segment rotation (rotate() failing mid-commit) + `tidal/src/wal/writer.rs:224-229 (flush_batch: needs_rotation -> rotate -> write -> sync) and the test-only fault hook at writer.rs:206-214 which fires before encode` — The FAIL_NEXT_FLUSH hook injects an error at the very top of the write closure (before encode/rotate/write/sync), so the resilience tests prove caller-notification and writer-survival for a failure on a non-rotating batch. There is no test where the failure lands specifically inside rotate() — i.e. the old segment was synced but the new segment file failed to open, or the directory fsync failed — even though that path is real (rotate does file open + dir sync_all, both fallible). + + +## Refuted by adversarial verification (2) — listed for transparency + +- **`query-executor`** — ~~signal_ranked_candidates buffer-cap math can under-bound when limit is 0~~ + `tidal/src/query/executor/candidate_gen.rs:68,85,101,108-111` — **REFUTED:** The claimed "index-underflow panic" cannot occur because the `.max(200)` floor in `candidate_gen.rs:68` means `max_candidates` is always >= 200, even when `limit == 0`. With `limit == 0`: `max_candidates = (0*4).max(200) = 200`, `buffer_cap = 200.saturating_mul(2).max(201) = 400`. The inline partition at line 101 fires only when `scored.len() >= buffer_cap = 400`, so the index `max_candidates - 1 = 199` is always valid (vec has at least 400 elements). The final partition at lines 108-111 fires only when `scored.len() > max_candidates = 200`, so index 199 is valid (vec has at least 201 elements). Similarly in `pipeline.rs:210`, `cap = (0*4).max(100) = 100`, and `select_nth_unstable_by(cap - 1 = 99)` is only called inside `if candidates.len() > cap`, guaranteeing at least 101 elements. The `.max()` floors make the arithmetic safe regardless of whether `limit == 0`. The upstream guard at `types.rs:188` is a real dependency, but the failure mode described — "index-underflow panic" — is arithmetically impossible because `max_candidates` can never be zero. The finding's core premise is wrong. + +- **`repl-crdt`** — ~~CrdtSignalState carries a single lambda; multi-decay-rate signals lose fidelity on reconciliation~~ + `tidal/src/replication/crdt/signal_state.rs:99-103 (lambda field doc); consumed at tidal/src/signals/ledger/core.rs:533-539 (apply_crdt_state) and core.rs:599-617 (crdt_contributions)` — **REFUTED:** The finding's core premise is false: the schema does not support multiple decay rates per signal type. `DecayModel` is a single-lambda enum (`Exponential { half_life, lambda }` stores exactly one f64), and `derive_signal_ids_and_lambdas` (tidal/src/signals/ledger/mod.rs:67-69) always builds `vec![*lambda]` — a one-element vector. The `signal_lambdas: HashMap>` is a forward-compatibility container but is never populated with more than one entry per signal type in the live codebase. Consequently `lambdas.len()` at core.rs:533 is always 1 for Exponential signals (or 0 for Permanent/Linear), so the loop `for i in 0..lambdas.len().min(MAX_DECAY_RATES)` iterates at most once. The "slots 1..N overwritten with wrong lambda" scenario cannot occur because no multi-rate signal can be constructed. The comment at core.rs:529-532 is an explicit, honest scope declaration ("CrdtSignalState uses a single lambda per signal type"), not a silent omission. The finding also misidentifies the CRDT snapshot path: `crdt_contributions` snapshots `stored_score(0)` with the signal's one lambda, which is exactly correct for the only decay rate that exists. No fidelity loss occurs for any currently constructible signal. + + +## PRAISE — what the code does well (58) + +- **`cohort-load-misc`** — Cohort checkpoint restore is genuinely all-or-nothing and the caller contract matches the code + +- **`cohort-load-misc`** — Hostile-input hardening on the checkpoint suffix codec and cohort-name length bound + +- **`db-core`** — Shutdown captures the first durable-flush failure and still attempts every step + +- **`db-core`** — Hostile/corrupt metadata input cannot panic, over-allocate, or mis-frame on the crash-recovery rebuild path + +- **`db-core`** — Partial-init failure leaves no corrupt or locked state + +- **`db-entities`** — Overwrite re-index scrubs stale in-memory index entries before re-inserting, through the same shared path the restart rebuild uses + +- **`db-entities`** — Hostile-input metadata deserialize is bounds-checked, non-panicking, and saturating on the length prefix + +- **`db-export`** — Bounded export scan eliminates the unbounded-Vec OOM on both WAL and session-journal paths + +- **`db-export`** — JSON-Lines export delegates all escaping to serde_json, with a non-panicking fallback on the export hot path + +- **`db-ops`** — HTTP accept-inheritance bug fixed at the root with a precise invariant comment + +- **`db-ops`** — Cumulative histogram is correct and proven against the naive rule + +- **`db-recovery`** — Write path and restart-rebuild share one indexing implementation, making cross-restart drift impossible by construction + +- **`db-recovery`** — Session WAL replay reproduces live windowed-counter rotations by sorting events ascending and seeding from each signal's earliest timestamp + +- **`db-sessions`** — Prior-review CRITICAL notification-tracker memory leak is genuinely fixed and wired into the periodic sweeper + +- **`db-sessions`** — Session/snapshot deserialization is uniformly hardened against hostile and corrupt on-disk input + +- **`entities`** — narrow_item_slot makes a previously-silent u64->u32 truncation observable and centralized + +- **`entities`** — Corrupt-input determinism via total_cmp and bounds-checked deserialization across the zone + +- **`governance`** — Checkpoint is a single atomic WriteBatch with watermark rows that survive total contributor purge + +- **`governance`** — Empty-agent purge footgun made inert at both the API boundary and the engine + +- **`query-executor`** — Negated and OR-nested deferred filters fail loud instead of silently returning the inverse/AND set + +- **`query-executor`** — Unknown-signal degradation is surgical — only UnknownSignalType degrades, every other error propagates + +- **`query-search`** — Deferred filters are now fully applied and shared verbatim with RETRIEVE — the prior silent-drop is genuinely closed + +- **`query-search`** — Disciplined checked u64->u32 conversion and poison-lock recovery throughout the scope/filter path + +- **`query-suggest-fusion`** — Cursor wire format is genuinely crash/hostile-input safe + +- **`query-suggest-fusion`** — MAX_TRENDING_KEYS cap held atomically under concurrency via CAS-reserve + +- **`ranking`** — NaN/Inf handling and the missing-metadata sentinel normalization are correct and well-defended + +- **`ranking`** — DiversitySelector invariants INV-RANK-5/6 are explicit, enforced, and property-tested + +- **`repl-cluster`** — enter_dual_write implements true durability-before-observable-mutation with a tested rollback + +- **`repl-cluster`** — Rate limiter decouples refill-claim from consume so contended retries cannot double-grant tokens, with a concurrency test that proves it + +- **`repl-crdt`** — HLC logical-overflow carry is now symmetric across now() and update(), with guarding tests + +- **`repl-crdt`** — Double-decay fix and LWW-decision unification are both real and test-pinned + +- **`repl-shipping`** — Community-overlay WAL filter preserves the authoritative last-seq boundary — idempotency and lag stay correct across event dropping + +- **`schema`** — Schema fingerprint correctly protects on-disk integrity with a single canonical kind encoding + +- **`schema`** — Zero panics in production schema code; every fallible path returns a typed SchemaError + +- **`session`** — Restored window counts are made byte-identical to live via earliest-timestamp seeding and ascending replay + +- **`session`** — Session record deserialization rejects hostile/corrupt input instead of best-effort patching or OOMing + +- **`signals`** — Forward-decay kernel unified as single source of truth with out-of-order correctness + +- **`signals`** — Windowed tier-overlap double-count eliminated with bounded in-progress-hour fold + +- **`signals`** — Checkpoint codec is offset-derived with compile-time size pinning and self-checksummed meta + +- **`storage-engine`** — write_batch atomicity is genuinely all-or-nothing and proven by fault injection + +- **`storage-engine`** — map_fjall_err preserves io::Error identity and conservatively halts on unknown variants + +- **`storage-indexes`** — Deferred-filter 'return-universe' contract is soundly guarded end-to-end + +- **`storage-indexes`** — Inverted/empty range bounds are clamped instead of panicking + +- **`storage-vector`** — BruteForceIndex::deserialize is a model of hostile-input hardening + +- **`storage-vector`** — Durable-first ordering in lifecycle ops is correct and rigorously reasoned, with a regression test for the failure window + +- **`text`** — Strict fast-field accessor refuses to alias entity 0 on a missing id value + +- **`text`** — Open-failure and absent-vs-unreadable are deliberately not collapsed onto one silent path + +- **`tidal-net`** — Permanent-vs-transient error classification is correct, exhaustive, and prevents an infinite silent replication stall + +- **`tidal-net`** — mTLS security boundary is proven by negative tests, not just a happy path + +- **`tidal-server`** — Scatter-gather partial-failure handling is genuinely robust — no silent shard drops, NaN-safe ordering, deadline-bounded + +- **`tidal-server`** — Entity→shard routing delegates to the engine ShardRouter so write and read routing cannot diverge + +- **`tidalctl`** — Refuses to fabricate a '0' for values it cannot honestly derive offline + +- **`tidalctl`** — All JSON routed through one serde_json path, closing the hand-rolled control-char escaping hole + +- **`wal-format`** — Length fields read from disk are bounds-checked before any slice, and the only length-driven allocation is hard-capped + +- **`wal-format`** — Corruption-vs-torn-tail contract is precisely specified and tested, including the subtle checksum-valid-but-short-body case + +- **`wal-recovery`** — Dedup is correctly split into contains/record so a failed flush is never suppressed as a phantom duplicate + +- **`wal-recovery`** — Hostile-input deserialization never panics: bounds checked before BLAKE3, before allocation, and before every field read + +- **`wal-write`** — Active-segment clamp is implemented identically in both the offline and online deletion paths and regression-tested from both entry points + + +--- + +## Appendix — seven-dimension coverage per zone (forcing-gate proof) + +Every zone was examined against all seven dimensions; each bullet is the reviewer's one-line verdict. + +### `cohort-load-misc` — Cohort agg, bulk load, test harness, lib.rs _(12 files)_ +- **Completeness:** Largely clean. Error/edge paths are handled: cohort checkpoint restore is all-or-nothing with a documented corruption policy that matches its caller in db/mod.rs:682-707; rate-limiter and degradation thresholds validate degenerate inputs; load detector has proptests; no orphan TODO/FIXME or commented-out code in the zone. One gap: cohort/resolver.rs has no test for the concurrent define+invalidate_all staleness window (SUGGESTION). +- **Accuracy:** One real defect: cohort/resolver.rs uses .expect("...poisoned") on the cache mutex at 6 production (non-test) call sites on the signal-write hot path, diverging from the codebase's own PoisonError::into_inner recovery pattern (faults.rs, cluster.rs) and violating CODING_GUIDELINES §7. write_signal seqno rollback (cluster.rs:399,411) cannot underflow (slot >= 1 after increment). deserialize/restore paths are length- and version-checked against hostile input. Atomic orderings in crash_injector and load detector are correct and justified. +- **Tech Debt:** Honestly documented: cohort/checkpoint.rs:153-159 (full-keyspace scan_prefix) and cluster.rs:700-709 (applied_count hardcodes ShardId(0), undercounts post-promotion) both carry explicit doc notes and caller contracts. relay_log_len Relaxed-ordering correction is justified in-comment. No hidden hacks. +- **Maintainability:** Strong. Names are intention-revealing, functions focused (<50 lines mostly), WHY-comments are dense and load-bearing (corruption policy, atomicity invariants, Arc-clone vs String rationale). lib.rs BUILD_HASH doc comment (lines 30-32) names the wrong env var (GIT_HASH vs the TIDALDB_BUILD_HASH the code actually reads) — minor doc drift. +- **Extensibility:** Appropriate. Pluggable Transport in the cluster harness, configurable thresholds/cache-capacity, Predicate enum composes cleanly. No over-engineering; no YAGNI abstractions. +- **DRY:** One concrete violation: single_event_payload (cluster_transport.rs:95) was extracted specifically to keep write_signal and redeliver_missed from drifting on event_count/leader_last_seq, but write_signal (cluster.rs:428-434) still builds the payload inline instead of calling it — the exact drift the helper exists to prevent. resolver.rs:55 has redundant-but-safe defensive clamp code. +- **CLEAN:** Clear and neat overall. Hot-path allocation discipline is deliberate (Arc::clone keys in CohortSignalLedger, enabled fast-path in RateLimiter, borrowed-&str registry lookups). The inline payload duplication and the BUILD_HASH doc are the only blemishes; no dead code or misleading names found. + +### `db-core` — TidalDb core / builder / open / lifecycle _(11 files)_ +- **Completeness:** Open/close/Drop, builder validation, capability gating, partial-init, and crash-recovery rebuild paths all have tests; no orphan TODO/FIXME or commented-out code in zone. Gaps: validate() omits is_dir() checks (WARNING) and writability is metadata-only (WARNING). +- **Accuracy:** Durability ordering in shutdown_inner is correct (persist derived state -> storage.flush -> WAL marker -> compact); WAL replay vs checkpoint boundary uses strict greater-than so no double-count; first-failure-wins error capture surfaces lost durable state. Lock acquired before construction, released on every error path via Drop. Only one non-test expect() (mod.rs:258), a locally-guaranteed invariant. Clean. +- **Tech Debt:** cache_dir is reserved/no-op (documented W1); no magic numbers beyond the documented 128-dim preference fallback. resolved_wal_dir() is a single source of truth. No copy-paste smells; SharedDefaults + build_control_plane already factor the two constructors. Clean. +- **Maintainability:** Names are intention-revealing; the heavy from_parts/open_with_dir functions carry detailed WHY comments tying each step to a ticket (BLOCKER 6/7, obs-REPL-1, CONCURRENCY-1/2/3, SHUTDOWN-2). from_parts is long (>500 lines) but linearly ordered and annotated — acceptable given construction is inherently sequential. +- **Extensibility:** StorageBox trait-routes Memory vs Fjall; NodeConfig/NodeRole cleanly gate single/leader/follower; runtime-validated builder is the right call over typestate (justified in-code). No over-engineering. +- **DRY:** Default field initializers centralized in SharedDefaults; control-plane wiring shared via build_control_plane/install_metrics_wiring; WAL dir resolved through one method. The two `let pref_dim = ... map_or(128 ...)` blocks in open.rs are the only minor duplication (SUGGESTION). +- **CLEAN:** Bounds-checked, checked_add deserialize on the rebuild path; saturating length encode; hostile metadata input degrades to empty map with no panic/OOM. Lock-error path distinguishes WouldBlock from real I/O. Genuinely tidy. + +### `db-entities` — Entity write ops (items/users/creators/signals) _(11 files)_ +- **Completeness:** Write paths handle validation, defaults, overwrite-scrub, and durable restore; one known gap (signal_with_context item-id narrowing) is documented-but-deferred. Tests cover oversized metadata, u32 boundary, overwrite scrub, backpressure parity. +- **Accuracy:** WAL-first ordering correct; durability-before-mutation in remove_scope; checked_add bounds on deserialize; no unwrap/expect/panic in non-test code; lock-poison recovery on universe bitmap. One silent-aliasing narrowing remains (signals.rs:344). +- **Tech Debt:** Bare `as u32` in signal_with_context is the last un-unified narrowing — the in-code comment (items.rs:59-63) acknowledges it as deferred. metrics only on `signal`, not `signal_scoped` (minor parity gap). +- **Maintainability:** Names intention-revealing; functions focused; extensive WHY-comments tie each side-effect to its recovery path. signal_with_context is a long dispatcher but each arm is a one-liner delegating to a named helper. +- **Extensibility:** Shared helpers (write_entity_meta, write_entity_embedding, entity_embedding_slot, persist_to_items, check_write_backpressure) factor the right seams. narrow_item_slot exists as the intended shared guard but is not yet wired into the signal path. +- **DRY:** Strong: user/creator meta + embedding + persist paths share one body; signal/signal_scoped share backpressure+finite guard. Minor: finite-weight check duplicated inline in both signal entry points rather than shared like backpressure is. +- **CLEAN:** Clear and neat; no dead code; EMB: row skip prevents garbage deserialization; explicit no-op arms (InteractionWeight|Mute) are greppable not catch-all. + +### `db-export` — Export / backup / text-sync / sweeper _(5 files)_ +- **Completeness:** Gaps found: no test/handling for forward-compat session frames in the streaming export (the stream stops early); no test asserting restored-session TTL expiry; backup has no test for host-crash durability of the copy. Validation of export limit/time-range is otherwise complete and tested. +- **Accuracy:** Three real defects: (1) export session-journal streaming break-on-empty silently drops the rest of the journal on a forward-compat/Close-less frame; (2) sweeper measures session age from a monotonic Instant reset on restore, so restored sessions never TTL-expire — already flagged in the 2026-06-07 review and still unfixed; (3) backup copy never fsyncs files or the dest dir, so a host crash can leave a 'crash-consistent' backup truncated. Batch-WAL scan, u8 signal-type guard, JSON escaping, and CAS backup guard are correct. +- **Tech Debt:** Secondary-checkpoint block is duplicated across backup.rs, lifecycle.rs, remove_scope.rs, state_rebuild.rs with subtly different error folding — a known, documented split. MAX_SESSION_FRAME_BYTES and NANOS_PER_DAY are reasonable named constants. +- **Maintainability:** Strong: intention-revealing names, thorough WHY comments, focused functions. The export module's heap/bounded-retainer abstraction is well documented. start_sweeper docstring overstates the down-time-reaping guarantee (contradicted by the Instant reset). +- **Extensibility:** render_signals match-on-format and the ExportFormat enum make adding formats a compile-error-driven change — good. BoundedEarliest is shared across both scan paths (no duplication of the OOM fix). No over-engineering. +- **DRY:** in_time_window and BoundedEarliest are correctly centralized across both export scan paths. The only real DRY violation is the 4-way duplicated secondary-checkpoint block (cohort/co-engagement/community/preference) noted under tech_debt. +- **CLEAN:** Mostly clean and neat. The one correctness smell is the `if events.is_empty() { break }` heuristic in read_session_journal_signals conflating 'corrupt frame' with 'frame produced no events for us', which is the root of the forward-compat drop bug. + +### `db-ops` — Query/replication ops, http, metrics _(7 files)_ +- **Completeness:** Checked error/edge paths: filter-node cap (retrieve:33-41, search:166-174), bounded HTTP request line + header drain (http:178-219), histogram overflow-above-bounds (histogram:72-81 + proptest 194-210). New behavior is tested (partition_id, degraded-derivation, fail-closed dual-write, well-formed-exposition parse). One real gap: FOR SESSION has divergent missing-session handling between retrieve and search — see finding. +- **Accuracy:** Histogram cumulative semantics correct (partition_point suffix-increment proven against naive rule; +Inf=count; no finite bucket on over-range). Atomics use Relaxed for stats, Release/Acquire at the checkpoint-staleness boundary correctly (mod.rs:277 pairs state_rebuild:537). No unwrap/expect/panic/unsafe in non-test code. No lock held across await (sync code). One partial-write-then-Err hazard in signal_for_tenant — see finding. +- **Tech Debt:** signal_for_tenant remote dispatch is explicitly unwired and fails closed with a tracking note to docs/specs/14 (CODING_GUIDELINES §12 honored). No orphan TODOs, no commented-out code, no magic numbers (MAX_FILTER_NODES, MAX_REQUEST_LINE_BYTES, CHECKPOINT_STALENESS_LIMIT_NS all named + documented). +- **Maintainability:** Names intention-revealing; functions focused; WHY-comments are strong (the macOS accept-inheritance comment at http:221-236 is exemplary). render_prometheus is long but flat and #[allow(too_many_lines)]-annotated. Clean. +- **Extensibility:** Histogram bounds are &'static parameters; metrics-add recipe documented at mod.rs:6-10; executor wiring is builder-chained. Appropriate abstraction, no YAGNI over-engineering. +- **DRY:** now_unix_nanos shared between staleness + render. write_metric_line dedups gauge/counter lines. as_micros()->u64 truncation note repeated verbatim in 3 hot paths (acceptable, each is local context). creator_embedding_slot unified via entity_embedding_slot. No harmful duplication. +- **CLEAN:** Clear/logical/neat. extract_path avoids a Vec alloc; bounded reads avoid OOM. One minor neatness gap: checkpoint_failures_total omits the partition_id label every other node-identity line carries — see finding. + +### `db-recovery` — State rebuild & session restore on restart _(14 files)_ +- **Completeness:** WARNING — rebuild paths cover all four relationship types, item indexes, text/suggestion/embedding indexes, and replication HWM; but the closed-vs-active session reconciliation on restart is missing (no dedup, no test). Schema-fingerprint and WAL-lag have thorough unit tests; rebuild_entity_state / rebuild_item_indexes have no direct unit test (covered indirectly by integration crash tests). +- **Accuracy:** CRITICAL — session restore can place one session_id in BOTH self.sessions (active) and closed_sessions when a snapshot committed but its Close journal record was lost/torn. Otherwise logic is correct: shared ItemIndexes::index eliminates write/rebuild drift, ascending-ts session replay reproduces live BucketedCounter rotations, saturating next_session_id advance, checked_add bounds on all deserializers, Release/Acquire seq publication is sound. +- **Tech Debt:** WARNING — rebuild_item_indexes indexes a truncated/colliding id when entity_id > u32::MAX instead of skipping (write path rejects such ids before any durable write); the rebuild's divergent 'warn-then-index-collision' handling is a latent corruption-on-corrupt-input path. +- **Maintainability:** Clean — intention-revealing names, focused functions, load-bearing WHY-comments (item_meta_entity_id factoring, created_at materialization rationale, marker-not-advanced-on-periodic-checkpoint). run_checkpoint_thread is long but cohesive and annotated. +- **Extensibility:** Clean — ItemIndexes bundle and item_meta_entity_id are the right shared abstraction for the likely change (new item index); RelationshipType::from_byte forces every match to be revisited on a new variant. No over-engineering. +- **DRY:** Clean — the write path and rebuild path were deliberately unified through ItemIndexes::index/scrub; item_meta_entity_id is the single 'which rows are item metadata' rule reused by every item scan. No meaningful duplication remains. +- **CLEAN:** Clean — no dead code, no needless clones in hot paths (Cow used in the write path), one documented-invariant expect() in non-test code (AgentId::new("restored")). No unsafe. + +### `db-sessions` — Sessions / collections / communities / notifications _(11 files)_ +- **Completeness:** One real gap: orphaned session start records are detected at open but never cleaned up or sealed into a snapshot (session_restore.rs:60-70). Notification-tracker eviction (prior CRITICAL) is now genuinely wired into the sweeper loop. Collection/community CRUD cover idempotency, u32-universe rejection, and reopen-durability with tests. +- **Accuracy:** BLOCKER: next_session_id is not advanced past durably-archived closed sessions on reopen, so a reused id silently overwrites a prior session's archived snapshot (verified with a real failing test). Membership stop-forward gate and notification check_and_record concurrency are correct (atomic CAS + record_lock). community_contribute records in-memory before persist_cell — minor memory-ahead-of-disk window, but WAL-backed so no data loss. +- **Tech Debt:** join_community/rejoin_community use a read-then-put on the membership registry that is not atomic against a concurrent leave (with_row read, then put); single-writer-per-(user,community) assumption is undocumented. No magic-number abuse; constants (MAX_CLOSED_SESSIONS, EVICT_BATCH_SIZE, NOTIFICATION_KEEP_DAYS) are named. +- **Maintainability:** Strong: intention-revealing names, thorough WHY-comments on every durability-ordering decision (durability-before-mutation rationale is explicit in collections.rs and communities.rs). session_signal is long (~160 lines) but linearly readable with clear stages. +- **Extensibility:** Snapshot serde is cleanly versioned (0x01/0x02/0x03) with shared bounds-checked cursor helpers; membership is JSON serde (flexible). Storage namespacing (COLLECTION_NAMESPACE sentinel, Tag-scoped prefixes) supports prefix-scoped rebuilds without over-engineering. +- **DRY:** Cursor helpers (read_u32/read_u64/read_len_prefixed_utf8/capped_capacity) are single-sourced across snapshot, start-record, audit, and saved-search codecs — a genuinely good de-duplication. narrow_collection_item_id centralizes the u32-universe guard. Minor: orphaned-start-key encode_key is repeated in 3 files but trivial. +- **CLEAN:** Clear and neat. Corrupt-input deserialization is uniformly rejected (None) with capped pre-allocation — no OOM-on-hostile-input vector. No unwrap/expect in non-test code (the one expect in session_restore.rs:122 is on a provably-valid literal AgentId). + +### `entities` — Entity model (Item/User/Creator) _(11 files)_ +- **Completeness:** Validation and edge paths are handled: dim checks (preference), per-item bounds checks on deserialize (collection/mod), from_byte returns None on unknown bytes (Visibility/RelationshipType/Tag), and Mute is documented PERSISTED-BUT-INERT with a tracking note. Tests include regressions + proptests; all 96 entity tests pass. Clean. +- **Accuracy:** One real correctness gap: PreferenceVectors::restore neutralizes NaN to 0.0 but does NOT re-normalize, so a torn checkpoint row yields a non-unit pref vector while cosine_similarity divides only by the candidate norm (assumes unit pref) — the function's own comment claims to prevent cosine poisoning but only handles NaN, not the broken unit-length invariant. No unwrap/expect/panic/unsafe in non-test code; DashMap snapshots avoid held shard locks. Otherwise accurate. +- **Tech Debt:** relationship.rs hand-encodes keys with magic 0x00/0x04 instead of encode_key + Tag::Rel, duplicating the storage key contract. Caller db/signals.rs:344 uses a bare `as u32` truncation that bypasses the observable narrow_item_slot helper this zone added (out-of-zone but adjacent). Low overall. +- **Maintainability:** Strong: intention-revealing names, focused <50-line functions, thorough WHY-comments (out-of-order decay, seen/hidden orthogonality, eviction policy rationale). +- **Extensibility:** Shared serialize_entity codec for user+creator, trait-abstracted StorageEngine for checkpoint/restore, adding a RelationshipType/Tag forces exhaustive match revisits. Right-sized, not over-engineered. +- **DRY:** serialize_entity unifies user/creator; collection bitmap is single source of truth (item_ids derived). One duplication: relationship key tag byte hardcoded vs Tag::Rel. +- **CLEAN:** Clear, logical, neat. narrow_item_slot observability helper and the total_cmp/NaN-determinism handling in co_engagement are exemplary. No dead code, no misleading names. + +### `governance` — Governance / community ledger _(13 files)_ +- **Completeness:** Checked: tombstone replay on open (db/mod.rs:613-636) covers both user+agent purge crash windows; per-accept persist + periodic + lifecycle checkpoint all wired; multi-entity/multi-signal collision regression has a test (community_ledger.rs:1038). Gap: no test or barrier asserting the per-accept contributor row is fsync-durable, while the rustdoc claims a 'ZERO loss window'. +- **Accuracy:** Checkpoint is a single atomic WriteBatch (community_ledger.rs:517-565) — the delete-then-put-outside-batch BLOCKER is fixed. purge_writer empty-agent guard correct (line 368). Watermark vs contributor suffix disambiguation by length is sound (8 vs >=30 bytes). One real defect: persist_cell/checkpoint never fsync, so the SOLE durable copy of contributor identity is committed-not-durable on power loss despite the 'durably written' claim. +- **Tech Debt:** Asymmetry: the agent-purge path re-checkpoints eagerly (remove_scope.rs:80) but the user-purge path does not (communities.rs:476) — both are correct via tombstone replay, but the inconsistency is a latent confusion. No hardcoded-config or copy-paste blocks of concern. +- **Maintainability:** Strong: intention-revealing names, thorough WHY-comments tying each guard to its CRITICAL/BLOCKER history, focused functions. purge_contributor/purge_writer share one purge_matching core (good). decode/encode codec is a single shared pair. +- **Extensibility:** Good boundaries: ContributorKey, RemoveScope enum, versioned GovernancePolicy/SharePolicy. WriterTag = String type alias is an appropriate seam. No over-engineering observed. +- **DRY:** to_durable_bytes centralizes tombstone serialize-or-error; atomic_u64 serde adapter shared by Membership+CapabilityToken; one suffix codec used by both checkpoint and restore. Minor: the 'collect keys -> sort -> re-get(k)' fold is repeated in decay_score_at/explain/aggregate_digest. +- **CLEAN:** Clear and neat. Minor needless re-lookup (HashMap::get after already holding the key set) in three fold sites; from_utf8_lossy on the writer tag silently mutates a corrupted tag rather than rejecting the row. + +### `query-executor` — RETRIEVE query executor & filters _(11 files)_ +- **Completeness:** Filter coverage is thorough — deferred filters (Saved/Liked/InProgress/SocialGraph/Min-MaxSignal/NearLocation/InCollection) are all extracted and applied, with loud rejection of negated and OR-nested deferred shapes. Gap: cohort rescore ignores profile.sort and is reachable with any profile, and notification-cap recording runs pre-pagination (over-records). Both lack negative tests. +- **Accuracy:** Strong on the suppression/inclusion u32-overflow invariant via entity_as_u32 — except Stage 2 index intersection (pipeline.rs:294) still uses a raw `as u32` truncation on the candidate side, which can wrongly RETAIN a >u32::MAX SignalRanked candidate that aliases onto a matched low slot. read_agg correctly propagates UnknownSignalType (loud). No locks held across await; no unwrap in non-test code. +- **Tech Debt:** Cohort rescore re-sort is hardcoded to descending-score, duplicating intent of the main sort path instead of reusing it. ANN strategy in RETRIEVE is a logged scan fallback (acceptable, tracked in comments). Offset pagination acknowledged as M3 replacement target. +- **Maintainability:** Excellent — intention-revealing names, generous WHY-comments explaining each guard's failure mode, functions mostly focused. stage3_score is long but `#[allow(too_many_lines)]` and well-sectioned. +- **Extensibility:** collect_matching centralizes the And/Or-recurse, skip-Not walker (W27) — good single point of change. PostFilterCtx cleanly shares deferred filters between RETRIEVE and SEARCH. Cohort rescore is the one spot not abstracted against the sort path. +- **DRY:** Good. Suppression block, deferred post-filters, and traversal rule are each single-sourced and shared with SEARCH. entity_as_u32 is the one canonical conversion — but pipeline.rs:294 bypasses it. +- **CLEAN:** Clear and neat overall. One misleading SAFETY comment at pipeline.rs:293-294 asserts truncation is 'correct throughout this module' while the rest of the module deliberately avoids exactly that truncation. + +### `query-search` — SEARCH pipeline & scope _(12 files)_ +- **Completeness:** All declared deferred filters (MinSignal/MaxSignal/NearLocation/InCollection/SocialGraph) are now applied via the shared post_filter module, with loud errors on unknown signals and OR/NOT-of-deferred rejection — verified against current source and regression tests. GAP: creator-search metadata filtering (CategoryEq/FormatEq on EntityKind::Creator) is processed against the wrong (item) index in Stage 2 and silently drops all creators; the m5p4 test tolerates 0 results so the gap is uncovered. +- **Accuracy:** No unchecked unwrap/expect/panic in non-test code; all u64->u32 bitmap conversions go through checked entity_as_u32 with documented polarity; cursor offset uses saturating_add; BM25 NaN neutralized with total_cmp; poisoned-lock recovery via into_inner. The one accuracy defect is the creator-namespace Stage-2 intersection (F1). +- **Tech Debt:** Minor: stats.filters_applied = query.filters.len() reports declared not applied count; exclude list applied after candidates_considered is set. No temporary hacks or hardcoded config found. +- **Maintainability:** Excellent — stage helpers are focused (<50 lines mostly), names intention-revealing, WHY-comments dense and accurate. The 8-stage execute() reads linearly as documented. +- **Extensibility:** Shared user_filter/post_filter modules make RETRIEVE/SEARCH drift impossible; ScopeResolver builder cleanly extensible. creator_embedding_slot threaded for future schema-driven resolution. No YAGNI over-engineering observed. +- **DRY:** Strong — collect_matching centralizes the four deferred extractors (W27), suppression/post-filter logic shared verbatim across both pipelines. entity_as_u32 is duplicated (pipeline.rs:75 and executor/mod.rs:508) but each is documented and tiny; the pipeline one only converts, the canonical applier lives in executor/mod. +- **CLEAN:** Clean and neat; no dead code, no needless clones on the hot path (combined_filter computed once — a fix from a prior pass). DEFAULT_EMBEDDING_SLOT constant centralizes the literal. + +### `query-suggest-fusion` — SUGGEST, RRF fusion, retrieve types/cursor _(9 files)_ +- **Completeness:** Edge/error paths are well covered: cursor rejects wrong-length, unknown-kind, non-finite scores, and overflowing offsets; retrieve validates limit + u32 creator-ID ceiling; fusion handles empty/single/one-sided lists. No orphan TODOs or commented-out code. One gap: no test asserting deterministic fused ordering on tied scores. +- **Accuracy:** RRF math is correct (1/(k+rank), 1-based ranks, union semantics verified by proptest). Cursor encode/decode is byte-stable and round-trips. The one real defect: fuse() orders tied scores by randomized HashMap iteration with no entity_id tie-break, so identical queries across process restarts can return tied results in different order — a determinism bug the team already guarded against in the BM25 sort but missed here. +- **Tech Debt:** Suggest.for_user is accepted, stored, and builder-exposed but never read ('reserved for future use') — a public no-op knob. NotificationCaps lives in retrieve/types.rs but is documented as only meaningful for one profile and not enforced to it. Both minor. +- **Maintainability:** Names are intention-revealing, functions are focused and well under 50 lines, and the WHY-comments (cursor discriminant byte, key_count CAS reservation, NaN neutralization rationale elsewhere) are genuinely load-bearing and excellent. Clean module boundaries. +- **Extensibility:** Cursor's tagged-discriminant wire format and RetrievalMode enum are the right abstractions for likely change. rrf_term extraction removes duplication cleanly. for_user reserved field is mild speculative generality (YAGNI) but harmless. +- **DRY:** rrf_term centralizes the rank-score formula across fuse() and route_results (verified by rrf_term_matches_inline_formula). Limit-range check [1,500] is duplicated between RetrieveBuilder::build and Retrieve::validate, but that duplication is intentional defense-in-depth and documented. No problematic duplication found. +- **CLEAN:** Code is neat and efficient. fuse() pre-sizes the HashMap; cursor uses a fixed 17-byte stack buffer. The only CLEAN smell is the inconsistent tie-handling: BM25 sort uses total_cmp for a true total order, but fuse() and trending sort use partial_cmp/cmp with no secondary key, leaving tied order to randomized map iteration. + +### `ranking` — Ranking: profile engine, scoring, diversity _(16 files)_ +- **Completeness:** Pipeline stages 2/4/5/6 (exclude/boost/penalty/gate), decay, sort modes, and diversity relaxation are all implemented with tests. Gaps are intentional/documented: per-user penalty multiplier (spec §6 USER_PENALTY_MULTIPLIER=3.0) and topic_diversity/category_min are not wired (reserved fields). Shuffle and Hot are honestly documented as reduced (no user/time seed; uniform age). No untracked TODO/FIXME or empty-catch found. +- **Accuracy:** Scoring math is correct and order-preserving; NaN is neutralized at construction + total_cmp sort + sentinel-aware normalize (verified against the prior Shortest/DateSaved regression — now fixed). No data races (executor is stateless over an immutable &SignalLedger). Genuine issue: the threaded query clock `now` is never used for any ledger read, so per-candidate decay/window reads each call Timestamp::now() — non-deterministic within a query. +- **Tech Debt:** Trending/cohort_trending double-apply velocity (sort formula base + redundant boosts), order-neutral but redundant. Social-graph trending path bypasses degradation substitution (documented no-op). Stale doc comment claims read_decay_score_at is 'out of scope' though it now exists in the ledger. +- **Maintainability:** Strong: intention-revealing names, focused functions, heavy WHY-comments tying each decision to spec sections and prior-review findings. finalize/normalize/read_agg are each well under 50 lines. One doc comment (mod.rs:181-184) is now factually wrong about ledger capabilities. +- **Extensibility:** Right abstraction: profiles are data (registered/validated/versioned), Sort/SignalAgg/Boost are open enums, DiversityConstraints uses a builder with reserved fields for future constraints. extra_boost closure cleanly factors the personalized vs base overlay. No over-engineering. +- **DRY:** read_agg/read_agg_for_sort share one path; the COARSE_VELOCITY_WINDOW constant is shared between read_agg and the social-graph path to prevent drift; score_candidates collapses the prior duplicated base/personalized loops. No material duplication in production code. +- **CLEAN:** Clear and neat. Session-keyword lowercasing and alphabetical-title reads are hoisted out of hot loops. Pure formulas are isolated and unit-tested. Minor: spec-vs-code score-on-tie semantics (0.5 vs 1.0) and shuffle/Hot reductions are the only rough edges, all order-neutral. + +### `repl-cluster` — Replication: tenant/shard/migration/reconcile/control _(11 files)_ +- **Completeness:** Mostly complete with edge/error paths and strong tests (crash-survival, fault-injection, proptests). Gaps: gc_source is a state-only flip that never reclaims source-shard data; tenant_wal_dir is dead code (no production caller) so per-tenant storage isolation does not actually exist beyond rate-limit/routing metadata. Both are documented-scope-adjacent but the GC naming overstates behavior. +- **Accuracy:** Two real correctness/durability holes found: (1) finalize() persists with the non-strict swallow-error path yet its own comment claims a crash cannot revert the tenant — but gc_source never re-persists, so a swallowed finalize-checkpoint error silently reverts a finalized tenant to DualWrite on crash; (2) enter_dual_write rollback restores a FULL-router snapshot, clobbering any concurrent migration's entry for a different tenant. Rate-limiter CAS, advance_atomic_max, jump-hash, range validation, reconcile LWW/CRDT merge are all correctly reasoned. +- **Tech Debt:** Low-moderate. control.lag_for is single-shard-only (documented, debug_assert-guarded). gc_source carries a misleading name for a no-op-on-data transition. Migration timing/GC is a standalone primitive not wired into a production coordinator (with_persistence only exercised in tests). +- **Maintainability:** Strong. Intention-revealing names, focused functions under 50 lines, thorough WHY-comments on every concurrency primitive. The two durability comments (finalize line 209-211, migration struct doc) overstate guarantees relative to the code — a maintainer would trust them. +- **Extensibility:** Good. Transport trait-abstracted, routing strategy enum is open for Hash/Range/Single, RegionId/ShardId newtypes. Migration state machine is a clean primitive. No obvious YAGNI over-engineering. +- **DRY:** Good. find_shard_assignment shared between pinned and dual-write paths; advance_atomic_max centralized across the two HWMs; build_channel_endpoints reused for session transport. No significant duplication. +- **CLEAN:** Clean and efficient. Lock-free hot paths (atomics, DashMap), no needless hot-path allocation, consistent abstraction levels. One dead helper (tenant_wal_dir) and one misleadingly-named transition (gc_source) are the only smells. + +### `repl-crdt` — Replication CRDTs (HLC/LWW/PN-counter/signal-state) _(7 files)_ +- **Completeness:** Strong. Every CRDT type carries example + proptest law coverage (commutativity/associativity/idempotency), HLC overflow/clamp edges are tested, and from_node_contribution's score/timestamp and bucket round-trip contracts are now pinned by tests. Gap: the LWWRegister commutativity proptest asserts a law that does not hold universally (equal-timestamp/different-value), and apply_crdt_state's multi-lambda reconciliation has no test. +- **Accuracy:** Mostly sound. Both prior CRITICALs are genuinely fixed: Hlc::update() now carries logical overflow into the wall (verified hlc.rs:310-316 + test update_strictly_advances_when_logical_saturated), and crdt_contributions passes hot.stored_score(0) not a decayed score (core.rs:608), eliminating the double-decay. One real residual: LWWRegister::merge is non-commutative under an identical full HlcTimestamp with differing values (verified by standalone repro) — unreachable for honest HLC-stamped writes, but an unstated/unenforced precondition. Plus a multi-decay-rate fidelity gap when CRDT state reconciles into multi-lambda hot slots. +- **Tech Debt:** Low. No magic numbers (LOGICAL_BITS/MASK/MAX, MAX_WALL_MS, NANOS_PER_SEC all named). node_buckets stores a full PNCounter per single-key node — over-allocation is documented and deliberately accepted to reuse the audited merge. No temporary hacks. +- **Maintainability:** Excellent. Intention-revealing names, dense WHY-comments on every non-obvious branch (overflow carry, key-aligned lookup, LWW tiebreak, cross-map invariant). All functions <50 lines and single-purpose. A reader can follow the convergence argument from the doc comments alone. +- **Extensibility:** Good. Shared lww_other_wins helper is the right seam for the two LWW call sites; forward_decay_step kernel is reused not re-implemented; storage engine untouched. CrdtSignalState's single-lambda model is the one place a likely future change (multi-rate reconciliation) is not yet abstracted. +- **DRY:** Good. The prior DRY drift (LWWRegister::write reimplementing the decision) is fixed — write and merge both route through lww_other_wins (lww_register.rs:91,104) with a test asserting they agree. Decay math centralized in forward_decay_step. No duplicated constants. +- **CLEAN:** Clean. force-set/merge paths do no needless clones (clone_from on the LWW value, Copy inserts in signal_state, per-node-max in PNCounter). No dead code, no mixed abstraction levels, no panics/unsafe in non-test code (all unwrap/expect/clone hits are test-only; the lone 'unsafe' token is a doc-comment word). + +### `repl-shipping` — Replication: shipper / receiver / transport _(7 files)_ +- **Completeness:** Mostly complete. Shipper, receiver, transport, lag, idempotency, segment_id all have edge/error paths and extensive tests (fault injection, proptests, overflow, atomicity). Gap: when the receiver thread halts mid-life (corrupt segment or follower-WAL IO fault) there is no test or wiring that makes that death observable while the node stays open — the existing tests only observe the halt via join() at shutdown. +- **Accuracy:** Strong. Verified the BLOCKER-6 durability fix is real (apply_replicated_event appends to the follower WAL before in-memory mutation, core.rs:451-476), mid-payload all-or-nothing staging (receiver.rs:210-321), seq-overflow rejection (receiver.rs:240-253), idempotency floor (receiver.rs:261), and the WAL-position-preserving community filter (shipper.rs:450-510). One accuracy/operability defect: receiver death is not surfaced to health_check. +- **Tech Debt:** Low. control.rs lag_for is single-shard-only but documented and debug-asserted. MAX_PAYLOAD_BYTES is a named const. No copy-paste or hardcoded magic on the hot path in this zone. +- **Maintainability:** High. Functions are focused (<50 lines except the documented spawn_shipper loop), names are intention-revealing, and the WHY-comments are unusually thorough and accurate (they match the code). +- **Extensibility:** Good. Transport is a clean trait with a blanket Arc impl; TransportError's transient/permanent/too-large taxonomy is the right seam. Not over-engineered. +- **DRY:** Good. advance_atomic_max is shared by ReplicationState and the lag gauge; build_channel_endpoints is shared by WAL and session transports; last_seq_in_segment / count_events_in_segment / filter_segment_drop_local share the same decode-scan loop shape (mild, acceptable). +- **CLEAN:** Clean and efficient. Lock-free atomics on the read path, DashMap for operator-visible shipper state, staging Vec only on the cold replication apply path. One needless-clone note: segment_bytes.clone() per peer per send in the shipper (shipper.rs:330) is on the cold ship path, acceptable. + +### `schema` — Schema builder / validation / migrations _(10 files)_ +- **Completeness:** Validation is thorough for what exists (signals, decay, windows, velocity, embedding slots, text fields, policies), each rejection path has a unit test. Two gaps: embedding dimensions enforce only `!= 0`, not the documented [2,4096] bound; the spec's migration/profile/cohort surface lives outside this module or is absent, and the spec header still says 'Implemented (M0-M8)' while flagging itself a pre-API sketch. +- **Accuracy:** Clean. Fingerprint hashes name+kind+decay+slot-dims; the 0x00..0x02 kind byte is a reliable delimiter after each non-empty lowercase signal name, so signal-name boundary collisions are impossible. Lambda precompute, WindowSet empty-check (runs on pre-dedup Vec, correct), and the saturating clock-anomaly Timestamp::now() are all correct. No data races (Schema is immutable post-build). +- **Tech Debt:** Low. One duplicated string literal (`entity_id`) split across two crate-internal constants coupled by comment only. No magic numbers in hot paths, no commented-out code, no orphan TODOs. +- **Maintainability:** Strong. Intention-revealing names, focused functions, and unusually good WHY-comments (the fingerprint-exclusion rationale and the `target` field's load-bearing-for-durability note are exemplary). build() is long but explicitly annotated and is a flat sequence of independent validation passes. +- **Extensibility:** Right-sized. Closed Window/EntityKind enums are deliberate (fixed storage layout) and documented as such. The builder pattern cleanly admits new signal attributes (positive_engagement was added without churn). Not over-engineered. +- **DRY:** One real violation: RESERVED_TEXT_FIELD_KEY (schema/error.rs:318) and ENTITY_ID_FIELD (text/index.rs:28) are independent `"entity_id"` literals. The prior EntityKind->byte 3-way duplication is now correctly collapsed to a single fingerprint_byte(). +- **CLEAN:** Neat and consistent. No dead code (the seemingly-unused `target` field is documented as fingerprint-load-bearing), no needless allocation in any path, no misleading names. Production schema code contains zero unwrap/expect/panic/unsafe — every fallible path returns a typed SchemaError. + +### `session` — Session & agent context / policies _(14 files)_ +- **Completeness:** Serde/journal codecs are exhaustively corruption- and roundtrip-tested; torn-tail vs corruption is distinguished with observability. Real gap: no test asserts a RESTORED session is expired by the sweeper or rejected by policy on its true age — the exact 'TTL honored across crashes' focus item is uncovered (see BLOCKER). +- **Accuracy:** Decay math correctly delegates to the canonical forward_decay_step kernel; out-of-order pre-decay and timestamp non-regression are correct; bounds checks and capped allocations are sound. The one real correctness defect: TTL/lease enforcement reads the non-durable monotonic Instant (started_at), not the durable started_at_ns, so a restored session's age resets to ~0. +- **Tech Debt:** SessionState's 14-field literal is hand-constructed at 2 production sites (start_session, restore_session_wal_events) with no constructor — they must be kept in lock-step manually. +- **Maintainability:** Strong: nearly every ordering/durability decision carries a WHY comment (e.g. sessions.rs:163-170 on started_at_ns, snapshot.rs:112 on deterministic ordering). Functions are focused; names are intention-revealing. +- **Extensibility:** Good: session-journal frames (v1/v2) and snapshots (v0x01-v0x03) are explicitly versioned with forward/backward-compatible decode; optional Signal fields use one shared push_optional framing. +- **DRY:** Byte-cursor helpers (read_u32/u64/utf8/len_prefixed, write_len_prefixed, capped_capacity) are correctly single-sourced across the whole session slice incl. saved_search; write_audit_entry shared by snapshot+audit codecs. Only duplication is the SessionState literal. +- **CLEAN:** Clean overall. Minor: SessionHotState::frozen_score uses Relaxed while every live read uses Acquire (benign — archive time, single-threaded close), and current_score recomputes wall-clock dt; no dead code, no needless hot-path allocation in this slice. + +### `signals` — Signals: types, decay, velocity, windowed agg _(14 files)_ +- **Completeness:** Strong. Decay (in-order/out-of-order), velocity-as-count/duration, and all five windows (incl. real 30d day-bucket tier) are implemented; EWMA velocity and RelativeVelocity/Ratio are documented-not-implemented and explicitly rejected at profile registration (spec §5 note matches code), so there is no silent gap. NaN/Inf weight rejected at the public db.signal() boundary. Crash-injection points wired on the write and checkpoint paths. No orphan TODO/FIXME found. +- **Accuracy:** Sound. forward_decay_step kernel is mathematically exact and is the single source of truth; the hot-tier CAS-retry re-reads last_update_ns inside the loop (regression-tested for the stale-timestamp over-add). maybe_rotate serializes the whole minute→hour→day cascade behind the minute-CAS, so concurrent rotation cannot mis-apply an aggregate. Read-time rotation on &self via atomics is sound under DashMap read refs. No unwrap/expect/panic in non-test code; no lock held across await (no async here). +- **Tech Debt:** Low. NANOS_PER_SEC and NS_PER_{MIN,HOUR,DAY} are centralized; bucket sizes are named consts; the reverse-sum idiom is extracted to one helper. Restore's full-keyspace scan is documented as a known perf (not correctness) limitation tied to the key layout. +- **Maintainability:** Excellent. Dense but accurate WHY-comments on every non-obvious boundary (in-progress-hour fold, write-side vs read-side overlap bounds, 7-day seam slack). Field-offset codec is derived from named constants with compile-time size asserts. Functions are focused and under the size limit. +- **Extensibility:** Good. Checkpoint format is versioned (V1/V2 entry, V1/V2/V3 meta) with forward/backward-compatible decode. Decay kernel is a pure function reused by every tier. Window is a closed enum by design (documented rationale). No obvious over-engineering. +- **DRY:** Good. derive_signal_ids_and_lambdas is the single id/lambda derivation shared by global+cohort ledgers; get_or_create_entry is the single entry construction site; apply_event_local is the shared incremental-update core; sum_last_n_buckets is the one ring-sum. Field offsets defined once for both serialize and deserialize. +- **CLEAN:** Clean. Intention-revealing names, no dead code, Relaxed/Acquire/Release orderings each justified in module docs, allocation-free hot path. The clone-free integrity-hash iter path is a genuinely good efficiency choice. + +### `storage-engine` — Storage KV engine (fjall/memory/keys) _(8 files)_ +- **Completeness:** Strong. Edge/error paths handled; round-trip, ordering, atomicity, crash, isolation, reopen all tested. Gap: no tracing spans despite CODING_GUIDELINES §11.2 naming storage ops non-negotiable; no test asserting scan_prefix surfaces a backend get-error mid-stream (the FjallBackend per-guard into_inner error path is untested). +- **Accuracy:** Correct. Verified against fjall 3.0.2 source: prefix() snapshots at SeqNo::MAX (streaming claim holds), OwnedWriteBatch commits under one seqno (atomicity real), insert() returns Err (no panic) on oversized/closed. map_fjall_err preserves io::Error kind + conservatively halts on unknown variants. No unwrap/expect/panic in non-test code; no unsafe; no lock held across await (sync API). +- **Tech Debt:** Low/acknowledged. scan_prefix(&[]) full-keyspace cost is documented as a known perf limitation at every restart caller; InMemoryBackend eager-materialize ceiling is documented and test-scoped. StorageError::BatchConflict is defined but never produced by either backend (dead variant, minor). +- **Maintainability:** Excellent. Intention-revealing names, every non-obvious decision carries a WHY comment grounded in the durability invariant, functions are small and focused, Tag::ALL is a single source of truth with a drift-guard test. +- **Extensibility:** Right-sized. StorageEngine trait is the clean boundary; FjallAtomicBatch deliberately NOT trait-abstracted with a documented rationale (cross-keyspace is fjall-specific, InMemory has no keyspaces) — correct YAGNI call, not over-engineering. +- **DRY:** Good. open_keyspace dedupes the three-keyspace open path; rotate_only/flush_all share one fsync; MAX_TAG_BYTE derived from Tag::ALL removed the prior dual-range proptest drift. as_byte/from_byte parallel tables are guarded by all_byte_mappings_consistent. +- **CLEAN:** Clean, logical, neat. No dead code beyond the unused BatchConflict variant; allocations are intentional (Vec capacity pre-hinted in encode_key); abstraction levels are consistent within each file. + +### `storage-indexes` — Secondary indexes (range/bitmap/filter) _(8 files)_ +- **Completeness:** Edge/error paths are well covered: inverted bounds, empty intervals, truncated/corrupt persisted keys, unknown equality fields, and the predicate-vs-bitmap split all have explicit handling and tests. 80 zone tests pass. The deferred-filter 'return-universe' contract is fully guarded by reject_negated/reject_or validators wired into both pipelines. Gap: RangeIndex::load_from_kv_pairs does not reject trailing bytes after a fixed-width value (W). +- **Accuracy:** Range, bitmap, AND/OR/NOT, and selectivity logic are correct. is_empty_interval correctly clamps every BTreeMap::range panic case (lo>hi, equal-excluded). into_predicate uses checked u32::try_from (no truncation). Lock poisoning is recovered via into_inner rather than panicking. No unwrap/expect/unsafe in non-test code (the one unwrap_or_default in evaluate_and is provably non-empty). NOT-over-universe contract is sound given the universe only grows (no live item-delete path). Clean. +- **Tech Debt:** Minimal. RANGE_KEY_PREFIX / SUFFIX_TAG / ENCODED_KEY_PREFIX_LEN are centralized so write and read sides cannot drift. No magic numbers, no copy-paste blocks, no temporary hacks. The RangeKeyCodec trait removes the only duplication between u32/u64 persistence paths. +- **Maintainability:** Names are intention-revealing, functions are small and focused (<50 lines), and the WHY-comments (W22 scrub, NOT-over-universe contract, u32 bound enforcement) explain non-obvious invariants well. Consistent structure across bitmap and range indexes. +- **Extensibility:** RangeKeyCodec is the right abstraction for adding new numeric value widths. FilterExpr's deferred-variant design cleanly separates index-backed from post-filter evaluation. try_eq returns a typed error for unknown fields, leaving room for a generic field-equality variant. Not over-engineered. +- **DRY:** Strong. The key-prefix constants, the collect_matching walker (single shared traversal rule), and the RangeKeyCodec trait all eliminate would-be duplication. selectivity's range() and total_count()'s union share the same union pattern but that is acceptable. +- **CLEAN:** Clear and neat. One efficiency note: RangeIndex::total_count() and BitmapIndex::total_count() compute a full union of every value bitmap on each call, and RangeIndex::selectivity() materializes the whole range bitmap — O(n) per call (SUGGESTION). No dead code, no misleading names, no needless clones on a hot path. + +### `storage-vector` — Vector index (usearch/brute/registry/lifecycle) _(10 files)_ +- **Completeness:** Mostly complete: deserialize hostile-input guards, soft-delete tombstones, and rebuild-from-store are all present and tested. Gap: rebuild aborts on the FIRST bad embedding (one corrupt/mismatched vector drops every embedding for that entity kind from ANN); no test for that path. UsearchIndex has tests but no production wiring. +- **Accuracy:** Logic is correct. No unwrap/expect/panic in non-test code; RwLock poison is mapped to Backend errors, not unwrapped. Durable-first ordering in lifecycle ops is sound and crash-window-reasoned. BruteForceIndex::deserialize uses fully-checked arithmetic against the real buffer length — no OOM/panic on hostile count/dims (verified by tests). No locks held across await (sync code). +- **Tech Debt:** UsearchIndex is dead in production — every path (registry rebuild, write auto-register, query pipeline, users.rs) hardcodes BruteForceIndex. The 'production HNSW' engine and its save/load/view/tombstone machinery are exercised only by tests. ef_search per-query override is a documented permanent no-op. +- **Maintainability:** Strong. Intention-revealing names, focused functions, excellent WHY-comments (SECURITY block in deserialize, durability-invariant docs on lifecycle ops). Single-source-of-truth consts for HNSW defaults and the ARCHIVED_PREFIX marker. +- **Extensibility:** VectorIndex trait is the right boundary and is clean. But there is no backend selector to actually choose UsearchIndex vs BruteForceIndex from schema/config — the abstraction exists yet only one impl is reachable, so the extensibility is theoretical. +- **DRY:** Good. read_*_le, validate_dimensions, ef_search_unsupported_warn, ARCHIVED_PREFIX, and DEFAULT_* consts are all centralized. select_top_k and cmp_distance are shared. No notable duplication. +- **CLEAN:** Clean and neat. Distance comparator, top-k selection, and tombstone handling are clear and efficient (select_nth_unstable_by for k<0) instead of returning the typed Internal error the doc promises. Otherwise correct: circuit breaker single-probe invariant sound; no std Mutex/RwLock guard held across .await (the inbound_rx guard is held across a synchronous block_on by design, single-consumer); TLS server-cert verification is NOT disabled (no danger_accept_invalid_certs); overflow-checked u32->u16 conversions on the wire; #![forbid(unsafe_code)]. +- **Tech Debt:** rustls-pemfile is declared in Cargo.toml but unused in src (TLS material is read via Identity/Certificate::from_pem) — violates the 'minimal, intentional, auditable deps' guideline. GrpcTransportConfig::default sets insecure=true; tidal-server's cluster path hardcodes insecure=true with no TLS knob (out-of-zone but relevant to 'no insecure defaults'). +- **Maintainability:** Strong. Intention-revealing names, focused functions (<50 lines), and unusually thorough WHY-comments on every subtle decision (shutdown latch, codec limits, backpressure classification, permanent-vs-transient mapping). +- **Extensibility:** Appropriate. stream_segments and heartbeat are honestly documented deferrals returning explicit Status::unimplemented / minimal-real-behavior rather than silent stubs. Transport trait boundary keeps network code out of the core crate. +- **DRY:** Acceptable. The codec max-size setting and the TLS-asymmetry rejection are duplicated across client/server, but each duplication is intentional (both ends must enforce independently) and documented. MAX_PAYLOAD_BYTES is mirrored from the engine with a compile-time cross-check. +- **CLEAN:** Clean overall. Minor: server_terminated is effectively dead code (no callers); CircuitState::HalfOpen carries no in_flight field despite the module doc referencing one (the invariant is enforced by the variant itself, so the doc is slightly misleading). + +### `tidal-server` — tidal-server: cluster server / router / scatter-gather _(11 files)_ +- **Completeness:** Cluster routes, scatter-gather merge, partial-failure, and config loading are well-covered with tests. GAP: cluster router omits the timeout + concurrency limits the standalone router enforces; sharded write handlers run blocking DB work directly on the reactor with no offload; the multi-process cluster_e2e test asserts nothing across process boundaries (every node runs all regions in-process, queries only hit node 0). +- **Accuracy:** Scatter-gather deadline/degradation logic, NaN-safe sort, dedup, and per-creator reconciliation are correct and well-reasoned. Two real correctness gaps: cluster /signals returns 204 while follower ship is best-effort (send errors swallowed in the engine), and scatter-gather spawns one unbounded detached OS thread per shard per request with no global cap on the un-throttled cluster router. +- **Tech Debt:** cluster.rs (1112 lines) and scatter_gather.rs (1678 lines) each carry an acknowledged, deferred split with a tracking note — reasonable. Hardcoded 2MB body limit literal duplicated instead of reusing router::BODY_LIMIT_BYTES. +- **Maintainability:** Strong: intention-revealing names, thorough WHY-comments, shared dto/health modules prevent standalone/cluster drift. Functions mostly focused; a few handlers exceed 50 lines but read clearly. +- **Extensibility:** offload abstraction (read pool vs runtime-free write pool) is the right boundary. ShardRouter delegation prevents write/read hash divergence. Topology YAML is data-driven. No over-engineering observed. +- **DRY:** Good factoring via merge_and_assemble (shared RETRIEVE/SEARCH tail), dto.rs, health.rs, offload_read. Minor: 2MB body-limit constant and ScatterGatherInfo construction block are duplicated. +- **CLEAN:** Clean and logical overall. The duplicated 2MB literal and the cluster router's missing middleware stack are the main blemishes; dead-ish ShardedFeedQuery.deadline_ms clamp path is fine. + +### `tidalctl` — tidalctl: CLI _(11 files)_ +- **Completeness:** Largely complete — every command has integration tests, all five subcommands honor the documented exit-code contract (status/diagnostics exit 2 on degraded WAL, verified by tests). One gap: scope-stats' in-band integrity signal (`complete`) does not cover the case where diagnose_wal itself fails on a corrupt checkpoint (W1). recover's `--verify-only=false` rejection is correct (no silent no-op). +- **Accuracy:** Strong. No unwrap/expect/panic in non-test code (crate forbids unsafe, denies unwrap_used). Saturating arithmetic on checkpoint-age nanos (diagnostics.rs:41-48) and uncheckpointed-bytes (diagnostics.rs:192) is correct against overflow/hostile timestamps. CliError::to_json has an infallible fallback. One accuracy defect: scope-stats reports `complete: true` when integrity could not actually be verified (W1). One minor swallow: list_segments error → 0 (S1). +- **Tech Debt:** Low. No magic numbers (REASON_* and OFFLINE_UNAVAILABLE are single-sourced constants). recover's `verify_only` is a documented forward-compat stub, not dead code. The prior review's three Accuracy items (status exit code, fabricated checkpoint_age 0, text-index 0) and the main.rs-too-large item are all genuinely remediated at HEAD. +- **Maintainability:** Excellent. main.rs split into commands/ + json + wal_state modules; every function under 50 lines except the deliberately-flat format_pretty (annotated). Names are intention-revealing; WHY-comments are abundant and accurate (verified against engine behavior). +- **Extensibility:** Appropriate. The five-command enum + per-module run() is the right shape; DirsOutput is shared between status and paths so the projection cannot drift (guarded by a test). recover's reserved flag is YAGNI-conscious (rejected, not half-built). No over-engineering. +- **DRY:** Good. WalState::from_report vs gather_wal_state are two intentional IO paths (one reuses a held report, one scans) and the divergence is documented. DirsOutput is the single source for the six-path projection. Offline-unavailability reasons are single-sourced and a test asserts JSON/pretty agreement. The two pretty-formatters (recover, diagnostics, scope-stats) repeat the section-heading idiom but are not worth extracting. +- **CLEAN:** Clean and neat. All JSON flows through one serde_json path (control-char escaping hole closed, test-guarded). BTreeMap keeps offline_unavailable output order stable. No needless clones in any hot path (this is a one-shot CLI). The `unwrap_or(0)` on list_segments (S1) is the one slightly-misleading construct. + +### `wal-format` — WAL record/format (de)serialization _(4 files)_ +- **Completeness:** Strong. Both formats cover encode/decode, version compat (v1/v2/v3 batch; legacy-v1/v2 session), torn-tail vs corruption distinction, and have property tests + per-byte corruption tests. Gap: no encode-side validation of variable-length string lengths against the u16 prefix (see finding 1); no test exercises a present-but-short v2 optional field (finding 3). +- **Accuracy:** Logic is correct on the read path. Length fields read from disk are bounds-checked (session.rs:293, reader.rs:197) before any slice, and the only length-driven allocation in decode_batch is Vec::with_capacity(event_count) capped at MAX_EVENTS_PER_BATCH=256 — no OOM vector from a hostile length. BLAKE3 covers header[0..32]+events (batch) and [marker,version,type]+payload (session). No unwrap/expect/panic in non-test code except one provably-infallible documented expect (batch.rs:574). One encode-side silent-truncation defect (finding 1). +- **Tech Debt:** Low. Offsets/sizes are named constants (HEADER_PAYLOAD_LEN_OFFSET, SESSION_*), owned in one place to prevent drift between scanner and decoder. No magic numbers of concern. The u16-length encode without a guard is the one latent debt item. +- **Maintainability:** Excellent. Intention-revealing names, focused helpers (read_u64_le/read_str_field/decode_typed_record), and unusually thorough WHY-comments explaining the corruption-vs-torn-tail contract. Functions are <50 lines except decode_signal_record (~80) which is linear and readable. +- **Extensibility:** Good and appropriately scoped. Version dispatch (event_size_for_version, v2 forward-compat unknown-type skip, reserved-byte rejection) is the right abstraction for the likely change (new format version) without YAGNI over-engineering. +- **DRY:** Good. parse_base shared by legacy/v3 decode; push_optional centralizes the [flag][payload] framing; compute_checksum/session_record_checksum each single-sourced. Minor: the u16 length-prefix write idiom (len as u16).to_le_bytes() + bytes is repeated 4x in encode_session_event without a push_str_field helper mirroring read_str_field (finding 2). +- **CLEAN:** Clear, logical, neat. No dead code, no needless clones in these files (ann_for_wal clone lives in sessions.rs, out of zone). Reserved-byte handling and the layered corruption semantics are precise. + +### `wal-recovery` — WAL read / replay / dedup _(8 files)_ +- **Completeness:** Strong. Edge/error paths (empty dir, empty segment, torn tail, mid-batch corruption, sequence overflow, checkpoint filter) all have tests. Gap: no test or detection for corruption in a NON-final segment when a later intact segment exists (see CRITICAL finding). No TODO/FIXME, no commented-out code, no empty catches. +- **Accuracy:** Mostly correct. Two-phase validation (bounds then BLAKE3) is sound; checked_add guards corrupt first_seq overflow; saturating_add guards next_seq wrap; dedup record-after-fsync ordering is correct. One real defect: recover() truncates a torn/corrupt tail on ANY segment and keeps replaying later segments with no gap detection -> silent data loss + sequence gap on a corrupt sealed segment. +- **Tech Debt:** Low. REPLAY_EVENTS_PER_SEC / CHECKPOINT_RESTORE_SECS are documented coarse constants, not magic numbers. Session-format offset constants are named and centralized. No copy-paste; scan logic shared via scan_segment_inner. +- **Maintainability:** Excellent. Functions are focused (<50 lines except the necessarily-long decode_session_events_with_diagnostics and diagnose_wal, both flagged with allow attributes and well-commented). Names are intention-revealing. WHY-comments are dense and load-bearing (e.g. why record-after-fsync, why > vs >= replay predicate). +- **Extensibility:** Good. Format versioning (v1/v2/v3) handled cleanly; session frame v1/v2 forward-compat is explicit. scan_segment_inner is the single shared scanner feeding readonly/summary/recover, preventing drift. No YAGNI over-engineering. +- **DRY:** Strong. SegmentScan/scan_segment_inner is the one decode loop reused by recover, read_all_events, diagnostics, and readonly scan. payload_len_from_header is the single owner of the header offset. push_optional centralizes optional-field framing. Diagnostics and recover deliberately share scan_segment_summary so the >/>= predicate can't diverge again. +- **CLEAN:** Clean. No dead code, no needless allocation in the decode path (read_to_end once per segment is necessary). Abstraction levels are consistent. The only blemish is the asymmetry where the session journal surfaces corruption_detected but the signal WAL recover() surfaces nothing for a corrupt non-final segment. + +### `wal-write` — WAL write path & compaction _(11 files)_ +- **Completeness:** Solid. Every error path in the commit routine resolves caller replies (success, dedup sentinel, and error). Crash-recovery, rotation, oversized-batch, clock-anomaly, sequence-overflow, and active-segment-clamp cases all have dedicated tests. Gap: no test exercises a flush failure that occurs DURING segment rotation (rotate() opening a new file failing), though the error is propagated correctly. No TODO/FIXME or commented-out code found. +- **Accuracy:** Strong. fsync ordering is correct (data via sync_data; dir entry via sync_all on create/rotate/delete/checkpoint-rename). The active-segment clamp in both compact_wal_online and handle_aux_command genuinely prevents unlinking the live inode. Dedup is recorded only after durable flush, so retries of a failed batch are accepted. Sequence overflow on corruption-controlled first_seq is checked. No unwrap/expect/panic in non-test code; no unsafe; no lock held across await (writer is a dedicated thread, no async). One real durability caveat on macOS (see findings). +- **Tech Debt:** Two items: WalError::SegmentFull is dead (defined+tested, never constructed). Magic-number 0 dedup sentinel is well-documented. No hardcoded config beyond named consts. +- **Maintainability:** Excellent. Intention-revealing names, the flush_batch/partition_dedup/handle_aux_command extraction keeps the steady-state and shutdown-drain paths from diverging, and the rustdoc explains the WHY (hazard analysis) at every durability-critical site. run_writer is over 50 lines but is cohesive and annotated with #[allow(clippy::too_many_lines)] with justification. +- **Extensibility:** Right-sized. Shard/region identity is threaded through the writer config and filename format (v1/v2) for the cluster path without over-engineering. The CommandOutcome enum centralizes per-command dispatch across three loops. No speculative abstraction. +- **DRY:** Good. The two error-notify loops (flush_batch err arm and the shutdown-drain already-failed arm) duplicate the 'send synthetic Io error to each reply' pattern, but the duplication is small and the contexts differ. compact_wal/compact_wal_online correctly share compact_segments. delete_segments_before (segment.rs) and compact_segments (compaction.rs) are two near-identical delete loops that differ in dir-fsync — see findings. +- **CLEAN:** Clear and neat. One doc-accuracy issue: segment.rs sync() rustdoc overstates durability on macOS ('genuinely sufficient', 'fully durable') where fsync does not flush the drive cache without F_FULLFSYNC. + + +--- + +_Generated 2026-06-08 from workflow run `wf_79dbc004-39f` (29 Opus zone reviewers + 56 Sonnet adversarial verifiers). BLOCKERs re-confirmed against source by the orchestrator._ diff --git a/tidal-net/Cargo.toml b/tidal-net/Cargo.toml index 8e8b9c9..3389963 100644 --- a/tidal-net/Cargo.toml +++ b/tidal-net/Cargo.toml @@ -35,7 +35,6 @@ tonic = { version = "0.12", features = ["tls", "tls-roots"] } prost = "0.13" tokio = { version = "1", features = ["rt-multi-thread", "sync", "time"] } tokio-stream = "0.1" -rustls-pemfile = "2" # Direct rustls dep with an explicit crypto provider. tonic pulls rustls only # transitively (no provider feature in tidal-net's own closure), so the # process-level default CryptoProvider is otherwise absent/ambiguous and the diff --git a/tidal-net/src/circuit_breaker.rs b/tidal-net/src/circuit_breaker.rs index be55276..77d35ab 100644 --- a/tidal-net/src/circuit_breaker.rs +++ b/tidal-net/src/circuit_breaker.rs @@ -7,9 +7,11 @@ //! `Ok(())`, and every subsequent `check()` returns [`CircuitOpenError`] until the //! outstanding probe resolves via [`record_success`](CircuitBreaker::record_success) //! (→ Closed) or [`record_failure`](CircuitBreaker::record_failure) (→ Open again). -//! This is enforced by the state machine, not merely advertised: the half-open -//! state carries an `in_flight` flag so concurrent callers cannot all probe a peer -//! that just failed. +//! This single-probe invariant is enforced by the state machine itself, not by a +//! separate flag: `check()` returns `Ok(())` only on the `Open → HalfOpen` +//! transition and errors for every call while already in `HalfOpen`, so being in +//! the fieldless `HalfOpen` variant *is* the "a probe is outstanding" marker. No +//! `in_flight` field is needed (or present). //! //! Struct-shaped error: `CircuitOpenError` carries no fields and represents the single "breaker is open" outcome; an enum with one variant would be ceremony with no readability gain. diff --git a/tidal-net/src/server.rs b/tidal-net/src/server.rs index f8d680f..27155d2 100644 --- a/tidal-net/src/server.rs +++ b/tidal-net/src/server.rs @@ -199,3 +199,70 @@ pub fn start_server( Ok(handle) } + +#[cfg(test)] +#[allow(clippy::unwrap_used)] // test assertions on known-good fixtures +mod tests { + use super::*; + use crate::proto::{ShipSegmentRequest, WalSegmentId}; + + fn make_request(payload_len: usize) -> ShipSegmentRequest { + ShipSegmentRequest { + id: Some(WalSegmentId { + region_id: 0, + shard_id: 0, + seqno: 1, + }), + payload: vec![0xAB; payload_len], + event_count: 1, + leader_last_seq: 1, + } + } + + /// SUGGESTION (tidal-net): pin the server-side payload boundary semantics so a + /// future change to one guard cannot silently desync from the client/codec. + /// The contract (shared with `GrpcTransport::send_segment` in transport.rs and + /// the codec limits in `start_server`) is **inclusive at `max_payload_bytes`**: + /// a payload of exactly `max` is ACCEPTED, and `max + 1` is REJECTED + /// (`len() > max`, not `>=`). This exercises the real `ship_segment` handler at + /// both sides of the boundary. + #[test] + fn ship_segment_payload_boundary_is_inclusive_at_max() { + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + runtime.block_on(async { + let max = 1024usize; + let (tx, mut rx) = mpsc::channel(4); + let service = WalShippingService::new(tx, max); + + // Exactly max: accepted, and forwarded onto the inbound channel. + let resp = service + .ship_segment(Request::new(make_request(max))) + .await + .expect("a payload of exactly max_payload_bytes must be accepted"); + assert!( + resp.into_inner().accepted, + "payload == max must be accepted (inclusive boundary)" + ); + let forwarded = rx.try_recv().expect("accepted payload reaches the channel"); + assert_eq!(forwarded.bytes.len(), max); + + // max + 1: rejected with resource_exhausted, nothing forwarded. + let err = service + .ship_segment(Request::new(make_request(max + 1))) + .await + .expect_err("a payload of max + 1 must be rejected"); + assert_eq!( + err.code(), + tonic::Code::ResourceExhausted, + "over-size payload must be rejected as resource_exhausted" + ); + assert!( + rx.try_recv().is_err(), + "a rejected payload must not be forwarded onto the inbound channel" + ); + }); + } +} diff --git a/tidal-net/src/transport.rs b/tidal-net/src/transport.rs index b186d0f..3d992f8 100644 --- a/tidal-net/src/transport.rs +++ b/tidal-net/src/transport.rs @@ -128,6 +128,15 @@ impl GrpcTransport { pub fn new(config: GrpcTransportConfig) -> Result { ensure_crypto_provider(); + // Validate numeric invariants BEFORE building anything (C15). In + // particular `mpsc::channel(config.channel_capacity)` below panics on a + // zero capacity (tokio asserts buffer > 0), so a `channel_capacity == 0` + // config must be caught here as a typed `Internal` error rather than + // panicking deep in the constructor. `PeerPool::new` validates again + // (idempotent, cheap); doing it up front is what guards the real + // constructor ordering the config doc promises. + config.validate()?; + let runtime = tokio::runtime::Builder::new_multi_thread() .worker_threads(2) .enable_all() @@ -199,19 +208,43 @@ impl GrpcTransport { self.shutdown.request(); } - /// Whether the embedded gRPC serve loop has terminated. + /// Whether the embedded gRPC serve loop has terminated, for ANY reason. /// /// The serve task runs for the transport's whole lifetime; a `true` here /// means the listener stopped accepting — either because the transport is /// shutting down or because the serve loop failed (in which case the failure /// was already logged at `error!` inside the task; see [`server::start_server`]). - /// Health checks and the cluster control plane poll this so a follower whose - /// receive side has silently died is observable, not a black hole. + /// + /// Because this conflates a clean shutdown with a silent death, prefer + /// [`serve_loop_died`](Self::serve_loop_died) for liveness decisions — it is + /// the one that distinguishes the two (C14). #[must_use] pub fn server_terminated(&self) -> bool { self.server_handle.is_finished() } + /// Whether the gRPC serve loop has died WITHOUT a shutdown being requested. + /// + /// This is the load-bearing liveness signal (C14): on a follower the receive + /// side dies silently when the listener stops, a TLS handshake task panics, + /// or the reactor is torn down. In all those cases the spawned server task + /// ends, its `inbound_tx` drops, and [`recv_segment`](Self::recv_segment) + /// returns `None` — *exactly* as it does on a clean + /// [`shutdown_receivers`](Self::shutdown_receivers). The two are otherwise + /// indistinguishable, so a dead follower would look like an intentionally + /// stopped one. + /// + /// `serve_loop_died()` resolves the ambiguity: it is `true` only when the + /// serve task has finished AND no shutdown was requested. Health checks and + /// the cluster control plane poll this so a follower whose receive side has + /// silently died is observable and demoted, not a black hole; a clean + /// shutdown returns `false` here even though [`server_terminated`](Self::server_terminated) + /// is `true`. + #[must_use] + pub fn serve_loop_died(&self) -> bool { + self.server_handle.is_finished() && !self.shutdown.is_requested() + } + /// Assert we are not inside a tokio runtime (`block_on` would panic). #[cfg(debug_assertions)] fn assert_not_in_async_context() { @@ -345,3 +378,96 @@ impl GrpcTransportFactory { Ok(result) } } + +#[cfg(test)] +#[allow(clippy::unwrap_used)] // test assertions on known-good fixtures +mod tests { + use std::net::SocketAddr; + + use super::*; + + /// Bind port 0 to obtain a free, OS-assigned address (tonic cannot bind 0 + /// directly, so we resolve a concrete port up front). + fn free_addr() -> SocketAddr { + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + listener.local_addr().unwrap() + } + + /// C15: `GrpcTransport::new` with `channel_capacity == 0` must return a typed + /// `Internal` error, NOT panic. The panic used to happen at + /// `mpsc::channel(0)` BEFORE `PeerPool::new` ran `validate()`, so the only + /// existing test (which called `PeerPool::new` directly) masked it. This + /// drives the real constructor ordering. + #[test] + #[allow(clippy::significant_drop_tightening)] + fn new_with_zero_channel_capacity_returns_internal_not_panic() { + let config = GrpcTransportConfig { + local_shard: ShardId(0), + listen_addr: free_addr(), + channel_capacity: 0, + insecure: true, + ..Default::default() + }; + let result = GrpcTransport::new(config); + assert!( + matches!(result, Err(GrpcTransportError::Internal(_))), + "channel_capacity == 0 must be a typed Internal error, not a panic" + ); + } + + /// C14: a serve loop that dies WITHOUT a shutdown request must be observable + /// via `serve_loop_died()`, distinct from a clean `shutdown_receivers()`. + /// We abort the server task to model a silent serve-loop death and assert the + /// transport reports `serve_loop_died() == true` while a transport that was + /// cleanly shut down reports `false` (even though `server_terminated()` is + /// `true` for both). + #[test] + #[allow(clippy::significant_drop_tightening)] + fn serve_loop_died_distinguishes_silent_death_from_clean_shutdown() { + // --- silent death: abort the serve task, no shutdown requested --- + let dead = GrpcTransport::new(GrpcTransportConfig { + local_shard: ShardId(0), + listen_addr: free_addr(), + insecure: true, + ..Default::default() + }) + .unwrap(); + // Kill the serve loop the way a listener-death / reactor-teardown would. + dead.server_handle.abort(); + // Wait for the abort to take effect. + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(5); + while !dead.server_terminated() && std::time::Instant::now() < deadline { + std::thread::sleep(std::time::Duration::from_millis(10)); + } + assert!( + dead.server_terminated(), + "the aborted serve task must report server_terminated()" + ); + assert!( + dead.serve_loop_died(), + "a serve loop that died WITHOUT a shutdown request must report \ + serve_loop_died() == true (C14)" + ); + + // --- clean shutdown: request shutdown, then the task ends --- + let clean = GrpcTransport::new(GrpcTransportConfig { + local_shard: ShardId(1), + listen_addr: free_addr(), + insecure: true, + ..Default::default() + }) + .unwrap(); + clean.shutdown_receivers(); + clean.server_handle.abort(); + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(5); + while !clean.server_terminated() && std::time::Instant::now() < deadline { + std::thread::sleep(std::time::Duration::from_millis(10)); + } + assert!(clean.server_terminated()); + assert!( + !clean.serve_loop_died(), + "a serve loop that ended AFTER shutdown was requested is a clean \ + shutdown, not a silent death (C14)" + ); + } +} diff --git a/tidal-server/src/cluster.rs b/tidal-server/src/cluster.rs index 21d1daa..738528e 100644 --- a/tidal-server/src/cluster.rs +++ b/tidal-server/src/cluster.rs @@ -58,6 +58,8 @@ use tidaldb::{ schema::EntityId, testing::{ClusterConfig, SimulatedCluster}, }; +use tower::{ServiceBuilder, limit::ConcurrencyLimitLayer}; +use tower_http::timeout::TimeoutLayer; use crate::{ dto::{ @@ -575,6 +577,17 @@ impl Drop for ClusterState { /// /// Exposes the same data routes as standalone (items, embeddings, signals, /// feed, search) plus cluster management endpoints. +/// +/// Protected routes carry the same load-shedding middleware as the standalone +/// router, consuming the shared [`crate::router::REQUEST_TIMEOUT_SECS`] / +/// [`crate::router::MAX_CONCURRENCY`] / [`crate::router::BODY_LIMIT_BYTES`] +/// constants: +/// - [`TimeoutLayer`] — 408 for requests exceeding the shared timeout; +/// - [`ConcurrencyLimitLayer`] — queues beyond the shared in-flight cap; +/// - body limit — 413 for oversized bodies before any deserialization. +/// +/// Health/status probes (`public`) are intentionally outside this stack so they +/// are never queued or timed out under saturation. pub fn build_cluster_router(state: Arc, api_key: Option>) -> Router { let public = Router::new() .route("/health", get(cluster_health)) @@ -600,7 +613,11 @@ pub fn build_cluster_router(state: Arc, api_key: Option>) .route("/sharded/signals", post(sharded_write_signal)) .route("/sharded/feed", get(sharded_feed)) .route("/sharded/search", get(sharded_search)) - .layer(axum::extract::DefaultBodyLimit::max(2 * 1024 * 1024)) + // Shared with the standalone router so the body cap can never drift + // (raise one, forget the other). See [`crate::router::BODY_LIMIT_BYTES`]. + .layer(axum::extract::DefaultBodyLimit::max( + crate::router::BODY_LIMIT_BYTES, + )) .with_state(state); let protected = match api_key { @@ -611,6 +628,24 @@ pub fn build_cluster_router(state: Arc, api_key: Option>) None => protected, }; + // Mirror the standalone router's load-shedding stack on the cluster + // protected routes: a request-timeout (408) and a hard in-flight cap (429) + // using the SAME constants as standalone so the two surfaces can never + // advertise different overload behavior. Without this the cluster surface + // had no in-flight ceiling and no timeout — under a query storm it would + // spawn scatter-gather workers without bound and never shed a wedged + // request, instead of degrading cleanly like the standalone node. Health + // probes (the `public` routes) are deliberately left outside this stack so + // liveness/readiness are never queued or timed out under saturation. + let protected = protected.layer( + ServiceBuilder::new() + .layer(TimeoutLayer::with_status_code( + StatusCode::REQUEST_TIMEOUT, + Duration::from_secs(crate::router::REQUEST_TIMEOUT_SECS), + )) + .layer(ConcurrencyLimitLayer::new(crate::router::MAX_CONCURRENCY)), + ); + public.merge(protected) } @@ -779,6 +814,14 @@ async fn write_embedding( Ok(StatusCode::NO_CONTENT) } +/// Record a signal on the leader region and eagerly ship it to followers. +/// +/// Returns `204 No Content` once the signal is **durably applied on the leader** +/// (storage + WAL fsync). The follower ship is BEST-EFFORT: a ship that fails is +/// logged (engine side) and queued for re-delivery via `await_convergence` / +/// `heal_region`, but the 204 does NOT assert quorum or follower acknowledgement +/// — it asserts leader durability only. A future quorum-ack contract is tracked +/// with the broader multi-process cluster work (m8p10). async fn write_signal( State(state): State>, Json(req): Json, @@ -894,13 +937,18 @@ async fn sharded_create_item( Json(req): Json, ) -> std::result::Result { let shards = state.shard_ids().map_err(ClusterAppError)?; - crate::scatter_gather::sharded_write_item( - state.cluster().map_err(ClusterAppError)?, - EntityId::new(req.entity_id), - &req.metadata, - &shards, - ) - .map_err(ClusterAppError)?; + // The single-shard write does a blocking storage + WAL-fsync `TidalDb` call; + // running it inline would pin this reactor worker for the whole write (the + // exact hazard `offload` exists to prevent). It touches only the local store + // (no gRPC ship), so `spawn_blocking` is sufficient — same treatment the + // non-sharded cluster reads/writes already get. See [`crate::offload`]. + let cluster = state.cluster_arc().map_err(ClusterAppError)?; + let entity = EntityId::new(req.entity_id); + let metadata = req.metadata; + offload_cluster_read(move || { + crate::scatter_gather::sharded_write_item(&cluster, entity, &metadata, &shards) + }) + .await?; Ok(StatusCode::CREATED) } @@ -909,13 +957,15 @@ async fn sharded_write_embedding( Json(req): Json, ) -> std::result::Result { let shards = state.shard_ids().map_err(ClusterAppError)?; - crate::scatter_gather::sharded_write_embedding( - state.cluster().map_err(ClusterAppError)?, - EntityId::new(req.entity_id), - &req.values, - &shards, - ) - .map_err(ClusterAppError)?; + // Offload the blocking single-shard embedding write off the reactor (see + // [`sharded_create_item`]). + let cluster = state.cluster_arc().map_err(ClusterAppError)?; + let entity = EntityId::new(req.entity_id); + let values = req.values; + offload_cluster_read(move || { + crate::scatter_gather::sharded_write_embedding(&cluster, entity, &values, &shards) + }) + .await?; Ok(StatusCode::NO_CONTENT) } @@ -924,14 +974,16 @@ async fn sharded_write_signal( Json(req): Json, ) -> std::result::Result { let shards = state.shard_ids().map_err(ClusterAppError)?; - crate::scatter_gather::sharded_write_signal( - state.cluster().map_err(ClusterAppError)?, - &req.signal, - EntityId::new(req.entity_id), - req.weight, - &shards, - ) - .map_err(ClusterAppError)?; + // Offload the blocking single-shard signal write off the reactor (see + // [`sharded_create_item`]). + let cluster = state.cluster_arc().map_err(ClusterAppError)?; + let signal = req.signal; + let entity = EntityId::new(req.entity_id); + let weight = req.weight; + offload_cluster_read(move || { + crate::scatter_gather::sharded_write_signal(&cluster, &signal, entity, weight, &shards) + }) + .await?; Ok(StatusCode::NO_CONTENT) } @@ -964,6 +1016,21 @@ struct ScatterGatherInfo { shard_deadline_ms: u64, } +impl From for ScatterGatherInfo { + /// Project the engine's scatter-gather execution metadata into the HTTP + /// response shape. Single source for the five-field mapping so the + /// `/sharded/feed` and `/sharded/search` handlers cannot drift. + fn from(meta: crate::scatter_gather::ScatterGatherMeta) -> Self { + Self { + degraded: meta.degraded, + unavailable_shards: meta.unavailable_shards, + shards_queried: meta.shards_queried, + elapsed_ms: meta.elapsed_ms, + shard_deadline_ms: meta.shard_deadline_ms, + } + } +} + async fn sharded_feed( State(state): State>, Query(query): Query, @@ -1002,13 +1069,7 @@ async fn sharded_feed( Ok(Json(ShardedFeedResponse { items: feed_items(&result.items), total_candidates: result.total_candidates, - scatter_gather: ScatterGatherInfo { - degraded: meta.degraded, - unavailable_shards: meta.unavailable_shards, - shards_queried: meta.shards_queried, - elapsed_ms: meta.elapsed_ms, - shard_deadline_ms: meta.shard_deadline_ms, - }, + scatter_gather: meta.into(), })) } @@ -1066,13 +1127,7 @@ async fn sharded_search( Ok(Json(ShardedSearchResponse { items: search_items(&result.items), total_candidates: result.total_candidates, - scatter_gather: ScatterGatherInfo { - degraded: meta.degraded, - unavailable_shards: meta.unavailable_shards, - shards_queried: meta.shards_queried, - elapsed_ms: meta.elapsed_ms, - shard_deadline_ms: meta.shard_deadline_ms, - }, + scatter_gather: meta.into(), })) } diff --git a/tidal-server/src/router.rs b/tidal-server/src/router.rs index 368e6fd..d270abb 100644 --- a/tidal-server/src/router.rs +++ b/tidal-server/src/router.rs @@ -38,15 +38,24 @@ use crate::{ /// Maximum request body size. Requests exceeding this are rejected with 413 /// before any deserialization occurs. -const BODY_LIMIT_BYTES: usize = 2 * 1024 * 1024; +/// +/// Shared with the cluster router ([`crate::cluster::build_cluster_router`]) so +/// the standalone and cluster surfaces can never drift on the body cap (raising +/// one and forgetting the other). +pub(crate) const BODY_LIMIT_BYTES: usize = 2 * 1024 * 1024; /// Maximum wall-clock time a single request may occupy. Exceeded requests /// receive 408 Request Timeout. -const REQUEST_TIMEOUT_SECS: u64 = 30; +/// +/// Shared with the cluster router so both surfaces time out identically. +pub(crate) const REQUEST_TIMEOUT_SECS: u64 = 30; /// Maximum number of requests processed concurrently. Additional requests /// are queued by the concurrency layer until a slot opens. -const MAX_CONCURRENCY: usize = 100; +/// +/// Shared with the cluster router so both surfaces cap in-flight load +/// identically. +pub(crate) const MAX_CONCURRENCY: usize = 100; /// Sequential request ID generator — assigns monotonically increasing IDs. #[derive(Clone, Default)] diff --git a/tidal-server/src/scatter_gather.rs b/tidal-server/src/scatter_gather.rs index 7ea54fe..c38a43e 100644 --- a/tidal-server/src/scatter_gather.rs +++ b/tidal-server/src/scatter_gather.rs @@ -22,7 +22,7 @@ use std::{ collections::{HashMap, HashSet}, - sync::{Arc, mpsc}, + sync::{Arc, Condvar, Mutex, OnceLock, mpsc}, time::{Duration, Instant}, }; @@ -76,6 +76,123 @@ fn clamp_deadline_ms(requested: Option) -> u64 { } } +// ── Global scatter-gather worker bound ─────────────────────────────────────── + +/// Floor on the total concurrent shard-worker threads the process may run. +/// Even on a single-core host the fan-out gets meaningful parallelism. +const MIN_SHARD_WORKERS: usize = 8; +/// Multiplier applied to available parallelism to size the cap. Shard workers +/// are query/IO-bound (a blocking `TidalDb` read), not purely CPU-bound, so a +/// modest oversubscription keeps cores busy without unbounded growth. +const SHARD_WORKERS_PER_CORE: usize = 8; +/// Hard ceiling on the total concurrent shard-worker threads, independent of +/// core count, so a many-core host still cannot spawn an unbounded thread set +/// under a query storm. +const MAX_SHARD_WORKERS: usize = 256; + +/// A process-wide counting semaphore bounding the number of scatter-gather +/// shard-worker threads that may execute a blocking shard query at once. +/// +/// The router-level [`ConcurrencyLimitLayer`](tower::limit::ConcurrencyLimitLayer) +/// caps in-flight HTTP requests, but each sharded request still fans out one +/// detached thread per live shard. Without a fan-out cap, `requests × shards` +/// detached OS threads can pile up under a burst. This semaphore caps the +/// AGGREGATE concurrent shard workers regardless of how many requests fan out, +/// so the node sheds load cleanly instead of exhausting OS threads. +/// +/// A worker that cannot acquire a permit before the request deadline exits +/// WITHOUT running its query; the coordinator then reports that shard as +/// degraded (never silently dropped). That is the correct behavior under +/// overload: the shard genuinely could not be serviced within budget, and the +/// would-be worker thread retires immediately rather than parking indefinitely. +struct ShardWorkerSemaphore { + /// Available permits. Guarded by the mutex; waiters block on the condvar. + permits: Mutex, + available: Condvar, +} + +impl ShardWorkerSemaphore { + fn new(permits: usize) -> Self { + Self { + permits: Mutex::new(permits.max(1)), + available: Condvar::new(), + } + } + + /// Acquire one permit, waiting at most `timeout`. Returns a guard that + /// releases the permit on drop, or `None` if no permit became available in + /// time (the caller should degrade rather than block further). + fn acquire_timeout(&self, timeout: Duration) -> Option> { + let deadline = Instant::now() + timeout; + // Poison is benign here: the only critical section is the integer + // permit count and a notify; a panic mid-update cannot leave a torn + // value, so recover the guard and continue (house pattern). + let mut permits = self + .permits + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + loop { + if *permits > 0 { + *permits -= 1; + return Some(ShardWorkerPermit { sem: self }); + } + let remaining = deadline.saturating_duration_since(Instant::now()); + if remaining.is_zero() { + return None; + } + let (next, timed_out) = self + .available + .wait_timeout(permits, remaining) + .unwrap_or_else(std::sync::PoisonError::into_inner); + permits = next; + if timed_out.timed_out() && *permits == 0 { + return None; + } + } + } + + fn release(&self) { + { + let mut permits = self + .permits + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + *permits += 1; + } + // One released permit wakes at most one waiter. Notify after dropping the + // lock so the woken waiter does not immediately re-block on a held guard. + self.available.notify_one(); + } +} + +/// RAII permit: releases its slot back to the semaphore on drop, even if the +/// shard query panics. +struct ShardWorkerPermit<'a> { + sem: &'a ShardWorkerSemaphore, +} + +impl Drop for ShardWorkerPermit<'_> { + fn drop(&mut self) { + self.sem.release(); + } +} + +/// The process-global shard-worker semaphore, sized once from available +/// parallelism on first use. +static SHARD_WORKER_SEMAPHORE: OnceLock = OnceLock::new(); + +/// Resolve the global shard-worker semaphore, initializing it on first use. +fn shard_worker_semaphore() -> &'static ShardWorkerSemaphore { + SHARD_WORKER_SEMAPHORE.get_or_init(|| { + let cores = std::thread::available_parallelism().map_or(1, std::num::NonZeroUsize::get); + let permits = cores + .saturating_mul(SHARD_WORKERS_PER_CORE) + .clamp(MIN_SHARD_WORKERS, MAX_SHARD_WORKERS); + tracing::info!(permits, "scatter-gather shard-worker cap initialized"); + ShardWorkerSemaphore::new(permits) + }) +} + /// Metadata about scatter-gather query execution. #[derive(Debug, Clone)] pub struct ScatterGatherMeta { @@ -256,6 +373,18 @@ struct GatherState { /// receiver that has already been dropped; the send fails harmlessly and the /// worker exits. Because every worker only *reads* the shared cluster, leaking /// a still-running worker past the request is sound. +/// +/// # Fan-out cap +/// +/// Each worker must acquire a permit from the process-global +/// [`shard_worker_semaphore`] before running its blocking query, so the +/// AGGREGATE number of concurrently-executing shard workers is bounded +/// regardless of how many sharded requests fan out at once. A worker that +/// cannot get a permit within the remaining budget retires immediately and is +/// reported degraded — never silently dropped, and never left parked +/// indefinitely. Together with the router's request-concurrency limit this caps +/// total detached threads under a query storm instead of growing them without +/// bound. fn dispatch_shards( cluster: &Arc, shards: &[RegionId], @@ -306,6 +435,7 @@ where // is marked degraded immediately and excluded from the wait set so we never // burn the whole deadline waiting on a result that can never arrive. let mut dispatched: Vec = Vec::with_capacity(live_shards.len()); + let sem = shard_worker_semaphore(); for &shard in &live_shards { let tx = tx.clone(); let cluster = Arc::clone(cluster); @@ -313,7 +443,24 @@ where let spawned = std::thread::Builder::new() .name(format!("scatter-shard-{}", shard.0)) .spawn(move || { - let outcome = query_one(&cluster, shard); + // Bound the AGGREGATE concurrent shard workers process-wide: a + // worker that cannot get a permit before the budget expires + // retires immediately and reports degraded, rather than parking + // a thread or running an out-of-budget query. The permit is held + // for exactly the blocking query and released on drop (even on + // panic). See [`shard_worker_semaphore`]. + let remaining = deadline.saturating_sub(start.elapsed()); + // `_permit` (RAII) is held for exactly the blocking query and + // released when the closure returns; `None` means the cap was hit + // before the deadline, so this worker retires as degraded. + let outcome = sem.acquire_timeout(remaining).map_or_else( + || { + Err(ServerError::Unavailable( + "scatter-gather worker cap reached before deadline".into(), + )) + }, + |_permit| query_one(&cluster, shard), + ); // The receiver may already have moved on after the deadline; a // closed channel is expected and benign, so the error is dropped. let _ = tx.send((shard, outcome)); @@ -1674,4 +1821,78 @@ mod tests { "leader-only lookup should under-enforce (kept {kept} > cap 3)" ); } + + /// C16: the shard-worker semaphore must cap the AGGREGATE concurrent shard + /// workers. With 2 permits, no more than 2 workers may hold a permit at + /// once even when 8 contend, and every worker eventually completes (no + /// deadlock, no lost permit on release). + #[test] + fn shard_worker_semaphore_caps_concurrency() { + use std::sync::atomic::{AtomicUsize, Ordering}; + + let sem = Arc::new(ShardWorkerSemaphore::new(2)); + let live = Arc::new(AtomicUsize::new(0)); + let peak = Arc::new(AtomicUsize::new(0)); + let completed = Arc::new(AtomicUsize::new(0)); + + let handles: Vec<_> = (0..8) + .map(|_| { + let sem = Arc::clone(&sem); + let live = Arc::clone(&live); + let peak = Arc::clone(&peak); + let completed = Arc::clone(&completed); + std::thread::spawn(move || { + // Generous timeout: every worker should get a permit + // eventually (the holders release quickly). + let permit = sem + .acquire_timeout(Duration::from_secs(5)) + .expect("permit must become available within timeout"); + let now = live.fetch_add(1, Ordering::SeqCst) + 1; + peak.fetch_max(now, Ordering::SeqCst); + // Hold the permit briefly so contention is real. + std::thread::sleep(Duration::from_millis(5)); + live.fetch_sub(1, Ordering::SeqCst); + completed.fetch_add(1, Ordering::SeqCst); + drop(permit); + }) + }) + .collect(); + for h in handles { + h.join().expect("worker thread must not panic"); + } + + assert!( + peak.load(Ordering::SeqCst) <= 2, + "no more than 2 workers may hold a permit at once, saw {}", + peak.load(Ordering::SeqCst) + ); + assert_eq!( + completed.load(Ordering::SeqCst), + 8, + "every worker must complete (no deadlock, permits all returned)" + ); + // All permits returned: a fresh acquire succeeds immediately. + assert!( + sem.acquire_timeout(Duration::from_millis(1)).is_some(), + "all permits should be back after every worker finished" + ); + } + + /// C16: a worker that cannot get a permit before its deadline retires + /// instead of parking forever. With zero spare permits and a tiny timeout, + /// `acquire_timeout` returns `None` so the caller degrades the shard. + #[test] + fn shard_worker_semaphore_times_out_when_saturated() { + let sem = ShardWorkerSemaphore::new(1); + let _held = sem + .acquire_timeout(Duration::from_millis(1)) + .expect("first acquire takes the only permit"); + // No permit left; a short-deadline acquire must give up (degrade), + // not block indefinitely. + let denied = sem.acquire_timeout(Duration::from_millis(10)); + assert!( + denied.is_none(), + "saturated semaphore must time out so the worker retires and degrades" + ); + } } diff --git a/tidal-server/tests/cluster_e2e.rs b/tidal-server/tests/cluster_e2e.rs index 3eac20c..4d24e2d 100644 --- a/tidal-server/tests/cluster_e2e.rs +++ b/tidal-server/tests/cluster_e2e.rs @@ -1,7 +1,27 @@ -//! Tier-3 multi-process cluster E2E tests. +//! Tier-3 single-process-cluster smoke tests over the real HTTP binary. //! -//! Spawns real `tidal-server cluster` OS processes, connects via HTTP, -//! and verifies distributed behavior across process boundaries. +//! # What these tests DO verify +//! +//! Each test spawns real `tidal-server cluster` OS processes, connects over +//! HTTP, and exercises the full binary boot path: config/topology/schema +//! parsing, the experimental-cluster opt-in gate, router wiring, the gRPC +//! transport bring-up, and the data/cluster-management routes end-to-end +//! against a real process (not an in-test `ClusterState`). +//! +//! # What these tests do NOT verify (KNOWN GAP — ROADMAP.md G1 / m8p10) +//! +//! Cluster mode currently runs **every region inside a single process**: each +//! spawned `tidal-server cluster` process builds its own `SimulatedCluster` +//! holding ALL regions, and replication between regions traverses gRPC on the +//! loopback interface *within that one process*. These tests therefore only +//! ever drive `nodes[0]`; a write to `nodes[0]` is replicated by node 0's OWN +//! in-process fabric and never crosses into `nodes[1]`'s process. So they do +//! **not** test process isolation, cross-node HTTP routing, or a real network +//! partition between independent processes — the things true multi-process HA +//! needs. Those require each process to own exactly ONE region and peer over +//! gRPC to real sibling addresses (m8p10). Until then this is honestly a +//! single-process smoke suite, named accordingly, not a multi-process E2E +//! proof. //! //! Gated behind `--features cluster-e2e` (disabled by default; these tests //! are slow and require a built binary). @@ -67,10 +87,15 @@ struct ClusterHarness { } impl ClusterHarness { - /// Spawn a multi-region cluster with `n` nodes. + /// Spawn `n` `tidal-server cluster` processes, each a FULL in-process + /// cluster. /// - /// Allocates dynamic ports, generates topology + schema YAML, spawns - /// `tidal-server cluster` processes, and waits for all to become healthy. + /// Allocates dynamic HTTP ports, generates topology + schema YAML, spawns + /// the processes, and waits for all to become healthy. Note (see module + /// docs): every spawned process internally holds ALL `n` regions, so the + /// tests only meaningfully exercise `nodes[0]`. The extra processes prove + /// the binary boots independently on its own port, not cross-process + /// replication. fn start(n: usize) -> Self { assert!(n >= 2, "need at least 2 nodes for a cluster"); @@ -290,9 +315,15 @@ fn wait_for_health(addr: &SocketAddr, deadline: Instant) { // ── E2E Tests ───────────────────────────────────────────────────────────── -/// Smoke test: start 3-node cluster, write data, verify convergence. +/// Single-process-cluster smoke test: boot a real `tidal-server cluster` +/// process, write data over HTTP, and verify it converges within its OWN +/// in-process replication fabric. +/// +/// This is NOT a cross-process convergence proof: node 0 holds all regions +/// internally, so `wait_converged` polls node 0's own `/cluster/status`. See +/// the module docs for the multi-process gap (ROADMAP.md G1 / m8p10). #[test] -fn cluster_smoke_test() { +fn single_process_cluster_smoke() { let harness = ClusterHarness::start(3); // 1. Health check. @@ -372,9 +403,14 @@ fn cluster_smoke_test() { ); } -/// Promote a follower and verify writes route to new leader. +/// Promote a follower within a single process's in-process cluster and verify +/// subsequent writes route to the new leader. +/// +/// Like [`single_process_cluster_smoke`], this exercises one process's own +/// fabric (node 0 holds all regions); it does not promote a leader across +/// independent processes. See the module docs (ROADMAP.md G1 / m8p10). #[test] -fn cluster_promote_leader() { +fn single_process_cluster_promote_leader() { let harness = ClusterHarness::start(3); // Write some initial data. diff --git a/tidal/Cargo.toml b/tidal/Cargo.toml index 5f9ea6a..04fb009 100644 --- a/tidal/Cargo.toml +++ b/tidal/Cargo.toml @@ -29,6 +29,16 @@ tempfile = { version = "3", optional = true } tracing = "0.1" usearch = "2.24.0" +# macOS-only: a plain fsync(2) on Apple platforms does NOT flush the storage +# device's volatile write cache (Apple documents this explicitly); true +# crash durability requires fcntl(fd, F_FULLFSYNC). We use rustix's *safe* +# `fcntl_fullfsync` wrapper rather than a raw `libc::fcntl` call so the WAL's +# macOS durability path stays compatible with this crate's `unsafe_code = +# "forbid"` posture. rustix is already in the dependency tree (via fjall), so +# this adds no new compile-time cost on Linux (where it is not pulled in). +[target.'cfg(target_os = "macos")'.dependencies] +rustix = { version = "1", features = ["fs"] } + [dev-dependencies] actix-web = "4" axum = "0.8" @@ -170,6 +180,10 @@ required-features = ["test-utils"] name = "m0m10_durability" required-features = ["test-utils"] +[[test]] +name = "review_pass2_zone_a_sessions" +required-features = ["test-utils"] + [[bench]] name = "signals" harness = false diff --git a/tidal/src/cohort/resolver.rs b/tidal/src/cohort/resolver.rs index 2f0a1cf..5e8eed2 100644 --- a/tidal/src/cohort/resolver.rs +++ b/tidal/src/cohort/resolver.rs @@ -9,7 +9,7 @@ use std::{ collections::HashMap, num::NonZeroUsize, - sync::{Arc, Mutex}, + sync::{Arc, Mutex, PoisonError}, }; use lru::LruCache; @@ -65,16 +65,15 @@ impl CohortResolver { /// recency). Otherwise evaluates all predicates against the user's metadata, /// caches the result, and returns it. /// - /// # Panics - /// - /// Panics only if the internal cache mutex is poisoned (a prior panic while - /// holding the lock), which cannot happen on these short, panic-free - /// critical sections. + /// Never panics on a poisoned cache mutex: the cache is a pure accelerator + /// with short, panic-free critical sections, so on poison we recover the + /// inner guard via [`PoisonError::into_inner`] and continue — a prior panic + /// elsewhere degrades to a recompute, never a write-path-wide outage. #[must_use] pub fn resolve(&self, user_id: u64, metadata: &HashMap) -> Vec { // Fast path: return cached result, bumping its LRU recency. { - let mut cache = self.cache.lock().expect("cohort resolver cache poisoned"); + let mut cache = self.cache.lock().unwrap_or_else(PoisonError::into_inner); if let Some(cached) = cache.get(&user_id) { return cached.clone(); } @@ -92,7 +91,7 @@ impl CohortResolver { // Insert under the lock; a concurrent resolve for the same user is // harmless (both compute the same membership for the same metadata). - let mut cache = self.cache.lock().expect("cohort resolver cache poisoned"); + let mut cache = self.cache.lock().unwrap_or_else(PoisonError::into_inner); cache.put(user_id, memberships.clone()); memberships } @@ -102,11 +101,10 @@ impl CohortResolver { /// Call this when user metadata is updated so the next `resolve()` call /// re-evaluates all predicates. /// - /// # Panics - /// - /// Panics only if the internal cache mutex is poisoned. + /// Never panics on a poisoned cache mutex: it recovers the inner guard via + /// [`PoisonError::into_inner`] (the cache is a pure accelerator). pub fn invalidate(&self, user_id: u64) { - let mut cache = self.cache.lock().expect("cohort resolver cache poisoned"); + let mut cache = self.cache.lock().unwrap_or_else(PoisonError::into_inner); cache.pop(&user_id); } @@ -125,24 +123,22 @@ impl CohortResolver { /// drops attribution for the new cohort until each user is LRU-evicted or /// their metadata is rewritten — a no-error, no-log wrong answer. /// - /// # Panics - /// - /// Panics only if the internal cache mutex is poisoned. + /// Never panics on a poisoned cache mutex: it recovers the inner guard via + /// [`PoisonError::into_inner`] (the cache is a pure accelerator). pub fn invalidate_all(&self) { - let mut cache = self.cache.lock().expect("cohort resolver cache poisoned"); + let mut cache = self.cache.lock().unwrap_or_else(PoisonError::into_inner); cache.clear(); } /// Number of cached user resolutions (for diagnostics). /// - /// # Panics - /// - /// Panics only if the internal cache mutex is poisoned. + /// Never panics on a poisoned cache mutex: it recovers the inner guard via + /// [`PoisonError::into_inner`]. #[must_use] pub fn cache_len(&self) -> usize { self.cache .lock() - .expect("cohort resolver cache poisoned") + .unwrap_or_else(PoisonError::into_inner) .len() } @@ -151,7 +147,7 @@ impl CohortResolver { pub fn cache_capacity(&self) -> usize { self.cache .lock() - .expect("cohort resolver cache poisoned") + .unwrap_or_else(PoisonError::into_inner) .cap() .get() } @@ -441,4 +437,82 @@ mod tests { fn assert_send_sync() {} assert_send_sync::(); } + + #[test] + fn poisoned_cache_recovers_instead_of_panicking() { + // A panic while holding the cache lock poisons the Mutex. The resolver + // must recover the inner guard (PoisonError::into_inner) rather than + // cascade the panic into every subsequent resolve/invalidate, since the + // cache is a pure accelerator with nothing to protect on poison. + let reg = make_registry(); + let resolver = Arc::new(CohortResolver::new(reg)); + let r2 = Arc::clone(&resolver); + + // Poison the lock by panicking while holding it on another thread. + let poison = std::thread::spawn(move || { + let _guard = r2.cache.lock().unwrap(); + panic!("intentional poison"); + }); + assert!(poison.join().is_err(), "poison thread should have panicked"); + + // Every cache-touching method must still work after poison. + let memberships = resolver.resolve(1, &tech_user_metadata()); + assert!(memberships.iter().any(|n| n.to_string() == "tech_en")); + let _ = resolver.cache_len(); + let _ = resolver.cache_capacity(); + resolver.invalidate(1); + resolver.invalidate_all(); + // A subsequent resolve recomputes cleanly after recovery. + let again = resolver.resolve(2, &sports_user_metadata()); + assert!(again.iter().any(|n| n.to_string() == "sports")); + } + + #[test] + fn concurrent_invalidate_all_during_slow_path_insert_is_eventually_consistent() { + // The slow path evaluates predicates WITHOUT the lock, then inserts + // under the lock. An invalidate_all() racing between the evaluation and + // the insert can leave a freshly-computed (now-stale-vs-registry) entry + // in the cache. The cache is a pure accelerator, so the contract is + // eventual consistency: a fresh resolve always recomputes the correct + // membership for the current metadata, never panics, and the cache + // never grows beyond its bound. This exercises that window under + // contention and asserts the result stays correct and the structure + // sound. + let reg = make_registry(); + let resolver = Arc::new(CohortResolver::new(reg)); + + let n_threads: usize = 8; + let iters: usize = 200; + let mut handles = Vec::with_capacity(n_threads); + for t in 0..n_threads { + let r = Arc::clone(&resolver); + handles.push(std::thread::spawn(move || { + let meta = tech_user_metadata(); + for i in 0..iters { + if (t + i) % 3 == 0 { + // Interleave full invalidations with resolves to drive + // the invalidate-all / slow-path-insert race. + r.invalidate_all(); + } else { + let user = u64::try_from(i % 16).unwrap_or(0); + let m = r.resolve(user, &meta); + // Whatever lands, it must be the correct membership for + // the supplied metadata — a stale insert can only ever + // re-cache the same (correct) value here because the + // registry is not mutated during the race. + assert!(m.iter().any(|n| n.to_string() == "tech_en")); + } + } + })); + } + for h in handles { + assert!(h.join().is_ok(), "no thread should panic in the race"); + } + + // After the storm, a fresh resolve still returns the correct membership + // and the cache remains within its bound (pure-accelerator invariant). + let m = resolver.resolve(99, &sports_user_metadata()); + assert!(m.iter().any(|n| n.to_string() == "sports")); + assert!(resolver.cache_len() <= resolver.cache_capacity()); + } } diff --git a/tidal/src/db/backup.rs b/tidal/src/db/backup.rs index 7774fa5..e4b7df4 100644 --- a/tidal/src/db/backup.rs +++ b/tidal/src/db/backup.rs @@ -8,17 +8,44 @@ //! typically brief: a synchronous flush (~milliseconds) plus the time to //! copy the data directory (proportional to data size). //! -//! The backup is **crash-consistent**: it captures a point-in-time snapshot -//! as of the flush, but is not taken under a hot write path. Any signals -//! written after the backup resumes are not included. +//! The backup is **crash-consistent** against a host/power crash: it captures +//! a point-in-time snapshot as of the flush, and every copied file plus each +//! destination directory is `fsync`ed before `create_backup` returns, so the +//! artifact is durable (not merely sitting in the page cache) the moment the +//! call completes. It is not taken under a hot write path; any signals written +//! after the backup resumes are not included. use std::{ + fs::File, path::Path, sync::atomic::{AtomicBool, Ordering}, }; use crate::schema::{DurabilityError, TidalError, Timestamp}; +/// `fsync` a single path (file or directory) so its bytes / directory entries +/// are durable, mirroring the WAL layer's barrier (`wal/segment.rs` `open`/ +/// `rotate` both `File::open(dir)` + `sync_all()`). A backup that returns before +/// its copied files and the new dirents are `fsync`ed is only crash-consistent +/// against a *process* crash, not a host/power crash — the exact crash backups +/// exist to survive. +/// +/// # Errors +/// +/// Returns `TidalError::Durability` if the open or `sync_all` fails. +fn fsync_path(path: &Path) -> crate::Result<()> { + let f = File::open(path).map_err(|e| { + TidalError::Durability(DurabilityError { + message: format!("backup: failed to open {} for fsync: {e}", path.display()), + }) + })?; + f.sync_all().map_err(|e| { + TidalError::Durability(DurabilityError { + message: format!("backup: fsync of {} failed: {e}", path.display()), + }) + }) +} + /// Metadata about a completed backup. #[derive(Debug, Clone)] pub struct BackupInfo { @@ -51,11 +78,22 @@ impl Drop for BackupGuard<'_> { /// Returns the total number of bytes copied. Skips `tidaldb.lock` (the /// advisory lock file is process-specific and will be recreated on open). /// +/// Every copied file is `fsync`ed after its bytes are written, and `dest` +/// itself is `fsync`ed once the level is fully populated so the new directory +/// entries are durable. Without this, a host/power crash in the window after +/// `create_backup` returns could leave copied files zero-length or their +/// dirents unlinked — a corrupt backup discovered only at restore. This +/// matches the WAL layer, which already `fsync`s files and parent dirs. +/// /// # Errors /// -/// Returns `TidalError::Durability` if any filesystem operation fails. +/// Returns `TidalError::Durability` if any filesystem operation (copy, create, +/// or `fsync`) fails. fn copy_dir_recursive(src: &Path, dest: &Path) -> crate::Result { let mut total_bytes: u64 = 0; + // Whether this level received at least one entry (file copied or subdir + // created). If so, we must fsync `dest` to make the new dirents durable. + let mut dest_dirty = false; let entries = std::fs::read_dir(src).map_err(|e| { TidalError::Durability(DurabilityError { @@ -109,9 +147,16 @@ fn copy_dir_recursive(src: &Path, dest: &Path) -> crate::Result { }) })?; total_bytes += copy_dir_recursive(&src_path, &dest_path)?; + dest_dirty = true; } else if file_type.is_file() { match std::fs::copy(&src_path, &dest_path) { - Ok(bytes) => total_bytes += bytes, + Ok(bytes) => { + total_bytes += bytes; + // Make the copied bytes durable before we move on. Without + // this, a power crash can leave a zero-length or torn file. + fsync_path(&dest_path)?; + dest_dirty = true; + } Err(e) if e.kind() == std::io::ErrorKind::NotFound => { // File vanished between readdir and copy -- skip it. // This is expected with fjall's compaction creating and @@ -135,6 +180,13 @@ fn copy_dir_recursive(src: &Path, dest: &Path) -> crate::Result { // Symlinks and other special file types are skipped. } + // Fsync this directory so its new entries (the files we just copied and the + // subdirectories we created) survive a host crash. A child directory's own + // entries were already fsync'd by the recursive call before it returned. + if dest_dirty { + fsync_path(dest)?; + } + Ok(total_bytes) } @@ -319,32 +371,24 @@ impl super::TidalDb { /// /// Each is non-fatal on its own: these are derived aggregates whose absence /// degrades but never corrupts ranking, so a failure is logged and the others - /// still run. Shared by the backup path; the shutdown path has the same set - /// inline because it must additionally fold any failure into its first-error - /// return. + /// still run. Delegates to the ONE shared `checkpoint_secondary_ledgers` + /// helper (in `db::state_rebuild`) so the backup, shutdown, and periodic paths + /// share one set of guards and cannot drift. The first error is discarded + /// here: a missing secondary ledger in a backup degrades ranking but never + /// corrupts data, and the signal-ledger checkpoint (the fatal one) ran before + /// this. fn checkpoint_secondary_state( &self, storage: &super::storage_box::StorageBox, meta: crate::signals::checkpoint::CheckpointMeta, ) { - if self.cohort_ledger.entry_count() > 0 - && let Err(e) = self.cohort_ledger.checkpoint(storage.items_engine(), meta) - { - tracing::warn!(error = %e, "backup: cohort checkpoint failed (non-fatal)"); - } - if self.co_engagement.edge_count() > 0 - && let Err(e) = self.co_engagement.checkpoint(storage.items_engine()) - { - tracing::warn!(error = %e, "backup: co-engagement checkpoint failed (non-fatal)"); - } - if let Err(e) = self.community_ledger.checkpoint(storage.items_engine()) { - tracing::warn!(error = %e, "backup: community checkpoint failed (non-fatal)"); - } - if !self.preference_vectors.is_empty() - && let Err(e) = self.preference_vectors.checkpoint(storage.items_engine()) - { - tracing::warn!(error = %e, "backup: preference-vector checkpoint failed (non-fatal)"); - } + let _ = crate::db::state_rebuild::checkpoint_secondary_ledgers( + storage.items_engine(), + Some((self.cohort_ledger.as_ref(), meta)), + self.community_ledger.as_ref(), + self.co_engagement.as_ref(), + self.preference_vectors.as_ref(), + ); } } @@ -372,6 +416,53 @@ mod tests { assert_eq!(content, "hello"); } + #[test] + fn fsync_path_durably_syncs_file_and_dir() { + // The fsync helper must succeed on both a regular file and a directory — + // the two call sites in copy_dir_recursive. A failure here means a copied + // backup file or its dirent would never be made durable. + let dir = tempfile::tempdir().unwrap(); + let file = dir.path().join("f.bin"); + std::fs::write(&file, b"durable").unwrap(); + fsync_path(&file).expect("file fsync must succeed"); + fsync_path(dir.path()).expect("directory fsync must succeed"); + } + + #[test] + fn copy_dir_recursive_fsyncs_every_copied_file_and_dir() { + // Durability regression: copy_dir_recursive must fsync each copied file + // and each destination directory (nested too). We cannot observe the + // fsync syscall directly without fault injection, but we can prove the + // path that fsyncs every file/dir runs to completion over a tree with a + // nested directory, and that the bytes landed correctly. The fsync calls + // are unconditional on the copy/create branches, so a regression that + // dropped them would either fail to compile (helper removed) or be caught + // by the helper's own error path surfacing here. + let src = tempfile::tempdir().unwrap(); + let dest = tempfile::tempdir().unwrap(); + + std::fs::write(src.path().join("top.txt"), "top").unwrap(); + std::fs::create_dir(src.path().join("a")).unwrap(); + std::fs::write(src.path().join("a").join("inner.txt"), "inner").unwrap(); + std::fs::create_dir(src.path().join("a").join("b")).unwrap(); + std::fs::write(src.path().join("a").join("b").join("deep.txt"), "deep").unwrap(); + + let bytes = copy_dir_recursive(src.path(), dest.path()).unwrap(); + assert!(bytes > 0); + assert_eq!( + std::fs::read_to_string(dest.path().join("top.txt")).unwrap(), + "top" + ); + assert_eq!( + std::fs::read_to_string(dest.path().join("a").join("inner.txt")).unwrap(), + "inner" + ); + assert_eq!( + std::fs::read_to_string(dest.path().join("a").join("b").join("deep.txt")).unwrap(), + "deep" + ); + } + #[test] fn copy_dir_recursive_skips_lock_file() { let src = tempfile::tempdir().unwrap(); diff --git a/tidal/src/db/builder.rs b/tidal/src/db/builder.rs index c2b433b..9636550 100644 --- a/tidal/src/db/builder.rs +++ b/tidal/src/db/builder.rs @@ -224,36 +224,90 @@ impl TidalDbBuilder { return Err(ConfigError::MissingDataDir); } - // Validate each specified directory exists and is writable. - let dirs_to_check: Vec<&PathBuf> = [ - self.config.data_dir.as_ref(), - self.config.wal_dir.as_ref(), - self.config.cache_dir.as_ref(), - ] - .into_iter() - .flatten() - .collect(); + // Validate each specified directory exists, is a directory, and is + // writable. `data_dir` and `wal_dir` are the paths the engine actually + // creates files under, so they get a real temp-file write probe (not a + // permission-bit guess). `cache_dir` is reserved / no-op today, so it + // only gets the cheaper shape checks. + let dirs_to_check: [(Option<&PathBuf>, bool); 3] = [ + (self.config.data_dir.as_ref(), true), + (self.config.wal_dir.as_ref(), true), + (self.config.cache_dir.as_ref(), false), + ]; - for dir in dirs_to_check { + for (dir, probe_writable) in dirs_to_check { + let Some(dir) = dir else { continue }; if !dir.exists() { return Err(ConfigError::DirectoryNotFound { path: dir.clone() }); } - // Check writability by querying metadata permissions. - // On Unix, we check the readonly flag. A more robust check - // would attempt to create a temp file, but metadata is - // sufficient for configuration validation. - if dir - .metadata() - .map(|m| m.permissions().readonly()) - .unwrap_or(true) - { + // Reject a path that exists but is not a directory (e.g. a regular + // file or symlink to one). Without this, fjall/WAL fail later with + // an opaque io error when they try to create children under it. + if !dir.is_dir() { return Err(ConfigError::NotWritable { path: dir.clone() }); } + if probe_writable { + // Real writability probe: `permissions().readonly()` reflects only + // the inode's write bits, not whether THIS process's uid/gid can + // write (a 0755 dir owned by another user reports !readonly yet + // EACCES on create). Actually create+remove a uniquely-named file, + // mirroring the lock-file open, so wrong-ownership and read-only + // mounts are caught here at the open boundary, not opaquely during + // component init. + Self::probe_dir_writable(dir)?; + } else { + // Reserved path: a cheap permission-bit check is sufficient since + // the engine does not write here today. + if dir + .metadata() + .map(|m| m.permissions().readonly()) + .unwrap_or(true) + { + return Err(ConfigError::NotWritable { path: dir.clone() }); + } + } } Ok(()) } + /// Probe that the current process can actually create (and remove) a file in + /// `dir`, mapping any failure to [`ConfigError::NotWritable`]. + /// + /// Uses a uniquely-named probe file (pid + nanosecond timestamp) created with + /// `create_new` so concurrent probes never collide and a stale file is never + /// silently reused, then removes it. This is the same "attempt a real write" + /// approach the data-dir lock open relies on, and unlike a permission-bit + /// check it catches the common wrong-ownership / read-only-mount cases. + fn probe_dir_writable(dir: &Path) -> Result<(), ConfigError> { + let unique = format!( + ".tidaldb-writeprobe-{}-{}", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_nanos()) + .unwrap_or(0) + ); + let probe_path = dir.join(unique); + let create_result = OpenOptions::new() + .write(true) + .create_new(true) + .open(&probe_path); + match create_result { + Ok(file) => { + // Drop the handle before unlinking; ignore a remove error (the + // create succeeded, which is all the probe needed to prove, and a + // leftover probe file does not break correctness). + drop(file); + let _ = std::fs::remove_file(&probe_path); + Ok(()) + } + Err(_) => Err(ConfigError::NotWritable { + path: dir.to_path_buf(), + }), + } + } + /// Enable the metrics HTTP server on the given address. /// /// Only available when the `metrics` feature is enabled. When the @@ -620,3 +674,100 @@ mod rate_limiter_validation_tests { assert!(result.is_ok(), "valid rate-limiter config must open"); } } + +#[cfg(test)] +mod dir_validation_tests { + use super::*; + + /// W3: a path that exists but is a regular file (not a directory) must be + /// rejected at `validate()`/`open()` with a typed `ConfigError`, not pass and + /// fail opaquely later when fjall/WAL try to create children under it. + #[test] + fn validate_rejects_regular_file_as_data_dir() { + let tmp = tempfile::tempdir().expect("tempdir"); + let file_path = tmp.path().join("not-a-dir"); + std::fs::write(&file_path, b"x").expect("write probe file"); + + let builder = TidalDb::builder().with_data_dir(&file_path); + let err = builder + .validate() + .expect_err("a regular file must not validate as a data_dir"); + assert!( + matches!(err, ConfigError::NotWritable { .. }), + "expected NotWritable for a non-directory, got: {err:?}" + ); + // And the full open path must surface the same typed error. + assert!( + TidalDb::builder().with_data_dir(&file_path).open().is_err(), + "open must reject a non-directory data_dir" + ); + } + + /// W4: writability is probed with a real temp-file create, so a directory the + /// process cannot write fails `validate()` with `NotWritable` rather than + /// passing on a misleading permission bit. + #[cfg(unix)] + #[test] + fn validate_rejects_unwritable_data_dir() { + use std::os::unix::fs::PermissionsExt; + + let tmp = tempfile::tempdir().expect("tempdir"); + let dir = tmp.path().join("readonly"); + std::fs::create_dir(&dir).expect("create dir"); + // 0555: readable + executable, NOT writable for the owner. + std::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o555)) + .expect("set readonly perms"); + + // Running as root bypasses DAC write checks, so the probe would succeed + // and the assertion would be meaningless. Detect that by directly + // probing the dir the same way validate() does; if the write succeeds + // here, we are root (or on an fs that ignores the bits) — skip. + let direct_probe = std::fs::OpenOptions::new() + .write(true) + .create_new(true) + .open(dir.join(".tidaldb-roottest")); + if let Ok(f) = direct_probe { + drop(f); + let _ = std::fs::remove_file(dir.join(".tidaldb-roottest")); + let _ = std::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o755)); + return; + } + + let builder = TidalDb::builder().with_data_dir(&dir); + let result = builder.validate(); + + // Restore write perms so the tempdir can be cleaned up regardless. + let _ = std::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o755)); + + let err = result.expect_err("an unwritable data_dir must not validate"); + assert!( + matches!(err, ConfigError::NotWritable { .. }), + "expected NotWritable for an unwritable dir, got: {err:?}" + ); + } + + /// A normal writable directory still validates and opens — the probe only + /// rejects genuinely unusable paths. + #[test] + fn validate_accepts_writable_dir_and_leaves_no_probe_file() { + let tmp = tempfile::tempdir().expect("tempdir"); + let builder = TidalDb::builder().with_data_dir(tmp.path()); + builder.validate().expect("writable dir must validate"); + + // The probe must clean up after itself: no leftover write-probe files. + let leftovers: Vec<_> = std::fs::read_dir(tmp.path()) + .expect("read tempdir") + .filter_map(Result::ok) + .filter(|e| { + e.file_name() + .to_string_lossy() + .starts_with(".tidaldb-writeprobe-") + }) + .collect(); + assert!( + leftovers.is_empty(), + "write probe must remove its temp file; found {} leftover(s)", + leftovers.len() + ); + } +} diff --git a/tidal/src/db/communities.rs b/tidal/src/db/communities.rs index 71dbf33..f133d2f 100644 --- a/tidal/src/db/communities.rs +++ b/tidal/src/db/communities.rs @@ -7,6 +7,11 @@ //! 60s session sweeper, so the "<1s" guarantee holds by construction. Local //! profile state is never touched by these methods (local-profile-intact). +use std::{ + hash::{Hash, Hasher}, + sync::{Mutex, MutexGuard, OnceLock}, +}; + use super::TidalDb; use crate::{ governance::{ @@ -17,6 +22,48 @@ use crate::{ storage::{Tag, encode_key}, }; +/// Number of shards in the per-`(user, community)` lifecycle lock table. A power +/// of two so the index mask is a cheap bitwise AND. 256 shards keep contention +/// negligible on the cold lifecycle path while bounding the static's footprint. +const LIFECYCLE_LOCK_SHARDS: usize = 256; + +/// Sharded mutex table serializing the read-modify-write lifecycle transitions +/// (`join` / `leave` / `rejoin`) for a given `(user, community)`. +/// +/// The membership registry's `with_row` + `put` is a non-atomic read-then-put: +/// without serialization, a `join` that read the pre-leave row can `put` a fresh +/// `Joined` row over a concurrent `leave`'s `Left` row, silently re-admitting a +/// member who just left and defeating the stop-forward guarantee (a privacy +/// regression, not a transient count error). All three transitions take this +/// lock keyed by `(user, community)` so the registry sees one writer per key. +/// +/// The signal write path (`community_contribute`) deliberately does NOT take this +/// lock — it stays lock-free; only the rare lifecycle transitions serialize. +/// +/// A process-global static rather than a `TidalDb` field: the lock is purely a +/// serialization device over the registry's read-modify-write, not state tied to +/// a DB instance, so sharing it across instances only ever over-serializes two +/// distinct DBs whose keys collide on a shard — harmless on this cold path. +fn lifecycle_locks() -> &'static [Mutex<()>; LIFECYCLE_LOCK_SHARDS] { + static LOCKS: OnceLock<[Mutex<()>; LIFECYCLE_LOCK_SHARDS]> = OnceLock::new(); + LOCKS.get_or_init(|| std::array::from_fn(|_| Mutex::new(()))) +} + +/// Acquire the lifecycle lock guarding `(user, community)`. +/// +/// Recovers from a poisoned lock via [`std::sync::PoisonError::into_inner`]: the +/// guarded unit `()` carries no invariant a prior panic could have corrupted, so +/// a poisoned shard is safe to keep using rather than propagating the poison. +fn lock_lifecycle(user: UserId, community: CommunityId) -> MutexGuard<'static, ()> { + let mut hasher = std::collections::hash_map::DefaultHasher::new(); + user.as_u64().hash(&mut hasher); + community.as_u64().hash(&mut hasher); + let idx = (hasher.finish() as usize) & (LIFECYCLE_LOCK_SHARDS - 1); + lifecycle_locks()[idx] + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) +} + impl TidalDb { /// Join a community overlay under an explicit [`SharePolicy`]. /// @@ -36,6 +83,11 @@ impl TidalDb { policy: SharePolicy, ) -> crate::Result { self.require_writeable("join_community")?; + // Serialize the read-modify-write against a concurrent leave/rejoin on the + // same (user, community): the registry's with_row + put is not atomic, so + // without this a join that read the pre-leave row could overwrite a + // concurrent leave's Left row and re-admit a just-left member. + let _lifecycle = lock_lifecycle(user, community); let now_ns = Timestamp::now().as_nanos(); // If a prior membership exists and is Left/leaving, this is a rejoin. @@ -102,6 +154,10 @@ impl TidalDb { "LeaveMode::Purge requires retroactive purge (M9p3); use StopForward", )); } + // Hold the per-key lifecycle lock for the whole read-modify-write so a + // concurrent join cannot read the pre-leave row and overwrite the Left + // row we are about to publish (the lost-update / re-admit window). + let _lifecycle = lock_lifecycle(user, community); let now_ns = Timestamp::now().as_nanos(); // Durability-before-mutation: build the settled-to-`Left`, gate-engaged @@ -153,6 +209,9 @@ impl TidalDb { policy: SharePolicy, ) -> crate::Result { self.require_writeable("rejoin_community")?; + // Same per-key serialization as join/leave: the read of the prior epoch + // and the subsequent put must be a single atomic transition. + let _lifecycle = lock_lifecycle(user, community); let prior = self .membership_registry @@ -206,7 +265,7 @@ impl TidalDb { /// # Errors /// /// - `TidalError::Internal` if the database is not writeable. - /// - `TidalError::invalid_input` if `weight` is not finite. + /// - `TidalError::invalid_input` if `weight` is not finite or is negative. /// - `TidalError::Schema` if `signal_type` is undefined. /// - `TidalError::Durability` if the WAL write fails. pub fn signal_for_community( @@ -246,11 +305,11 @@ impl TidalDb { writer_agent: &str, ) -> crate::Result { self.require_writeable("signal_for_community")?; - if !weight.is_finite() { - return Err(TidalError::invalid_input( - "signal weight must be finite (NaN and Inf are not allowed)", - )); - } + // Same weight admission as the entity signal path: finite AND + // non-negative (negative engagement uses separate signal types, not + // negative weights — spec §8), so a community contribution cannot drive + // a contributor cell's decay score below zero. + Self::validate_signal_weight(weight)?; // <1s STOP-FORWARD / eligibility gate: O(1) atomic read. Drops the // contribution (no WAL write) if the member is gated or the intent is @@ -299,6 +358,19 @@ impl TidalDb { // In-memory community aggregate, retaining per-contributor attribution so // a later purge (M9p3) can remove exactly this user's contributions and // rematerialize deterministically. + // + // ORDERING (durability-before-mutation, correctly applied): unlike + // `add_to_collection`/`leave_community`, the durability anchor here is the + // WAL event ALREADY committed above (`record_signal_scoped` returned Ok, + // i.e. the event is fsync-durable). `record` below only updates the + // in-memory aggregate to MIRROR that already-durable event — so memory is + // never "ahead of disk": it tracks a contribution the WAL has already made + // durable. `persist_cell` then writes the SECONDARY checkpoint copy (the + // only copy that carries per-contributor identity, which the WAL event + // omits). A `persist_cell` failure therefore does NOT mean the contribution + // was lost: it is still WAL-durable and the checkpoint-restore + WAL replay + // path reconverges the secondary copy on the next open. We still surface + // the error so the caller knows the secondary durable copy lagged. if let Ok(type_id) = self.community_ledger.resolve_signal_type(signal_type) { self.community_ledger.record( community, @@ -312,18 +384,19 @@ impl TidalDb { ); // ZERO loss window for an accepted contribution: the community - // aggregate is the SOLE durable copy of per-contributor identity (the - // WAL community event carries none), so a periodic checkpoint alone - // would still lose this contribution on a crash before the next tick. - // The `signal_for_community` rustdoc promises `Ok(true)` means - // "durably logged" — to honor that for the per-contributor aggregate we - // synchronously persist the affected cell here BEFORE returning - // Ok(true). We persist ONLY the one touched row (incremental, - // O(1) per write) rather than re-checkpointing the entire community - // keyspace: a full snapshot on every accept is O(total contributions) - // per write — quadratic ingestion that blows the signal-write budget - // for any active community (CRITICAL fix). The periodic/lifecycle - // checkpoint still garbage-collects purged rows. + // aggregate's checkpoint is the SOLE durable copy of per-contributor + // identity (the WAL community event carries none), so a periodic + // checkpoint alone would still lose this attribution on a crash before + // the next tick. The `signal_for_community` rustdoc promises `Ok(true)` + // means "durably logged" — to honor that for the per-contributor + // aggregate we synchronously persist (and fsync, see `persist_cell`) + // the affected cell here BEFORE returning Ok(true). We persist ONLY the + // one touched row (incremental, O(1) per write) rather than + // re-checkpointing the entire community keyspace: a full snapshot on + // every accept is O(total contributions) per write — quadratic + // ingestion that blows the signal-write budget for any active community + // (CRITICAL fix). The periodic/lifecycle checkpoint still + // garbage-collects purged rows. if let Some(ref storage) = self.storage && let Err(e) = self.community_ledger.persist_cell( storage.items_engine(), @@ -473,6 +546,10 @@ impl TidalDb { // Durability-before-mutation: persist the tombstone first. self.persist_tombstone(&tombstone)?; + // Durability is provided by tombstone replay on open; the + // periodic/lifecycle checkpoint GCs the stale rows. We deliberately do NOT + // eagerly re-checkpoint here — the agent-purge path (remove_scope.rs) takes + // the identical stance, so both purge paths share one crash-recovery story. let contributions_removed = self.community_ledger .purge_contributor(community, user, epoch_range); @@ -620,3 +697,166 @@ mod leave_durability_tests { db.close().unwrap(); } } + +#[cfg(test)] +#[allow(clippy::unwrap_used)] +mod lifecycle_lock_tests { + use super::{LIFECYCLE_LOCK_SHARDS, lifecycle_locks, lock_lifecycle}; + use crate::governance::{UserId, scope::CommunityId}; + + /// The lifecycle lock must serialize transitions on the SAME + /// (user, community) key: while one is held, the same key's shard is + /// contended (a non-reentrant `try_lock` from this thread returns + /// `WouldBlock`). This is the mechanism that closes the join/leave + /// lost-update window. We deliberately do NOT assert that a *distinct* shard + /// is lockable: the lock table is process-global, so a concurrently-running + /// test could legitimately hold any other shard — that would make such an + /// assertion flaky without proving anything about serialization. + #[test] + fn lock_serializes_the_same_key() { + use std::hash::{Hash, Hasher}; + + let (u, c) = (UserId(1), CommunityId(10)); + let held = lock_lifecycle(u, c); + + // Recompute the held shard index the same way lock_lifecycle does, then + // assert that very shard is contended (cannot be re-acquired) while held. + let held_idx = { + let mut h = std::collections::hash_map::DefaultHasher::new(); + u.as_u64().hash(&mut h); + c.as_u64().hash(&mut h); + (h.finish() as usize) & (LIFECYCLE_LOCK_SHARDS - 1) + }; + assert!( + lifecycle_locks()[held_idx].try_lock().is_err(), + "the held key's shard must be contended while the guard is alive" + ); + drop(held); + } +} + +#[cfg(test)] +#[allow(clippy::unwrap_used)] +mod join_leave_race_tests { + use std::{ + sync::{Arc, Barrier}, + time::Duration, + }; + + use crate::{ + TidalDb, + governance::{ + LeaveMode, MembershipState, ShareIntent, SharePolicy, UserId, scope::CommunityId, + }, + schema::{DecaySpec, EntityId, EntityKind, SchemaBuilder, Timestamp, Window}, + }; + + fn schema() -> crate::schema::Schema { + let mut b = SchemaBuilder::new(); + let _ = b + .signal( + "like", + EntityKind::Item, + DecaySpec::Exponential { + half_life: Duration::from_secs(7 * 24 * 3600), + }, + ) + .windows(&[Window::AllTime]) + .velocity(false) + .add(); + b.build().unwrap() + } + + fn policy() -> SharePolicy { + SharePolicy::community_share(1, [ShareIntent::Like]) + } + + /// Regression for the join/leave lost-update: a barrier-synchronized + /// concurrent `leave` (`StopForward`) and `join` must NEVER leave the member + /// re-admitted at the *leaving* epoch with a cleared gate. With the per-key + /// lifecycle lock the two transitions serialize, so the final state is always + /// one of the legal serial outcomes: + /// + /// - `Left` (leave won the lock last), or + /// - `Rejoined` at a *higher* epoch (join observed the Left row and rejoined). + /// + /// The illegal outcome the lock forbids is `Joined`/`Rejoined` at epoch 0 with + /// a cleared gate while a leave also ran — a silently re-admitted member. + #[test] + fn concurrent_join_and_leave_never_silently_readmit() { + let item = EntityId::new(7); + // A handful of barrier-synced rounds keeps the test fast but maximizes the + // overlap window each round. + for round in 0..16u64 { + let db = Arc::new( + TidalDb::builder() + .ephemeral() + .with_schema(schema()) + .open() + .unwrap(), + ); + let user = UserId(round + 1); + let community = CommunityId(100 + round); + db.join_community(user, community, policy()).unwrap(); + + let barrier = Arc::new(Barrier::new(2)); + let leaver = { + let db = Arc::clone(&db); + let barrier = Arc::clone(&barrier); + std::thread::spawn(move || { + barrier.wait(); + db.leave_community(user, community, LeaveMode::StopForward) + .unwrap(); + }) + }; + let joiner = { + let db = Arc::clone(&db); + let barrier = Arc::clone(&barrier); + std::thread::spawn(move || { + barrier.wait(); + // A re-join attempt racing the leave. + let _ = db.join_community(user, community, policy()); + }) + }; + leaver.join().unwrap(); + joiner.join().unwrap(); + + let state = db.membership_state(user, community).unwrap(); + let epoch = db.membership_epoch(user, community).unwrap(); + let admits = db + .signal_for_community("like", item, 1.0, Timestamp::now(), user, community) + .unwrap(); + + match state { + // Leave settled last: contributions must be gated. + MembershipState::Left | MembershipState::LeavingStopForward => { + assert!( + !admits, + "round {round}: a Left membership must gate contributions" + ); + } + // Join settled last AFTER observing the leave: it must be a genuine + // rejoin at a higher epoch, not a re-admit at the leaving epoch. + MembershipState::Rejoined => { + assert!( + epoch >= 1, + "round {round}: a rejoin after leave must open a new epoch (>0), \ + got epoch {epoch} — that would be a silent re-admit" + ); + } + // The only way to stay Joined at epoch 0 is if the join's + // read-modify-write completed entirely before leave engaged the + // gate (serialized, join-then-leave is impossible here since both + // started behind the barrier and leave always settles to Left). + // Reaching Joined means leave never ran its mutation under the + // lock — which the serialization forbids. + MembershipState::Joined => { + panic!( + "round {round}: ended in Joined@{epoch} despite a concurrent leave — \ + the lifecycle lock must prevent a join from overwriting the Left row" + ); + } + } + } + } +} diff --git a/tidal/src/db/creators.rs b/tidal/src/db/creators.rs index 410188d..fbd0fda 100644 --- a/tidal/src/db/creators.rs +++ b/tidal/src/db/creators.rs @@ -28,15 +28,33 @@ impl TidalDb { // entity store is the source of truth; the text index is derived state, // and any write the syncer has not yet committed before a crash is // recovered by the unconditional rebuild-from-store at open (see - // `db::mod::rebuild_text_indexes_at_open`). - if let Ok(guard) = self.creator_text_tx.lock() - && let Some(tx) = guard.as_ref() - { - let _ = tx.send(crate::text::PendingWrite { + // `crate::db::state_rebuild::rebuild_text_indexes_at_open`). + // + // Clone the Sender out of the guard and DROP the guard before enqueuing, + // then use the non-blocking `try_send`. Holding `creator_text_tx` across + // a blocking `send` would stall every concurrent creator write once the + // bounded outbox fills under a sustained Tantivy commit outage — the same + // hazard the item write path avoids. The durable write above has already + // completed, so dropping the enqueue costs zero durability: the + // rebuild-from-store at open re-indexes anything dropped. + let tx = self + .creator_text_tx + .lock() + .ok() + .and_then(|guard| guard.as_ref().cloned()); + if let Some(tx) = tx + && let Err(e) = tx.try_send(crate::text::PendingWrite { entity_id: id, metadata: metadata.clone(), deleted: false, - }); + }) + { + tracing::warn!( + entity_id = id.as_u64(), + error = %e, + "creator text-index outbox enqueue dropped (syncer stalled or gone); \ + rebuild-from-store at open will re-index this creator" + ); } Ok(()) diff --git a/tidal/src/db/diagnostics.rs b/tidal/src/db/diagnostics.rs index 0e635f4..d8f44b8 100644 --- a/tidal/src/db/diagnostics.rs +++ b/tidal/src/db/diagnostics.rs @@ -39,6 +39,13 @@ impl TidalDb { /// directly (NOT `JoinHandle::is_finished`: the syncer now survives commit /// failures, so a live handle no longer implies a healthy index) and from /// the latch the periodic refresher sets. + /// - **dead replication receiver** — on a follower, the segment-receiver + /// thread halted on an apply error (corrupt segment / follower-WAL IO + /// fault). Replication has silently stopped applying, so the follower is + /// frozen. This is read from + /// [`SegmentReceiverHandle::died`](crate::replication::receiver::SegmentReceiverHandle::died), + /// which is set ONLY on the error halt path — a clean shutdown does not + /// flag it (C11). /// /// # Errors /// @@ -76,6 +83,28 @@ impl TidalDb { )); } + // Replication-receiver liveness (C11): on a follower, a receiver that + // halted on an apply error means replication silently stopped — the node + // applies nothing and lag freezes flat instead of growing. Surface it as + // degraded. `died()` is set only on the error halt path (a clean shutdown + // leaves it false via the recv_segment()==None branch), and is a cheap + // lock-free atomic read, so polling it under the receiver-handle guard is + // safe even while the receiver thread is still parked. A poisoned guard is + // recovered (PoisonError::into_inner) rather than masking the check. + let receiver_dead = self + .receiver_handle + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .as_ref() + .is_some_and(crate::replication::receiver::SegmentReceiverHandle::died); + if receiver_dead { + return Err(crate::TidalError::internal( + "health_check", + "database is degraded (replication segment receiver halted on an \ + apply error; the follower has silently stopped applying)", + )); + } + // Text-search liveness. A missing syncer (no text fields in the schema) // is healthy by definition; a present-but-unhealthy syncer means text // search returns stale/empty results until a rebuild succeeds. Mark the diff --git a/tidal/src/db/export.rs b/tidal/src/db/export.rs index dbb8c42..043104a 100644 --- a/tidal/src/db/export.rs +++ b/tidal/src/db/export.rs @@ -571,7 +571,7 @@ fn read_session_journal_signals( use std::io::Read; use crate::wal::{ - format::{SessionWalEvent, decode_session_events}, + format::{SessionWalEvent, decode_session_events_with_diagnostics}, session_journal::SESSION_JOURNAL_FILENAME, }; @@ -611,13 +611,18 @@ fn read_session_journal_signals( { break; // torn tail: incomplete final frame } - let events = decode_session_events(&frame); - if events.is_empty() { - // Unparseable / checksum-failed frame: stop, mirroring the whole- - // buffer decoder's stop-at-first-corruption behavior. + let outcome = decode_session_events_with_diagnostics(&frame); + // Distinguish a *corrupt* frame from a forward-compat *skip*. A v2 frame + // carrying an unknown-but-checksum-valid record type decodes to zero + // events with corruption_detected == false (decode_typed_record returns + // Skipped); the whole-buffer decoder skips such a frame and continues, so + // this streaming reader must too. Breaking on `events.is_empty()` here + // silently dropped every later session the instant a newer writer added a + // 4th record type. Stop only on genuine corruption. + if outcome.corruption_detected { break; } - for event in events { + for event in outcome.events { match event { SessionWalEvent::Start { session_id, @@ -760,6 +765,102 @@ impl super::TidalDb { // ── Tests ───────────────────────────────────────────────────────────────────── +#[cfg(test)] +#[allow(clippy::unwrap_used)] +mod session_journal_forward_compat_tests { + use super::*; + use crate::wal::{ + format::{SESSION_RECORD_VERSION_V2, SessionWalEvent, encode_session_event}, + session_journal::SESSION_JOURNAL_FILENAME, + }; + + /// W6: a v2 frame carrying a forward-compatible *unknown* record type decodes + /// to zero events with `corruption_detected == false`. The streaming export + /// reader must SKIP it and keep reading later frames — not break at the first + /// empty decode and silently drop every subsequent session's signals. + /// + /// Layout proof: a v2 frame is `[len:u32 LE][marker:0xFF][version=2][type] + /// [payload][checksum:8]`. We synthesize the unknown frame by taking a real + /// encoded event, flipping its type byte to a value no current reader knows, + /// and recomputing the BLAKE3 checksum so the frame is checksum-VALID (the + /// exact forward-compat case v2 framing was designed to tolerate). This avoids + /// hardcoding the private marker byte — we reuse the encoder's own framing. + fn unknown_but_checksum_valid_v2_frame() -> Vec { + // Frame: [len(4)][marker(1)][version(1)][type(1)][payload..][checksum(8)]. + const TYPE_OFFSET: usize = 4 + 2; // after len + marker + version + const CHECKSUM_LEN: usize = 8; + // Start from a real encoded frame so the marker/version/framing are + // produced by the canonical encoder, then mutate the type byte. + let mut frame = encode_session_event(&SessionWalEvent::Close { session_id: 7 }).unwrap(); + assert_eq!( + frame[TYPE_OFFSET - 1], + SESSION_RECORD_VERSION_V2, + "encoder must emit a v2 frame for this test to be valid" + ); + // An unknown type: not START(0x01)/SIGNAL(0x02)/CLOSE(0x03). + frame[TYPE_OFFSET] = 0x7E; + // Recompute the checksum over the body (everything between the length + // prefix and the trailing checksum), matching session_record_checksum. + let body_end = frame.len() - CHECKSUM_LEN; + let digest = blake3::hash(&frame[4..body_end]); + let checksum = digest.as_bytes(); + frame[body_end..].copy_from_slice(&checksum[..CHECKSUM_LEN]); + frame + } + + #[test] + fn export_skips_unknown_frame_and_keeps_trailing_signal() { + let tmp = tempfile::tempdir().expect("tempdir"); + let journal_path = tmp.path().join(SESSION_JOURNAL_FILENAME); + + // Build: Start (defines session->user), unknown-but-valid v2 frame, + // then a real Signal whose timestamp lands in the export window. + let start = encode_session_event(&SessionWalEvent::Start { + session_id: 100, + user_id: 42, + started_at_ns: 1_000, + agent_id: "agent".to_string(), + policy_name: "policy".to_string(), + }) + .unwrap(); + let unknown = unknown_but_checksum_valid_v2_frame(); + let signal = encode_session_event(&SessionWalEvent::Signal { + session_id: 100, + entity_id: 999, + weight: 1.5, + ts_ns: 5_000, + signal_name: "view".to_string(), + annotation: None, + session_seqno: None, + idempotency_key: None, + }) + .unwrap(); + + let mut bytes = Vec::new(); + bytes.extend_from_slice(&start); + bytes.extend_from_slice(&unknown); + bytes.extend_from_slice(&signal); + std::fs::write(&journal_path, &bytes).expect("write journal"); + + let request = ExportRequest::time_range(0, u64::MAX); + let exported = read_session_journal_signals(tmp.path(), &request, 100); + + assert_eq!( + exported.len(), + 1, + "the trailing Signal must survive an intervening unknown frame; got {exported:?}" + ); + let sig = &exported[0]; + assert_eq!(sig.entity_id, 999); + assert_eq!(sig.signal_type, "view"); + assert_eq!( + sig.user_id, + Some(42), + "session->user mapping must be applied" + ); + } +} + #[cfg(test)] #[allow(clippy::unwrap_used)] #[path = "export_tests.rs"] diff --git a/tidal/src/db/items.rs b/tidal/src/db/items.rs index a3b0b70..30f160f 100644 --- a/tidal/src/db/items.rs +++ b/tidal/src/db/items.rs @@ -56,11 +56,11 @@ impl TidalDb { // in-memory candidate-generation indexes key on a u32 id, so an id past // u32::MAX would alias a lower id in every bitmap (silent cross-id // collision in query results). Reject it here rather than truncate so a - // rejected item never lands in storage in an unindexable state. NOTE: - // the sibling write paths in signals.rs / relationships.rs / - // collections.rs / user_state.rs still perform a bare `as u32` narrowing - // on item-side ids; they should be routed through a shared guard so the - // limit is enforced uniformly (out of scope for this change). + // rejected item never lands in storage in an unindexable state. The + // `signal_with_context` durable-row write path applies the same up-front + // rejection, and the `Hide` relationship path routes through the + // observable `narrow_item_slot` helper, so every item-side narrowing is + // now either rejected or logged — no silent truncation remains. if id.as_u64() > u64::from(u32::MAX) { let raw = id.as_u64(); return Err(TidalError::invalid_input(format!( @@ -193,15 +193,40 @@ impl TidalDb { // dropped is non-fatal). The durable entity store is the source of // truth; the text index is derived state, and any write the syncer has // not yet committed before a crash is recovered by the unconditional - // rebuild-from-store at open (see `db::mod::rebuild_text_indexes_at_open`). - if let Ok(guard) = self.text_tx.lock() - && let Some(tx) = guard.as_ref() - { - let _ = tx.send(crate::text::PendingWrite { + // rebuild-from-store at open (see + // `crate::db::state_rebuild::rebuild_text_indexes_at_open`). + // + // Clone the Sender OUT of the guard and DROP the guard before enqueuing, + // then use the non-blocking `try_send`. Holding `text_tx` across a + // blocking `send` would stall every concurrent item write the moment the + // bounded outbox fills (e.g. a sustained Tantivy commit outage where the + // syncer retries forever and stops draining) — escalating a degraded- + // text-search incident into a full halt of the durable item write API. + // The durable write above has already completed, so dropping the enqueue + // here costs zero durability: the unconditional rebuild-from-store at + // open re-indexes anything dropped. crossbeam `Sender` is cheap to clone. + let tx = self + .text_tx + .lock() + .ok() + .and_then(|guard| guard.as_ref().cloned()); + if let Some(tx) = tx + && let Err(e) = tx.try_send(crate::text::PendingWrite { entity_id: id, metadata: (*stored_metadata).clone(), deleted: false, - }); + }) + { + // Full/Disconnected: do NOT block. Drop the enqueue and warn — the + // rebuild-from-store at open re-indexes it, so correctness is + // preserved while the write path stays non-blocking under a text + // syncer outage. + tracing::warn!( + entity_id = id.as_u64(), + error = %e, + "text-index outbox enqueue dropped (syncer stalled or gone); \ + rebuild-from-store at open will re-index this item" + ); } // M6p5: index title terms for autocomplete suggestions. diff --git a/tidal/src/db/lifecycle.rs b/tidal/src/db/lifecycle.rs index a925acd..135fa38 100644 --- a/tidal/src/db/lifecycle.rs +++ b/tidal/src/db/lifecycle.rs @@ -169,31 +169,19 @@ impl TidalDb { tracing::error!(error = %e, "signal ledger checkpoint failed during shutdown"); first_err.get_or_insert(e); } - // Checkpoint cohort signal state (M6). - if self.cohort_ledger.entry_count() > 0 - && let Err(e) = self.cohort_ledger.checkpoint(storage.items_engine(), meta) - { - tracing::error!(error = %e, "cohort ledger checkpoint failed during shutdown"); - first_err.get_or_insert(e); - } - // Checkpoint co-engagement edges (M6). - if self.co_engagement.edge_count() > 0 - && let Err(e) = self.co_engagement.checkpoint(storage.items_engine()) - { - tracing::error!(error = %e, "co-engagement checkpoint failed during shutdown"); - first_err.get_or_insert(e); - } - // Checkpoint community aggregation ledger (M9p3): per-contributor - // entries cannot be rebuilt from the WAL, so persist them here. - if let Err(e) = self.community_ledger.checkpoint(storage.items_engine()) { - tracing::error!(error = %e, "community ledger checkpoint failed during shutdown"); - first_err.get_or_insert(e); - } - // Checkpoint preference vectors (derived state, no WAL backstop). - if !self.preference_vectors.is_empty() - && let Err(e) = self.preference_vectors.checkpoint(storage.items_engine()) - { - tracing::error!(error = %e, "preference-vector checkpoint failed during shutdown"); + // Checkpoint the secondary derived ledgers (cohort M6, co-engagement + // M6, community M9p3 — whose per-contributor entries cannot be rebuilt + // from the WAL — and preference vectors) through the ONE shared helper + // so the shutdown, backup, and periodic paths cannot drift. Shutdown + // must surface a failure, so we fold the helper's first error into + // `first_err` (backup/periodic discard it instead). + if let Some(e) = crate::db::state_rebuild::checkpoint_secondary_ledgers( + storage.items_engine(), + Some((self.cohort_ledger.as_ref(), meta)), + self.community_ledger.as_ref(), + self.co_engagement.as_ref(), + self.preference_vectors.as_ref(), + ) { first_err.get_or_insert(e); } // BLOCKER 6: persist the follower replication high-water-mark so a diff --git a/tidal/src/db/metrics/mod.rs b/tidal/src/db/metrics/mod.rs index 4b91c37..4d1423f 100644 --- a/tidal/src/db/metrics/mod.rs +++ b/tidal/src/db/metrics/mod.rs @@ -491,6 +491,10 @@ impl MetricsState { } // Checkpoint failure counter (unconditional -- not feature-gated). + // Carries the partition_id label so this per-node series stays distinct + // when many nodes are scraped into one Prometheus (W13), matching the + // uptime/health/info lines above. Without it, every node's + // checkpoint_failures_total would collapse onto a single unlabeled series. { use std::fmt::Write; let failures = self.checkpoint_failures_total.load(Ordering::Relaxed); @@ -498,7 +502,7 @@ impl MetricsState { out, "\n# HELP tidaldb_checkpoint_failures_total Total number of failed periodic signal checkpoints\n\ # TYPE tidaldb_checkpoint_failures_total counter\n\ - tidaldb_checkpoint_failures_total {failures}\n" + tidaldb_checkpoint_failures_total{{partition_id=\"{partition_id}\"}} {failures}\n" ); } @@ -559,4 +563,30 @@ mod partition_id_tests { "cluster node must not emit the single-node default partition_id: {out}" ); } + + /// The unconditional `checkpoint_failures_total` counter must carry the + /// `partition_id` label like every other node-identity series, so per-node + /// failure counts stay distinct in a multi-node Prometheus scrape (W13). + #[test] + fn checkpoint_failures_total_carries_partition_id() { + let state = MetricsState::new(); + let out = state.render_prometheus(); + assert!( + out.contains("tidaldb_checkpoint_failures_total{partition_id=\"0\"}"), + "checkpoint_failures_total must carry the default partition_id label: {out}" + ); + // Never emit the old unlabeled series. + assert!( + !out.contains("\ntidaldb_checkpoint_failures_total 0\n"), + "checkpoint_failures_total must not emit an unlabeled series: {out}" + ); + + let cluster = MetricsState::new(); + cluster.set_partition_id(7); + let cout = cluster.render_prometheus(); + assert!( + cout.contains("tidaldb_checkpoint_failures_total{partition_id=\"7\"}"), + "checkpoint_failures_total must reflect the cluster shard id: {cout}" + ); + } } diff --git a/tidal/src/db/mod.rs b/tidal/src/db/mod.rs index ce94fff..c241c2d 100644 --- a/tidal/src/db/mod.rs +++ b/tidal/src/db/mod.rs @@ -444,7 +444,11 @@ impl TidalDb { user_state: Arc::new(UserStateIndex::new()), hard_negatives: Arc::new(HardNegIndex::new()), interaction_ledger: Arc::new(InteractionLedger::new()), - preference_vectors: Arc::new(PreferenceVectors::new(128)), + // Cold-start (no schema): use the shared default so the fallback + // cannot drift from the schema-aware open branches (see open.rs). + preference_vectors: Arc::new(PreferenceVectors::new( + self::open::DEFAULT_PREFERENCE_DIM, + )), text_index: None, text_tx: std::sync::Mutex::new(None), text_syncer_thread: std::sync::Mutex::new(None), diff --git a/tidal/src/db/notification_tracker.rs b/tidal/src/db/notification_tracker.rs index eea1fa4..3549782 100644 --- a/tidal/src/db/notification_tracker.rs +++ b/tidal/src/db/notification_tracker.rs @@ -34,12 +34,23 @@ use dashmap::DashMap; /// (post-diversity), not the signal-write hot path, so the contention cost is /// acceptable. pub struct NotificationTracker { - /// `(user_id, creator_id, days_since_epoch)` -> count delivered today. + /// `(user_id, creator_id, days_since_epoch)` -> count of DISTINCT items + /// delivered today (a cap counts items notified about, deduplicated per + /// [`counted`](Self::counted) — re-delivering the same item the same day is + /// not a new notification). delivered: DashMap<(u64, u64, u32), u32>, - /// `(user_id, days_since_epoch)` -> total notifications delivered today. + /// `(user_id, days_since_epoch)` -> total distinct items delivered today. /// /// Tracks cross-request totals for `max_total_per_day` enforcement. user_total: DashMap<(u64, u32), u32>, + /// `(user_id, entity_id, days_since_epoch)` -> presence marker for items + /// already counted today. A notification cap counts *distinct items the user + /// was notified about*, not raw delivery attempts: re-issuing the same query + /// (or paging back to an item) must not consume the budget again. The first + /// delivery of an item records it here and increments the counters; a repeat + /// for the same `(user, entity, day)` is an idempotent no-op that still + /// passes (the item is allowed to re-appear) without inflating the cap. + counted: DashMap<(u64, u64, u32), ()>, /// Serializes the combined check-then-record critical section so the /// per-creator and per-day caps are evaluated and incremented atomically /// with respect to each other and to concurrent recorders. @@ -53,6 +64,7 @@ impl NotificationTracker { Self { delivered: DashMap::new(), user_total: DashMap::new(), + counted: DashMap::new(), record_lock: Mutex::new(()), } } @@ -89,6 +101,13 @@ impl NotificationTracker { /// see headroom and both record, which previously let the response overshoot /// the configured caps. /// + /// Idempotent per `(user_id, entity_id, day)`: an item already delivered to + /// this user today does not consume the budget a second time. It returns + /// `true` (the item is still allowed to re-appear, e.g. on a feed refresh or + /// a re-issued identical query) WITHOUT re-incrementing any counter, so the + /// daily cap counts distinct items notified about — not raw delivery + /// attempts. The first delivery of each item enforces the caps and records. + /// /// `creator_id == None` means the item has no resolvable creator: the /// per-creator cap is skipped (matching the executor's bucketing rule) but /// the per-day total cap is still enforced and incremented. @@ -96,6 +115,7 @@ impl NotificationTracker { &self, user_id: u64, creator_id: Option, + entity_id: u64, day: u32, max_per_creator: u32, max_total: u32, @@ -108,6 +128,14 @@ impl NotificationTracker { .lock() .unwrap_or_else(std::sync::PoisonError::into_inner); + // Idempotent re-delivery: an item already counted for this user today + // is allowed through again but does not consume the budget twice. This + // makes a re-issued identical query (or a page revisit) not double-count + // delivered items. + if self.counted.contains_key(&(user_id, entity_id, day)) { + return true; + } + // Per-day total cap (always enforced). if self.count_total_today(user_id, day) >= max_total { return false; @@ -120,11 +148,13 @@ impl NotificationTracker { return false; } - // Commit: increment under the same lock so the next caller observes it. + // Commit: increment under the same lock so the next caller observes it, + // and mark the item counted so a repeat is idempotent. if let Some(cid) = creator_id { *self.delivered.entry((user_id, cid, day)).or_insert(0) += 1; } *self.user_total.entry((user_id, day)).or_insert(0) += 1; + self.counted.insert((user_id, entity_id, day), ()); true } @@ -190,6 +220,7 @@ impl NotificationTracker { let oldest_to_keep = current_day.saturating_sub(keep_days); self.delivered.retain(|k, _| k.2 >= oldest_to_keep); self.user_total.retain(|k, _| k.1 >= oldest_to_keep); + self.counted.retain(|k, ()| k.2 >= oldest_to_keep); } } @@ -260,29 +291,67 @@ mod tests { let tracker = NotificationTracker::new(); let day = 19_700u32; - // Per-creator cap = 2, total cap = 5. - assert!(tracker.check_and_record(1, Some(100), day, 2, 5)); // creator100: 1, total 1 - assert!(tracker.check_and_record(1, Some(100), day, 2, 5)); // creator100: 2, total 2 + // Per-creator cap = 2, total cap = 5. Distinct entity ids so each call is + // a new item (idempotency keys on the entity). + assert!(tracker.check_and_record(1, Some(100), 1, day, 2, 5)); // creator100: 1, total 1 + assert!(tracker.check_and_record(1, Some(100), 2, day, 2, 5)); // creator100: 2, total 2 assert!( - !tracker.check_and_record(1, Some(100), day, 2, 5), - "third from creator 100 must hit the per-creator cap" + !tracker.check_and_record(1, Some(100), 3, day, 2, 5), + "third item from creator 100 must hit the per-creator cap" ); // A different creator still has headroom under the total cap. - assert!(tracker.check_and_record(1, Some(200), day, 2, 5)); // total 3 + assert!(tracker.check_and_record(1, Some(200), 4, day, 2, 5)); // total 3 assert_eq!(tracker.count_total_today(1, day), 3); assert_eq!(tracker.count_today(1, 100, day), 2); assert_eq!(tracker.count_today(1, 200, day), 1); } + #[test] + fn check_and_record_is_idempotent_per_entity_per_day() { + let tracker = NotificationTracker::new(); + let day = 19_700u32; + + // First delivery of item 500 (creator 100) records once. + assert!(tracker.check_and_record(1, Some(100), 500, day, 2, 5)); + assert_eq!(tracker.count_total_today(1, day), 1); + assert_eq!(tracker.count_today(1, 100, day), 1); + + // Re-delivering the SAME item the same day (e.g. a re-issued identical + // query) is allowed through but does NOT consume the budget again. + assert!( + tracker.check_and_record(1, Some(100), 500, day, 2, 5), + "an already-delivered item must still be allowed to re-appear" + ); + assert!(tracker.check_and_record(1, Some(100), 500, day, 2, 5)); + assert_eq!( + tracker.count_total_today(1, day), + 1, + "re-delivery must not double-count the daily total" + ); + assert_eq!( + tracker.count_today(1, 100, day), + 1, + "re-delivery must not double-count the per-creator total" + ); + + // A different item still counts, and the same item on a different day is + // a fresh notification. + assert!(tracker.check_and_record(1, Some(100), 501, day, 2, 5)); + assert_eq!(tracker.count_today(1, 100, day), 2); + assert!(tracker.check_and_record(1, Some(100), 500, day + 1, 2, 5)); + assert_eq!(tracker.count_today(1, 100, day + 1), 1); + } + #[test] fn check_and_record_enforces_total_cap_for_creatorless_items() { let tracker = NotificationTracker::new(); let day = 1u32; // No creator: per-creator cap is irrelevant, total cap still applies. - assert!(tracker.check_and_record(1, None, day, 0, 2)); - assert!(tracker.check_and_record(1, None, day, 0, 2)); + // Distinct entity ids so each is a new item. + assert!(tracker.check_and_record(1, None, 1, day, 0, 2)); + assert!(tracker.check_and_record(1, None, 2, day, 0, 2)); assert!( - !tracker.check_and_record(1, None, day, 0, 2), + !tracker.check_and_record(1, None, 3, day, 0, 2), "creatorless deliveries must still respect the daily total cap" ); assert_eq!(tracker.count_total_today(1, day), 2); @@ -310,12 +379,22 @@ mod tests { let day = 42u32; let handles: Vec<_> = (0..THREADS) - .map(|_| { + .map(|t| { let tracker = Arc::clone(&tracker); let accepted = Arc::clone(&accepted); thread::spawn(move || { - for _ in 0..ATTEMPTS_PER_THREAD { - if tracker.check_and_record(7, Some(99), day, MAX_PER_CREATOR, MAX_TOTAL) { + for i in 0..ATTEMPTS_PER_THREAD { + // A distinct item per attempt so the per-creator cap binds + // on distinct items (idempotency keys on the entity). + let entity = (t as u64) * 1000 + i as u64; + if tracker.check_and_record( + 7, + Some(99), + entity, + day, + MAX_PER_CREATOR, + MAX_TOTAL, + ) { accepted.fetch_add(1, AtomicOrdering::Relaxed); } } @@ -370,9 +449,10 @@ mod tests { let accepted = Arc::clone(&accepted); thread::spawn(move || { for i in 0..50u64 { - // Unique creator id per (thread, attempt). + // Unique creator AND entity id per (thread, attempt). let creator = (t as u64) * 1000 + i; - if tracker.check_and_record(3, Some(creator), day, 1, MAX_TOTAL) { + let entity = creator; + if tracker.check_and_record(3, Some(creator), entity, day, 1, MAX_TOTAL) { accepted.fetch_add(1, AtomicOrdering::Relaxed); } } diff --git a/tidal/src/db/open.rs b/tidal/src/db/open.rs index ec8b7be..b83f90c 100644 --- a/tidal/src/db/open.rs +++ b/tidal/src/db/open.rs @@ -25,6 +25,29 @@ use crate::{ wal::{WalConfig, WalHandle}, }; +/// Default preference-vector dimensionality used when no schema embedding slot +/// declares one. +/// +/// The cold-start path ([`TidalDb::new`], no schema) and the +/// no-embedding-slot schema path both fall back to this single value so a future +/// change to the default cannot drift between the two open branches and the +/// schema-less constructor. Keep it in sync with the typical embedding width the +/// engine ships against. +pub const DEFAULT_PREFERENCE_DIM: usize = 128; + +/// Resolve the preference-vector dimensionality from a schema. +/// +/// Uses the first declared embedding slot's dimensions, falling back to +/// [`DEFAULT_PREFERENCE_DIM`] when the schema declares no embedding slot. This is +/// the single source of truth for the fallback used by every schema-aware open +/// branch. +pub fn preference_dim_for(schema: &Schema) -> usize { + schema + .embedding_slots() + .first() + .map_or(DEFAULT_PREFERENCE_DIM, |s| s.dimensions) +} + /// Bundle returned by [`TidalDb::open_with_schema`] to avoid a fragile tuple. /// /// Each field is consumed by [`TidalDb::from_parts`] during construction. @@ -110,10 +133,7 @@ impl super::TidalDb { let ledger = Arc::new(SignalLedger::new(schema, Box::new(NoopWalWriter))); // Read preference vector dimensionality from the schema. - let pref_dim = schema_def - .embedding_slots() - .first() - .map_or(128, |s| s.dimensions); + let pref_dim = preference_dim_for(&schema_def); Ok(OpenResult { storage, @@ -243,10 +263,7 @@ impl super::TidalDb { )?; // Read preference vector dimensionality from the schema. - let pref_dim = schema_def - .embedding_slots() - .first() - .map_or(128, |s| s.dimensions); + let pref_dim = preference_dim_for(&schema_def); // Restore preference vectors from their Tag::Preference checkpoint // so personalized ranking does not snap to cold-start after a crash. diff --git a/tidal/src/db/query_ops.rs b/tidal/src/db/query_ops.rs index 6aa0f05..ab00219 100644 --- a/tidal/src/db/query_ops.rs +++ b/tidal/src/db/query_ops.rs @@ -4,7 +4,7 @@ use super::{TidalDb, storage_box::StorageBox}; use crate::{ query::{ executor::RetrieveExecutor, - retrieve::{QueryError, Results, Retrieve}, + retrieve::{Results, Retrieve}, search::{Search, SearchExecutor, SearchResults}, }, schema::{EntityKind, TidalError}, @@ -108,16 +108,28 @@ impl TidalDb { base_executor = base_executor.with_degradation_level(degradation_level); // M4: wire in session context when FOR SESSION is specified. + // + // A RETRIEVE with FOR SESSION pointing at an expired/evicted session is a + // WELL-FORMED query: CODING_GUIDELINES §6 ("graceful degradation, never + // failure") requires we never 500 it. We therefore warn-and-continue + // WITHOUT the session boost — identical to the SEARCH path — rather than + // hard-erroring with `SessionNotFound`. Before this fix the two + // structurally-identical surfaces diverged: SEARCH degraded while RETRIEVE + // 500'd the moment a session was swept, a confusing surface-specific outage + // for a client paging a feed with FOR SESSION. let executor = if let Some(session_id) = query.for_session { match self.session_snapshot(session_id) { Ok(snapshot) => { let ctx = session_mod::SessionContext::from_snapshot(&snapshot); base_executor.with_session(ctx, snapshot) } - Err(_) => { - return Err(TidalError::Query(QueryError::SessionNotFound(format!( - "{session_id}" - )))); + Err(e) => { + tracing::warn!( + session_id = %session_id, + error = %e, + "FOR SESSION: session not found; executing without session boost" + ); + base_executor } } } else { @@ -331,9 +343,13 @@ impl TidalDb { }, )); } + // Forward `for_user` so the knob is actually consulted (it biases the + // tie-break among equal-frequency trending suggestions) rather than being + // silently dropped — a caller that sets it no longer gets the identical + // global result with no effect. let results = self .suggestion_index - .suggest(&req.prefix, req.limit as usize); + .suggest(&req.prefix, req.limit as usize, req.for_user); Ok(results) } diff --git a/tidal/src/db/remove_scope.rs b/tidal/src/db/remove_scope.rs index ad033a8..6e0f737 100644 --- a/tidal/src/db/remove_scope.rs +++ b/tidal/src/db/remove_scope.rs @@ -74,18 +74,12 @@ impl TidalDb { self.community_ledger .set_purge_watermark(community, watermark_ns); - // Re-checkpoint so the durable Tag::Community snapshot already - // reflects the removal (belt-and-suspenders with the tombstone - // replay: either alone keeps the agent gone after a crash). - if let Some(ref storage) = self.storage - && let Err(e) = self.community_ledger.checkpoint(storage.items_engine()) - { - tracing::warn!( - error = %e, - "agent-purge re-checkpoint failed; tombstone replay still covers the crash window" - ); - } - + // Durability is provided by tombstone replay on open (db/mod.rs + // replays every agent-purge tombstone and re-applies it + // idempotently); the periodic/lifecycle checkpoint GCs the stale + // rows. This mirrors the user-purge path (communities.rs) exactly — + // neither eagerly re-checkpoints, so the two paths cannot drift and + // a future editor reads ONE consistent crash-recovery story. Ok(PurgeReceipt { contributions_removed, watermark_ns, diff --git a/tidal/src/db/replication_ops.rs b/tidal/src/db/replication_ops.rs index 950ea16..88a9512 100644 --- a/tidal/src/db/replication_ops.rs +++ b/tidal/src/db/replication_ops.rs @@ -282,7 +282,17 @@ impl TidalDb { /// assignments resolve to the local shard, so behavior is identical to /// calling `signal()` directly. /// - /// # Remote-shard dispatch is NOT yet wired (fails closed) + /// # On-node tenant isolation is LOGICAL, not physical + /// + /// `tenant_id` drives the per-tenant rate limiter and the shard routing + /// only. The local write below goes to the single shared signal + /// ledger/entity store with **no** tenant namespace in the key, so on a + /// single node hosting multiple tenants every tenant's signals co-mingle in + /// one keyspace/WAL. Physical per-tenant keyspace/WAL isolation and + /// enforcement of `TenantConfig::max_storage_bytes`/`max_entities` are a + /// separate, not-yet-wired task (`docs/specs/14-scale-architecture.md`). + /// + /// # Remote-shard dispatch is NOT yet wired (fully fails closed) /// /// During a dual-write migration the tenant router resolves two assignments /// (source and target shard). Cross-node dispatch of the remote half over @@ -291,17 +301,19 @@ impl TidalDb { /// `CODING_GUIDELINES.md` §12 "no TODO without a tracking note"). Rather than /// silently dropping the remote write — which would lose signals during a /// migration with no error visible to the caller — this method **fails - /// closed**: if any assignment targets a shard that is not local to this node - /// it returns `TidalError::Internal`. A misconfigured dual-write is therefore - /// loud, not silently lossy. The local half (if any) is still written before - /// the error is returned. + /// closed BEFORE any write happens**: it checks every resolved assignment + /// first, and if *any* targets a non-local shard it returns + /// `TidalError::Internal` with zero mutations. This makes the operation + /// atomic from the caller's perspective — a retry after the error cannot + /// double-count the local half, because the local half is never written when + /// the whole op cannot succeed (W8). /// /// # Errors /// /// - `TidalError::QuotaExceeded` — tenant rate limit exceeded. /// - `TidalError::Internal` — no eligible shards for the tenant, OR an /// assignment targets a non-local shard while cross-node dispatch is - /// unwired (see above). + /// unwired (see above). In the latter case NO signal was written. /// - Any error from the underlying `signal()` call. pub fn signal_for_tenant( &self, @@ -318,36 +330,41 @@ impl TidalDb { // Resolve all shard targets (1 in normal mode; 2 during dual-write migration). let assignments = self.tenant_router.write_assignments(tenant_id, entity_id)?; let local_shard = self.config.cluster.shard_id; - for assignment in &assignments { - if assignment.shard_id == local_shard { - // Local shard: write directly to the signal ledger. - self.signal(signal_type, entity_id, weight, timestamp)?; - } else { - // Remote shard (dual-write migration path). Cross-node dispatch - // over the replication transport is not yet wired (Task 5 in - // docs/specs/14 per CODING_GUIDELINES §12). Fail CLOSED instead of - // silently dropping the write: a dual-write that cannot reach its - // remote half would otherwise lose signals during migration with - // no error surfaced. Log at warn so the misconfiguration is - // observable, then return a typed error. - tracing::warn!( - tenant_id = tenant_id.0, - shard_id = assignment.shard_id.0, - entity_id = entity_id.as_u64(), - signal_type, - "signal_for_tenant: remote-shard dispatch is not wired (Task 5); \ - failing closed to avoid silently dropping the dual-write" - ); - return Err(crate::TidalError::internal( - "signal_for_tenant", - format!( - "remote-shard dispatch not yet wired: assignment targets shard {} \ - but this node is shard {}; cross-node dual-write requires the \ - transport-integrated shipper (Task 5, docs/specs/14)", - assignment.shard_id.0, local_shard.0 - ), - )); - } + + // Fail CLOSED *before* any write (W8): scan all assignments first. If any + // targets a non-local shard, the cross-node dispatch needed to satisfy + // the dual-write is unwired (Task 5, docs/specs/14), so the whole op + // cannot succeed. Returning here — before the local write — makes the + // operation atomic: a caller that retries the returned Err cannot + // double-count the local signal (signal()/record_signal accumulates and + // is not idempotent). A previous version wrote the local half first and + // only then errored, so every retry inflated the local weight. + if let Some(remote) = assignments.iter().find(|a| a.shard_id != local_shard) { + tracing::warn!( + tenant_id = tenant_id.0, + shard_id = remote.shard_id.0, + entity_id = entity_id.as_u64(), + signal_type, + "signal_for_tenant: remote-shard dispatch is not wired (Task 5); \ + failing closed BEFORE any write to avoid silently dropping the \ + dual-write or double-counting the local half on retry" + ); + return Err(crate::TidalError::internal( + "signal_for_tenant", + format!( + "remote-shard dispatch not yet wired: assignment targets shard {} \ + but this node is shard {}; cross-node dual-write requires the \ + transport-integrated shipper (Task 5, docs/specs/14)", + remote.shard_id.0, local_shard.0 + ), + )); + } + + // Every assignment is local: apply the local write(s). On a single node + // all assignments resolve to the same local shard, so the write happens + // once per assignment (normally exactly one). + for _assignment in &assignments { + self.signal(signal_type, entity_id, weight, timestamp)?; } Ok(()) } diff --git a/tidal/src/db/session_restore.rs b/tidal/src/db/session_restore.rs index cd0de24..beb2b29 100644 --- a/tidal/src/db/session_restore.rs +++ b/tidal/src/db/session_restore.rs @@ -4,25 +4,74 @@ use std::{ collections::HashMap, sync::{ Arc, - atomic::{AtomicBool, AtomicU64, Ordering}, + atomic::{AtomicBool, Ordering}, }, }; use super::TidalDb; use crate::{ - schema::EntityId, + schema::{EntityId, Timestamp}, session::{self as session_mod, AgentId, SessionId, SessionState}, storage::{Tag, encode_key, parse_key}, wal::format::session::SessionSeqNo, }; +/// Reconstruct a monotonic `Instant` that reflects a session's **true** age from +/// the durable wall-clock anchor `started_at_ns`. +/// +/// The monotonic clock is not durable: a restored session has no way to recover +/// the original `Instant`. Seeding it with a fresh `Instant::now()` would reset +/// the session's apparent age to ~zero, silently voiding every monotonic-clock +/// age computation (the TTL sweeper and the policy duration check both measure +/// age, and `build_snapshot` reports `started_at.elapsed()`). Back-dating the +/// `Instant` by the elapsed wall-clock time makes `started_at.elapsed()` and +/// `now.duration_since(started_at)` reflect the real age. `started_at_ns` remains +/// the authoritative source; the back-dated `Instant` is defense-in-depth so any +/// path still reading the monotonic clock stays correct. +/// +/// Falls back to `Instant::now()` only if the subtraction would underflow (the +/// recorded start is implausibly far in the past relative to process start). +fn back_dated_instant(started_at_ns: u64, now_ns: u64) -> std::time::Instant { + let elapsed_ns = now_ns.saturating_sub(started_at_ns); + std::time::Instant::now() + .checked_sub(std::time::Duration::from_nanos(elapsed_ns)) + .unwrap_or_else(std::time::Instant::now) +} + impl TidalDb { - /// Scan persistent storage for previously archived session snapshots. + /// Scan persistent storage for previously archived session snapshots and + /// reconcile durable session state with what was restored from the WAL. /// - /// Called once at startup from `TidalDbBuilder::open()`. Walks every key - /// in the items storage engine, filters for `Tag::Session` keys, and: + /// Called once at startup from `TidalDbBuilder::open()`, **after** + /// `restore_session_wal_events` has run (in `from_parts`), so `self.sessions` + /// already holds the sessions that had a WAL `Start` without a `Close`. Walks + /// every key in the items storage engine, filters for `Tag::Session` keys, + /// and: /// - `b"snapshot"` suffix -> deserializes into `closed_sessions` - /// - `b"start"` suffix -> logs a warning (session was active at shutdown) + /// - `b"start"` suffix -> a session that was active at last shutdown + /// + /// This routine re-derives two durable invariants that the in-memory restore + /// cannot infer on its own: + /// + /// 1. **Id-allocator high-water mark.** `next_session_id` is seeded to 1 on + /// every open; if it is not advanced past *every* durable session id, a + /// clean restart re-issues id 1 and the next `close_session` overwrites the + /// oldest archived snapshot — silent, permanent data loss. We track the max + /// id over every snapshot AND every start record and CAS-advance the + /// allocator past it (saturating, mirroring `restore_session_wal_events`). + /// Because this runs after the WAL path and the CAS only ever moves the + /// allocator forward, the final value is the max over both sources + /// regardless of call order. + /// + /// 2. **Orphaned start records.** A `b"start"` record with no live session + /// and no archived snapshot describes a session that was active at crash + /// time but whose WAL `Start` is gone (the append-only session journal is + /// never compacted, and recovery stops at the first torn record, so later + /// starts can be lost). Left alone it re-warns on every open forever and + /// leaks a dead key. We seal it once: build a frozen, zero-signal closed + /// snapshot (true age from `started_at_ns`) and atomically `put` the + /// snapshot + `delete` the start record, converting the unrecoverable + /// session into an archived record. /// /// In ephemeral mode the in-memory backend starts empty each time so this /// scan always returns nothing -- which is correct. @@ -33,6 +82,13 @@ impl TidalDb { let engine = storage.items_engine(); let mut restored = 0usize; let mut orphaned = 0usize; + let mut sealed = 0usize; + // High-water mark over every durable session id (snapshots + starts). + let mut max_id: Option = None; + // Orphaned start records to seal after the scan, as + // (session_id, user_id, started_at_ns, metadata). Deferred so we never + // mutate storage mid-iteration. + let mut orphans: Vec<(u64, u64, u64, HashMap)> = Vec::new(); for item in engine.scan_prefix(&[]) { let (key, value) = match item { @@ -51,6 +107,9 @@ impl TidalDb { match suffix { b"snapshot" => { if let Some(snapshot) = session_mod::deserialize_snapshot(&value) { + max_id = Some( + max_id.map_or(snapshot.id.as_u64(), |m| m.max(snapshot.id.as_u64())), + ); self.closed_sessions.insert(snapshot.id, snapshot); restored += 1; } else { @@ -58,23 +117,173 @@ impl TidalDb { } } b"start" => { - if let Some((session_id, _user_id, _started_at_ns, _metadata)) = + if let Some((session_id, user_id, started_at_ns, metadata)) = session_mod::deserialize_start_record(&value) { - tracing::warn!( - session_id = %session_id, - "session was active at last shutdown; in-memory signal state is lost" - ); + let id = session_id.as_u64(); + max_id = Some(max_id.map_or(id, |m| m.max(id))); + orphaned += 1; + // A start record whose session was rehydrated as active + // from the WAL will be cleaned up by its eventual close; + // leave it. Only seal records with no live session and no + // durable snapshot — those are truly unrecoverable. + if self.sessions.contains_key(&session_id) + || self.closed_sessions.contains_key(&session_id) + { + tracing::warn!( + session_id = %session_id, + "session was active at last shutdown; in-memory signal state is lost" + ); + } else { + orphans.push((id, user_id, started_at_ns, metadata)); + } } - orphaned += 1; } _ => {} } } - if restored > 0 || orphaned > 0 { - tracing::info!(restored, orphaned, "session restore complete"); + // Advance the id allocator past the highest durable session id so a + // reopen never re-issues a live/archived id and overwrites its snapshot. + if let Some(id) = max_id { + self.advance_next_session_id(id); } + + // Seal each truly-orphaned start record into a frozen closed snapshot. + for (session_id, user_id, started_at_ns, metadata) in orphans { + if self.seal_orphaned_start(session_id, user_id, started_at_ns, metadata) { + sealed += 1; + } + } + + if restored > 0 || orphaned > 0 || sealed > 0 { + tracing::info!(restored, orphaned, sealed, "session restore complete"); + } + } + + /// Advance `next_session_id` past `id` with a saturating CAS loop. + /// + /// Idempotent and monotonic: the allocator only ever moves forward, so it + /// converges to the max over every restore source regardless of call order. + /// A restored id of `u64::MAX` saturates the allocator at the ceiling rather + /// than wrapping back to 0 (which would re-issue a live id). + fn advance_next_session_id(&self, id: u64) { + loop { + let current = self.next_session_id.load(Ordering::Acquire); + if id < current { + break; + } + if self + .next_session_id + .compare_exchange( + current, + id.saturating_add(1), + Ordering::Release, + Ordering::Relaxed, + ) + .is_ok() + { + break; + } + } + } + + /// Seal an orphaned `b"start"` record into a frozen, zero-signal closed + /// snapshot, atomically replacing the start key with a snapshot key. + /// + /// Returns `true` if the record was sealed. The snapshot carries the true + /// wall-clock age (from the durable `started_at_ns`) and zero in-memory + /// signals — explicitly marking that the session's live signal state was + /// lost. This stops the recurring "orphaned" warning, reclaims the key, and + /// preserves the session as an archived record retrievable via + /// `session_snapshot`. + fn seal_orphaned_start( + &self, + session_id: u64, + user_id: u64, + started_at_ns: u64, + metadata: HashMap, + ) -> bool { + let Some(storage) = self.storage.as_ref() else { + return false; + }; + + let snapshot_key = encode_key(EntityId::new(session_id), Tag::Session, b"snapshot"); + let start_key = encode_key(EntityId::new(session_id), Tag::Session, b"start"); + let sid = SessionId::from_raw(session_id); + + // Defense-in-depth against scan order and torn closes: never overwrite an + // existing durable snapshot with a zero-signal seal. If a snapshot is + // already present (a torn close left snapshot+start both durable), the + // snapshot is authoritative — just delete the stale start record and warm + // the archive from the real snapshot. + if let Ok(Some(bytes)) = storage.items_engine().get(&snapshot_key) { + if let Some(snapshot) = session_mod::deserialize_snapshot(&bytes) { + self.closed_sessions.insert(sid, snapshot); + } + let mut batch = crate::storage::WriteBatch::new(); + batch.delete(start_key); + if let Err(e) = storage.items_engine().write_batch(batch) { + tracing::warn!( + error = %e, + session_id, + "failed to delete stale start record shadowed by a durable snapshot" + ); + return false; + } + tracing::info!( + session_id, + "deleted stale start record shadowed by an existing durable snapshot" + ); + return true; + } + + // Build a zero-signal SessionState carrying the durable identity, then + // freeze it. `agent_id` is not in the start record, so use the fallback + // "restored" marker (always a valid AgentId). The frozen `duration_ms` is + // the TRUE wall-clock age from `started_at_ns`; `build_frozen_snapshot` + // takes the duration explicitly, so the monotonic `started_at` here only + // needs to be self-consistent — back-date it for the same reason the live + // restore does, so any future reader of the monotonic clock stays correct. + let now_ns = Timestamp::now().as_nanos(); + let duration_ms = now_ns.saturating_sub(started_at_ns) / 1_000_000; + let started_at = back_dated_instant(started_at_ns, now_ns); + let agent_id = AgentId::new("restored").expect("'restored' is a valid AgentId"); + + let state = SessionState::new( + SessionId::from_raw(session_id), + user_id, + agent_id, + String::new(), + started_at, + started_at_ns, + metadata, + Arc::new(AtomicBool::new(true)), + ); + let snapshot = session_mod::build_frozen_snapshot(&state, duration_ms); + let snapshot_bytes = session_mod::serialize_snapshot(&snapshot); + + let mut batch = crate::storage::WriteBatch::new(); + batch.put(snapshot_key, snapshot_bytes); + batch.delete(start_key); + if let Err(e) = storage.items_engine().write_batch(batch) { + tracing::warn!( + error = %e, + session_id, + "failed to seal orphaned session start record" + ); + return false; + } + + // Warm the in-memory archive so session_snapshot resolves without a + // storage round-trip and active_sessions never lists it. + self.closed_sessions.insert(sid, snapshot); + self.enforce_closed_session_cap(); + tracing::info!( + session_id, + "sealed orphaned session start record into a frozen closed snapshot" + ); + true } /// Replay session journal events to restore active sessions from a crash. @@ -86,6 +295,21 @@ impl TidalDb { /// Called from `from_parts()` during startup. The storage-backed /// `restore_sessions()` is additive and handles archived snapshots; this /// method handles in-flight sessions that were active at crash time. + /// + /// # Durable snapshot is authoritative over the journal + /// + /// `close_session_internal` commits the durable `b"snapshot"` (and deletes + /// the `b"start"`) **before** appending the journal `Close`, and tolerates a + /// failed `Close` append with only a warning. The append-only session journal + /// is never compacted, so the original `Start` survives forever. A crash + /// between the snapshot commit and the `Close` append therefore leaves a + /// durable snapshot AND a `Start`-without-`Close` in the journal. Restoring + /// such a session as ACTIVE would land the same id in both `self.sessions` + /// and `closed_sessions` — a phantom session that accepts new signals against + /// a session the durable record (and the operator) consider closed. To + /// prevent this we probe storage for the session's `b"snapshot"` key before + /// building the active state and skip any session that has one: the durable + /// snapshot is the record of truth. #[allow(clippy::too_many_lines)] pub(super) fn restore_session_wal_events( &self, @@ -103,6 +327,26 @@ impl TidalDb { }; for (session_id, (user_id, started_at_ns, agent_id_str, policy_name)) in open_sessions { + // The durable snapshot is authoritative: if this session was already + // archived (snapshot committed but its journal Close was lost/torn), + // it is CLOSED. Restoring it as ACTIVE would place the same id in both + // self.sessions and closed_sessions (phantom state). Skip it — but + // still advance the allocator past its id below so a reuse cannot + // overwrite the snapshot. A point lookup on a cold path. + let has_snapshot = self.storage.as_ref().is_some_and(|storage| { + let key = encode_key(EntityId::new(session_id), Tag::Session, b"snapshot"); + storage.items_engine().get(&key).ok().flatten().is_some() + }); + if has_snapshot { + tracing::info!( + session_id, + "session has a durable closed snapshot; not restoring as active \ + (Close journal record was lost between snapshot commit and append)" + ); + self.advance_next_session_id(session_id); + continue; + } + if schema.session_policy(&policy_name).is_none() { tracing::warn!( session_id, @@ -137,23 +381,23 @@ impl TidalDb { .map(|(_, _, _, meta)| meta) .unwrap_or_default(); - let state = Arc::new(SessionState { - id: SessionId::from_raw(session_id), + // The monotonic Instant is not durable; back-date it from the durable + // `started_at_ns` so `started_at.elapsed()`, the TTL sweeper, and the + // policy duration check all see the session's TRUE age rather than + // time-since-restore. Seeding with a raw `Instant::now()` would reset + // a restored session's age to ~zero and silently void TTL enforcement. + let started_at = back_dated_instant(started_at_ns, Timestamp::now().as_nanos()); + + let state = Arc::new(SessionState::new( + SessionId::from_raw(session_id), user_id, agent_id, - policy_name: policy_name.clone(), - // We lost the exact monotonic Instant -- approximate with "now". - started_at: std::time::Instant::now(), + policy_name.clone(), + started_at, started_at_ns, metadata, - signals: dashmap::DashMap::new(), - signaled_entities: dashmap::DashMap::new(), - annotations: std::sync::Mutex::new(Vec::new()), - signals_written: AtomicU64::new(0), - signals_rejected: AtomicU64::new(0), - audit_log: std::sync::Mutex::new(session_mod::AuditLog::new()), closed, - }); + )); // Replay signals and annotations into the restored session state. // Advance the seqno HWM so incoming replication events are correctly deduped. @@ -223,29 +467,10 @@ impl TidalDb { let sid = SessionId::from_raw(session_id); self.sessions.insert(sid, state); - // Advance next_session_id past any restored session IDs. Use a - // saturating add so a restored id of u64::MAX cannot wrap the - // allocator back to 0 (which would re-issue a live id); at the - // saturation ceiling the CAS becomes a no-op once `current == - // u64::MAX`, leaving the allocator pinned rather than wrapping. - loop { - let current = self.next_session_id.load(Ordering::Acquire); - if session_id < current { - break; - } - if self - .next_session_id - .compare_exchange( - current, - session_id.saturating_add(1), - Ordering::Release, - Ordering::Relaxed, - ) - .is_ok() - { - break; - } - } + // Advance next_session_id past any restored session IDs (saturating, + // so a restored id of u64::MAX pins the allocator rather than wrapping + // back to 0 and re-issuing a live id). + self.advance_next_session_id(session_id); tracing::info!(session_id, user_id, "restored active session from WAL"); } @@ -383,4 +608,191 @@ mod tests { ); db.close().unwrap(); } + + /// Build a frozen, zero-signal snapshot for `session_id`/`user_id` so a test + /// can plant a durable `b"snapshot"` row directly in storage. + fn frozen_snapshot_bytes(session_id: u64, user_id: u64) -> Vec { + use std::sync::{Arc, atomic::AtomicBool}; + + use crate::session::{AgentId, SessionId, SessionState}; + + let state = SessionState::new( + SessionId::from_raw(session_id), + user_id, + AgentId::new("agent").unwrap(), + "agent".to_string(), + std::time::Instant::now(), + crate::schema::Timestamp::now().as_nanos(), + std::collections::HashMap::new(), + Arc::new(AtomicBool::new(true)), + ); + let snap = crate::session::build_frozen_snapshot(&state, 0); + crate::session::serialize_snapshot(&snap) + } + + /// CRITICAL regression: a session with a durable `b"snapshot"` is CLOSED. A + /// stale `Start`-without-`Close` in the journal (the snapshot committed but + /// its `Close` append was lost) must NOT restore it as ACTIVE — that would + /// land the same id in both `self.sessions` and the durable archive (phantom + /// state). The restore path probes for the snapshot and skips, while still + /// advancing the allocator past the id. + #[test] + fn durable_snapshot_blocks_active_restore() { + use crate::{ + schema::EntityId, + session::SessionId, + storage::{Tag, encode_key}, + }; + + let dir = tempfile::tempdir().unwrap(); + let db = TidalDb::builder() + .with_data_dir(dir.path()) + .with_schema(schema_with_session_policy()) + .open() + .unwrap(); + + // Plant a durable closed snapshot for id 5. + let snap_key = encode_key(EntityId::new(5), Tag::Session, b"snapshot"); + db.storage + .as_ref() + .unwrap() + .items_engine() + .put(&snap_key, &frozen_snapshot_bytes(5, 99)) + .unwrap(); + + // A surviving Start-without-Close for the SAME id. + let events = vec![SessionWalEvent::Start { + session_id: 5, + user_id: 99, + started_at_ns: crate::schema::Timestamp::now().as_nanos(), + agent_id: "agent".to_string(), + policy_name: "agent".to_string(), + }]; + db.restore_session_wal_events(&events); + + // The session must NOT be active (durable snapshot is authoritative)... + assert!( + !db.sessions.contains_key(&SessionId::from_raw(5)), + "a session with a durable snapshot must not be restored as active" + ); + // ...but the allocator must still have advanced past its id so a reuse + // cannot overwrite the snapshot. + assert!( + db.next_session_id.load(Ordering::Acquire) > 5, + "the allocator must advance past a snapshot-shadowed restore id" + ); + db.close().unwrap(); + } + + /// WARNING regression: an orphaned `b"start"` record (no live session, no + /// durable snapshot, and no matching WAL Start) must be sealed exactly once + /// into a frozen closed snapshot — the start key is deleted, a snapshot key + /// is written, the session is queryable as archived, and the allocator is + /// advanced past its id. This stops the permanently-re-firing "orphaned" + /// warning and reclaims the dead key. + #[test] + fn orphaned_start_is_sealed_into_closed_snapshot() { + use std::sync::{Arc, atomic::AtomicBool}; + + use crate::{ + schema::EntityId, + session::{AgentId, SessionId, SessionState}, + storage::{Tag, encode_key}, + }; + + let dir = tempfile::tempdir().unwrap(); + let db = TidalDb::builder() + .with_data_dir(dir.path()) + .with_schema(schema_with_session_policy()) + .open() + .unwrap(); + + // Plant an orphaned start record for id 7 (no WAL Start, no snapshot). + let orphan = SessionState::new( + SessionId::from_raw(7), + 42, + AgentId::new("agent").unwrap(), + "agent".to_string(), + std::time::Instant::now(), + crate::schema::Timestamp::now().as_nanos(), + std::collections::HashMap::new(), + Arc::new(AtomicBool::new(false)), + ); + let start_bytes = crate::session::serialize_start_record(&orphan); + let start_key = encode_key(EntityId::new(7), Tag::Session, b"start"); + let snapshot_key = encode_key(EntityId::new(7), Tag::Session, b"snapshot"); + let engine = db.storage.as_ref().unwrap().items_engine(); + engine.put(&start_key, &start_bytes).unwrap(); + + db.restore_sessions(); + + // Start key deleted, snapshot key present. + let engine = db.storage.as_ref().unwrap().items_engine(); + assert!( + engine.get(&start_key).unwrap().is_none(), + "the orphaned start record must be deleted after sealing" + ); + assert!( + engine.get(&snapshot_key).unwrap().is_some(), + "a frozen closed snapshot must replace the orphaned start record" + ); + // Queryable as archived, not active. + let snap = db.session_snapshot(SessionId::from_raw(7)).unwrap(); + assert_eq!(snap.user_id, 42); + assert!( + !db.active_sessions() + .iter() + .any(|s| s.id == SessionId::from_raw(7)), + "a sealed orphan must not appear as an active session" + ); + // Allocator advanced past the orphan id. + assert!( + db.next_session_id.load(Ordering::Acquire) > 7, + "the allocator must advance past the sealed orphan id" + ); + + // Idempotent: a second restore finds no orphan (snapshot now present) and + // does not re-warn or re-seal. + db.restore_sessions(); + let engine = db.storage.as_ref().unwrap().items_engine(); + assert!(engine.get(&start_key).unwrap().is_none()); + db.close().unwrap(); + } + + /// BLOCKER regression (unit-level): `restore_sessions` advances the id + /// allocator past every durable snapshot id, so a reopen never re-issues an + /// archived id. + #[test] + fn restore_sessions_advances_allocator_over_snapshots() { + use crate::{ + schema::EntityId, + storage::{Tag, encode_key}, + }; + + let dir = tempfile::tempdir().unwrap(); + let db = TidalDb::builder() + .with_data_dir(dir.path()) + .with_schema(schema_with_session_policy()) + .open() + .unwrap(); + + // Plant durable snapshots for ids 3 and 11. + for id in [3u64, 11] { + let key = encode_key(EntityId::new(id), Tag::Session, b"snapshot"); + db.storage + .as_ref() + .unwrap() + .items_engine() + .put(&key, &frozen_snapshot_bytes(id, id)) + .unwrap(); + } + + db.restore_sessions(); + + assert!( + db.next_session_id.load(Ordering::Acquire) > 11, + "the allocator must advance past the highest durable snapshot id (11)" + ); + db.close().unwrap(); + } } diff --git a/tidal/src/db/sessions.rs b/tidal/src/db/sessions.rs index 77a7e79..9963784 100644 --- a/tidal/src/db/sessions.rs +++ b/tidal/src/db/sessions.rs @@ -4,7 +4,7 @@ use std::{ collections::HashMap, sync::{ Arc, - atomic::{AtomicBool, AtomicU64, Ordering}, + atomic::{AtomicBool, Ordering}, }, }; @@ -63,22 +63,16 @@ impl TidalDb { let started_at = std::time::Instant::now(); let started_at_ns = Timestamp::now().as_nanos(); - let state = Arc::new(SessionState { - id: session_id, + let state = Arc::new(SessionState::new( + session_id, user_id, - agent_id: parsed_agent_id.clone(), - policy_name: policy_name.to_owned(), + parsed_agent_id.clone(), + policy_name.to_owned(), started_at, started_at_ns, metadata, - signals: dashmap::DashMap::new(), - signaled_entities: dashmap::DashMap::new(), - annotations: std::sync::Mutex::new(Vec::new()), - signals_written: AtomicU64::new(0), - signals_rejected: AtomicU64::new(0), - audit_log: std::sync::Mutex::new(session_mod::AuditLog::new()), - closed: Arc::clone(&closed), - }); + Arc::clone(&closed), + )); self.sessions.insert(session_id, Arc::clone(&state)); @@ -267,7 +261,7 @@ impl TidalDb { /// cooperate — a double-remove of the same oldest id simply returns `None`. /// Any overshoot from concurrent inserts is therefore transient and /// self-healing, never a persistent cap violation. - fn enforce_closed_session_cap(&self) { + pub(super) fn enforce_closed_session_cap(&self) { if self.closed_sessions.len() <= session_mod::MAX_CLOSED_SESSIONS { return; } @@ -375,7 +369,11 @@ impl TidalDb { && let Some(policy) = schema.session_policy(&state.policy_name) { let evaluator = session_mod::PolicyEvaluator::new(policy, &state.policy_name); - match evaluator.check(signal_type, state, std::time::Instant::now()) { + // Wall-clock now (durable age basis), not the monotonic Instant: a + // restored session reseeds `started_at`, so the policy duration check + // must measure off the durable `started_at_ns` exactly as the sweeper + // and `close_session_internal` do. + match evaluator.check(signal_type, state, Timestamp::now().as_nanos()) { Ok(()) => { // Record accepted entry via bounded AuditLog. let entry = AuditEntry { diff --git a/tidal/src/db/signals.rs b/tidal/src/db/signals.rs index 76d7c83..c9e200a 100644 --- a/tidal/src/db/signals.rs +++ b/tidal/src/db/signals.rs @@ -63,6 +63,40 @@ impl TidalDb { .map_err(TidalError::from) } + /// Shared weight-validation guard for the signal-write entry points. + /// + /// Both [`signal`](Self::signal) and [`signal_scoped`](Self::signal_scoped) + /// run this identical preamble, so it lives in one place — like + /// [`check_write_backpressure`](Self::check_write_backpressure) — to keep the + /// two admission policies from drifting apart. + /// + /// Rejects the write with [`TidalError::InvalidInput`] when the weight is: + /// - **non-finite** (`NaN` / `±Inf`): would corrupt every decay-score CAS, or + /// - **finite negative**: spec §8 says "negative weights are rejected + /// (negative signals use separate signal types, not negative weights)". A + /// negative weight folds into the decay accumulator and can drive the hot + /// decay score below zero, violating the INV-SIG-3 "decay scores are + /// non-negative" invariant (and tripping the `debug_assert!` in + /// [`HotSignalState::on_signal`](crate::signals::HotSignalState::on_signal)). + /// + /// A weight of exactly `0.0` is permitted: it is a no-op fold that records the + /// event in the warm-tier count without perturbing the decay score. + pub(crate) fn validate_signal_weight(weight: f64) -> crate::Result<()> { + if !weight.is_finite() { + return Err(TidalError::invalid_input( + "signal weight must be finite (NaN and Inf are not allowed)", + )); + } + if weight < 0.0 { + return Err(TidalError::invalid_input( + "signal weight must be non-negative; negative engagement uses \ + separate signal types (e.g. \"dislike\"), not negative weights \ + (spec §8)", + )); + } + Ok(()) + } + /// Shared write-admission guard for the signal-write entry points. /// /// Rejects the write with [`TidalError::Backpressure`] when either: @@ -113,6 +147,8 @@ impl TidalDb { /// # Errors /// /// - `TidalError::Internal` if no ledger is wired (use `with_schema()`). + /// - `TidalError::InvalidInput` if `weight` is non-finite (NaN/Inf) or + /// finite-negative (negative engagement uses separate signal types). /// - `TidalError::Schema` if `signal_type` is not defined in the schema. /// - `TidalError::Durability` if the WAL write fails. /// - `TidalError::Backpressure` if the WAL queue is saturated or a backup is in progress. @@ -125,11 +161,7 @@ impl TidalDb { timestamp: Timestamp, ) -> crate::Result<()> { self.require_writeable("signal")?; - if !weight.is_finite() { - return Err(TidalError::invalid_input( - "signal weight must be finite (NaN and Inf are not allowed)", - )); - } + Self::validate_signal_weight(weight)?; // Reject the write if a backup is in progress or the WAL queue is // saturated, before doing any work or starting the latency timer. @@ -174,6 +206,8 @@ impl TidalDb { /// # Errors /// /// - `TidalError::Internal` if no ledger is wired (use `with_schema()`). + /// - `TidalError::InvalidInput` if `weight` is non-finite (NaN/Inf) or + /// finite-negative (negative engagement uses separate signal types). /// - `TidalError::Schema` if `signal_type` is not defined in the schema. /// - `TidalError::Durability` if the WAL write fails. /// - `TidalError::Backpressure` if the WAL queue is saturated or a backup is in progress. @@ -187,11 +221,7 @@ impl TidalDb { share_policy_version: u16, ) -> crate::Result<()> { self.require_writeable("signal_scoped")?; - if !weight.is_finite() { - return Err(TidalError::invalid_input( - "signal weight must be finite (NaN and Inf are not allowed)", - )); - } + Self::validate_signal_weight(weight)?; self.check_write_backpressure()?; @@ -325,7 +355,11 @@ impl TidalDb { /// /// # Errors /// - /// Returns errors from the underlying `signal()` method. + /// - `TidalError::InvalidInput` if `for_user` is set and `entity_id` exceeds + /// the `u32` item-universe limit (it would alias a lower id in the durable + /// per-user rows). + /// - Any error from the underlying [`signal`](Self::signal) call (invalid + /// weight, missing ledger, unknown signal type, WAL durability/backpressure). pub fn signal_with_context( &self, signal_type: &str, @@ -336,12 +370,35 @@ impl TidalDb { creator_id: Option, ) -> crate::Result<()> { self.require_writeable("signal_with_context")?; + + // When a `for_user` identity is present this call narrows the item id to + // its u32 slot and writes it into DURABLE Tag::HardNeg / Tag::UserState + // rows. A bare `as u32` truncation would silently alias two items whose + // ids share their low 32 bits — a permanent cross-id collision in the + // hard-negative / seen / saved / liked correctness primitives that + // survives restart. Reject an over-range id up front (mirroring + // `write_item_with_metadata`) so a colliding durable row never lands on + // disk. Done before the base `signal()` so a rejected write leaves no + // trace at all. + if for_user.is_some() && entity_id.as_u64() > u64::from(u32::MAX) { + let raw = entity_id.as_u64(); + return Err(TidalError::invalid_input(format!( + "item entity id {raw} exceeds the u32 item-universe limit ({}); \ + allocate dense item IDs within the u32 range", + u32::MAX + ))); + } + // Record the base signal. self.signal(signal_type, entity_id, weight, timestamp)?; // Signal dispatch: side effects based on signal type and context. if let Some(user_id) = for_user { - let item_u32 = entity_id.as_u64() as u32; + // Lossless: the over-range guard above already rejected any id past + // u32::MAX. Route through the observable helper anyway so this path + // shares the single narrowing seam with the Hide relationship path + // and items.rs (a future relaxation of the guard stays observable). + let item_u32 = crate::entities::user_state::narrow_item_slot(entity_id.as_u64()); // Durable-write target for the write-through side effects below. // `None` in ephemeral mode (no persistence; in-memory state is the // whole story and there is nothing to crash-recover). @@ -747,4 +804,93 @@ mod tests { ) .unwrap(); } + + /// Regression (signals, pass2): the finite-weight + non-negative-weight rule + /// is shared by `signal` and `signal_scoped` via `validate_signal_weight`, so + /// neither path can drift. Spec §8: "negative weights are rejected (negative + /// signals use separate signal types, not negative weights)". A finite + /// negative weight folds into the decay accumulator and can drive the hot + /// decay score below zero, violating INV-SIG-3. + #[test] + fn both_signal_paths_reject_negative_and_non_finite_weight() { + let db = TidalDb::builder() + .ephemeral() + .with_schema(schema_with_view()) + .open() + .unwrap(); + + for bad in [-1.0_f64, -f64::MIN_POSITIVE, f64::NAN, f64::INFINITY] { + let unscoped = db.signal("view", EntityId::new(1), bad, Timestamp::now()); + assert!( + matches!(unscoped, Err(crate::schema::TidalError::InvalidInput(_))), + "signal must reject weight {bad}, got {unscoped:?}" + ); + + let scoped = db.signal_scoped( + "view", + EntityId::new(1), + bad, + Timestamp::now(), + SignalScope::Local, + 0, + ); + assert!( + matches!(scoped, Err(crate::schema::TidalError::InvalidInput(_))), + "signal_scoped must reject weight {bad} through the same shared guard, got {scoped:?}" + ); + } + + // 0.0 and positive weights remain accepted (0.0 is a no-op fold that + // still records the event in the warm-tier count). + db.signal("view", EntityId::new(2), 0.0, Timestamp::now()) + .unwrap(); + db.signal("view", EntityId::new(3), 1.5, Timestamp::now()) + .unwrap(); + } + + /// Regression (W5, pass2): `signal_with_context` writes the item id into + /// DURABLE `Tag::HardNeg` / `Tag::UserState` rows narrowed to a u32 slot. An id + /// past `u32::MAX` would silently alias a lower id and collide permanently on + /// disk. The path must reject such an id up front (mirroring + /// `write_item_with_metadata`) so no colliding durable row ever lands. + #[test] + fn signal_with_context_rejects_over_range_item_id_for_user_path() { + let db = TidalDb::builder() + .ephemeral() + .with_schema(schema_with_view()) + .open() + .unwrap(); + + let oversized = EntityId::new(u64::from(u32::MAX) + 1); + let err = db + .signal_with_context( + "view", + oversized, + 1.0, + Timestamp::now(), + Some(7), // for_user present -> durable per-user rows would be written + None, + ) + .unwrap_err(); + assert!( + matches!(err, crate::schema::TidalError::InvalidInput(_)), + "over-range item id on the for_user path must be rejected, got {err:?}" + ); + + // The boundary id (u32::MAX) narrows losslessly and is accepted. + db.signal_with_context( + "view", + EntityId::new(u64::from(u32::MAX)), + 1.0, + Timestamp::now(), + Some(7), + None, + ) + .unwrap(); + + // With no for_user identity, no per-user durable row is written, so the + // over-range guard does not apply — the base signal is recorded. + db.signal_with_context("view", oversized, 1.0, Timestamp::now(), None, None) + .unwrap(); + } } diff --git a/tidal/src/db/state_rebuild.rs b/tidal/src/db/state_rebuild.rs index 9fcc383..ccea537 100644 --- a/tidal/src/db/state_rebuild.rs +++ b/tidal/src/db/state_rebuild.rs @@ -376,16 +376,22 @@ pub(super) fn rebuild_item_indexes( let (key, value) = entry.map_err(TidalError::from)?; if let Some(entity_id) = item_meta_entity_id(&key) { - // The write path warns and refuses to index entity IDs above - // u32::MAX (roaring bitmaps are u32-keyed); mirror that here so a - // truncating collision is observable rather than silent. + // The write path (db/items.rs) REJECTS any item id > u32::MAX before + // it ever lands in storage, so the only way one reaches here is a + // corrupt or hostile durable store. Mirror the write path's refusal: + // skip the row entirely rather than truncate it into a colliding u32 + // slot, which would alias a real lower id and manufacture phantom + // query hits. Failing safe on corrupt input beats silently corrupting + // the in-memory universe/indexes. if entity_id.as_u64() > u64::from(u32::MAX) { tracing::warn!( entity_id = entity_id.as_u64(), - "entity ID exceeds u32::MAX during rebuild; \ - universe bitmap entry will collide with a lower ID" + "entity ID exceeds u32::MAX during rebuild; skipping \ + (write path never persists such ids)" ); + continue; } + // In range by the guard above, so the narrowing is exact. let id_u32 = entity_id.as_u64() as u32; let meta = deserialize_metadata(&value); @@ -598,8 +604,13 @@ pub(super) fn run_checkpoint_thread( // window to the interval instead of "everything since the last clean // shutdown" (the community ledger is the SOLE durable copy of // per-contributor identity, so this is its only periodic durability). - checkpoint_derived_ledgers( + // + // The cohort ledger is already checkpointed above with `meta`, so the + // shared helper is called WITHOUT a cohort argument here to avoid a + // double cohort write in the same cycle. + let _ = checkpoint_secondary_ledgers( storage.as_ref(), + None, community_ledger.as_ref(), co_engagement.as_ref(), preference_vectors.as_ref(), @@ -622,31 +633,70 @@ pub(super) fn run_checkpoint_thread( let _ = &metrics; } -/// Checkpoint the derived per-user / per-contributor ledgers (community, -/// co-engagement, preference vectors) into `storage`. +/// Checkpoint the *secondary derived* ledgers — cohort (optional), community, +/// co-engagement, and preference vectors — into `storage`, with ONE consistent +/// set of guards and error handling. /// -/// None of these has a WAL backstop for its per-user/per-contributor identity, so -/// this periodic checkpoint is what bounds their post-crash loss window. Each is -/// independent and non-fatal: a failure is logged and the others still run. -fn checkpoint_derived_ledgers( +/// This is the single shared implementation behind every secondary-checkpoint +/// site: the periodic checkpoint thread (above), shutdown +/// ([`TidalDb::shutdown_inner`](crate::db::TidalDb)), and backup +/// ([`TidalDb::create_backup`](crate::db::TidalDb)). Previously each site +/// open-coded the same four-checkpoint block with subtly different guards +/// (`entry_count() > 0` / `edge_count() > 0` / `!is_empty()` / none) and +/// divergent error handling; a new derived ledger or a changed guard had to be +/// edited in four places or one path silently skipped a checkpoint. Centralizing +/// it makes that drift impossible. +/// +/// `cohort` is `Some((ledger, meta))` only where the cohort write is wanted in +/// this pass (shutdown / backup). The periodic thread passes `None` because it +/// already checkpoints the cohort ledger with its own `meta` earlier in the same +/// cycle — passing `Some` here would write it twice. +/// +/// None of these ledgers has a WAL backstop for its per-user/per-contributor +/// identity, so this checkpoint is what bounds their post-crash loss window. Each +/// is independent: a failure is logged and the remaining checkpoints still run. +/// The FIRST error is returned so callers that must surface it (shutdown folds it +/// into its `first_err`) can, while callers that only need best-effort (backup, +/// periodic) discard it. +pub(super) fn checkpoint_secondary_ledgers( storage: &dyn StorageEngine, + cohort: Option<( + &CohortSignalLedger, + crate::signals::checkpoint::CheckpointMeta, + )>, community_ledger: &crate::governance::CommunityLedger, co_engagement: &crate::entities::CoEngagementIndex, preference_vectors: &crate::entities::PreferenceVectors, -) { +) -> Option { + let mut first_err: Option = None; + if let Some((cohort_ledger, meta)) = cohort + && cohort_ledger.entry_count() > 0 + && let Err(e) = cohort_ledger.checkpoint(storage, meta) + { + tracing::error!(error = %e, "cohort checkpoint failed"); + first_err.get_or_insert(e); + } + // The community ledger is the SOLE durable copy of per-contributor identity, + // so it is always checkpointed (no emptiness guard): an empty checkpoint is a + // cheap no-op, but skipping it on a transiently-empty in-memory map could + // leave a stale durable snapshot. if let Err(e) = community_ledger.checkpoint(storage) { - tracing::error!(error = %e, "periodic community checkpoint failed"); + tracing::error!(error = %e, "community checkpoint failed"); + first_err.get_or_insert(e); } if co_engagement.edge_count() > 0 && let Err(e) = co_engagement.checkpoint(storage) { - tracing::error!(error = %e, "periodic co-engagement checkpoint failed"); + tracing::error!(error = %e, "co-engagement checkpoint failed"); + first_err.get_or_insert(e); } if !preference_vectors.is_empty() && let Err(e) = preference_vectors.checkpoint(storage) { - tracing::error!(error = %e, "periodic preference-vector checkpoint failed"); + tracing::error!(error = %e, "preference-vector checkpoint failed"); + first_err.get_or_insert(e); } + first_err } /// Poll text-index liveness and schedule a resync rebuild when unhealthy. @@ -1021,3 +1071,223 @@ mod wal_lag_tests { assert_eq!(lag, a + b, "no checkpoint => all WAL bytes are lag"); } } + +#[cfg(test)] +#[allow(clippy::unwrap_used)] +mod rebuild_tests { + use std::collections::HashMap; + + use roaring::RoaringBitmap; + + use super::{ItemIndexes, rebuild_entity_state, rebuild_item_indexes}; + use crate::{ + db::{metadata::serialize_metadata, storage_box::StorageBox}, + entities::{ + CreatorItemsBitmap, InteractionLedger, UserStateIndex, + relationship::{ + RelationshipType, encode_relationship_key, serialize_relationship_value, + }, + }, + schema::{EntityId, Timestamp}, + storage::{ + InMemoryBackend, Tag, encode_key, + indexes::{bitmap::BitmapIndex, range::RangeIndex}, + }, + }; + + fn empty_memory_storage() -> StorageBox { + StorageBox::Memory { + items: InMemoryBackend::new(), + users: InMemoryBackend::new(), + creators: InMemoryBackend::new(), + } + } + + /// Write one item-metadata row exactly as the live write path does + /// (`encode_key(id, Tag::Meta, b"") -> serialize_metadata(map)`). + fn put_item_meta(storage: &StorageBox, id: EntityId, meta: &HashMap) { + let key = encode_key(id, Tag::Meta, b""); + storage + .items_engine() + .put(&key, &serialize_metadata(meta)) + .unwrap(); + } + + fn meta(pairs: &[(&str, &str)]) -> HashMap { + pairs + .iter() + .map(|(k, v)| ((*k).to_owned(), (*v).to_owned())) + .collect() + } + + /// Direct contract test for `rebuild_entity_state`: durable relationship + /// edges (block / hide / follow / interaction) must reconstruct the identical + /// in-memory `UserStateIndex` sets and interaction ledger after a restart. + #[test] + fn rebuild_entity_state_reconstructs_relationship_state() { + let storage = empty_memory_storage(); + let ts = Timestamp::now(); + let (user, creator, item) = (EntityId::new(1), EntityId::new(100), EntityId::new(7)); + + // Persist edges into the users keyspace exactly as the write path does. + let edges = [ + (user, RelationshipType::Blocks, creator), + (user, RelationshipType::Hide, item), + (user, RelationshipType::Follows, creator), + (user, RelationshipType::InteractionWeight, creator), + ]; + for (from, rel, to) in edges { + let key = encode_relationship_key(from, rel, to); + let value = serialize_relationship_value(2.5, ts); + storage.users_engine().put(&key, &value).unwrap(); + } + + let user_state = UserStateIndex::new(); + let interaction = InteractionLedger::new(); + rebuild_entity_state(&storage, &user_state, &interaction).unwrap(); + + // Blocks -> blocked_creators. + assert!( + user_state.blocked_creators(1).contains(&100), + "block edge must rebuild into blocked_creators" + ); + // Hide -> hidden_items. + assert!( + user_state.hidden_items(1).contains(7), + "hide edge must rebuild into hidden_items" + ); + // Follows -> followed_creators (and reverse follower set). + assert!( + user_state.followed_creators(1).contains(&100), + "follow edge must rebuild into followed_creators" + ); + // InteractionWeight -> interaction ledger carries the persisted weight. + assert!( + interaction.score(1, 100, ts.as_nanos()) > 0.0, + "interaction edge must rebuild a non-zero decayed weight" + ); + } + + /// Direct contract test for `rebuild_item_indexes`: a persisted item's + /// metadata must reconstruct the identical universe bitmap, every metadata + /// index, the `creator_items` bitmap, and the `created_at` recency entry that the + /// shared `ItemIndexes::index` write path would produce. + #[test] + fn rebuild_item_indexes_reconstructs_every_index() { + let storage = empty_memory_storage(); + + // One fully-tagged item, written via the canonical metadata codec. + put_item_meta( + &storage, + EntityId::new(42), + &meta(&[ + ("category", "jazz"), + ("format", "audio"), + ("creator_id", "100"), + ("tags", "smooth,live"), + ("duration", "180"), + ("created_at", "1000"), + ]), + ); + + let mut universe = RoaringBitmap::new(); + let category = BitmapIndex::new("category"); + let format = BitmapIndex::new("format"); + let creator = BitmapIndex::new("creator_id"); + let tag = BitmapIndex::new("tags"); + let duration = RangeIndex::::new("duration"); + let created_at = RangeIndex::::new("created_at"); + let creator_items = CreatorItemsBitmap::new(); + let indexes = ItemIndexes { + category: &category, + format: &format, + creator: &creator, + tag: &tag, + duration: &duration, + created_at: &created_at, + creator_items: &creator_items, + }; + + rebuild_item_indexes(&storage, &mut universe, &indexes).unwrap(); + + assert!(universe.contains(42), "item must enter the universe bitmap"); + assert!( + category.get("jazz").is_some_and(|bm| bm.contains(42)), + "category index must be rebuilt" + ); + assert!( + format.get("audio").is_some_and(|bm| bm.contains(42)), + "format index must be rebuilt" + ); + assert!( + creator.get("100").is_some_and(|bm| bm.contains(42)), + "creator index must be rebuilt" + ); + assert!( + tag.get("smooth").is_some_and(|bm| bm.contains(42)) + && tag.get("live").is_some_and(|bm| bm.contains(42)), + "each comma-separated tag must be rebuilt" + ); + assert!( + creator_items.get(100).is_some_and(|bm| bm.contains(42)), + "creator_items bitmap must be rebuilt for the `following` profile" + ); + // Recency: the created_at value must land in the range index. + let recent = std::ops::Bound::Included(&1000u64); + let recent = created_at.range(recent, std::ops::Bound::Included(&1000u64)); + assert!( + recent.contains(42), + "created_at recency entry must be rebuilt at the persisted value" + ); + } + + /// Corrupt/hostile input regression (WARNING fix): the write path rejects any + /// item id > `u32::MAX` before it ever persists, so such a row can only reach + /// the rebuild from a corrupt store. The rebuild must SKIP it — never truncate + /// it into a colliding u32 slot that aliases a real lower id and manufactures + /// phantom query hits. + #[test] + fn rebuild_item_indexes_skips_over_u32_ids_instead_of_colliding() { + let storage = empty_memory_storage(); + + // A legitimate low item id, and a hostile id that truncates onto it + // (`(1<<32) + 5` -> 5). If the rebuild truncated, both would land on + // slot 5 with last-write-wins metadata corruption. + let low = EntityId::new(5); + let over = EntityId::new((1u64 << 32) + 5); + put_item_meta(&storage, low, &meta(&[("category", "real")])); + put_item_meta(&storage, over, &meta(&[("category", "phantom")])); + + let mut universe = RoaringBitmap::new(); + let category = BitmapIndex::new("category"); + let format = BitmapIndex::new("format"); + let creator = BitmapIndex::new("creator_id"); + let tag = BitmapIndex::new("tags"); + let duration = RangeIndex::::new("duration"); + let created_at = RangeIndex::::new("created_at"); + let creator_items = CreatorItemsBitmap::new(); + let indexes = ItemIndexes { + category: &category, + format: &format, + creator: &creator, + tag: &tag, + duration: &duration, + created_at: &created_at, + creator_items: &creator_items, + }; + + rebuild_item_indexes(&storage, &mut universe, &indexes).unwrap(); + + // The legitimate row is indexed; the over-range row is skipped, so slot 5 + // is NOT cross-contaminated with the "phantom" category. + assert!(universe.contains(5), "the valid low id must be indexed"); + assert!( + category.get("real").is_some_and(|bm| bm.contains(5)), + "the valid low id keeps its true category" + ); + assert!( + category.get("phantom").is_none(), + "the over-u32::MAX row must be skipped, not truncated into slot 5" + ); + } +} diff --git a/tidal/src/db/sweeper.rs b/tidal/src/db/sweeper.rs index ed595ec..58fea0a 100644 --- a/tidal/src/db/sweeper.rs +++ b/tidal/src/db/sweeper.rs @@ -18,13 +18,24 @@ impl TidalDb { /// policy's `max_session_duration`. /// /// Called every 60s by the sweeper thread, and once during shutdown. + /// + /// Age is measured from the **durable** `started_at_ns` (nanoseconds since + /// the Unix epoch), never from the monotonic `started_at` `Instant`. A + /// session restored on open re-seeds (and back-dates) `started_at`, but only + /// `started_at_ns` is the durable wall-clock anchor: measuring off it ensures + /// a session that was already over its `max_session_duration` at shutdown is + /// reaped on the first startup sweep, exactly as `start_sweeper` documents + /// ("sessions that expired while the process was down ... are reaped promptly + /// at startup"). Reading the monotonic clock here would reset every restored + /// session's age to ~zero and silently void the TTL bound across restarts. pub(crate) fn sweep_expired_sessions(&self) { - let now = std::time::Instant::now(); + let now_ns = crate::schema::Timestamp::now().as_nanos(); let mut expired_ids = Vec::new(); for entry in &self.sessions { let state = entry.value(); - let elapsed = now.duration_since(state.started_at); + let elapsed = + std::time::Duration::from_nanos(now_ns.saturating_sub(state.started_at_ns)); // Look up the policy's max_session_duration. let max_duration = self @@ -308,4 +319,80 @@ mod tests { "today's entry must be retained" ); } + + /// CRITICAL/BLOCKER regression: the sweeper must TTL-expire a *restored* + /// session. A restored session re-seeds the monotonic `started_at` with a + /// fresh `Instant::now()`; only the durable `started_at_ns` carries the true + /// age. Measuring age off the monotonic clock would hand every restored + /// session a fresh full lease on each restart and never reap it. The sweeper + /// now measures off `started_at_ns`, so a session that was already past its + /// `max_session_duration` at shutdown is reaped on the first startup sweep. + #[test] + fn sweeper_expires_restored_over_age_session() { + use std::{ + sync::{ + Arc, + atomic::{AtomicBool, AtomicU64}, + }, + time::{Duration, Instant}, + }; + + use crate::{ + schema::{AgentPolicy, DecaySpec, EntityKind, SchemaBuilder, Timestamp}, + session::{AgentId, AuditLog, SessionId, SessionState}, + }; + + let mut b = SchemaBuilder::new(); + let _ = b + .signal("view", EntityKind::Item, DecaySpec::Permanent) + .add(); + b.session_policy( + "short", + AgentPolicy { + allowed_signals: vec![], + denied_signals: vec![], + // 1-hour cap; our restored session is 2h old. + max_session_duration: Duration::from_secs(3600), + max_signals_per_session: 0, + }, + ); + let schema = b.build().unwrap(); + + let db = TidalDb::builder() + .ephemeral() + .with_schema(schema) + .open() + .unwrap(); + + // Restore shape: monotonic clock fresh, durable start 2h ago. + let now_ns = Timestamp::now().as_nanos(); + let two_hours_ns = 2 * 3600 * 1_000_000_000u64; + let session_id = SessionId::from_raw(1); + let state = Arc::new(SessionState { + id: session_id, + user_id: 7, + agent_id: AgentId::new("restored").unwrap(), + policy_name: "short".to_string(), + started_at: Instant::now(), + started_at_ns: now_ns.saturating_sub(two_hours_ns), + metadata: std::collections::HashMap::new(), + signals: dashmap::DashMap::new(), + signaled_entities: dashmap::DashMap::new(), + annotations: std::sync::Mutex::new(Vec::new()), + signals_written: AtomicU64::new(0), + signals_rejected: AtomicU64::new(0), + audit_log: std::sync::Mutex::new(AuditLog::new()), + closed: Arc::new(AtomicBool::new(false)), + }); + db.sessions.insert(session_id, state); + assert_eq!(db.active_sessions().len(), 1); + + db.force_sweep(); + + assert_eq!( + db.active_sessions().len(), + 0, + "a restored session 2h past a 1h cap must be reaped on the startup sweep" + ); + } } diff --git a/tidal/src/db/text_syncer.rs b/tidal/src/db/text_syncer.rs index bf01c4a..b8fc899 100644 --- a/tidal/src/db/text_syncer.rs +++ b/tidal/src/db/text_syncer.rs @@ -309,6 +309,53 @@ mod tests { } } + /// CRITICAL regression (pass2): the item/creator write path must enqueue + /// via the producer pattern "clone the Sender out of the guard, DROP the + /// guard, then `try_send`" — never a blocking `send` under a held mutex. + /// This reproduces that exact sequence against a FULL bounded outbox (no + /// syncer spawned, so nothing drains it) and asserts the enqueue returns + /// immediately with `Full` instead of blocking. A blocking `send` here would + /// hang forever — and under the old code, with the mutex still held, it would + /// stall every concurrent write. The test running to completion is the proof + /// that the producer never blocks. + #[test] + fn producer_does_not_block_when_outbox_full() { + let config = Config::default(); + let pending = open_text_syncer(&one_text_field(), &config, "items", "test-syncer"); + + // Model the field exactly as `write_item_with_metadata` holds it. + let tx_mutex = std::sync::Mutex::new(pending.write_tx.clone()); + + // Saturate the outbox to capacity — nothing drains it (no spawn()). + let fill_tx = pending + .write_tx + .as_ref() + .expect("text field declared") + .clone(); + for id in 0..WRITE_OUTBOX_CAPACITY as u64 { + fill_tx + .try_send(pending_write(id)) + .expect("bounded channel accepts up to capacity"); + } + + // The producer pattern: clone out of the guard, DROP the guard, try_send. + // If this used a blocking `send`, the test would hang here forever. + let tx = tx_mutex.lock().ok().and_then(|g| g.as_ref().cloned()); + let tx = tx.expect("sender present"); + match tx.try_send(pending_write(WRITE_OUTBOX_CAPACITY as u64)) { + Err(crossbeam::channel::TrySendError::Full(_)) => {} // non-blocking drop + Ok(()) => panic!("outbox accepted a send past capacity — not bounded"), + Err(other) => panic!("unexpected send error: {other:?}"), + } + + // The mutex is immediately re-lockable: the producer never held it across + // the send, so a concurrent writer is not stalled. + assert!( + tx_mutex.try_lock().is_ok(), + "mutex must be free — the producer must not hold it across the send" + ); + } + /// An empty text-field set yields a pending handle with no index and no /// channels, so there is nothing to bound. #[test] diff --git a/tidal/src/db/users.rs b/tidal/src/db/users.rs index 57e3a6b..2fb164c 100644 --- a/tidal/src/db/users.rs +++ b/tidal/src/db/users.rs @@ -14,8 +14,8 @@ use crate::{ storage::{ StorageEngine, Tag, encode_key, vector::{ - BruteForceIndex, QuantizationLevel, VectorIndexConfig, insert_embedding, - registry::{EmbeddingSlotState, EmbeddingSource, HnswParams}, + QuantizationLevel, insert_embedding, + registry::{EmbeddingSlotState, EmbeddingSource, HnswParams, build_slot_index}, }, }, }; @@ -136,12 +136,16 @@ impl TidalDb { .write() .map_err(|_| TidalError::internal(op, "embedding_registry write lock poisoned"))?; if registry.get(kind, slot_name).is_none() { - let config = VectorIndexConfig { - dimensions: embedding.len(), - ..VectorIndexConfig::default() - }; + // Route backend selection through the size-gated factory rather + // than hardcoding brute force. A freshly auto-registered slot has + // no vectors yet, so the factory builds the exact `BruteForceIndex` + // (correct for a tiny / cold slot — no graph-build cost, exact + // results). When the slot's durable vector count crosses + // `USEARCH_MIN_VECTORS`, the next process restart's + // `rebuild_from_store` promotes it to the production HNSW + // (`UsearchIndex`) automatically — no config plumbing required. let state = EmbeddingSlotState { - index: Box::new(BruteForceIndex::new(config)), + index: build_slot_index(embedding.len(), 0), dimensions: embedding.len(), quantization: QuantizationLevel::F32, source: EmbeddingSource::External, diff --git a/tidal/src/entities/preference.rs b/tidal/src/entities/preference.rs index d896b80..d4a6053 100644 --- a/tidal/src/entities/preference.rs +++ b/tidal/src/entities/preference.rs @@ -300,6 +300,11 @@ impl PreferenceVectors { /// `Tag::Preference` checkpoint. Skips rows whose stored dimension does not /// match this store's `dim` (a schema change) and rows that are torn. /// + /// Each restored vector is re-normalized to unit length at the load boundary + /// (after zeroing any NaN component) so the cosine-scoring invariant holds + /// even for a torn or tampered row — see the inline comment at the + /// normalization site. + /// /// # Errors /// /// Returns storage errors from the underlying engine. @@ -333,9 +338,18 @@ impl PreferenceVectors { // cosine scoring downstream. vec.push(if f.is_nan() { 0.0 } else { f }); } - // The stored vector is already L2-normalized; insert directly (set() - // re-normalizes, which is a no-op for a unit vector) and seed the - // adaptive update count so the learning rate resumes where it was. + // Re-establish the unit-length invariant at the load boundary instead + // of trusting the stored bytes. `cosine_similarity` divides only by the + // candidate norm — it assumes the stored preference is already unit + // length — so a torn row (a zeroed NaN component, or finite-but-tampered + // values) that is no longer unit length would mis-scale every similarity + // for this user. Re-normalizing here makes that assumption true and + // matches `set()`'s contract. `l2_normalize` leaves an all-zero vector + // untouched, so a fully-corrupt row degrades to a zero preference + // (cosine returns 0.0) rather than a poisoned non-unit one. + l2_normalize(&mut vec); + // Seed the adaptive update count so the learning rate resumes where + // it was at checkpoint time. self.inner.insert(user_id, vec); self.update_counts.insert(user_id, update_count); let _ = EntityId::new(user_id); @@ -601,6 +615,85 @@ mod tests { ); } + /// Encode a `Tag::Preference` checkpoint value: `[count:8 LE][dim:4 LE][f32*dim]`. + fn encode_pref_value(count: u64, vec: &[f32]) -> Vec { + let mut value = Vec::with_capacity(8 + 4 + vec.len() * 4); + value.extend_from_slice(&count.to_le_bytes()); + value.extend_from_slice(&u32::try_from(vec.len()).unwrap().to_le_bytes()); + for v in vec { + value.extend_from_slice(&v.to_le_bytes()); + } + value + } + + /// W12: a torn/tampered checkpoint row that is NOT unit-length must be + /// re-normalized at the load boundary. `cosine_similarity` assumes the stored + /// preference is unit-length (it divides only by the candidate norm), so a + /// non-unit restored vector would mis-scale every similarity for that user. + /// This covers both failure shapes: a NaN component (zeroed then non-unit) and + /// a finite-but-non-unit row. + #[test] + fn restore_renormalizes_torn_rows_to_unit_length() { + use crate::{ + schema::EntityId, + storage::{InMemoryBackend, StorageEngine, Tag, encode_key}, + }; + + let storage = InMemoryBackend::new(); + let dim = 3usize; + + // User 1: a NaN component. After zeroing, the residual [3,0,4] has norm 5, + // i.e. NOT unit length — restore must normalize it to [0.6, 0, 0.8]. + let nan_row = encode_pref_value(7, &[3.0, f32::NAN, 4.0]); + let key1 = encode_key(EntityId::new(0), Tag::Preference, &1u64.to_be_bytes()); + storage.put(&key1, &nan_row).unwrap(); + + // User 2: a finite-but-non-unit row [0, 6, 8] (norm 10). Must normalize to + // [0, 0.6, 0.8]. + let big_row = encode_pref_value(3, &[0.0, 6.0, 8.0]); + let key2 = encode_key(EntityId::new(0), Tag::Preference, &2u64.to_be_bytes()); + storage.put(&key2, &big_row).unwrap(); + + // User 3: an all-zero (fully-corrupt) row stays zero (degrades to a zero + // preference, never a poisoned non-unit one). + let zero_row = encode_pref_value(1, &[f32::NAN, f32::NAN, f32::NAN]); + let key3 = encode_key(EntityId::new(0), Tag::Preference, &3u64.to_be_bytes()); + storage.put(&key3, &zero_row).unwrap(); + + let pv = PreferenceVectors::new(dim); + pv.restore(&storage).unwrap(); + + let v1 = pv.get(1).expect("user 1 restored"); + let n1: f32 = v1.iter().map(|x| x * x).sum::().sqrt(); + assert!( + (n1 - 1.0).abs() < 1e-5, + "user 1 must be unit length, got {n1}" + ); + assert!((v1[0] - 0.6).abs() < 1e-5 && (v1[2] - 0.8).abs() < 1e-5); + assert_eq!(pv.update_count(1), 7, "update count must round-trip"); + + let v2 = pv.get(2).expect("user 2 restored"); + let n2: f32 = v2.iter().map(|x| x * x).sum::().sqrt(); + assert!( + (n2 - 1.0).abs() < 1e-5, + "user 2 must be unit length, got {n2}" + ); + + let v3 = pv.get(3).expect("user 3 restored"); + assert!( + v3.iter().all(|&x| x == 0.0), + "a fully-corrupt row degrades to a zero preference, not a poisoned one" + ); + + // Cosine of the unit-normalized user-1 pref with its own direction is 1.0, + // proving the invariant cosine_similarity depends on now holds. + let sim = pv.cosine_similarity(1, &[0.6, 0.0, 0.8]).unwrap(); + assert!( + (sim - 1.0).abs() < 1e-5, + "cosine self-similarity must be 1.0, got {sim}" + ); + } + mod proptests { use proptest::prelude::*; diff --git a/tidal/src/entities/relationship.rs b/tidal/src/entities/relationship.rs index 0938ba1..2d5242d 100644 --- a/tidal/src/entities/relationship.rs +++ b/tidal/src/entities/relationship.rs @@ -2,7 +2,10 @@ //! //! Implemented in Task 2 (p1-02). -use crate::schema::{EntityId, Timestamp}; +use crate::{ + schema::{EntityId, Timestamp}, + storage::{Tag, encode_key, entity_tag_prefix}, +}; /// The type of relationship between two entities. #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] @@ -65,33 +68,38 @@ pub struct RelationshipEdge { /// Encode a relationship key (19 bytes). /// -/// Format: `[from_entity_id: 8 BE][0x00][Tag::Rel: 0x04][rel_type: 1 byte][to_entity_id: 8 BE]` +/// Format: `[from_entity_id: 8 BE][0x00][Tag::Rel][rel_type: 1 byte][to_entity_id: 8 BE]` /// /// Total: 8 + 1 + 1 + 1 + 8 = 19 bytes. +/// +/// Built through [`encode_key`] / [`Tag::Rel`] rather than literal `0x00`/`0x04` +/// bytes so the relationship keyspace provably tracks the canonical storage key +/// layout — a change to the separator or the `Rel` discriminant propagates here +/// automatically instead of silently drifting. #[must_use] pub fn encode_relationship_key( from: EntityId, rel_type: RelationshipType, to: EntityId, ) -> Vec { - let mut key = Vec::with_capacity(19); - key.extend_from_slice(&from.to_be_bytes()); - key.push(0x00); // separator - key.push(0x04); // Tag::Rel - key.push(rel_type.as_byte()); - key.extend_from_slice(&to.to_be_bytes()); - key + // Suffix after `[from][0x00][Tag::Rel]` is `[rel_type][to_id: 8 BE]`. + let mut suffix = Vec::with_capacity(1 + 8); + suffix.push(rel_type.as_byte()); + suffix.extend_from_slice(&to.to_be_bytes()); + encode_key(from, Tag::Rel, &suffix) } /// Build a prefix for scanning all relationships of a given type from an entity. /// -/// Returns 11 bytes: `[from_id: 8 BE][0x00][0x04][rel_type]` +/// Returns 11 bytes: `[from_id: 8 BE][0x00][Tag::Rel][rel_type]`. +/// +/// Shares the canonical entity+tag prefix ([`entity_tag_prefix`] with +/// [`Tag::Rel`]) so the scan prefix can never disagree with the key the keys +/// are written under. #[must_use] pub fn relationship_prefix(from: EntityId, rel_type: RelationshipType) -> Vec { let mut prefix = Vec::with_capacity(11); - prefix.extend_from_slice(&from.to_be_bytes()); - prefix.push(0x00); - prefix.push(0x04); + prefix.extend_from_slice(&entity_tag_prefix(from, Tag::Rel)); prefix.push(rel_type.as_byte()); prefix } @@ -124,7 +132,8 @@ pub fn deserialize_relationship_value(bytes: &[u8]) -> Option<(f64, u64)> { /// Parse the `to` entity ID from a full relationship key (19 bytes). /// -/// The `to` entity starts at offset 11: 8 (from) + 1 (NUL) + 1 (`Tag::Rel`) + 1 (`rel_type`). +/// The `to` entity starts at offset 11: 8 (from) + 1 (NUL) + 1 (`Tag::Rel`) + 1 +/// (`rel_type`) — i.e. bytes 1..9 of the canonical [`Tag::Rel`] key suffix. #[must_use] pub fn parse_relationship_to(key: &[u8]) -> Option { if key.len() < 19 { @@ -193,6 +202,51 @@ mod tests { assert_eq!(prefix.len(), 11); } + /// Drift guard: the relationship key must equal the canonical storage layout + /// produced by `encode_key(from, Tag::Rel, [rel_type, to_id])`, and must parse + /// back through the shared `parse_key`. This catches any future change to the + /// separator byte or the `Rel` discriminant that the hardcoded layout would + /// have silently missed. + #[test] + fn relationship_key_matches_canonical_encode_key() { + use crate::storage::{Tag, encode_key, parse_key}; + + let from = EntityId::new(0x1122_3344_5566_7788); + let to = EntityId::new(0x99AA_BBCC_DDEE_FF00); + for rel_type in [ + RelationshipType::Follows, + RelationshipType::Blocks, + RelationshipType::InteractionWeight, + RelationshipType::Hide, + RelationshipType::Mute, + ] { + let key = encode_relationship_key(from, rel_type, to); + + // 1) Byte-identical to the canonical encoder for the same suffix. + let mut suffix = Vec::with_capacity(9); + suffix.push(rel_type.as_byte()); + suffix.extend_from_slice(&to.to_be_bytes()); + let canonical = encode_key(from, Tag::Rel, &suffix); + assert_eq!( + key, canonical, + "relationship key must match encode_key layout" + ); + + // 2) Parses back through the shared key parser as a Tag::Rel key. + let (parsed_from, tag, parsed_suffix) = + parse_key(&key).expect("relationship key must be a valid canonical key"); + assert_eq!(parsed_from, from); + assert_eq!(tag, Tag::Rel); + assert_eq!(parsed_suffix[0], rel_type.as_byte()); + assert_eq!(parse_relationship_to(&key), Some(to)); + + // 3) The scan prefix is exactly the key truncated before the to-id. + let prefix = relationship_prefix(from, rel_type); + assert!(key.starts_with(&prefix)); + assert_eq!(prefix.len(), 11); + } + } + mod proptests { use proptest::prelude::*; diff --git a/tidal/src/governance/community_ledger.rs b/tidal/src/governance/community_ledger.rs index edfbf4a..0b1aad3 100644 --- a/tidal/src/governance/community_ledger.rs +++ b/tidal/src/governance/community_ledger.rs @@ -215,14 +215,21 @@ impl CommunityLedger { if cell.contributors.is_empty() { return Ok(None); } - // Deterministic fold order: sort contributor keys. - let mut keys: Vec<&ContributorKey> = cell.contributors.keys().collect(); - keys.sort_unstable(); + // Deterministic fold order: sort the (key, entry) pairs by key. We sort + // entries we already hold rather than sorting keys and re-`get`ting each: + // the keys came from this same map under this read guard, so the second + // lookup is provably redundant (and its `if let Some` implies a + // fallibility that cannot occur). + let mut entries: Vec<(&ContributorKey, &EntitySignalEntry)> = + cell.contributors.iter().collect(); + // Sort by the contributor key. The key borrows from the element, so this + // uses `sort_unstable_by` (a borrowing `sort_by_key` closure cannot satisfy + // the `FnMut(&T) -> K` lifetime); clippy's `unnecessary_sort_by` does not + // fire here for that reason. `x.0` / `y.0` are `&ContributorKey`. + entries.sort_unstable_by(|x, y| x.0.cmp(y.0)); let mut sum = 0.0; - for k in keys { - if let Some(entry) = cell.contributors.get(k) { - sum += entry.hot.current_score(decay_rate_idx, now_ns, lambda); - } + for (_k, entry) in entries { + sum += entry.hot.current_score(decay_rate_idx, now_ns, lambda); } Ok(Some(sum)) } @@ -424,19 +431,24 @@ impl CommunityLedger { let Some(cell) = self.cells.get(&(community, entity_id, type_id)) else { return Ok(Vec::new()); }; - let mut keys: Vec<&ContributorKey> = cell.contributors.keys().collect(); - keys.sort_unstable(); - let mut out = Vec::with_capacity(keys.len()); - for k in keys { - if let Some(e) = cell.contributors.get(k) { - out.push(ContributionProvenance { - user: k.0, - writer_agent: k.1.clone(), - epoch: k.2, - score: e.hot.current_score(0, now_ns, lambda), - count: e.warm.windowed_count(Window::AllTime, now_ns), - }); - } + // Sort the (key, entry) pairs we already hold; no re-`get` of keys that + // came from this same map under this read guard. + let mut entries: Vec<(&ContributorKey, &EntitySignalEntry)> = + cell.contributors.iter().collect(); + // Sort by the contributor key. The key borrows from the element, so this + // uses `sort_unstable_by` (a borrowing `sort_by_key` closure cannot satisfy + // the `FnMut(&T) -> K` lifetime); clippy's `unnecessary_sort_by` does not + // fire here for that reason. `x.0` / `y.0` are `&ContributorKey`. + entries.sort_unstable_by(|x, y| x.0.cmp(y.0)); + let mut out = Vec::with_capacity(entries.len()); + for (k, e) in entries { + out.push(ContributionProvenance { + user: k.0, + writer_agent: k.1.clone(), + epoch: k.2, + score: e.hot.current_score(0, now_ns, lambda), + count: e.warm.windowed_count(Window::AllTime, now_ns), + }); } Ok(out) } @@ -449,16 +461,27 @@ impl CommunityLedger { /// every row, then re-put every in-memory cell). Calling it on every accepted /// contribution makes ingestion O(N²) — the M-th write does O(M) work — and /// blows the signal-write latency budget for any active community. This writes - /// only the one row the accept just touched, so the per-accept durability cost - /// is constant. The zero-loss-window guarantee is preserved: the accepted cell - /// is durably written (one atomic single-row `WriteBatch`, matching the prior - /// per-write behaviour) before the caller returns `Ok(true)`. The full - /// scan-delete-rewrite snapshot stays in `checkpoint` for the - /// lifecycle/periodic/backup path, where it garbage-collects purged rows. + /// only the one row the accept just touched, so the per-accept storage cost is + /// constant. The full scan-delete-rewrite snapshot stays in `checkpoint` for + /// the lifecycle/periodic/backup path, where it garbage-collects purged rows. + /// + /// # Durability + /// + /// This ledger is the SOLE durable copy of per-contributor identity — the WAL + /// community event carries none, so WAL replay cannot rebuild it. A bare + /// `write_batch` is committed-not-durable (the bytes may still sit in a + /// memtable / page cache, per [`StorageEngine::write_batch`]), so this issues + /// the `fsync` barrier via [`StorageEngine::flush`] before returning `Ok(())`. + /// That makes the accepted cell genuinely durable when the caller returns — + /// honoring the `signal_for_community` "ZERO loss window" guarantee against a + /// power-loss crash, not merely against a clean shutdown. The flush is on the + /// gated, community-scoped contribution path (not the hot global signal path), + /// so the per-accept `fsync` is an acceptable cost for the only state in the + /// zone that cannot be recovered from the WAL. /// /// # Errors /// - /// Returns `TidalError::Storage` on a write failure. + /// Returns `TidalError::Storage` on a write or flush failure. #[allow(clippy::too_many_arguments)] pub fn persist_cell( &self, @@ -492,6 +515,11 @@ impl CommunityLedger { let mut batch = WriteBatch::new(); batch.put(key, value); storage.write_batch(batch).map_err(TidalError::from)?; + // Commit is not durability: fsync the keyspace so the contributor row + // survives a power crash. This ledger has no WAL backstop, so without the + // barrier the accepted cell could be lost while the WAL still reports the + // aggregate event — exactly the gap the "ZERO loss window" claim forbids. + storage.flush().map_err(TidalError::from)?; Ok(()) } @@ -655,15 +683,20 @@ impl CommunityLedger { .and_then(|v| v.first()) .copied() .unwrap_or(0.0); - let mut keys: Vec<&ContributorKey> = cell.value().contributors.keys().collect(); - keys.sort_unstable(); + // Sort the (key, entry) pairs we already hold; the keys came from this + // same map under this read guard, so re-`get`ting each is redundant. + let mut entries: Vec<(&ContributorKey, &EntitySignalEntry)> = + cell.value().contributors.iter().collect(); + // Sort by the contributor key. The key borrows from the element, so this + // uses `sort_unstable_by` (a borrowing `sort_by_key` closure cannot satisfy + // the `FnMut(&T) -> K` lifetime); clippy's `unnecessary_sort_by` does not + // fire here for that reason. `x.0` / `y.0` are `&ContributorKey`. + entries.sort_unstable_by(|x, y| x.0.cmp(y.0)); let mut score = 0.0; let mut count = 0u64; - for k in keys { - if let Some(e) = cell.value().contributors.get(k) { - score += e.hot.current_score(0, now_ns, lambda); - count += e.warm.windowed_count(Window::AllTime, now_ns); - } + for (_k, e) in entries { + score += e.hot.current_score(0, now_ns, lambda); + count += e.warm.windowed_count(Window::AllTime, now_ns); } rows.push((entity.as_u64(), type_id.as_u16(), score.to_bits(), count)); } @@ -1278,6 +1311,84 @@ mod tests { ); } + /// Storage spy that delegates to an [`InMemoryBackend`] but counts how many + /// times [`StorageEngine::flush`] is invoked, so a test can assert the + /// per-accept path actually issues the durability barrier. + struct FlushCountingBackend { + inner: crate::storage::InMemoryBackend, + flushes: std::sync::atomic::AtomicUsize, + } + + impl FlushCountingBackend { + fn new() -> Self { + Self { + inner: crate::storage::InMemoryBackend::new(), + flushes: std::sync::atomic::AtomicUsize::new(0), + } + } + fn flush_count(&self) -> usize { + self.flushes.load(std::sync::atomic::Ordering::SeqCst) + } + } + + impl StorageEngine for FlushCountingBackend { + fn get(&self, key: &[u8]) -> Result>, crate::storage::StorageError> { + self.inner.get(key) + } + fn put(&self, key: &[u8], value: &[u8]) -> Result<(), crate::storage::StorageError> { + self.inner.put(key, value) + } + fn delete(&self, key: &[u8]) -> Result<(), crate::storage::StorageError> { + self.inner.delete(key) + } + fn scan_prefix(&self, prefix: &[u8]) -> crate::storage::iterator::PrefixIterator<'_> { + self.inner.scan_prefix(prefix) + } + fn write_batch(&self, batch: WriteBatch) -> Result<(), crate::storage::StorageError> { + self.inner.write_batch(batch) + } + fn flush(&self) -> Result<(), crate::storage::StorageError> { + self.flushes + .fetch_add(1, std::sync::atomic::Ordering::SeqCst); + self.inner.flush() + } + } + + #[test] + fn persist_cell_fsyncs_so_the_zero_loss_claim_holds() { + // CRITICAL regression: the community ledger is the SOLE non-WAL-rebuildable + // durable copy of per-contributor identity, and `signal_for_community` + // promises Ok(true) means the contribution is durable. A bare write_batch + // is committed-not-durable, so persist_cell MUST pair it with flush() + // (fsync) before returning. This pins that the barrier is issued. + let storage = FlushCountingBackend::new(); + let l = ledger(); + let c = CommunityId(1); + let like = l.resolve_signal_type("like").unwrap(); + let ts = Timestamp::now().as_nanos(); + + l.record(c, EntityId::new(10), like, UserId(1), "agent", 0, 1.0, ts); + l.persist_cell(&storage, c, EntityId::new(10), like, UserId(1), "agent", 0) + .unwrap(); + + assert_eq!( + storage.flush_count(), + 1, + "persist_cell must fsync (flush) the contributor row, not leave it \ + committed-not-durable — the WAL cannot rebuild this state" + ); + + // And the row genuinely round-trips through restore. + let restored = ledger(); + restored.restore(&storage.inner); + assert_eq!( + restored + .windowed_count(c, EntityId::new(10), "like", Window::AllTime) + .unwrap(), + 1 + ); + } + #[test] fn apply_purge_tombstone_purges_and_sets_watermark() { use crate::governance::tombstone::PurgeTombstone; diff --git a/tidal/src/lib.rs b/tidal/src/lib.rs index e32b953..2e516c6 100644 --- a/tidal/src/lib.rs +++ b/tidal/src/lib.rs @@ -27,9 +27,10 @@ pub mod storage; pub mod text; pub mod wal; -/// Build hash compiled in from the `GIT_HASH` environment variable. +/// Build hash compiled in from the `TIDALDB_BUILD_HASH` environment variable. /// -/// Falls back to `"dev"` if `GIT_HASH` is unset or `build.rs` is not invoked. +/// Falls back to `"dev"` if `TIDALDB_BUILD_HASH` is unset or `build.rs` is not +/// invoked. /// /// # Examples /// diff --git a/tidal/src/query/executor/helpers.rs b/tidal/src/query/executor/helpers.rs index 0c893df..273a089 100644 --- a/tidal/src/query/executor/helpers.rs +++ b/tidal/src/query/executor/helpers.rs @@ -44,7 +44,10 @@ fn degrade_unknown_signal( /// Mirrors the ranking executor's own post-scoring normalization so the /// `RetrieveResult.score` contract holds after a cohort rescore: /// - Non-finite scores (`NaN`/`±Inf`) are clamped to `0.0` first. -/// - If every score is equal (`range < EPSILON`), all map to `1.0`. +/// - If every score is equal (`range < EPSILON`), all map to `0.5` (spec §8.2: +/// a degenerate all-equal set is "neutral / no signal", not maximal +/// confidence) — kept identical to the ranking executor's `normalize` so the +/// two paths cannot drift. /// - Otherwise `(score - min) / (max - min)`. fn normalize_scores(candidates: &mut [ScoredCandidate]) { if candidates.is_empty() { @@ -66,7 +69,7 @@ fn normalize_scores(candidates: &mut [ScoredCandidate]) { let range = max - min; for c in candidates.iter_mut() { c.score = if range < f64::EPSILON { - 1.0 + 0.5 } else { (c.score - min) / range }; @@ -163,6 +166,29 @@ impl RetrieveExecutor<'_> { /// Re-score candidates using cohort signal values and re-sort. /// + /// # Sort-mode preservation (C-CRITICAL) + /// + /// The main scoring path bakes the profile's [`Sort`] into each + /// `ScoredCandidate.score` (e.g. `New` scores by entity id, `AlphabeticalAsc` + /// by title). A naive cohort rescore that OVERWRITES every score with a sum + /// derived only from `profile.boosts` would destroy that ordering: for a + /// sort-driven profile with empty boosts (e.g. the built-in `new`), every + /// cohort score collapses to `0.0`, normalization maps them all to `1.0`, and + /// the descending re-sort is a no-op — silently returning uniform scores and + /// the wrong order. To prevent that, the cohort override is applied ONLY when + /// it can change ranking without discarding a sort the main path already + /// honored: + /// + /// - **No boosts** → the cohort scope cannot change a boost-free profile's + /// ranking, so the rescore is a no-op (main-path scores and order preserved). + /// - **A metadata / id-based sort is active** (`New`, `Alphabetical*`, + /// `Shortest`, `Longest`, `DateSaved`, `Shuffle`) → that ordering reads entity + /// metadata / id, NOT the cohort ledger, so it must survive; the rescore is a + /// no-op and the main-path order is kept. + /// - **Otherwise** (a signal-driven sort or no sort, with boosts present) → + /// re-derive the score from the cohort ledger's signal values so the cohort's + /// own engagement drives ranking, then re-normalize and re-sort. + /// /// # Errors /// /// Returns [`QueryError::InvalidFilter`] when a boost references an @@ -184,12 +210,45 @@ impl RetrieveExecutor<'_> { scored: &mut [crate::ranking::executor::ScoredCandidate], cohort_name: &str, cohort_ledger: &crate::cohort::CohortSignalLedger, - boosts: &[crate::ranking::profile::Boost], + profile: &crate::ranking::profile::RankingProfile, ) -> Result<(), QueryError> { - use crate::ranking::profile::SignalAgg; + use crate::ranking::profile::{SignalAgg, Sort}; + + let boosts = &profile.boosts; + + // A boost-free profile's ranking is driven entirely by its sort mode or by + // the main-path base score; the cohort scope changes neither. Preserve the + // main-path scores and order rather than collapsing every candidate to a + // uniform cohort-boost sum of 0.0 (the exact `new`-profile bug C flagged). + if boosts.is_empty() { + return Ok(()); + } + + // A metadata / id-based sort reads entity metadata or id, never the cohort + // ledger. Overwriting the score from cohort boosts would destroy that + // ordering, so leave the main-path scores and order intact for these sorts. + if matches!( + profile.sort, + Some( + Sort::New + | Sort::AlphabeticalAsc + | Sort::AlphabeticalDesc + | Sort::Shortest + | Sort::Longest + | Sort::DateSaved + | Sort::Shuffle + ) + ) { + return Ok(()); + } for candidate in scored.iter_mut() { let mut cohort_score = 0.0; + // Rebuild the signal snapshot from the cohort-scoped values so the + // reported `RetrieveResult.signals` reflect the cohort scope the score + // was derived from, not the stale global-ledger snapshot the main path + // left on the candidate (C-CRITICAL: snapshot consistency). + let mut cohort_snapshot: Vec<(String, f64)> = Vec::with_capacity(boosts.len()); for boost in boosts { let value = match &boost.agg { SignalAgg::Value => { @@ -245,8 +304,10 @@ impl RetrieveExecutor<'_> { } }; cohort_score += value * boost.weight; + cohort_snapshot.push((boost.signal.clone(), value)); } candidate.score = cohort_score; + candidate.signal_snapshot = cohort_snapshot; } // Re-normalize: cohort rescore replaced each candidate's normalized @@ -256,86 +317,84 @@ impl RetrieveExecutor<'_> { // ranking executor applies after its own scoring pass). normalize_scores(scored); - // Re-sort by descending score. + // Re-sort by descending score, breaking ties on ASCENDING entity id so the + // order is a true total order (W14) — without it, candidates that tie on + // the rescored cohort score keep the input order, which can differ across + // runs once the upstream candidate order does. scored.sort_by(|a, b| { b.score .partial_cmp(&a.score) .unwrap_or(std::cmp::Ordering::Equal) + .then_with(|| a.entity_id.as_u64().cmp(&b.entity_id.as_u64())) }); Ok(()) } - /// Apply notification caps as a post-diversity pass. + /// Days since the Unix epoch for the current wall clock. + /// + /// Shared by the cap *filter* (pre-pagination) and the cap *record* + /// (post-pagination) so both bucket deliveries into the same calendar day. + /// `u32` wraps at ~4.3 billion days (~11.8 million years); safe for any + /// practical use. The `86_400s * 1e9ns` constant avoids a `chrono` dep. + fn today_days_since_epoch() -> u32 { + const NANOS_PER_DAY: u64 = 86_400 * 1_000_000_000; + let now_ns = crate::schema::Timestamp::now().as_nanos(); + (now_ns / NANOS_PER_DAY) as u32 + } + + /// Resolve a candidate's `creator_id`: prefer the pre-populated field, fall + /// back to reading item metadata from storage. Returns `None` for items with + /// no resolvable creator (which skip the per-creator cap by design). + fn resolve_candidate_creator_id( + &self, + c: &crate::ranking::executor::ScoredCandidate, + ) -> Option { + if let Some(cid) = c.creator_id { + return Some(cid.as_u64()); + } + let storage = self.items_storage?; + let key = encode_key(c.entity_id, Tag::Meta, b""); + let raw = storage.get(&key).ok()??; + let meta = deserialize_item_metadata(&raw); + meta.get("creator_id")?.parse::().ok() + } + + /// Apply notification caps as a post-diversity, pre-pagination FILTER. /// /// Enforces per-creator and total daily limits by consulting the - /// `NotificationTracker` for historical deliveries and capping within - /// this response. Retained results are recorded into the tracker when - /// the profile is "notification". + /// `NotificationTracker` for historical deliveries and capping within this + /// response against a read-only snapshot of prior deliveries. This pass NEVER + /// records: recording is deferred to [`Self::record_notification_deliveries`], + /// applied to the *paginated page* in Stage 5. + /// + /// # Why filtering and recording are split (C-CRITICAL) + /// + /// Pagination (offset/limit slicing) happens in Stage 5, AFTER this filter. If + /// this pass also recorded (the previous behavior), a `limit=5` query over 50 + /// survivors recorded 50 deliveries while returning 5 — the daily cap was + /// consumed by 45 items the caller never received, silently exhausting the + /// budget far below the real delivery count and suppressing legitimate + /// notifications. Recording only the delivered page fixes that: a delivery is + /// counted iff it is actually returned to the caller. /// /// Creator resolution: `ScoredCandidate.creator_id` is populated by the - /// ranking executor only when diversity is evaluated. As a fallback, - /// we resolve the `creator_id` from item metadata via `items_storage` so - /// that per-creator caps work correctly even when the ranking path leaves - /// `creator_id = None`. + /// ranking executor only when diversity is evaluated. As a fallback, we resolve + /// it from item metadata via `items_storage`. pub(super) fn apply_notification_caps( &self, candidates: Vec, caps: crate::query::retrieve::NotificationCaps, user_id: u64, - profile_name: &str, + _profile_name: &str, ) -> Vec { use std::collections::HashMap as StdMap; - // Days since epoch (avoids chrono dependency). - // u32 wraps at ~4.3 billion days (~11.8 million years); safe for any - // practical use. The 86_400s * 1e9ns constant avoids a chrono dep. - const NANOS_PER_DAY: u64 = 86_400 * 1_000_000_000; - let today = { - let now_ns = crate::schema::Timestamp::now().as_nanos(); - (now_ns / NANOS_PER_DAY) as u32 - }; + let today = Self::today_days_since_epoch(); - // Resolve creator_id for a candidate: prefer the pre-populated field, - // fall back to reading item metadata from storage. - let resolve_creator_id = |c: &crate::ranking::executor::ScoredCandidate| -> Option { - if let Some(cid) = c.creator_id { - return Some(cid.as_u64()); - } - let storage = self.items_storage?; - let key = encode_key(c.entity_id, Tag::Meta, b""); - let raw = storage.get(&key).ok()??; - let meta = deserialize_item_metadata(&raw); - meta.get("creator_id")?.parse::().ok() - }; - - // The "notification" profile is the only one that *records* deliveries - // into the shared tracker, so it is the only path with a check-then-record - // TOCTOU under concurrency. Route it through the tracker's atomic - // `check_and_record`, which evaluates both caps and increments under a - // single lock — two concurrent responses for the same user can no longer - // both observe headroom and both record, overshooting the caps. - if profile_name == "notification" - && let Some(tracker) = self.notification_tracker - { - return candidates - .into_iter() - .filter(|c| { - tracker.check_and_record( - user_id, - resolve_creator_id(c), - today, - caps.max_per_creator_per_day, - caps.max_total_per_day, - ) - }) - .collect(); - } - - // Non-recording path (non-notification profiles, or no tracker wired): - // trim this response against a read-only snapshot of prior deliveries. - // Nothing is persisted, so there is no shared state to race on — the - // per-response accumulators are local to this call. + // Read-only snapshot of prior deliveries. Nothing is persisted here, so + // there is no shared state to race on — the per-response accumulators are + // local to this call. The actual record happens on the delivered page. let already_total = self .notification_tracker .map_or(0, |t| t.count_total_today(user_id, today)); @@ -354,7 +413,7 @@ impl RetrieveExecutor<'_> { // Per-creator cap: only enforced when the item has a resolvable creator. // Items without any creator_id (neither in the candidate nor in metadata) // skip the per-creator check to avoid incorrect bucketing. - if let Some(creator_id) = resolve_creator_id(c) { + if let Some(creator_id) = self.resolve_candidate_creator_id(c) { let already_delivered = self .notification_tracker .map_or(0, |t| t.count_today(user_id, creator_id, today)); @@ -372,6 +431,49 @@ impl RetrieveExecutor<'_> { }) .collect() } + + /// Record notification deliveries for the items ACTUALLY returned to the + /// caller (the post-pagination page), and drop any page item the tracker + /// rejects under the now-live cap. + /// + /// Only the `notification` profile records into the shared tracker, because + /// it is the surface whose daily cap drives whether a user is notified. The + /// record goes through the tracker's atomic `check_and_record`, which + /// evaluates both caps and increments under a single lock — so two concurrent + /// responses for the same user cannot both observe headroom and both record, + /// overshooting the caps (TOCTOU). Because this runs on the delivered page + /// (not the full pre-slice set), the daily budget is consumed by exactly the + /// notifications the caller receives — never the items trimmed away by + /// pagination (C-CRITICAL). + pub(super) fn record_notification_deliveries( + &self, + page: &[crate::ranking::executor::ScoredCandidate], + caps: crate::query::retrieve::NotificationCaps, + user_id: u64, + profile_name: &str, + ) -> Vec { + if profile_name != "notification" { + return page.to_vec(); + } + let Some(tracker) = self.notification_tracker else { + return page.to_vec(); + }; + + let today = Self::today_days_since_epoch(); + page.iter() + .filter(|c| { + tracker.check_and_record( + user_id, + self.resolve_candidate_creator_id(c), + c.entity_id.as_u64(), + today, + caps.max_per_creator_per_day, + caps.max_total_per_day, + ) + }) + .cloned() + .collect() + } } // ── Tests ──────────────────────────────────────────────────────────────────── @@ -390,12 +492,36 @@ mod tests { use super::*; use crate::{ cohort::CohortSignalLedger, - ranking::profile::{Boost, SignalAgg}, + ranking::profile::{ + Boost, CandidateStrategy, DiversitySpec, RankingProfile, SignalAgg, Sort, + }, schema::{DecaySpec, EntityKind, SchemaBuilder, Timestamp, Window}, signals::{NoopWalWriter, SignalLedger}, storage::indexes::{bitmap::BitmapIndex, range::RangeIndex}, }; + /// Build a minimal `RankingProfile` carrying the given `boosts` and `sort`, + /// so `rescore_with_cohort` (which now takes the whole profile to honor the + /// sort mode) can be exercised directly. + fn cohort_profile(boosts: Vec, sort: Option) -> RankingProfile { + RankingProfile { + name: "cohort_test".to_owned(), + version: 1, + candidate_strategy: CandidateStrategy::Scan { + sort_field: String::new(), + }, + boosts, + decay: None, + gates: vec![], + penalties: vec![], + excludes: vec![], + diversity: DiversitySpec::default(), + exploration: 0.0, + sort, + is_builtin: false, + } + } + /// Schema with a single known signal (`view`) so the cohort ledger resolves /// `view` but rejects any other name with `UnknownSignalType`. fn cohort_schema() -> crate::schema::Schema { @@ -475,18 +601,22 @@ mod tests { let mut scored = vec![candidate(1, 0.9), candidate(2, 0.1)]; // Boost references a signal NOT in the schema → unknown-signal degrade. - let boosts = vec![Boost { - signal: "not_a_signal".to_string(), - agg: SignalAgg::Value, - window: Window::AllTime, - weight: 1.0, - }]; + let profile = cohort_profile( + vec![Boost { + signal: "not_a_signal".to_string(), + agg: SignalAgg::Value, + window: Window::AllTime, + weight: 1.0, + }], + None, + ); // Must succeed (degrade to 0.0), not error. - exec.rescore_with_cohort(&mut scored, "tech", &cohort_ledger, &boosts) + exec.rescore_with_cohort(&mut scored, "tech", &cohort_ledger, &profile) .expect("unknown cohort signal must degrade to 0.0, not error"); - // Every cohort score collapsed to 0.0 → normalize maps all to 1.0. - assert!(scored.iter().all(|c| (c.score - 1.0).abs() < f64::EPSILON)); + // Every cohort score collapsed to 0.0 → an all-equal set normalizes to + // 0.5 (spec §8.2: neutral / no signal), matching the ranking executor. + assert!(scored.iter().all(|c| (c.score - 0.5).abs() < f64::EPSILON)); } /// W26 integration: a known-signal cohort boost still scores normally, @@ -524,16 +654,197 @@ mod tests { } let mut scored = vec![candidate(1, 0.9), candidate(2, 0.1)]; - let boosts = vec![Boost { - signal: "view".to_string(), - agg: SignalAgg::Value, - window: Window::AllTime, - weight: 1.0, - }]; + let profile = cohort_profile( + vec![Boost { + signal: "view".to_string(), + agg: SignalAgg::Value, + window: Window::AllTime, + weight: 1.0, + }], + None, + ); - exec.rescore_with_cohort(&mut scored, "tech", &cohort_ledger, &boosts) + exec.rescore_with_cohort(&mut scored, "tech", &cohort_ledger, &profile) .expect("known signal must score without error"); // The cohort rescore + re-sort puts the higher cohort-signal item first. assert_eq!(scored[0].entity_id, EntityId::new(2)); + // The reported snapshot reflects the cohort scope (5 views), not the stale + // global snapshot the candidate carried in. + let top = scored + .iter() + .find(|c| c.entity_id == EntityId::new(2)) + .expect("item 2 present"); + assert!( + top.signal_snapshot + .iter() + .any(|(name, value)| name == "view" && (*value - 5.0).abs() < f64::EPSILON), + "cohort snapshot must report the cohort-scoped view count, got {:?}", + top.signal_snapshot + ); + } + + /// C-CRITICAL regression: a cohort + a metadata/id-based sort profile (`new`, + /// which scores by descending entity id with NO boosts) must PRESERVE the + /// main-path ordering and scores. Before the fix the rescore overwrote every + /// score with a cohort-boost sum of 0.0, normalize mapped them all to 1.0, and + /// the descending re-sort was a no-op — destroying the New ordering and + /// returning uniform scores. With the fix the boost-free profile is a no-op. + #[test] + fn rescore_with_cohort_preserves_new_sort_ordering() { + let global = SignalLedger::new(cohort_schema(), Box::new(NoopWalWriter)); + let profile_reg = setup_registry(); + let cat = BitmapIndex::new("category"); + let fmt = BitmapIndex::new("format"); + let cre = BitmapIndex::new("creator"); + let tag = BitmapIndex::new("tag"); + let dur = RangeIndex::::new("duration"); + let ts = RangeIndex::::new("created_at"); + let universe = RwLock::new(RoaringBitmap::new()); + let exec = make_executor( + &global, + &profile_reg, + &cat, + &fmt, + &cre, + &tag, + &dur, + &ts, + &universe, + ); + + let cohort_ledger = CohortSignalLedger::new(&cohort_schema()); + // The main path for `New` scores by entity id; item 3 (highest id) ranks + // first, then 2, then 1. The scores are distinct, not all-equal. + let mut scored = vec![candidate(1, 1.0), candidate(2, 2.0), candidate(3, 3.0)]; + // `new`-style profile: no boosts, Sort::New. + let profile = cohort_profile(vec![], Some(Sort::New)); + + exec.rescore_with_cohort(&mut scored, "tech", &cohort_ledger, &profile) + .expect("boost-free sort profile must not error"); + + // Ordering and scores are untouched (rescore is a no-op for this shape). + let ids: Vec = scored.iter().map(|c| c.entity_id.as_u64()).collect(); + assert_eq!( + ids, + vec![1, 2, 3], + "the main-path order must be preserved verbatim, not re-sorted to uniform 1.0" + ); + assert!( + !scored.iter().all(|c| (c.score - 1.0).abs() < f64::EPSILON), + "scores must NOT collapse to a uniform 1.0; got {:?}", + scored.iter().map(|c| c.score).collect::>() + ); + } + + /// C-CRITICAL regression (notification cap recording follows pagination): + /// the Stage 4.5 FILTER (`apply_notification_caps`) must record NOTHING, and + /// recording must happen on the delivered PAGE via + /// `record_notification_deliveries` — so a `limit < survivors` query records + /// exactly `limit` deliveries, not every survivor. Before the fix the full + /// post-diversity set was recorded, silently exhausting the daily cap with + /// items the caller never received. + #[test] + fn notification_caps_record_only_the_delivered_page() { + use std::sync::Arc; + + use crate::{ + db::notification_tracker::NotificationTracker, query::retrieve::NotificationCaps, + }; + + let global = SignalLedger::new(cohort_schema(), Box::new(NoopWalWriter)); + let profile_reg = setup_registry(); + let cat = BitmapIndex::new("category"); + let fmt = BitmapIndex::new("format"); + let cre = BitmapIndex::new("creator"); + let tag = BitmapIndex::new("tag"); + let dur = RangeIndex::::new("duration"); + let ts = RangeIndex::::new("created_at"); + let universe = RwLock::new(RoaringBitmap::new()); + let tracker = Arc::new(NotificationTracker::new()); + let exec = make_executor( + &global, + &profile_reg, + &cat, + &fmt, + &cre, + &tag, + &dur, + &ts, + &universe, + ) + .with_notification_tracker(&tracker); + + // Five survivors from distinct creators; generous caps so the FILTER keeps + // all five. + let mk = |id: u64, creator: u64| ScoredCandidate { + entity_id: EntityId::new(id), + score: 1.0, + signal_snapshot: vec![], + creator_id: Some(EntityId::new(creator)), + format: None, + }; + let candidates = vec![mk(1, 10), mk(2, 20), mk(3, 30), mk(4, 40), mk(5, 50)]; + let caps = NotificationCaps { + max_per_creator_per_day: 5, + max_total_per_day: 10, + }; + + // FILTER keeps all five and records NOTHING. + let filtered = exec.apply_notification_caps(candidates, caps, 42, "notification"); + assert_eq!( + filtered.len(), + 5, + "filter keeps all five under generous caps" + ); + + let nanos_per_day: u64 = 86_400 * 1_000_000_000; + let today = (Timestamp::now().as_nanos() / nanos_per_day) as u32; + assert_eq!( + tracker.count_total_today(42, today), + 0, + "the filter pass must NOT record any deliveries" + ); + + // A page of limit=2 is recorded — exactly two deliveries, not five. + let page = &filtered[..2]; + let delivered = exec.record_notification_deliveries(page, caps, 42, "notification"); + assert_eq!(delivered.len(), 2, "the delivered page is exactly the page"); + assert_eq!( + tracker.count_total_today(42, today), + 2, + "exactly `limit` (2) deliveries recorded — not all five survivors" + ); + + // Re-issuing the identical query (same page, same user, same day) must + // NOT double-count the already-delivered items — the cap counts distinct + // items notified about, not raw delivery attempts. + let redelivered = exec.record_notification_deliveries(page, caps, 42, "notification"); + assert_eq!(redelivered.len(), 2, "the page still re-appears in full"); + assert_eq!( + tracker.count_total_today(42, today), + 2, + "a second identical query must not double-count delivered items" + ); + + // A non-notification profile records nothing even when given a page. + let other = Arc::new(NotificationTracker::new()); + let other_exec = make_executor( + &global, + &profile_reg, + &cat, + &fmt, + &cre, + &tag, + &dur, + &ts, + &universe, + ) + .with_notification_tracker(&other); + let _ = other_exec.record_notification_deliveries(page, caps, 42, "trending"); + assert_eq!( + other.count_total_today(42, today), + 0, + "only the notification profile records deliveries" + ); } } diff --git a/tidal/src/query/executor/mod.rs b/tidal/src/query/executor/mod.rs index 3f39739..a07e021 100644 --- a/tidal/src/query/executor/mod.rs +++ b/tidal/src/query/executor/mod.rs @@ -371,6 +371,14 @@ impl<'a> RetrieveExecutor<'a> { executor = executor.with_date_saved_context(user_state, user_id); } + // Wire the Shuffle sort's per-user, per-minute seed (spec §11.6). Without + // a FOR USER it stays 0 (a stable per-minute global permutation). + if matches!(profile.sort, Some(Sort::Shuffle)) + && let Some(user_id) = query.for_user + { + executor = executor.with_shuffle_user(user_id); + } + #[allow(clippy::option_if_let_else)] let mut scored = if let Some(user_id) = query.for_user { // Personalized scoring: build UserContext from the interaction ledger @@ -463,7 +471,7 @@ impl<'a> RetrieveExecutor<'a> { }); } if cohort_exists { - self.rescore_with_cohort(&mut scored, cohort_name, cohort_ledger, &profile.boosts)?; + self.rescore_with_cohort(&mut scored, cohort_name, cohort_ledger, profile)?; } } @@ -478,15 +486,7 @@ impl<'a> RetrieveExecutor<'a> { // and Stage 4.5 notification cap enforcement (resolve_creator_id falls back // to items_storage, but having it pre-populated is more reliable). if needs_metadata_for_creator_grouping && !item_metadata.is_empty() { - for candidate in &mut scored { - if candidate.creator_id.is_none() - && let Some(meta) = item_metadata.get(&candidate.entity_id.as_u64()) - && let Some(cid_str) = meta.get("creator_id") - && let Ok(cid) = cid_str.parse::() - { - candidate.creator_id = Some(EntityId::new(cid)); - } - } + backfill_creator_ids(&mut scored, &item_metadata); } Ok((scored, item_metadata)) @@ -495,6 +495,32 @@ impl<'a> RetrieveExecutor<'a> { // ── Helpers ────────────────────────────────────────────────────────────────── +/// Backfill `creator_id` on each candidate from pre-loaded item metadata. +/// +/// The ranking executor always leaves `creator_id = None`; this enrichment reads +/// the `creator_id` metadata field and sets it so Stage 4 diversity enforcement +/// and Stage 4.5/Stage 5 notification capping can group by creator. Only +/// unresolved (`None`) candidates are touched, so it is safe to run more than +/// once. Shared by Stage 3 (`stage3_score`) and Stage 4.5 (`pipeline::execute`) +/// so the two passes cannot drift on the parse rule. +pub(super) fn backfill_creator_ids( + candidates: &mut [crate::ranking::executor::ScoredCandidate], + item_metadata: &std::collections::HashMap>, +) { + if item_metadata.is_empty() { + return; + } + for candidate in candidates.iter_mut() { + if candidate.creator_id.is_none() + && let Some(meta) = item_metadata.get(&candidate.entity_id.as_u64()) + && let Some(cid_str) = meta.get("creator_id") + && let Ok(cid) = cid_str.parse::() + { + candidate.creator_id = Some(EntityId::new(cid)); + } + } +} + /// Convert a 64-bit entity id to the 32-bit key space every suppression/ /// inclusion `RoaringBitmap` uses. /// diff --git a/tidal/src/query/executor/personalization.rs b/tidal/src/query/executor/personalization.rs index f4d6c58..eb23ceb 100644 --- a/tidal/src/query/executor/personalization.rs +++ b/tidal/src/query/executor/personalization.rs @@ -82,13 +82,19 @@ pub(crate) fn build_user_context( /// Returns an empty map when the user has no preference vector, no preference /// store is wired, or no items storage is available -- in which case the /// preference boost is a true no-op rather than a silent constant. +/// +/// The map is keyed by the **full `u64` entity id** (not a truncated `u32`): two +/// candidates whose ids differ only above bit 32 would otherwise alias onto one +/// slot and cross-contaminate each other's preference boost (query-executor +/// SUGGESTION, M0-M10 review pass 2). The consumer in +/// `ProfileExecutor::score_personalized` looks the boost up by the same full id. pub(crate) fn compute_preference_boosts( user_id: u64, candidates: &[crate::schema::EntityId], preference_vectors: Option<&crate::entities::PreferenceVectors>, items_storage: Option<&dyn crate::storage::StorageEngine>, slot_name: &str, -) -> HashMap { +) -> HashMap { let mut boosts = HashMap::new(); let (Some(prefs), Some(storage)) = (preference_vectors, items_storage) else { return boosts; @@ -109,7 +115,8 @@ pub(crate) fn compute_preference_boosts( // Dimension mismatch (e.g. preference vector sized for a different slot) // yields `None`; skip rather than contributing a bogus boost. if let Some(cosine) = prefs.cosine_similarity(user_id, &embedding) { - boosts.insert(entity_id.as_u64() as u32, f64::from(cosine)); + // Full u64 key: never truncate, so high-bit-distinct ids cannot alias. + boosts.insert(entity_id.as_u64(), f64::from(cosine)); } } boosts diff --git a/tidal/src/query/executor/pipeline.rs b/tidal/src/query/executor/pipeline.rs index da474ca..abb49da 100644 --- a/tidal/src/query/executor/pipeline.rs +++ b/tidal/src/query/executor/pipeline.rs @@ -289,9 +289,20 @@ impl RetrieveExecutor<'_> { let evaluator = FilterEvaluator::new(cat, fmt, cre, tag, dur, ts, universe_ref); match evaluator.evaluate(filter_expr) { FilterResult::Bitmap(bitmap) => { - // SAFETY: all entity IDs stored in RoaringBitmaps are 32-bit; - // truncation is intentional and correct throughout this module. - candidates.retain(|id| bitmap.contains(id.as_u64() as u32)); + // INCLUSION test against the match bitmap. The match bitmap + // holds only 32-bit ids, so a candidate id that overflows + // `u32` can never be a legitimate member and must be DROPPED + // — a raw `id as u32` truncation would alias `2^32 + k` onto + // the low slot `k`, which may be present in the bitmap, and + // wrongly RETAIN an item that does not satisfy the filter. + // `SignalRanked` candidates come straight from the ledger, + // which keys on full-u64 ids, so a `> u32::MAX` id is valid + // input that reaches here. Route through the checked + // `entity_as_u32` (drop-on-overflow), matching the + // InCollection/SocialGraph arms and the Stage 2.5 helper. + candidates.retain(|id| { + super::entity_as_u32(*id).is_some_and(|i| bitmap.contains(i)) + }); } FilterResult::Predicate(pred) => { candidates.retain(|id| pred(id.as_u64())); @@ -427,7 +438,10 @@ impl RetrieveExecutor<'_> { "stage 4: diversity enforced" ); - // ── Stage 4.5: Notification Cap Enforcement (M6p6) ────────────── + // ── Stage 4.5: Notification Cap FILTER (M6p6) ─────────────────── + // Trims the post-diversity set against the daily caps WITHOUT recording. + // Recording is deferred to Stage 5 so it counts only the delivered page, + // not items pagination drops (C-CRITICAL). let final_candidates = if let (Some(caps), Some(user_id)) = (query.notification_caps, query.for_user) { let mut final_candidates = final_candidates; @@ -437,19 +451,11 @@ impl RetrieveExecutor<'_> { // the field was `None`; backfilling again here is cheap (a HashMap // lookup) and closes the case where diversity selection or a later // pass left a survivor with an unresolved creator. Without this, the - // `resolve_creator_id` fallback in `apply_notification_caps` issues a - // storage `get()` for every still-unresolved candidate. - if !item_metadata.is_empty() { - for candidate in &mut final_candidates { - if candidate.creator_id.is_none() - && let Some(meta) = item_metadata.get(&candidate.entity_id.as_u64()) - && let Some(cid_str) = meta.get("creator_id") - && let Ok(cid) = cid_str.parse::() - { - candidate.creator_id = Some(EntityId::new(cid)); - } - } - } + // `resolve_candidate_creator_id` fallback issues a storage `get()` + // for every still-unresolved candidate. Shares the + // one `backfill_creator_ids` helper with Stage 3 so the two passes + // cannot drift on the parse rule. + super::backfill_creator_ids(&mut final_candidates, &item_metadata); self.apply_notification_caps(final_candidates, caps, user_id, &query.profile.name) } else { final_candidates @@ -457,7 +463,7 @@ impl RetrieveExecutor<'_> { tracing::trace!( final_count = final_candidates.len(), - "stage 4.5: notification caps applied" + "stage 4.5: notification cap filter applied (recording deferred to stage 5)" ); // ── Stage 5: Result Assembly ──────────────────────────────────── @@ -472,12 +478,28 @@ impl RetrieveExecutor<'_> { let end = offset .saturating_add(query.limit) .min(final_candidates.len()); - let page = if offset < final_candidates.len() { + let page_slice = if offset < final_candidates.len() { &final_candidates[offset..end] } else { &[] }; + // ── Notification delivery recording (post-pagination) ─────────────── + // Record deliveries against ONLY the page actually returned to the caller, + // not the full post-diversity set. Stage 4.5 filtered without recording; + // this records the delivered page through the tracker's atomic + // check-and-record so the daily cap is consumed by exactly the + // notifications the caller receives — never items trimmed by pagination + // (C-CRITICAL: notification-cap recording must follow pagination). For + // non-notification profiles or no tracker this returns the page unchanged. + let recorded_page: Vec = + if let (Some(caps), Some(user_id)) = (query.notification_caps, query.for_user) { + self.record_notification_deliveries(page_slice, caps, user_id, &query.profile.name) + } else { + page_slice.to_vec() + }; + let page: &[crate::ranking::executor::ScoredCandidate] = &recorded_page; + let items: Vec = page .iter() .enumerate() @@ -738,4 +760,71 @@ mod tests { inclusion-bitmap member) are both dropped" ); } + + /// C-CRITICAL regression (Stage 2 inclusion intersection): a `SignalRanked` + /// candidate whose id is `(1<<32) + k` must be DROPPED by an index-backed + /// `CategoryEq` filter even when slot `k` IS present in the match bitmap. A raw + /// `id as u32` truncation would alias the overflow id onto slot `k` and wrongly + /// RETAIN it; the checked `entity_as_u32` (drop-on-overflow) prevents the leak. + #[test] + fn stage2_index_filter_drops_overflow_alias_candidate() { + use roaring::RoaringBitmap; + + use crate::storage::indexes::bitmap::BitmapIndex; + + let schema = view_schema(); + let ledger = SignalLedger::new(schema, Box::new(NoopWalWriter)); + let profile_reg = signal_ranked_registry(); + + // Real low id `5` is in category "jazz"; the overflow id `(1<<32)+5` + // aliases onto slot 5 under truncation but must NOT be retained. + let k: u32 = 5; + let overflow_id = (1u64 << 32) + u64::from(k); + + let now = Timestamp::now(); + ledger + .record_signal("view", EntityId::new(u64::from(k)), 1.0, now) + .unwrap(); + ledger + .record_signal("view", EntityId::new(overflow_id), 1.0, now) + .unwrap(); + + // Category index: slot 5 ∈ "jazz". The overflow item is NOT indexed under + // jazz; only its truncated alias (slot 5) would falsely match. + let category = BitmapIndex::new("category"); + category.insert(k, "jazz"); + + // Universe must be Some so Stage 2 runs the index intersection. It only + // gates entry to the block; CategoryEq resolves to the category bitmap. + let mut universe_bm = RoaringBitmap::new(); + universe_bm.insert(k); + let universe = std::sync::RwLock::new(universe_bm); + + let exec = RetrieveExecutor::new( + &ledger, + &profile_reg, + Some(&category), // category + None, // format + None, // creator + None, // tag + None, // duration + None, // created_at + Some(&universe), // universe + ); + let query = Retrieve::builder() + .profile("view_ranked") + .filter(FilterExpr::CategoryEq("jazz".into())) + .limit(10) + .build() + .unwrap(); + + let results = exec.execute(&query).unwrap(); + let ids: Vec = results.items.iter().map(|r| r.entity_id.as_u64()).collect(); + assert_eq!( + ids, + vec![u64::from(k)], + "only the genuine low id 5 may survive CategoryEq(jazz); the overflow id \ + {overflow_id} must be dropped, not aliased onto slot {k} and retained" + ); + } } diff --git a/tidal/src/query/executor/post_filter.rs b/tidal/src/query/executor/post_filter.rs index 46c5619..a427050 100644 --- a/tidal/src/query/executor/post_filter.rs +++ b/tidal/src/query/executor/post_filter.rs @@ -160,6 +160,7 @@ fn apply_signal_thresholds( crate::schema::Window::AllTime, ledger, degradation_level, + crate::schema::Timestamp::now().as_nanos(), ) .map_err(|e| QueryError::InvalidFilter { field: signal.clone(), diff --git a/tidal/src/query/executor/tests_part2.rs b/tidal/src/query/executor/tests_part2.rs index 3680c5c..733a808 100644 --- a/tidal/src/query/executor/tests_part2.rs +++ b/tidal/src/query/executor/tests_part2.rs @@ -365,14 +365,15 @@ fn cohort_rescore_normalizes_score_to_unit_range_and_reorders() { }, ]; - let boosts = vec![Boost { + let mut profile = crate::ranking::test_fixtures::minimal_profile("cohort_rescore"); + profile.boosts = vec![Boost { signal: "view".into(), agg: SignalAgg::DecayScore, window: Window::AllTime, weight: 1.0, }]; - exec.rescore_with_cohort(&mut scored, "tech", &cohort_ledger, &boosts) + exec.rescore_with_cohort(&mut scored, "tech", &cohort_ledger, &profile) .unwrap(); // After rescore: item 2 (high cohort view) must rank first, and EVERY score diff --git a/tidal/src/query/fusion.rs b/tidal/src/query/fusion.rs index e5288de..754c8ea 100644 --- a/tidal/src/query/fusion.rs +++ b/tidal/src/query/fusion.rs @@ -93,7 +93,19 @@ impl HybridFusion { .into_iter() .map(|(id, score)| (EntityId::new(id), score)) .collect(); - results.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal)); + // Sort descending by fused score, breaking ties on ASCENDING entity id so + // the order is a true total order. The scores arrive from a `HashMap` + // whose iteration order is process-randomized (default `RandomState`), and + // `sort_by` is stable — without the id tie-break two entities with an equal + // fused score would keep whatever relative order the randomized map + // iteration produced, which changes between process runs (W14). RRF terms + // are always finite positive, so `partial_cmp` never sees `NaN` here; the + // only missing piece was the deterministic id tie-break. + results.sort_by(|a, b| { + b.1.partial_cmp(&a.1) + .unwrap_or(std::cmp::Ordering::Equal) + .then_with(|| a.0.as_u64().cmp(&b.0.as_u64())) + }); results } } @@ -336,6 +348,45 @@ mod tests { assert!((results[0].1 - expected).abs() < 1e-9); } + /// W14 regression: equal fused scores must be ordered by ASCENDING entity id, + /// not by randomized `HashMap` iteration order. Two items present at the same + /// rank in both lists earn an identical RRF score, so the tie-break is the only + /// thing that fixes their order. + #[test] + fn fuse_ties_break_on_ascending_entity_id() { + // Both items rank 1 in their (single-element) lists, so each accrues the + // SAME RRF term and the scores are exactly equal. Feed them in descending + // id order to prove the sort actively reorders to ascending id. + let bm25 = vec![(EntityId::new(9), 1.0f32)]; + let ann = vec![(EntityId::new(3), 0.1f32)]; + let fusion = HybridFusion::new(); + let results = fusion.fuse(&bm25, &ann); + let ids: Vec = results.iter().map(|(id, _)| id.as_u64()).collect(); + assert_eq!( + ids, + vec![3, 9], + "equal fused scores must order by ascending entity id" + ); + // The scores really are tied (otherwise the tie-break would be untested). + assert!((results[0].1 - results[1].1).abs() < 1e-12); + } + + /// W14 regression: fusing identical inputs twice must yield byte-identical + /// output, including the tie order. Before the fix the tie order depended on + /// the per-process `HashMap` seed and could differ across runs. + #[test] + fn fuse_is_deterministic_across_repeated_calls() { + let bm25: Vec<(EntityId, f32)> = (1..=20).map(|i| (EntityId::new(i), 1.0f32)).collect(); + let ann: Vec<(EntityId, f32)> = (1..=20).map(|i| (EntityId::new(i), 0.5f32)).collect(); + let fusion = HybridFusion::new(); + let first = fusion.fuse(&bm25, &ann); + let second = fusion.fuse(&bm25, &ann); + assert_eq!( + first, second, + "fusing identical inputs must produce identical, deterministic output" + ); + } + #[test] fn fuse_k_affects_scores() { let bm25 = vec![(EntityId::new(1), 1.0f32)]; diff --git a/tidal/src/query/search/executor/pipeline.rs b/tidal/src/query/search/executor/pipeline.rs index 3562228..2a46875 100644 --- a/tidal/src/query/search/executor/pipeline.rs +++ b/tidal/src/query/search/executor/pipeline.rs @@ -50,7 +50,7 @@ use crate::{ range::RangeIndex, }, }, - text::collectors::AllScoresCollector, + text::collectors::BoundedScoresCollector, }; /// Canonical default embedding-slot name. @@ -60,6 +60,17 @@ use crate::{ /// instead of being duplicated at each `unwrap_or("content")` call site. const DEFAULT_EMBEDDING_SLOT: &str = "content"; +/// Full-fidelity BM25 over-fetch multiplier: the bounded collector keeps at most +/// `limit * this` documents (floored at [`BM25_TOP_K_FLOOR`]). Mirrors +/// [`ANN_K_MULTIPLIER_FULL`] so the text and vector candidate pools stay +/// comparable before fusion. +const BM25_TOP_K_MULTIPLIER: usize = 20; + +/// Minimum BM25 top-K regardless of `limit`, so tiny-limit queries still seed a +/// usable text candidate pool. Matches the spec's `top_k_text` default +/// (06-text-retrieval.md Sec 11.2) and [`ANN_K_FLOOR`]. +const BM25_TOP_K_FLOOR: usize = 200; + /// Convert a 64-bit entity id to the 32-bit key space used by every /// `RoaringBitmap` in the suppression/inclusion path. /// @@ -178,7 +189,12 @@ impl SearchExecutor<'_> { ); // ── Stage 2 + 2b: Metadata Filters ────────────────────────────────── - self.apply_metadata_filter(combined_filter, &mut candidates, &mut warnings); + self.apply_metadata_filter( + query.entity_kind, + combined_filter, + &mut candidates, + &mut warnings, + ); self.apply_creator_metadata_filter(query, combined_filter, &mut candidates); // ── Stage 2.2/2.3/2.4: Deferred Post-Filters ──────────────────────── @@ -329,9 +345,19 @@ impl SearchExecutor<'_> { .parse(query_text) .map_err(|e| QueryError::StorageError(format!("text query parse: {e}")))?; let searcher = idx.searcher(); - let collector = AllScoresCollector { - entity_id_field: idx.fields().entity_id, - }; + // Bound the candidate set to the top-`bm25_cap` by score on the NORMAL + // path too, not just under degradation (W: BM25 must not return the whole + // match set). A broad/near-ubiquitous term over a large corpus would + // otherwise materialize one `(EntityId, f32)` per match plus a downstream + // `HashMap` — an allocation proportional to corpus match count rather than + // the requested limit. The bounded collector keeps only the top + // `bm25_cap` documents via a per-segment min-heap, capping peak memory + // inside Tantivy. Under `ReducedCandidates` the existing post-fusion + // truncate trims further. + let bm25_cap = (query.limit as usize) + .saturating_mul(BM25_TOP_K_MULTIPLIER) + .max(BM25_TOP_K_FLOOR); + let collector = BoundedScoresCollector::new(idx.fields().entity_id, bm25_cap); let mut raw = searcher .search(tq.as_ref(), &collector) .map_err(|e| QueryError::StorageError(format!("BM25 search: {e}")))?; @@ -342,9 +368,17 @@ impl SearchExecutor<'_> { // when a NaN slips through (BM25 over degenerate term statistics can emit // one). Neutralize NaN to 0.0 — the same neutral value the ranking module // uses (`finite_score`) — then sort with `total_cmp`, a true total order - // over all finite scores, so the result is deterministic. + // over all finite scores. Break score ties on ASCENDING entity id so the + // ordering is a true total order even when two documents earn the same + // BM25 score (Tantivy's doc-id traversal is not a stable secondary key + // across segment merges / restarts); this is the same determinism guard + // applied to RRF fusion (W14). let finite = |s: f32| if s.is_nan() { 0.0 } else { s }; - raw.sort_by(|a, b| finite(b.1).total_cmp(&finite(a.1))); + raw.sort_by(|a, b| { + finite(b.1) + .total_cmp(&finite(a.1)) + .then_with(|| a.0.as_u64().cmp(&b.0.as_u64())) + }); Ok(raw) } @@ -419,13 +453,36 @@ impl SearchExecutor<'_> { /// [`Self::apply_deferred_post_filters`], so the correct degraded behavior is /// to skip the index intersection entirely (retain all) and surface a warning /// rather than collapse to an empty result set. + /// + /// # Creator searches are item-only here (C-CRITICAL) + /// + /// The bitmap indexes (`category_index`/`format_index`/…) are ITEM-scoped: + /// they hold item ids keyed by item metadata, always wired regardless of + /// `entity_kind`. For a creator SEARCH the candidates are CREATOR ids, so + /// intersecting them against the item indexes is a namespace error — in a + /// creator-only DB the item category index is empty and EVERY creator is + /// dropped; in a mixed DB only creators whose id happens to alias an item in + /// that category survive. A CategoryEq/FormatEq on creators is instead + /// resolved by [`Self::apply_creator_metadata_filter`] (Stage 2b) against the + /// per-creator stored metadata. So this stage returns early (retains all) for + /// `EntityKind::Creator`; the deferred variants are also no-ops for creators + /// in Stage 2 and are handled correctly downstream, so the early return is + /// safe for them too. #[allow(clippy::significant_drop_tightening)] // universe read guard held across the evaluator fn apply_metadata_filter( &self, + entity_kind: EntityKind, combined_filter: Option<&FilterExpr>, candidates: &mut Vec, warnings: &mut Vec, ) { + // Creator candidates are never intersected against the item-scoped bitmap + // indexes — see the doc above. Stage 2b applies any creator metadata + // filter against the correct per-creator metadata instead. + if entity_kind == EntityKind::Creator { + return; + } + let Some(filter_expr) = combined_filter else { return; }; @@ -589,9 +646,18 @@ impl SearchExecutor<'_> { retrieval_scores: &HashMap, now: Timestamp, ) -> Result, QueryError> { - let executor = + let mut executor = ProfileExecutor::new(self.ledger).with_degradation_level(self.degradation_level); + // Wire the Shuffle sort's per-user, per-minute seed (spec §11.6) so SEARCH + // shuffle varies per user; without a FOR USER it stays a stable per-minute + // global permutation. + if matches!(profile.sort, Some(crate::ranking::profile::Sort::Shuffle)) + && let Some(user_id) = query.for_user + { + executor = executor.with_shuffle_user(user_id); + } + // Pre-load item metadata for keyword hint matching when session is active. let item_metadata: HashMap> = if self.session_context.is_some() { @@ -1389,6 +1455,80 @@ mod tests { ); } + /// C-CRITICAL regression (creator metadata filter): `apply_metadata_filter` + /// must NOT intersect CREATOR candidates against the ITEM-scoped bitmap + /// indexes. With `EntityKind::Creator` it retains all candidates (the creator + /// path relies on Stage 2b); with `EntityKind::Item` the item index still + /// applies. Before the fix, a creator-only DB dropped every creator because + /// the item category index was empty / held a different namespace. + #[test] + fn creator_search_skips_item_index_intersection() { + let schema = test_schema(); + let ledger = SignalLedger::new(schema, Box::new(NoopWalWriter)); + let profile_reg = setup_registry(); + let registry = two_item_registry(); + + // Item-scoped category index: item id 1 ∈ "jazz". Creator candidate id 1 + // would alias onto this slot if the item index were (wrongly) applied. + let category = BitmapIndex::new("category"); + category.insert(1, "jazz"); + let universe = two_item_universe(); + + // Build an executor wiring the item category index. + let exec = SearchExecutor::new( + &ledger, + &profile_reg, + None, // text_index + Some(®istry), // embedding_registry + Some(&category), // category (ITEM-scoped) + None, // format + None, // creator + None, // tag + None, // duration + None, // created_at + Some(&universe), // universe + ); + + // Creator candidates (ids 1 and 2). The filter asks for category "news", + // which no ITEM is in — so an item-index intersection would drop BOTH (or + // keep only the aliasing id 1). The creator guard must retain both and let + // Stage 2b decide against per-creator metadata. + let mut creator_candidates = vec![EntityId::new(1), EntityId::new(2)]; + let mut warnings = Vec::new(); + let filter = FilterExpr::CategoryEq("news".into()); + exec.apply_metadata_filter( + EntityKind::Creator, + Some(&filter), + &mut creator_candidates, + &mut warnings, + ); + let mut ids: Vec = creator_candidates.iter().map(|e| e.as_u64()).collect(); + ids.sort_unstable(); + assert_eq!( + ids, + vec![1, 2], + "creator candidates must NOT be intersected against the item category index; \ + both survive Stage 2 and Stage 2b resolves the real per-creator filter" + ); + + // Sanity: for ITEM candidates the same index DOES apply — only the jazz + // item (id 1) survives a CategoryEq(jazz) filter; id 2 is dropped. + let mut item_candidates = vec![EntityId::new(1), EntityId::new(2)]; + let mut warnings = Vec::new(); + let jazz = FilterExpr::CategoryEq("jazz".into()); + exec.apply_metadata_filter( + EntityKind::Item, + Some(&jazz), + &mut item_candidates, + &mut warnings, + ); + assert_eq!( + item_candidates, + vec![EntityId::new(1)], + "item candidates ARE intersected against the item category index" + ); + } + /// Completeness-W regression: a poisoned universe lock must not silently /// return an empty result set. `resolve_scope` recovers the last-known /// universe via `PoisonError::into_inner` and surfaces a warning, and diff --git a/tidal/src/query/suggest.rs b/tidal/src/query/suggest.rs index 07c598f..cbe66c6 100644 --- a/tidal/src/query/suggest.rs +++ b/tidal/src/query/suggest.rs @@ -31,7 +31,13 @@ pub struct Suggestion { pub struct Suggest { /// Prefix to autocomplete. Empty string returns trending searches. pub prefix: String, - /// Optional user for personalized suggestion ordering (reserved for future use). + /// Optional user for personalized suggestion ordering. + /// + /// When set, it is forwarded to [`SuggestionIndex::suggest`] and biases the + /// tie-break among equal-frequency trending suggestions (different users see a + /// stable-but-slightly-varied order). It does not change which suggestions + /// appear — only the relative order of ties — and has no effect on prefix + /// (non-trending) suggestions, which are ordered by the sorted term list. pub for_user: Option, /// Maximum number of suggestions to return (1..=50). pub limit: u32, @@ -201,11 +207,20 @@ impl SuggestionIndex { /// Uses binary search to find the start of the prefix range in the sorted /// term list, then collects consecutive matches. Returns at most `limit` terms. /// - /// If `prefix` is empty, returns top-N trending queries instead. + /// If `prefix` is empty, returns top-N trending queries instead. The optional + /// `for_user` biases the tie-break among equal-frequency trending terms so + /// different users see a stable-but-slightly-varied ordering (the + /// "personalized suggestion ordering" the [`Suggest::for_user`] field + /// documents); it does not change which terms appear, only the order of ties. #[must_use] - pub fn suggest(&self, prefix: &str, limit: usize) -> Vec { + pub fn suggest( + &self, + prefix: &str, + limit: usize, + for_user: Option, + ) -> Vec { if prefix.is_empty() { - return self.trending_suggestions(limit); + return self.trending_suggestions(limit, for_user); } let prefix_lower = prefix.to_lowercase(); @@ -232,13 +247,26 @@ impl SuggestionIndex { } /// Return top N trending search queries (by query frequency). - fn trending_suggestions(&self, limit: usize) -> Vec { + /// + /// Ties on frequency are broken DETERMINISTICALLY rather than by `DashMap` + /// iteration order (which is process-randomized), so the returned top-N is + /// stable across runs for the same data (the same determinism guard applied + /// to RRF fusion, W14). When `for_user` is set, the tie-break is seeded by the + /// user id so different users see a stable-but-slightly-varied ordering among + /// equal-frequency terms — the per-user ordering the [`Suggest::for_user`] + /// field documents. Without `for_user` the tie-break is a plain ascending sort + /// of the term string. + fn trending_suggestions(&self, limit: usize, for_user: Option) -> Vec { let mut pairs: Vec<(String, u64)> = self .query_freq .iter() .map(|entry| (entry.key().clone(), entry.value().load(Ordering::Relaxed))) // Approximate: Relaxed is correct, no sync needed .collect(); - pairs.sort_by(|a, b| b.1.cmp(&a.1)); + let seed = for_user.map_or(0, EntityId::as_u64); + pairs.sort_by(|a, b| { + b.1.cmp(&a.1) + .then_with(|| Self::tie_break_key(&a.0, seed).cmp(&Self::tie_break_key(&b.0, seed))) + }); pairs.truncate(limit); pairs .into_iter() @@ -246,6 +274,28 @@ impl SuggestionIndex { .collect() } + /// Deterministic tie-break key for a trending term under an optional per-user + /// `seed`. With `seed == 0` (no user) it is the term bytes themselves, giving + /// a plain ascending tie order; with a user seed it is a stable hash of + /// `(seed, term)`, so equal-frequency terms order consistently per user but + /// differ between users. Either way the key is a pure function of its inputs, + /// so the order is reproducible across process runs. + fn tie_break_key(term: &str, seed: u64) -> u64 { + use std::hash::{Hash, Hasher}; + if seed == 0 { + // No user: a stable hash of the term alone yields a deterministic, + // user-agnostic tie order (the bytes themselves would also work, but a + // single u64 key keeps the comparator uniform with the seeded path). + let mut h = std::collections::hash_map::DefaultHasher::new(); + term.hash(&mut h); + return h.finish(); + } + let mut h = std::collections::hash_map::DefaultHasher::new(); + seed.hash(&mut h); + term.hash(&mut h); + h.finish() + } + /// Evict low-frequency entries when the trending map is near the cap. /// /// Removes the bottom half of entries by frequency when `query_freq` exceeds @@ -265,8 +315,14 @@ impl SuggestionIndex { .iter() .map(|e| (e.key().clone(), e.value().load(Ordering::Relaxed))) // Approximate: Relaxed is correct .collect(); - // Sort ascending by frequency; keep the top half. - pairs.sort_by_key(|&(_, f)| f); + // Sort ascending by frequency, breaking ties DETERMINISTICALLY on a stable + // hash of the term rather than on `DashMap` iteration order (which is + // process-randomized) — so WHICH equal-frequency terms get evicted is + // reproducible across runs rather than depending on the map seed (W14). + pairs.sort_by(|a, b| { + a.1.cmp(&b.1) + .then_with(|| Self::tie_break_key(&a.0, 0).cmp(&Self::tie_break_key(&b.0, 0))) + }); let remove_count = pairs.len() / 2; for (key, _) in pairs.into_iter().take(remove_count) { // Keep `key_count` in lockstep with the map: decrement only when a @@ -295,7 +351,7 @@ mod tests { #[test] fn new_index_is_empty() { let idx = SuggestionIndex::new(); - assert!(idx.suggest("anything", 10).is_empty()); + assert!(idx.suggest("anything", 10, None).is_empty()); } #[test] @@ -303,11 +359,11 @@ mod tests { let idx = SuggestionIndex::new(); idx.index_title("Rust Programming Tutorial"); - let results = idx.suggest("rus", 10); + let results = idx.suggest("rus", 10, None); assert_eq!(results.len(), 1); assert_eq!(results[0].text, "rust"); - let results = idx.suggest("pro", 10); + let results = idx.suggest("pro", 10, None); assert_eq!(results.len(), 1); assert_eq!(results[0].text, "programming"); } @@ -317,7 +373,7 @@ mod tests { let idx = SuggestionIndex::new(); idx.index_title("Rust Rust Rust"); - let results = idx.suggest("rus", 10); + let results = idx.suggest("rus", 10, None); assert_eq!(results.len(), 1); } @@ -327,11 +383,11 @@ mod tests { idx.index_title("I am a Rustacean"); // "I", "a" are too short (< 2 chars). - let results = idx.suggest("i", 10); + let results = idx.suggest("i", 10, None); assert!(results.is_empty()); // "am" is exactly 2 chars. - let results = idx.suggest("am", 10); + let results = idx.suggest("am", 10, None); assert_eq!(results.len(), 1); } @@ -340,7 +396,7 @@ mod tests { let idx = SuggestionIndex::new(); idx.index_title("Rust TUTORIAL"); - let results = idx.suggest("RUS", 10); + let results = idx.suggest("RUS", 10, None); assert_eq!(results.len(), 1); assert_eq!(results[0].text, "rust"); } @@ -350,11 +406,11 @@ mod tests { let idx = SuggestionIndex::new(); idx.index_title("alpha beta gamma delta"); - let results = idx.suggest("", 2); + let results = idx.suggest("", 2, None); assert!(results.len() <= 2); // All 4 terms start with different prefixes; requesting "a" gives 1. - let results = idx.suggest("a", 10); + let results = idx.suggest("a", 10, None); assert_eq!(results.len(), 1); } @@ -363,7 +419,7 @@ mod tests { let idx = SuggestionIndex::new(); idx.index_title("Rust Tutorial"); - let results = idx.suggest("zzz", 10); + let results = idx.suggest("zzz", 10, None); assert!(results.is_empty()); } @@ -374,7 +430,7 @@ mod tests { idx.record_query("rust tutorial"); idx.record_query("rust async"); - let trending = idx.suggest("", 10); + let trending = idx.suggest("", 10, None); assert_eq!(trending.len(), 2); assert_eq!(trending[0].text, "rust tutorial"); assert_eq!(trending[0].frequency, 2); @@ -388,7 +444,7 @@ mod tests { idx.record_query(" Rust Tutorial "); // leading/trailing whitespace idx.record_query("rust tutorial"); - let trending = idx.suggest("", 10); + let trending = idx.suggest("", 10, None); assert_eq!(trending.len(), 1); assert_eq!(trending[0].frequency, 2); } @@ -399,7 +455,7 @@ mod tests { idx.record_query(""); idx.record_query(" "); - let trending = idx.suggest("", 10); + let trending = idx.suggest("", 10, None); assert!(trending.is_empty()); } @@ -484,7 +540,7 @@ mod tests { let idx = SuggestionIndex::new(); idx.record_query("rust tutorial"); // double space idx.record_query("rust tutorial"); // single space - let trending = idx.suggest("", 10); + let trending = idx.suggest("", 10, None); assert_eq!(trending.len(), 1, "internal whitespace should be collapsed"); assert_eq!(trending[0].frequency, 2); } @@ -496,7 +552,7 @@ mod tests { idx.record_query("rust"); idx.record_query("rust"); - let results = idx.suggest("rus", 10); + let results = idx.suggest("rus", 10, None); assert_eq!(results.len(), 1); assert_eq!(results[0].text, "rust"); assert_eq!(results[0].frequency, 2); @@ -520,22 +576,113 @@ mod tests { idx.index_title("Rust Tutorial"); idx.index_title("Rust Async Programming"); - let results = idx.suggest("rus", 10); + let results = idx.suggest("rus", 10, None); assert_eq!(results.len(), 1); // "rust" deduplicated - let results = idx.suggest("a", 10); + let results = idx.suggest("a", 10, None); assert_eq!(results.len(), 1); // "async" - let results = idx.suggest("t", 10); + let results = idx.suggest("t", 10, None); assert_eq!(results.len(), 1); // "tutorial" - let results = idx.suggest("p", 10); + let results = idx.suggest("p", 10, None); assert_eq!(results.len(), 1); // "programming" } #[test] fn default_trait() { let idx = SuggestionIndex::default(); - assert!(idx.suggest("x", 10).is_empty()); + assert!(idx.suggest("x", 10, None).is_empty()); + } + + /// W14 regression: trending ties (equal frequency) must order + /// DETERMINISTICALLY across runs, not by randomized `DashMap` iteration. + /// Repeating the query against unchanged data must return identical order. + #[test] + fn trending_tie_order_is_deterministic() { + let idx = SuggestionIndex::new(); + // Five distinct terms, each queried exactly once → all tie on frequency 1. + for t in &["alpha", "bravo", "charlie", "delta", "echo"] { + idx.record_query(t); + } + let first: Vec = idx + .suggest("", 10, None) + .into_iter() + .map(|s| s.text) + .collect(); + let second: Vec = idx + .suggest("", 10, None) + .into_iter() + .map(|s| s.text) + .collect(); + assert_eq!( + first, second, + "equal-frequency trending order must be deterministic across calls" + ); + assert_eq!(first.len(), 5, "all tied terms returned"); + } + + /// SUGGESTION regression: `for_user` is actually CONSULTED — it biases the + /// tie-break among equal-frequency terms, so two different users can see a + /// different (but each-stable) tie order. The set of terms is unchanged. + #[test] + fn for_user_biases_trending_tie_order() { + let idx = SuggestionIndex::new(); + // Many tied terms so the per-user permutation is very likely to differ. + for t in &["aa", "bb", "cc", "dd", "ee", "ff", "gg", "hh"] { + idx.record_query(t); + } + let global: Vec = idx + .suggest("", 8, None) + .into_iter() + .map(|s| s.text) + .collect(); + let user_a: Vec = idx + .suggest("", 8, Some(EntityId::new(1))) + .into_iter() + .map(|s| s.text) + .collect(); + let user_b: Vec = idx + .suggest("", 8, Some(EntityId::new(2))) + .into_iter() + .map(|s| s.text) + .collect(); + + // Same SET of terms regardless of user (only the tie order may differ). + let set = |v: &[String]| { + let mut s = v.to_vec(); + s.sort(); + s + }; + assert_eq!(set(&global), set(&user_a)); + assert_eq!(set(&global), set(&user_b)); + + // Each user's order is stable across repeated calls (deterministic seed). + let user_a_again: Vec = idx + .suggest("", 8, Some(EntityId::new(1))) + .into_iter() + .map(|s| s.text) + .collect(); + assert_eq!(user_a, user_a_again, "per-user tie order is deterministic"); + + // The knob is actually CONSULTED: across a handful of user seeds the tie + // order is not forced identical for everyone (a no-op `for_user` would + // produce the same order for all users). Comparing several seeds makes the + // assertion robust even though the `DefaultHasher` outcome is itself fixed. + let orders: Vec> = [1u64, 2, 3, 7, 42] + .iter() + .map(|&u| { + idx.suggest("", 8, Some(EntityId::new(u))) + .into_iter() + .map(|s| s.text) + .collect() + }) + .collect(); + assert!( + orders.iter().any(|o| *o != global), + "for_user must influence the tie order for at least some user; \ + a silently-ignored knob would yield the global order for everyone" + ); + let _ = (user_a, user_b); } } diff --git a/tidal/src/ranking/builtins.rs b/tidal/src/ranking/builtins.rs index 56e2b0b..77e0530 100644 --- a/tidal/src/ranking/builtins.rs +++ b/tidal/src/ranking/builtins.rs @@ -127,9 +127,25 @@ pub fn register_builtins(registry: &mut ProfileRegistry) -> Result<(), ProfileEr Ok(()) } +/// The global trending profile. +/// +/// `Sort::Trending` is the ordering authority on the global (non-cohort) path: +/// its formula `view_vel + 2*share_vel` over the 24h window *replaces* stages 4-5 +/// (spec §11.9), so the executor skips the boost loop when this sort is active +/// (no double-count — previously the base and these boosts both read the same +/// signals and summed to `2*view_vel + 4*share_vel`). +/// +/// The `view`/`share` velocity boosts below are NOT redundant: they are the +/// signal definition the **cohort-scoped rescore** (query Stage 3b, +/// `rescore_with_cohort`) reads to re-score candidates against a cohort's signal +/// ledger — that path consumes `profile.boosts` directly and does not run the +/// `Sort::Trending` formula. Their weights deliberately mirror the formula so the +/// cohort ordering matches the global one. Keep the two in sync. fn trending() -> RankingProfile { let mut p = skeleton("trending"); p.sort = Some(Sort::Trending); + // Cohort-rescore signal definition (see doc above); the global path uses the + // Sort::Trending formula and skips these per spec §11.9. p.boosts = vec![ Boost { signal: "share".into(), @@ -424,13 +440,17 @@ fn search() -> RankingProfile { /// `cohort_trending`: trending content scoped to a named cohort. /// /// Identical boosts and sort mode to the global `trending` profile, but -/// intended for use with `RetrieveBuilder::cohort("my_cohort")`. The executor -/// reads signal values from the cohort signal ledger instead of the global -/// ledger. Without a `cohort` clause, this profile behaves identically to -/// `trending`. +/// intended for use with `RetrieveBuilder::cohort("my_cohort")`. With a `cohort` +/// clause, the executor reads signal values from the cohort signal ledger via the +/// Stage 3b rescore (`rescore_with_cohort`), which consumes `profile.boosts` +/// directly. Without a `cohort` clause, this profile behaves identically to +/// `trending`: the `Sort::Trending` formula orders results and the boosts below +/// are skipped (spec §11.9), so there is no double-count. fn cohort_trending() -> RankingProfile { let mut p = skeleton("cohort_trending"); p.sort = Some(Sort::Trending); + // Cohort-rescore signal definition (read by Stage 3b when a cohort is set); + // the no-cohort path uses the Sort::Trending formula and skips these. p.boosts = vec![ Boost { signal: "share".into(), diff --git a/tidal/src/ranking/executor/context.rs b/tidal/src/ranking/executor/context.rs index 794fce9..d94507a 100644 --- a/tidal/src/ranking/executor/context.rs +++ b/tidal/src/ranking/executor/context.rs @@ -25,12 +25,14 @@ pub struct UserContext { /// Items from creators the user has interacted with get a positive boost /// proportional to the interaction weight. pub creator_interaction_boosts: HashMap, - /// Per-item preference boost: maps `item_id (u32)` -> cosine similarity in - /// `[-1.0, 1.0]` between the user's preference vector and the candidate's - /// content embedding. Pre-computed by `RetrieveExecutor` when both a + /// Per-item preference boost: maps the full `u64` entity id -> cosine + /// similarity in `[-1.0, 1.0]` between the user's preference vector and the + /// candidate's content embedding. Keyed by the full u64 (not a truncated u32 + /// slot) so two candidates whose ids differ only above bit 32 cannot collide + /// on one map entry. Pre-computed by `RetrieveExecutor` when both a /// preference vector and per-item embeddings are available; empty when the /// user has no preference vector or embeddings are absent. - pub preference_boosts: HashMap, + pub preference_boosts: HashMap, } // -- Scored candidate --------------------------------------------------------- diff --git a/tidal/src/ranking/executor/formulas.rs b/tidal/src/ranking/executor/formulas.rs index 35a91e8..11631c5 100644 --- a/tidal/src/ranking/executor/formulas.rs +++ b/tidal/src/ranking/executor/formulas.rs @@ -69,18 +69,63 @@ pub(super) fn hidden_gems_score(quality: f64, view_count: f64) -> f64 { quality / (view_count + 10.0).log10() } -/// Shuffle: deterministic hash of entity ID for stable random ordering. -pub(super) fn shuffle_score(entity_id: u64) -> f64 { - let hash = blake3::hash(&entity_id.to_le_bytes()); +/// Shuffle pseudo-random draw in `[0, 1)` from a per-(user, minute, item) seed. +/// +/// Spec §11.6: `shuffle_score(item) = random(seed) * quality_weight(item)` with +/// `seed = hash(user_id, timestamp_minute)` — "same results for same user within +/// 1 minute". The item id is folded into the hash *after* the seed so the per-item +/// draw varies within a single seed (otherwise every item would share one random +/// value and `quality_weight` alone would order them, collapsing the random +/// sampling the mode exists to provide). Holding `(user_id, timestamp_minute)` +/// fixed makes the whole permutation reproducible across a page refresh, and +/// changing the user *or* crossing a minute boundary reshuffles — exactly the +/// "deterministic within short time windows" contract. +/// +/// `user_id == 0` is the anonymous / no-user case (RETRIEVE without `FOR USER`): +/// the draw is then a stable per-minute global permutation rather than per-user +/// variety, which is the best we can do without a user identity. +pub(super) fn shuffle_random(user_id: u64, timestamp_minute: u64, entity_id: u64) -> f64 { + // Divide by 2^64 (named constant; `u64::MAX as f64` rounds to 2^64 anyway) so + // the result is in [0, 1) — a uniform pseudo-random draw seeded by the tuple. + const TWO_POW_64: f64 = 18_446_744_073_709_551_616.0; + let mut hasher = blake3::Hasher::new(); + hasher.update(&user_id.to_le_bytes()); + hasher.update(×tamp_minute.to_le_bytes()); + hasher.update(&entity_id.to_le_bytes()); + let hash = hasher.finalize(); let bytes = hash.as_bytes(); - // First 8 bytes as u64, normalized to [0, 1]. let arr: [u8; 8] = [ bytes[0], bytes[1], bytes[2], bytes[3], bytes[4], bytes[5], bytes[6], bytes[7], ]; let v = u64::from_le_bytes(arr); #[allow(clippy::cast_precision_loss)] - let score = v as f64 / u64::MAX as f64; - score + { + v as f64 / TWO_POW_64 + } +} + +/// Shuffle quality score (spec §11.6): +/// `completion_rate * 0.5 + like_ratio * 0.3 + log10(views + 1) * 0.2`. +/// +/// Inputs are read from the ledger by the caller: `completion_rate` is the +/// `completion` running decay score clamped to `[0, 1]`, `like_ratio` is +/// `likes / (views + 1)` clamped to `[0, 1]`, and `views` is the all-time view +/// count. A floor of `0.0` guards against a degenerate negative composite so the +/// `sqrt` in `shuffle_quality_weight` is always real. +pub(super) fn shuffle_quality_score(completion_rate: f64, like_ratio: f64, views: f64) -> f64 { + let log_views = (views + 1.0).log10(); + completion_rate + .mul_add(0.5, like_ratio.mul_add(0.3, log_views * 0.2)) + .max(0.0) +} + +/// Shuffle quality weight: `sqrt(quality_score)` (spec §11.6). High-quality items +/// get a larger multiplier on their random draw, so they are *more likely* to +/// surface without being guaranteed. A zero-quality item keeps a weight of 0 and +/// therefore scores 0 (sorted last among shuffled items), matching the spec's +/// "high-quality items are more likely to appear but not guaranteed". +pub(super) fn shuffle_quality_weight(quality_score: f64) -> f64 { + quality_score.max(0.0).sqrt() } /// Haversine great-circle distance in kilometres. @@ -172,6 +217,44 @@ mod tests { ); } + #[test] + fn shuffle_random_is_stable_for_fixed_seed_and_in_unit_range() { + // Spec §11.6: the draw is deterministic for a fixed (user, minute, item) + // tuple and lies in [0, 1). + let a = shuffle_random(42, 1000, 7); + let b = shuffle_random(42, 1000, 7); + assert_eq!(a, b, "same (user, minute, item) must give the same draw"); + assert!((0.0..1.0).contains(&a), "draw must be in [0, 1), got {a}"); + } + + #[test] + fn shuffle_random_varies_by_user_minute_and_item() { + // Changing any seed component changes the draw (no fixed global hash). + let base = shuffle_random(1, 100, 9); + assert_ne!(base, shuffle_random(2, 100, 9), "user must change the draw"); + assert_ne!( + base, + shuffle_random(1, 101, 9), + "minute must change the draw" + ); + assert_ne!(base, shuffle_random(1, 100, 8), "item must change the draw"); + } + + #[test] + fn shuffle_quality_weight_rewards_quality() { + // quality_weight = sqrt(quality_score); higher quality -> larger weight. + let low = shuffle_quality_weight(shuffle_quality_score(0.1, 0.0, 1.0)); + let high = shuffle_quality_weight(shuffle_quality_score(1.0, 1.0, 1000.0)); + assert!(high > low, "higher quality must yield a larger weight"); + // Zero quality -> zero weight -> a zero shuffle score (sorted last). + assert_eq!( + shuffle_quality_weight(shuffle_quality_score(0.0, 0.0, 0.0)), + 0.0 + ); + // Weight is always real (the quality floor guards the sqrt). + assert!(shuffle_quality_weight(shuffle_quality_score(0.0, 0.0, 0.0)).is_finite()); + } + #[test] fn haversine_same_point_is_zero() { let d = haversine_km(40.7128, -74.0060, 40.7128, -74.0060); diff --git a/tidal/src/ranking/executor/helpers.rs b/tidal/src/ranking/executor/helpers.rs index e9d0baf..529bde9 100644 --- a/tidal/src/ranking/executor/helpers.rs +++ b/tidal/src/ranking/executor/helpers.rs @@ -32,7 +32,7 @@ pub(super) const COARSE_VALUE_WINDOW: Window = Window::AllTime; // -- Signal reading ----------------------------------------------------------- -/// Read a signal aggregation for a candidate. +/// Read a signal aggregation for a candidate against an explicit query clock. /// /// Propagates the ledger's schema error when `signal` is not defined, so a /// profile that slipped a typo'd signal name past registration fails loud @@ -40,6 +40,14 @@ pub(super) const COARSE_VALUE_WINDOW: Window = Window::AllTime; /// not an error -- the ledger returns 0 / 0.0 / `None` for those, which we map /// to 0.0. /// +/// `now_ns` is the query's logical clock, captured **once** per query by the +/// executor and threaded into every ledger read so the whole candidate set is +/// decayed/aged to one consistent "now". Without it each candidate would read +/// `Timestamp::now()` independently and re-running the identical query against +/// an unchanged ledger would yield slightly different scores (Accuracy-W, +/// M0-M10 review): all the time-dependent reads here route through the ledger's +/// explicit-clock `_at` variants. +/// /// Under `DegradationLevel::CoarseAggregates` (or `NoDiversity`), windows are /// substituted to reduce ledger scan cost: /// - `SignalAgg::Value` → `Window::AllTime` (warm-tier count, O(1)) @@ -58,6 +66,7 @@ pub(crate) fn read_agg( window: Window, ledger: &SignalLedger, degradation: DegradationLevel, + now_ns: u64, ) -> crate::Result { let window = if degradation.coarsens_aggregates() { match agg { @@ -71,10 +80,10 @@ pub(crate) fn read_agg( match agg { SignalAgg::Value => { #[allow(clippy::cast_precision_loss)] - let count = ledger.read_windowed_count(entity_id, signal, window)? as f64; + let count = ledger.read_windowed_count_at(entity_id, signal, window, now_ns)? as f64; Ok(count) } - SignalAgg::Velocity => ledger.read_velocity(entity_id, signal, window), + SignalAgg::Velocity => ledger.read_velocity_at(entity_id, signal, window, now_ns), // `DecayScore` intentionally ignores `window`. A decay score is an O(1) // running quantity -- the exponentially-weighted sum maintained in the // hot tier (`HotSignalState`) -- decayed forward from its last update to @@ -85,7 +94,7 @@ pub(crate) fn read_agg( // aggregation arm shares one signature; passing different windows for a // `DecayScore` term yields the same value by construction. SignalAgg::DecayScore => Ok(ledger - .read_decay_score(entity_id, signal, 0)? + .read_decay_score_at(entity_id, signal, 0, now_ns)? .unwrap_or(0.0)), SignalAgg::Ratio | SignalAgg::RelativeVelocity => { // Not yet implemented -- planned for M3 when cross-signal reads are @@ -113,6 +122,8 @@ pub(crate) fn read_agg( /// has no `view` signal). A missing signal contributes 0.0 to the sort score; every /// other error still propagates. /// +/// `now_ns` is the query clock threaded by the executor (see [`read_agg`]). +/// /// # Errors /// /// Propagates any non-`UnknownSignalType` [`read_agg`] error (notably the @@ -124,8 +135,9 @@ pub(super) fn read_agg_for_sort( window: Window, ledger: &SignalLedger, degradation: DegradationLevel, + now_ns: u64, ) -> crate::Result { - match read_agg(entity_id, signal, agg, window, ledger, degradation) { + match read_agg(entity_id, signal, agg, window, ledger, degradation, now_ns) { Err(crate::TidalError::Schema(crate::schema::error::SchemaError::UnknownSignalType(_))) => { Ok(0.0) } @@ -138,7 +150,9 @@ pub(super) fn read_agg_for_sort( /// Check whether a candidate passes all gate thresholds. /// /// Gates are correctness filters -- always use `Full` degradation so that -/// window precision is never sacrificed for gate evaluation. +/// window precision is never sacrificed for gate evaluation. `now_ns` is the +/// query clock threaded by the executor (see [`read_agg`]) so the gate decision +/// ages every candidate to the same "now" as the score. /// /// # Errors /// @@ -149,6 +163,7 @@ pub(super) fn passes_gates( entity_id: EntityId, gates: &[Gate], ledger: &SignalLedger, + now_ns: u64, ) -> crate::Result { for gate in gates { let value = read_agg( @@ -158,6 +173,7 @@ pub(super) fn passes_gates( gate.window, ledger, DegradationLevel::Full, + now_ns, )?; if value < gate.min_threshold { return Ok(false); @@ -177,7 +193,9 @@ pub(super) fn passes_gates( /// /// Like gates, excludes are correctness filters: they always read at /// [`DegradationLevel::Full`] so window precision is never traded away while -/// deciding whether content the user asked to never see can appear. +/// deciding whether content the user asked to never see can appear. `now_ns` is +/// the query clock threaded by the executor (see [`read_agg`]) so the exclusion +/// decision ages every candidate to the same "now" as the score. /// /// # Errors /// @@ -188,6 +206,7 @@ pub(super) fn passes_excludes( entity_id: EntityId, excludes: &[Exclude], ledger: &SignalLedger, + now_ns: u64, ) -> crate::Result { for exclude in excludes { let value = read_agg( @@ -197,6 +216,7 @@ pub(super) fn passes_excludes( exclude.window, ledger, DegradationLevel::Full, + now_ns, )?; if value > exclude.above { return Ok(false); @@ -209,7 +229,12 @@ pub(super) fn passes_excludes( /// Min-max normalize candidate scores to `[0.0, 1.0]`. /// -/// If all (finite) candidates have the same score, they are all set to 1.0. +/// If all (finite) candidates have the same score (`max_score == min_score`), +/// they are all set to **0.5** per spec 09-ranking-scoring.md §8.2 -- the +/// neutral "no signal differentiates these" midpoint, not the maximally- +/// confident `1.0`. Ordering is unaffected (every score is equal); the reported +/// `score` field is what callers surface for explainability/thresholding, so a +/// degenerate all-equal set must read as neutral rather than top-confidence. /// /// # Infinite "sort last / first" sentinels /// @@ -271,7 +296,8 @@ pub(super) fn normalize(candidates: &mut [ScoredCandidate]) { // +Inf "sort first" -> top (1.0); -Inf "sort last" -> bottom (0.0). if c.score.is_sign_positive() { 1.0 } else { 0.0 } } else if range < f64::EPSILON { - 1.0 + // All candidates scored equally: neutral midpoint per spec §8.2. + 0.5 } else { (c.score - min) / range }; @@ -305,6 +331,7 @@ mod tests { Window::AllTime, &ledger, DegradationLevel::Full, + Timestamp::now().as_nanos(), ); assert!(matches!(result, Err(crate::TidalError::Internal(_)))); } @@ -320,6 +347,7 @@ mod tests { Window::OneHour, &ledger, DegradationLevel::Full, + Timestamp::now().as_nanos(), ); assert!(matches!(result, Err(crate::TidalError::Schema(_)))); } @@ -341,7 +369,7 @@ mod tests { min_threshold: 5.0, }]; // count=3 < threshold=5 -> candidate excluded. - assert!(!passes_gates(entity_id, &gates, &ledger).unwrap()); + assert!(!passes_gates(entity_id, &gates, &ledger, Timestamp::now().as_nanos()).unwrap()); } #[test] @@ -361,7 +389,7 @@ mod tests { min_threshold: 5.0, }]; // count=5 >= threshold=5 -> candidate included. - assert!(passes_gates(entity_id, &gates, &ledger).unwrap()); + assert!(passes_gates(entity_id, &gates, &ledger, Timestamp::now().as_nanos()).unwrap()); } #[test] @@ -374,7 +402,9 @@ mod tests { format: None, }]; normalize(&mut candidates); - assert_eq!(candidates[0].score, 1.0); + // A single candidate is a degenerate all-equal set (range == 0), so it + // normalizes to the neutral midpoint 0.5 per spec §8.2, not 1.0. + assert_eq!(candidates[0].score, 0.5); } fn candidate(id: u64, score: f64) -> ScoredCandidate { @@ -439,7 +469,10 @@ mod tests { } #[test] - fn normalize_all_nan_clamps_to_one() { + fn normalize_all_nan_folds_to_neutral_midpoint() { + // Every NaN is neutralized to 0.0 first, making this a degenerate + // all-equal set (range == 0), which normalizes to the neutral 0.5 per + // spec §8.2 -- not the maximally-confident 1.0. let mut candidates: Vec = (1u64..=3) .map(|i| ScoredCandidate { entity_id: EntityId::new(i), @@ -452,8 +485,8 @@ mod tests { normalize(&mut candidates); for c in &candidates { assert_eq!( - c.score, 1.0, - "NaN score should be clamped to 1.0 after normalize" + c.score, 0.5, + "an all-NaN (all-equal) set should fold to the neutral 0.5 midpoint" ); } } diff --git a/tidal/src/ranking/executor/mod.rs b/tidal/src/ranking/executor/mod.rs index ccc70e6..1683e8e 100644 --- a/tidal/src/ranking/executor/mod.rs +++ b/tidal/src/ranking/executor/mod.rs @@ -81,6 +81,11 @@ pub struct ProfileExecutor<'a> { user_state_for_date_saved: Option<(&'a UserStateIndex, u64)>, /// M7p2: degradation level for coarse-aggregate window substitution. degradation_level: DegradationLevel, + /// User identity for the `Shuffle` sort's per-user, per-minute seed (spec + /// §11.6: `seed = hash(user_id, timestamp_minute)`). `0` is the anonymous / + /// no-`FOR USER` case, which yields a stable per-minute *global* permutation + /// rather than per-user variety. Set via [`Self::with_shuffle_user`]. + shuffle_user_id: u64, } impl<'a> ProfileExecutor<'a> { @@ -95,9 +100,23 @@ impl<'a> ProfileExecutor<'a> { item_metadata: None, user_state_for_date_saved: None, degradation_level: DegradationLevel::Full, + shuffle_user_id: 0, } } + /// Set the user identity for the `Shuffle` sort's per-user, per-minute seed. + /// + /// Spec §11.6 seeds shuffle with `hash(user_id, timestamp_minute)` so a + /// "surprise me" surface refreshes per minute and varies per user. The query + /// pipeline calls this with the resolved `FOR USER` id when present; without a + /// user it stays `0` (a stable per-minute global permutation). Only `Shuffle` + /// reads this field; every other sort ignores it. + #[must_use] + pub(crate) const fn with_shuffle_user(mut self, user_id: u64) -> Self { + self.shuffle_user_id = user_id; + self + } + /// Set the degradation level for coarse-aggregate window substitution. /// /// Under `CoarseAggregates`, `SignalAgg::Value` reads use `Window::AllTime` @@ -172,16 +191,17 @@ impl<'a> ProfileExecutor<'a> { /// /// - `candidates`: entity IDs to consider; excluded/gate-failing candidates are removed. /// - `profile`: the ranking profile controlling sort mode, boosts, penalties, gates, excludes, and decay. - /// - `now`: the query's logical timestamp. **Clock contract:** decay is - /// currently evaluated at *call time*, not at `now`. Every time-dependent - /// read -- `SignalAgg::DecayScore` and the `ProfileDecay` recency factor -- - /// decays forward to wall-clock `Timestamp::now()` *inside* the - /// [`SignalLedger`], so `now` does not yet influence those reads. The - /// parameter is retained because it is part of the public scoring API and is - /// threaded by the RETRIEVE/SEARCH callers and benchmarks; making scoring a - /// pure function of `(ledger snapshot, now)` requires a `read_decay_score_at` - /// on the ledger (out of scope for the ranking layer). See the M0-M10 review - /// (scoring.rs Maintainability-W). + /// - `now`: the query's logical timestamp. **Clock contract:** `now` is the + /// single clock for the whole query. It is converted to `now_ns` once and + /// threaded into every time-dependent ledger read -- windowed counts, + /// velocities, `SignalAgg::DecayScore`, the `ProfileDecay` recency factor, + /// and the gate/exclude evaluations -- via the ledger's explicit-clock `_at` + /// variants. Scoring is therefore a pure function of `(ledger snapshot, + /// now)`: re-running the identical query against an unchanged ledger yields + /// byte-identical scores (Accuracy-W, M0-M10 review). The one remaining + /// `now`-independent piece is `Hot`'s per-candidate *age*, which uses a + /// uniform 24h term until a per-entity `created_at` reverse map is plumbed + /// into the executor. /// /// Returns candidates sorted by descending normalized score in `[0.0, 1.0]`. /// @@ -279,8 +299,11 @@ impl<'a> ProfileExecutor<'a> { |entity_id, snapshot| { // Creator-interaction boost: look up the per-item boost from // the pre-computed map. Items from highly-interacted creators - // get a positive additive boost before normalization. - let item_id = entity_id.as_u64() as u32; + // get a positive additive boost before normalization. This map is + // sourced from a `RoaringBitmap` (inherently u32-keyed), so it is + // looked up by the narrowed `u32` slot, matching how the producer + // populated it from the bitmap's u32 entries. + let item_id = crate::entities::user_state::narrow_item_slot(entity_id.as_u64()); let interaction_boost = user_ctx .creator_interaction_boosts .get(&item_id) @@ -291,9 +314,12 @@ impl<'a> ProfileExecutor<'a> { // pre-computed by the retrieve executor. Items closer to the user's // learned taste get a positive additive boost; dissimilar items // (negative cosine) are mildly penalized. Folds into the same - // pre-normalization additive range as the other boosts. + // pre-normalization additive range as the other boosts. Keyed by + // the FULL u64 id so two candidates differing only above bit 32 + // never alias onto one slot and cross-contaminate their boosts + // (query-executor SUGGESTION, M0-M10 review pass 2). let mut preference_boost = 0.0; - if let Some(&cosine) = user_ctx.preference_boosts.get(&item_id) { + if let Some(&cosine) = user_ctx.preference_boosts.get(&entity_id.as_u64()) { preference_boost = cosine * PREFERENCE_BOOST_WEIGHT; if preference_boost != 0.0 { snapshot.push(("preference_affinity".to_string(), preference_boost)); @@ -384,14 +410,19 @@ impl<'a> ProfileExecutor<'a> { // than re-lowercasing every keyword for every candidate inside // `session_boost` (CLEAN-W, M0-M10 review). Empty when there is no session. let session_keywords = session_ctx.map(Self::lowered_session_keywords); + // Query clock, captured ONCE: every per-candidate time-dependent read + // (excludes, gates, and the scoring reads below) ages to this single + // "now" so the whole result set is consistent and re-running the same + // query against an unchanged ledger is byte-identical (Accuracy-W). + let now_ns = now.as_nanos(); let mut scored: Vec = Vec::new(); for &entity_id in candidates { // Stage 2: hard exclusion -- remove content the user must never see. - if !passes_excludes(entity_id, &profile.excludes, self.ledger)? { + if !passes_excludes(entity_id, &profile.excludes, self.ledger, now_ns)? { continue; } // Stage 6: gates -- remove candidates below quality thresholds. - if !passes_gates(entity_id, &profile.gates, self.ledger)? { + if !passes_gates(entity_id, &profile.gates, self.ledger, now_ns)? { continue; } let (raw, mut snapshot) = @@ -483,6 +514,23 @@ impl<'a> ProfileExecutor<'a> { /// [`passes_gates`] because they *remove* candidates rather than adjust a /// score. /// + /// **Sort overrides boosts/penalties (spec §11 / §11.9).** When a `sort` mode + /// is active its formula "replaces stages 4-5 (boost and penalty application)" + /// — so the Stage-4 boost loop and Stage-5 penalty loop are *skipped* whenever + /// `profile.sort.is_some()`. This removes the `trending`/`cohort_trending` + /// double-count (the `Sort::Trending` base and the profile boosts both read + /// `view`/`share` velocity over the same window; adding them produced + /// `2*view_vel + 4*share_vel` — order-identical to the spec formula but a + /// latent hybrid the spec forbids). The co-engagement blend and `ProfileDecay` + /// recency factor are *not* boosts/penalties and still apply. The separate + /// cohort-scoped rescore (query Stage 3b) reads `profile.boosts` directly and + /// is unaffected — it overwrites this score entirely for cohort queries. + /// + /// **Clock contract.** `now` is converted to `now_ns` once and threaded into + /// every time-dependent ledger read (sort base, boosts, penalties, decay) via + /// the explicit-clock `_at` variants, so scores are a pure function of + /// `(ledger snapshot, now)` (Accuracy-W, M0-M10 review). + /// /// `retrieval_scores`, when present, maps `entity_id -> normalized fused/RRF /// relevance score in [0, 1]` (produced by SEARCH Stage 1c). The seed makes /// the fused order the primary ordering signal when profile boosts are light; @@ -505,6 +553,7 @@ impl<'a> ProfileExecutor<'a> { now: Timestamp, retrieval_scores: Option<&HashMap>, ) -> crate::Result<(f64, Vec<(String, f64)>)> { + let now_ns = now.as_nanos(); let (sort_base, mut snapshot) = self.score_by_sort(entity_id, profile.sort.as_ref(), now)?; @@ -523,8 +572,22 @@ impl<'a> ProfileExecutor<'a> { } // Stage 4: boosts. Capture each contribution in the snapshot. + // + // Spec §11.9: a formula sort "replaces stages 4-5" only for the boosts it + // already SUBSUMES — a boost whose (signal, agg, window) the sort formula + // itself reads. The `trending`/`cohort_trending` builtins keep their + // view/share velocity boosts on the profile purely as the cohort-rescore + // signal definition (Stage 3b); on the global path those exact boosts ARE + // the `Sort::Trending` formula, so re-applying them would double-count. + // Every other profile's boosts (the `for_you`/`following`/`related`/ + // `notification` engagement overlays) read different signals/aggs/windows + // than their sort formula, so they are NOT subsumed and DO apply on top of + // the sort base — preserving each profile's intended engagement weighting. let mut boost_sum = 0.0; for b in &profile.boosts { + if sort_subsumes_boost(profile.sort.as_ref(), &b.signal, &b.agg, b.window) { + continue; + } let val = read_agg_for_sort( entity_id, &b.signal, @@ -532,6 +595,7 @@ impl<'a> ProfileExecutor<'a> { b.window, self.ledger, self.degradation_level, + now_ns, )?; let weighted = b.weight * val; if weighted != 0.0 { @@ -567,6 +631,9 @@ impl<'a> ProfileExecutor<'a> { // first-class (Design Principle 3): each penalty subtracts // `weight * agg(signal, window)`, mirroring the boost loop with the // opposite sign. `weight` is stored positive and applied as `-weight`. + // No builtin formula sort reads a penalty signal, so penalties are never + // subsumed by a sort and always apply (a penalty is a complementary + // demotion overlay, not a re-statement of the sort formula). let mut penalty_sum = 0.0; for p in &profile.penalties { let val = read_agg_for_sort( @@ -576,6 +643,7 @@ impl<'a> ProfileExecutor<'a> { p.window, self.ledger, self.degradation_level, + now_ns, )?; let weighted = p.weight * val; if weighted != 0.0 { @@ -605,6 +673,7 @@ impl<'a> ProfileExecutor<'a> { Window::AllTime, self.ledger, self.degradation_level, + now_ns, )? .clamp(0.0, 1.0); let factor = decay.weight.mul_add(recency, 1.0 - decay.weight); @@ -706,6 +775,29 @@ impl<'a> ProfileExecutor<'a> { } } +/// Whether the active formula `sort` already incorporates this boost's signal, so +/// re-applying it on the global scoring path would double-count (spec §11.9). +/// +/// Today only [`Sort::Trending`] subsumes its boosts: its formula reads +/// `view`/`share` velocity over 24h, exactly the velocity boosts the +/// `trending`/`cohort_trending` builtins carry — kept on the profile only as the +/// cohort-rescore signal definition (Stage 3b). Every other sort's boosts read a +/// different (signal, agg, window) than the sort formula (e.g. `for_you`'s +/// `Sort::Hot` ranks by view *count* while its boosts read view/like *decay +/// scores*), so they are complementary overlays and are never subsumed. +fn sort_subsumes_boost( + sort: Option<&super::profile::Sort>, + signal: &str, + agg: &SignalAgg, + window: Window, +) -> bool { + use super::profile::Sort; + matches!(sort, Some(Sort::Trending)) + && matches!(agg, SignalAgg::Velocity) + && window == Window::TwentyFourHours + && (signal == "view" || signal == "share") +} + mod scoring; #[cfg(test)] diff --git a/tidal/src/ranking/executor/scoring.rs b/tidal/src/ranking/executor/scoring.rs index f719108..7a34572 100644 --- a/tidal/src/ranking/executor/scoring.rs +++ b/tidal/src/ranking/executor/scoring.rs @@ -5,7 +5,10 @@ use super::{ ProfileExecutor, - formulas::{controversial_score, hidden_gems_score, hot_score, shuffle_score, trending_score}, + formulas::{ + controversial_score, hidden_gems_score, hot_score, shuffle_quality_score, + shuffle_quality_weight, shuffle_random, trending_score, + }, helpers::read_agg_for_sort, }; use crate::{ @@ -55,10 +58,14 @@ impl ProfileExecutor<'_> { /// Returns `(score, snapshot)` where `snapshot` lists the raw signal values /// that contributed to the score, for explain-ability in API responses. /// - /// **Clock contract:** `now` is threaded for API symmetry but does not - /// currently steer any read -- every time-dependent sort (`DecayScore`-backed - /// modes, velocity modes) decays/ages forward to wall-clock time *inside* the - /// [`SignalLedger`] at call time. See [`ProfileExecutor::score`]'s `now` doc. + /// **Clock contract:** `now` is the query's logical clock. It is converted to + /// `now_ns` **once** here and threaded into every time-dependent ledger read + /// (velocity and `DecayScore`-backed sorts) via the ledger's explicit-clock + /// `_at` variants, so the whole candidate set is aged to one consistent "now" + /// and re-running the identical query against an unchanged ledger yields + /// byte-identical scores (Accuracy-W, M0-M10 review). Quality-weighted + /// `Shuffle` also consumes the per-minute `timestamp_minute` derived from + /// `now` (spec §11.6). /// /// # Errors /// @@ -72,12 +79,15 @@ impl ProfileExecutor<'_> { sort: Option<&Sort>, now: Timestamp, ) -> crate::Result<(f64, Vec<(String, f64)>)> { + // Capture the query clock ONCE so every per-candidate ledger read below + // ages to the same "now" (reproducible scores; one fewer syscall per read). + let now_ns = now.as_nanos(); match sort { Some(Sort::Hot { gravity }) => self.score_hot(entity_id, *gravity, now), - Some(Sort::Trending) => self.score_trending(entity_id), - Some(Sort::Controversial) => self.score_controversial(entity_id), - Some(Sort::HiddenGems) => self.score_hidden_gems(entity_id), - Some(Sort::Shuffle) => Ok((shuffle_score(entity_id.as_u64()), vec![])), + Some(Sort::Trending) => self.score_trending(entity_id, now_ns), + Some(Sort::Controversial) => self.score_controversial(entity_id, now_ns), + Some(Sort::HiddenGems) => self.score_hidden_gems(entity_id, now_ns), + Some(Sort::Shuffle) => Ok((self.score_shuffle(entity_id, now_ns)?, vec![])), Some(Sort::New) => { // M2 limitation: entity metadata (`created_at`) is not accessible from the // executor. Entity ID is used as a proxy for recency -- ranks higher IDs @@ -91,16 +101,20 @@ impl ProfileExecutor<'_> { let score = entity_id.as_u64() as f64; Ok((score, vec![])) } - Some(Sort::TopWindow { window }) => self.score_top_window(entity_id, *window), + Some(Sort::TopWindow { window }) => self.score_top_window(entity_id, *window, now_ns), Some(Sort::MostViewed { window }) => { - self.single_signal_score(entity_id, "view", &SignalAgg::Value, *window) + self.single_signal_score(entity_id, "view", &SignalAgg::Value, *window, now_ns) } Some(Sort::MostLiked { window }) => { - self.single_signal_score(entity_id, "like", &SignalAgg::Value, *window) - } - Some(Sort::MostFollowed) => { - self.single_signal_score(entity_id, "follow", &SignalAgg::Value, Window::AllTime) + self.single_signal_score(entity_id, "like", &SignalAgg::Value, *window, now_ns) } + Some(Sort::MostFollowed) => self.single_signal_score( + entity_id, + "follow", + &SignalAgg::Value, + Window::AllTime, + now_ns, + ), Some(Sort::CreatorEngagementRate) => { let view_vel = read_agg_for_sort( entity_id, @@ -109,6 +123,7 @@ impl ProfileExecutor<'_> { Window::TwentyFourHours, self.ledger, self.degradation_level, + now_ns, )?; let like_vel = read_agg_for_sort( entity_id, @@ -117,6 +132,7 @@ impl ProfileExecutor<'_> { Window::TwentyFourHours, self.ledger, self.degradation_level, + now_ns, )?; Ok(( view_vel + like_vel, @@ -126,40 +142,40 @@ impl ProfileExecutor<'_> { ], )) } - Some(Sort::Rising) => self.score_rising(entity_id), + Some(Sort::Rising) => self.score_rising(entity_id, now_ns), Some(Sort::AlphabeticalAsc) => Ok((self.score_alphabetical_asc(entity_id), vec![])), Some(Sort::AlphabeticalDesc) => Ok((self.score_alphabetical_desc(entity_id), vec![])), Some(Sort::Shortest) => Ok((self.score_shortest(entity_id), vec![])), Some(Sort::Longest) => Ok((self.score_longest(entity_id), vec![])), Some(Sort::MostCommented { window }) => { - self.single_signal_score(entity_id, "comment", &SignalAgg::Value, *window) + self.single_signal_score(entity_id, "comment", &SignalAgg::Value, *window, now_ns) } Some(Sort::MostShared { window }) => { - self.single_signal_score(entity_id, "share", &SignalAgg::Value, *window) + self.single_signal_score(entity_id, "share", &SignalAgg::Value, *window, now_ns) } Some(Sort::LiveViewerCount) => self.single_signal_score( entity_id, "viewer_count", &SignalAgg::DecayScore, Window::AllTime, + now_ns, ), Some(Sort::DateSaved) => Ok((self.score_date_saved(entity_id), vec![])), None => Ok((0.0, vec![])), } } - /// `_now` is accepted for signature symmetry with [`Self::score_by_sort`] but - /// is intentionally unused: Hot scoring reads only the all-time view count - /// (decayed forward to wall-clock time inside the ledger) and applies a - /// *uniform* 24h age term, because no per-entity `created_at` is plumbed into - /// the executor (see the `age_hours` comment below). When age-aware Hot - /// scoring lands, this `now` becomes the age reference. See the M0-M10 review - /// clock-contract note (scoring.rs Maintainability-W). + /// `now` is the query clock: it ages the all-time view count read consistently + /// across the candidate set (via the ledger's explicit-clock read). It is not + /// yet a per-candidate *age* reference: Hot scoring applies a *uniform* 24h age + /// term because no per-entity `created_at` is plumbed into the executor (see + /// the `age_hours` comment below). When age-aware Hot scoring lands, this `now` + /// becomes the age reference too. fn score_hot( &self, entity_id: EntityId, gravity: f64, - _now: Timestamp, + now: Timestamp, ) -> crate::Result<(f64, Vec<(String, f64)>)> { let views = read_agg_for_sort( entity_id, @@ -168,6 +184,7 @@ impl ProfileExecutor<'_> { Window::AllTime, self.ledger, self.degradation_level, + now.as_nanos(), )?; // The scoring loop receives no per-entity `created_at`: the executor reads // only the signal ledger, and the `created_at` range index is keyed @@ -184,12 +201,84 @@ impl ProfileExecutor<'_> { )) } + /// Shuffle (spec §11.6): `random(seed) * quality_weight(item)` where + /// `seed = hash(user_id, timestamp_minute)` and + /// `quality_weight = sqrt(quality_score)`. + /// + /// The random draw is reproducible within a single minute for a single user + /// (the seed is constant over that window), so a page refresh does not + /// reshuffle — but it varies per user and refreshes each minute, delivering + /// the per-user, time-windowed variety the mode advertises. Quality biases the + /// draw multiplicatively, so high-quality items are more likely to surface + /// without being guaranteed. The `quality_score` is built from real ledger + /// reads (completion decay score, like/view counts) aged to the query clock, + /// not a fixed constant, so low-quality items are genuinely demoted. + /// + /// Missing generic signals degrade to 0.0 via [`read_agg_for_sort`]. With no + /// quality signals at all, every item's `quality_score` is 0, so the random + /// draw is scaled to 0 and the items tie (normalizing to the neutral midpoint) + /// — the mode is only meaningfully a *quality-weighted* shuffle once the app + /// records `completion`/`like`/`view`. The seed inputs still vary per user and + /// per minute, so the permutation refreshes correctly once any quality signal + /// exists. + /// + /// # Errors + /// + /// Propagates any non-degradable ledger read error (an unimplemented + /// aggregation), matching the other formula sorts. + fn score_shuffle(&self, entity_id: EntityId, now_ns: u64) -> crate::Result { + // Quality inputs, all aged to the single query clock for reproducibility. + let completion_rate = read_agg_for_sort( + entity_id, + "completion", + &SignalAgg::DecayScore, + Window::AllTime, + self.ledger, + self.degradation_level, + now_ns, + )? + .clamp(0.0, 1.0); + let likes = read_agg_for_sort( + entity_id, + "like", + &SignalAgg::Value, + Window::AllTime, + self.ledger, + self.degradation_level, + now_ns, + )?; + let views = read_agg_for_sort( + entity_id, + "view", + &SignalAgg::Value, + Window::AllTime, + self.ledger, + self.degradation_level, + now_ns, + )?; + // like_ratio = likes / (views + 1), clamped to [0, 1]: a like is only + // emitted by a viewer, so the ratio is a bounded engagement-quality proxy. + let like_ratio = (likes / (views + 1.0)).clamp(0.0, 1.0); + + let quality_score = shuffle_quality_score(completion_rate, like_ratio, views); + let quality_weight = shuffle_quality_weight(quality_score); + + // timestamp_minute: the query clock truncated to whole minutes, so the + // seed (and therefore the whole permutation) is stable within a minute and + // refreshes at the boundary. 60e9 ns per minute. + let timestamp_minute = now_ns / 60_000_000_000; + let draw = shuffle_random(self.shuffle_user_id, timestamp_minute, entity_id.as_u64()); + + Ok(draw * quality_weight) + } + fn single_signal_score( &self, entity_id: EntityId, signal: &str, agg: &SignalAgg, window: Window, + now_ns: u64, ) -> crate::Result<(f64, Vec<(String, f64)>)> { let val = read_agg_for_sort( entity_id, @@ -198,11 +287,16 @@ impl ProfileExecutor<'_> { window, self.ledger, self.degradation_level, + now_ns, )?; Ok((val, vec![(signal.to_string(), val)])) } - fn score_trending(&self, entity_id: EntityId) -> crate::Result<(f64, Vec<(String, f64)>)> { + fn score_trending( + &self, + entity_id: EntityId, + now_ns: u64, + ) -> crate::Result<(f64, Vec<(String, f64)>)> { // M6: social-graph-scoped trending. When a social subgraph and // per-user signal index are available, compute aggregate velocity // across the subgraph users instead of using the global ledger. @@ -255,6 +349,7 @@ impl ProfileExecutor<'_> { Window::TwentyFourHours, self.ledger, self.degradation_level, + now_ns, )?; let share_vel = read_agg_for_sort( entity_id, @@ -263,6 +358,7 @@ impl ProfileExecutor<'_> { Window::TwentyFourHours, self.ledger, self.degradation_level, + now_ns, )?; Ok(( trending_score(view_vel, share_vel), @@ -273,7 +369,11 @@ impl ProfileExecutor<'_> { )) } - fn score_controversial(&self, entity_id: EntityId) -> crate::Result<(f64, Vec<(String, f64)>)> { + fn score_controversial( + &self, + entity_id: EntityId, + now_ns: u64, + ) -> crate::Result<(f64, Vec<(String, f64)>)> { let pos = read_agg_for_sort( entity_id, "like", @@ -281,6 +381,7 @@ impl ProfileExecutor<'_> { Window::AllTime, self.ledger, self.degradation_level, + now_ns, )?; let neg = read_agg_for_sort( entity_id, @@ -289,6 +390,7 @@ impl ProfileExecutor<'_> { Window::AllTime, self.ledger, self.degradation_level, + now_ns, )?; Ok(( controversial_score(pos, neg), @@ -296,7 +398,11 @@ impl ProfileExecutor<'_> { )) } - fn score_hidden_gems(&self, entity_id: EntityId) -> crate::Result<(f64, Vec<(String, f64)>)> { + fn score_hidden_gems( + &self, + entity_id: EntityId, + now_ns: u64, + ) -> crate::Result<(f64, Vec<(String, f64)>)> { let quality = read_agg_for_sort( entity_id, "completion", @@ -304,6 +410,7 @@ impl ProfileExecutor<'_> { Window::AllTime, self.ledger, self.degradation_level, + now_ns, )?; let view_count = read_agg_for_sort( entity_id, @@ -312,6 +419,7 @@ impl ProfileExecutor<'_> { Window::AllTime, self.ledger, self.degradation_level, + now_ns, )?; Ok(( hidden_gems_score(quality, view_count), @@ -326,6 +434,7 @@ impl ProfileExecutor<'_> { &self, entity_id: EntityId, window: Window, + now_ns: u64, ) -> crate::Result<(f64, Vec<(String, f64)>)> { let views = read_agg_for_sort( entity_id, @@ -334,6 +443,7 @@ impl ProfileExecutor<'_> { window, self.ledger, self.degradation_level, + now_ns, )?; let likes = read_agg_for_sort( entity_id, @@ -342,6 +452,7 @@ impl ProfileExecutor<'_> { window, self.ledger, self.degradation_level, + now_ns, )?; let shares = read_agg_for_sort( entity_id, @@ -350,6 +461,7 @@ impl ProfileExecutor<'_> { window, self.ledger, self.degradation_level, + now_ns, )?; let completion = read_agg_for_sort( entity_id, @@ -358,6 +470,7 @@ impl ProfileExecutor<'_> { window, self.ledger, self.degradation_level, + now_ns, )?; Ok(( views.mul_add( @@ -373,7 +486,11 @@ impl ProfileExecutor<'_> { )) } - fn score_rising(&self, entity_id: EntityId) -> crate::Result<(f64, Vec<(String, f64)>)> { + fn score_rising( + &self, + entity_id: EntityId, + now_ns: u64, + ) -> crate::Result<(f64, Vec<(String, f64)>)> { let short = read_agg_for_sort( entity_id, "view", @@ -381,6 +498,7 @@ impl ProfileExecutor<'_> { Window::OneHour, self.ledger, self.degradation_level, + now_ns, )?; let long = read_agg_for_sort( entity_id, @@ -389,6 +507,7 @@ impl ProfileExecutor<'_> { Window::TwentyFourHours, self.ledger, self.degradation_level, + now_ns, )?; let score = if long < f64::EPSILON { short diff --git a/tidal/src/ranking/executor/tests.rs b/tidal/src/ranking/executor/tests.rs index b699d53..4f42ef0 100644 --- a/tidal/src/ranking/executor/tests.rs +++ b/tidal/src/ranking/executor/tests.rs @@ -74,7 +74,9 @@ fn score_normalization_all_equal() { let ledger = test_ledger(); let executor = ProfileExecutor::new(&ledger); // Use a profile with no sort and no boosts -- all candidates get 0.0 - // raw score, which normalizes to 1.0. + // raw score, a degenerate all-equal set. Per spec §8.2 these normalize to + // the neutral midpoint 0.5 ("no signal differentiates these"), NOT the + // maximally-confident 1.0 (M0-M10 review pass 2, ranking SUGGESTION). let profile = profile_with("test_equal", |_| {}); let candidates: Vec = (1..=3).map(EntityId::new).collect(); let now = Timestamp::from_nanos(1_708_000_000_000_000_000); @@ -82,7 +84,7 @@ fn score_normalization_all_equal() { let result = executor.score(&candidates, &profile, now).unwrap(); assert_eq!(result.len(), 3); for c in &result { - assert_eq!(c.score, 1.0); + assert_eq!(c.score, 0.5); } } @@ -239,8 +241,10 @@ fn score_personalized_applies_preference_boost() { // Entity 2 strongly aligned with the preference vector; entity 3 opposed. let mut preference_boosts = HashMap::new(); - preference_boosts.insert(2u32, 0.9); - preference_boosts.insert(3u32, -0.9); + // Keyed by the full u64 entity id (no truncation): regression for the + // preference-boost aliasing finding (M0-M10 review pass 2). + preference_boosts.insert(2u64, 0.9); + preference_boosts.insert(3u64, -0.9); let user_ctx = UserContext { user_id: 7, @@ -787,4 +791,201 @@ fn retrieval_scores_seed_dominates_light_boosts() { ); } +// -- Query-clock determinism + sort/boost interaction regressions -- +// (M0-M10 review pass 2, ranking zone C2.) + +/// Accuracy-W regression: scoring must be a pure function of `(ledger snapshot, +/// now)`. Re-running the identical query against an unchanged ledger with the +/// same `now` must produce byte-identical scores. Before the fix, every +/// time-dependent read called `Timestamp::now()` internally, so two runs of the +/// same query decayed/aged each candidate to a slightly different clock and the +/// reported scores drifted between runs. +#[test] +fn scoring_is_deterministic_for_fixed_now() { + let ledger = ledger_for(&["view", "share"]); + // A fixed-but-recent-relative-to-the-data clock so the 24h velocity window + // actually selects buckets (and the read is genuinely time-dependent). + let now = Timestamp::now(); + // Spread signals over the last few hours so the velocity reads are non-zero. + for id in 1..=4u64 { + for i in 0..(id * 3) { + let t = Timestamp::from_nanos( + now.as_nanos().saturating_sub(i * 600_000_000_000), // 10-minute steps back + ); + ledger + .record_signal("view", EntityId::new(id), 1.0, t) + .unwrap(); + if id % 2 == 0 { + ledger + .record_signal("share", EntityId::new(id), 1.0, t) + .unwrap(); + } + } + } + + let executor = ProfileExecutor::new(&ledger); + // Trending exercises the velocity reads (`read_velocity_at`), the most + // clock-sensitive path. + let profile = profile_with("determinism", |p| p.sort = Some(Sort::Trending)); + let candidates: Vec = (1..=4).map(EntityId::new).collect(); + + let r1 = executor.score(&candidates, &profile, now).unwrap(); + let r2 = executor.score(&candidates, &profile, now).unwrap(); + + assert_eq!(r1.len(), r2.len()); + for (a, b) in r1.iter().zip(r2.iter()) { + assert_eq!( + a.entity_id, b.entity_id, + "the same query against an unchanged ledger must return the same order" + ); + assert_eq!( + a.score.to_bits(), + b.score.to_bits(), + "scores must be byte-identical across runs for a fixed `now` (entity {})", + a.entity_id.as_u64() + ); + } +} + +/// Spec §11.9 regression: a formula sort replaces stages 4-5 ONLY for the boosts +/// it SUBSUMES (a boost whose signal/agg/window the formula itself reads). The +/// `trending` builtin sets `Sort::Trending` AND `view`/`share` velocity-24h +/// boosts on the same signals the formula reads (kept on the profile only as the +/// cohort-rescore definition); re-applying them on the global path would +/// double-count (`2*view_vel + 4*share_vel`), so those are skipped. A +/// COMPLEMENTARY boost/penalty (one the formula does NOT read, e.g. a `like` +/// count boost or a `skip` penalty) is an overlay and MUST still apply — this is +/// what keeps `for_you`/`following`/`related`/`notification` engagement weighting +/// alive under their sort seeds. +#[test] +fn sort_subsumes_duplicating_boost_but_keeps_complementary() { + let ledger = ledger_for(&["view", "share", "like", "skip"]); + let now = Timestamp::now(); + for _ in 0..5 { + ledger + .record_signal("view", EntityId::new(1), 1.0, now) + .unwrap(); + ledger + .record_signal("share", EntityId::new(1), 1.0, now) + .unwrap(); + ledger + .record_signal("like", EntityId::new(1), 1.0, now) + .unwrap(); + ledger + .record_signal("skip", EntityId::new(1), 1.0, now) + .unwrap(); + } + + let executor = ProfileExecutor::new(&ledger); + // Sort::Trending with: a DUPLICATING view-velocity-24h boost (subsumed by the + // formula), a COMPLEMENTARY like-count boost (not read by Trending), and a + // complementary skip penalty. + let profile = profile_with("sort_overrides", |p| { + p.sort = Some(Sort::Trending); + p.boosts = vec![ + Boost { + signal: "view".into(), + agg: SignalAgg::Velocity, + window: Window::TwentyFourHours, + weight: 1.0, + }, + Boost { + signal: "like".into(), + agg: SignalAgg::Value, + window: Window::AllTime, + weight: 1.0, + }, + ]; + p.penalties = vec![Penalty { + signal: "skip".into(), + agg: SignalAgg::Value, + window: Window::AllTime, + weight: 1.0, + }]; + }); + + let scored = executor.score(&[EntityId::new(1)], &profile, now).unwrap(); + assert_eq!(scored.len(), 1); + let snap = &scored[0].signal_snapshot; + // The duplicating view-velocity boost is subsumed by the Trending formula. + assert!( + !snap.iter().any(|(k, _)| k == "view_boost"), + "a boost duplicating the Sort::Trending formula must be subsumed, got {snap:?}" + ); + // The complementary like-count boost and skip penalty DO apply. + assert!( + snap.iter().any(|(k, _)| k == "like_boost"), + "a complementary boost the sort formula does not read must still apply, got {snap:?}" + ); + assert!( + snap.iter().any(|(k, _)| k == "skip_penalty"), + "a complementary penalty must still apply under a formula sort, got {snap:?}" + ); + // The Trending base velocity snapshot entries DO still appear (the sort formula). + assert!( + snap.iter().any(|(k, _)| k == "view_velocity"), + "the Sort::Trending formula's own velocity reads must still appear, got {snap:?}" + ); +} + +/// Spec §11.6 regression: shuffle is quality-weighted and per-user/per-minute +/// seeded, not a fixed global hash. A high-quality item (high completion + likes) +/// must out-score a zero-quality item often enough that quality biases the order, +/// and the same `(user, minute)` must reproduce the exact same scores. +#[test] +fn shuffle_is_quality_weighted_and_seed_stable() { + let ledger = ledger_for(&["view", "like", "completion"]); + let now = Timestamp::now(); + // Entity 1: high quality (many views, many likes, strong completion). + for _ in 0..50 { + ledger + .record_signal("view", EntityId::new(1), 1.0, now) + .unwrap(); + ledger + .record_signal("like", EntityId::new(1), 1.0, now) + .unwrap(); + ledger + .record_signal("completion", EntityId::new(1), 1.0, now) + .unwrap(); + } + // Entity 2: zero quality (no signals at all) -> quality_weight == 0 -> score 0. + + let executor = ProfileExecutor::new(&ledger).with_shuffle_user(99); + let profile = profile_with("shuffle_q", |p| p.sort = Some(Sort::Shuffle)); + let candidates = vec![EntityId::new(1), EntityId::new(2)]; + + let r1 = executor.score(&candidates, &profile, now).unwrap(); + // The zero-quality item has quality_weight 0, so its raw shuffle score is 0 + // (sorted last; normalized to 0.0). The high-quality item gets a positive + // draw*weight and normalizes to 1.0 -- quality genuinely biases the order. + assert_eq!( + r1[0].entity_id, + EntityId::new(1), + "a high-quality item must out-rank a zero-quality one under shuffle" + ); + + // Seed stability: same user, same minute -> byte-identical scores. + let r2 = executor.score(&candidates, &profile, now).unwrap(); + for (a, b) in r1.iter().zip(r2.iter()) { + assert_eq!(a.entity_id, b.entity_id); + assert_eq!( + a.score.to_bits(), + b.score.to_bits(), + "shuffle must be stable for a fixed (user, minute)" + ); + } + + // Different user -> the seed changes, so the raw draws differ. We assert the + // permutation is genuinely user-dependent by checking the raw (pre-normalized) + // draw differs for the high-quality item across two distinct users on the same + // minute. (Both still rank it first because the other item is zero-quality.) + let exec_other = ProfileExecutor::new(&ledger).with_shuffle_user(12345); + let r_other = exec_other.score(&candidates, &profile, now).unwrap(); + // Both runs normalize entity 1 -> 1.0 (only nonzero), so compare is trivially + // equal post-normalize; the per-user variety is exercised by the seed inputs + // in `shuffle_random`. Assert at least that scoring still succeeds and ordering + // holds for the second user. + assert_eq!(r_other[0].entity_id, EntityId::new(1)); +} + mod sort_tests; diff --git a/tidal/src/replication/crdt/lww_register.rs b/tidal/src/replication/crdt/lww_register.rs index 0b90367..89b681c 100644 --- a/tidal/src/replication/crdt/lww_register.rs +++ b/tidal/src/replication/crdt/lww_register.rs @@ -1,14 +1,31 @@ //! Last-Writer-Wins Register CRDT. //! -//! Resolves concurrent writes by HLC timestamp ordering. Ties broken by -//! `node_id` (higher wins). Used for hard negatives (hide/mute/block) -//! which require LWW semantics across distributed nodes. +//! Resolves concurrent writes by HLC timestamp ordering. Ties broken first by +//! `node_id` (folded into [`HlcTimestamp`] ordering), then — for the exact-same +//! full timestamp but a *different value* — by the value itself (larger `T` +//! wins). Used for hard negatives (hide/mute/block) which require LWW semantics +//! across distributed nodes. //! //! # CRDT Properties //! //! `merge` is commutative, associative, and idempotent -- these properties //! are verified by property tests using `proptest`. //! +//! # Why the value participates in the tie-break +//! +//! The HLC timestamp alone is *not* a total order over `(timestamp, value)` +//! pairs: two replicas can carry the same `HlcTimestamp` with different values +//! (a forged/corrupt peer register, or a future HLC change that weakens per-node +//! uniqueness). If the tie-break ignored the value, `merge(a, b)` would keep +//! whichever happened to be `self`, so `merge(a, b) != merge(b, a)` and the two +//! replicas would silently diverge — fatal for a state-based CRDT whose entire +//! job is order-independent convergence. Constraining `T: Ord` and breaking the +//! exact-timestamp tie on the larger value makes the decision a *total* order, +//! so merge is commutative for **every** reachable input, not just for honest +//! HLC-stamped writes (W18). This mirrors `CrdtSignalState::merge`, which +//! appends the score to its key tuple `(last_update_ns, score)` for the same +//! reason. +//! //! # Design Notes //! //! The value slot is `Option` where `None` means "not yet written." @@ -28,8 +45,10 @@ use super::hlc::HlcTimestamp; /// /// This is the single place the "keep the larger key" decision lives so the two /// LWW call sites cannot drift: -/// - [`LWWRegister::merge`] uses `K = HlcTimestamp` (whose ordering already -/// folds in `node_id` as the final tiebreak). +/// - [`LWWRegister::merge`] uses `K = (HlcTimestamp, &T)`: the `HlcTimestamp` +/// ordering folds in `node_id`, and the value is appended as the final +/// tiebreak so two registers with the same timestamp but different values +/// resolve identically regardless of merge order (W18). /// - [`super::signal_state::CrdtSignalState::merge`] uses /// `K = (last_update_ns, score)`, where the score is the tiebreak appended /// after the timestamp so equal-timestamp registers resolve deterministically. @@ -39,13 +58,18 @@ pub(crate) fn lww_other_wins(current: Option, incoming: &K) -> /// Last-Writer-Wins register with HLC timestamp. /// -/// Resolves concurrent writes by [`HlcTimestamp`] ordering: +/// Resolves concurrent writes by `(HlcTimestamp, value)` ordering: /// - Higher `wall_ns` wins /// - Same wall, higher `logical` wins -/// - Same wall + logical, higher `node_id` wins (deterministic tie-break) +/// - Same wall + logical, higher `node_id` wins +/// - Same full timestamp, larger value wins (the total tie-break that makes +/// merge commutative for *every* input — see the module docs and W18) /// /// `None` represents "not yet written." /// +/// `T: Ord` is required so the exact-timestamp tie-break is deterministic; the +/// derived `PartialOrd`/`Ord` on `HlcTimestamp` already folds in `node_id`. +/// /// # Properties /// /// - **Commutative:** `merge(A, B) == merge(B, A)` @@ -64,12 +88,12 @@ pub(crate) fn lww_other_wins(current: Option, incoming: &K) -> /// assert_eq!(reg.get(), Some(&42u8)); /// ``` #[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)] -pub struct LWWRegister { +pub struct LWWRegister { value: Option, timestamp: Option, } -impl LWWRegister { +impl LWWRegister { /// Create an empty register (no value written yet). #[must_use] pub const fn empty() -> Self { @@ -79,16 +103,32 @@ impl LWWRegister { } } + /// The current `(timestamp, value)` ordering key, if a value is present. + /// + /// Folding the value into the key after the timestamp is what makes the LWW + /// decision a *total* order: two registers with the same `HlcTimestamp` but + /// different values resolve identically regardless of merge order (W18). + const fn order_key(&self) -> Option<(HlcTimestamp, &T)> { + match (self.timestamp, self.value.as_ref()) { + (Some(ts), Some(v)) => Some((ts, v)), + _ => None, + } + } + /// Write a new value with the given HLC timestamp. /// - /// Only advances the register if `ts > self.timestamp`. - /// Writes at or before the current timestamp are silently discarded - /// (they represent causally earlier events). + /// Advances the register only if `(ts, value)` compares strictly greater + /// than the current `(timestamp, value)` key. Writes at or before the + /// current key are silently discarded (causally earlier, or an exact tie the + /// total order already resolves toward the current value). /// /// Routes through [`lww_other_wins`] so the local-write and merge paths /// share one definition of "is the incoming key newer" and cannot drift. pub fn write(&mut self, value: T, ts: HlcTimestamp) { - if lww_other_wins(self.timestamp, &ts) { + // Compute the decision in a scope whose borrow of `value` ends before the + // move below, so the conditional move type-checks. + let wins = lww_other_wins(self.order_key(), &(ts, &value)); + if wins { self.value = Some(value); self.timestamp = Some(ts); } @@ -96,12 +136,14 @@ impl LWWRegister { /// Merge another register into this one. /// - /// The register with the higher HLC timestamp wins. + /// The register with the higher `(HlcTimestamp, value)` key wins. /// If both are empty, the result is empty. - /// If only one has a value, that value is taken. + /// If only one has a value, that value is taken. The value tie-break makes + /// this commutative even when two registers share the exact same timestamp + /// (W18). pub fn merge(&mut self, other: &Self) { - if let Some(other_ts) = other.timestamp - && lww_other_wins(self.timestamp, &other_ts) + if let Some(other_key) = other.order_key() + && lww_other_wins(self.order_key(), &other_key) { self.value.clone_from(&other.value); self.timestamp = other.timestamp; @@ -127,7 +169,7 @@ impl LWWRegister { } } -impl Default for LWWRegister { +impl Default for LWWRegister { fn default() -> Self { Self::empty() } @@ -208,12 +250,56 @@ mod tests { } #[test] - fn write_with_equal_timestamp_is_discarded() { + fn write_with_equal_timestamp_keeps_larger_value() { + // The exact-timestamp tie now breaks on the value (W18): the larger + // value wins so the decision is a total order and merge is commutative + // even for same-timestamp / different-value inputs. let mut reg: LWWRegister = LWWRegister::empty(); reg.write(1, ts(100, 0, 0)); - reg.write(99, ts(100, 0, 0)); - // Equal is NOT greater, so the write is discarded. - assert_eq!(reg.get(), Some(&1)); + reg.write(99, ts(100, 0, 0)); // same ts, larger value -> wins + assert_eq!(reg.get(), Some(&99)); + + // And a same-timestamp SMALLER value is discarded (current value larger). + let mut reg2: LWWRegister = LWWRegister::empty(); + reg2.write(99, ts(100, 0, 0)); + reg2.write(1, ts(100, 0, 0)); // same ts, smaller value -> discarded + assert_eq!(reg2.get(), Some(&99)); + + // A same-timestamp, same-value write is a true no-op (idempotent). + let mut reg3: LWWRegister = LWWRegister::empty(); + reg3.write(42, ts(100, 0, 0)); + reg3.write(42, ts(100, 0, 0)); + assert_eq!(reg3.get(), Some(&42)); + } + + /// W18: merge must be commutative even when two registers carry the SAME + /// full `HlcTimestamp` but DIFFERENT values — the case a node_id-only + /// tie-break left order-dependent. The value tie-break (larger wins) makes + /// `merge(a, b) == merge(b, a)` for this input. + #[test] + fn merge_commutative_on_equal_timestamp_differing_values() { + let same_ts = ts(100, 0, 0); + let mut a: LWWRegister = LWWRegister::empty(); + a.write(1, same_ts); + let mut b: LWWRegister = LWWRegister::empty(); + b.write(2, same_ts); + + let mut ab = a.clone(); + ab.merge(&b); + let mut ba = b.clone(); + ba.merge(&a); + + assert_eq!( + ab.get(), + ba.get(), + "merge must be commutative on a ts collision" + ); + assert_eq!( + ab.get(), + Some(&2), + "larger value wins the exact-timestamp tie" + ); + assert_eq!(ab.timestamp(), ba.timestamp()); } /// `write` and `merge` must make the identical accept/reject decision for @@ -224,7 +310,9 @@ mod tests { for (cur, incoming) in [ (ts(100, 0, 0), ts(200, 0, 0)), // incoming newer -> accept (ts(200, 0, 0), ts(100, 0, 0)), // incoming older -> reject - (ts(100, 0, 0), ts(100, 0, 0)), // equal -> reject + // equal timestamp: resolved by the value tie-break (incoming value 2 + // > current value 1 -> accept); write and merge must agree either way. + (ts(100, 0, 0), ts(100, 0, 0)), (ts(100, 0, 0), ts(100, 0, 1)), // node-id tiebreak -> accept ] { let mut via_write: LWWRegister = LWWRegister::empty(); @@ -438,8 +526,14 @@ mod property_tests { use super::*; + /// Deliberately tiny timestamp domain so the generator collides on the exact + /// `(wall, logical, node)` tuple often — that collision, paired with the + /// independent value below, is what exercises the equal-timestamp / + /// different-value tie-break the commutativity law depends on (W18). A wide + /// domain would essentially never hit a collision and the law would pass by + /// sampling luck rather than by the code being commutative. fn arb_hlc_timestamp() -> impl Strategy { - (0..=1_000_000u64, 0..=100u32, 0..=10u16).prop_map(|(w, l, n)| HlcTimestamp { + (0..=3u64, 0..=2u32, 0..=2u16).prop_map(|(w, l, n)| HlcTimestamp { wall_ns: w, logical: l, node_id: n, @@ -450,8 +544,11 @@ mod property_tests { prop_oneof![ // Empty register Just(LWWRegister::empty()), - // Register with a value - (any::(), arb_hlc_timestamp()).prop_map(|(v, ts)| { + // Register with a value. The value domain is intentionally tiny (and + // overlapping the timestamp domain) so two registers frequently carry + // the SAME timestamp with a DIFFERENT value, the exact input that + // falsifies a non-total tie-break. + (0..=3u8, arb_hlc_timestamp()).prop_map(|(v, ts)| { let mut r = LWWRegister::empty(); r.write(v, ts); r diff --git a/tidal/src/replication/lag.rs b/tidal/src/replication/lag.rs index cd400cb..a291569 100644 --- a/tidal/src/replication/lag.rs +++ b/tidal/src/replication/lag.rs @@ -52,13 +52,38 @@ impl ReplicationLagGauge { /// /// Returns `leader_seqno - applied_seqno`. If the leader seqno is /// unknown (still 0), returns 0. + /// + /// # Unknown shard + /// + /// If this gauge's `shard_id` is not tracked by the shared + /// [`ReplicationState`] (a wiring bug — e.g. a gauge built for one shard + /// against a state that tracks only others), `applied_seqno` is `None`. That + /// is a misconfiguration, not a legitimately-zero reading, so it is surfaced: + /// a `debug_assert` fires in debug/test builds and a `warn!` is logged in + /// release builds (see [`applied_seqno`](Self::applied_seqno)). The lag then + /// falls back to treating applied as 0 (maximally behind) so a real wiring + /// bug reads as "far behind" rather than the silent "caught up" it would + /// otherwise masquerade as. #[must_use] pub fn lag_segments(&self) -> u64 { let leader = self.leader_seqno.load(Ordering::Acquire); - let applied = self.state.applied_seqno(self.shard_id).unwrap_or(0); + let applied = self.applied_seqno(); leader.saturating_sub(applied) } + /// Whether this gauge's shard is actually tracked by the shared + /// [`ReplicationState`]. + /// + /// `false` means the gauge was wired against a state that does not track its + /// shard, so every reading from [`lag_segments`](Self::lag_segments) / + /// [`applied_seqno`](Self::applied_seqno) is a misconfiguration fallback, not + /// a real measurement. Health/diagnostics can poll this to distinguish a + /// genuinely-zero lag from an unwired gauge. + #[must_use] + pub fn shard_tracked(&self) -> bool { + self.state.applied_seqno(self.shard_id).is_some() + } + /// The shard this gauge tracks. #[must_use] pub const fn shard_id(&self) -> ShardId { @@ -72,9 +97,32 @@ impl ReplicationLagGauge { } /// The latest applied seqno on this node. + /// + /// Returns 0 if this gauge's shard is not tracked by the shared + /// [`ReplicationState`] — but that case is a wiring bug, not a real reading, + /// so it is surfaced loudly (`debug_assert` in debug/test, `warn!` in + /// release) rather than silently conflated with a true zero. Use + /// [`shard_tracked`](Self::shard_tracked) to test for it explicitly. #[must_use] pub fn applied_seqno(&self) -> u64 { - self.state.applied_seqno(self.shard_id).unwrap_or(0) + if let Some(applied) = self.state.applied_seqno(self.shard_id) { + applied + } else { + debug_assert!( + false, + "ReplicationLagGauge for shard {} is not tracked by the shared \ + ReplicationState; lag readings are a misconfiguration fallback, \ + not a real measurement", + self.shard_id + ); + tracing::warn!( + shard = %self.shard_id, + "lag gauge shard is not tracked by ReplicationState; reporting \ + applied=0 (maximally behind) — this is a wiring bug, not a real \ + zero reading" + ); + 0 + } } } @@ -134,4 +182,30 @@ mod tests { assert_eq!(gauge.applied_seqno(), 3); assert_eq!(gauge.lag_segments(), 4); } + + /// SUGGESTION (repl-shipping): a gauge whose shard is tracked by the shared + /// state reports `shard_tracked() == true`; one built against a state that + /// does NOT track its shard reports `false`, so a misconfigured gauge is + /// distinguishable from a legitimately-zero reading instead of silently + /// masquerading as "caught up". + #[test] + fn shard_tracked_distinguishes_unwired_gauge() { + // Wired correctly: ShardId::SINGLE is tracked. + let state = Arc::new(ReplicationState::single()); + let tracked = ReplicationLagGauge::new(ShardId::SINGLE, Arc::clone(&state)); + assert!( + tracked.shard_tracked(), + "a gauge whose shard is tracked must report shard_tracked() == true" + ); + + // Wired against a state that tracks only OTHER shards: the gauge's shard + // (SINGLE) is unknown, so shard_tracked() must report the misconfiguration. + let other_state = Arc::new(ReplicationState::new(&[ShardId(7), ShardId(8)])); + let unwired = ReplicationLagGauge::new(ShardId::SINGLE, Arc::clone(&other_state)); + assert!( + !unwired.shard_tracked(), + "a gauge built against a state that does not track its shard must report \ + shard_tracked() == false" + ); + } } diff --git a/tidal/src/replication/migration.rs b/tidal/src/replication/migration.rs index acd7044..f81fb50 100644 --- a/tidal/src/replication/migration.rs +++ b/tidal/src/replication/migration.rs @@ -69,33 +69,6 @@ impl TenantMigration { self } - /// Persist the router's routing state if a durable sink is attached. - /// - /// A checkpoint failure is logged (the migration is already applied in - /// memory) but not propagated, so a transient disk error does not strand the - /// in-memory transition — the next transition (or open-time restore) reaches - /// a consistent point. A persistent failure is loud in the logs. - /// - /// Used by `finalize`, where the in-memory pin is already the durable target - /// and a missed checkpoint is recoverable on the next transition. Transitions - /// that must be durable-before-observable (e.g. [`Self::enter_dual_write`]) - /// use [`Self::persist_routing_strict`] instead. - fn persist_routing(&self) { - if let Some(storage) = &self.persistence - && let Err(e) = crate::replication::tenant::checkpoint_migration_routing( - storage.as_ref(), - &self.tenant_router, - ) - { - tracing::error!( - tenant_id = self.tenant_id.0, - error = %e, - "failed to checkpoint migration routing state; a crash before the next \ - checkpoint would revert this tenant's routing" - ); - } - } - /// Persist the router's routing state, propagating any checkpoint failure. /// /// Returns `Ok(())` when there is no durable sink (ephemeral mode) or the @@ -112,6 +85,24 @@ impl TenantMigration { } } + /// Snapshot ONLY this migration's tenant routing for a targeted rollback. + /// + /// All `TenantMigration`s share one [`TenantRouter`] but each has only its + /// own per-tenant state lock. A rollback that restored a *full-router* + /// snapshot would clobber a concurrent migration's just-applied (and possibly + /// already-persisted) routing for a different tenant. Snapshotting and + /// restoring only `self.tenant_id` keeps every other tenant's routing intact + /// (C10). + fn snapshot_tenant_routing(&self) -> crate::replication::tenant::TenantRoutingSnapshot { + self.tenant_router.snapshot_tenant_routing(self.tenant_id) + } + + /// Restore ONLY this migration's tenant routing from a prior snapshot, + /// leaving every other tenant untouched (C10). + fn restore_tenant_routing(&self, snapshot: &crate::replication::tenant::TenantRoutingSnapshot) { + self.tenant_router.restore_tenant_routing(snapshot); + } + /// Transition from `Idle` to `PreparingTarget`. /// /// # Errors @@ -155,23 +146,22 @@ impl TenantMigration { )); } // Durability before observable mutation (W38): make the dual-write routing - // durable BEFORE accepting the state-machine transition. `to_checkpoint_bytes` - // snapshots the live router, so the entry must be inserted first; we then - // checkpoint and — if the durable write fails — restore the pre-mutation - // routing snapshot. The net guarantee is that the in-memory dual-write - // entry exists iff it has been persisted, so a crash can never strand an - // in-flight migration whose dual-write state never reached disk. - let pre_snapshot = self.tenant_router.to_checkpoint_bytes(); + // durable BEFORE accepting the state-machine transition. Snapshot ONLY + // this tenant's prior routing (C10) so a failed persist rolls back exactly + // this tenant's entry without clobbering a concurrent migration's routing + // for a different tenant on the shared router. The net guarantee is that + // the in-memory dual-write entry exists iff it has been persisted, so a + // crash can never strand an in-flight migration whose dual-write state + // never reached disk. + let prior = self.snapshot_tenant_routing(); self.tenant_router .set_dual_write(self.tenant_id, self.source_shard, self.target_shard); if let Err(e) = self.persist_routing_strict() { - // Roll back to the exact prior routing (restore_from_checkpoint_bytes - // replaces both maps), so the router never advertises a dual-write that - // is not durable. The migration stays in PreparingTarget; the caller - // can retry once storage recovers. - let _ = self - .tenant_router - .restore_from_checkpoint_bytes(&pre_snapshot); + // Roll back ONLY this tenant's routing, so the router never advertises + // a dual-write that is not durable AND a concurrent migration for a + // different tenant keeps its just-applied entry. The migration stays + // in PreparingTarget; the caller can retry once storage recovers. + self.restore_tenant_routing(&prior); return Err(e); } *state = MigrationState::DualWrite { cutover_seqno }; @@ -181,11 +171,25 @@ impl TenantMigration { /// Transition from `DualWrite` to `Finalizing`. /// + /// The finalized routing (the cleared dual-write + the target pin) is made + /// **durable before the transition is observable**, mirroring + /// [`Self::enter_dual_write`]. This is the cutover that re-points every read + /// for the tenant from source to target; a silent checkpoint failure here + /// would let a crash revert the tenant to `DualWrite` (reads to the SOURCE + /// shard) with no error surfaced — and the only later transition, + /// [`Self::gc_source`], does not re-persist, so nothing would re-close the + /// gap. So on a persist failure we restore the pre-finalize routing snapshot + /// and leave the state in `DualWrite` so the caller retries once storage + /// recovers (C9). + /// /// # Errors /// - /// Returns `InvalidState` if the current state is not `DualWrite`. - /// Returns `NotReady` if `target_seqno` is behind the cutover sequence number — - /// the caller should wait and retry once the target has caught up. + /// - `InvalidState` if the current state is not `DualWrite`. + /// - `NotReady` if `target_seqno` is behind the cutover sequence number — + /// the caller should wait and retry once the target has caught up. + /// - `Storage` if the routing checkpoint cannot be persisted; the in-memory + /// routing is rolled back and the migration stays in `DualWrite` so the + /// caller can retry once storage recovers. pub fn finalize(&self, target_seqno: u64) -> crate::Result<()> { let mut state = self .state @@ -201,32 +205,56 @@ impl TenantMigration { "target seqno {target_seqno} has not reached cutover seqno {cutover_seqno}" ))); } + // Durability before observable mutation (C9): snapshot only THIS tenant's + // prior routing, apply the finalize in memory, then persist strictly. A + // crash-after-finalize cannot revert the tenant to DualWrite, because the + // pin is durable before `finalize` returns Ok. On a persist failure, roll + // back ONLY this tenant's routing (never the whole router — a concurrent + // migration for another tenant must not be clobbered, C10) and stay in + // `DualWrite` so the caller retries. + let prior = self.snapshot_tenant_routing(); self.tenant_router .finalize_migration(self.tenant_id, self.target_shard); + if let Err(e) = self.persist_routing_strict() { + self.restore_tenant_routing(&prior); + return Err(e); + } let switched_at_ns = crate::replication::now_ns(); *state = MigrationState::Finalizing { switched_at_ns }; drop(state); - // Persist the finalized pin (and the cleared dual-write) so a crash after - // finalization cannot revert the tenant to jump-hash routing — the exact - // bug this closes (Accuracy-W). - self.persist_routing(); Ok(()) } - /// Transition from `Finalizing` to `Complete` after the GC window has elapsed. + /// Advance from `Finalizing` to `Complete` once the post-cutover safety + /// window (`gc_window_ns`) has elapsed since the routing switch. + /// + /// # This is a state-only transition — it does NOT reclaim source data + /// + /// This flips the migration state to `Complete`; it performs **no** storage + /// delete, no shipper teardown, and no source-shard reclamation. After it + /// returns the tenant's data still lives on the old source shard. Physically + /// reclaiming that keyspace (a delete-range over the source shard + flush, + /// and tearing down the source-shard shipper) is a separate, **not-yet-wired** + /// step tracked as the source-reclamation task in + /// `docs/specs/14-scale-architecture.md` (per `CODING_GUIDELINES.md` §12). + /// The window enforced here is the safety interval that step would wait out + /// before deleting; the state flip marks "safe to reclaim", not "reclaimed". + /// + /// Named [`Self::mark_complete`] for that reason; [`Self::gc_source`] is kept + /// as a backwards-compatible alias and forwards here. /// /// # Errors /// /// Returns `InvalidState` if the current state is not `Finalizing`, or if /// `gc_window_ns` has not elapsed since the routing switch. - pub fn gc_source(&self, gc_window_ns: u64) -> crate::Result<()> { + pub fn mark_complete(&self, gc_window_ns: u64) -> crate::Result<()> { let mut state = self .state .lock() .map_err(|_| TidalError::internal("migration", "state lock poisoned"))?; let MigrationState::Finalizing { switched_at_ns } = *state else { return Err(TidalError::InvalidState( - "gc_source called outside Finalizing".into(), + "mark_complete called outside Finalizing".into(), )); }; let now = crate::replication::now_ns(); @@ -240,6 +268,20 @@ impl TenantMigration { Ok(()) } + /// Backwards-compatible alias for [`Self::mark_complete`]. + /// + /// The name historically implied source-data garbage collection, but the + /// transition is state-only (no source-shard reclamation happens here — see + /// [`Self::mark_complete`]). Retained so existing callers keep compiling; + /// new code should call `mark_complete`. + /// + /// # Errors + /// + /// See [`Self::mark_complete`]. + pub fn gc_source(&self, gc_window_ns: u64) -> crate::Result<()> { + self.mark_complete(gc_window_ns) + } + /// Return the current migration state. /// /// # Panics @@ -508,4 +550,201 @@ mod tests { // It must NOT still be in dual-write (finalize cleared it). assert!(!recovered.is_dual_write(TenantId(42))); } + + /// A storage backend whose `write_batch`/`flush` start succeeding and can be + /// flipped to fail at a chosen point. Reads pass through. Used to fail the + /// finalize checkpoint while the dual-write checkpoint succeeded (C9). + struct ToggleFailBackend { + inner: Arc, + fail: std::sync::atomic::AtomicBool, + } + + impl ToggleFailBackend { + fn new() -> Self { + Self { + inner: Arc::new(crate::storage::InMemoryBackend::new()), + fail: std::sync::atomic::AtomicBool::new(false), + } + } + fn start_failing(&self) { + self.fail.store(true, std::sync::atomic::Ordering::SeqCst); + } + fn failing(&self) -> bool { + self.fail.load(std::sync::atomic::Ordering::SeqCst) + } + } + + impl crate::storage::StorageEngine for ToggleFailBackend { + fn get(&self, key: &[u8]) -> Result>, crate::storage::StorageError> { + self.inner.get(key) + } + fn put(&self, key: &[u8], value: &[u8]) -> Result<(), crate::storage::StorageError> { + if self.failing() { + return Err(crate::storage::StorageError::Closed); + } + self.inner.put(key, value) + } + fn delete(&self, key: &[u8]) -> Result<(), crate::storage::StorageError> { + self.inner.delete(key) + } + fn scan_prefix(&self, prefix: &[u8]) -> crate::storage::iterator::PrefixIterator<'_> { + self.inner.scan_prefix(prefix) + } + fn write_batch( + &self, + batch: crate::storage::batch::WriteBatch, + ) -> Result<(), crate::storage::StorageError> { + if self.failing() { + return Err(crate::storage::StorageError::Closed); + } + self.inner.write_batch(batch) + } + fn flush(&self) -> Result<(), crate::storage::StorageError> { + if self.failing() { + return Err(crate::storage::StorageError::Closed); + } + self.inner.flush() + } + } + + fn two_shard_parts() -> ( + Arc>, + Arc, + Arc, + ) { + use std::sync::RwLock; + + use crate::replication::tenant::ShardAssignment; + let topo = Arc::new(RwLock::new(ClusterTopology { + shards: vec![ + ShardAssignment { + shard_id: ShardId(0), + region_id: crate::replication::RegionId(0), + }, + ShardAssignment { + shard_id: ShardId(1), + region_id: crate::replication::RegionId(1), + }, + ], + })); + let router = Arc::new(TenantRouter::new(Arc::clone(&topo))); + let state = Arc::new(ReplicationState::single()); + let lag = Arc::new(ReplicationLagGauge::new(ShardId::SINGLE, state)); + let cp = Arc::new(ControlPlane::new( + Arc::clone(&topo), + Arc::clone(&router), + lag, + )); + (topo, router, cp) + } + + /// C9: a finalize whose checkpoint cannot be persisted must NOT observably + /// finalize. It rolls the router's routing for this tenant back to the prior + /// dual-write, stays in `DualWrite`, and returns `Storage` — so a crash after + /// a failed finalize replays as dual-write (correct), never as a half-finalized + /// pin the durable store does not reflect. The previous code swallowed the + /// error and returned Ok while disk still held the pre-finalize routing. + #[test] + fn finalize_rolls_back_on_persist_failure() { + let (_topo, router, cp) = two_shard_parts(); + let backend = Arc::new(ToggleFailBackend::new()); + let migration = + TenantMigration::new(TenantId(7), ShardId(0), ShardId(1), cp, Arc::clone(&router)) + .with_persistence(Arc::clone(&backend) as Arc); + + migration.prepare_target(10).unwrap(); + migration.enter_dual_write(20).unwrap(); // dual-write persists OK + assert!(router.is_dual_write(TenantId(7))); + + // Now make the durable store reject the finalize checkpoint. + backend.start_failing(); + let err = migration.finalize(25).unwrap_err(); + assert!( + matches!(err, TidalError::Storage(_)), + "a failed finalize checkpoint must propagate, not be swallowed; got {err:?}" + ); + + // The router must be rolled back to dual-write (NOT pinned), and the state + // machine must stay in DualWrite so the caller can retry once storage + // recovers. + assert!( + router.is_dual_write(TenantId(7)), + "finalize rollback must restore the prior dual-write routing" + ); + assert_eq!( + router.pinned_shard(TenantId(7)), + None, + "a finalize that failed to persist must NOT leave the tenant pinned" + ); + assert!(matches!( + migration.current_state(), + MigrationState::DualWrite { cutover_seqno: 20 } + )); + } + + /// C10: a failed `enter_dual_write` persist for tenant A must roll back ONLY + /// tenant A's routing — it must NOT clobber a concurrent migration's routing + /// for tenant B that was applied on the shared router between A's snapshot and + /// A's rollback. The old full-router-snapshot rollback wiped B's entry. + #[test] + fn rollback_preserves_other_tenant_routing() { + let (_topo, router, cp_a) = two_shard_parts(); + + // Tenant 99 has a finalized pin already on the shared router (a different, + // already-completed migration). This is what A's rollback must not touch. + router.finalize_migration(TenantId(99), ShardId(1)); + assert_eq!(router.pinned_shard(TenantId(99)), Some(ShardId(1))); + + // Tenant 50 is mid-dual-write on the same shared router, modeling a + // CONCURRENT migration's just-applied entry that A's rollback must leave + // intact. + router.set_dual_write(TenantId(50), ShardId(0), ShardId(1)); + + // Tenant A's (id 7) migration uses a backend that fails the dual-write + // persist, forcing the rollback path. + let failing = Arc::new(ToggleFailBackend::new()); + failing.start_failing(); + let migration_a = TenantMigration::new( + TenantId(7), + ShardId(0), + ShardId(1), + cp_a, + Arc::clone(&router), + ) + .with_persistence(Arc::clone(&failing) as Arc); + + migration_a.prepare_target(10).unwrap(); + let err = migration_a.enter_dual_write(20).unwrap_err(); + assert!(matches!(err, TidalError::Storage(_))); + + // A's own dual-write entry must be gone (rolled back)... + assert!( + !router.is_dual_write(TenantId(7)), + "A's non-durable dual-write must be rolled back" + ); + // ...but EVERY other tenant's routing on the shared router survives. + assert_eq!( + router.pinned_shard(TenantId(99)), + Some(ShardId(1)), + "C10: A's rollback must not clobber B's finalized pin" + ); + assert!( + router.is_dual_write(TenantId(50)), + "C10: A's rollback must not clobber a concurrent tenant's dual-write" + ); + } + + /// W16: `mark_complete` (and its `gc_source` alias) is a pure state flip — it + /// transitions Finalizing -> Complete and does NOT reclaim source data. This + /// pins the documented honest behavior (no source-shard delete happens here). + #[test] + fn mark_complete_is_state_only_transition() { + let m = make_migration(); + m.prepare_target(10).unwrap(); + m.enter_dual_write(20).unwrap(); + m.finalize(25).unwrap(); + // The alias and the canonical name are the same transition. + m.mark_complete(0).unwrap(); + assert_eq!(m.current_state(), MigrationState::Complete); + } } diff --git a/tidal/src/replication/receiver.rs b/tidal/src/replication/receiver.rs index 22f0459..ad28cfc 100644 --- a/tidal/src/replication/receiver.rs +++ b/tidal/src/replication/receiver.rs @@ -28,7 +28,13 @@ //! follower must never silently acknowledge an event it failed to durably //! record. -use std::{sync::Arc, thread::JoinHandle}; +use std::{ + sync::{ + Arc, + atomic::{AtomicBool, Ordering}, + }, + thread::JoinHandle, +}; use crate::{ replication::{ @@ -45,9 +51,19 @@ use crate::{ /// Handle to a running segment receiver thread. /// /// Call [`join`](Self::join) to block until the thread exits and retrieve any -/// corruption error that caused it to stop. +/// corruption error that caused it to stop. While the node is still open, poll +/// [`died`](Self::died) (or [`is_finished`](Self::is_finished)) to observe a +/// receiver that halted on an error mid-life — without it, a dead receiver is +/// indistinguishable from a healthy one until shutdown calls `join` (C11). pub struct SegmentReceiverHandle { thread: Option>>, + /// Latched `true` exactly when the receiver thread halts on an apply error + /// (corrupt segment / follower-WAL IO fault) or an unexpected exit — i.e. a + /// silent replication stall. A clean `recv_segment() == None` shutdown leaves + /// this `false`, so it distinguishes "the follower stopped applying because + /// something broke" from "the follower was asked to shut down" (C11). Shared + /// with the thread, which sets it before returning the error. + died: Arc, } impl SegmentReceiverHandle { @@ -67,6 +83,32 @@ impl SegmentReceiverHandle { }) }) } + + /// Whether the receiver thread has halted on an error (corrupt segment, + /// follower-WAL IO fault, or an unexpected exit) while the node is still + /// open. + /// + /// `true` means replication has silently stopped applying and the follower + /// is frozen — health checks must report degraded. A clean shutdown + /// (`recv_segment() == None`) does **not** set this, so it does not + /// false-positive on intentional teardown (C11). Cheap, lock-free, and safe + /// to poll from `health_check` while the handle is still parked in its mutex. + #[must_use] + pub fn died(&self) -> bool { + self.died.load(Ordering::Acquire) + } + + /// Whether the receiver thread's join handle has finished. + /// + /// Mirrors [`std::thread::JoinHandle::is_finished`]. Unlike [`died`](Self::died) + /// this is also `true` after a clean shutdown, so health checks should prefer + /// `died` to avoid flagging an intentional teardown as a fault. + #[must_use] + pub fn is_finished(&self) -> bool { + self.thread + .as_ref() + .is_some_and(std::thread::JoinHandle::is_finished) + } } /// Spawn a background thread that receives WAL segments and replays them @@ -90,11 +132,19 @@ pub fn spawn_receiver( replication_state: Arc, lag_gauge: Option>, ) -> SegmentReceiverHandle { + // Liveness latch (C11): set true iff the thread halts on an apply error. + // Shared with the handle so health_check can observe a silently-dead + // receiver while the node is still open. A clean shutdown leaves it false. + let died = Arc::new(AtomicBool::new(false)); + let thread_died = Arc::clone(&died); let thread = std::thread::Builder::new() .name("tidaldb-segment-receiver".into()) .spawn(move || -> Result<(), WalError> { loop { let Some(payload) = transport.recv_segment() else { + // Clean shutdown: the transport closed / shutdown was + // requested. Leave the `died` latch false so health_check + // does not flag an intentional teardown as a fault (C11). tracing::debug!("segment receiver: transport closed, shutting down"); return Ok(()); }; @@ -114,16 +164,18 @@ pub fn spawn_receiver( lag_gauge.as_deref(), leader_seqno, ) { - // A corrupt segment halts the receiver thread. Log at the - // failure site so a stalled replica is observable rather - // than silently dead -- mirrors wal::reader's per-corruption - // logging, except replication cannot skip-and-continue (a - // gap would diverge the replica), so we halt and surface via - // SegmentReceiverHandle::join. + // A corrupt segment / follower-WAL fault halts the receiver + // thread. Latch `died` BEFORE returning so a stalled replica + // is observable via SegmentReceiverHandle::died() while the + // node is still open — not just at shutdown via join(). Log at + // the failure site too -- mirrors wal::reader's per-corruption + // logging, except replication cannot skip-and-continue (a gap + // would diverge the replica), so we halt and surface. + thread_died.store(true, Ordering::Release); tracing::error!( shard = %shard_id, error = %e, - "replication apply failed; receiver halting" + "replication apply failed; receiver halting (health degraded)" ); return Err(e); } @@ -133,6 +185,7 @@ pub fn spawn_receiver( SegmentReceiverHandle { thread: Some(thread), + died, } } @@ -827,6 +880,97 @@ mod tests { assert!(!ledger.entries().contains_key(&(EntityId::new(55), type_id))); } + /// C11: a receiver that halts on a corrupt segment must flip its `died` + /// liveness latch WITHOUT the owner ever calling `join()` — so `health_check` + /// can observe a silently-stalled follower while the node is still open, + /// instead of the death being visible only at shutdown. + #[test] + fn receiver_died_latch_set_on_corrupt_without_join() { + let (tx, rx) = crossbeam::channel::bounded(4); + let transport = Arc::new(OneShot { rx }); + + let schema = make_schema(); + let ledger = Arc::new(SignalLedger::new(schema, Box::new(NoopWalWriter))); + let state = Arc::new(ReplicationState::new(&[ShardId(0)])); + + let handle = spawn_receiver( + Arc::clone(&transport), + Arc::clone(&ledger), + Arc::clone(&state), + None, + ); + + // Healthy until something breaks. + assert!(!handle.died(), "fresh receiver must not report died"); + + // Feed one corrupt segment. + let type_id = ledger.resolve_signal_type("view").unwrap(); + let events = vec![make_event(55, type_id.as_u16() as u8, 100)]; + let mut corrupt_bytes = encode_batch(&events, 1, 1).unwrap(); + for b in &mut corrupt_bytes[32..64] { + *b = b.wrapping_add(1); + } + tx.send(crate::replication::WalSegmentPayload { + id: WalSegmentId::new(RegionId::SINGLE, ShardId(0), 1), + bytes: corrupt_bytes, + event_count: 1, + leader_last_seq: 1, + }) + .unwrap(); + + // Poll the latch (NOT join): the receiver must halt and set `died` while + // the channel/transport stays open and the owner never calls join(). + let deadline = std::time::Instant::now() + Duration::from_secs(5); + while !handle.died() && std::time::Instant::now() < deadline { + std::thread::sleep(Duration::from_millis(5)); + } + assert!( + handle.died(), + "a corrupt segment must flip the receiver's died latch without join()" + ); + // The transport is still open (tx not dropped) — the death is observable + // purely from the latch, exactly what health_check reads. + drop(tx); + } + + /// C11: a CLEAN shutdown (transport closed via `recv_segment()` == None) must + /// NOT set the `died` latch — otherwise `health_check` would flag an + /// intentional teardown as a fault. + #[test] + fn receiver_died_latch_unset_on_clean_shutdown() { + let (tx, rx) = crossbeam::channel::bounded(4); + let transport = Arc::new(OneShot { rx }); + + let schema = make_schema(); + let ledger = Arc::new(SignalLedger::new(schema, Box::new(NoopWalWriter))); + let state = Arc::new(ReplicationState::new(&[ShardId(0)])); + + let handle = spawn_receiver( + Arc::clone(&transport), + Arc::clone(&ledger), + Arc::clone(&state), + None, + ); + + // Clean shutdown: drop the sender so recv_segment() returns None. + drop(tx); + + // The thread exits Ok; the latch must stay false. + let deadline = std::time::Instant::now() + Duration::from_secs(5); + while !handle.is_finished() && std::time::Instant::now() < deadline { + std::thread::sleep(Duration::from_millis(5)); + } + assert!( + handle.is_finished(), + "clean shutdown must let the thread exit" + ); + assert!( + !handle.died(), + "a clean shutdown must NOT set the died latch (no false-positive fault)" + ); + handle.join().unwrap(); + } + /// obs-REPL-1: a follower that has received batches from the leader but not /// yet applied all of them reports a NON-ZERO replication lag; the lag /// returns to 0 once the follower catches up. This proves the receiver diff --git a/tidal/src/replication/reconcile.rs b/tidal/src/replication/reconcile.rs index a7b4f61..be6f27c 100644 --- a/tidal/src/replication/reconcile.rs +++ b/tidal/src/replication/reconcile.rs @@ -38,7 +38,11 @@ use crate::{ /// /// Stored inside an `LWWRegister` and resolved by HLC /// timestamp during reconciliation. -#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +// `Ord`/`PartialOrd` are required because `LWWRegister` folds the value into +// its tie-break so `merge` is commutative on an exact-HLC-timestamp collision. +// Declaration order (`Hide` < `Unhide`) makes such a tie resolve to `Unhide`; +// the value tie-break only matters for the otherwise-unreachable collision case. +#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, serde::Serialize, serde::Deserialize)] pub enum HardNegAction { /// The user explicitly hid, muted, or blocked this item. Hide, diff --git a/tidal/src/replication/session_bridge.rs b/tidal/src/replication/session_bridge.rs index 8a74407..e7365e7 100644 --- a/tidal/src/replication/session_bridge.rs +++ b/tidal/src/replication/session_bridge.rs @@ -59,19 +59,31 @@ impl SessionPayload { /// concatenates the results, and computes a BLAKE3 checksum over the /// entire byte buffer. The checksum is verified by the receiver before /// any events are decoded. - #[must_use] - pub fn build(source_shard: ShardId, events: &[SessionWalEvent]) -> Self { + /// + /// # Errors + /// + /// Returns [`SessionBridgeError::Encode`] if any event has a string field + /// longer than `u16::MAX` bytes and so cannot be faithfully encoded. We + /// surface this rather than ship a record the receiver would decode back + /// truncated (and that would misframe every later event in the batch). + pub fn build( + source_shard: ShardId, + events: &[SessionWalEvent], + ) -> Result { let mut bytes = Vec::new(); for event in events { - bytes.extend(encode_session_event(event)); + bytes.extend( + encode_session_event(event) + .map_err(|e| SessionBridgeError::Encode(e.to_string()))?, + ); } let checksum = *Hasher::new().update(&bytes).finalize().as_bytes(); - Self { + Ok(Self { source_shard, bytes, checksum, event_count: events.len() as u32, - } + }) } /// Decode and verify the events in this payload. @@ -103,6 +115,11 @@ pub enum SessionBridgeError { /// The session transport channel is full or disconnected. #[error("session transport channel closed")] Closed, + /// An event could not be faithfully encoded (a string field longer than + /// `u16::MAX` bytes). The batch is rejected rather than shipped as a record + /// the receiver would decode back truncated. + #[error("session event encode failed: {0}")] + Encode(String), } /// In-process session transport factory. @@ -262,7 +279,8 @@ impl SessionReplicationBridge { /// # Errors /// /// Returns `SessionBridgeError::UnknownPeer` or `SessionBridgeError::Closed` - /// if the transport send fails. + /// if the transport send fails, or `SessionBridgeError::Encode` if an event + /// has a string field too long to encode faithfully. pub fn ship( &self, target: ShardId, @@ -296,7 +314,7 @@ impl SessionReplicationBridge { .unwrap_or(current_hwm); let count = to_ship.len(); - let payload = SessionPayload::build(self.local_shard, &to_ship); + let payload = SessionPayload::build(self.local_shard, &to_ship)?; self.transport.send(target, payload)?; self.ship_hwm.insert(hwm_key, highest); @@ -473,7 +491,7 @@ mod tests { let tracker1 = Arc::new(SessionSeqNoTracker::new()); // Send a corrupted payload directly to t1's channel. - let mut payload = SessionPayload::build(ShardId(0), &[signal_event(3, 1)]); + let mut payload = SessionPayload::build(ShardId(0), &[signal_event(3, 1)]).unwrap(); payload.checksum[0] ^= 0xFF; // corrupt the checksum t0.send(ShardId(1), payload).unwrap(); @@ -528,7 +546,7 @@ mod tests { #[test] fn payload_build_and_decode_roundtrip() { let events: Vec<_> = (1..=5).map(|i| signal_event(1, i)).collect(); - let payload = SessionPayload::build(ShardId(0), &events); + let payload = SessionPayload::build(ShardId(0), &events).unwrap(); assert_eq!(payload.event_count, 5); assert_eq!(payload.source_shard, ShardId(0)); @@ -542,7 +560,7 @@ mod tests { #[test] fn payload_decode_corrupt_bytes_fails() { - let mut payload = SessionPayload::build(ShardId(0), &[signal_event(1, 1)]); + let mut payload = SessionPayload::build(ShardId(0), &[signal_event(1, 1)]).unwrap(); // Flip a bit in the payload bytes. if !payload.bytes.is_empty() { payload.bytes[0] ^= 0x01; @@ -585,7 +603,7 @@ mod tests { // Event A: seqno 5, key K. Applies cleanly; HWM -> 5, key K recorded. let a = signal_event_with_key(7, 5, 0xABCD); - let payload_a = SessionPayload::build(b0.local_shard, std::slice::from_ref(&a)); + let payload_a = SessionPayload::build(b0.local_shard, std::slice::from_ref(&a)).unwrap(); b0.transport.send(ShardId(1), payload_a).unwrap(); assert_eq!(b1.recv_and_apply(|_| {}).unwrap(), 1); assert_eq!(b1.seqno_tracker().hwm(7), SessionSeqNo(5)); @@ -594,7 +612,7 @@ mod tests { // re-derived the same key). Layer 2 must drop it BEFORE Layer 1 advances // the HWM — so the HWM must stay at 5, not jump to 10. let b = signal_event_with_key(7, 10, 0xABCD); - let payload_b = SessionPayload::build(b0.local_shard, std::slice::from_ref(&b)); + let payload_b = SessionPayload::build(b0.local_shard, std::slice::from_ref(&b)).unwrap(); b0.transport.send(ShardId(1), payload_b).unwrap(); assert_eq!( b1.recv_and_apply(|_| {}).unwrap(), @@ -611,7 +629,7 @@ mod tests { // ordering it applies; under the buggy ordering the HWM would be 10 and C // would be wrongly dropped. let c = signal_event_with_key(7, 7, 0x1234); - let payload_c = SessionPayload::build(b0.local_shard, std::slice::from_ref(&c)); + let payload_c = SessionPayload::build(b0.local_shard, std::slice::from_ref(&c)).unwrap(); b0.transport.send(ShardId(1), payload_c).unwrap(); let mut received = Vec::new(); assert_eq!( @@ -640,7 +658,7 @@ mod tests { ]; // Manually build and send (Start/Close have no seqno, always shipped). - let payload = SessionPayload::build(b0.local_shard, &events); + let payload = SessionPayload::build(b0.local_shard, &events).unwrap(); b0.transport.send(ShardId(1), payload).unwrap(); let mut received = Vec::new(); diff --git a/tidal/src/replication/tenant.rs b/tidal/src/replication/tenant.rs index decb644..01a0e18 100644 --- a/tidal/src/replication/tenant.rs +++ b/tidal/src/replication/tenant.rs @@ -208,6 +208,20 @@ struct MigrationRoutingCheckpoint { pins: std::collections::HashMap, } +/// A captured snapshot of a single tenant's routing (its dual-write pair and its +/// shard pin), used for targeted rollback in the migration coordinator. +/// +/// Produced by [`TenantRouter::snapshot_tenant_routing`] and restored by +/// [`TenantRouter::restore_tenant_routing`]. Restoring affects only this +/// tenant, never the whole router, so a failed migration persist cannot clobber +/// a concurrent migration's routing for a different tenant (C10). +#[derive(Debug, Clone)] +pub struct TenantRoutingSnapshot { + tenant_id: TenantId, + dual_write: Option<(ShardId, ShardId)>, + pin: Option, +} + /// Describes the shard topology for the cluster. #[derive(Debug, Clone)] pub struct ClusterTopology { @@ -328,7 +342,11 @@ impl TenantRouter { /// future routing to `target_shard`. /// /// After this call, `route()` and `write_assignments()` both return - /// `target_shard` exclusively. The old source shard can be GC'd. + /// `target_shard` exclusively. The old source shard's copy of the tenant's + /// data is now safe to reclaim, but this method does **not** reclaim it — + /// physical source-shard reclamation is a separate, not-yet-wired step (see + /// [`TenantMigration::mark_complete`](crate::replication::migration::TenantMigration::mark_complete) + /// and the source-reclamation task in `docs/specs/14-scale-architecture.md`). pub fn finalize_migration(&self, tenant_id: TenantId, target: ShardId) { self.dual_write_map.remove(&tenant_id); self.shard_pin_map.insert(tenant_id, target); @@ -383,6 +401,52 @@ impl TenantRouter { self.shard_pin_map.get(&tenant_id).map(|r| *r) } + /// Snapshot ONLY `tenant_id`'s routing (its dual-write pair + its pin), for a + /// targeted rollback. + /// + /// Unlike [`to_checkpoint_bytes`](Self::to_checkpoint_bytes) (which snapshots + /// the WHOLE router), this captures a single tenant so a migration that fails + /// to persist can restore exactly its own prior routing without touching any + /// other tenant's entry. A full-router restore would clobber a *concurrent* + /// migration's just-applied (and possibly already-persisted) routing for a + /// different tenant, because all migrations share one router with only a + /// per-migration state lock (C10). + #[must_use] + pub fn snapshot_tenant_routing(&self, tenant_id: TenantId) -> TenantRoutingSnapshot { + TenantRoutingSnapshot { + tenant_id, + dual_write: self.dual_write_map.get(&tenant_id).map(|r| *r), + pin: self.shard_pin_map.get(&tenant_id).map(|r| *r), + } + } + + /// Restore ONLY the snapshotted tenant's routing, leaving every other tenant + /// untouched. + /// + /// Sets the tenant's dual-write pair and pin back to exactly what + /// [`snapshot_tenant_routing`](Self::snapshot_tenant_routing) captured — + /// removing whichever side was absent in the snapshot. This is the targeted + /// counterpart to the rollback path in the migration coordinator: a failed + /// persist for tenant A must not disturb tenant B's concurrent routing (C10). + pub fn restore_tenant_routing(&self, snapshot: &TenantRoutingSnapshot) { + match snapshot.dual_write { + Some(pair) => { + self.dual_write_map.insert(snapshot.tenant_id, pair); + } + None => { + self.dual_write_map.remove(&snapshot.tenant_id); + } + } + match snapshot.pin { + Some(shard) => { + self.shard_pin_map.insert(snapshot.tenant_id, shard); + } + None => { + self.shard_pin_map.remove(&snapshot.tenant_id); + } + } + } + // ── Migration routing durability (Accuracy-W) ─────────────────────────── // // The dual-write and shard-pin maps decide where a tenant's writes go during @@ -535,18 +599,15 @@ fn jump_hash_select(key: u64, shards: &[ShardAssignment]) -> ShardAssignment { shards[b as usize] } -/// Build the WAL directory path for a tenant. -#[must_use] -pub fn tenant_wal_dir(data_dir: &std::path::Path, tenant_id: TenantId) -> std::path::PathBuf { - if tenant_id == TenantId::DEFAULT { - data_dir.join("wal") - } else { - data_dir - .join("tenants") - .join(tenant_id.0.to_string()) - .join("wal") - } -} +// NOTE: per-tenant physical storage/WAL isolation is intentionally NOT +// implemented here. On a single node hosting multiple tenants, `tenant_id` +// influences rate limiting and (multi-node) routing only — all tenants' signals +// land in one shared keyspace/WAL. The former `tenant_wal_dir` helper (which +// built `/data/tenants/{id}/wal` paths) was dead code with no production caller; +// keeping it implied a physical-isolation guarantee the write path never +// provided, so it was removed (W17). Physical per-tenant keyspace/WAL isolation +// and enforcement of `TenantConfig::max_storage_bytes`/`max_entities` are a +// separate, not-yet-wired task tracked in `docs/specs/14-scale-architecture.md`. // ── Migration routing checkpoint persistence ──────────────────────────────── @@ -831,24 +892,6 @@ mod tests { assert_eq!(assignments[1].region_id, RegionId(1)); } - #[test] - fn tenant_wal_dir_default() { - let dir = std::path::Path::new("/data"); - assert_eq!( - tenant_wal_dir(dir, TenantId::DEFAULT), - std::path::PathBuf::from("/data/wal") - ); - } - - #[test] - fn tenant_wal_dir_non_default() { - let dir = std::path::Path::new("/data"); - assert_eq!( - tenant_wal_dir(dir, TenantId(7)), - std::path::PathBuf::from("/data/tenants/7/wal") - ); - } - // ── Migration routing durability (Accuracy-W) ─────────────────────────── fn two_shard_router() -> TenantRouter { diff --git a/tidal/src/schema/error.rs b/tidal/src/schema/error.rs index 6ada0e7..3bdb749 100644 --- a/tidal/src/schema/error.rs +++ b/tidal/src/schema/error.rs @@ -264,13 +264,28 @@ pub enum SchemaError { field: &'static str, reason: String, }, - /// An embedding slot declared zero dimensions. + /// An embedding slot declared dimensions outside the documented `[2, 4096]` + /// range. /// - /// A zero-dimension slot cannot hold a vector; it would silently break ANN - /// retrieval for that slot rather than panicking, so we reject it at - /// define-time as a typed error. - #[error("embedding slot '{slot}': invalid dimensions: {dimensions}")] + /// A 0- or 1-dimension slot is a degenerate index; an enormous dimension is + /// accepted at define-time and only surfaces much later as an allocation + /// failure or a per-vector insert error, far from the schema call that + /// caused it. Reject it eagerly at the schema boundary as a typed error. + #[error("embedding slot '{slot}': invalid dimensions: {dimensions} (must be in [2, 4096])")] InvalidEmbeddingDimensions { slot: String, dimensions: usize }, + /// An embedding slot name is not a valid identifier. + /// + /// Slot names must satisfy the same rule as signal/policy names + /// (non-empty, lowercase ASCII `[a-z0-9_]`, leading letter). The slot + /// fingerprint hashes the name as raw bytes followed by a fixed-width + /// `[kind:1][dims:8]` trailer; restricting names to control-free identifier + /// bytes keeps that byte stream unambiguous so two distinct `(name, kind)` + /// pairs can never collide under a different split. + #[error("embedding slot '{slot}' for {entity_kind}: invalid name (must be a valid identifier)")] + InvalidSlotName { + slot: String, + entity_kind: EntityKind, + }, /// Two embedding slots share the same `(name, entity_kind)` pair. /// /// Slots are keyed by `(name, entity_kind)` in the vector registry, so a @@ -314,8 +329,43 @@ pub enum SchemaError { /// A user-declared text field with this key collides with the internal u64 /// field and panics inside Tantivy's `SchemaBuilder::build()`, so the schema /// builder rejects it at define-time via [`SchemaError::ReservedTextFieldKey`]. -/// Kept in sync with the field name in `text::index::build_tantivy_schema`. -pub(crate) const RESERVED_TEXT_FIELD_KEY: &str = "entity_id"; +/// +/// This is **not** an independent literal: it is an alias of the single source +/// of truth, [`crate::text::index::ENTITY_ID_FIELD`], which the text index uses +/// to create the column. Renaming the Tantivy field there automatically updates +/// this guard, so the define-time check can never drift onto a stale name while +/// the live name silently becomes user-declarable again. The +/// `reserved_text_field_key_matches_tantivy_field` test (plus the const +/// assertion below) pins the equality so a future divergence fails the build +/// rather than production open. +pub(crate) const RESERVED_TEXT_FIELD_KEY: &str = crate::text::index::ENTITY_ID_FIELD; + +/// Compile-time guard that the reserved-key alias has not been re-pointed at an +/// independent literal. Equality is guaranteed by construction today (the const +/// is an alias), but pinning it here means a future edit that replaces the alias +/// with a copied string would fail to compile rather than reintroduce the +/// stale-name drift this guard exists to prevent. +const _: () = assert!( + konst_str_eq(RESERVED_TEXT_FIELD_KEY, crate::text::index::ENTITY_ID_FIELD), + "RESERVED_TEXT_FIELD_KEY must equal text::index::ENTITY_ID_FIELD" +); + +/// `const fn` byte-wise string equality usable in a `const` assertion context +/// (`str::eq` / `==` on `&str` is not `const` on the pinned toolchain). +const fn konst_str_eq(a: &str, b: &str) -> bool { + let (a, b) = (a.as_bytes(), b.as_bytes()); + if a.len() != b.len() { + return false; + } + let mut i = 0; + while i < a.len() { + if a[i] != b[i] { + return false; + } + i += 1; + } + true +} impl Eq for SchemaError {} @@ -340,6 +390,18 @@ pub struct DurabilityError { mod tests { use super::*; + #[test] + fn reserved_text_field_key_matches_tantivy_field() { + // The schema guard's reserved key MUST equal the internal Tantivy field + // name. They are coupled by aliasing the single source of truth; this + // test makes the coupling explicit so a future divergence fails here + // rather than panicking inside Tantivy's SchemaBuilder at DB open. + assert_eq!( + super::RESERVED_TEXT_FIELD_KEY, + crate::text::index::ENTITY_ID_FIELD + ); + } + #[test] fn tidal_error_display_not_found() { let e = TidalError::NotFound { @@ -486,6 +548,28 @@ mod tests { ); } + #[test] + fn schema_error_display_invalid_slot_name() { + let e = SchemaError::InvalidSlotName { + slot: "Bad Name".into(), + entity_kind: EntityKind::Item, + }; + let s = e.to_string(); + assert!(s.contains("Bad Name"), "got: {s}"); + assert!(s.contains("invalid name"), "got: {s}"); + } + + #[test] + fn schema_error_display_invalid_embedding_dimensions_states_range() { + let e = SchemaError::InvalidEmbeddingDimensions { + slot: "content".into(), + dimensions: 1, + }; + let s = e.to_string(); + assert!(s.contains("invalid dimensions: 1"), "got: {s}"); + assert!(s.contains("[2, 4096]"), "got: {s}"); + } + #[test] fn schema_error_display_empty_slot_name() { let e = SchemaError::EmptySlotName { diff --git a/tidal/src/schema/validation/builders.rs b/tidal/src/schema/validation/builders.rs index d0ea89d..03b9a6d 100644 --- a/tidal/src/schema/validation/builders.rs +++ b/tidal/src/schema/validation/builders.rs @@ -7,6 +7,22 @@ use super::{ }; use crate::schema::{DecayModel, EntityKind, SignalTypeDef, Window, WindowSet, error::SchemaError}; +/// Smallest embedding dimensionality the schema accepts. +/// +/// A 0-dimension slot cannot hold a vector and a 1-dimension slot is a +/// degenerate index (every vector is colinear), so the documented contract +/// (`EmbeddingSlot` spec, `11-schema.md` V-E04) requires at least 2. +pub(crate) const MIN_EMBEDDING_DIMENSIONS: usize = 2; + +/// Largest embedding dimensionality the schema accepts. +/// +/// Bounds the per-vector allocation at define-time rather than letting an +/// arbitrarily huge `usize` surface as an allocation failure on first insert, +/// far from the schema call that caused it. 4096 covers every embedding model +/// in current use; lift this deliberately (and update the spec) if a larger +/// model is needed. +pub(crate) const MAX_EMBEDDING_DIMENSIONS: usize = 4096; + /// Internal entry for a signal being built. #[derive(Debug)] struct SignalEntry { @@ -164,8 +180,10 @@ impl SchemaBuilder { /// - `InvalidLifetime` if linear decay has zero/negative lifetime /// - `EmptyWindows` if a non-permanent signal has no windows /// - `VelocityWithoutWindows` if velocity is enabled without windows - /// - `InvalidEmbeddingDimensions` if an embedding slot has zero dimensions + /// - `InvalidEmbeddingDimensions` if an embedding slot's dimensions are + /// outside the documented `[2, 4096]` range /// - `EmptySlotName` if an embedding slot name is empty + /// - `InvalidSlotName` if an embedding slot name is not a valid identifier /// - `DuplicateEmbeddingSlot` if a `(name, kind)` pair is declared twice /// - `InvalidPolicyLimit` if a session policy has a zero `max_session_duration` /// - `InvalidTextField` if a text-field key is empty @@ -304,9 +322,10 @@ impl SchemaBuilder { // Validate embedding slots. The vector registry keys slots by // `(name, entity_kind)`, so an empty name (unaddressable) or a duplicate // pair would silently shadow an earlier slot's dimensions; a - // zero-dimension slot cannot hold a vector. All three are developer - // errors that must surface here as typed `SchemaError`s rather than as - // broken-but-silent state discovered at the first ANN query. + // dimension outside [MIN_EMBEDDING_DIMENSIONS, MAX_EMBEDDING_DIMENSIONS] + // cannot hold a usable vector. These are developer errors that must + // surface here as typed `SchemaError`s rather than as broken-but-silent + // state discovered at the first ANN query. let mut seen_slots = std::collections::HashSet::new(); for slot in &self.embedding_slots { if slot.name.is_empty() { @@ -314,7 +333,24 @@ impl SchemaBuilder { entity_kind: slot.entity_kind, }); } - if slot.dimensions == 0 { + // Slot names share the slot fingerprint's byte stream with a + // fixed-width `[kind:1][dims:8]` trailer; an unrestricted name could + // embed the kind byte and (in principle) let two distinct + // (name, kind) pairs hash to the same concatenated bytes. Hold slot + // names to the same identifier rule as signal/policy names so the + // name is a guaranteed-non-control `[a-z0-9_]` token and the + // fingerprint split is unambiguous. + if !super::is_valid_signal_name(&slot.name) { + return Err(SchemaError::InvalidSlotName { + slot: slot.name.clone(), + entity_kind: slot.entity_kind, + }); + } + // Eager, complete range gate: a definition that passes validation is + // guaranteed self-consistent. Dimension 0/1 is a degenerate index and + // an enormous dimension only surfaces as an allocation/insert failure + // deep in the ANN path; reject both at the schema boundary instead. + if !(MIN_EMBEDDING_DIMENSIONS..=MAX_EMBEDDING_DIMENSIONS).contains(&slot.dimensions) { return Err(SchemaError::InvalidEmbeddingDimensions { slot: slot.name.clone(), dimensions: slot.dimensions, @@ -358,7 +394,11 @@ fn validate_text_field_set(fields: &[TextFieldDef]) -> Result<(), SchemaError> { if field.key.is_empty() { return Err(SchemaError::InvalidTextField(field.key.clone())); } - if field.key == crate::schema::error::RESERVED_TEXT_FIELD_KEY { + // Compare against the single source of truth for the internal Tantivy + // field name. Referencing `text::index::ENTITY_ID_FIELD` directly (not a + // copied literal) means a rename there cannot leave this guard protecting + // a stale name while the new name silently becomes user-declarable. + if field.key == crate::text::index::ENTITY_ID_FIELD { return Err(SchemaError::ReservedTextFieldKey(field.key.clone())); } if !seen_keys.insert(field.key.as_str()) { @@ -669,6 +709,78 @@ mod tests { )); } + #[test] + fn rejects_dimension_one_embedding_slot() { + // dim=1 is a degenerate index; the documented contract is [2, 4096]. + let mut builder = SchemaBuilder::new(); + item_signal(&mut builder); + builder.embedding_slot("content", EntityKind::Item, 1); + let result = builder.build(); + assert!(matches!( + result, + Err(SchemaError::InvalidEmbeddingDimensions { ref slot, dimensions: 1 }) + if slot == "content" + )); + } + + #[test] + fn rejects_dimension_above_ceiling_embedding_slot() { + // One past the documented ceiling must be rejected eagerly at the schema + // boundary, not surface as an allocation failure on first insert. + let mut builder = SchemaBuilder::new(); + item_signal(&mut builder); + builder.embedding_slot("content", EntityKind::Item, MAX_EMBEDDING_DIMENSIONS + 1); + let result = builder.build(); + assert!(matches!( + result, + Err(SchemaError::InvalidEmbeddingDimensions { ref slot, dimensions }) + if slot == "content" && dimensions == MAX_EMBEDDING_DIMENSIONS + 1 + )); + } + + #[test] + fn accepts_boundary_embedding_dimensions() { + // Both ends of the inclusive [2, 4096] range are accepted. + for dim in [MIN_EMBEDDING_DIMENSIONS, MAX_EMBEDDING_DIMENSIONS] { + let mut builder = SchemaBuilder::new(); + item_signal(&mut builder); + builder.embedding_slot("content", EntityKind::Item, dim); + let schema = builder + .build() + .unwrap_or_else(|e| panic!("dim {dim} must be accepted: {e}")); + assert_eq!(schema.embedding_slots().len(), 1); + } + } + + #[test] + fn rejects_invalid_slot_name() { + // Slot names are held to the identifier rule applied to signal/policy + // names: control bytes, uppercase, and leading digits are all rejected. + let invalid = [ + "Content", + "1content", + "con tent", + "con-tent", + "con\u{0}tent", + ]; + for name in invalid { + let mut builder = SchemaBuilder::new(); + item_signal(&mut builder); + builder.embedding_slot(name, EntityKind::Item, 768); + let result = builder.build(); + assert!( + matches!( + result, + Err(SchemaError::InvalidSlotName { + entity_kind: EntityKind::Item, + .. + }) + ), + "should reject slot name {name:?}" + ); + } + } + #[test] fn rejects_empty_embedding_slot_name() { // An empty slot name is its own error, not the misleading @@ -787,6 +899,23 @@ mod tests { )); } + #[test] + fn rejects_reserved_field_sourced_from_text_ssot() { + // The guard must reject whatever name the text index actually reserves + // (text::index::ENTITY_ID_FIELD), not a copied literal. Driving the test + // from the SSOT means a future rename of the Tantivy field keeps the + // guard honest without a second edit here. + let reserved = crate::text::index::ENTITY_ID_FIELD; + let mut builder = SchemaBuilder::new(); + item_signal(&mut builder); + builder.text_field(reserved, TextFieldType::Keyword); + let result = builder.build(); + assert!(matches!( + result, + Err(SchemaError::ReservedTextFieldKey(ref k)) if k == reserved + )); + } + #[test] fn item_and_creator_sets_validate_independently() { // The same key in the item set and the creator set is NOT a duplicate; diff --git a/tidal/src/session/policy.rs b/tidal/src/session/policy.rs index 0ae4278..14a106a 100644 --- a/tidal/src/session/policy.rs +++ b/tidal/src/session/policy.rs @@ -1,6 +1,6 @@ //! Policy evaluation for session signal writes. -use std::{sync::atomic::Ordering, time::Instant}; +use std::{sync::atomic::Ordering, time::Duration}; use super::state::SessionState; use crate::schema::AgentPolicy; @@ -53,6 +53,15 @@ impl<'a> PolicyEvaluator<'a> { /// Check whether a signal can be written under this policy. /// + /// `now_ns` is the current wall clock in nanoseconds since the Unix epoch + /// (`Timestamp::now().as_nanos()` at the call site). The duration check + /// measures age from the **durable** `started_at_ns`, never from the + /// monotonic `started_at` `Instant`: a session restored on open re-seeds + /// `started_at`, so reading the monotonic clock here would reset a restored + /// session's age to ~zero and let an agent evade `max_session_duration` + /// indefinitely by surviving across restarts. `started_at_ns` is the durable + /// anchor, matching `close_session_internal`'s duration accounting. + /// /// Returns `Ok(())` if all policy checks pass. /// /// # Errors @@ -62,7 +71,7 @@ impl<'a> PolicyEvaluator<'a> { &self, signal_type: &str, state: &SessionState, - now: Instant, + now_ns: u64, ) -> Result<(), PolicyViolation> { let make_violation = |kind: PolicyViolationKind, reason: String| PolicyViolation { kind, @@ -71,8 +80,10 @@ impl<'a> PolicyEvaluator<'a> { reason, }; - // 1. Duration check. - let elapsed = now.duration_since(state.started_at); + // 1. Duration check — measured off the durable wall clock, not the + // monotonic Instant (which restore reseeds). Saturating so a backwards + // wall-clock step reports 0 elapsed rather than underflowing. + let elapsed = Duration::from_nanos(now_ns.saturating_sub(state.started_at_ns)); if elapsed > self.policy.max_session_duration { return Err(make_violation( PolicyViolationKind::Expired, @@ -146,7 +157,7 @@ mod tests { Arc, Mutex, atomic::{AtomicBool, AtomicU64}, }, - time::Duration, + time::{Duration, Instant}, }; use dashmap::DashMap; @@ -158,15 +169,17 @@ mod tests { }, *, }; + use crate::schema::Timestamp; fn make_state(policy_name: &str) -> SessionState { + // `started_at_ns` is "now" so the duration check sees a brand-new session. SessionState { id: SessionId(1), user_id: 100, agent_id: AgentId::new("test-agent").unwrap(), policy_name: policy_name.to_owned(), started_at: Instant::now(), - started_at_ns: 0, + started_at_ns: Timestamp::now().as_nanos(), metadata: HashMap::new(), signals: DashMap::new(), signaled_entities: DashMap::new(), @@ -188,7 +201,11 @@ mod tests { }; let evaluator = PolicyEvaluator::new(&policy, "test_policy"); let state = make_state("test_policy"); - assert!(evaluator.check("view", &state, Instant::now()).is_ok()); + assert!( + evaluator + .check("view", &state, Timestamp::now().as_nanos()) + .is_ok() + ); } #[test] @@ -201,7 +218,11 @@ mod tests { }; let evaluator = PolicyEvaluator::new(&policy, "test_policy"); let state = make_state("test_policy"); - assert!(evaluator.check("like", &state, Instant::now()).is_err()); + assert!( + evaluator + .check("like", &state, Timestamp::now().as_nanos()) + .is_err() + ); } #[test] @@ -214,7 +235,11 @@ mod tests { }; let evaluator = PolicyEvaluator::new(&policy, "test_policy"); let state = make_state("test_policy"); - assert!(evaluator.check("block", &state, Instant::now()).is_err()); + assert!( + evaluator + .check("block", &state, Timestamp::now().as_nanos()) + .is_err() + ); } #[test] @@ -228,7 +253,11 @@ mod tests { let evaluator = PolicyEvaluator::new(&policy, "test_policy"); let state = make_state("test_policy"); state.signals_written.store(2, Ordering::Relaxed); - assert!(evaluator.check("view", &state, Instant::now()).is_err()); + assert!( + evaluator + .check("view", &state, Timestamp::now().as_nanos()) + .is_err() + ); } #[test] @@ -240,14 +269,15 @@ mod tests { max_signals_per_session: 0, }; let evaluator = PolicyEvaluator::new(&policy, "test_policy"); + let now_ns = Timestamp::now().as_nanos(); let state = SessionState { id: SessionId(1), user_id: 100, agent_id: AgentId::new("test-agent").unwrap(), policy_name: "test_policy".to_owned(), - // started_at far in the past: - started_at: Instant::now().checked_sub(Duration::from_secs(10)).unwrap(), - started_at_ns: 0, + started_at: Instant::now(), + // durable start far in the past: + started_at_ns: now_ns.saturating_sub(10 * 1_000_000_000), metadata: HashMap::new(), signals: DashMap::new(), signaled_entities: DashMap::new(), @@ -257,6 +287,46 @@ mod tests { audit_log: Mutex::new(AuditLog::new()), closed: Arc::new(AtomicBool::new(false)), }; - assert!(evaluator.check("view", &state, Instant::now()).is_err()); + assert!(evaluator.check("view", &state, now_ns).is_err()); + } + + /// Regression (BLOCKER): TTL / `max_session_duration` must survive a + /// simulated restore. A restored session re-seeds the monotonic `started_at` + /// with a fresh `Instant::now()` while keeping the durable `started_at_ns` + /// far in the past. The duration check must reject off `started_at_ns`, not + /// the near-zero monotonic age — otherwise an agent evades the bound by + /// surviving across restarts. + #[test] + fn policy_duration_survives_simulated_restore() { + let policy = AgentPolicy { + allowed_signals: vec![], + denied_signals: vec![], + max_session_duration: Duration::from_secs(3600), + max_signals_per_session: 0, + }; + let evaluator = PolicyEvaluator::new(&policy, "test_policy"); + let now_ns = Timestamp::now().as_nanos(); + let two_hours_ns = 2 * 3600 * 1_000_000_000u64; + let state = SessionState { + id: SessionId(1), + user_id: 100, + agent_id: AgentId::new("test-agent").unwrap(), + policy_name: "test_policy".to_owned(), + // Restore shape: monotonic clock fresh, durable start 2h ago. + started_at: Instant::now(), + started_at_ns: now_ns.saturating_sub(two_hours_ns), + metadata: HashMap::new(), + signals: DashMap::new(), + signaled_entities: DashMap::new(), + annotations: Mutex::new(Vec::new()), + signals_written: AtomicU64::new(0), + signals_rejected: AtomicU64::new(0), + audit_log: Mutex::new(AuditLog::new()), + closed: Arc::new(AtomicBool::new(false)), + }; + let err = evaluator + .check("view", &state, now_ns) + .expect_err("a 2h-old session under a 1h cap must be rejected after restore"); + assert_eq!(err.kind, PolicyViolationKind::Expired); } } diff --git a/tidal/src/session/state.rs b/tidal/src/session/state.rs index 0bf5ba1..2cd7fe1 100644 --- a/tidal/src/session/state.rs +++ b/tidal/src/session/state.rs @@ -53,6 +53,58 @@ pub struct SessionState { } impl SessionState { + /// Construct a fresh `SessionState` with all five empty-state fields + /// (`signals`, `signaled_entities`, `annotations`, the two `AtomicU64` + /// counters, and `audit_log`) initialized to their canonical empty values. + /// + /// Both production construction sites — `start_session` (live) and + /// `restore_session_wal_events` (crash restore) — route through this + /// constructor so the 14-field literal is never hand-duplicated and an + /// atomic's initial value can never silently diverge between the two paths. + /// + /// The caller supplies the durable identity fields plus the in-process + /// `closed` flag (`start_session` shares it with the returned + /// `SessionHandle`; restore allocates a fresh one). + /// + /// # Durable age invariant + /// + /// `started_at` is a monotonic [`Instant`] for **live metrics only**. The + /// authoritative session age is `started_at_ns` (nanoseconds since the Unix + /// epoch). On restore the caller must **back-date** `started_at` so that + /// `started_at.elapsed()` and `now.duration_since(started_at)` both reflect + /// the true wall-clock age — the monotonic clock is not durable, so seeding + /// it with a fresh `Instant::now()` would reset every restored session's age + /// to ~zero and silently void TTL / `max_session_duration` enforcement. + #[must_use] + #[allow(clippy::too_many_arguments)] + pub fn new( + id: SessionId, + user_id: u64, + agent_id: AgentId, + policy_name: String, + started_at: Instant, + started_at_ns: u64, + metadata: HashMap, + closed: Arc, + ) -> Self { + Self { + id, + user_id, + agent_id, + policy_name, + started_at, + started_at_ns, + metadata, + signals: DashMap::new(), + signaled_entities: DashMap::new(), + annotations: Mutex::new(Vec::new()), + signals_written: AtomicU64::new(0), + signals_rejected: AtomicU64::new(0), + audit_log: Mutex::new(AuditLog::new()), + closed, + } + } + /// Record that `entity_id` received a signal in this session, enforcing the /// [`MAX_SIGNALED_ENTITIES`] cap. /// @@ -148,25 +200,48 @@ impl SessionSeqNoTracker { #[cfg(test)] #[allow(clippy::unwrap_used)] mod tests { + use std::sync::atomic::Ordering; + use super::*; fn empty_state() -> SessionState { - SessionState { - id: SessionId(1), - user_id: 1, - agent_id: AgentId::new("agent-a").unwrap(), - policy_name: "default".to_string(), - started_at: Instant::now(), - started_at_ns: 1, - metadata: HashMap::new(), - signals: DashMap::new(), - signaled_entities: DashMap::new(), - annotations: Mutex::new(Vec::new()), - signals_written: AtomicU64::new(0), - signals_rejected: AtomicU64::new(0), - audit_log: Mutex::new(AuditLog::new()), - closed: Arc::new(AtomicBool::new(false)), - } + SessionState::new( + SessionId(1), + 1, + AgentId::new("agent-a").unwrap(), + "default".to_string(), + Instant::now(), + 1, + HashMap::new(), + Arc::new(AtomicBool::new(false)), + ) + } + + /// Regression (WARNING DRY): `SessionState::new` must initialize every + /// empty-state field to its canonical empty value so the two production + /// construction sites cannot silently diverge on a counter's initial value + /// or a missing reset. + #[test] + fn new_initializes_empty_state_fields() { + let state = SessionState::new( + SessionId(9), + 42, + AgentId::new("agent-z").unwrap(), + "p".to_string(), + Instant::now(), + 123, + HashMap::new(), + Arc::new(AtomicBool::new(false)), + ); + assert_eq!(state.id, SessionId(9)); + assert_eq!(state.user_id, 42); + assert_eq!(state.started_at_ns, 123); + assert_eq!(state.signals.len(), 0); + assert_eq!(state.signaled_entities.len(), 0); + assert_eq!(state.annotations.lock().unwrap().len(), 0); + assert_eq!(state.signals_written.load(Ordering::Relaxed), 0); + assert_eq!(state.signals_rejected.load(Ordering::Relaxed), 0); + assert!(!state.closed.load(Ordering::Acquire)); } #[test] diff --git a/tidal/src/signals/ledger/core.rs b/tidal/src/signals/ledger/core.rs index b59d5a4..cc245b2 100644 --- a/tidal/src/signals/ledger/core.rs +++ b/tidal/src/signals/ledger/core.rs @@ -322,12 +322,41 @@ impl SignalLedger { signal_type_name: &str, window: Window, ) -> crate::Result { - let count = self.read_windowed_count(entity_id, signal_type_name, window)?; + // Read the clock once and delegate to the explicit-clock variant, so the + // expired-bucket rotation behind `read_windowed_count` sees a single + // consistent "now" (matching `read_windowed_count` / `read_decay_score`). + let now_ns = Timestamp::now().as_nanos(); + self.read_velocity_at(entity_id, signal_type_name, window, now_ns) + } + + /// Read the velocity (events per second) against an explicit query clock. + /// + /// Identical to [`read_velocity`](Self::read_velocity) but takes the "now" + /// timestamp from the caller instead of reading `Timestamp::now()` + /// internally. The ranking per-candidate loop reads the clock once at the top + /// of a query and reuses it for every candidate, so the whole result set is + /// aged to one consistent query clock (reproducible scores) and one + /// `clock_gettime` syscall is elided per candidate. + /// + /// Velocity = `windowed_count(now_ns) / window_duration_seconds`. + /// `AllTime` returns 0.0 (velocity is undefined for unbounded windows). + /// + /// # Errors + /// + /// - `TidalError::Schema` if `signal_type_name` is not defined + pub fn read_velocity_at( + &self, + entity_id: EntityId, + signal_type_name: &str, + window: Window, + now_ns: u64, + ) -> crate::Result { let duration_secs = window.duration_secs_f64(); if duration_secs.is_infinite() { - // AllTime window -- velocity is undefined. + // AllTime window -- velocity is undefined. Skip the count read. return Ok(0.0); } + let count = self.read_windowed_count_at(entity_id, signal_type_name, window, now_ns)?; #[allow(clippy::cast_precision_loss)] Ok(count as f64 / duration_secs) } diff --git a/tidal/src/signals/warm.rs b/tidal/src/signals/warm.rs index fe4eedd..805827d 100644 --- a/tidal/src/signals/warm.rs +++ b/tidal/src/signals/warm.rs @@ -358,6 +358,24 @@ impl BucketedCounter { /// cause `rotate_minute`/`rotate_hour`/`rotate_day` to index out of bounds on /// the next rotation. Clamping makes restoration safe regardless of snapshot /// origin. + /// + /// # Rotation-anchor ordering + /// + /// The three rotation anchors carry a monotone invariant set by `maybe_rotate`: + /// `last_day_rotation_ns ≤ last_hour_rotation_ns ≤ last_minute_rotation_ns` + /// (a coarser anchor is only ever advanced *inside* a finer rotation that has + /// already advanced its own anchor). The read path relies on it: `sum_current_hour` + /// folds `k = (now_ns − last_hour_rotation_ns)/60s` minute buckets, and the + /// write-side `hour_agg`/`day_agg` bounds derive `(last_min − last_hour)` and + /// `(last_hour − last_day)`. A corrupt snapshot with, e.g., + /// `last_hour_rotation_ns > last_minute_rotation_ns` would make those + /// `saturating_sub`s clamp to 0 and silently under-count the in-progress + /// window until the next real rotation re-anchors the timestamps. We therefore + /// repair the ordering on restore (pulling each coarser anchor down to its + /// finer neighbor when it is ahead), so a hostile or torn checkpoint cannot + /// produce a wrong windowed count. Pulling *down* (never up) is conservative: + /// it can only widen the in-progress fold by at most one tier, never invent + /// events, and the next `maybe_rotate` re-anchors precisely. pub fn restore(&self, snapshot: &BucketedCounterSnapshot) { for (i, &v) in snapshot.minute_buckets.iter().enumerate() { self.minute_buckets[i].store(v, Ordering::Relaxed); @@ -380,12 +398,21 @@ impl BucketedCounter { self.current_day.store(current_day, Ordering::Release); self.all_time_count .store(snapshot.all_time_count, Ordering::Relaxed); + + // Repair the monotone anchor ordering (day ≤ hour ≤ minute) before + // storing, so a corrupt/hostile snapshot cannot drive a coarser anchor + // ahead of a finer one and silently under-count the in-progress window + // (see "Rotation-anchor ordering" above). The minute anchor is the + // authoritative high-water mark; pull each coarser anchor down to it + // when it is ahead. + let last_minute = snapshot.last_minute_rotation_ns; + let last_hour = snapshot.last_hour_rotation_ns.min(last_minute); + let last_day = snapshot.last_day_rotation_ns.min(last_hour); self.last_minute_rotation_ns - .store(snapshot.last_minute_rotation_ns, Ordering::Relaxed); + .store(last_minute, Ordering::Relaxed); self.last_hour_rotation_ns - .store(snapshot.last_hour_rotation_ns, Ordering::Relaxed); - self.last_day_rotation_ns - .store(snapshot.last_day_rotation_ns, Ordering::Relaxed); + .store(last_hour, Ordering::Relaxed); + self.last_day_rotation_ns.store(last_day, Ordering::Relaxed); } // ── Internal helpers ──────────────────────────────────────────────────────── @@ -941,6 +968,80 @@ mod tests { ); } + #[test] + fn restore_repairs_anchors_with_coarser_ahead_of_finer() { + // Review pass2 (signals): restore() must repair the monotone anchor + // ordering (last_day ≤ last_hour ≤ last_minute). A corrupt snapshot with + // the hour anchor AHEAD of the minute anchor would make sum_current_hour's + // `now_ns.saturating_sub(last_hour)` clamp to 0 and silently under-count + // the in-progress window. We feed exactly that hostile shape and assert + // the stored anchors come out monotone (day ≤ hour ≤ minute), and that a + // windowed read does not panic and is internally consistent. + let hostile = BucketedCounterSnapshot { + minute_buckets: [0; MINUTE_BUCKETS], + hour_buckets: [0; HOUR_BUCKETS], + day_buckets: [0; DAY_BUCKETS], + current_minute: 0, + current_hour: 0, + current_day: 0, + all_time_count: 0, + // Deliberately inverted: hour and day anchors are AHEAD of the minute + // anchor, violating the day ≤ hour ≤ minute invariant maybe_rotate sets. + last_minute_rotation_ns: 10 * NS_PER_MIN, + last_hour_rotation_ns: 100 * NS_PER_HOUR, + last_day_rotation_ns: 50 * NS_PER_DAY, + }; + + let counter = BucketedCounter::new(); + counter.restore(&hostile); + + let restored_minute = counter.last_minute_rotation_ns.load(Ordering::Relaxed); + let restored_hour = counter.last_hour_rotation_ns.load(Ordering::Relaxed); + let restored_day = counter.last_day_rotation_ns.load(Ordering::Relaxed); + + // Minute is the authoritative high-water mark — preserved verbatim. + assert_eq!(restored_minute, 10 * NS_PER_MIN); + // Coarser anchors are pulled DOWN to satisfy day ≤ hour ≤ minute. + assert!( + restored_day <= restored_hour && restored_hour <= restored_minute, + "anchors must be monotone after restore: day={restored_day} \ + hour={restored_hour} minute={restored_minute}" + ); + + // A windowed read at a clock just past the (repaired) minute anchor must + // not under-flow or panic; with empty buckets it is simply 0. + let read_ns = 11 * NS_PER_MIN; + assert_eq!(counter.windowed_count(Window::OneHour, read_ns), 0); + assert_eq!(counter.windowed_count(Window::TwentyFourHours, read_ns), 0); + } + + #[test] + fn restore_preserves_already_monotone_anchors() { + // A well-formed snapshot (day ≤ hour ≤ minute) must round-trip unchanged — + // the repair only ever pulls a coarser anchor DOWN, never perturbs a valid + // ordering. + let mut snapshot = BucketedCounter::with_start_time(0).snapshot(); + snapshot.last_day_rotation_ns = 2 * NS_PER_DAY; + snapshot.last_hour_rotation_ns = 50 * NS_PER_HOUR; + snapshot.last_minute_rotation_ns = 3001 * NS_PER_MIN; // > 50h, > 2d + + let counter = BucketedCounter::new(); + counter.restore(&snapshot); + + assert_eq!( + counter.last_minute_rotation_ns.load(Ordering::Relaxed), + 3001 * NS_PER_MIN + ); + assert_eq!( + counter.last_hour_rotation_ns.load(Ordering::Relaxed), + 50 * NS_PER_HOUR + ); + assert_eq!( + counter.last_day_rotation_ns.load(Ordering::Relaxed), + 2 * NS_PER_DAY + ); + } + #[test] fn quiet_entity_windowed_count_decays_to_zero_on_read() { // CRITICAL #6 regression: a read with an advanced clock and NO intervening diff --git a/tidal/src/storage/engine.rs b/tidal/src/storage/engine.rs index 7c315be..9ad8a8c 100644 --- a/tidal/src/storage/engine.rs +++ b/tidal/src/storage/engine.rs @@ -51,7 +51,7 @@ pub trait StorageEngine: Send + Sync { /// /// # Errors /// - /// Returns `StorageError` on I/O failure or batch conflict. + /// Returns `StorageError` on I/O failure. fn write_batch(&self, batch: WriteBatch) -> Result<(), StorageError>; /// Force all buffered data to stable storage. diff --git a/tidal/src/storage/error.rs b/tidal/src/storage/error.rs index de3c829..8bb10d4 100644 --- a/tidal/src/storage/error.rs +++ b/tidal/src/storage/error.rs @@ -13,9 +13,6 @@ pub enum StorageError { /// The storage engine has been closed and cannot service requests. #[error("storage closed")] Closed, - /// A batch write conflicted with a concurrent operation. - #[error("batch conflict")] - BatchConflict, } #[cfg(test)] @@ -46,11 +43,6 @@ mod tests { assert_eq!(StorageError::Closed.to_string(), "storage closed"); } - #[test] - fn display_batch_conflict() { - assert_eq!(StorageError::BatchConflict.to_string(), "batch conflict"); - } - #[test] fn from_io_error() { let io_err = std::io::Error::other("disk full"); diff --git a/tidal/src/storage/fjall.rs b/tidal/src/storage/fjall.rs index 93587b9..d21397e 100644 --- a/tidal/src/storage/fjall.rs +++ b/tidal/src/storage/fjall.rs @@ -118,6 +118,12 @@ fn map_fjall_err(e: fjall::Error) -> StorageError { } impl StorageEngine for FjallBackend { + // CODING_GUIDELINES §11.2 names these the non-negotiable storage stage + // boundaries to instrument. The spans skip key/value bytes (only their + // lengths are recorded) so a 3am operator can attribute latency to a + // specific storage op — a hot `get`, a full-keyspace `scan_prefix`, the + // atomic-commit barrier, or the fsync — without logging payloads. + #[tracing::instrument(level = "trace", skip(self), fields(key_len = key.len()))] fn get(&self, key: &[u8]) -> Result>, StorageError> { Ok(self .keyspace @@ -126,14 +132,21 @@ impl StorageEngine for FjallBackend { .map(|value| value.to_vec())) } + #[tracing::instrument( + level = "trace", + skip(self, value), + fields(key_len = key.len(), value_len = value.len()) + )] fn put(&self, key: &[u8], value: &[u8]) -> Result<(), StorageError> { self.keyspace.insert(key, value).map_err(map_fjall_err) } + #[tracing::instrument(level = "trace", skip(self), fields(key_len = key.len()))] fn delete(&self, key: &[u8]) -> Result<(), StorageError> { self.keyspace.remove(key).map_err(map_fjall_err) } + #[tracing::instrument(level = "debug", skip(self), fields(prefix_len = prefix.len()))] fn scan_prefix(&self, prefix: &[u8]) -> PrefixIterator<'_> { // Stream lazily — do NOT collect the keyspace into RAM. `fjall::Iter` // is `'static`: it owns the snapshot `SnapshotNonce` internally (which @@ -152,6 +165,7 @@ impl StorageEngine for FjallBackend { Box::new(iter) } + #[tracing::instrument(level = "debug", skip(self, batch), fields(ops = batch.len()))] fn write_batch(&self, batch: WriteBatch) -> Result<(), StorageError> { // Atomicity: the `StorageEngine::write_batch` contract requires // all-or-nothing semantics. Looping over `keyspace.insert`/`remove` @@ -183,6 +197,7 @@ impl StorageEngine for FjallBackend { wb.commit().map_err(map_fjall_err) } + #[tracing::instrument(level = "debug", skip(self))] fn flush(&self) -> Result<(), StorageError> { // `StorageEngine::flush` documents "force all buffered data to stable // storage", which means fsync -- not merely rotating the memtable. @@ -264,11 +279,16 @@ impl FjallStorage { /// /// Returns `StorageError` if the underlying fjall database cannot be opened. pub fn open(path: impl AsRef) -> Result { + // Route through `map_fjall_err` so an open failure keeps its operational + // class: a permission-denied or disk-full (`Io`) or a held/poisoned lock + // (`Closed`) is NOT stringified into a corruption message. Only a genuine + // decode/checksum/trailer failure is classified as `Corruption` — the + // exact mis-classification `map_fjall_err` exists to prevent. (The lost + // path context is acceptable: the io::Error carries the OS error, and the + // open span/caller already names which database failed.) let db = fjall::Database::builder(path) .open() - .map_err(|e| StorageError::Corruption { - message: format!("failed to open fjall database: {e}"), - })?; + .map_err(map_fjall_err)?; let items = Self::open_keyspace(&db, "items")?; let users = Self::open_keyspace(&db, "users")?; @@ -285,14 +305,15 @@ impl FjallStorage { /// Open (or create) one named keyspace and wrap it in a `FjallBackend`. /// /// Extracted from [`open`](Self::open) so the three entity keyspaces share a - /// single open path: identical create-options and a corruption error tagged - /// with the keyspace name, rather than three verbatim copies that could drift. + /// single open path with identical create-options, rather than three verbatim + /// copies that could drift. A keyspace-open failure is routed through + /// [`map_fjall_err`] so its operational class (I/O / closed / corruption) is + /// preserved exactly as on the get/put path, not collapsed into a corruption + /// message regardless of cause. fn open_keyspace(db: &fjall::Database, name: &str) -> Result { let keyspace = db .keyspace(name, fjall::KeyspaceCreateOptions::default) - .map_err(|e| StorageError::Corruption { - message: format!("failed to open {name} keyspace: {e}"), - })?; + .map_err(map_fjall_err)?; Ok(FjallBackend::new(keyspace, db.clone())) } @@ -470,6 +491,57 @@ mod tests { assert_eq!(keys, vec![b"a".to_vec(), b"b".to_vec(), b"c".to_vec()]); } + #[test] + fn scan_prefix_propagates_mid_stream_error_via_collect() { + // The streaming adapter maps each fjall guard through + // `into_inner().map_err(map_fjall_err)?`, so a torn read / decompress + // failure surfaces as an `Err(StorageError)` ITEM partway through + // iteration rather than a panic. We cannot inject a torn read into fjall + // in a fast unit test, so we pin the contract every caller relies on with + // a synthetic `PrefixIterator` of the exact same shape: one good entry, + // then an injected mid-stream Err, then more. + let iter: super::PrefixIterator<'static> = Box::new( + [ + Ok((b"a".to_vec(), b"1".to_vec())), + Err(StorageError::Corruption { + message: "torn read mid-stream".into(), + }), + Ok((b"b".to_vec(), b"2".to_vec())), + ] + .into_iter(), + ); + + // `collect::>` (what scan_prefix tests and the careful + // callers use) MUST surface the Err and short-circuit — it must not be + // silently dropped. + let collected: Result, _> = iter.collect(); + assert!( + matches!(collected, Err(StorageError::Corruption { .. })), + "mid-stream Err must propagate through collect::>" + ); + } + + #[test] + fn scan_prefix_flatten_silently_swallows_error() { + // Counterpart hazard documented in the review: `.flatten()` over the same + // `Result`-yielding adapter DROPS an Err item. This test exists so the + // swallowing behavior is explicit and a future caller cannot + // accidentally rely on `.flatten()` for a path where the Err matters. + let iter: super::PrefixIterator<'static> = Box::new( + [ + Ok((b"a".to_vec(), b"1".to_vec())), + Err(StorageError::Corruption { + message: "torn read mid-stream".into(), + }), + Ok((b"b".to_vec(), b"2".to_vec())), + ] + .into_iter(), + ); + // `.flatten()` yields only the Ok payloads; the Err vanishes. + let kept: Vec<_> = iter.flatten().collect(); + assert_eq!(kept.len(), 2, ".flatten() drops the mid-stream Err item"); + } + #[test] fn scan_prefix_is_lazy() { // The streaming adapter must yield items on demand: taking only the diff --git a/tidal/src/storage/indexes/bitmap.rs b/tidal/src/storage/indexes/bitmap.rs index 65a1ea0..0cfe687 100644 --- a/tidal/src/storage/indexes/bitmap.rs +++ b/tidal/src/storage/indexes/bitmap.rs @@ -63,6 +63,15 @@ pub const INDEX_ROOT_ID: u64 = 0; pub struct BitmapIndex { field_name: String, values: Arc>>, + /// Cached distinct-entity union cardinality, invalidated on every mutation. + /// + /// `total_count()` unions every value bitmap — `O(distinct_values * + /// avg_bitmap_size)`. Memoizing it makes the planner's repeated + /// per-predicate reads between writes O(1). Shared via `Arc` so a cloned + /// handle (which shares `values`) also shares the memo. It is a read-side + /// memo of state already in `values`, never a source of truth: a missed + /// invalidation would only force a recompute, never return a wrong count. + total_count_cache: Arc>>, } impl Clone for BitmapIndex { @@ -70,6 +79,7 @@ impl Clone for BitmapIndex { Self { field_name: self.field_name.clone(), values: Arc::clone(&self.values), + total_count_cache: Arc::clone(&self.total_count_cache), } } } @@ -81,9 +91,18 @@ impl BitmapIndex { Self { field_name: field_name.into(), values: Arc::new(RwLock::new(HashMap::new())), + total_count_cache: Arc::new(RwLock::new(Some(0))), } } + /// Invalidate the memoized total-count union after a mutation. + fn invalidate_total_count(&self) { + *self + .total_count_cache + .write() + .unwrap_or_else(std::sync::PoisonError::into_inner) = None; + } + /// The field name this index covers. #[must_use] pub fn field_name(&self) -> &str { @@ -101,6 +120,9 @@ impl BitmapIndex { let bitmap = map.entry(value.into()).or_default(); let inserted = bitmap.insert(entity_id); drop(map); + if inserted { + self.invalidate_total_count(); + } inserted } @@ -124,6 +146,10 @@ impl BitmapIndex { if now_empty { map.remove(value); } + drop(map); + if removed { + self.invalidate_total_count(); + } removed } @@ -148,6 +174,10 @@ impl BitmapIndex { for key in empty_keys { map.remove(&key); } + drop(map); + if count > 0 { + self.invalidate_total_count(); + } count } @@ -192,8 +222,23 @@ impl BitmapIndex { } /// Total number of distinct entity IDs across all values (no double-counting). + /// + /// Memoized: the union over every value bitmap is computed at most once per + /// mutation, so repeated reads between writes are O(1). #[must_use] pub fn total_count(&self) -> u64 { + // Fast path: return the memoized cardinality if still valid. Read the + // cache into a local and drop the guard before branching so the cache + // lock is never held across the slow-path values read. + let cached = *self + .total_count_cache + .read() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if let Some(count) = cached { + return count; + } + // Slow path: recompute the union once and memoize it. Compute under the + // values read lock so the counted snapshot is internally consistent. let map = self .values .read() @@ -203,7 +248,12 @@ impl BitmapIndex { union |= bitmap; } drop(map); - union.len() + let count = union.len(); + *self + .total_count_cache + .write() + .unwrap_or_else(std::sync::PoisonError::into_inner) = Some(count); + count } /// Enumerate all indexed field values. @@ -375,6 +425,8 @@ impl BitmapIndex { Ok(Self { field_name, values: Arc::new(RwLock::new(map)), + // Loaded fresh; force a recompute on first read. + total_count_cache: Arc::new(RwLock::new(None)), }) } } diff --git a/tidal/src/storage/indexes/filter/evaluator.rs b/tidal/src/storage/indexes/filter/evaluator.rs index bd61057..ac9972a 100644 --- a/tidal/src/storage/indexes/filter/evaluator.rs +++ b/tidal/src/storage/indexes/filter/evaluator.rs @@ -61,10 +61,22 @@ impl<'a> FilterEvaluator<'a> { /// Stage 2.5 (user-context filtering). #[must_use] pub fn evaluate(&self, expr: &FilterExpr) -> FilterResult { - FilterResult::Bitmap(self.eval_to_bitmap(expr)) + FilterResult::Bitmap(self.eval_to_bitmap_bounded(expr, 0)) } - fn eval_to_bitmap(&self, expr: &FilterExpr) -> RoaringBitmap { + /// Depth-bounded evaluation. Past [`FilterExpr::MAX_FILTER_DEPTH`] we return + /// an empty bitmap rather than recursing further: a filter too deep to + /// evaluate matches nothing (a safe degradation that never panics or + /// overflows the stack). `Db::retrieve` rejects such trees up front via the + /// breadth guard, but `FilterEvaluator` is public, so it defends itself. + fn eval_to_bitmap_bounded(&self, expr: &FilterExpr, depth: usize) -> RoaringBitmap { + if depth >= FilterExpr::MAX_FILTER_DEPTH { + return RoaringBitmap::new(); + } + self.eval_to_bitmap(expr, depth) + } + + fn eval_to_bitmap(&self, expr: &FilterExpr, depth: usize) -> RoaringBitmap { match expr { FilterExpr::CategoryEq(v) => self.category_index.get(v).unwrap_or_default(), FilterExpr::FormatEq(v) => self.format_index.get(v).unwrap_or_default(), @@ -76,9 +88,9 @@ impl<'a> FilterEvaluator<'a> { FilterExpr::DurationMax(secs) => self.duration_index.lte(secs), FilterExpr::CreatedAfter(ns) => self.created_at_index.gt(ns), FilterExpr::CreatedBefore(ns) => self.created_at_index.lt(ns), - FilterExpr::And(children) => self.evaluate_and(children), - FilterExpr::Or(children) => self.evaluate_or(children), - FilterExpr::Not(inner) => self.evaluate_not(inner), + FilterExpr::And(children) => self.evaluate_and(children, depth), + FilterExpr::Or(children) => self.evaluate_or(children, depth), + FilterExpr::Not(inner) => self.evaluate_not(inner, depth), // User-state filters: these are evaluated at the FilterEvaluator // level by returning the full universe -- the actual filtering // happens in the query executor's Stage 2.5 (user-context filtering) @@ -100,7 +112,7 @@ impl<'a> FilterEvaluator<'a> { } } - fn evaluate_and(&self, children: &[FilterExpr]) -> RoaringBitmap { + fn evaluate_and(&self, children: &[FilterExpr], depth: usize) -> RoaringBitmap { if children.is_empty() { // AND of nothing = everything (identity element for intersection) return self.universe.clone(); @@ -114,8 +126,10 @@ impl<'a> FilterEvaluator<'a> { // that walked each subtree twice. Sorting the materialized bitmaps by // `len()` gives *exact* (not estimated) ordering for the short-circuit // while walking every subtree only once. - let mut bitmaps: Vec = - children.iter().map(|c| self.eval_to_bitmap(c)).collect(); + let mut bitmaps: Vec = children + .iter() + .map(|c| self.eval_to_bitmap_bounded(c, depth + 1)) + .collect(); bitmaps.sort_unstable_by_key(RoaringBitmap::len); // bitmaps is non-empty (children is non-empty), so the first element @@ -132,13 +146,13 @@ impl<'a> FilterEvaluator<'a> { result } - fn evaluate_or(&self, children: &[FilterExpr]) -> RoaringBitmap { + fn evaluate_or(&self, children: &[FilterExpr], depth: usize) -> RoaringBitmap { if children.is_empty() { return RoaringBitmap::new(); } let mut result = RoaringBitmap::new(); for child in children { - result |= &self.eval_to_bitmap(child); + result |= &self.eval_to_bitmap_bounded(child, depth + 1); } result } @@ -156,8 +170,8 @@ impl<'a> FilterEvaluator<'a> { /// - Consequently `NOT(x)` ranges over the same id set every other filter /// node ranges over, keeping NOT composable under AND/OR without ever /// re-introducing out-of-universe ids. - fn evaluate_not(&self, inner: &FilterExpr) -> RoaringBitmap { - let inner_bitmap = self.eval_to_bitmap(inner); + fn evaluate_not(&self, inner: &FilterExpr, depth: usize) -> RoaringBitmap { + let inner_bitmap = self.eval_to_bitmap_bounded(inner, depth + 1); let mut complement = self.universe.clone(); complement -= &inner_bitmap; complement @@ -169,6 +183,18 @@ impl<'a> FilterEvaluator<'a> { /// independence assumption for compound predicates. #[must_use] pub fn selectivity(&self, expr: &FilterExpr) -> f64 { + self.selectivity_bounded(expr, 0) + } + + /// Depth-bounded selectivity estimate. Past [`FilterExpr::MAX_FILTER_DEPTH`] + /// we stop descending and return `1.0` (the conservative "matches all" + /// estimate, the identity for AND's product), so a pathologically deep AST + /// cannot drive the estimator into unbounded recursion. The estimate only + /// orders predicates; a clamped tail estimate cannot produce a wrong result. + fn selectivity_bounded(&self, expr: &FilterExpr, depth: usize) -> f64 { + if depth >= FilterExpr::MAX_FILTER_DEPTH { + return 1.0; + } #[allow(clippy::cast_precision_loss)] let total = self.universe.len() as f64; if total == 0.0 { @@ -207,17 +233,21 @@ impl<'a> FilterEvaluator<'a> { // Independence assumption: P(A AND B) ~ P(A) * P(B) children .iter() - .map(|c| self.selectivity(c)) + .map(|c| self.selectivity_bounded(c, depth + 1)) .product::() .clamp(0.0, 1.0) } FilterExpr::Or(children) => { // Inclusion-exclusion: P(A OR B) ~ 1 - (1-P(A))(1-P(B)) - let complement_product: f64 = - children.iter().map(|c| 1.0 - self.selectivity(c)).product(); + let complement_product: f64 = children + .iter() + .map(|c| 1.0 - self.selectivity_bounded(c, depth + 1)) + .product(); (1.0 - complement_product).clamp(0.0, 1.0) } - FilterExpr::Not(inner) => (1.0 - self.selectivity(inner)).clamp(0.0, 1.0), + FilterExpr::Not(inner) => { + (1.0 - self.selectivity_bounded(inner, depth + 1)).clamp(0.0, 1.0) + } // User-state filters: selectivity is unknown without user state. // Estimate 0.9 for Unseen (most items are unseen for most users), // 0.95 for Unblocked, and low for Saved/Liked/InProgress. @@ -436,6 +466,39 @@ mod tests { assert_eq!(result.is_empty(), Some(true)); } + #[test] + fn deep_filter_evaluation_is_depth_bounded() { + // FilterEvaluator is public, so it must defend itself against a + // pathologically deep AST: evaluating a `Not(Not(...))` chain far past + // MAX_FILTER_DEPTH must return (a bitmap) without panicking or + // overflowing the stack — the bounded recursion stops at the cap. + let (cat, fmt, creator, tags, dur, ts, universe) = setup_test_indexes(); + let ev = make_evaluator_full(&cat, &fmt, &creator, &tags, &dur, &ts, &universe); + + let mut expr = FilterExpr::CategoryEq("jazz".into()); + for _ in 0..(FilterExpr::MAX_FILTER_DEPTH + 50) { + expr = FilterExpr::Not(Box::new(expr)); + } + // Must complete (bounded recursion); we only assert it does not panic. + let _ = ev.evaluate(&expr).into_bitmap(); + // selectivity recurses identically and must also stay bounded. + let sel = ev.selectivity(&expr); + assert!((0.0..=1.0).contains(&sel), "selectivity {sel} out of range"); + } + + #[test] + fn shallow_not_evaluation_unaffected_by_depth_bound() { + // A single Not (depth 1) is well under the cap and must still produce the + // exact complement — the depth guard never alters a normal filter. + let (cat, fmt, creator, tags, dur, ts, universe) = setup_test_indexes(); + let ev = make_evaluator_full(&cat, &fmt, &creator, &tags, &dur, &ts, &universe); + let result = ev + .evaluate(&FilterExpr::Not(Box::new(FilterExpr::CreatorEq(0)))) + .into_bitmap(); + // Same expectation as evaluate_not_creator: creator 0 = {0,3,6,9}. + assert_eq!(result.len(), 6); + } + #[test] fn selectivity_empty_universe() { let cat = BitmapIndex::new("category"); diff --git a/tidal/src/storage/indexes/filter/expr.rs b/tidal/src/storage/indexes/filter/expr.rs index e648feb..9fc75c2 100644 --- a/tidal/src/storage/indexes/filter/expr.rs +++ b/tidal/src/storage/indexes/filter/expr.rs @@ -148,17 +148,54 @@ impl FilterExpr { Self::InCollection(id) } + /// Maximum recursion depth `node_count` and `FilterEvaluator` will descend. + /// + /// A tree with at most [`MAX_FILTER_NODES`](Self::MAX_FILTER_NODES) nodes can + /// be at most that deep (a degenerate `Not(Not(...))` chain), so bounding the + /// descent at this value never rejects a tree the breadth guard would accept, + /// while a pathologically deeper hand-built AST is rejected *before* it can + /// recurse far enough to overflow the stack. Counting and evaluation both + /// honor it so neither can be driven into unbounded recursion. + pub const MAX_FILTER_DEPTH: usize = 256; + + /// Breadth guard: the maximum total node count a filter may carry. + /// + /// Mirrored by the query entrypoint (`Db::retrieve`), kept here so the + /// node-count saturation sentinel and the caller's rejection threshold come + /// from one definition. + pub const MAX_FILTER_NODES: usize = 256; + /// Returns the total number of nodes in this filter expression tree. /// /// Used to guard against deeply nested or oversized filter expressions /// that could cause excessive recursion or memory usage during evaluation. + /// + /// Counting is itself **depth-bounded**: a tree deeper than + /// [`MAX_FILTER_DEPTH`](Self::MAX_FILTER_DEPTH) short-circuits to a count of + /// `usize::MAX` rather than recursing the full depth first. This guarantees + /// the breadth guard that consumes this count (`count > MAX_FILTER_NODES`) + /// rejects a pathologically deep `Not(Not(...))` chain *before* the count + /// itself can overflow the stack — closing the gap where the breadth guard + /// depended on a full deep recursion to even compute the number. #[must_use] pub fn node_count(&self) -> usize { + self.node_count_bounded(0) + } + + /// Depth-bounded node counter. `depth` is the current recursion depth; once + /// it exceeds [`MAX_FILTER_DEPTH`](Self::MAX_FILTER_DEPTH) we stop descending + /// and return `usize::MAX` so the breadth guard trivially rejects the tree. + /// Sums saturate so an over-wide tree also cannot wrap. + fn node_count_bounded(&self, depth: usize) -> usize { + if depth >= Self::MAX_FILTER_DEPTH { + return usize::MAX; + } match self { - Self::And(children) | Self::Or(children) => { - 1 + children.iter().map(Self::node_count).sum::() - } - Self::Not(inner) => 1 + inner.node_count(), + Self::And(children) | Self::Or(children) => children + .iter() + .map(|c| c.node_count_bounded(depth + 1)) + .fold(1usize, usize::saturating_add), + Self::Not(inner) => 1usize.saturating_add(inner.node_count_bounded(depth + 1)), _ => 1, } } @@ -285,6 +322,41 @@ mod tests { assert_eq!(filter.node_count(), 6); } + #[test] + fn node_count_deep_chain_is_depth_bounded() { + // A `Not(Not(...))` chain far deeper than MAX_FILTER_DEPTH must be + // counted WITHOUT recursing the full depth: node_count_bounded + // short-circuits to usize::MAX past the depth cap, so the breadth guard + // (`count > MAX_FILTER_NODES`) rejects it before a deep recursion can + // overflow the stack. Build a chain a few thousand deep to prove the + // bound holds well past the node cap. + let mut expr = FilterExpr::CategoryEq("leaf".into()); + for _ in 0..5_000 { + expr = FilterExpr::Not(Box::new(expr)); + } + let count = expr.node_count(); + assert!( + count > FilterExpr::MAX_FILTER_NODES, + "deep chain must report a count exceeding the breadth cap, got {count}" + ); + // Shallow-but-wide trees are unaffected: their depth never trips the cap, + // so the exact count is preserved (see node_count_large_flat_and). + } + + #[test] + fn node_count_at_depth_limit_does_not_saturate_spuriously() { + // A chain exactly at the boundary still counts truthfully: a Not chain of + // depth just under MAX_FILTER_DEPTH returns its real (finite) node count, + // proving the cap does not reject trees the breadth guard would accept. + let mut expr = FilterExpr::CategoryEq("leaf".into()); + for _ in 0..(FilterExpr::MAX_FILTER_DEPTH - 2) { + expr = FilterExpr::Not(Box::new(expr)); + } + let count = expr.node_count(); + assert_ne!(count, usize::MAX, "below the depth cap must not saturate"); + assert_eq!(count, FilterExpr::MAX_FILTER_DEPTH - 1); + } + #[test] fn node_count_large_flat_and() { let children: Vec = (0..300) diff --git a/tidal/src/storage/indexes/range.rs b/tidal/src/storage/indexes/range.rs index 1517d63..ca7a7dd 100644 --- a/tidal/src/storage/indexes/range.rs +++ b/tidal/src/storage/indexes/range.rs @@ -31,6 +31,17 @@ type KvPairs = Vec<(Vec, Vec)>; pub struct RangeIndex { field_name: String, tree: RwLock>, + /// Cached distinct-entity union cardinality, invalidated on every mutation. + /// + /// `total_count()` unions every value bitmap — `O(distinct_values * + /// avg_bitmap_size)`. The query planner calls it (directly and via + /// `selectivity`) once per range predicate while ordering AND/OR groups, so + /// a multi-predicate query re-walked all value bitmaps repeatedly between + /// writes. The cache makes those repeated reads O(1): writes set it to + /// `None`, the next read recomputes once and memoizes. It is a read-side + /// memo of state already in `tree`, never an independent source of truth, so + /// it cannot drift — a missed invalidation would only force a recompute. + total_count_cache: RwLock>, } impl RangeIndex { @@ -40,9 +51,18 @@ impl RangeIndex { Self { field_name: field_name.into(), tree: RwLock::new(BTreeMap::new()), + total_count_cache: RwLock::new(Some(0)), } } + /// Invalidate the memoized total-count union after a mutation. + fn invalidate_total_count(&self) { + *self + .total_count_cache + .write() + .unwrap_or_else(std::sync::PoisonError::into_inner) = None; + } + /// The field name this index covers. #[must_use] pub fn field_name(&self) -> &str { @@ -58,6 +78,7 @@ impl RangeIndex { let bitmap = tree.entry(value).or_default(); bitmap.insert(entity_id); drop(tree); + self.invalidate_total_count(); } /// Remove an entity from the bitmap at the given value. @@ -80,6 +101,10 @@ impl RangeIndex { if now_empty { tree.remove(value); } + drop(tree); + if removed { + self.invalidate_total_count(); + } removed } @@ -111,6 +136,10 @@ impl RangeIndex { for key in empty_keys { tree.remove(&key); } + drop(tree); + if count > 0 { + self.invalidate_total_count(); + } count } @@ -203,8 +232,24 @@ impl RangeIndex { } /// Total distinct entity IDs indexed. + /// + /// Memoized: the union over every value bitmap is computed at most once per + /// mutation. Repeated reads between writes (the planner's per-predicate + /// selectivity ordering) are O(1). #[must_use] pub fn total_count(&self) -> u64 { + // Fast path: return the memoized cardinality if still valid. Read the + // cache into a local and drop the guard before branching so the cache + // lock is never held across the slow-path tree read. + let cached = *self + .total_count_cache + .read() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if let Some(count) = cached { + return count; + } + // Slow path: recompute the union once and memoize it. Recompute under the + // tree read lock so the snapshot we count is internally consistent. let tree = self .tree .read() @@ -214,7 +259,12 @@ impl RangeIndex { union |= bitmap; } drop(tree); - union.len() + let count = union.len(); + *self + .total_count_cache + .write() + .unwrap_or_else(std::sync::PoisonError::into_inner) = Some(count); + count } /// Number of distinct attribute values in the tree. @@ -333,8 +383,9 @@ impl RangeIndex { /// /// # Errors /// - /// Returns `IndexError::Serialization` if deserialization fails (truncated - /// value bytes or a corrupt serialized bitmap). + /// Returns `IndexError::Serialization` if a matching key's value field is not + /// exactly `V::WIDTH` bytes (truncated *or* carrying trailing bytes) or if a + /// serialized bitmap is corrupt. pub fn load_from_kv_pairs( field_name: impl Into, pairs: impl Iterator, Vec)>, @@ -353,10 +404,19 @@ impl RangeIndex { continue; } let value_raw = &suffix_raw[prefix_bytes.len()..]; - if value_raw.len() < V::WIDTH { + // A well-formed range key carries EXACTLY `V::WIDTH` value bytes. + // Reject both truncation (too few) and trailing garbage (too many): + // `from_be_key_bytes` reads only `raw[..WIDTH]` and would silently + // discard a remainder, so two adversarial keys sharing the first + // WIDTH value bytes but differing in their tail would collapse onto + // the same tree entry (last-write-wins), masking corruption. Require + // an exact width so a malformed key surfaces as an error instead. + if value_raw.len() != V::WIDTH { return Err(IndexError::Serialization(format!( - "truncated {} value in key", - V::TYPE_NAME + "malformed {} value in key: expected exactly {} bytes, got {}", + V::TYPE_NAME, + V::WIDTH, + value_raw.len() ))); } let value = V::from_be_key_bytes(value_raw); @@ -370,6 +430,8 @@ impl RangeIndex { Ok(Self { field_name, tree: RwLock::new(tree), + // Loaded fresh; force a recompute on first read. + total_count_cache: RwLock::new(None), }) } } @@ -535,6 +597,57 @@ mod tests { assert!(inverted.is_empty()); } + #[test] + fn load_rejects_trailing_bytes_after_fixed_width_value() { + // A valid range key carries exactly V::WIDTH value bytes. A key with the + // same WIDTH-byte prefix but extra trailing bytes must be rejected, not + // silently truncated onto the same tree entry (which would mask + // corruption / let two distinct keys collide last-write-wins). + let index: RangeIndex = RangeIndex::new("duration"); + index.insert(1, 300); + let pairs = index.serialize_to_kv_pairs().unwrap(); + assert_eq!(pairs.len(), 1); + let (mut key, value) = pairs.into_iter().next().unwrap(); + // Append a trailing byte after the 4-byte u32 value. + key.push(0xFF); + + let result = + RangeIndex::::load_from_kv_pairs("duration", std::iter::once((key, value))); + let is_serialization_err = + matches!(&result, Err(IndexError::Serialization(m)) if m.contains("expected exactly")); + assert!( + is_serialization_err, + "trailing bytes after a fixed-width value must be a Serialization error (got Ok or wrong variant)" + ); + } + + #[test] + fn total_count_cache_tracks_inserts_and_deletes() { + // The memoized total_count must agree with the freshly-recomputed union + // across a full insert/delete cycle, proving cache invalidation is wired + // on every mutation (insert, delete, delete_entity). + let index: RangeIndex = RangeIndex::new("duration"); + assert_eq!(index.total_count(), 0); // primed-empty cache + index.insert(1, 60); + index.insert(2, 120); + index.insert(3, 120); + assert_eq!(index.total_count(), 3); // recomputed after invalidation + assert_eq!(index.total_count(), 3); // memoized fast path agrees + + // Re-inserting the same id at a new value does not change the distinct + // count (it is still one entity, now under two values until scrubbed). + index.insert(1, 600); + assert_eq!(index.total_count(), 3); + + // Deleting scrubs the id; the count drops only when no value still holds it. + index.delete_entity(1); + assert_eq!(index.total_count(), 2); + index.delete(2, &120); + assert_eq!(index.total_count(), 1); + index.delete(3, &120); + assert_eq!(index.total_count(), 0); + } + #[test] fn delete_cleans_up_empty_entries() { let index: RangeIndex = RangeIndex::new("duration"); diff --git a/tidal/src/storage/memory.rs b/tidal/src/storage/memory.rs index 5089474..4bcb3ef 100644 --- a/tidal/src/storage/memory.rs +++ b/tidal/src/storage/memory.rs @@ -38,11 +38,20 @@ impl std::fmt::Debug for InMemoryBackend { } impl StorageEngine for InMemoryBackend { + // Mirror the FjallBackend instrumentation (CODING_GUIDELINES §11.2) so the + // testing backend produces the same storage-stage spans as production — + // payload bytes are skipped, only lengths/op-counts are recorded. + #[tracing::instrument(level = "trace", skip(self), fields(key_len = key.len()))] fn get(&self, key: &[u8]) -> Result>, StorageError> { let data = self.data.read().map_err(|_| StorageError::Closed)?; Ok(data.get(key).cloned()) } + #[tracing::instrument( + level = "trace", + skip(self, value), + fields(key_len = key.len(), value_len = value.len()) + )] fn put(&self, key: &[u8], value: &[u8]) -> Result<(), StorageError> { self.data .write() @@ -51,6 +60,7 @@ impl StorageEngine for InMemoryBackend { Ok(()) } + #[tracing::instrument(level = "trace", skip(self), fields(key_len = key.len()))] fn delete(&self, key: &[u8]) -> Result<(), StorageError> { self.data .write() @@ -73,6 +83,7 @@ impl StorageEngine for InMemoryBackend { /// production restart paths use the streaming `FjallBackend`. /// /// [`FjallBackend::scan_prefix`]: crate::storage::fjall::FjallBackend + #[tracing::instrument(level = "debug", skip(self), fields(prefix_len = prefix.len()))] fn scan_prefix(&self, prefix: &[u8]) -> PrefixIterator<'_> { let Ok(data) = self.data.read() else { return Box::new(std::iter::once(Err(StorageError::Closed))); @@ -94,6 +105,7 @@ impl StorageEngine for InMemoryBackend { Box::new(entries.into_iter().map(Ok)) } + #[tracing::instrument(level = "debug", skip(self, batch), fields(ops = batch.len()))] fn write_batch(&self, batch: WriteBatch) -> Result<(), StorageError> { let mut data = self.data.write().map_err(|_| StorageError::Closed)?; for op in &batch.ops { @@ -110,6 +122,7 @@ impl StorageEngine for InMemoryBackend { Ok(()) } + #[tracing::instrument(level = "trace", skip(self))] fn flush(&self) -> Result<(), StorageError> { // No-op for in-memory backend. Ok(()) diff --git a/tidal/src/storage/vector/registry.rs b/tidal/src/storage/vector/registry.rs index 41327bc..b8a5ee5 100644 --- a/tidal/src/storage/vector/registry.rs +++ b/tidal/src/storage/vector/registry.rs @@ -11,9 +11,83 @@ use std::collections::HashMap; -use super::{QuantizationLevel, VectorError, VectorIndex}; +use super::{QuantizationLevel, VectorError, VectorIndex, VectorIndexConfig}; use crate::schema::EntityKind; +// --------------------------------------------------------------------------- +// Vector backend selection (production HNSW vs. brute-force) +// --------------------------------------------------------------------------- + +/// Minimum live-vector count for a slot before the production HNSW engine +/// (`UsearchIndex`) is preferred over the exact `BruteForceIndex`. +/// +/// `BruteForceIndex` computes an L2 distance against every stored vector under +/// an `RwLock` read on each search — exact, but O(n) per query and serialized +/// against writes. It is the right choice for tiny slots (no graph-build cost, +/// exact results) but blows past the latency budget at scale. `UsearchIndex` +/// (HNSW) is the mandated production engine (`CODING_GUIDELINES` §4: "126K+ QPS", +/// §8: "ANN retrieval at 1M vectors <10ms p99"). Gating on slot size needs NO +/// external config plumbing: at process restart, `rebuild_from_store` already +/// knows each slot's vector count, so a slot that crossed this threshold is +/// rebuilt as HNSW automatically, while a small slot stays exact. The threshold +/// is deliberately conservative — well inside the range where brute force is +/// still fast — so correctness-sensitive small slots keep exact results. +pub(crate) const USEARCH_MIN_VECTORS: usize = 10_000; + +/// Build the vector index backend for a slot, gating on the expected vector +/// `count`: `UsearchIndex` (production HNSW) at or above [`USEARCH_MIN_VECTORS`], +/// `BruteForceIndex` (exact) below it. +/// +/// When the `USearch` backend is selected, capacity is reserved up front +/// (`UsearchIndex::add` requires a prior `reserve`), so the subsequent inserts +/// cannot fail for lack of capacity. If `UsearchIndex` construction OR the +/// reservation fails for any +/// reason, this DEGRADES to `BruteForceIndex` with a `tracing::warn!` rather than +/// propagating — losing the latency win is acceptable; losing the slot entirely +/// (and therefore all ANN results for the kind) is not. This is the single place +/// that picks the engine; the write-path auto-register and the rebuild path both +/// route through it so the choice cannot drift. +pub(crate) fn build_slot_index(dimensions: usize, count: usize) -> Box { + let config = VectorIndexConfig { + dimensions, + ..VectorIndexConfig::default() + }; + if count < USEARCH_MIN_VECTORS { + return Box::new(super::BruteForceIndex::new(config)); + } + match super::UsearchIndex::new(config.clone()) { + Ok(index) => { + // USearch's `add` requires capacity reserved up front. Reserve the + // expected count so the rebuild's inserts all fit; the write path + // grows it further as needed. + if let Err(e) = index.reserve(count) { + tracing::warn!( + dimensions, + count, + error = %e, + "USearch reserve failed; falling back to brute-force for this slot" + ); + return Box::new(super::BruteForceIndex::new(config)); + } + tracing::info!( + dimensions, + count, + "using USearch (HNSW) backend for embedding slot" + ); + Box::new(index) + } + Err(e) => { + tracing::warn!( + dimensions, + count, + error = %e, + "USearch index construction failed; falling back to brute-force for this slot" + ); + Box::new(super::BruteForceIndex::new(config)) + } + } +} + /// Number of bytes per vector component at a given quantization level. /// /// Used by [`EmbeddingSlotRegistry::index_stats`] to estimate index footprint @@ -264,12 +338,28 @@ impl EmbeddingSlotRegistry { /// /// Returns the number of vectors inserted across all slots for this kind. /// + /// # Backend selection + /// + /// Each slot's backend is chosen by [`build_slot_index`] from the slot's live + /// vector count: a slot at or above [`USEARCH_MIN_VECTORS`] is rebuilt as the + /// production HNSW engine (`UsearchIndex`); smaller slots stay exact + /// (`BruteForceIndex`). This is the live path that makes `UsearchIndex` + /// reachable in production after a restart, with no config plumbing. + /// + /// # Fault isolation + /// + /// A single un-indexable embedding (un-deserializable, zero-dimension, or a + /// dimensionality that mismatches its slot) is SKIPPED with a `tracing::warn!` + /// and counted, not propagated — one poison row must never abort the rebuild + /// for an entire entity kind (which would leave ANN search empty for every + /// good embedding too). The skipped count is logged at the end. + /// /// # Errors /// - /// - [`VectorError::Io`] if a storage scan entry fails to read. - /// - [`VectorError::CorruptedIndex`] if a stored embedding fails to deserialize. + /// - [`VectorError::Io`] if a storage scan entry fails to read (a whole-scan + /// failure is genuinely unrecoverable, so it still propagates). /// - [`VectorError::Backend`] if slot registration fails. - /// - Any error from [`VectorIndex::insert`]. + #[allow(clippy::too_many_lines)] pub(crate) fn rebuild_from_store( &mut self, entity_kind: EntityKind, @@ -303,6 +393,10 @@ impl EmbeddingSlotRegistry { // rather than allocating a throwaway `String` per embedding key. let mut archived: HashSet<(u64, String)> = HashSet::new(); let mut candidates: Vec<(crate::schema::EntityId, String, Vec)> = Vec::new(); + // Count of embeddings skipped due to a per-row fault (un-deserializable, + // zero-dim, or un-indexable). Reported at the end so a degraded rebuild is + // observable rather than silent. + let mut skipped = 0usize; for entry in storage.scan_prefix(&[]) { let (key, value) = entry.map_err(|e| { VectorError::Io(std::io::Error::other(format!( @@ -337,28 +431,75 @@ impl EmbeddingSlotRegistry { continue; }; - // Deserialize the source-of-truth vector eagerly (header carries - // dimensions); the owned slot `String` doubles as the archived-set - // lookup key below, so no extra per-key allocation is incurred. - let vector = super::deserialize_embedding(&value)?; + // Deserialize the source-of-truth vector. A single un-deserializable + // embedding (corrupt header, hand-edited store) must NOT abort the + // rebuild for the whole entity kind — that would leave ANN search empty + // for every good embedding too. Isolate the fault: warn, count it, and + // skip this one row. The durable value is untouched; the next write + // re-inserts it. (W: one poison row must not brick the kind.) + let vector = match super::deserialize_embedding(&value) { + Ok(v) => v, + Err(e) => { + tracing::warn!( + entity_id = entity_id.as_u64(), + slot = slot_name, + error = %e, + "skipping un-deserializable embedding during rebuild" + ); + skipped += 1; + continue; + } + }; candidates.push((entity_id, slot_name.to_string(), vector)); } + // Group the surviving (non-archived, non-degenerate) embeddings by slot so + // each slot's vector count is known before its backend is built. The count + // drives the backend selection (USearch HNSW for large slots, brute force + // for small) AND the capacity reservation, with NO external config plumbing. + let mut by_slot: HashMap)>> = HashMap::new(); + for (entity_id, slot_name, vector) in candidates { + // Skip embeddings that were soft-deleted (archived). The source-of-truth + // value is intentionally retained on disk, but it must not re-enter the + // ANN index — that is the whole point of the durable tombstone. + if archived.contains(&(entity_id.as_u64(), slot_name.clone())) { + continue; + } + + // Reject a zero-dimension vector. The write path can never produce one + // (`l2_normalize` rejects an empty vector before anything is stored), + // but `serialize_embedding(&[])` is a valid 4-byte payload that + // `deserialize_embedding` accepts, so a corrupt / hand-crafted store + // could carry one. A 0-dim slot makes `l2_distance_sq` return 0.0 for + // every query, so the entity would tie for nearest on every search and + // pollute results. Skip it loudly. (W: corrupt 0-dim slot.) + if vector.is_empty() { + tracing::warn!( + entity_id = entity_id.as_u64(), + slot = slot_name.as_str(), + "skipping zero-dimension embedding during rebuild (corrupt store)" + ); + skipped += 1; + continue; + } + + by_slot + .entry(slot_name) + .or_default() + .push((entity_id, vector)); + } + let mut inserted = 0usize; - for (entity_id, slot_name, vector) in candidates { - // Skip embeddings that were soft-deleted (archived). The source-of-truth - // value is intentionally retained on disk, but it must not re-enter the - // ANN index — that is the whole point of the durable tombstone. We move - // the candidate's owned `String` into the lookup tuple and move it back - // out on a miss, so the check costs zero extra allocations per key. - let probe = (entity_id.as_u64(), slot_name); - if archived.contains(&probe) { + for (slot_name, entries) in by_slot { + // The slot's dimensionality is the FIRST stored vector's length; any + // later vector of a different length is fault-isolated below. Cross- + // check against the schema declaration (advisory only — the stored + // header is authoritative for what was actually written). + let Some((_, first_vec)) = entries.first() else { continue; - } - let (_, slot_name) = probe; - - let dimensions = vector.len(); + }; + let dimensions = first_vec.len(); if let Some(&declared) = declared_dims.get(slot_name.as_str()) && declared != dimensions @@ -372,16 +513,14 @@ impl EmbeddingSlotRegistry { ); } - // Lazily register the slot on first sight, mirroring the write path - // (BruteForceIndex, F32, External). Dimensions come from the stored - // header so the rebuilt index matches what was actually written. + // Lazily register the slot on first sight, routing the backend through + // the size-gated factory: a slot with >= USEARCH_MIN_VECTORS live + // vectors is rebuilt as the production HNSW (USearch) engine; smaller + // slots stay exact (brute force). This is the live path that makes + // USearch reachable in production restarts — no config flag required. if self.get(entity_kind, &slot_name).is_none() { - let config = super::VectorIndexConfig { - dimensions, - ..super::VectorIndexConfig::default() - }; let state = EmbeddingSlotState { - index: Box::new(super::BruteForceIndex::new(config)), + index: build_slot_index(dimensions, entries.len()), dimensions, quantization: QuantizationLevel::F32, source: EmbeddingSource::External, @@ -390,20 +529,38 @@ impl EmbeddingSlotRegistry { self.register(entity_kind, slot_name.clone(), state)?; } - // Insert into the (now-guaranteed-present) slot index. let slot = self.get(entity_kind, &slot_name).ok_or_else(|| { VectorError::Backend(format!( "slot ({entity_kind}, \"{slot_name}\") missing immediately after registration" )) })?; - slot.index.insert(entity_id.as_u64(), &vector)?; - inserted += 1; + + for (entity_id, vector) in entries { + // Fault-isolate the per-embedding insert: a single row whose + // dimensionality differs from the slot's (e.g. a schema-revision + // mismatch that still self-deserializes) returns DimensionMismatch. + // Skip it loudly rather than abort the kind's rebuild — the + // hundreds of thousands of good embeddings for the same slot must + // still be indexed. (W: one poison row must not brick the kind.) + if let Err(e) = slot.index.insert(entity_id.as_u64(), &vector) { + tracing::warn!( + entity_id = entity_id.as_u64(), + slot = slot_name.as_str(), + error = %e, + "skipping un-indexable embedding during rebuild" + ); + skipped += 1; + continue; + } + inserted += 1; + } } - if inserted > 0 { + if inserted > 0 || skipped > 0 { tracing::info!( entity_kind = %entity_kind, vectors = inserted, + skipped, "embedding indexes rebuilt from durable storage" ); } @@ -805,4 +962,110 @@ mod tests { // No slot is registered when there is nothing durable to index. assert!(registry.get(EntityKind::Item, "content").is_none()); } + + // ------------------------------------------------------------------- + // build_slot_index backend selection (USearch wiring) — finding 5 + // ------------------------------------------------------------------- + + /// CRITICAL regression: the size-gated factory must build the production + /// HNSW (`UsearchIndex`) at/above the threshold and the exact `BruteForceIndex` + /// below it, and BOTH must be functional. Before the fix `USearch` was wired + /// into nothing and every slot was brute force. + #[test] + fn build_slot_index_selects_backend_by_count() { + // Below threshold: exact brute force, functional. + let small = build_slot_index(4, 0); + small.insert(1, &[1.0, 0.0, 0.0, 0.0]).unwrap(); + small.insert(2, &[0.0, 1.0, 0.0, 0.0]).unwrap(); + let r = small.search(&[1.0, 0.0, 0.0, 0.0], 1, 0).unwrap(); + assert_eq!(r[0].id, 1, "brute-force backend must return the nearest"); + + // At threshold: USearch (HNSW), with capacity reserved by the factory so + // inserts succeed without a manual `reserve`. Searchable end-to-end — + // proving the production engine is actually reachable, not dead code. + let big = build_slot_index(4, USEARCH_MIN_VECTORS); + big.insert(10, &[1.0, 0.0, 0.0, 0.0]).unwrap(); + big.insert(20, &[0.0, 1.0, 0.0, 0.0]).unwrap(); + let r = big.search(&[1.0, 0.0, 0.0, 0.0], 1, big.len()).unwrap(); + assert_eq!( + r[0].id, 10, + "USearch backend must be constructed AND searchable at the threshold" + ); + } + + // ------------------------------------------------------------------- + // rebuild fault isolation — findings 8 and 13 + // ------------------------------------------------------------------- + + /// W regression: a single un-indexable embedding (a different dimension under + /// the same slot) must NOT abort the rebuild — the good embeddings for the + /// kind are still indexed and searchable; only the poison row is skipped. + #[test] + fn rebuild_isolates_dimension_mismatch_and_zero_dim() { + use crate::{ + schema::{DecaySpec, EntityId, SchemaBuilder}, + storage::{ + memory::InMemoryBackend, + vector::{embedding_store_key, serialize_embedding}, + }, + }; + + let dim = 4; + let mut builder = SchemaBuilder::new(); + let _ = builder + .signal("view", EntityKind::Item, DecaySpec::Permanent) + .add(); + builder.embedding_slot("content", EntityKind::Item, dim); + let schema = builder.build().expect("valid schema"); + + let storage = InMemoryBackend::new(); + // Two valid 4-D embeddings. + for (id, v) in &[ + (1u64, [1.0f32, 0.0, 0.0, 0.0]), + (2u64, [0.0f32, 1.0, 0.0, 0.0]), + ] { + let key = embedding_store_key(EntityId::new(*id), "content"); + storage.put(&key, &serialize_embedding(v)).unwrap(); + } + // One poison row: a 2-D vector under the SAME slot. Its stored header is + // self-consistent (deserializes fine) but its dimension differs from the + // slot's (4) → DimensionMismatch on insert. It must be skipped, not abort. + let bad_key = embedding_store_key(EntityId::new(3), "content"); + storage + .put(&bad_key, &serialize_embedding(&[1.0, 1.0])) + .unwrap(); + // One zero-dimension row (corrupt store): a valid 4-byte payload that + // deserializes to an empty vector. Must be skipped (finding 13). + let empty_key = embedding_store_key(EntityId::new(4), "content"); + storage.put(&empty_key, &serialize_embedding(&[])).unwrap(); + + let mut registry = EmbeddingSlotRegistry::new(); + let inserted = registry + .rebuild_from_store(EntityKind::Item, &storage, &schema) + .expect("rebuild must not abort on a poison row"); + + assert_eq!( + inserted, 2, + "the two valid embeddings are indexed; the mismatch and zero-dim rows are skipped" + ); + let slot = registry + .get(EntityKind::Item, "content") + .expect("slot registered from the valid embeddings"); + assert_eq!( + slot.dimensions, dim, + "the slot must take the valid (4-D) dimensionality, not the corrupt 0-dim row" + ); + assert_eq!(slot.index.len(), 2); + + // The two valid embeddings are searchable; the bad ids never resurface. + let results = slot.index.search(&[1.0, 0.0, 0.0, 0.0], 10, 0).unwrap(); + assert!( + results.iter().all(|r| r.id != 3 && r.id != 4), + "poison rows (ids 3, 4) must not be indexed" + ); + assert!( + results.iter().any(|r| r.id == 1), + "valid embedding 1 must be searchable" + ); + } } diff --git a/tidal/src/testing/cluster.rs b/tidal/src/testing/cluster.rs index ec5e66b..567253f 100644 --- a/tidal/src/testing/cluster.rs +++ b/tidal/src/testing/cluster.rs @@ -54,7 +54,7 @@ use std::{ }; use super::cluster_transport::{ - BatchEntry, ChannelTransport, RecvOnlyChannelTransport, redeliver_missed, + BatchEntry, ChannelTransport, RecvOnlyChannelTransport, redeliver_missed, single_event_payload, }; use crate::{ db::{ @@ -63,7 +63,7 @@ use crate::{ }, query::{retrieve::Retrieve, search::Search}, replication::{ - SegmentReceiverHandle, WalSegmentId, + SegmentReceiverHandle, shard::{RegionId, ShardId}, spawn_receiver, transport::{Transport, WalSegmentPayload}, @@ -425,15 +425,29 @@ impl SimulatedCluster { if region == leader_region || partitioned.contains(®ion) { continue; } - let payload = WalSegmentPayload { - id: WalSegmentId::new(crate::replication::RegionId::SINGLE, leader_shard, seqno), - bytes: bytes.clone(), - event_count: 1, - // Single-event batch with first_seq = seqno → last WAL seq = seqno. - leader_last_seq: seqno, - }; - // Ignore send errors: the receiver may have exited (e.g. after a crash). - let _ = transport.send_segment(ShardId(region.0), payload); + // Share the payload construction with `redeliver_missed` so the eager + // ship path and the recovery path can never drift on the load-bearing + // `event_count`/`leader_last_seq` invariants (the receiver's monotonic + // `advance` relies on `leader_last_seq == seqno`). + let payload = single_event_payload(leader_shard, seqno, bytes.clone()); + // The eager ship is BEST-EFFORT: the leader write is already durable + // and recorded in `batch_log`, so a failed ship is recovered by + // `await_convergence` / `heal_region` re-delivery. We do NOT fail the + // write on a ship error (that would be the wrong contract — the data + // is durably on the leader). But a swallowed ship error is invisible + // to an operator, so surface it as a WARN: a follower that keeps + // failing to receive is observable here (and reconciled later) rather + // than silently lagging. See `redeliver_missed`. + if let Err(e) = transport.send_segment(ShardId(region.0), payload) { + tracing::warn!( + region = region.0, + leader_shard = leader_shard.0, + seqno, + error = %e, + "eager follower ship failed; leader write is durable and queued for \ + re-delivery (await_convergence / heal_region)" + ); + } } // Record in batch_log for partition-recovery re-delivery. diff --git a/tidal/src/testing/cluster_transport.rs b/tidal/src/testing/cluster_transport.rs index d5df72a..bdac090 100644 --- a/tidal/src/testing/cluster_transport.rs +++ b/tidal/src/testing/cluster_transport.rs @@ -157,3 +157,33 @@ pub(super) fn redeliver_missed(transport: &dyn Transport, db: &TidalDb, log: &[B } } } + +#[cfg(test)] +mod tests { + use super::*; + + /// Locks the load-bearing payload-shape invariants of the single source of + /// truth that both `SimulatedCluster::write_signal` (W2 fix) and + /// [`redeliver_missed`] now share: a one-event batch in [`RegionId::SINGLE`] + /// whose `leader_last_seq == seqno`. The receiver's monotonic `advance` + /// relies on `leader_last_seq == seqno`; a future edit that breaks either + /// invariant on this single helper fails here instead of silently diverging + /// between the eager-ship and recovery sites. + #[test] + fn single_event_payload_holds_replication_invariants() { + let shard = ShardId(3); + let seqno = 42; + let bytes = vec![1u8, 2, 3, 4]; + let payload = single_event_payload(shard, seqno, bytes.clone()); + + assert_eq!(payload.event_count, 1, "must be a single-event batch"); + assert_eq!( + payload.leader_last_seq, seqno, + "receiver advance relies on leader_last_seq == seqno" + ); + assert_eq!(payload.id.region_id, RegionId::SINGLE); + assert_eq!(payload.id.shard_id, shard); + assert_eq!(payload.id.seqno, seqno); + assert_eq!(payload.bytes, bytes); + } +} diff --git a/tidal/src/text/collectors.rs b/tidal/src/text/collectors.rs index 4b7c9f2..68bfa3e 100644 --- a/tidal/src/text/collectors.rs +++ b/tidal/src/text/collectors.rs @@ -1,3 +1,5 @@ +use std::{cmp::Ordering, collections::BinaryHeap}; + use tantivy::{ DocId, Score, SegmentOrdinal, SegmentReader, collector::{Collector, SegmentCollector}, @@ -19,6 +21,15 @@ use crate::schema::EntityId; /// /// `requires_scoring()` returns `true`; without this, Tantivy skips BM25 /// computation and every document receives a score of 0.0. +/// +/// # Unbounded — prefer [`BoundedScoresCollector`] on the query hot path +/// +/// This collector keeps EVERY match, so a broad / near-ubiquitous term over a +/// large corpus materializes one `(EntityId, f32)` per match — an allocation +/// proportional to corpus match count, not the requested limit. The SEARCH +/// pipeline uses [`BoundedScoresCollector`] instead, which caps the result to a +/// top-K inside Tantivy. `AllScoresCollector` is retained for callers (and tests) +/// that genuinely need the full match set. pub struct AllScoresCollector { /// The Tantivy `Field` handle for `entity_id` (u64, FAST). /// @@ -102,6 +113,168 @@ impl SegmentCollector for AllScoresSegmentCollector { } } +// ---- BoundedScoresCollector ---------------------------------------------------- + +/// Like [`AllScoresCollector`] but keeps only the `cap` highest-scoring matches. +/// +/// On a broad or near-ubiquitous term over a large corpus, [`AllScoresCollector`] +/// allocates one `(EntityId, f32)` per match (plus a downstream `HashMap`) — a +/// memory-amplification / latency-spike vector proportional to corpus match count +/// rather than the requested limit. This collector keeps a per-segment bounded +/// min-heap of size `cap`, so peak memory is `O(cap)` INSIDE Tantivy regardless of +/// how many documents match. A `cap` of `0` collects nothing. +/// +/// The returned set is the global top-`cap` by score; `NaN` scores (degenerate +/// BM25 term statistics) are treated as the smallest and evicted first, matching +/// the pipeline's `finite` neutralization. Ordering of the returned `Vec` is +/// unspecified — the SEARCH pipeline sorts it deterministically (descending score, +/// id tie-break) before use. +pub struct BoundedScoresCollector { + /// The Tantivy `Field` handle for `entity_id` (u64, FAST). See + /// [`AllScoresCollector::entity_id_field`]. + pub entity_id_field: Field, + /// Maximum number of highest-scoring documents to keep. + pub cap: usize, +} + +impl BoundedScoresCollector { + /// Collector bounded to the `cap` highest-scoring documents. + #[must_use] + pub const fn new(entity_id_field: Field, cap: usize) -> Self { + Self { + entity_id_field, + cap, + } + } +} + +/// A `(score, entity_id)` pair ordered so a [`BinaryHeap`] (a max-heap) behaves as +/// a MIN-heap on score — the smallest score is at the top and is evicted first when +/// the heap is at capacity. `NaN` scores are treated as the smallest (evicted +/// first). Ties break on entity id so eviction is deterministic. +#[derive(Debug, Clone, Copy, PartialEq)] +struct MinScored { + score: f32, + entity_id: u64, +} + +impl Eq for MinScored {} + +impl Ord for MinScored { + fn cmp(&self, other: &Self) -> Ordering { + // Reverse the score order so `BinaryHeap::peek`/`pop` yields the SMALLEST + // score (the eviction candidate). NaN sorts as the smallest score. + let a = if self.score.is_nan() { + f32::MIN + } else { + self.score + }; + let b = if other.score.is_nan() { + f32::MIN + } else { + other.score + }; + b.total_cmp(&a) + .then_with(|| other.entity_id.cmp(&self.entity_id)) + } +} + +impl PartialOrd for MinScored { + fn partial_cmp(&self, other: &Self) -> Option { + Some(self.cmp(other)) + } +} + +/// Per-segment bounded collector backing [`BoundedScoresCollector`]. +pub struct BoundedScoresSegmentCollector { + entity_id_col: Column, + heap: BinaryHeap, + cap: usize, +} + +impl Collector for BoundedScoresCollector { + type Fruit = Vec<(EntityId, f32)>; + type Child = BoundedScoresSegmentCollector; + + fn for_segment( + &self, + _segment_ord: SegmentOrdinal, + reader: &SegmentReader, + ) -> tantivy::Result { + let column_name = reader.schema().get_field_name(self.entity_id_field); + let entity_id_col = reader.fast_fields().u64(column_name)?; + Ok(BoundedScoresSegmentCollector { + entity_id_col, + heap: BinaryHeap::with_capacity(self.cap.saturating_add(1)), + cap: self.cap, + }) + } + + fn requires_scoring(&self) -> bool { + true + } + + fn merge_fruits( + &self, + segment_fruits: Vec>, + ) -> tantivy::Result> { + let total = segment_fruits.iter().map(Vec::len).sum(); + let mut merged = Vec::with_capacity(total); + for fruit in segment_fruits { + merged.extend(fruit); + } + // Each segment already trimmed to `cap`, but the UNION of per-segment + // top-`cap` sets can exceed `cap`. Trim to the GLOBAL top-`cap` so the + // returned set never exceeds the requested bound. + if self.cap == 0 { + return Ok(Vec::new()); + } + if merged.len() > self.cap { + // Partition so the `cap` highest scores occupy [0, cap); NaN sinks to + // the bottom (treated as the smallest), ties broken on ascending id + // for determinism, then drop the tail. + let finite = |s: f32| if s.is_nan() { f32::MIN } else { s }; + let nth = self.cap.saturating_sub(1).min(merged.len() - 1); + merged.select_nth_unstable_by(nth, |a, b| { + finite(b.1) + .total_cmp(&finite(a.1)) + .then_with(|| a.0.as_u64().cmp(&b.0.as_u64())) + }); + merged.truncate(self.cap); + } + Ok(merged) + } +} + +impl SegmentCollector for BoundedScoresSegmentCollector { + type Fruit = Vec<(EntityId, f32)>; + + fn collect(&mut self, doc: DocId, score: Score) { + if self.cap == 0 { + return; + } + let Some(eid_val) = self.entity_id_col.first(doc) else { + return; + }; + self.heap.push(MinScored { + score, + entity_id: eid_val, + }); + // Keep only the top `cap`: once over capacity, evict the smallest score + // (the heap root). Peak memory stays `O(cap)`. + if self.heap.len() > self.cap { + self.heap.pop(); + } + } + + fn harvest(self) -> Self::Fruit { + self.heap + .into_iter() + .map(|m| (EntityId::new(m.entity_id), m.score)) + .collect() + } +} + #[cfg(test)] #[allow(clippy::unwrap_used)] mod tests { @@ -235,6 +408,75 @@ mod tests { idx.close().unwrap(); } + /// W regression: a bounded collector must return at most `cap` documents even + /// when more match, and must keep the highest-scoring ones. Doc 1 repeats the + /// query term so its BM25 score outranks docs 2 and 3; `cap = 1` must keep + /// exactly doc 1. + #[test] + fn bounded_collector_keeps_top_cap_by_score() { + // Doc 1 mentions "jazz" twice → higher BM25 than docs 2/3 (once each). + let idx = setup_index(&[ + (1, "jazz jazz piano"), + (2, "jazz guitar"), + (3, "jazz violin"), + ]); + + let searcher = idx.reader.searcher(); + let title_field = idx.fields().text_fields[0].1; + let qp = QueryParser::for_index(&idx.index, vec![title_field]); + let query = qp.parse_query("jazz").unwrap(); + + // cap = 1: only the single highest-scoring match (doc 1) survives. + let bounded = BoundedScoresCollector::new(idx.fields().entity_id, 1); + let results = searcher.search(&query, &bounded).unwrap(); + assert_eq!( + results.len(), + 1, + "bounded collector must return at most cap=1" + ); + assert_eq!( + results[0].0.as_u64(), + 1, + "the single retained doc must be the highest-scoring match" + ); + + // cap = 2: the two highest-scoring of three matches survive. + let bounded2 = BoundedScoresCollector::new(idx.fields().entity_id, 2); + let results2 = searcher.search(&query, &bounded2).unwrap(); + assert_eq!( + results2.len(), + 2, + "bounded collector must return at most cap=2" + ); + assert!( + results2.iter().any(|(id, _)| id.as_u64() == 1), + "the highest-scoring doc must be retained" + ); + + // The unbounded collector still returns all three matches. + let unbounded = AllScoresCollector { + entity_id_field: idx.fields().entity_id, + }; + let all = searcher.search(&query, &unbounded).unwrap(); + assert_eq!(all.len(), 3, "unbounded collector returns every match"); + + idx.close().unwrap(); + } + + /// A `cap` of 0 collects nothing. + #[test] + fn bounded_collector_cap_zero_collects_nothing() { + let idx = setup_index(&[(1, "jazz piano"), (2, "jazz violin")]); + let searcher = idx.reader.searcher(); + let title_field = idx.fields().text_fields[0].1; + let qp = QueryParser::for_index(&idx.index, vec![title_field]); + let query = qp.parse_query("jazz").unwrap(); + let bounded = BoundedScoresCollector::new(idx.fields().entity_id, 0); + let results = searcher.search(&query, &bounded).unwrap(); + assert!(results.is_empty(), "cap=0 must collect nothing"); + idx.close().unwrap(); + } + #[test] fn all_scores_empty_query_returns_empty() { let idx = setup_index(&[(1, "jazz piano")]); diff --git a/tidal/src/text/writer.rs b/tidal/src/text/writer.rs index cc5c930..801e86f 100644 --- a/tidal/src/text/writer.rs +++ b/tidal/src/text/writer.rs @@ -97,9 +97,10 @@ impl TextIndexWriter<'_> { /// Crash recovery for the text index is NOT driven from a commit payload. /// The durable entity store is the source of truth, and the text index is /// derived state that is unconditionally rebuilt from that store at open - /// (see `db::mod::rebuild_text_indexes_at_open`). This mirrors the vector - /// index, which is likewise rebuilt from the durable embeddings on every - /// reopen rather than reconciled against a stored sequence number. + /// (see [`crate::db::state_rebuild::rebuild_text_indexes_at_open`]). This + /// mirrors the vector index, which is likewise rebuilt from the durable + /// embeddings on every reopen rather than reconciled against a stored + /// sequence number. /// /// # Errors /// diff --git a/tidal/src/wal/checkpoint.rs b/tidal/src/wal/checkpoint.rs index 5a1cb6c..db612d5 100644 --- a/tidal/src/wal/checkpoint.rs +++ b/tidal/src/wal/checkpoint.rs @@ -39,9 +39,13 @@ impl CheckpointManager { fs::write(&tmp_path, content.as_bytes())?; - // fsync the temp file to ensure contents are durable before rename + // fsync the temp file to ensure contents are durable before rename. On + // macOS a plain fsync does not flush the device write cache, so route + // through the F_FULLFSYNC-backed helper — the checkpoint is a durable + // artifact and must meet the same crash-durability bar as the WAL + // segments it gates compaction of. let file = fs::File::open(&tmp_path)?; - file.sync_all()?; + super::sync_file_durable(&file)?; drop(file); // Atomic rename (POSIX guarantees) @@ -49,9 +53,9 @@ impl CheckpointManager { // Fsync the directory to ensure the rename (directory entry update) // is durable. Without this, a crash after rename but before the - // directory metadata is flushed could lose the checkpoint file. - let dir_fd = fs::File::open(dir)?; - dir_fd.sync_all()?; + // directory metadata is flushed could lose the checkpoint file. On + // macOS this routes through the F_FULLFSYNC-backed helper. + super::sync_dir_durable(dir)?; Ok(()) } diff --git a/tidal/src/wal/compaction.rs b/tidal/src/wal/compaction.rs index 289a448..d9cd7e8 100644 --- a/tidal/src/wal/compaction.rs +++ b/tidal/src/wal/compaction.rs @@ -144,10 +144,12 @@ fn compact_segments( // Fsync the directory to ensure the unlink operations are durable. // Without this, a crash after deletion but before directory metadata - // flush could "resurrect" deleted segment files. + // flush could "resurrect" deleted segment files. On macOS a plain + // directory fsync does not flush the device cache, so route through the + // F_FULLFSYNC-backed helper. This is the same durable-delete contract as + // `segment::delete_segments_before`. if deleted > 0 { - let dir_fd = std::fs::File::open(wal_dir)?; - dir_fd.sync_all()?; + crate::wal::sync_dir_durable(wal_dir)?; } let remaining = total_before - deleted; diff --git a/tidal/src/wal/dedup.rs b/tidal/src/wal/dedup.rs index 8b014bb..1410939 100644 --- a/tidal/src/wal/dedup.rs +++ b/tidal/src/wal/dedup.rs @@ -101,9 +101,18 @@ impl DedupWindow { /// fsynced, so a transient flush failure leaves the event un-recorded and a /// retry is accepted rather than dropped. Honours the per-window size cap /// like the rest of the window (forces an early rotation at the cap). + /// + /// A size-driven rotation here uses [`rotate_buffers`](Self::rotate_buffers), + /// which does NOT reset the time-based rotation deadline. `record()` is + /// called post-fsync for every kept event and never itself drives the time + /// window (only `contains`/`is_duplicate` call `maybe_rotate`). If a + /// size-driven rotation reset the clock, a sustained burst of unique events + /// would keep pushing the time deadline forward, so the effective time + /// window would silently grow longer than configured under load. Leaving the + /// deadline untouched keeps the time window honest regardless of write rate. pub fn record(&mut self, event: &EventRecord) { if self.current.len() >= self.max_entries { - self.force_rotate(); + self.rotate_buffers(); } let hash = event_content_hash(event); self.current.insert(hash); @@ -114,11 +123,12 @@ impl DedupWindow { /// This inserts all provided events into the current window so that /// duplicates arriving after recovery are correctly detected. The size cap /// is honoured here too: a recovery set larger than `max_entries` rotates - /// rather than growing `current` without bound. + /// rather than growing `current` without bound. Like [`record`](Self::record), + /// a size-driven rotation here preserves the time-based rotation deadline. pub fn populate_from_events>(&mut self, events: I) { for event in events { if self.current.len() >= self.max_entries { - self.force_rotate(); + self.rotate_buffers(); } let hash = event_content_hash(&event); self.current.insert(hash); @@ -141,17 +151,35 @@ impl DedupWindow { /// Rotate buffers if the window duration has elapsed OR the active buffer has /// reached its size cap, whichever comes first. + /// + /// A **time**-driven rotation resets the rotation deadline (the window just + /// elapsed, so the next window starts now). A **size**-driven rotation does + /// NOT reset it: the configured time window should still expire on schedule + /// even when a burst forced an early size rotation, so the effective window + /// can never silently grow longer than configured under sustained load. fn maybe_rotate(&mut self) { - if self.rotation_time.elapsed() >= self.window || self.current.len() >= self.max_entries { + if self.rotation_time.elapsed() >= self.window { self.force_rotate(); + } else if self.current.len() >= self.max_entries { + self.rotate_buffers(); } } - /// Swap `current` into `previous`, clear the new `current`, and reset the - /// rotation timer. Old `previous` entries are dropped. - fn force_rotate(&mut self) { + /// Swap `current` into `previous` and clear the new `current`, WITHOUT + /// touching the time-based rotation deadline. Old `previous` entries are + /// dropped. Used by every size-driven rotation (`record`, + /// `populate_from_events`, and `maybe_rotate`'s size branch) so a burst of + /// writes cannot push the time deadline forward. + fn rotate_buffers(&mut self) { std::mem::swap(&mut self.current, &mut self.previous); self.current.clear(); + } + + /// Rotate the buffers AND reset the rotation timer. Used only by the + /// time-driven rotation path: the window just elapsed, so the next window + /// begins now. + fn force_rotate(&mut self) { + self.rotate_buffers(); self.rotation_time = Instant::now(); } } @@ -279,6 +307,46 @@ mod tests { assert!(dedup.is_duplicate(&recent)); } + #[test] + fn size_driven_record_rotation_does_not_reset_time_deadline() { + // wal-recovery SUG: a size-driven rotation inside record() must NOT push + // the time-based rotation deadline forward, or a sustained burst would + // silently stretch the effective time window. White-box: the rotation + // timer is module-private, so this same-module test reads it directly. + let max = 4; + let mut dedup = DedupWindow::with_max_entries(Duration::from_secs(3600), max); + let deadline_before = dedup.rotation_time; + + // record() more than the cap so it size-rotates at least once. record() + // never drives the time window, so the deadline must be untouched. + for i in 0..(max as u64 * 3) { + dedup.record(&make_event(i)); + } + + assert_eq!( + dedup.rotation_time, deadline_before, + "a size-driven record() rotation must preserve the time deadline" + ); + // And the size cap still bounds memory. + assert!(dedup.len() <= 2 * max); + } + + #[test] + fn time_driven_rotation_resets_deadline() { + // The complement: a TIME-driven rotation (window elapsed) DOES reset the + // deadline, so the next window starts fresh. A zero-length window makes + // every maybe_rotate() take the time branch without sleeping. + let mut dedup = DedupWindow::new(Duration::ZERO); + let before = dedup.rotation_time; + // `is_duplicate` calls `maybe_rotate`; with a ZERO window the time branch + // fires and `force_rotate` updates the deadline to "now". + let _ = dedup.is_duplicate(&make_event(1)); + assert!( + dedup.rotation_time >= before, + "a time-driven rotation must advance the deadline to now" + ); + } + #[test] fn populate_from_events_honours_size_cap() { let max = 16; diff --git a/tidal/src/wal/diagnostics.rs b/tidal/src/wal/diagnostics.rs index 0dfb240..77df578 100644 --- a/tidal/src/wal/diagnostics.rs +++ b/tidal/src/wal/diagnostics.rs @@ -472,16 +472,17 @@ mod tests { std::fs::create_dir_all(&wal_dir).unwrap(); let mut bytes = Vec::new(); - bytes.extend(encode_session_event(&SessionWalEvent::Start { - session_id: 1, - user_id: 10, - started_at_ns: 100, - agent_id: "agent".to_string(), - policy_name: "policy".to_string(), - })); - bytes.extend(encode_session_event(&SessionWalEvent::Close { - session_id: 1, - })); + bytes.extend( + encode_session_event(&SessionWalEvent::Start { + session_id: 1, + user_id: 10, + started_at_ns: 100, + agent_id: "agent".to_string(), + policy_name: "policy".to_string(), + }) + .unwrap(), + ); + bytes.extend(encode_session_event(&SessionWalEvent::Close { session_id: 1 }).unwrap()); std::fs::write(wal_dir.join(SESSION_JOURNAL_FILENAME), &bytes).unwrap(); let report = diagnose_wal(dir.path()).unwrap(); @@ -509,8 +510,9 @@ mod tests { started_at_ns: 100, agent_id: "agent".to_string(), policy_name: "policy".to_string(), - }); - let mut corrupt = encode_session_event(&SessionWalEvent::Close { session_id: 2 }); + }) + .unwrap(); + let mut corrupt = encode_session_event(&SessionWalEvent::Close { session_id: 2 }).unwrap(); // Flip the first payload byte so the per-record BLAKE3 checksum fails. // // A v2 session frame is laid out as diff --git a/tidal/src/wal/error.rs b/tidal/src/wal/error.rs index 08c884c..7b624ab 100644 --- a/tidal/src/wal/error.rs +++ b/tidal/src/wal/error.rs @@ -10,9 +10,6 @@ pub enum WalError { /// Data corruption detected (BLAKE3 mismatch, invalid magic, etc.). #[error("WAL corruption: {message}")] Corruption { message: String }, - /// Current segment is full; internal signal to trigger rotation. - #[error("WAL segment full")] - SegmentFull, /// Invalid WAL configuration supplied at open time. /// /// Returned (rather than panicking or crashing the writer thread) when a @@ -58,11 +55,6 @@ mod tests { assert_eq!(e.to_string(), "WAL corruption: bad checksum"); } - #[test] - fn display_segment_full() { - assert_eq!(WalError::SegmentFull.to_string(), "WAL segment full"); - } - #[test] fn display_closed() { assert_eq!(WalError::Closed.to_string(), "WAL closed"); diff --git a/tidal/src/wal/format/session.rs b/tidal/src/wal/format/session.rs index 96986b2..383e01e 100644 --- a/tidal/src/wal/format/session.rs +++ b/tidal/src/wal/format/session.rs @@ -1,5 +1,7 @@ // ── Session journal record types ──────────────────────────────────────────── +use crate::wal::error::WalError; + /// Record type discriminant for session start events. pub const SESSION_RECORD_START: u8 = 0x01; /// Record type discriminant for session signal events. @@ -62,21 +64,53 @@ const SESSION_SIGNAL_FIXED_PREFIX: usize = 32; /// whatever `write_payload` appends. This is the single framing used by every /// optional Signal field (annotation, session seqno, idempotency key), so the /// `[flag][payload]` convention lives in exactly one place instead of being -/// hand-rolled per field. +/// hand-rolled per field. `write_payload` is fallible so an optional string +/// field (the annotation) can reject an over-length value via [`push_str_field`] +/// without being silently truncated. fn push_optional( buf: &mut Vec, value: Option<&T>, - write_payload: impl FnOnce(&mut Vec, &T), -) { - match value { - Some(v) => { - buf.push(1u8); - write_payload(buf, v); - } - None => buf.push(0u8), + write_payload: impl FnOnce(&mut Vec, &T) -> Result<(), WalError>, +) -> Result<(), WalError> { + if let Some(v) = value { + buf.push(1u8); + write_payload(buf, v) + } else { + buf.push(0u8); + Ok(()) } } +/// Append a length-prefixed UTF-8 string field as `[len: u16 LE][bytes]`. +/// +/// This is the single encode-side owner of the `[len: u16][bytes]` framing, +/// mirroring the decode-side [`read_str_field`]. Hand-rolling +/// `(s.len() as u16).to_le_bytes()` per field (as the four call sites used to) +/// is exactly the bug this fixes: a `len() > u16::MAX` silently *wraps* the +/// 16-bit length while the full byte payload is still appended, producing a +/// frame whose own checksum passes but that `read_str_field` reads back +/// truncated — silently dropping the over-length field's tail AND misframing +/// every later record in the journal. Encoding must never produce a record it +/// cannot faithfully decode, so we reject the over-length field here with a +/// typed error rather than casting. +/// +/// # Errors +/// +/// Returns [`WalError::Corruption`] if `s.len() > u16::MAX` (the largest length +/// the 16-bit prefix can represent). +fn push_str_field(buf: &mut Vec, field: &str, value: &str) -> Result<(), WalError> { + let len = u16::try_from(value.len()).map_err(|_| WalError::Corruption { + message: format!( + "session journal {field} field is {} bytes, exceeds the u16 length-prefix maximum of {}", + value.len(), + u16::MAX + ), + })?; + buf.extend_from_slice(&len.to_le_bytes()); + buf.extend_from_slice(value.as_bytes()); + Ok(()) +} + /// Compute the per-record checksum over the framed record body. /// /// Input is the in-`len` body excluding the trailing checksum, i.e. @@ -167,8 +201,18 @@ pub enum SessionWalEvent { /// `[if has_annotation: annotation_len: u16 LE, annotation: bytes]` /// /// **Close payload**: `[session_id: u64 LE]` -#[must_use] -pub fn encode_session_event(event: &SessionWalEvent) -> Vec { +/// +/// # Errors +/// +/// Returns [`WalError::Corruption`] if any variable-length string field +/// (`agent_id`, `policy_name`, `signal_name`, or `annotation`) is longer than +/// `u16::MAX` bytes. The length prefix is a `u16`, so an over-length field +/// could only be encoded by truncating its length count — producing a frame +/// whose checksum passes but that decodes back truncated and misframes every +/// later record. The format layer is self-defending: it refuses to emit a +/// record it cannot faithfully decode, regardless of what the write API let +/// through upstream. +pub fn encode_session_event(event: &SessionWalEvent) -> Result, WalError> { // Encode the body (marker + version + type + payload) first, then prepend // the length prefix and append the per-record checksum trailer. let mut payload = Vec::new(); @@ -186,10 +230,8 @@ pub fn encode_session_event(event: &SessionWalEvent) -> Vec { payload.extend_from_slice(&session_id.to_le_bytes()); payload.extend_from_slice(&user_id.to_le_bytes()); payload.extend_from_slice(&started_at_ns.to_le_bytes()); - payload.extend_from_slice(&(agent_id.len() as u16).to_le_bytes()); - payload.extend_from_slice(agent_id.as_bytes()); - payload.extend_from_slice(&(policy_name.len() as u16).to_le_bytes()); - payload.extend_from_slice(policy_name.as_bytes()); + push_str_field(&mut payload, "agent_id", agent_id)?; + push_str_field(&mut payload, "policy_name", policy_name)?; } SessionWalEvent::Signal { session_id, @@ -206,20 +248,20 @@ pub fn encode_session_event(event: &SessionWalEvent) -> Vec { payload.extend_from_slice(&entity_id.to_le_bytes()); payload.extend_from_slice(&weight.to_le_bytes()); payload.extend_from_slice(&ts_ns.to_le_bytes()); - payload.extend_from_slice(&(signal_name.len() as u16).to_le_bytes()); - payload.extend_from_slice(signal_name.as_bytes()); + push_str_field(&mut payload, "signal_name", signal_name)?; push_optional(&mut payload, annotation.as_ref(), |buf, ann| { - buf.extend_from_slice(&(ann.len() as u16).to_le_bytes()); - buf.extend_from_slice(ann.as_bytes()); - }); + push_str_field(buf, "annotation", ann) + })?; // m8p4: session_seqno (optional u64) push_optional(&mut payload, session_seqno.as_ref(), |buf, seqno| { buf.extend_from_slice(&seqno.0.to_le_bytes()); - }); + Ok(()) + })?; // m8p4: idempotency_key (optional u128) push_optional(&mut payload, idempotency_key.as_ref(), |buf, key| { buf.extend_from_slice(&key.to_le_bytes()); - }); + Ok(()) + })?; } SessionWalEvent::Close { session_id } => { payload.push(SESSION_RECORD_CLOSE); @@ -234,7 +276,7 @@ pub fn encode_session_event(event: &SessionWalEvent) -> Vec { buf.extend_from_slice(&len.to_le_bytes()); buf.extend(payload); buf.extend_from_slice(&checksum); - buf + Ok(buf) } /// Decode all session events from a session journal file's contents. @@ -544,11 +586,24 @@ fn decode_signal_record(bytes: &[u8], pos: &mut usize, end: usize) -> Option end { + return None; + } let v = read_u64_le(bytes, pos); Some(SessionSeqNo(v)) } else { @@ -558,11 +613,16 @@ fn decode_signal_record(bytes: &[u8], pos: &mut usize, end: usize) -> Option end { + return None; + } let v = u128::from_le_bytes([ bytes[*pos], bytes[*pos + 1], @@ -607,6 +667,15 @@ fn decode_signal_record(bytes: &[u8], pos: &mut usize, end: usize) -> Option Vec { + encode_session_event(event).expect("fixed test events encode losslessly") + } + #[test] fn session_start_roundtrip() { let event = SessionWalEvent::Start { @@ -616,7 +685,7 @@ mod tests { agent_id: "test-agent".to_string(), policy_name: "default_policy".to_string(), }; - let encoded = encode_session_event(&event); + let encoded = enc(&event); let decoded = decode_session_events(&encoded); assert_eq!(decoded.len(), 1); assert_eq!(decoded[0], event); @@ -634,7 +703,7 @@ mod tests { session_seqno: None, idempotency_key: None, }; - let encoded = encode_session_event(&event); + let encoded = enc(&event); let decoded = decode_session_events(&encoded); assert_eq!(decoded.len(), 1); assert_eq!(decoded[0], event); @@ -652,7 +721,7 @@ mod tests { session_seqno: None, idempotency_key: None, }; - let encoded = encode_session_event(&event); + let encoded = enc(&event); let decoded = decode_session_events(&encoded); assert_eq!(decoded.len(), 1); assert_eq!(decoded[0], event); @@ -661,7 +730,7 @@ mod tests { #[test] fn session_close_roundtrip() { let event = SessionWalEvent::Close { session_id: 42 }; - let encoded = encode_session_event(&event); + let encoded = enc(&event); let decoded = decode_session_events(&encoded); assert_eq!(decoded.len(), 1); assert_eq!(decoded[0], event); @@ -702,7 +771,7 @@ mod tests { let mut all_bytes = Vec::new(); for event in &events { - all_bytes.extend(encode_session_event(event)); + all_bytes.extend(enc(event)); } let decoded = decode_session_events(&all_bytes); @@ -721,7 +790,7 @@ mod tests { agent_id: "agent".to_string(), policy_name: "policy".to_string(), }; - let encoded = encode_session_event(&event); + let encoded = enc(&event); // Truncate the record mid-way. let truncated = &encoded[..encoded.len() / 2]; @@ -739,8 +808,8 @@ mod tests { agent_id: "agent".to_string(), policy_name: "policy".to_string(), }; - let mut all_bytes = encode_session_event(&e1); - let e2_bytes = encode_session_event(&e2); + let mut all_bytes = enc(&e1); + let e2_bytes = enc(&e2); // Add partial second record. all_bytes.extend_from_slice(&e2_bytes[..e2_bytes.len() / 2]); @@ -774,7 +843,7 @@ mod tests { session_seqno: Some(SessionSeqNo(7)), idempotency_key: Some(0xDEAD_BEEF_CAFE_BABE_u128), }; - let encoded = encode_session_event(&event); + let encoded = enc(&event); let decoded = decode_session_events(&encoded); assert_eq!(decoded.len(), 1); assert_eq!(decoded[0], event); @@ -793,7 +862,7 @@ mod tests { session_seqno: None, idempotency_key: None, }; - let encoded = encode_session_event(&event); + let encoded = enc(&event); let decoded = decode_session_events(&encoded); assert_eq!(decoded[0], event); } @@ -806,7 +875,7 @@ mod tests { fn encode_session_event_v1_legacy(event: &SessionWalEvent) -> Vec { // Encode a v2 record, then strip the marker/version prefix and the // checksum trailer to reconstruct the exact v1 byte layout. - let v2 = encode_session_event(event); + let v2 = enc(event); // v2 body starts after the length prefix with [marker, version, type, ...]. let body = &v2[SESSION_LEN_PREFIX_LEN..v2.len() - SESSION_CHECKSUM_LEN]; // Drop marker(1) + version(1) to leave [type, payload...]. @@ -821,7 +890,7 @@ mod tests { #[test] fn session_v2_record_has_checksum_trailer() { let event = SessionWalEvent::Close { session_id: 42 }; - let encoded = encode_session_event(&event); + let encoded = enc(&event); // len(4) + marker(1) + version(1) + type(1) + session_id(8) + checksum(8) assert_eq!( encoded.len(), @@ -866,7 +935,7 @@ mod tests { let v2_event = SessionWalEvent::Close { session_id: 1 }; let mut bytes = encode_session_event_v1_legacy(&legacy_event); - bytes.extend(encode_session_event(&v2_event)); + bytes.extend(enc(&v2_event)); let decoded = decode_session_events(&bytes); assert_eq!(decoded.len(), 2); @@ -897,8 +966,8 @@ mod tests { idempotency_key: None, }; - let r1 = encode_session_event(&e1); - let mut r2 = encode_session_event(&e2); + let r1 = enc(&e1); + let mut r2 = enc(&e2); // Corrupt the first payload byte of r2, just past the length prefix and // the marker/version/type frame header. let flip = SESSION_LEN_PREFIX_LEN + SESSION_FRAME_PREFIX_LEN; @@ -921,7 +990,7 @@ mod tests { fn session_corrupted_checksum_trailer_detected() { // Mangle only the checksum trailer (not the body): still detected. let event = SessionWalEvent::Close { session_id: 7 }; - let mut encoded = encode_session_event(&event); + let mut encoded = enc(&event); let last = encoded.len() - 1; encoded[last] ^= 0x01; @@ -942,7 +1011,7 @@ mod tests { agent_id: "agent".to_string(), policy_name: "policy".to_string(), }; - let encoded = encode_session_event(&event); + let encoded = enc(&event); let truncated = &encoded[..encoded.len() - 3]; let outcome = decode_session_events_with_diagnostics(truncated); @@ -978,7 +1047,7 @@ mod tests { ]; let mut bytes = Vec::new(); for e in &events { - bytes.extend(encode_session_event(e)); + bytes.extend(enc(e)); } let outcome = decode_session_events_with_diagnostics(&bytes); assert_eq!(outcome.events, events); @@ -1007,7 +1076,7 @@ mod tests { session_seqno: None, idempotency_key: None, }; - let encoded = encode_session_event(&event); + let encoded = enc(&event); let decoded = decode_session_events(&encoded); assert_eq!(decoded.len(), 1); match &decoded[0] { @@ -1050,9 +1119,7 @@ mod tests { frame.extend_from_slice(&payload); frame.extend_from_slice(&checksum); // Follow it with a real Close so we can prove decoding continued. - frame.extend(encode_session_event(&SessionWalEvent::Close { - session_id: 9, - })); + frame.extend(enc(&SessionWalEvent::Close { session_id: 9 })); let outcome = decode_session_events_with_diagnostics(&frame); assert!( @@ -1101,7 +1168,7 @@ mod tests { // A valid record followed by a checksum-valid-but-short v2 frame keeps // the first record and stops cleanly without flagging corruption. let e1 = SessionWalEvent::Close { session_id: 5 }; - let mut bytes = encode_session_event(&e1); + let mut bytes = enc(&e1); let mut payload = vec![ SESSION_RECORD_VERSIONED, @@ -1148,4 +1215,116 @@ mod tests { ); assert_eq!(outcome.corruption_count, 1); } + + // ── W30: over-length string fields are rejected, never silently truncated ── + + #[test] + fn session_over_length_annotation_is_rejected() { + // An annotation longer than u16::MAX bytes cannot fit the 16-bit length + // prefix. The pre-fix bug cast `len() as u16`, wrapping the count while + // appending the full bytes — a frame whose checksum passed but decoded + // back truncated, misframing every later record. The encoder must now + // reject it with a typed error rather than emit an unrecoverable record. + let event = SessionWalEvent::Signal { + session_id: 1, + entity_id: 2, + weight: 1.0, + ts_ns: 3, + signal_name: "reward".to_string(), + annotation: Some("x".repeat(70_000)), + session_seqno: None, + idempotency_key: None, + }; + let result = encode_session_event(&event); + match result { + Err(WalError::Corruption { message }) => { + assert!( + message.contains("annotation") && message.contains("70000"), + "unexpected error message: {message}" + ); + } + other => panic!("expected Corruption for an over-length annotation, got {other:?}"), + } + } + + #[test] + fn session_over_length_signal_name_is_rejected() { + let event = SessionWalEvent::Signal { + session_id: 1, + entity_id: 2, + weight: 1.0, + ts_ns: 3, + signal_name: "s".repeat(usize::from(u16::MAX) + 1), + annotation: None, + session_seqno: None, + idempotency_key: None, + }; + assert!(matches!( + encode_session_event(&event), + Err(WalError::Corruption { .. }) + )); + } + + #[test] + fn session_max_length_field_round_trips() { + // Exactly u16::MAX bytes is the largest field the prefix can represent + // and MUST round-trip losslessly — the rejection boundary is exclusive. + let annotation = "y".repeat(usize::from(u16::MAX)); + let event = SessionWalEvent::Signal { + session_id: 1, + entity_id: 2, + weight: 1.0, + ts_ns: 3, + signal_name: "view".to_string(), + annotation: Some(annotation), + session_seqno: None, + idempotency_key: None, + }; + let encoded = encode_session_event(&event).expect("u16::MAX-length field must encode"); + let decoded = decode_session_events(&encoded); + assert_eq!(decoded.len(), 1); + assert_eq!(decoded[0], event); + } + + // ── wal-format SUG: present-but-short optional field stops, not silent None ── + + #[test] + fn session_present_but_short_seqno_is_clean_stop_not_silent_none() { + // Hand-build a checksum-valid v2 Signal frame whose session_seqno flag is + // SET but whose 8-byte value is truncated. A genuine writer never emits + // this (it always writes the full value when it sets the flag); only a + // forged frame with a recomputed checksum can. The decoder must treat it + // as a structural mismatch (clean stop, no event) rather than silently + // decoding the present-but-short field to `None` and returning a Signal. + let mut payload = vec![ + SESSION_RECORD_VERSIONED, + SESSION_RECORD_VERSION_V2, + SESSION_RECORD_SIGNAL, + ]; + payload.extend_from_slice(&1u64.to_le_bytes()); // session_id + payload.extend_from_slice(&2u64.to_le_bytes()); // entity_id + payload.extend_from_slice(&1.0f64.to_le_bytes()); // weight + payload.extend_from_slice(&3u64.to_le_bytes()); // ts_ns + payload.extend_from_slice(&4u16.to_le_bytes()); // signal_name len = 4 + payload.extend_from_slice(b"view"); // signal_name + payload.push(0u8); // has_annotation = false + payload.push(1u8); // has_seqno = true ... + payload.extend_from_slice(&[0xAA, 0xBB]); // ... but only 2 of 8 bytes + let checksum = session_record_checksum(&payload); + let len = (payload.len() + SESSION_CHECKSUM_LEN) as u32; + let mut frame = Vec::new(); + frame.extend_from_slice(&len.to_le_bytes()); + frame.extend_from_slice(&payload); + frame.extend_from_slice(&checksum); + + let outcome = decode_session_events_with_diagnostics(&frame); + assert!( + outcome.events.is_empty(), + "a present-but-short optional field must NOT decode to a silent-None Signal" + ); + // Checksum-valid bytes were proven intact, so this is a structural + // (forward-compat/torn) mismatch, a clean stop — not on-disk corruption. + assert!(!outcome.corruption_detected); + assert_eq!(outcome.corruption_count, 0); + } } diff --git a/tidal/src/wal/mod.rs b/tidal/src/wal/mod.rs index 4c967f5..5fed51b 100644 --- a/tidal/src/wal/mod.rs +++ b/tidal/src/wal/mod.rs @@ -48,6 +48,57 @@ use crate::replication::{RegionId, ShardId}; /// Default channel capacity for the writer command channel. const DEFAULT_CHANNEL_CAPACITY: usize = 10_000; +/// Flush a file's data to **stable storage**, defeating the drive write cache. +/// +/// On Linux this is `File::sync_data()` (`fdatasync`), which already pushes the +/// bytes past the OS page cache to the device. On **macOS**, however, std's +/// `sync_data`/`sync_all` issues a plain `fsync(2)`, which Apple documents as +/// NOT flushing the storage device's volatile write cache — a power-loss can +/// still lose bytes the WAL has "fsynced" and acknowledged to callers. There, +/// durable crash safety requires `fcntl(fd, F_FULLFSYNC)`, which we issue via +/// rustix's safe `fcntl_fullfsync` wrapper (this crate is `unsafe_code = +/// "forbid"`, so a raw `libc::fcntl` is not an option). +/// +/// Use this for every WAL data sync that must survive a hard crash (segment +/// appends, checkpoint temp). For directory-entry durability use +/// [`sync_dir_durable`]. +pub(crate) fn sync_file_durable(file: &std::fs::File) -> Result<(), WalError> { + #[cfg(target_os = "macos")] + { + // F_FULLFSYNC flushes the device write cache; plain fsync on macOS does + // not. A safe wrapper keeps us within `unsafe_code = "forbid"`. + rustix::fs::fcntl_fullfsync(file).map_err(|e| WalError::Io(e.into()))?; + Ok(()) + } + #[cfg(not(target_os = "macos"))] + { + file.sync_data()?; + Ok(()) + } +} + +/// Make a directory's metadata (entry creations/renames/unlinks) durable. +/// +/// Same platform split as [`sync_file_durable`]: on macOS a plain directory +/// `fsync` does not flush the device cache, so we issue `F_FULLFSYNC` on the +/// directory FD; elsewhere `File::sync_all()` is sufficient. Without a durable +/// directory sync, a crash after a create/rename/unlink but before the metadata +/// reaches the platter could resurrect a deleted segment or lose a freshly +/// created one's name. +pub(crate) fn sync_dir_durable(dir: &std::path::Path) -> Result<(), WalError> { + let dir_fd = std::fs::File::open(dir)?; + #[cfg(target_os = "macos")] + { + rustix::fs::fcntl_fullfsync(&dir_fd).map_err(|e| WalError::Io(e.into()))?; + Ok(()) + } + #[cfg(not(target_os = "macos"))] + { + dir_fd.sync_all()?; + Ok(()) + } +} + /// A signal event to be appended to the WAL. /// /// This is the public write type. It maps 1:1 to the internal diff --git a/tidal/src/wal/reader.rs b/tidal/src/wal/reader.rs index e8eb445..2b8451b 100644 --- a/tidal/src/wal/reader.rs +++ b/tidal/src/wal/reader.rs @@ -23,14 +23,35 @@ pub struct RecoveryResult { /// /// Reads the checkpoint (if any), then scans all segment files from the /// checkpoint position forward. Validates each batch with two-phase -/// checking (magic + bounds, then BLAKE3). Truncates any corrupted tail. +/// checking (magic + bounds, then BLAKE3). +/// +/// # Corruption is only repairable in the FINAL segment +/// +/// A torn or corrupt *tail* is the normal signature of a crash mid-append, and +/// it can only legitimately occur in the segment a writer was actively +/// appending to — i.e. the last (highest-`first_seq`) segment. Recovery +/// truncates that torn tail and replays the rest, exactly as before. +/// +/// A tail-corrupt **non-final** segment is a different beast entirely: that +/// segment was already sealed and synced (it rotated when a higher segment was +/// created), so a missing/garbled trailing batch there is silent bit-rot, not a +/// crash artifact. Blindly truncating it and replaying the later segments would +/// produce a state with a contiguous run of *missing* sequences — silent data +/// loss in the durability-critical path, returned as a clean `Ok`. So recovery +/// surfaces it as [`WalError::Corruption`] instead, mirroring the +/// sequence-overflow handling and the session-journal corruption signal. It +/// also enforces cross-segment sequence continuity: segment N+1's first batch +/// must begin exactly where segment N ended, and a gap is likewise +/// `Corruption`. An operator then learns at `open()` that bytes were lost, rather +/// than getting a silently-truncated database. /// /// Returns the replayed events and the next sequence number to assign. /// /// # Errors /// /// Returns `WalError::Io` on filesystem failure, or `WalError::Corruption` -/// if a segment is corrupted in a way that cannot be recovered by truncation. +/// if a non-final segment has a corrupt tail, if there is a gap in the +/// cross-segment sequence space, or if a forged `first_seq` overflows `u64`. pub fn recover(dir: &Path) -> Result { // WAL open is a writable path: sweep any checkpoint temp orphaned by a crash // between a prior write and its rename. `CheckpointManager::read` is kept @@ -67,9 +88,58 @@ pub fn recover(dir: &Path) -> Result { // a segment's filename `first_seq` is only its *lowest* sequence, and it may // hold events past the checkpoint, so the post-checkpoint filter is applied // per event below (`event_seq > checkpoint_seq`), not per segment. - for (_seg_first_seq, seg_path) in &segments { - let scan_result = recover_segment(seg_path)?; - for (header, events) in scan_result { + // + // Cross-segment continuity: each non-first segment's first batch must begin + // exactly where the previous segment's last event ended. `expected_first_seq` + // tracks that boundary; a mismatch is a gap in the durable sequence space and + // is surfaced as `Corruption` rather than silently replayed past. + let last_idx = segments.len() - 1; + let mut expected_first_seq: Option = None; + for (idx, (_seg_first_seq, seg_path)) in segments.iter().enumerate() { + let is_final = idx == last_idx; + + // Scan WITHOUT mutating first, so we can decide truncate-vs-error from + // the tail-corruption flag before touching the file. Only the final + // segment's torn tail is a legitimate crash artifact we may repair. + let summary = scan_segment_summary(seg_path)?; + if summary.tail_corrupt { + if is_final { + // Crash mid-append on the live segment: repair the torn tail in + // place (single-threaded recovery owns the dir here) and keep + // the valid batches we already scanned. + recover_segment(seg_path)?; + } else { + // A sealed, already-synced segment with a corrupt tail is silent + // bit-rot, NOT a crash artifact. Truncating it would drop a + // contiguous run of sequences and leave a gap; surface it loudly. + return Err(WalError::Corruption { + message: format!( + "non-final WAL segment {} has a corrupt/truncated tail; \ + truncating it would silently drop sequences covered by \ + later segments — refusing to recover past the gap", + seg_path.display() + ), + }); + } + } + + for (header, events) in summary.batches { + // Enforce cross-segment continuity at the FIRST batch of each + // segment after the first: it must begin where the previous segment + // ended. A gap means sequences were lost between segments. + if let Some(expected) = expected_first_seq.take() + && header.first_seq != expected + { + return Err(WalError::Corruption { + message: format!( + "WAL sequence gap at segment {}: expected first_seq={expected}, \ + found {} — a contiguous run of sequences is missing", + seg_path.display(), + header.first_seq + ), + }); + } + for (i, event) in events.into_iter().enumerate() { // `header.first_seq` is read from on-disk segment bytes, which // may be corruption-controlled. Adding the in-batch index with @@ -99,6 +169,8 @@ pub fn recover(dir: &Path) -> Result { if candidate > next_seq { next_seq = candidate; } + // The next event/segment must continue from one past this one. + expected_first_seq = Some(candidate); } } } @@ -557,4 +629,118 @@ mod tests { assert_eq!(result.events.len(), 2); assert_eq!(result.next_seq, 3); } + + #[test] + fn recover_corrupt_tail_in_non_final_segment_is_corruption_not_silent_loss() { + // C17 regression: a SEALED (non-final) segment with a corrupt/truncated + // tail is silent bit-rot, not a crash artifact. Truncating it and + // replaying the later segment would drop a contiguous run of sequences + // and return a clean `Ok` — exactly the silent data loss this guards + // against. recover() must surface it as `Corruption` instead, AND must + // NOT truncate the sealed segment on disk. + let dir = tempfile::tempdir().expect("tempdir creation should succeed"); + + // Segment 1 (will be NON-final): a valid batch + garbage tail. + let events1 = vec![sample_event(1, 1000)]; + let batch1 = encode_batch(&events1, 1, 1000).expect("encode should succeed"); + let mut seg1_data = batch1; + seg1_data.extend_from_slice(&[0xDE, 0xAD, 0xBE, 0xEF, 0x00, 0x11]); + let seg1 = super::super::segment::segment_filename(ShardId::SINGLE, 1); + let seg1_path = dir.path().join(&seg1); + fs::write(&seg1_path, &seg1_data).expect("write should succeed"); + + // Segment 2 (final): a clean, valid batch at a higher first_seq. + let events2 = vec![sample_event(2, 2000)]; + let batch2 = encode_batch(&events2, 2, 2000).expect("encode should succeed"); + let seg2 = super::super::segment::segment_filename(ShardId::SINGLE, 2); + fs::write(dir.path().join(seg2), &batch2).expect("write should succeed"); + + let result = recover(dir.path()); + match result { + Err(WalError::Corruption { message }) => { + assert!( + message.contains("non-final") || message.contains("corrupt"), + "unexpected corruption message: {message}" + ); + } + other => panic!( + "expected Corruption for a tail-corrupt non-final segment, got: {:?}", + other.map(|r| r.events.len()) + ), + } + + // The sealed non-final segment must be left intact on disk — recovery + // refused, so it must not have silently truncated the bit-rotted bytes. + let seg1_len_after = fs::metadata(&seg1_path) + .expect("metadata should succeed") + .len(); + assert_eq!( + seg1_len_after, + seg1_data.len() as u64, + "a non-final corrupt segment must NOT be truncated by recover()" + ); + } + + #[test] + fn recover_cross_segment_sequence_gap_is_corruption() { + // Two intact segments, but segment 2 begins at a first_seq that leaves a + // hole after segment 1's last event. A contiguous run of sequences is + // missing between them; recover() must surface a gap rather than replay + // past it as if nothing were lost. + let dir = tempfile::tempdir().expect("tempdir creation should succeed"); + + // Segment 1: one event at seq 1 (so it ends expecting next_seq=2). + let batch1 = encode_batch(&[sample_event(1, 1000)], 1, 1000).expect("encode"); + let seg1 = super::super::segment::segment_filename(ShardId::SINGLE, 1); + fs::write(dir.path().join(seg1), &batch1).expect("write should succeed"); + + // Segment 2: starts at seq 5, skipping seqs 2,3,4 entirely. + let batch2 = encode_batch(&[sample_event(5, 5000)], 5, 5000).expect("encode"); + let seg2 = super::super::segment::segment_filename(ShardId::SINGLE, 5); + fs::write(dir.path().join(seg2), &batch2).expect("write should succeed"); + + match recover(dir.path()) { + Err(WalError::Corruption { message }) => { + assert!( + message.contains("gap") || message.contains("missing"), + "unexpected corruption message: {message}" + ); + } + other => panic!( + "expected Corruption for a cross-segment sequence gap, got: {:?}", + other.map(|r| r.events.len()) + ), + } + } + + #[test] + fn recover_two_contiguous_segments_with_torn_final_tail_succeeds() { + // The complement of the C17 case: when the torn tail is in the FINAL + // (live) segment it IS a legitimate crash artifact, so recovery repairs + // it and replays the contiguous prefix. This keeps the legitimate + // crash-mid-append path working after the non-final guard was added. + let dir = tempfile::tempdir().expect("tempdir creation should succeed"); + + let batch1 = encode_batch(&[sample_event(1, 1000)], 1, 1000).expect("encode"); + let seg1 = super::super::segment::segment_filename(ShardId::SINGLE, 1); + fs::write(dir.path().join(seg1), &batch1).expect("write should succeed"); + + // Final segment: valid batch at seq 2 + torn tail. + let batch2 = encode_batch(&[sample_event(2, 2000)], 2, 2000).expect("encode"); + let mut seg2_data = batch2.clone(); + seg2_data.extend_from_slice(&[0xDE, 0xAD, 0xBE, 0xEF]); + let seg2 = super::super::segment::segment_filename(ShardId::SINGLE, 2); + let seg2_path = dir.path().join(&seg2); + fs::write(&seg2_path, &seg2_data).expect("write should succeed"); + + let result = recover(dir.path()).expect("recover should succeed"); + assert_eq!(result.events.len(), 2); + assert_eq!(result.next_seq, 3); + + // The final segment's torn tail was repaired in place. + let seg2_len_after = fs::metadata(&seg2_path) + .expect("metadata should succeed") + .len(); + assert_eq!(seg2_len_after, batch2.len() as u64); + } } diff --git a/tidal/src/wal/segment.rs b/tidal/src/wal/segment.rs index 9634cd6..7e5cb90 100644 --- a/tidal/src/wal/segment.rs +++ b/tidal/src/wal/segment.rs @@ -172,10 +172,11 @@ impl SegmentWriter { // Fsync the parent directory so the new directory entry is durable. // Without this, a crash after file creation but before the directory - // metadata is flushed could lose the segment file entirely. + // metadata is flushed could lose the segment file entirely. On macOS a + // plain directory fsync does not flush the device cache, so this routes + // through the F_FULLFSYNC-backed helper. if is_new { - let dir_fd = File::open(dir)?; - dir_fd.sync_all()?; + super::sync_dir_durable(dir)?; } let metadata = file.metadata()?; @@ -207,31 +208,35 @@ impl SegmentWriter { /// Sync this segment's written **data** to stable storage. /// - /// Uses `File::sync_data()` (`fdatasync` on Linux, `fsync` on macOS), which - /// flushes the file's data plus the size-changing metadata needed to read - /// that data back. For an append to an **existing** segment — the steady - /// state — this is genuinely sufficient: there is no directory change, so a - /// single `sync()` makes the freshly appended batch fully durable. + /// Routes through [`crate::wal::sync_file_durable`], which flushes the file's + /// data to the device, defeating the drive write cache. On Linux that is + /// `fdatasync`; on **macOS** a plain `fsync(2)` does NOT flush the device's + /// volatile write cache (Apple documents this), so the helper issues + /// `fcntl(fd, F_FULLFSYNC)` instead — without it a power-loss could lose a + /// batch the WAL had already "fsynced" and acknowledged. For an append to an + /// **existing** segment — the steady state — this makes the freshly appended + /// batch fully durable on every supported platform: there is no directory + /// change, so a single `sync()` suffices. /// - /// It does NOT, however, make a newly created segment's **directory entry** - /// durable. `fdatasync` says nothing about the parent directory, so a crash - /// right after a `create + sync()` could lose the file's name even though - /// its bytes reached the platter. Directory-entry durability for new and - /// rotated segments is handled separately by the explicit parent-directory - /// `sync_all()` in [`SegmentWriter::open`] (on first creation) and - /// [`SegmentWriter::rotate`] (on rotation); segment *deletions* are made - /// durable by the parent-directory fsync in - /// [`crate::wal::compaction`]. Together those sites complete the durability - /// story that this method deliberately covers only the data half of — do - /// not drop them on the assumption that `sync()` already fsyncs the - /// directory. + /// It does NOT make a newly created segment's **directory entry** durable. + /// A data sync says nothing about the parent directory, so a crash right + /// after a `create + sync()` could lose the file's name even though its + /// bytes reached the platter. Directory-entry durability for new and rotated + /// segments is handled separately by the explicit parent-directory sync in + /// [`SegmentWriter::open`] (on first creation) and [`SegmentWriter::rotate`] + /// (on rotation); segment *deletions* are made durable by the + /// parent-directory fsync in [`delete_segments_before`] and + /// [`crate::wal::compaction`]. All of those route through + /// [`crate::wal::sync_dir_durable`], so the directory half of the durability + /// story is also F_FULLFSYNC-backed on macOS. Together those sites complete + /// the story this method deliberately covers only the data half of — do not + /// drop them on the assumption that `sync()` already fsyncs the directory. /// /// # Errors /// /// Returns `WalError::Io` on sync failure. pub fn sync(&self) -> Result<(), WalError> { - self.file.sync_data()?; - Ok(()) + super::sync_file_durable(&self.file) } /// Whether the segment has reached its size threshold and should be rotated. @@ -276,9 +281,9 @@ impl SegmentWriter { // Fsync the parent directory so the new segment's directory entry // is durable. Without this, a crash after file creation but before - // the directory metadata is flushed could lose the new segment. - let dir_fd = File::open(&self.dir)?; - dir_fd.sync_all()?; + // the directory metadata is flushed could lose the new segment. On + // macOS this routes through the F_FULLFSYNC-backed helper. + super::sync_dir_durable(&self.dir)?; self.file = file; self.current_size = 0; @@ -289,22 +294,94 @@ impl SegmentWriter { /// Delete all segment files whose first sequence number is less than `before_seq`. /// +/// After the unlinks, the WAL directory is fsynced via +/// [`crate::wal::sync_dir_durable`] so the deletions are durable — matching +/// [`crate::wal::compaction::compact_segments`], the other durable-delete +/// routine. Without that directory sync a crash after a `remove_file` but +/// before the directory metadata reached the platter could resurrect a deleted +/// segment file. (Resurrection is benign here — the segment is covered by the +/// checkpoint and replays idempotently — but the two deletion paths must agree +/// on durability so a reader cannot mistake the asymmetry for a bug, and so a +/// future change that makes resurrection non-idempotent does not silently +/// regress.) +/// /// # Errors /// /// Returns `WalError::Io` on filesystem failure. Partial deletion may occur -/// if an error is encountered mid-way. +/// if an error is encountered mid-way; the directory fsync still runs for any +/// unlinks that did complete. pub fn delete_segments_before(dir: &Path, before_seq: u64) -> Result { let segments = list_segments(dir)?; let mut deleted = 0; + let mut result = Ok(()); for (seq, path) in segments { if seq < before_seq { - fs::remove_file(&path)?; + if let Err(e) = fs::remove_file(&path) { + result = Err(WalError::Io(e)); + break; + } deleted += 1; } } + // Make the unlinks durable, mirroring compaction's directory fsync. Run this + // even when a mid-way unlink failed, so the deletions that DID complete are + // durable before we surface the error. + if deleted > 0 { + super::sync_dir_durable(dir)?; + } + result?; Ok(deleted) } #[cfg(test)] #[path = "segment_tests.rs"] mod tests; + +#[cfg(test)] +#[allow(clippy::unwrap_used)] +mod durable_delete_tests { + use super::*; + + /// W32 regression: `delete_segments_before` must make its unlinks durable by + /// fsyncing the WAL directory (matching `compaction::compact_segments`), so + /// the two deletion paths agree on durability. We cannot observe the fsync + /// itself without crash injection, but we pin that the routine still deletes + /// exactly the older segments and that the directory-fsync code path executes + /// cleanly (it would error if the helper or its dir-open regressed). + #[test] + fn delete_segments_before_deletes_older_and_dir_sync_runs() { + let dir = tempfile::tempdir().expect("tempdir creation should succeed"); + for &seq in &[1u64, 50, 100] { + let _ = SegmentWriter::open(dir.path(), ShardId::SINGLE, seq, 1024) + .expect("seed segment open should succeed"); + } + + // floor = 100 -> seqs 1 and 50 are deleted, 100 survives. The directory + // fsync runs because deleted > 0; a regression in the helper or its + // dir-open would surface as an Err here. + let deleted = delete_segments_before(dir.path(), 100).expect("delete should succeed"); + assert_eq!(deleted, 2); + + let remaining = list_segments(dir.path()).expect("list should succeed"); + assert_eq!(remaining.len(), 1); + assert_eq!(remaining[0].0, 100); + } + + /// When nothing matches the floor, no directory fsync is issued (deleted == + /// 0) and the routine is a clean no-op. + #[test] + fn delete_segments_before_noop_skips_dir_sync() { + let dir = tempfile::tempdir().expect("tempdir creation should succeed"); + let _ = SegmentWriter::open(dir.path(), ShardId::SINGLE, 100, 1024) + .expect("seed segment open should succeed"); + + let deleted = delete_segments_before(dir.path(), 50).expect("delete should succeed"); + assert_eq!(deleted, 0); + assert_eq!( + list_segments(dir.path()) + .expect("list should succeed") + .len(), + 1 + ); + } +} diff --git a/tidal/src/wal/session_journal.rs b/tidal/src/wal/session_journal.rs index ba389f8..8b5f25a 100644 --- a/tidal/src/wal/session_journal.rs +++ b/tidal/src/wal/session_journal.rs @@ -14,8 +14,9 @@ use std::{ path::{Path, PathBuf}, }; -use super::format::{ - SessionWalEvent, decode_session_events_with_diagnostics, encode_session_event, +use super::{ + error::WalError, + format::{SessionWalEvent, decode_session_events_with_diagnostics, encode_session_event}, }; /// The session journal file name within the WAL directory. @@ -72,20 +73,27 @@ impl SessionJournal { /// Append a session event to the journal and fsync. /// - /// Durability semantics are unchanged from the previous buffered - /// implementation: the record is fully written and `sync_data`-flushed to - /// stable storage before this returns. A crash after return guarantees the - /// record is recoverable; a crash before return leaves at most a torn tail, - /// which [`recover`](Self::recover) discards cleanly. + /// The record is fully written and durably flushed to stable storage before + /// this returns. A crash after return guarantees the record is recoverable; + /// a crash before return leaves at most a torn tail, which + /// [`recover`](Self::recover) discards cleanly. The flush routes through + /// [`crate::wal::sync_file_durable`], so on macOS it issues `F_FULLFSYNC` + /// (defeating the device write cache) rather than a plain `fsync` that could + /// lose an acknowledged record on power loss. /// /// # Errors /// - /// Returns `std::io::Error` on write or sync failure. - pub fn append(&mut self, event: &SessionWalEvent) -> Result<(), std::io::Error> { - let encoded = encode_session_event(event); + /// Returns [`WalError::Corruption`] if the event cannot be faithfully + /// encoded (a string field longer than `u16::MAX`; see + /// [`encode_session_event`]), or [`WalError::Io`] on write or sync failure. + /// The fallible encode means an oversized field is surfaced to the caller + /// instead of silently writing a record the decoder cannot read back. + pub fn append(&mut self, event: &SessionWalEvent) -> Result<(), WalError> { + let encoded = encode_session_event(event)?; self.file.write_all(&encoded)?; - // fsync the file descriptor for durability before returning. - self.file.sync_data()?; + // Durable flush before returning (F_FULLFSYNC on macOS, fdatasync + // elsewhere) so an acknowledged session record survives a hard crash. + crate::wal::sync_file_durable(&self.file)?; Ok(()) } diff --git a/tidal/src/wal/writer.rs b/tidal/src/wal/writer.rs index 2908595..da17ee0 100644 --- a/tidal/src/wal/writer.rs +++ b/tidal/src/wal/writer.rs @@ -649,3 +649,75 @@ fn handle_session_command(cmd: WalCommand, journal: &mut Option) #[allow(clippy::unwrap_used, clippy::similar_names)] #[path = "writer_tests.rs"] mod tests; + +#[cfg(test)] +#[allow(clippy::unwrap_used)] +mod rotation_failure_tests { + use crossbeam::channel::bounded; + + use super::*; + use crate::wal::segment::{SegmentWriter, segment_filename}; + + fn make_event(id: u64) -> EventRecord { + EventRecord::signal(id, 1, 1.0, 1_000_000_000) + } + + /// wal-write SUG: cover a flush failure that lands specifically inside + /// `rotate()` (old segment synced, new segment file fails to open), not just + /// the pre-encode fault the `FAIL_NEXT_FLUSH` hook injects. + /// + /// We force the failure deterministically with real filesystem semantics: + /// `max_size = 0` makes `needs_rotation()` true on the first flush, and we + /// pre-create a *directory* at the exact path the new segment file would take + /// (`wal-{batch_seq:020}.seg`) so `rotate()`'s `OpenOptions::open` fails. The + /// contract under test is the same as every other flush-error path: the + /// caller's reply channel must carry the error rather than being dropped (a + /// dropped channel would surface as a misleading `Closed`). + #[test] + fn flush_batch_error_inside_rotate_notifies_callers() { + let dir = tempfile::tempdir().expect("tempdir creation should succeed"); + + // max_size = 0 -> needs_rotation() is true immediately, so the very first + // flush attempts a rotate before writing. + let mut segment = + SegmentWriter::open(dir.path(), ShardId::SINGLE, 1, 0).expect("open should succeed"); + + let config = WriterConfig { + dir: dir.path().to_path_buf(), + segment_size: 0, + batch_size: 100, + batch_timeout: Duration::from_millis(10), + dedup_window: Duration::from_secs(30), + session_journal_path: None, + shard_id: ShardId::SINGLE, + region_id: RegionId::SINGLE, + }; + + // flush_batch rotates to `segment_filename(SINGLE, batch_seq)` where + // batch_seq = 1 (matching SegmentWriter::open's first_seq). Pre-create a + // DIRECTORY at that path so opening it as a file fails — a real, not + // mocked, I/O fault landing inside rotate(). + let collision = dir.path().join(segment_filename(ShardId::SINGLE, 1)); + std::fs::remove_file(&collision).expect("seed segment file should exist to remove"); + std::fs::create_dir(&collision).expect("create blocking directory should succeed"); + + let (reply_tx, reply_rx) = bounded(1); + let kept_events = vec![make_event(1)]; + let kept_replies = vec![reply_tx]; + + let result = flush_batch(&mut segment, &config, 1, &kept_events, kept_replies); + + assert!( + result.is_err(), + "a rotate() failure inside flush must propagate, got {result:?}" + ); + // The caller must be notified with the error, not left on a dropped channel. + let reply = reply_rx + .recv() + .expect("reply channel must NOT be dropped when rotate() fails"); + assert!( + matches!(reply, Err(WalError::Io(_))), + "caller must receive the rotate I/O error, got {reply:?}" + ); + } +} diff --git a/tidal/tests/m8p3_crdt.rs b/tidal/tests/m8p3_crdt.rs index 98f3331..444e9ff 100644 --- a/tidal/tests/m8p3_crdt.rs +++ b/tidal/tests/m8p3_crdt.rs @@ -357,9 +357,12 @@ proptest! { fn hard_neg_latest_always_wins( writes in prop::collection::vec(arb_action_write(), 2..=10), ) { - // Find the expected winner: the write with the maximum timestamp. + // Find the expected winner: the write with the maximum timestamp, with + // an exact-timestamp tie broken on the larger action value — matching + // the commutative tie-break LWWRegister::merge now applies so the law + // holds even on the (otherwise unreachable for honest HLCs) collision. let (expected_action, expected_ts) = writes.iter() - .max_by_key(|(_, ts)| *ts) + .max_by(|a, b| a.1.cmp(&b.1).then_with(|| a.0.cmp(&b.0))) .unwrap(); // Build individual registers and merge them all together. diff --git a/tidal/tests/m8p5_multitenancy.rs b/tidal/tests/m8p5_multitenancy.rs index 8c9d74d..cf0a1d3 100644 --- a/tidal/tests/m8p5_multitenancy.rs +++ b/tidal/tests/m8p5_multitenancy.rs @@ -263,7 +263,9 @@ fn test_dual_write_routing_returns_both_shards() { /// `signal_for_tenant` must FAIL CLOSED when a dual-write resolves a non-local /// shard while cross-node dispatch is unwired — never silently drop the remote -/// half. The local half is still written before the error surfaces. +/// half. No write happens — the op fails closed atomically (it detects the +/// non-local assignment before touching the ledger), so a retry cannot +/// double-count the local signal. #[test] fn test_signal_for_tenant_fails_closed_on_unwired_remote_shard() { use std::time::Duration; @@ -312,15 +314,15 @@ fn test_signal_for_tenant_fails_closed_on_unwired_remote_shard() { "expected Internal (fail-closed) error, got: {err}" ); - // The LOCAL half was still applied before the error: the signal is visible - // on this node even though the remote dispatch failed closed. + // NOTHING was written: the op fails closed atomically before any ledger + // mutation, so a caller that retries on Err cannot double-count the local + // signal. The item therefore has no decay score on this node. let score = db .read_decay_score(item, "view", 0) - .expect("read local decay score") - .expect("entity must have a decay score after the local write"); + .expect("read local decay score"); assert!( - score > 0.0, - "local-shard half of the dual-write must still be applied (got score {score})" + score.is_none() || matches!(score, Some(s) if s == 0.0), + "fail-closed must apply zero writes (got score {score:?})" ); } diff --git a/tidal/tests/review_pass2_creator_search_filter.rs b/tidal/tests/review_pass2_creator_search_filter.rs new file mode 100644 index 0000000..ccffe2e --- /dev/null +++ b/tidal/tests/review_pass2_creator_search_filter.rs @@ -0,0 +1,112 @@ +#![allow(clippy::unwrap_used)] +//! M0–M10 review pass 2, zone C regression: a creator SEARCH with a +//! CategoryEq/FormatEq filter must NOT be intersected against the ITEM-scoped +//! bitmap indexes. +//! +//! Before the fix, `apply_metadata_filter` (Stage 2) ran a `FilterEvaluator` over +//! the item category/format indexes and `candidates.retain(|id| bitmap.contains(id))` +//! against CREATOR ids — the wrong namespace. In a mixed DB only creators whose +//! id coincidentally aliased an item in that category survived; in a creator-only +//! DB every creator was dropped. The Stage 2b creator-metadata post-filter (an +//! AND-style retain) cannot resurrect candidates Stage 2 already removed, so a +//! documented feature ("filter creators by category") returned zero/arbitrary +//! results with no error. This test builds a DB with BOTH items (in category +//! "jazz") and creators (with category "news") and proves the matching creators +//! are returned. + +use std::{collections::HashMap, time::Duration}; + +use tidaldb::{ + TidalDb, + query::search::Search, + schema::{DecaySpec, EntityId, EntityKind, SchemaBuilder, TextFieldType, Window}, + storage::indexes::filter::FilterExpr, +}; + +fn schema() -> tidaldb::schema::Schema { + let mut builder = SchemaBuilder::new(); + let _ = builder + .signal( + "view", + EntityKind::Item, + DecaySpec::Exponential { + half_life: Duration::from_secs(7 * 24 * 3600), + }, + ) + .windows(&[Window::AllTime]) + .add(); + let _ = builder + .signal( + "follow", + EntityKind::Creator, + DecaySpec::Exponential { + half_life: Duration::from_secs(30 * 24 * 3600), + }, + ) + .windows(&[Window::AllTime]) + .add(); + // Creators are retrieved by BM25 over their "name" text field. + builder.creator_text_field("name", TextFieldType::Text); + builder.build().unwrap() +} + +#[test] +fn creator_search_by_category_returns_matching_creators() { + let db = TidalDb::builder() + .ephemeral() + .with_schema(schema()) + .open() + .unwrap(); + + // ITEMS in category "jazz" — these populate the item-scoped category index. + // Their ids deliberately overlap the creator id range so a (buggy) intersection + // against the item index would alias creator ids onto these item slots. + for id in 1..=3u64 { + let mut item_meta = HashMap::new(); + item_meta.insert("title".to_string(), format!("jazz track {id}")); + item_meta.insert("category".to_string(), "jazz".to_string()); + db.write_item_with_metadata(EntityId::new(id), &item_meta) + .unwrap(); + } + + // CREATORS with category metadata "news". Their names all contain "news" so + // a BM25 query on "news" retrieves them. + for id in 1..=3u64 { + let mut creator_meta = HashMap::new(); + creator_meta.insert("name".to_string(), format!("news anchor {id}")); + creator_meta.insert("category".to_string(), "news".to_string()); + db.write_creator(EntityId::new(id), &creator_meta).unwrap(); + } + // One creator in a different category — must be excluded by the filter. + let mut sports_meta = HashMap::new(); + sports_meta.insert("name".to_string(), "news roundup 99".to_string()); + sports_meta.insert("category".to_string(), "sports".to_string()); + db.write_creator(EntityId::new(99), &sports_meta).unwrap(); + + db.flush_creator_text_index().unwrap(); + + // SEARCH creators for "news" filtered to category "news". The matching news + // creators (1, 2, 3) must be returned — NOT dropped by an item-index + // intersection (the item category index holds only "jazz" item ids). + let query = Search::builder() + .entity_kind(EntityKind::Creator) + .query("news") + .filter(FilterExpr::eq("category", "news")) + .limit(20) + .build() + .unwrap(); + + let results = db.search(&query).unwrap(); + let mut ids: Vec = results.items.iter().map(|r| r.entity_id.as_u64()).collect(); + ids.sort_unstable(); + assert_eq!( + ids, + vec![1, 2, 3], + "creators in category 'news' must be returned (not dropped by the item-index \ + intersection); the 'sports' creator 99 is correctly excluded by the metadata filter. \ + Got {ids:?}, warnings: {:?}", + results.warnings + ); + + db.close().unwrap(); +} diff --git a/tidal/tests/review_pass2_d_replication.rs b/tidal/tests/review_pass2_d_replication.rs new file mode 100644 index 0000000..5794c37 --- /dev/null +++ b/tidal/tests/review_pass2_d_replication.rs @@ -0,0 +1,110 @@ +//! Review pass-2, Zone D (Replication + tidal-net transport) regression tests. +//! +//! W8: `signal_for_tenant` must fail CLOSED *before any write* when a dual-write +//! resolves a non-local shard, so a caller that retries the returned error +//! cannot double-count the local half (signal accumulation is not idempotent). +//! The companion case proves a fully-local resolution still writes. + +#![allow(clippy::unwrap_used)] + +use std::time::Duration; + +use tidaldb::{ + TidalDb, TidalError, + replication::{RegionId, ShardAssignment, ShardId, TenantId}, + schema::{DecaySpec, EntityId, EntityKind, SchemaBuilder, Timestamp, Window}, +}; + +fn open_db_with_view_signal() -> TidalDb { + let mut builder = SchemaBuilder::new(); + let _ = builder + .signal( + "view", + EntityKind::Item, + DecaySpec::Exponential { + half_life: Duration::from_secs(7 * 24 * 3600), + }, + ) + .windows(&[Window::AllTime]) + .velocity(false) + .add(); + let schema = builder.build().unwrap(); + + TidalDb::builder() + .ephemeral() + .with_schema(schema) + .open() + .expect("ephemeral db opens") +} + +/// W8: a dual-write that resolves a non-local shard must fail closed with ZERO +/// local writes — the previous behavior wrote the local half and then errored, +/// so a retry double-counted the signal. After the error, the entity must have +/// NO decay score, proving the local half was never applied. +#[test] +fn signal_for_tenant_remote_shard_fails_closed_with_zero_writes() { + let db = open_db_with_view_signal(); + + // This node is the default local shard (ShardId::SINGLE = 0). Add a second + // shard and put the default tenant into dual-write so a write resolves BOTH + // the local shard 0 and the non-local shard 1. + db.control_plane().update_topology(ShardAssignment { + shard_id: ShardId(1), + region_id: RegionId(1), + }); + db.tenant_router() + .set_dual_write(TenantId::DEFAULT, ShardId(0), ShardId(1)); + + let item = EntityId::new(42); + let err = db + .signal_for_tenant(TenantId::DEFAULT, "view", item, 1.0, Timestamp::now()) + .expect_err("remote-shard dual-write must fail closed, not silently drop"); + assert!( + matches!(err, TidalError::Internal(_)), + "expected Internal (fail-closed) error, got: {err}" + ); + + // The local half must NOT have been applied: a retry of the Err cannot + // double-count because nothing was written. The entity has no decay score. + let score = db + .read_decay_score(item, "view", 0) + .expect("read local decay score"); + assert!( + score.is_none() || score == Some(0.0), + "fail-closed must write nothing locally; got score {score:?}" + ); + + // Retrying the failed op repeatedly must still leave nothing written — the + // exact retry-amplification the W8 fix removes. + for _ in 0..5 { + let _ = db.signal_for_tenant(TenantId::DEFAULT, "view", item, 1.0, Timestamp::now()); + } + let score_after_retries = db + .read_decay_score(item, "view", 0) + .expect("read local decay score after retries"); + assert!( + score_after_retries.is_none() || score_after_retries == Some(0.0), + "retries of a fail-closed op must not accumulate any local weight; got {score_after_retries:?}" + ); +} + +/// Companion: when every resolved assignment is local (normal, non-migrating +/// tenant) the local write still lands — fail-closed only triggers on a +/// non-local assignment. +#[test] +fn signal_for_tenant_all_local_writes_succeed() { + let db = open_db_with_view_signal(); + + let item = EntityId::new(7); + db.signal_for_tenant(TenantId::DEFAULT, "view", item, 1.0, Timestamp::now()) + .expect("a fully-local tenant write must succeed"); + + let score = db + .read_decay_score(item, "view", 0) + .expect("read local decay score") + .expect("a local write must produce a decay score"); + assert!( + score > 0.0, + "local write must accumulate a positive score, got {score}" + ); +} diff --git a/tidal/tests/review_pass2_query_for_session.rs b/tidal/tests/review_pass2_query_for_session.rs new file mode 100644 index 0000000..77da30e --- /dev/null +++ b/tidal/tests/review_pass2_query_for_session.rs @@ -0,0 +1,90 @@ +#![allow(clippy::unwrap_used, clippy::doc_markdown)] +//! M0–M10 review pass 2, zone C regression: FOR SESSION missing-session handling +//! must NOT diverge between RETRIEVE and SEARCH. +//! +//! A RETRIEVE/SEARCH with `FOR SESSION ` pointing at an expired/evicted +//! session is a WELL-FORMED query. Per CODING_GUIDELINES §6 ("graceful +//! degradation, never failure") the engine must execute it WITHOUT the session +//! boost rather than hard-erroring. Before the fix RETRIEVE returned +//! `Err(SessionNotFound)` while the structurally-identical SEARCH degraded — a +//! confusing surface-specific outage the moment a session was swept. This test +//! proves both surfaces now degrade identically. + +use std::{collections::HashMap, time::Duration}; + +use tidaldb::{ + SessionId, TidalDb, + query::{retrieve::Retrieve, search::Search}, + schema::{DecaySpec, EntityId, EntityKind, SchemaBuilder, Timestamp, Window}, +}; + +fn test_schema() -> tidaldb::schema::Schema { + let mut builder = SchemaBuilder::new(); + let _ = builder + .signal( + "view", + EntityKind::Item, + DecaySpec::Exponential { + half_life: Duration::from_secs(7 * 24 * 3600), + }, + ) + .windows(&[Window::AllTime]) + .add(); + builder.build().unwrap() +} + +fn test_db() -> TidalDb { + TidalDb::builder() + .ephemeral() + .with_schema(test_schema()) + .open() + .unwrap() +} + +#[test] +fn retrieve_and_search_degrade_identically_on_missing_session() { + let db = test_db(); + + // Seed one item with a `view` signal so both queries have something to rank. + let now = Timestamp::now(); + db.signal("view", EntityId::new(1), 1.0, now).unwrap(); + let mut meta = HashMap::new(); + meta.insert("title".to_string(), "jazz piano".to_string()); + db.write_item_with_metadata(EntityId::new(1), &meta) + .unwrap(); + + // A session id that was never started (the "swept session" case). + let missing = SessionId::from_raw(9_999_999); + + // RETRIEVE FOR SESSION must NOT error — it degrades to no-boost. + let retrieve = Retrieve::builder() + .profile("new") + .for_session(missing) + .limit(10) + .build() + .unwrap(); + let r = db.retrieve(&retrieve); + assert!( + r.is_ok(), + "RETRIEVE with a missing session must degrade gracefully, got {:?}", + r.err() + ); + + // SEARCH FOR SESSION must likewise degrade (this already worked; + // asserted here to lock the two surfaces together). + let search = Search::builder() + .query("jazz") + .using_profile("search") + .for_session(missing) + .limit(10) + .build() + .unwrap(); + let s = db.search(&search); + assert!( + s.is_ok(), + "SEARCH with a missing session must degrade gracefully, got {:?}", + s.err() + ); + + db.close().unwrap(); +} diff --git a/tidal/tests/review_pass2_storage_indexes_bitmap_cache.rs b/tidal/tests/review_pass2_storage_indexes_bitmap_cache.rs new file mode 100644 index 0000000..e788bf3 --- /dev/null +++ b/tidal/tests/review_pass2_storage_indexes_bitmap_cache.rs @@ -0,0 +1,56 @@ +#![allow(clippy::doc_markdown)] +//! Regression test for the pass-2 review finding: +//! "storage-indexes — total_count() and selectivity() recompute a full bitmap +//! union on every call." +//! +//! `BitmapIndex::total_count()` is now memoized (the union over every value +//! bitmap is computed at most once per mutation). This test pins the +//! correctness side of that optimization: the memoized count must always agree +//! with the freshly-recomputed union across a full insert / delete / +//! delete_entity cycle, including a cloned handle that shares the memo. + +use tidaldb::storage::indexes::BitmapIndex; + +#[test] +fn total_count_cache_tracks_mutations() { + let index = BitmapIndex::new("category"); + // Primed-empty cache returns 0 without a recompute. + assert_eq!(index.total_count(), 0); + + index.insert(1, "jazz"); + index.insert(2, "blues"); + index.insert(3, "jazz"); + // Invalidated by each insert, recomputed once, then memoized. + assert_eq!(index.total_count(), 3); + assert_eq!(index.total_count(), 3); // fast path agrees with slow path + + // Adding entity 1 to a SECOND value bitmap does not change the distinct + // count: it is still three distinct entities in the union. + index.insert(1, "blues"); + assert_eq!(index.total_count(), 3); + + // Removing entity 1 from only one of its two value bitmaps keeps it in the + // union (still present under "jazz"), so the distinct count is unchanged. + assert!(index.delete(1, "blues")); + assert_eq!(index.total_count(), 3); + + // delete_entity scrubs every value bitmap, so the count finally drops. + assert!(index.delete_entity(1) > 0); + assert_eq!(index.total_count(), 2); +} + +#[test] +fn cloned_handle_shares_total_count_memo() { + // BitmapIndex::clone shares the underlying values Arc AND the memo Arc, so a + // mutation through one handle must invalidate the other handle's cache. + let a = BitmapIndex::new("format"); + a.insert(10, "video"); + assert_eq!(a.total_count(), 1); + + let b = a.clone(); + assert_eq!(b.total_count(), 1); + + // Mutate through `a`; `b` must observe the change, not a stale memo. + a.insert(20, "audio"); + assert_eq!(b.total_count(), 2, "clone must see the shared invalidation"); +} diff --git a/tidal/tests/review_pass2_zone_a_sessions.rs b/tidal/tests/review_pass2_zone_a_sessions.rs new file mode 100644 index 0000000..3dae526 --- /dev/null +++ b/tidal/tests/review_pass2_zone_a_sessions.rs @@ -0,0 +1,249 @@ +//! Zone A (session lifecycle & sweeper) regression tests for the +//! M0-M10 code-review pass-2 findings. +//! +//! These are end-to-end persistence tests: they open a persistent database, +//! exercise the session lifecycle, shut down, and reopen — the only way to +//! reproduce the two BLOCKERs and the CRITICALs, which only manifest across a +//! restart. + +#![allow( + clippy::unwrap_used, + clippy::too_many_lines, + clippy::doc_markdown, + clippy::needless_collect +)] + +use std::{collections::HashMap, time::Duration}; + +use tidaldb::{ + AgentPolicy, SessionId, TidalDb, + schema::{DecaySpec, EntityId, EntityKind, SchemaBuilder, Timestamp}, +}; + +/// Schema with one signal type and a single long-TTL session policy named +/// "agent", used by every test in this file. +fn schema() -> tidaldb::schema::Schema { + let mut b = SchemaBuilder::new(); + let _ = b + .signal("view", EntityKind::Item, DecaySpec::Permanent) + .add(); + b.session_policy( + "agent", + AgentPolicy { + allowed_signals: vec!["view".to_string()], + denied_signals: vec![], + max_session_duration: Duration::from_secs(3600), + max_signals_per_session: 10_000, + }, + ); + b.build().unwrap() +} + +/// BLOCKER regression: a clean restart must NOT re-issue an archived session id +/// and overwrite its durable snapshot. +/// +/// Reproduces the reviewer's two assertions: +/// (1) a new session after reopen gets an id strictly greater than the +/// previously-closed id, and +/// (2) `session_snapshot` of the old id still returns the ORIGINAL +/// user_id / signals_written — the archive was not destroyed. +#[test] +fn reopen_does_not_reissue_archived_session_id() { + let dir = tempfile::tempdir().unwrap(); + + // Phase 1: start + signal (user 1, 5 signals) + close, then shut down. + let first_id; + { + let db = TidalDb::builder() + .with_data_dir(dir.path()) + .with_schema(schema()) + .open() + .unwrap(); + + let handle = db + .start_session(1, "agent", "agent", HashMap::new()) + .unwrap(); + first_id = handle.id; + for i in 1..=5u64 { + db.session_signal( + &handle, + "view", + EntityId::new(i), + 1.0, + Timestamp::now(), + None, + ) + .unwrap(); + } + db.close_session(handle).unwrap(); + + // Snapshot is archived with the true author + count. + let snap = db.session_snapshot(first_id).unwrap(); + assert_eq!(snap.user_id, 1); + assert_eq!(snap.signals_written, 5); + + db.close().unwrap(); + } + + // Phase 2: reopen and start a NEW session for a different user. + { + let db = TidalDb::builder() + .with_data_dir(dir.path()) + .with_schema(schema()) + .open() + .unwrap(); + + let handle = db + .start_session(2, "agent", "agent", HashMap::new()) + .unwrap(); + let second_id = handle.id; + + // (1) The new id must be strictly greater than the archived id — the + // allocator was advanced past the durable snapshot on reopen. + assert!( + second_id.as_u64() > first_id.as_u64(), + "reopened session id {second_id} must exceed the archived id {first_id}; \ + the allocator was reseeded to 1 and re-issued the archived id" + ); + + db.close_session(handle).unwrap(); + + // (2) The original archived snapshot is intact — not overwritten by the + // reused id's new (user 2, 0-signal) close. + let snap = db.session_snapshot(first_id).unwrap(); + assert_eq!( + snap.user_id, 1, + "the user-1 archived snapshot must survive a restart" + ); + assert_eq!( + snap.signals_written, 5, + "the user-1 archived signal count must survive a restart" + ); + + db.close().unwrap(); + } +} + +/// End-to-end companion to the in-module "snapshot is authoritative" guard: +/// after a clean close + reopen, the session is absent from `active_sessions()` +/// and resolves only as an archived snapshot. (The torn-Close phantom-state +/// case — snapshot present AND Start-without-Close in the journal — is covered +/// directly by `session_restore.rs::durable_snapshot_blocks_active_restore`, +/// which can inject that exact state through the internal restore path.) +#[test] +fn closed_session_is_not_restored_as_active() { + let dir = tempfile::tempdir().unwrap(); + + let closed_id; + { + let db = TidalDb::builder() + .with_data_dir(dir.path()) + .with_schema(schema()) + .open() + .unwrap(); + + let handle = db + .start_session(1, "agent", "agent", HashMap::new()) + .unwrap(); + closed_id = handle.id; + db.session_signal( + &handle, + "view", + EntityId::new(1), + 1.0, + Timestamp::now(), + None, + ) + .unwrap(); + db.close_session(handle).unwrap(); + db.close().unwrap(); + } + + // Reopen: the durable snapshot for `closed_id` is present. Even if a stale + // Start-without-Close survived in the journal, the snapshot is authoritative + // and the session must NOT come back as active. + { + let db = TidalDb::builder() + .with_data_dir(dir.path()) + .with_schema(schema()) + .open() + .unwrap(); + + let active_ids: Vec = db + .active_sessions() + .into_iter() + .map(|s| s.id.as_u64()) + .collect(); + assert!( + !active_ids.contains(&closed_id.as_u64()), + "a closed session (durable snapshot present) must never restore as active" + ); + + // It is still resolvable as a closed/archived session. + let snap = db.session_snapshot(closed_id).unwrap(); + assert_eq!(snap.id, closed_id); + + db.close().unwrap(); + } +} + +/// BLOCKER/CRITICAL regression: a session that was already past its +/// `max_session_duration` at shutdown must be TTL-reaped on the first startup +/// sweep after reopen — the durable age (`started_at_ns`) survives the restart, +/// so the back-dated monotonic clock makes the sweeper see the true age. +/// +/// Uses a fresh session under the public API; the focused unit tests in +/// `sweeper.rs` and `policy.rs` cover the simulated-2h-old-restore arithmetic +/// directly (the public API cannot dial `started_at_ns` into the past). Here we +/// assert the live invariant: a session within its TTL survives a sweep and is +/// listed active after reopen, proving restore back-dates rather than expiring +/// a perfectly fresh session. +#[test] +fn restored_session_within_ttl_survives_sweep() { + let dir = tempfile::tempdir().unwrap(); + let session_id: SessionId; + { + let db = TidalDb::builder() + .with_data_dir(dir.path()) + .with_schema(schema()) + .open() + .unwrap(); + let handle = db + .start_session(1, "agent", "agent", HashMap::new()) + .unwrap(); + session_id = handle.id; + db.session_signal( + &handle, + "view", + EntityId::new(1), + 1.0, + Timestamp::now(), + None, + ) + .unwrap(); + // Deliberately do NOT close — leave it active so the WAL replays it. + db.close().unwrap(); + } + + { + let db = TidalDb::builder() + .with_data_dir(dir.path()) + .with_schema(schema()) + .open() + .unwrap(); + // The fresh, within-TTL session is restored active and survives a sweep + // (back-dated started_at reflects its true near-zero age, not an instant + // expiry from a misread clock). + db.force_sweep(); + let active: Vec = db + .active_sessions() + .into_iter() + .map(|s| s.id.as_u64()) + .collect(); + assert!( + active.contains(&session_id.as_u64()), + "a within-TTL restored session must survive the startup sweep" + ); + db.close().unwrap(); + } +} diff --git a/tidal/tests/storage.rs b/tidal/tests/storage.rs index e7f5053..58d95ed 100644 --- a/tidal/tests/storage.rs +++ b/tidal/tests/storage.rs @@ -260,7 +260,6 @@ fn storage_error_display_all_variants() { assert!(err.to_string().contains("bad data")); assert_eq!(StorageError::Closed.to_string(), "storage closed"); - assert_eq!(StorageError::BatchConflict.to_string(), "batch conflict"); } // ============================================================================= diff --git a/tidalctl/src/commands/scope_stats.rs b/tidalctl/src/commands/scope_stats.rs index 7284792..ba1f1a8 100644 --- a/tidalctl/src/commands/scope_stats.rs +++ b/tidalctl/src/commands/scope_stats.rs @@ -2,7 +2,7 @@ use serde::Serialize; -use crate::{CliError, json::render_json}; +use crate::{CliError, EXIT_DEGRADED, json::render_json}; /// Per-scope signal-event tally over a WAL directory. #[derive(Default)] @@ -27,11 +27,50 @@ impl ScopeStats { } } +/// Tri-state integrity of the WAL scan, so the operator can distinguish a +/// clean scan from a torn tail from one whose integrity could NOT be verified +/// at all (e.g. a corrupt `checkpoint.meta` that makes the cross-check fail). +/// +/// Folding `unverified` into `clean` is exactly the false positive W29 fixes: +/// the tally read may succeed off the segments while the integrity diagnosis +/// itself errors out, so claiming `complete` then is silently-wrong reporting +/// on a durability-adjacent surface. +#[derive(Serialize, Clone, Copy, PartialEq, Eq, Debug)] +#[serde(rename_all = "snake_case")] +enum Integrity { + /// Diagnosis succeeded with zero inconsistencies — the tally is whole. + Clean, + /// Diagnosis succeeded but found a torn/corrupt tail — the tally undercounts. + Torn, + /// Diagnosis itself failed (e.g. corrupt `checkpoint.meta`) — integrity + /// could not be proven, so the tally must NOT be claimed complete. + Unverified, +} + +impl Integrity { + /// Only a proven-clean scan is `complete`. A torn tail OR an unverifiable + /// store both yield `false` — we never assert completeness we cannot prove. + const fn is_complete(self) -> bool { + matches!(self, Self::Clean) + } + + const fn describe(self) -> &'static str { + match self { + Self::Clean => "tally covers every WAL batch", + Self::Torn => "torn/corrupt tail — tally undercounts", + Self::Unverified => "integrity could not be verified — do not trust the tally", + } + } +} + /// `scope-stats` command output: the per-scope tally plus a tail-integrity /// indicator so the operator can tell whether the tally is complete. #[derive(Serialize)] struct ScopeStatsOutput { - wal_segments: usize, + /// Number of WAL segment files, or `None` (JSON `null`, pretty `"unknown"`) + /// when the segment listing could not be read — never a fabricated `0` that + /// reads as "no segments" while a non-zero `total_events` was just tallied. + wal_segments: Option, scope_stats: ScopeStatsBody, } @@ -49,7 +88,16 @@ impl ScopeStatsOutput { out.push_str("WAL Scope Stats\n"); out.push_str("===============\n\n"); - let _ = writeln!(out, "WAL Segments: {}", self.wal_segments); + match self.wal_segments { + Some(n) => { + let _ = writeln!(out, "WAL Segments: {n}"); + } + // Honest "unknown" rather than a fabricated 0: the listing failed + // after the events were already tallied (e.g. a concurrent rollover). + None => { + let _ = writeln!(out, "WAL Segments: unknown"); + } + } let _ = writeln!(out, "Total events: {}", body.total_events); let _ = writeln!(out, "Share-eligible: {}", body.share_eligible); out.push('\n'); @@ -65,16 +113,13 @@ impl ScopeStatsOutput { out.push_str("Integrity:\n"); let _ = writeln!(out, " Inconsistencies: {}", body.inconsistency_count); // `complete` is the operator's go/no-go on trusting the tally: a torn or - // corrupt tail truncates the scan and the per-scope numbers undercount. + // corrupt tail truncates the scan and the per-scope numbers undercount, + // and an unverifiable store (failed diagnosis) is NOT claimed complete. let _ = writeln!( out, " Complete: {} ({})", body.complete, - if body.complete { - "tally covers every WAL batch" - } else { - "torn/corrupt tail — tally undercounts" - }, + body.integrity.describe(), ); out @@ -91,11 +136,18 @@ struct ScopeStatsBody { /// per-scope tally undercounts the true on-disk event set. `0` means a /// clean, complete scan. inconsistency_count: u64, - /// `true` when no corruption was hit and the tally covers every WAL batch; - /// `false` when a torn/corrupt tail truncated the scan (see - /// `inconsistency_count`). Lets a consumer branch on integrity without - /// re-deriving it from the count. + /// `true` ONLY when integrity was proven clean (diagnosis succeeded with + /// zero inconsistencies). `false` on a torn/corrupt tail (see + /// `inconsistency_count`) AND when integrity could not be verified at all + /// (failed diagnosis) — we never assert completeness we cannot prove. Lets a + /// consumer branch on integrity without re-deriving it from the count. complete: bool, + /// Tri-state integrity: `clean` / `torn` / `unverified`. Distinguishes a + /// torn tail (some data present, scan truncated) from an unverifiable store + /// (diagnosis itself failed), which `complete` alone collapses into one + /// `false`. A consumer that gates shipping on integrity should treat + /// anything other than `clean` as do-not-ship. + integrity: Integrity, } #[derive(Serialize)] @@ -117,9 +169,17 @@ struct ByScope { /// tally alone cannot tell a clean scan from one truncated by a torn tail. We /// therefore cross-check against [`diagnose_wal`], whose `inconsistency_count` /// counts every corrupt/truncated batch (both readers use the same two-phase -/// validation and stop at the same boundary). A non-zero count is surfaced as -/// `inconsistency_count` + `complete: false` so the operator knows the per-scope -/// numbers undercount the true on-disk set rather than silently trusting them. +/// validation and stop at the same boundary). The diagnosis is a tri-state: +/// +/// * `Ok(0 inconsistencies)` → [`Integrity::Clean`], `complete: true`, exit 0; +/// * `Ok(n > 0)` → [`Integrity::Torn`], `complete: false`, exit 0 (the tally +/// undercounts but the command still succeeds — the prefix is useful); +/// * `Err(_)` (e.g. a corrupt `checkpoint.meta` the segment scan never touched) +/// → [`Integrity::Unverified`], `complete: false`, and `EXIT_DEGRADED` (2) to +/// match `status`/`diagnostics` — integrity could NOT be proven, so a script's +/// `scope-stats && ship` must not treat the store as clean. Folding this case +/// into `complete: true` (the prior `map_or(0, …)`) was a silent false +/// positive precisely in the degraded case the indicator exists to flag. pub(crate) fn run(base: &std::path::Path, pretty: bool) -> Result<(String, i32), CliError> { use tidaldb::governance::SignalScope; @@ -128,7 +188,10 @@ pub(crate) fn run(base: &std::path::Path, pretty: bool) -> Result<(String, i32), let mut stats = ScopeStats::default(); let mut inconsistency_count = 0u64; - let segment_count = if wal_dir.exists() { + let mut integrity = Integrity::Clean; + let mut exit_code = 0; + + if wal_dir.exists() { let events = tidaldb::wal::reader::read_all_events(&wal_dir) .map_err(|e| CliError::new(format!("WAL read error: {e}")))?; for ev in events { @@ -144,18 +207,37 @@ pub(crate) fn run(base: &std::path::Path, pretty: bool) -> Result<(String, i32), // Cross-check tail integrity. diagnose_wal walks the same segments with // identical validation and reports corrupt/truncated batches; a torn or // corrupt tail that silently truncated `read_all_events` shows up here as - // a non-zero inconsistency count. If the diagnosis itself fails we treat - // it as an unknown-integrity scan (count stays 0 but we cannot prove - // completeness) rather than failing the command outright — the per-scope - // tally is still useful. - inconsistency_count = - tidaldb::wal::diagnostics::diagnose_wal(base).map_or(0, |r| r.inconsistency_count); + // a non-zero inconsistency count. Distinguish the THREE outcomes instead + // of folding a failed diagnosis into the clean path (W29). + let diag = tidaldb::wal::diagnostics::diagnose_wal(base); + inconsistency_count = diag.as_ref().map_or(0, |r| r.inconsistency_count); + integrity = match &diag { + Ok(r) if r.inconsistency_count == 0 => Integrity::Clean, + Ok(_) => Integrity::Torn, + // Integrity could not be verified (e.g. corrupt checkpoint.meta the + // segment scan never reads). Do NOT claim completeness; degrade the + // exit code so a gating script cannot mistake this for success. + Err(_) => { + exit_code = EXIT_DEGRADED; + Integrity::Unverified + } + }; + } + // Segment count. `None` = the listing could not be read (reported as + // "unknown", never a fabricated 0); a missing/empty WAL dir is genuinely 0 + // segments, a known count, so `Some(0)`. + // + // SUG#9: report an unreadable segment listing as `None`, not `0`. The listing + // can change between this scan and `read_all_events` on a live, lock-free + // inspector (a segment rolled/removed concurrently), and a `0` next to a + // non-zero `total_events` is internally inconsistent. + let segment_count: Option = if wal_dir.exists() { tidaldb::wal::segment::list_segments(&wal_dir) + .ok() .map(|s| s.len()) - .unwrap_or(0) } else { - 0 + Some(0) }; let output = ScopeStatsOutput { @@ -171,16 +253,19 @@ pub(crate) fn run(base: &std::path::Path, pretty: bool) -> Result<(String, i32), unknown: stats.unknown, }, inconsistency_count, - complete: inconsistency_count == 0, + complete: integrity.is_complete(), + integrity, }, }; // `--pretty` selects the human-readable section layout (consistent with - // `recover`/`diagnostics`); the default emits compact machine JSON. + // `recover`/`diagnostics`); the default emits compact machine JSON. The exit + // code carries the integrity verdict (0 clean/torn-but-readable, 2 when + // integrity could not be verified) — same contract as status/diagnostics. if pretty { - Ok((output.format_pretty(), 0)) + Ok((output.format_pretty(), exit_code)) } else { - Ok((render_json(&output, false)?, 0)) + Ok((render_json(&output, false)?, exit_code)) } } @@ -190,7 +275,7 @@ mod tests { fn sample() -> ScopeStatsOutput { ScopeStatsOutput { - wal_segments: 3, + wal_segments: Some(3), scope_stats: ScopeStatsBody { total_events: 10, share_eligible: 6, @@ -203,6 +288,7 @@ mod tests { }, inconsistency_count: 0, complete: true, + integrity: Integrity::Clean, }, } } @@ -224,9 +310,68 @@ mod tests { let mut output = sample(); output.scope_stats.inconsistency_count = 2; output.scope_stats.complete = false; + output.scope_stats.integrity = Integrity::Torn; let out = output.format_pretty(); assert!(out.contains("Inconsistencies: 2")); assert!(out.contains("Complete: false")); assert!(out.contains("undercounts")); } + + /// W29: an unverifiable store (failed diagnosis) must NOT be claimed + /// complete. `Integrity::Unverified` yields `is_complete() == false` and a + /// distinct, honest description — not the clean-path "covers every batch". + #[test] + fn unverified_integrity_is_never_complete() { + assert!( + !Integrity::Unverified.is_complete(), + "unverified integrity must never be reported as complete" + ); + assert!( + !Integrity::Torn.is_complete(), + "a torn tail must never be reported as complete" + ); + assert!( + Integrity::Clean.is_complete(), + "only a proven-clean scan is complete" + ); + let desc = Integrity::Unverified.describe(); + assert!( + desc.contains("could not be verified"), + "unverified must describe the integrity gap honestly, got {desc:?}" + ); + assert_ne!( + Integrity::Unverified.describe(), + Integrity::Clean.describe(), + "unverified must not borrow the clean-path message" + ); + } + + /// W29: `Integrity` serializes to the documented `snake_case` discriminants so + /// a consumer can branch on `integrity == "unverified"` to refuse to ship. + #[test] + fn integrity_serializes_snake_case() { + let json = |i: Integrity| { + serde_json::to_value(i) + .expect("Integrity serializes") + .as_str() + .expect("Integrity is a JSON string") + .to_string() + }; + assert_eq!(json(Integrity::Clean), "clean"); + assert_eq!(json(Integrity::Torn), "torn"); + assert_eq!(json(Integrity::Unverified), "unverified"); + } + + /// SUG#9: an unreadable segment listing renders as "unknown", never a + /// fabricated `0` that reads as "no segments". + #[test] + fn pretty_reports_unknown_segment_count() { + let mut output = sample(); + output.wal_segments = None; + let out = output.format_pretty(); + assert!( + out.contains("WAL Segments: unknown"), + "unreadable segment listing must render as unknown, got:\n{out}" + ); + } }