# Review: Re-materialization after Purge ## Summary Implementation is complete. All 9 tasks are delivered. 1299 lib tests pass, 6 new integration tests pass, 8 existing retroactive purge tests pass. Clippy clean, fmt clean. ## Files Changed ### New Files - `tidal/src/cohort/rematerialization/mod.rs` — engine orchestrator, `RematerializationConfig`, `RematerializationHandle`, `start()`, `worker_loop()` - `tidal/src/cohort/rematerialization/job.rs` — `PurgeJob`, `PurgeJobQueue`, `PurgeJobStatus`, `JobId` - `tidal/src/cohort/rematerialization/replay.rs` — `filtered_replay()`, `ExclusionKey`, `ReplayResult`, `ReplayStats` - `tidal/src/cohort/rematerialization/swap.rs` — `atomic_swap()` - `tidal/src/cohort/rematerialization/audit.rs` — `AuditLogWriter`, `PurgeAuditEntry`, BLAKE3-framed binary format - `tidal/src/cohort/rematerialization/metrics.rs` — `RematerializationMetrics`, `RematerializationMetricsSnapshot` - `tidal/src/db/rematerialization.rs` — public API: `submit_purge_job`, `purge_job_status`, `rematerialization_metrics` - `tidal/tests/m9_purge_remat.rs` — 6 integration tests ### Modified Files - `tidal/src/cohort/ledger.rs` — added `remove_entry()`, `drain_community_into()` - `tidal/src/cohort/mod.rs` — added `pub mod contribution`, `pub mod purge`, `pub mod rematerialization` - `tidal/src/db/mod.rs` — added 3 fields, `mod rematerialization`, engine start in `from_parts`, `mod rematerialization` - `tidal/src/db/lifecycle.rs` — engine shutdown before WAL in `shutdown_inner()` - `tidal/src/cohort/purge.rs` — fixed `purge_retracts_from_ledger` test (ancient timestamp) - `tidal/src/entities/revocation.rs` — replaced `uuid::Uuid` with `[u8; 16]` (not in Cargo.toml) - `tidal/src/query/retrieve/types/tests.rs` — added `community: None` to 3 direct `Retrieve` struct initializers - `tidal/src/ranking/executor/mod.rs` — fixed `if let Some` on `Result` (bootstrapped code bug) - `tidal/src/entities/community.rs` — fixed temporary-dropped-while-borrowed (DashMap entry) ## Design Review ### Correctness - **ExclusionKey 4-tuple** (`entity_id`, `signal_type_id`, `weight_bits`, `timestamp_nanos`) precisely identifies WAL events to skip. Exact match avoids false exclusions. - **Atomic swap** uses `DashMap::remove + insert` to transfer `EntitySignalEntry` ownership without cloning (`HotSignalState` contains `AtomicU64`, not `Clone`). - **Score floor**: `HotSignalState::retract()` uses saturating subtraction, so scores never go negative. - **Idempotency**: Second `submit_purge_job` + completed job correctly processes an empty contribution log, producing a no-op swap. Score unchanged. - **Crash safety**: `requeue_interrupted()` at startup re-queues any `Running` jobs found in the queue. ### Concurrency - Background worker is a single thread — no per-job parallelism needed. Job queue mutex is only held briefly (claim + update). - `DashMap` shard locking ensures ranking queries on disjoint entries proceed during the atomic swap. - Worker polls every 500ms, responsive to shutdown within one sleep interval. ### Observability - `RematerializationMetrics` AtomicU64 counters: `jobs_pending`, `jobs_succeeded`, `jobs_failed`, `last_job_duration_ms`, `last_job_entries_updated`. - Audit log: BLAKE3-framed binary entries with verification checksum over live ledger state. - Tracing spans: `info!` on success, `warn!` on failure, `error!` on audit log open failure. ### Error Handling - WAL read errors propagate as `TidalError::Internal` and mark job `Failed` (retried up to 5 times). - Storage write failures in `request_community_purge` return `TidalError::Internal`; in-memory retraction already applied (eventual consistency). - Audit log write failure is non-fatal (warn log only) to prevent blocking job completion. - Mutex poison produces `TidalError::Internal` in public API methods. ## Issues Found and Fixed 1. **purge_retracts_from_ledger test**: Pre-existing bootstrapped test used `ts = 1_000_000_000u64` (nanoseconds = ~1970 epoch). Signal fully decays over 55 years. Fixed to `Timestamp::now()`. 2. **Retrieve struct direct initializers**: Bootstrapped `CommunityContext` field added to `Retrieve` struct but 3 test sites constructed the struct directly without `community: None`. Fixed. 3. **ranking/executor/mod.rs**: Bootstrapped code used `if let Some(type_id) = self.ledger.resolve_signal_type(...)` but `resolve_signal_type` returns `Result`, not `Option`. Fixed to `if let Ok`. 4. **entities/revocation.rs**: Bootstrapped code imported `uuid::Uuid` which is not in `Cargo.toml`. Replaced with `[u8; 16]` backed by `Timestamp::now().as_nanos()` + `AtomicU64` counter. 5. **entities/community.rs**: DashMap entry temporary dropped while borrowed. Fixed with explicit `let mut entry` binding (clippy `significant_drop_tightening`). ## Verdict APPROVED. Implementation is correct, complete, and consistent with the spec and design. All tests pass.