jordan 6f26d03c77 feat(m9): implement purge re-materialization engine

Adds the M9 purge re-materialization feature: a WAL-replay background
engine that rebuilds community cohort aggregates for a (user, community)
pair after retroactive signal purge, restoring ranking correctness without
modifying the immutable WAL.

Key additions:
- cohort::rematerialization module: PurgeJobQueue, RematerializationEngine,
  WAL replay, atomic CohortSignalLedger swap, BLAKE3 audit log, metrics counters
- TidalDb::{submit_purge_job, purge_job_status, rematerialization_metrics}
  public API (db/rematerialization.rs)
- Engine auto-starts in persistent mode; clean shutdown before WAL teardown
- 6 integration tests in tests/m9_purge_remat.rs covering ephemeral and
  persistent modes, job lifecycle, and multi-job independence
- Split oversized files to comply with 600-line limit: db/mod.rs →
  db/from_parts.rs, entities/revocation.rs → revocation/{mod,tests}.rs,
  schema/validation/builders.rs → builders/{mod,tests}.rs,
  signals/warm.rs → warm/{mod,tests,proptests}.rs
- Fix pre-existing bootstrap errors: export AuditKind from session module,
  add overrides_rejected to SessionSnapshot deserialization

2026-03-03 19:18:16 -07:00

4.8 KiB

Raw Blame History

Review: Re-materialization after Purge

Summary

Implementation is complete. All 9 tasks are delivered. 1299 lib tests pass, 6 new integration tests pass, 8 existing retroactive purge tests pass. Clippy clean, fmt clean.

Files Changed

New Files

tidal/src/cohort/rematerialization/mod.rs — engine orchestrator, RematerializationConfig, RematerializationHandle, start(), worker_loop()
tidal/src/cohort/rematerialization/job.rs — PurgeJob, PurgeJobQueue, PurgeJobStatus, JobId
tidal/src/cohort/rematerialization/replay.rs — filtered_replay(), ExclusionKey, ReplayResult, ReplayStats
tidal/src/cohort/rematerialization/swap.rs — atomic_swap()
tidal/src/cohort/rematerialization/audit.rs — AuditLogWriter, PurgeAuditEntry, BLAKE3-framed binary format
tidal/src/cohort/rematerialization/metrics.rs — RematerializationMetrics, RematerializationMetricsSnapshot
tidal/src/db/rematerialization.rs — public API: submit_purge_job, purge_job_status, rematerialization_metrics
tidal/tests/m9_purge_remat.rs — 6 integration tests

Modified Files

tidal/src/cohort/ledger.rs — added remove_entry(), drain_community_into()
tidal/src/cohort/mod.rs — added pub mod contribution, pub mod purge, pub mod rematerialization
tidal/src/db/mod.rs — added 3 fields, mod rematerialization, engine start in from_parts, mod rematerialization
tidal/src/db/lifecycle.rs — engine shutdown before WAL in shutdown_inner()
tidal/src/cohort/purge.rs — fixed purge_retracts_from_ledger test (ancient timestamp)
tidal/src/entities/revocation.rs — replaced uuid::Uuid with [u8; 16] (not in Cargo.toml)
tidal/src/query/retrieve/types/tests.rs — added community: None to 3 direct Retrieve struct initializers
tidal/src/ranking/executor/mod.rs — fixed if let Some on Result (bootstrapped code bug)
tidal/src/entities/community.rs — fixed temporary-dropped-while-borrowed (DashMap entry)

Design Review

Correctness

ExclusionKey 4-tuple (entity_id, signal_type_id, weight_bits, timestamp_nanos) precisely identifies WAL events to skip. Exact match avoids false exclusions.
Atomic swap uses DashMap::remove + insert to transfer EntitySignalEntry ownership without cloning (HotSignalState contains AtomicU64, not Clone).
Score floor: HotSignalState::retract() uses saturating subtraction, so scores never go negative.
Idempotency: Second submit_purge_job + completed job correctly processes an empty contribution log, producing a no-op swap. Score unchanged.
Crash safety: requeue_interrupted() at startup re-queues any Running jobs found in the queue.

Concurrency

Background worker is a single thread — no per-job parallelism needed. Job queue mutex is only held briefly (claim + update).
DashMap shard locking ensures ranking queries on disjoint entries proceed during the atomic swap.
Worker polls every 500ms, responsive to shutdown within one sleep interval.

Observability

RematerializationMetrics AtomicU64 counters: jobs_pending, jobs_succeeded, jobs_failed, last_job_duration_ms, last_job_entries_updated.
Audit log: BLAKE3-framed binary entries with verification checksum over live ledger state.
Tracing spans: info! on success, warn! on failure, error! on audit log open failure.

Error Handling

WAL read errors propagate as TidalError::Internal and mark job Failed (retried up to 5 times).
Storage write failures in request_community_purge return TidalError::Internal; in-memory retraction already applied (eventual consistency).
Audit log write failure is non-fatal (warn log only) to prevent blocking job completion.
Mutex poison produces TidalError::Internal in public API methods.

Issues Found and Fixed

purge_retracts_from_ledger test: Pre-existing bootstrapped test used ts = 1_000_000_000u64 (nanoseconds = ~1970 epoch). Signal fully decays over 55 years. Fixed to Timestamp::now().
Retrieve struct direct initializers: Bootstrapped CommunityContext field added to Retrieve struct but 3 test sites constructed the struct directly without community: None. Fixed.
ranking/executor/mod.rs: Bootstrapped code used if let Some(type_id) = self.ledger.resolve_signal_type(...) but resolve_signal_type returns Result, not Option. Fixed to if let Ok.
entities/revocation.rs: Bootstrapped code imported uuid::Uuid which is not in Cargo.toml. Replaced with [u8; 16] backed by Timestamp::now().as_nanos() + AtomicU64 counter.
entities/community.rs: DashMap entry temporary dropped while borrowed. Fixed with explicit let mut entry binding (clippy significant_drop_tightening).

Verdict

APPROVED. Implementation is correct, complete, and consistent with the spec and design. All tests pass.

4.8 KiB Raw Blame History