jordan 6f26d03c77 feat(m9): implement purge re-materialization engine

Adds the M9 purge re-materialization feature: a WAL-replay background
engine that rebuilds community cohort aggregates for a (user, community)
pair after retroactive signal purge, restoring ranking correctness without
modifying the immutable WAL.

Key additions:
- cohort::rematerialization module: PurgeJobQueue, RematerializationEngine,
  WAL replay, atomic CohortSignalLedger swap, BLAKE3 audit log, metrics counters
- TidalDb::{submit_purge_job, purge_job_status, rematerialization_metrics}
  public API (db/rematerialization.rs)
- Engine auto-starts in persistent mode; clean shutdown before WAL teardown
- 6 integration tests in tests/m9_purge_remat.rs covering ephemeral and
  persistent modes, job lifecycle, and multi-job independence
- Split oversized files to comply with 600-line limit: db/mod.rs →
  db/from_parts.rs, entities/revocation.rs → revocation/{mod,tests}.rs,
  schema/validation/builders.rs → builders/{mod,tests}.rs,
  signals/warm.rs → warm/{mod,tests,proptests}.rs
- Fix pre-existing bootstrap errors: export AuditKind from session module,
  add overrides_rejected to SessionSnapshot deserialization

2026-03-03 19:18:16 -07:00

2.4 KiB

Raw Blame History

Spec: Re-materialization after Purge

Problem

When a user requests retroactive removal of their signal contributions from a community cohort, the immediate in-memory retraction (PurgeCoordinator) only subtracts contributions currently tracked in the bounded CohortContributionLog ring buffer. If contributions were evicted from the ring buffer before the purge request, the live CohortSignalLedger still reflects those contributions. A full WAL replay is required to rebuild the correct aggregate state.

Goals

Correctness: After a purge job completes, the community's CohortSignalLedger must converge to the state it would have if the purged user had never contributed any signals.
Non-blocking: WAL replay runs in a background thread. Live signal writes and ranking queries are never blocked.
Auditability: Every purge job produces a tamper-evident audit log entry with a BLAKE3 verification checksum.
Observability: Job lifecycle (Pending → Running → Succeeded / Failed) and engine-level counters are queryable via public API.
Crash safety: Jobs interrupted by a crash are requeued on the next startup.

Non-Goals

Cross-shard WAL replay (single-node scope).
Persistent job queue (queue is in-memory; restarted jobs must be re-submitted if the process crashes before processing).
Real-time progress reporting (callers poll purge_job_status).

Key Invariants

Cohort ledger scores must never go below 0.0 after a purge.
Purging user A must not alter user B's contributions.
Purging the same user twice is safe and idempotent.

Public API Surface

// Submit a background re-materialization job.
db.submit_purge_job(user_id, community_id, signal_type_filter) -> Result<JobId>

// Poll job status.
db.purge_job_status(&job_id) -> Result<Option<PurgeJobStatus>>

// Snapshot engine counters.
db.rematerialization_metrics() -> Result<RematerializationMetricsSnapshot>

Acceptance Criteria

submit_purge_job returns a JobId without blocking.
purge_job_status returns Pending immediately after submission.
In persistent mode, jobs transition to Succeeded within 10 seconds for an empty WAL.
After a job succeeds, cohort ledger reflects the purged user's contributions removed.
rematerialization_metrics reflects jobs processed.
Interrupted (Running) jobs are requeued on startup.
Audit log entries are written after each job completion.

2.4 KiB Raw Blame History