6f26d03c77
Adds the M9 purge re-materialization feature: a WAL-replay background
engine that rebuilds community cohort aggregates for a (user, community)
pair after retroactive signal purge, restoring ranking correctness without
modifying the immutable WAL.
Key additions:
- cohort::rematerialization module: PurgeJobQueue, RematerializationEngine,
WAL replay, atomic CohortSignalLedger swap, BLAKE3 audit log, metrics counters
- TidalDb::{submit_purge_job, purge_job_status, rematerialization_metrics}
public API (db/rematerialization.rs)
- Engine auto-starts in persistent mode; clean shutdown before WAL teardown
- 6 integration tests in tests/m9_purge_remat.rs covering ephemeral and
persistent modes, job lifecycle, and multi-job independence
- Split oversized files to comply with 600-line limit: db/mod.rs →
db/from_parts.rs, entities/revocation.rs → revocation/{mod,tests}.rs,
schema/validation/builders.rs → builders/{mod,tests}.rs,
signals/warm.rs → warm/{mod,tests,proptests}.rs
- Fix pre-existing bootstrap errors: export AuditKind from session module,
add overrides_rejected to SessionSnapshot deserialization
49 lines
2.4 KiB
Markdown
49 lines
2.4 KiB
Markdown
# Spec: Re-materialization after Purge
|
|
|
|
## Problem
|
|
|
|
When a user requests retroactive removal of their signal contributions from a community cohort, the immediate in-memory retraction (`PurgeCoordinator`) only subtracts contributions currently tracked in the bounded `CohortContributionLog` ring buffer. If contributions were evicted from the ring buffer before the purge request, the live `CohortSignalLedger` still reflects those contributions. A full WAL replay is required to rebuild the correct aggregate state.
|
|
|
|
## Goals
|
|
|
|
1. **Correctness**: After a purge job completes, the community's `CohortSignalLedger` must converge to the state it would have if the purged user had never contributed any signals.
|
|
2. **Non-blocking**: WAL replay runs in a background thread. Live signal writes and ranking queries are never blocked.
|
|
3. **Auditability**: Every purge job produces a tamper-evident audit log entry with a BLAKE3 verification checksum.
|
|
4. **Observability**: Job lifecycle (Pending → Running → Succeeded / Failed) and engine-level counters are queryable via public API.
|
|
5. **Crash safety**: Jobs interrupted by a crash are requeued on the next startup.
|
|
|
|
## Non-Goals
|
|
|
|
- Cross-shard WAL replay (single-node scope).
|
|
- Persistent job queue (queue is in-memory; restarted jobs must be re-submitted if the process crashes before processing).
|
|
- Real-time progress reporting (callers poll `purge_job_status`).
|
|
|
|
## Key Invariants
|
|
|
|
- Cohort ledger scores must never go below 0.0 after a purge.
|
|
- Purging user A must not alter user B's contributions.
|
|
- Purging the same user twice is safe and idempotent.
|
|
|
|
## Public API Surface
|
|
|
|
```rust
|
|
// Submit a background re-materialization job.
|
|
db.submit_purge_job(user_id, community_id, signal_type_filter) -> Result<JobId>
|
|
|
|
// Poll job status.
|
|
db.purge_job_status(&job_id) -> Result<Option<PurgeJobStatus>>
|
|
|
|
// Snapshot engine counters.
|
|
db.rematerialization_metrics() -> Result<RematerializationMetricsSnapshot>
|
|
```
|
|
|
|
## Acceptance Criteria
|
|
|
|
- [ ] `submit_purge_job` returns a `JobId` without blocking.
|
|
- [ ] `purge_job_status` returns `Pending` immediately after submission.
|
|
- [ ] In persistent mode, jobs transition to `Succeeded` within 10 seconds for an empty WAL.
|
|
- [ ] After a job succeeds, cohort ledger reflects the purged user's contributions removed.
|
|
- [ ] `rematerialization_metrics` reflects jobs processed.
|
|
- [ ] Interrupted (Running) jobs are requeued on startup.
|
|
- [ ] Audit log entries are written after each job completion.
|