jordan 6f26d03c77 feat(m9): implement purge re-materialization engine

Adds the M9 purge re-materialization feature: a WAL-replay background
engine that rebuilds community cohort aggregates for a (user, community)
pair after retroactive signal purge, restoring ranking correctness without
modifying the immutable WAL.

Key additions:
- cohort::rematerialization module: PurgeJobQueue, RematerializationEngine,
  WAL replay, atomic CohortSignalLedger swap, BLAKE3 audit log, metrics counters
- TidalDb::{submit_purge_job, purge_job_status, rematerialization_metrics}
  public API (db/rematerialization.rs)
- Engine auto-starts in persistent mode; clean shutdown before WAL teardown
- 6 integration tests in tests/m9_purge_remat.rs covering ephemeral and
  persistent modes, job lifecycle, and multi-job independence
- Split oversized files to comply with 600-line limit: db/mod.rs →
  db/from_parts.rs, entities/revocation.rs → revocation/{mod,tests}.rs,
  schema/validation/builders.rs → builders/{mod,tests}.rs,
  signals/warm.rs → warm/{mod,tests,proptests}.rs
- Fix pre-existing bootstrap errors: export AuditKind from session module,
  add overrides_rejected to SessionSnapshot deserialization

2026-03-03 19:18:16 -07:00

3.8 KiB

Raw Blame History

Security Audit: Re-materialization after Purge

Scope

This audit covers:

tidal/src/cohort/rematerialization/ (all 6 files)
tidal/src/db/rematerialization.rs
Modifications to tidal/src/cohort/ledger.rs and tidal/src/db/mod.rs

Threat Model

The re-materialization engine runs as a background thread with access to:

The live CohortSignalLedger (read/write)
The CohortContributionLog (drain)
WAL files on disk (read-only replay)
An audit log file on disk (append-only write)

Findings

P1: No Authorization on `submit_purge_job`

submit_purge_job(user_id, community_id, filter) accepts any user_id from any caller. There is no check that the caller has permission to purge that user's data.

Risk: A caller could purge another user's cohort contributions without authorization.

Mitigation: TidalDB is an embedded database, not a multi-tenant network service. Authorization is the responsibility of the embedding application layer, which must validate that the requesting user matches the user_id before calling this API. This is consistent with the established pattern for all other user-scoped operations (request_community_purge, list_purge_manifests, etc.). No in-DB auth is warranted at this scope.

Status: Accepted — embedding layer responsibility.

P2: Audit Log Integrity

The BLAKE3-framed binary audit log uses only the first 4 bytes of the BLAKE3 hash as the checksum. This is a 32-bit checksum, not cryptographic integrity.

Risk: A determined adversary with filesystem write access could craft a forged audit entry that passes the 4-byte checksum check.

Mitigation: The full 32-byte BLAKE3 hash is not stored due to file size concerns in read_audit_log. The 4-byte checksum detects accidental corruption (the primary threat model). For regulatory environments requiring non-repudiation, the embedding layer should extend the audit log with external signing (e.g., append to a signed ledger). The audit log is append-only and the verification_checksum field in each entry uses the full BLAKE3 hash of the live ledger state.

Status: Accepted — noted for future hardening if compliance requires full tamper-evidence.

P3: WAL Read is Read-Only

filtered_replay opens WAL segments with read-only I/O. It does not mutate the WAL. Correct.

P4: No Unbounded Memory Growth in PurgeJobQueue

PurgeJobQueue stores all jobs (including Succeeded and PermanentlyFailed) indefinitely in memory. In a system with high purge volume this could grow without bound.

Risk: Memory exhaustion if thousands of purge jobs are submitted without restarting the process.

Mitigation: The job queue is in-memory and intentionally not persisted. In practice, purge operations are rare (user GDPR/CCPA requests). Embedders who anticipate high purge volumes should periodically restart the process or implement a job eviction policy. The current design is appropriate for the expected use case.

Status: Accepted for current scope — documented for future work.

P5: Mutex Poison Handling

All PurgeJobQueue mutex access in public API methods propagates poison as TidalError::Internal. Worker thread panics propagate through std::thread::JoinHandle::join() which returns Err. This is handled gracefully in shutdown_inner().

Status: No issue.

P6: Path Traversal in `audit_log_path`

audit_log_path is constructed as config.data_dir.join("purge_audit.log") — a fixed filename derived from a user-controlled data_dir. No user-supplied suffix is used. No path traversal risk.