6f26d03c77
Adds the M9 purge re-materialization feature: a WAL-replay background
engine that rebuilds community cohort aggregates for a (user, community)
pair after retroactive signal purge, restoring ranking correctness without
modifying the immutable WAL.
Key additions:
- cohort::rematerialization module: PurgeJobQueue, RematerializationEngine,
WAL replay, atomic CohortSignalLedger swap, BLAKE3 audit log, metrics counters
- TidalDb::{submit_purge_job, purge_job_status, rematerialization_metrics}
public API (db/rematerialization.rs)
- Engine auto-starts in persistent mode; clean shutdown before WAL teardown
- 6 integration tests in tests/m9_purge_remat.rs covering ephemeral and
persistent modes, job lifecycle, and multi-job independence
- Split oversized files to comply with 600-line limit: db/mod.rs →
db/from_parts.rs, entities/revocation.rs → revocation/{mod,tests}.rs,
schema/validation/builders.rs → builders/{mod,tests}.rs,
signals/warm.rs → warm/{mod,tests,proptests}.rs
- Fix pre-existing bootstrap errors: export AuditKind from session module,
add overrides_rejected to SessionSnapshot deserialization
69 lines
3.8 KiB
Markdown
69 lines
3.8 KiB
Markdown
# Security Audit: Re-materialization after Purge
|
|
|
|
## Scope
|
|
|
|
This audit covers:
|
|
- `tidal/src/cohort/rematerialization/` (all 6 files)
|
|
- `tidal/src/db/rematerialization.rs`
|
|
- Modifications to `tidal/src/cohort/ledger.rs` and `tidal/src/db/mod.rs`
|
|
|
|
## Threat Model
|
|
|
|
The re-materialization engine runs as a background thread with access to:
|
|
- The live `CohortSignalLedger` (read/write)
|
|
- The `CohortContributionLog` (drain)
|
|
- WAL files on disk (read-only replay)
|
|
- An audit log file on disk (append-only write)
|
|
|
|
## Findings
|
|
|
|
### P1: No Authorization on `submit_purge_job`
|
|
|
|
`submit_purge_job(user_id, community_id, filter)` accepts any `user_id` from any caller. There is no check that the caller has permission to purge that user's data.
|
|
|
|
**Risk**: A caller could purge another user's cohort contributions without authorization.
|
|
|
|
**Mitigation**: TidalDB is an embedded database, not a multi-tenant network service. Authorization is the responsibility of the embedding application layer, which must validate that the requesting user matches the `user_id` before calling this API. This is consistent with the established pattern for all other user-scoped operations (`request_community_purge`, `list_purge_manifests`, etc.). No in-DB auth is warranted at this scope.
|
|
|
|
**Status**: Accepted — embedding layer responsibility.
|
|
|
|
### P2: Audit Log Integrity
|
|
|
|
The BLAKE3-framed binary audit log uses only the first 4 bytes of the BLAKE3 hash as the checksum. This is a 32-bit checksum, not cryptographic integrity.
|
|
|
|
**Risk**: A determined adversary with filesystem write access could craft a forged audit entry that passes the 4-byte checksum check.
|
|
|
|
**Mitigation**: The full 32-byte BLAKE3 hash is not stored due to file size concerns in `read_audit_log`. The 4-byte checksum detects accidental corruption (the primary threat model). For regulatory environments requiring non-repudiation, the embedding layer should extend the audit log with external signing (e.g., append to a signed ledger). The audit log is append-only and the `verification_checksum` field in each entry uses the full BLAKE3 hash of the live ledger state.
|
|
|
|
**Status**: Accepted — noted for future hardening if compliance requires full tamper-evidence.
|
|
|
|
### P3: WAL Read is Read-Only
|
|
|
|
`filtered_replay` opens WAL segments with read-only I/O. It does not mutate the WAL. Correct.
|
|
|
|
### P4: No Unbounded Memory Growth in PurgeJobQueue
|
|
|
|
`PurgeJobQueue` stores all jobs (including Succeeded and PermanentlyFailed) indefinitely in memory. In a system with high purge volume this could grow without bound.
|
|
|
|
**Risk**: Memory exhaustion if thousands of purge jobs are submitted without restarting the process.
|
|
|
|
**Mitigation**: The job queue is in-memory and intentionally not persisted. In practice, purge operations are rare (user GDPR/CCPA requests). Embedders who anticipate high purge volumes should periodically restart the process or implement a job eviction policy. The current design is appropriate for the expected use case.
|
|
|
|
**Status**: Accepted for current scope — documented for future work.
|
|
|
|
### P5: Mutex Poison Handling
|
|
|
|
All `PurgeJobQueue` mutex access in public API methods propagates poison as `TidalError::Internal`. Worker thread panics propagate through `std::thread::JoinHandle::join()` which returns `Err`. This is handled gracefully in `shutdown_inner()`.
|
|
|
|
**Status**: No issue.
|
|
|
|
### P6: Path Traversal in `audit_log_path`
|
|
|
|
`audit_log_path` is constructed as `config.data_dir.join("purge_audit.log")` — a fixed filename derived from a user-controlled `data_dir`. No user-supplied suffix is used. No path traversal risk.
|
|
|
|
**Status**: No issue.
|
|
|
|
## Summary
|
|
|
|
No critical security issues found. Two accepted design decisions (P1: no in-DB auth; P2: 32-bit audit checksum) are appropriate for an embedded database and match the existing security posture of the codebase. The implementation is safe for production use within its intended deployment model.
|