jordan/tidaldb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
261d78d1f1
tidaldb/.sdlc/features/m9-purge-rematerialization/tasks.md
jordan 6f26d03c77 feat(m9): implement purge re-materialization engine 
Adds the M9 purge re-materialization feature: a WAL-replay background
engine that rebuilds community cohort aggregates for a (user, community)
pair after retroactive signal purge, restoring ranking correctness without
modifying the immutable WAL.

Key additions:
- cohort::rematerialization module: PurgeJobQueue, RematerializationEngine,
  WAL replay, atomic CohortSignalLedger swap, BLAKE3 audit log, metrics counters
- TidalDb::{submit_purge_job, purge_job_status, rematerialization_metrics}
  public API (db/rematerialization.rs)
- Engine auto-starts in persistent mode; clean shutdown before WAL teardown
- 6 integration tests in tests/m9_purge_remat.rs covering ephemeral and
  persistent modes, job lifecycle, and multi-job independence
- Split oversized files to comply with 600-line limit: db/mod.rs →
  db/from_parts.rs, entities/revocation.rs → revocation/{mod,tests}.rs,
  schema/validation/builders.rs → builders/{mod,tests}.rs,
  signals/warm.rs → warm/{mod,tests,proptests}.rs
- Fix pre-existing bootstrap errors: export AuditKind from session module,
  add overrides_rejected to SessionSnapshot deserialization
2026-03-03 19:18:16 -07:00

3.9 KiB
Raw Blame History

Tasks: Re-materialization after Purge

T1: PurgeJob and PurgeJobQueue

File: tidal/src/cohort/rematerialization/job.rs

  • JobId = [u8; 16], new_job_id() (timestamp+counter), job_id_to_string()
  • PurgeJobStatus enum: Pending, Running, Succeeded, Failed, PermanentlyFailed
  • PurgeJob struct with all lifecycle fields
  • PurgeJobQueue: DashMap + FIFO order Vec; submit, claim_pending, update_status, requeue_interrupted, requeue_failed, list_jobs, job_status, pending_count
  • Status: COMPLETE

T2: RematerializationMetrics

File: tidal/src/cohort/rematerialization/metrics.rs

  • RematerializationMetrics (AtomicU64 fields): jobs_pending, jobs_succeeded, jobs_failed, last_job_duration_ms, last_job_entries_updated
  • RematerializationMetricsSnapshot (plain u64/usize)
  • RematerializationMetrics::new() -> Arc<Self>, snapshot()
  • Status: COMPLETE

T3: Filtered WAL Replay

File: tidal/src/cohort/rematerialization/replay.rs

  • ExclusionKey { entity_id, signal_type_id, weight_bits, timestamp_nanos } 4-tuple
  • filtered_replay(wal_dir, schema, community_id, contributions, signal_type_filter) -> Result<ReplayResult>
  • Scans WAL segments; builds scratch CohortSignalLedger; excludes matched events
  • ReplayStats { events_replayed, events_excluded }
  • Status: COMPLETE

T4: Atomic Ledger Swap

File: tidal/src/cohort/rematerialization/swap.rs

  • atomic_swap(live, scratch, community_id) -> u64
  • Uses drain_community_into (no Clone needed); removes zero-event entries from live
  • Status: COMPLETE

T5: Audit Log

File: tidal/src/cohort/rematerialization/audit.rs

  • AuditLogWriter::open(path), append(&entry) -> Result<()>
  • BLAKE3-framed binary: [magic:4][body_len:4 LE][body:JSON][checksum:4]
  • PurgeAuditEntry, make_success_entry, make_failure_entry, read_audit_log
  • compute_verification_checksum_direct — BLAKE3 over sorted live ledger entries
  • Status: COMPLETE

T6: CohortSignalLedger Extensions

File: tidal/src/cohort/ledger.rs

  • remove_entry(cohort, entity_id, type_id) — removes one DashMap entry
  • drain_community_into(live, community_id) -> Vec<(EntityId, SignalTypeId)> — ownership transfer via DashMap::remove + insert
  • Status: COMPLETE

T7: RematerializationEngine Worker

File: tidal/src/cohort/rematerialization/mod.rs

  • RematerializationConfig struct
  • RematerializationHandle with shutdown() and Drop
  • start(config) -> RematerializationHandle — spawns tidal-remat thread
  • worker_loop: polls queue every 500ms, executes jobs, exponential backoff on failure
  • Status: COMPLETE

T8: TidalDb Integration

Files: tidal/src/db/mod.rs, tidal/src/db/rematerialization.rs, tidal/src/db/lifecycle.rs

  • Added purge_job_queue, rematerialization_metrics, rematerialization_handle fields
  • Engine started in from_parts() when config.data_dir is Some
  • Engine stopped in shutdown_inner() before WAL shutdown
  • Interrupted jobs requeued on startup
  • Public methods: submit_purge_job, purge_job_status, rematerialization_metrics
  • Status: COMPLETE

T9: Integration Tests

File: tidal/tests/m9_purge_remat.rs

  • 6 tests: submit_purge_job_returns_job_id, purge_job_status_pending_after_submit, unknown_job_id_returns_none, rematerialization_metrics_snapshot_is_valid, job_succeeds_in_persistent_mode, multiple_jobs_processed_independently
  • Status: COMPLETE

Bugfixes Applied

  • Fixed purge_retracts_from_ledger test in cohort/purge.rs (ancient timestamp caused full signal decay)
  • Fixed query/retrieve/types/tests.rs direct Retrieve struct initializers missing community field
  • Fixed entities/community.rs temporary drop while borrowed (DashMap entry)
  • Fixed ranking/executor/mod.rs if let Some on Result (bootstrapped code used wrong arm)
  • Replaced uuid::Uuid in entities/revocation.rs with [u8; 16] backed by timestamp+AtomicU64