tidaldb/tidal/docs/planning/milestone-9/phase-3/OVERVIEW.md

m9p3: Retroactive Purge and Deterministic Rematerialization

Delivers

The purge_prior_contributions(user, community, epoch_range) API and the deterministic rematerialization engine behind it. After this phase a user who has left a community (m9p2 stop_forward) can retroactively remove every contribution they ever shared into that community's overlay -- and the community's hot+warm signal aggregates are rebuilt deterministically from the surviving WAL events, so two identical purges produce byte-identical checkpoints. Purge is durable (Tag::PurgeTombstone), CRDT-safe (scope_tombstones on CrdtSignalState preserve commutativity/associativity/idempotence across partition heals), and auditable (community queries surface a purge_watermark). The local-profile-intact guarantee from m9p1 is upheld throughout: Local-scope signals are never matched by a community purge.

This is the XL "remove and rebuild" phase. It introduces no new wire format (WAL V3 landed in m9p1; membership epochs landed in m9p2); it composes those primitives with a new replay/recompute path.

Deliverables:

• TidalDb::purge_prior_contributions(user_id, community_id, epoch_range) -> PurgeReceipt (new db/purge.rs)
• PurgeTombstone { user_id, community_id, epoch_range, hlc, watermark_seqno } persisted to Tag::PurgeTombstone
• RematerializeEngine (new governance/rematerialize.rs): replays community WAL sorted by (hlc, seqno), drops tombstoned events, recomputes hot+warm tiers from scratch
• CrdtSignalState gains scope_tombstones: BTreeMap<ScopeTombstoneKey, HlcTimestamp> so merge stays a true CRDT under purge
• purge_watermark: Option<u64> added to Results.policy_metadata (the PolicyMetadata struct introduced in m9p2)
• Hard-negative leak guard: apply_replication_unhide and reconcile honor purge tombstones so a late pre-purge unhide cannot revive a purged hard negative
• Determinism invariant: double-purge -> BLAKE3-equal checkpoint payloads (property test)

Dependencies

• Requires: m9p2 complete (MembershipState, Membership, MembershipEpoch, epoch ranges, PolicyMetadata on Results, db/communities.rs join/leave/rejoin, stop_forward_at_ns gate); m9p1 complete (SignalScope, CommunityId, SignalProvenance, WAL V3 per-event scope/writer_agent/share_policy_version/membership_epoch); M8 CRDT (CrdtSignalState, HlcTimestamp, ReconciliationEngine, StateSnapshot)
• Files modified:
  • tidal/src/replication/crdt/signal_state.rs -- add scope_tombstones; thread tombstone application through merge
  • tidal/src/replication/reconcile.rs -- stop-forward / purge ordering: tombstone HLC dominates a late unhide (hard-negative leak guard)
  • tidal/src/signals/ledger/core.rs -- apply_wal_event accepts a purge filter; restore/replay machinery reused by rematerialize
  • tidal/src/entities/hard_neg.rs -- apply_replication_unhide consults purge tombstone before reviving an item
  • tidal/src/query/retrieve/types.rs -- PolicyMetadata.purge_watermark
  • tidal/src/db/mod.rs -- purge_tombstones registry field; wire purge.rs and rematerialize.rs
  • tidal/src/storage/keys.rs -- add Tag::PurgeTombstone = 0x0E
  • tidal/src/lib.rs -- re-export purge / rematerialize types under governance
• Files created:
  • tidal/src/db/purge.rs -- purge_prior_contributions, PurgeReceipt, tombstone write path
  • tidal/src/governance/rematerialize.rs -- RematerializeEngine, PurgeTombstone, ScopeTombstoneKey, sorted replay + recompute

Research References

• docs/research/tidaldb_wal.md -- WAL replay determinism, segment ordering
• docs/research/tidaldb_signal_ledger.md -- running-decay recompute, hot/warm tier rebuild from raw events
• thoughts.md -- Part II.1 (WAL convergence), Part V.5 (quarantine-first removal), CRDT merge laws (Kulkarni et al. 2014, HLC)
• tidal/docs/planning/milestone-8/phase-3 -- CrdtSignalState merge laws (commutative/associative/idempotent) this phase must preserve
• tidal/docs/planning/milestone-9/phase-1 & phase-2 -- SignalScope, CommunityId, Membership, MembershipEpoch, PolicyMetadata (GIVEN; do not redefine)

Acceptance Criteria (Phase Level)

• TidalDb::purge_prior_contributions(user_id: u64, community_id: CommunityId, epoch_range: RangeInclusive<u32>) -> Result<PurgeReceipt> exists and is #[must_use] on its return
• A PurgeTombstone { user_id, community_id, epoch_range, hlc, watermark_seqno } is durably written to Tag::PurgeTombstone (0x0E) before the rematerialization job runs; survives close/reopen
• Purge triggers rematerialization synchronously and the returned PurgeReceipt reports events_dropped, entities_rebuilt, and watermark_seqno
• Rematerialization replays the community WAL in a total order of (hlc, seqno), skipping every event whose (user_id, community_id, membership_epoch) falls under any tombstone; surviving events recompute hot+warm tiers from zero
• Determinism: running the same purge twice over the same WAL yields BLAKE3-equal checkpoint payloads (hash_checkpoint_payload); property test asserts equality across >= 64 generated WAL/tombstone sequences
• CRDT merge with scope_tombstones is still commutative, associative, and idempotent (proptest, mirroring signal_state.rs law tests); a tombstone with a dominating HLC always wins regardless of merge order
• Local-scope events are never selected by a community purge (local-profile-intact guarantee); a purge of community C leaves the user's local decay scores and windowed counts bit-identical
• Hard-negative leak guard: after purge + simulated partition heal, a pre-purge unhide with an HLC older than the purge tombstone does NOT revive a purged hard negative (apply_replication_unhide returns false)
• Community queries expose purge_watermark: Some(seqno) in Results.policy_metadata after a purge; pre-purge queries report None
• Crash injection mid-tombstone-write and mid-rematerialize recovers to a consistent state on reopen (either purge fully applied or not at all)
• All m8/m9p1/m9p2 tests pass unchanged; cargo fmt clean, cargo clippy -p tidaldb -D warnings clean, cargo test -p tidaldb --lib green; tests are fast

Task Execution Order

Task 01: Tombstone Types ──────────┐
  (PurgeTombstone,                  │
   ScopeTombstoneKey, Tag 0x0E)     │
                                    ├──> Task 03: scope_tombstones on CrdtSignalState
Task 02: Rematerialize Engine ─────┤        (merge-law preserving)
  (sorted (hlc,seqno) replay,       │              │
   recompute hot+warm)              │              v
                                    ├──> Task 04: purge_prior_contributions API
                                    │        (db/purge.rs: write tombstone -> run engine -> receipt)
                                    │              │
                                    │              v
                                    └──> Task 05: Hard-negative leak guard
                                             (reconcile + apply_replication_unhide honor tombstone)
                                                   │
                                                   v
                                       Task 06: Purge watermark in query metadata
                                                   │
                                                   v
                                       Task 07: Determinism + crash property tests

Tasks 01 and 02 are parallelizable (disjoint files: governance/rematerialize.rs type vs replay logic). Task 03 depends on 01 (tombstone key). Task 04 depends on 01+02+03 (it is the integration seam). Tasks 05, 06, 07 depend on 04.

Module Location

File	Status	Contains
tidal/src/governance/rematerialize.rs	NEW	RematerializeEngine, PurgeTombstone, ScopeTombstoneKey, RematerializeReport, sorted replay
tidal/src/db/purge.rs	NEW	TidalDb::purge_prior_contributions, PurgeReceipt, tombstone persistence
tidal/src/replication/crdt/signal_state.rs	MODIFIED	scope_tombstones: BTreeMap<ScopeTombstoneKey, HlcTimestamp>; tombstone-aware merge
tidal/src/replication/reconcile.rs	MODIFIED	Purge-tombstone ordering in plan; tombstone HLC dominates late unhide
tidal/src/signals/ledger/core.rs	MODIFIED	apply_wal_event_filtered (purge predicate); replay reuse for rematerialize
tidal/src/entities/hard_neg.rs	MODIFIED	apply_replication_unhide consults purge tombstone before reviving
tidal/src/query/retrieve/types.rs	MODIFIED	PolicyMetadata.purge_watermark: Option<u64>
tidal/src/storage/keys.rs	MODIFIED	Tag::PurgeTombstone = 0x0E
tidal/src/db/mod.rs	MODIFIED	purge_tombstones registry field; module wiring
tidal/src/lib.rs	MODIFIED	governance re-exports for purge types

Technical Design

Tombstone types (governance/rematerialize.rs)

use std::collections::BTreeMap;
use std::ops::RangeInclusive;

use crate::governance::scope::CommunityId;
use crate::replication::crdt::HlcTimestamp;

/// A durable record that a user's prior contributions to a community, over a
/// closed range of membership epochs, must be removed from community aggregates.
///
/// Tombstones are write-once and monotone: rematerialization skips any event
/// whose `(user_id, community_id, membership_epoch)` is covered, and CRDT merge
/// keeps the tombstone with the dominating [`HlcTimestamp`]. Persisted under
/// [`Tag::PurgeTombstone`](crate::storage::Tag) keyed by `(community_id, user_id)`.
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
pub struct PurgeTombstone {
    /// The user whose contributions are being removed.
    pub user_id: u64,
    /// The community overlay the purge targets. Never `Local` scope.
    pub community_id: CommunityId,
    /// Inclusive membership-epoch range to purge (from m9p2 epochs).
    pub epoch_range: RangeInclusive<u32>,
    /// Causal stamp of the purge. Dominates any earlier unhide on merge.
    pub hlc: HlcTimestamp,
    /// WAL sequence number at which the purge took effect (the watermark
    /// surfaced in query metadata for auditability).
    pub watermark_seqno: u64,
}

impl PurgeTombstone {
    /// Whether this tombstone covers a given contribution.
    ///
    /// A community purge NEVER matches `Local`-scope contributions: the caller
    /// only ever feeds community-scoped events here, but this stays explicit so
    /// a routing bug cannot silently delete local history.
    #[must_use]
    pub fn covers(&self, user_id: u64, community_id: CommunityId, epoch: u32) -> bool {
        self.user_id == user_id
            && self.community_id == community_id
            && self.epoch_range.contains(&epoch)
    }
}

/// Key identifying a CRDT scope tombstone within a `(entity, signal_type)`
/// state. Ordered so merge is a deterministic fold over a `BTreeMap`.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash,
         serde::Serialize, serde::Deserialize)]
pub struct ScopeTombstoneKey {
    pub community_id: CommunityId,
    pub user_id: u64,
    /// Upper bound of the purged epoch range (covers `0..=epoch_hi`).
    pub epoch_hi: u32,
}

Rematerialization engine (governance/rematerialize.rs)

/// Outcome of a deterministic rematerialization pass.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct RematerializeReport {
    /// Events skipped because a tombstone covered them.
    pub events_dropped: u64,
    /// Events replayed into the rebuilt aggregates.
    pub events_replayed: u64,
    /// Distinct `(entity, signal_type)` ledger entries rebuilt.
    pub entities_rebuilt: u64,
    /// WAL sequence number the rebuild observed as the tail.
    pub watermark_seqno: u64,
}

/// Replays a community's WAL in deterministic `(hlc, seqno)` order, drops
/// tombstoned contributions, and recomputes hot+warm signal tiers from scratch.
///
/// Determinism is the load-bearing property: two passes over the same WAL with
/// the same tombstone set MUST produce byte-identical checkpoint payloads. This
/// requires (1) a total order on events -- `(hlc, seqno)`, never `DashMap`
/// iteration order -- and (2) forward-only decay application so out-of-order
/// events never advance `last_update_ns` (the `HotSignalState::on_signal`
/// invariant from m1p4 is reused verbatim).
pub struct RematerializeEngine<'a> {
    storage: &'a dyn crate::storage::StorageEngine,
    tombstones: &'a [PurgeTombstone],
}

impl<'a> RematerializeEngine<'a> {
    #[must_use]
    pub fn new(
        storage: &'a dyn crate::storage::StorageEngine,
        tombstones: &'a [PurgeTombstone],
    ) -> Self {
        Self { storage, tombstones }
    }

    /// Rebuild the community signal aggregates for `community_id` into a fresh
    /// [`SignalLedger`], returning the rebuilt ledger and a report.
    ///
    /// Steps:
    /// 1. Scan community WAL events (V3 carries `scope`, `membership_epoch`).
    /// 2. Sort by `(hlc, seqno)` -- a total order, stable across runs.
    /// 3. For each event, skip if any tombstone `covers(user, community, epoch)`.
    /// 4. Apply survivors via `SignalLedger::apply_wal_event` (forward decay).
    ///
    /// # Errors
    ///
    /// Returns `TidalError::Storage` on scan failure, `TidalError::Internal`
    /// on a corrupt WAL record.
    pub fn rematerialize(
        &self,
        community_id: CommunityId,
    ) -> crate::Result<(crate::signals::SignalLedger, RematerializeReport)> {
        // ... sorted replay; see Task 02.
        todo!()
    }
}

Purge API (db/purge.rs)

/// Receipt returned by [`TidalDb::purge_prior_contributions`].
#[derive(Debug, Clone, PartialEq, Eq)]
#[must_use = "a purge receipt records the watermark callers need for audit/query metadata"]
pub struct PurgeReceipt {
    /// The tombstone that was durably written.
    pub tombstone: PurgeTombstone,
    /// Events removed by rematerialization.
    pub events_dropped: u64,
    /// Ledger entries rebuilt.
    pub entities_rebuilt: u64,
    /// The purge watermark seqno (also surfaced in query `policy_metadata`).
    pub watermark_seqno: u64,
}

impl TidalDb {
    /// Retroactively remove a user's prior contributions to a community overlay
    /// and deterministically rebuild that community's affected aggregates.
    ///
    /// Precondition (enforced, not assumed): the user must be in
    /// `MembershipState::Left` or `LeavingStopForward` for `community_id`
    /// (m9p2). `community_id` is never `Local` scope -- local history is
    /// untouchable by community purge.
    ///
    /// Order of operations (crash-safe):
    /// 1. Allocate `watermark_seqno` = current WAL tail and an `HlcTimestamp`.
    /// 2. Durably write the [`PurgeTombstone`] to `Tag::PurgeTombstone`, fsync.
    /// 3. Run [`RematerializeEngine`] and atomically checkpoint the rebuilt
    ///    community ledger (replacing the diverged hot/warm state).
    /// 4. Record the watermark for query `policy_metadata`.
    ///
    /// A crash between (2) and (3) is recovered on reopen by re-running
    /// rematerialization from the durable tombstone -- the operation is
    /// idempotent because the tombstone is the source of truth.
    ///
    /// # Errors
    ///
    /// - `TidalError::NotFound` if the user has no membership for the community.
    /// - `TidalError::Query` if the user is still `Joined` (must stop-forward first).
    /// - `TidalError::Storage` / `TidalError::Internal` on persistence failure.
    pub fn purge_prior_contributions(
        &self,
        user_id: u64,
        community_id: CommunityId,
        epoch_range: std::ops::RangeInclusive<u32>,
    ) -> crate::Result<PurgeReceipt> {
        // ... see Task 04.
        todo!()
    }
}

CRDT scope tombstones (replication/crdt/signal_state.rs)

// Added field on CrdtSignalState (a true state-based CRDT):
//
// /// Scope tombstones keep this state a CRDT under retroactive purge.
// /// LWW per key by HlcTimestamp; merge takes the dominating stamp. A
// /// BTreeMap (not HashMap) so the merge fold is deterministic order.
// scope_tombstones: BTreeMap<ScopeTombstoneKey, HlcTimestamp>,

impl CrdtSignalState {
    /// Record a purge tombstone. Idempotent; LWW by `HlcTimestamp`.
    pub fn apply_scope_tombstone(&mut self, key: ScopeTombstoneKey, hlc: HlcTimestamp) {
        let slot = self.scope_tombstones.entry(key).or_insert(hlc);
        if hlc > *slot {
            *slot = hlc;
        }
    }

    // merge() gains a final fold that unions scope_tombstones, keeping the
    // dominating HlcTimestamp per key -- this preserves commutativity,
    // associativity, and idempotence exactly like the existing node_buckets
    // PNCounter merge. Tombstone application is order-independent (max wins).
}

Hard-negative leak guard (entities/hard_neg.rs)

impl HardNegIndex {
    /// As `apply_replication_unhide`, but a pending purge tombstone whose HLC
    /// dominates the unhide blocks the revive. This is the leak guard: a late
    /// pre-purge `unhide` arriving after a heal must NOT bring a purged hard
    /// negative back. Returns `true` only if the item was actually removed.
    #[must_use]
    pub fn apply_replication_unhide_guarded(
        &self,
        user_id: u64,
        item_id: u32,
        ts: HlcTimestamp,
        purge_hlc: Option<HlcTimestamp>,
    ) -> bool {
        if matches!(purge_hlc, Some(p) if ts <= p) {
            return false; // tombstone dominates; do not revive.
        }
        self.apply_replication_unhide(user_id, item_id, ts)
    }
}

Query metadata (query/retrieve/types.rs)

// PolicyMetadata is introduced in m9p2 (carries membership_epoch). m9p3 adds:
//
// /// WAL seqno at which the most recent purge for this community took effect.
// /// `None` if no purge has been applied. Auditability: a client can prove a
// /// query reflects state at or after a given purge.
// pub purge_watermark: Option<u64>,

Notes

Determinism is the single load-bearing invariant

Rematerialization MUST sort events by (hlc, seqno) and apply forward-decay identically each run. Never iterate DashMap for replay order -- it is hash-sharded and non-deterministic. The known hazard (m8p3 / m1p4): an out-of-order event must NOT advance last_update_ns in HotSignalState (signals/hot.rs); it pre-decays its own weight instead. Reuse that exact path. Verify by BLAKE3-hashing checkpoint payloads after two identical purges (hash_checkpoint_payload in signals/checkpoint/integrity.rs) and asserting equality.

CRDT merge laws survive scope tombstones

CrdtSignalState is the project's canonical state-based CRDT (commutative, associative, idempotent -- see its proptests). Adding scope_tombstones must not break these. Tombstone application is max-by-HLC per key, which is exactly the algebra of the existing node_buckets PNCounter merge: order-independent and idempotent. Extend the existing proptest strategy to generate tombstones and re-assert prop_merge_commutative / prop_merge_associative / prop_merge_idempotent_on_replay. Use a BTreeMap (not HashMap) for the tombstone field so the merge fold and any serialization are deterministic.

Local-profile-intact: the silent-data-loss trap

A community purge must filter strictly on SignalScope::Community(community_id). A routing bug that lets a community purge match Local-scope events is silent data loss of the user's own profile. PurgeTombstone::covers is intentionally explicit, and every UAT asserts the local feed is bit-identical across the purge. Local-scope events never shipped (m9p1) and are never present in the community WAL the engine scans, but the filter is belt-and-suspenders.

<1s gates are NOT the sweeper

m9p2's stop-forward gate and m10p2's revocation are O(1) atomic reads on the write path. Purge is a heavier (potentially multi-segment replay) operation and is explicitly NOT on that latency budget -- but it must never route through the 60s sweeper (db/sweeper.rs) for correctness. The stop-forward gate already guarantees no new contributions enter; purge then cleans history at its own pace.

Crash safety: tombstone is the source of truth

Write the tombstone durably BEFORE running rematerialization. If we crash mid- rebuild, reopen replays rematerialization from the durable tombstone -- idempotent because the tombstone, not the rebuilt aggregate, is authoritative. Use the existing crash-injection points (CheckpointPreFlush, WalPreAggregate) to prove recovery rebuilds identical state. WAL/checkpoint backward-compat is non-negotiable: this phase reads V1/V2/V3 events; it adds no new wire format.

Hard-negative leak guard ordering

apply_replication_unhide (hard_neg.rs:118) uses union/LWW: a late pre-purge unhide with a higher HLC than the original hide would normally revive the item. The purge tombstone's HLC must dominate, so reconcile.rs (plan, ~line 233) and the guarded unhide path compare against the purge HLC first. A purged item stays purged across partition heal and failover -- this is the explicit UAT requirement that hard negatives from U do not leak back after replay.

Done When

A user who has stop-forwarded out of community C can call purge_prior_contributions(user, C, epoch_range), see a durable tombstone in Tag::PurgeTombstone, get a PurgeReceipt with the watermark seqno, and observe that C's ranking outputs no longer reflect any of their purged contributions. Re-running the identical purge produces a byte-identical checkpoint (BLAKE3-equal). The user's local profile decay scores and windowed counts are unchanged. After a simulated partition heal, a pre-purge unhide does not revive a purged hard negative. Community queries report purge_watermark: Some(seqno). All prior M8/M9 tests pass; cargo fmt, cargo clippy -p tidaldb -D warnings, and cargo test -p tidaldb --lib are clean.