3bcfb3c576
- Add BUILD.bazel across tidal, tidal-net, tidal-server, tidalctl for bzlmod build - Add tidal/ crate docs (README, CHANGELOG, CONTRIBUTING, AGENTS, CLAUDE, API, ARCHITECTURE) and ai-lookup reference - Add docker standalone/cluster/deploy images, compose, and prometheus config - Harden WAL (batch format, writer, dedup, diagnostics), text syncer/collectors, and vector registry - Expand tidalctl CLI and tests; restructure WAL/visibility integration test suites - Refine tidal-net transport/client/server and tidal-server cluster/scatter-gather
418 lines
22 KiB
Markdown
418 lines
22 KiB
Markdown
# m9p3: Retroactive Purge and Deterministic Rematerialization
|
|
|
|
## Delivers
|
|
|
|
The `purge_prior_contributions(user, community, epoch_range)` API and the
|
|
deterministic rematerialization engine behind it. After this phase a user who
|
|
has left a community (m9p2 `stop_forward`) can retroactively remove every
|
|
contribution they ever shared into that community's overlay -- and the community's
|
|
hot+warm signal aggregates are rebuilt *deterministically* from the surviving
|
|
WAL events, so two identical purges produce byte-identical checkpoints. Purge is
|
|
durable (`Tag::PurgeTombstone`), CRDT-safe (`scope_tombstones` on
|
|
`CrdtSignalState` preserve commutativity/associativity/idempotence across
|
|
partition heals), and auditable (community queries surface a `purge_watermark`).
|
|
The local-profile-intact guarantee from m9p1 is upheld throughout: `Local`-scope
|
|
signals are never matched by a community purge.
|
|
|
|
This is the XL "remove and rebuild" phase. It introduces no new wire format
|
|
(WAL V3 landed in m9p1; membership epochs landed in m9p2); it composes those
|
|
primitives with a new replay/recompute path.
|
|
|
|
Deliverables:
|
|
- `TidalDb::purge_prior_contributions(user_id, community_id, epoch_range)` -> `PurgeReceipt` (new `db/purge.rs`)
|
|
- `PurgeTombstone { user_id, community_id, epoch_range, hlc, watermark_seqno }` persisted to `Tag::PurgeTombstone`
|
|
- `RematerializeEngine` (new `governance/rematerialize.rs`): replays community WAL sorted by `(hlc, seqno)`, drops tombstoned events, recomputes hot+warm tiers from scratch
|
|
- `CrdtSignalState` gains `scope_tombstones: BTreeMap<ScopeTombstoneKey, HlcTimestamp>` so merge stays a true CRDT under purge
|
|
- `purge_watermark: Option<u64>` added to `Results.policy_metadata` (the `PolicyMetadata` struct introduced in m9p2)
|
|
- Hard-negative leak guard: `apply_replication_unhide` and `reconcile` honor purge tombstones so a late pre-purge unhide cannot revive a purged hard negative
|
|
- Determinism invariant: double-purge -> BLAKE3-equal checkpoint payloads (property test)
|
|
|
|
## Dependencies
|
|
|
|
- **Requires:** m9p2 complete (`MembershipState`, `Membership`, `MembershipEpoch`, `epoch` ranges, `PolicyMetadata` on `Results`, `db/communities.rs` join/leave/rejoin, `stop_forward_at_ns` gate); m9p1 complete (`SignalScope`, `CommunityId`, `SignalProvenance`, WAL V3 per-event `scope`/`writer_agent`/`share_policy_version`/`membership_epoch`); M8 CRDT (`CrdtSignalState`, `HlcTimestamp`, `ReconciliationEngine`, `StateSnapshot`)
|
|
- **Files modified:**
|
|
- `tidal/src/replication/crdt/signal_state.rs` -- add `scope_tombstones`; thread tombstone application through `merge`
|
|
- `tidal/src/replication/reconcile.rs` -- stop-forward / purge ordering: tombstone HLC dominates a late unhide (hard-negative leak guard)
|
|
- `tidal/src/signals/ledger/core.rs` -- `apply_wal_event` accepts a purge filter; `restore`/replay machinery reused by rematerialize
|
|
- `tidal/src/entities/hard_neg.rs` -- `apply_replication_unhide` consults purge tombstone before reviving an item
|
|
- `tidal/src/query/retrieve/types.rs` -- `PolicyMetadata.purge_watermark`
|
|
- `tidal/src/db/mod.rs` -- `purge_tombstones` registry field; wire `purge.rs` and `rematerialize.rs`
|
|
- `tidal/src/storage/keys.rs` -- add `Tag::PurgeTombstone = 0x0E`
|
|
- `tidal/src/lib.rs` -- re-export `purge` / `rematerialize` types under `governance`
|
|
- **Files created:**
|
|
- `tidal/src/db/purge.rs` -- `purge_prior_contributions`, `PurgeReceipt`, tombstone write path
|
|
- `tidal/src/governance/rematerialize.rs` -- `RematerializeEngine`, `PurgeTombstone`, `ScopeTombstoneKey`, sorted replay + recompute
|
|
|
|
## Research References
|
|
|
|
- `docs/research/tidaldb_wal.md` -- WAL replay determinism, segment ordering
|
|
- `docs/research/tidaldb_signal_ledger.md` -- running-decay recompute, hot/warm tier rebuild from raw events
|
|
- `thoughts.md` -- Part II.1 (WAL convergence), Part V.5 (quarantine-first removal), CRDT merge laws (Kulkarni et al. 2014, HLC)
|
|
- `tidal/docs/planning/milestone-8/phase-3` -- `CrdtSignalState` merge laws (commutative/associative/idempotent) this phase must preserve
|
|
- `tidal/docs/planning/milestone-9/phase-1` & `phase-2` -- `SignalScope`, `CommunityId`, `Membership`, `MembershipEpoch`, `PolicyMetadata` (GIVEN; do not redefine)
|
|
|
|
## Acceptance Criteria (Phase Level)
|
|
|
|
- [ ] `TidalDb::purge_prior_contributions(user_id: u64, community_id: CommunityId, epoch_range: RangeInclusive<u32>) -> Result<PurgeReceipt>` exists and is `#[must_use]` on its return
|
|
- [ ] A `PurgeTombstone { user_id, community_id, epoch_range, hlc, watermark_seqno }` is durably written to `Tag::PurgeTombstone` (0x0E) before the rematerialization job runs; survives close/reopen
|
|
- [ ] Purge triggers rematerialization synchronously and the returned `PurgeReceipt` reports `events_dropped`, `entities_rebuilt`, and `watermark_seqno`
|
|
- [ ] Rematerialization replays the community WAL in a total order of `(hlc, seqno)`, skipping every event whose `(user_id, community_id, membership_epoch)` falls under any tombstone; surviving events recompute hot+warm tiers from zero
|
|
- [ ] **Determinism:** running the same purge twice over the same WAL yields BLAKE3-equal checkpoint payloads (`hash_checkpoint_payload`); property test asserts equality across >= 64 generated WAL/tombstone sequences
|
|
- [ ] CRDT merge with `scope_tombstones` is still commutative, associative, and idempotent (proptest, mirroring `signal_state.rs` law tests); a tombstone with a dominating HLC always wins regardless of merge order
|
|
- [ ] `Local`-scope events are never selected by a community purge (local-profile-intact guarantee); a purge of community C leaves the user's local decay scores and windowed counts bit-identical
|
|
- [ ] Hard-negative leak guard: after purge + simulated partition heal, a pre-purge `unhide` with an HLC older than the purge tombstone does NOT revive a purged hard negative (`apply_replication_unhide` returns `false`)
|
|
- [ ] Community queries expose `purge_watermark: Some(seqno)` in `Results.policy_metadata` after a purge; pre-purge queries report `None`
|
|
- [ ] Crash injection mid-tombstone-write and mid-rematerialize recovers to a consistent state on reopen (either purge fully applied or not at all)
|
|
- [ ] All m8/m9p1/m9p2 tests pass unchanged; `cargo fmt` clean, `cargo clippy -p tidaldb -D warnings` clean, `cargo test -p tidaldb --lib` green; tests are fast
|
|
|
|
## Task Execution Order
|
|
|
|
```
|
|
Task 01: Tombstone Types ──────────┐
|
|
(PurgeTombstone, │
|
|
ScopeTombstoneKey, Tag 0x0E) │
|
|
├──> Task 03: scope_tombstones on CrdtSignalState
|
|
Task 02: Rematerialize Engine ─────┤ (merge-law preserving)
|
|
(sorted (hlc,seqno) replay, │ │
|
|
recompute hot+warm) │ v
|
|
├──> Task 04: purge_prior_contributions API
|
|
│ (db/purge.rs: write tombstone -> run engine -> receipt)
|
|
│ │
|
|
│ v
|
|
└──> Task 05: Hard-negative leak guard
|
|
(reconcile + apply_replication_unhide honor tombstone)
|
|
│
|
|
v
|
|
Task 06: Purge watermark in query metadata
|
|
│
|
|
v
|
|
Task 07: Determinism + crash property tests
|
|
```
|
|
|
|
Tasks 01 and 02 are parallelizable (disjoint files: `governance/rematerialize.rs`
|
|
type vs replay logic). Task 03 depends on 01 (tombstone key). Task 04 depends on
|
|
01+02+03 (it is the integration seam). Tasks 05, 06, 07 depend on 04.
|
|
|
|
## Module Location
|
|
|
|
| File | Status | Contains |
|
|
|------|--------|----------|
|
|
| `tidal/src/governance/rematerialize.rs` | NEW | `RematerializeEngine`, `PurgeTombstone`, `ScopeTombstoneKey`, `RematerializeReport`, sorted replay |
|
|
| `tidal/src/db/purge.rs` | NEW | `TidalDb::purge_prior_contributions`, `PurgeReceipt`, tombstone persistence |
|
|
| `tidal/src/replication/crdt/signal_state.rs` | MODIFIED | `scope_tombstones: BTreeMap<ScopeTombstoneKey, HlcTimestamp>`; tombstone-aware `merge` |
|
|
| `tidal/src/replication/reconcile.rs` | MODIFIED | Purge-tombstone ordering in `plan`; tombstone HLC dominates late unhide |
|
|
| `tidal/src/signals/ledger/core.rs` | MODIFIED | `apply_wal_event_filtered` (purge predicate); replay reuse for rematerialize |
|
|
| `tidal/src/entities/hard_neg.rs` | MODIFIED | `apply_replication_unhide` consults purge tombstone before reviving |
|
|
| `tidal/src/query/retrieve/types.rs` | MODIFIED | `PolicyMetadata.purge_watermark: Option<u64>` |
|
|
| `tidal/src/storage/keys.rs` | MODIFIED | `Tag::PurgeTombstone = 0x0E` |
|
|
| `tidal/src/db/mod.rs` | MODIFIED | `purge_tombstones` registry field; module wiring |
|
|
| `tidal/src/lib.rs` | MODIFIED | `governance` re-exports for purge types |
|
|
|
|
## Technical Design
|
|
|
|
### Tombstone types (`governance/rematerialize.rs`)
|
|
|
|
```rust
|
|
use std::collections::BTreeMap;
|
|
use std::ops::RangeInclusive;
|
|
|
|
use crate::governance::scope::CommunityId;
|
|
use crate::replication::crdt::HlcTimestamp;
|
|
|
|
/// A durable record that a user's prior contributions to a community, over a
|
|
/// closed range of membership epochs, must be removed from community aggregates.
|
|
///
|
|
/// Tombstones are write-once and monotone: rematerialization skips any event
|
|
/// whose `(user_id, community_id, membership_epoch)` is covered, and CRDT merge
|
|
/// keeps the tombstone with the dominating [`HlcTimestamp`]. Persisted under
|
|
/// [`Tag::PurgeTombstone`](crate::storage::Tag) keyed by `(community_id, user_id)`.
|
|
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
|
pub struct PurgeTombstone {
|
|
/// The user whose contributions are being removed.
|
|
pub user_id: u64,
|
|
/// The community overlay the purge targets. Never `Local` scope.
|
|
pub community_id: CommunityId,
|
|
/// Inclusive membership-epoch range to purge (from m9p2 epochs).
|
|
pub epoch_range: RangeInclusive<u32>,
|
|
/// Causal stamp of the purge. Dominates any earlier unhide on merge.
|
|
pub hlc: HlcTimestamp,
|
|
/// WAL sequence number at which the purge took effect (the watermark
|
|
/// surfaced in query metadata for auditability).
|
|
pub watermark_seqno: u64,
|
|
}
|
|
|
|
impl PurgeTombstone {
|
|
/// Whether this tombstone covers a given contribution.
|
|
///
|
|
/// A community purge NEVER matches `Local`-scope contributions: the caller
|
|
/// only ever feeds community-scoped events here, but this stays explicit so
|
|
/// a routing bug cannot silently delete local history.
|
|
#[must_use]
|
|
pub fn covers(&self, user_id: u64, community_id: CommunityId, epoch: u32) -> bool {
|
|
self.user_id == user_id
|
|
&& self.community_id == community_id
|
|
&& self.epoch_range.contains(&epoch)
|
|
}
|
|
}
|
|
|
|
/// Key identifying a CRDT scope tombstone within a `(entity, signal_type)`
|
|
/// state. Ordered so merge is a deterministic fold over a `BTreeMap`.
|
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash,
|
|
serde::Serialize, serde::Deserialize)]
|
|
pub struct ScopeTombstoneKey {
|
|
pub community_id: CommunityId,
|
|
pub user_id: u64,
|
|
/// Upper bound of the purged epoch range (covers `0..=epoch_hi`).
|
|
pub epoch_hi: u32,
|
|
}
|
|
```
|
|
|
|
### Rematerialization engine (`governance/rematerialize.rs`)
|
|
|
|
```rust
|
|
/// Outcome of a deterministic rematerialization pass.
|
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
|
pub struct RematerializeReport {
|
|
/// Events skipped because a tombstone covered them.
|
|
pub events_dropped: u64,
|
|
/// Events replayed into the rebuilt aggregates.
|
|
pub events_replayed: u64,
|
|
/// Distinct `(entity, signal_type)` ledger entries rebuilt.
|
|
pub entities_rebuilt: u64,
|
|
/// WAL sequence number the rebuild observed as the tail.
|
|
pub watermark_seqno: u64,
|
|
}
|
|
|
|
/// Replays a community's WAL in deterministic `(hlc, seqno)` order, drops
|
|
/// tombstoned contributions, and recomputes hot+warm signal tiers from scratch.
|
|
///
|
|
/// Determinism is the load-bearing property: two passes over the same WAL with
|
|
/// the same tombstone set MUST produce byte-identical checkpoint payloads. This
|
|
/// requires (1) a total order on events -- `(hlc, seqno)`, never `DashMap`
|
|
/// iteration order -- and (2) forward-only decay application so out-of-order
|
|
/// events never advance `last_update_ns` (the `HotSignalState::on_signal`
|
|
/// invariant from m1p4 is reused verbatim).
|
|
pub struct RematerializeEngine<'a> {
|
|
storage: &'a dyn crate::storage::StorageEngine,
|
|
tombstones: &'a [PurgeTombstone],
|
|
}
|
|
|
|
impl<'a> RematerializeEngine<'a> {
|
|
#[must_use]
|
|
pub fn new(
|
|
storage: &'a dyn crate::storage::StorageEngine,
|
|
tombstones: &'a [PurgeTombstone],
|
|
) -> Self {
|
|
Self { storage, tombstones }
|
|
}
|
|
|
|
/// Rebuild the community signal aggregates for `community_id` into a fresh
|
|
/// [`SignalLedger`], returning the rebuilt ledger and a report.
|
|
///
|
|
/// Steps:
|
|
/// 1. Scan community WAL events (V3 carries `scope`, `membership_epoch`).
|
|
/// 2. Sort by `(hlc, seqno)` -- a total order, stable across runs.
|
|
/// 3. For each event, skip if any tombstone `covers(user, community, epoch)`.
|
|
/// 4. Apply survivors via `SignalLedger::apply_wal_event` (forward decay).
|
|
///
|
|
/// # Errors
|
|
///
|
|
/// Returns `TidalError::Storage` on scan failure, `TidalError::Internal`
|
|
/// on a corrupt WAL record.
|
|
pub fn rematerialize(
|
|
&self,
|
|
community_id: CommunityId,
|
|
) -> crate::Result<(crate::signals::SignalLedger, RematerializeReport)> {
|
|
// ... sorted replay; see Task 02.
|
|
todo!()
|
|
}
|
|
}
|
|
```
|
|
|
|
### Purge API (`db/purge.rs`)
|
|
|
|
```rust
|
|
/// Receipt returned by [`TidalDb::purge_prior_contributions`].
|
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
|
#[must_use = "a purge receipt records the watermark callers need for audit/query metadata"]
|
|
pub struct PurgeReceipt {
|
|
/// The tombstone that was durably written.
|
|
pub tombstone: PurgeTombstone,
|
|
/// Events removed by rematerialization.
|
|
pub events_dropped: u64,
|
|
/// Ledger entries rebuilt.
|
|
pub entities_rebuilt: u64,
|
|
/// The purge watermark seqno (also surfaced in query `policy_metadata`).
|
|
pub watermark_seqno: u64,
|
|
}
|
|
|
|
impl TidalDb {
|
|
/// Retroactively remove a user's prior contributions to a community overlay
|
|
/// and deterministically rebuild that community's affected aggregates.
|
|
///
|
|
/// Precondition (enforced, not assumed): the user must be in
|
|
/// `MembershipState::Left` or `LeavingStopForward` for `community_id`
|
|
/// (m9p2). `community_id` is never `Local` scope -- local history is
|
|
/// untouchable by community purge.
|
|
///
|
|
/// Order of operations (crash-safe):
|
|
/// 1. Allocate `watermark_seqno` = current WAL tail and an `HlcTimestamp`.
|
|
/// 2. Durably write the [`PurgeTombstone`] to `Tag::PurgeTombstone`, fsync.
|
|
/// 3. Run [`RematerializeEngine`] and atomically checkpoint the rebuilt
|
|
/// community ledger (replacing the diverged hot/warm state).
|
|
/// 4. Record the watermark for query `policy_metadata`.
|
|
///
|
|
/// A crash between (2) and (3) is recovered on reopen by re-running
|
|
/// rematerialization from the durable tombstone -- the operation is
|
|
/// idempotent because the tombstone is the source of truth.
|
|
///
|
|
/// # Errors
|
|
///
|
|
/// - `TidalError::NotFound` if the user has no membership for the community.
|
|
/// - `TidalError::Query` if the user is still `Joined` (must stop-forward first).
|
|
/// - `TidalError::Storage` / `TidalError::Internal` on persistence failure.
|
|
pub fn purge_prior_contributions(
|
|
&self,
|
|
user_id: u64,
|
|
community_id: CommunityId,
|
|
epoch_range: std::ops::RangeInclusive<u32>,
|
|
) -> crate::Result<PurgeReceipt> {
|
|
// ... see Task 04.
|
|
todo!()
|
|
}
|
|
}
|
|
```
|
|
|
|
### CRDT scope tombstones (`replication/crdt/signal_state.rs`)
|
|
|
|
```rust
|
|
// Added field on CrdtSignalState (a true state-based CRDT):
|
|
//
|
|
// /// Scope tombstones keep this state a CRDT under retroactive purge.
|
|
// /// LWW per key by HlcTimestamp; merge takes the dominating stamp. A
|
|
// /// BTreeMap (not HashMap) so the merge fold is deterministic order.
|
|
// scope_tombstones: BTreeMap<ScopeTombstoneKey, HlcTimestamp>,
|
|
|
|
impl CrdtSignalState {
|
|
/// Record a purge tombstone. Idempotent; LWW by `HlcTimestamp`.
|
|
pub fn apply_scope_tombstone(&mut self, key: ScopeTombstoneKey, hlc: HlcTimestamp) {
|
|
let slot = self.scope_tombstones.entry(key).or_insert(hlc);
|
|
if hlc > *slot {
|
|
*slot = hlc;
|
|
}
|
|
}
|
|
|
|
// merge() gains a final fold that unions scope_tombstones, keeping the
|
|
// dominating HlcTimestamp per key -- this preserves commutativity,
|
|
// associativity, and idempotence exactly like the existing node_buckets
|
|
// PNCounter merge. Tombstone application is order-independent (max wins).
|
|
}
|
|
```
|
|
|
|
### Hard-negative leak guard (`entities/hard_neg.rs`)
|
|
|
|
```rust
|
|
impl HardNegIndex {
|
|
/// As `apply_replication_unhide`, but a pending purge tombstone whose HLC
|
|
/// dominates the unhide blocks the revive. This is the leak guard: a late
|
|
/// pre-purge `unhide` arriving after a heal must NOT bring a purged hard
|
|
/// negative back. Returns `true` only if the item was actually removed.
|
|
#[must_use]
|
|
pub fn apply_replication_unhide_guarded(
|
|
&self,
|
|
user_id: u64,
|
|
item_id: u32,
|
|
ts: HlcTimestamp,
|
|
purge_hlc: Option<HlcTimestamp>,
|
|
) -> bool {
|
|
if matches!(purge_hlc, Some(p) if ts <= p) {
|
|
return false; // tombstone dominates; do not revive.
|
|
}
|
|
self.apply_replication_unhide(user_id, item_id, ts)
|
|
}
|
|
}
|
|
```
|
|
|
|
### Query metadata (`query/retrieve/types.rs`)
|
|
|
|
```rust
|
|
// PolicyMetadata is introduced in m9p2 (carries membership_epoch). m9p3 adds:
|
|
//
|
|
// /// WAL seqno at which the most recent purge for this community took effect.
|
|
// /// `None` if no purge has been applied. Auditability: a client can prove a
|
|
// /// query reflects state at or after a given purge.
|
|
// pub purge_watermark: Option<u64>,
|
|
```
|
|
|
|
## Notes
|
|
|
|
### Determinism is the single load-bearing invariant
|
|
|
|
Rematerialization MUST sort events by `(hlc, seqno)` and apply forward-decay
|
|
identically each run. Never iterate `DashMap` for replay order -- it is
|
|
hash-sharded and non-deterministic. The known hazard (m8p3 / m1p4): an
|
|
out-of-order event must NOT advance `last_update_ns` in `HotSignalState`
|
|
(`signals/hot.rs`); it pre-decays its own weight instead. Reuse that exact path.
|
|
Verify by BLAKE3-hashing checkpoint payloads after two identical purges
|
|
(`hash_checkpoint_payload` in `signals/checkpoint/integrity.rs`) and asserting
|
|
equality.
|
|
|
|
### CRDT merge laws survive scope tombstones
|
|
|
|
`CrdtSignalState` is the project's canonical state-based CRDT (commutative,
|
|
associative, idempotent -- see its proptests). Adding `scope_tombstones` must
|
|
not break these. Tombstone application is max-by-HLC per key, which is exactly
|
|
the algebra of the existing `node_buckets` `PNCounter` merge: order-independent
|
|
and idempotent. Extend the existing proptest strategy to generate tombstones and
|
|
re-assert `prop_merge_commutative` / `prop_merge_associative` /
|
|
`prop_merge_idempotent_on_replay`. Use a `BTreeMap` (not `HashMap`) for the
|
|
tombstone field so the merge fold and any serialization are deterministic.
|
|
|
|
### Local-profile-intact: the silent-data-loss trap
|
|
|
|
A community purge must filter strictly on `SignalScope::Community(community_id)`.
|
|
A routing bug that lets a community purge match `Local`-scope events is silent
|
|
data loss of the user's own profile. `PurgeTombstone::covers` is intentionally
|
|
explicit, and every UAT asserts the local feed is bit-identical across the purge.
|
|
`Local`-scope events never shipped (m9p1) and are never present in the community
|
|
WAL the engine scans, but the filter is belt-and-suspenders.
|
|
|
|
### <1s gates are NOT the sweeper
|
|
|
|
m9p2's stop-forward gate and m10p2's revocation are O(1) atomic reads on the
|
|
write path. Purge is a heavier (potentially multi-segment replay) operation and
|
|
is explicitly NOT on that latency budget -- but it must never route through the
|
|
60s sweeper (`db/sweeper.rs`) for correctness. The stop-forward gate already
|
|
guarantees no new contributions enter; purge then cleans history at its own pace.
|
|
|
|
### Crash safety: tombstone is the source of truth
|
|
|
|
Write the tombstone durably BEFORE running rematerialization. If we crash mid-
|
|
rebuild, reopen replays rematerialization from the durable tombstone -- idempotent
|
|
because the tombstone, not the rebuilt aggregate, is authoritative. Use the
|
|
existing crash-injection points (`CheckpointPreFlush`, `WalPreAggregate`) to
|
|
prove recovery rebuilds identical state. WAL/checkpoint backward-compat is
|
|
non-negotiable: this phase reads V1/V2/V3 events; it adds no new wire format.
|
|
|
|
### Hard-negative leak guard ordering
|
|
|
|
`apply_replication_unhide` (`hard_neg.rs:118`) uses union/LWW: a late pre-purge
|
|
unhide with a higher HLC than the original hide would normally revive the item.
|
|
The purge tombstone's HLC must dominate, so `reconcile.rs` (`plan`, ~line 233)
|
|
and the guarded unhide path compare against the purge HLC first. A purged item
|
|
stays purged across partition heal and failover -- this is the explicit UAT
|
|
requirement that hard negatives from U do not leak back after replay.
|
|
|
|
## Done When
|
|
|
|
A user who has stop-forwarded out of community C can call
|
|
`purge_prior_contributions(user, C, epoch_range)`, see a durable tombstone in
|
|
`Tag::PurgeTombstone`, get a `PurgeReceipt` with the watermark seqno, and observe
|
|
that C's ranking outputs no longer reflect any of their purged contributions.
|
|
Re-running the identical purge produces a byte-identical checkpoint (BLAKE3-equal).
|
|
The user's local profile decay scores and windowed counts are unchanged. After a
|
|
simulated partition heal, a pre-purge unhide does not revive a purged hard
|
|
negative. Community queries report `purge_watermark: Some(seqno)`. All prior
|
|
M8/M9 tests pass; `cargo fmt`, `cargo clippy -p tidaldb -D warnings`, and
|
|
`cargo test -p tidaldb --lib` are clean.
|