tidaldb/tidal/docs/planning/milestone-10/phase-3/OVERVIEW.md

# m10p3: Provenance, Explainability, and Remove-by-Scope

## Delivers

The read-side audit and revocation surface that closes the M9/M10 arc:
every ranking-affecting signal carries a `SignalProvenance` record
(`writer`, `scope`, `policy_version`, `membership_epoch`), users can remove
contributions by scope (`agent` / `community` / `session` / `local`) with
surgical precision (never globally), the explainability endpoint attributes
each top-ranked item to its policy-allowed contributing signals, and all of
these outcomes survive replay and failover deterministically via the same
tombstone + CRDT machinery built in M9p3. This is the "make influence
attributable and revocable" phase -- no new write primitives, but every
ranking output becomes explainable and every contribution becomes removable
by scope without destroying local history.

Deliverables:
- `SignalProvenance` (from M9p1) attached to every ledger entry via a parallel
  `DashMap<(EntityId, SignalTypeId), SignalProvenance>` in `SignalLedger`
- `RetrieveResult.signals` / `Signal` carry provenance so callers can audit
  per-signal `(writer, scope, policy_version, membership_epoch)`
- `TidalDb::remove_from_personalization(scope: RemoveScope)` -- precise,
  non-global deletion reusing the M9p3 rematerialization engine filtered by scope
- Explainability endpoint (`GET /explain`) attributing top items to
  policy-allowed signals with `(writer, policy_version, applied_weight)`
- `Results.explainability_context` carrying the per-item attribution graph
- Remove-by-scope writes `ScopeTombstone`s; CRDT `scope_tombstones` make merge
  idempotent so partition -> heal preserves removals (hard-negative leak guard)
- Multi-node UAT (`tidal/tests/m10_uat.rs`): partition -> heal -> assert removed
  agent contributions stay removed and local profile is untouched

## Dependencies

- **Requires:** M10p2 complete (`CapabilityToken`, `revoke_capability`,
  `PolicyEvaluator` capability phase). M10p1 complete (`GovernancePolicy`,
  `policy_version` threaded into results). M9p3 complete (`PurgeTombstone`,
  `governance/rematerialize.rs` deterministic replay engine, `CrdtSignalState`
  `scope_tombstones` extension point). M9p1 complete (the six shared primitives:
  `SignalScope`, `SignalProvenance`, `Membership`, `SharePolicy`,
  `CapabilityToken`, `GovernancePolicy`).
- **Files modified:**
  - `tidal/src/signals/ledger/core.rs` -- parallel provenance `DashMap`;
    `record_signal` / `apply_wal_event` thread provenance
  - `tidal/src/query/retrieve/types.rs` -- `Signal` gains provenance;
    `Results` gains `explainability_context`
  - `tidal/src/ranking/executor/mod.rs` -- carry provenance through scoring
  - `tidal/src/ranking/executor/context.rs` -- `ScoredCandidate` provenance slot
  - `tidal/src/db/http.rs` -- `GET /explain` route (feature = `metrics`)
  - `tidal/src/replication/reconcile.rs` -- scope-tombstone gating in `plan`/`apply`
  - `tidal/src/replication/crdt/signal_state.rs` -- `scope_tombstones` in `merge`
  - `tidal/src/entities/hard_neg.rs` -- `apply_replication_unhide` honors scope tombstone
  - `tidal/src/db/mod.rs` -- wire the remove-by-scope entry point
- **Files created:**
  - `tidal/src/db/remove_scope.rs` -- `RemoveScope`, `remove_from_personalization`
  - `tidal/src/query/explain.rs` -- `ExplainabilityContext`, `ItemAttribution`,
    `SignalAttribution`, `explain_results`
  - `tidal/tests/m10_uat.rs` -- M10 end-to-end UAT (governance + remove-by-scope)

## Research References

- `docs/research/tidaldb_signal_ledger.md` -- ledger entry layout, the
  parallel-map pattern for per-entry metadata
- `docs/research/tidaldb_wal.md` -- tombstone replay ordering `(hlc, seqno)`
- `thoughts.md` -- Part II.1 (WAL convergence), Part V.14 (post-scoring passes),
  hard-negative leak invariants
- `/tmp/m9m10_brief.md` -- sections 1.2 (`SignalProvenance`), 4.5
  (local-profile-intact), 4.6 (hard-negative leak on replay/failover)

## Acceptance Criteria (Phase Level)

- [ ] Every ledger entry that affects ranking has a `SignalProvenance` reachable
      in O(1) via the parallel `DashMap<(EntityId, SignalTypeId), SignalProvenance>`;
      a write with no explicit provenance defaults to `SignalScope::Local`,
      `writer_agent_id` = the system agent, current `membership_epoch`
- [ ] `RetrieveResult.signals[i]` exposes `provenance: SignalProvenance` so a
      caller can read `(writer_agent_id, scope, share_policy_version, membership_epoch)`
      for each contributing signal
- [ ] `remove_from_personalization(RemoveScope::Agent(agent_id))` removes only
      that agent's contributions; an integration test asserts the user's
      local-scope (`SignalScope::Local`) decay scores and windowed counts are
      bit-identical before and after (local-profile-intact)
- [ ] `RemoveScope` covers `Agent(AgentId)`, `Community(CommunityId)`,
      `Session(SessionId)`, and `Local`; each routes through the M9p3
      rematerialization engine with a scope-typed filter (no global wipe path)
- [ ] `GET /explain?user=<id>&profile=<name>&limit=<n>` returns, for each
      top-ranked item, the contributing signals with `(writer, policy_version,
      applied_weight)`; signals excluded by `GovernancePolicy` are omitted from
      the attribution (only policy-allowed signals are explained)
- [ ] Remove-by-scope writes `ScopeTombstone`s to `Tag::PurgeTombstone`; a
      double-invocation produces a BLAKE3-identical checkpoint payload
      (deterministic, idempotent)
- [ ] Multi-node UAT: partition -> remove agent scope on one node -> heal ->
      after convergence the removed agent's contributions are absent on all
      nodes AND no hard negative from the removed scope leaks back
- [ ] CRDT merge with `scope_tombstones` remains commutative, associative, and
      idempotent (proptest); a late pre-removal unhide cannot revive a
      tombstoned item (tombstone HLC dominates)
- [ ] `cargo fmt` clean, `cargo clippy -p tidaldb -D warnings` clean, all lib
      tests + `m10_uat` pass; tests are fast

## Task Execution Order

```
Task 01: Provenance map in SignalLedger ───┐
  (parallel DashMap, default Local)         │
                                            ├──> Task 03: remove_from_personalization
Task 02: Provenance on read surface ───────┤        (db/remove_scope.rs, reuse M9p3
  (Signal + ScoredCandidate provenance,     │         rematerialize, scope filter)
   executor threading)                      │              │
                                            │              v
                                            └──> Task 04: Explainability endpoint
                                                     (query/explain.rs + GET /explain)
                                                          │
                                                          v
                                            Task 05: Scope tombstones + CRDT merge
                                              (reconcile + signal_state + hard_neg)
                                                          │
                                                          v
                                            Task 06: M10 UAT (single + multi-node)
```

Tasks 01 and 02 are parallelizable (write-side map vs read-side surface), but
02 reads the map 01 introduces, so commit 01 first. Task 03 depends on 01 (the
provenance is the deletion key) and the M9p3 rematerialization engine. Task 04
depends on 01+02 (provenance is what it attributes). Task 05 makes 03's removals
durable across replay. Task 06 verifies the whole arc end-to-end.

## Module Location

| File | Status | Contains |
|------|--------|----------|
| `tidal/src/signals/ledger/core.rs` | MODIFIED | `provenance: DashMap<(EntityId, SignalTypeId), SignalProvenance>`; `record_signal_with_provenance`, `apply_wal_event` provenance threading, `provenance_for(entity, type)` |
| `tidal/src/query/retrieve/types.rs` | MODIFIED | `Signal.provenance: SignalProvenance`; `Results.explainability_context: Option<ExplainabilityContext>` |
| `tidal/src/ranking/executor/context.rs` | MODIFIED | `ScoredCandidate.signal_provenance: Vec<(String, SignalProvenance)>` |
| `tidal/src/ranking/executor/mod.rs` | MODIFIED | Thread provenance from ledger -> `ScoredCandidate` -> `RetrieveResult` |
| `tidal/src/db/remove_scope.rs` | NEW | `RemoveScope` enum, `TidalDb::remove_from_personalization`, scope-filtered tombstone writes |
| `tidal/src/query/explain.rs` | NEW | `ExplainabilityContext`, `ItemAttribution`, `SignalAttribution`, `explain_results` |
| `tidal/src/db/http.rs` | MODIFIED | `GET /explain` route + JSON renderer (feature = `metrics`) |
| `tidal/src/replication/reconcile.rs` | MODIFIED | `StateSnapshot` carries `ScopeTombstone`s; `plan`/`apply` gate by tombstone HLC |
| `tidal/src/replication/crdt/signal_state.rs` | MODIFIED | `scope_tombstones` field; `merge` applies max-HLC tombstone, drops dominated contributions |
| `tidal/src/entities/hard_neg.rs` | MODIFIED | `apply_replication_unhide` rejects unhide dominated by a scope tombstone |
| `tidal/src/db/mod.rs` | MODIFIED | Re-export `RemoveScope`; wire `remove_from_personalization` |
| `tidal/tests/m10_uat.rs` | NEW | M10 UAT: trusted/experimental agent, policy reweight, revoke + remove-by-scope, three views, partition/heal |

## Technical Design

### Provenance attachment in `SignalLedger`

`SignalProvenance` is the M9p1 primitive (`tidal/src/governance/provenance.rs`),
referenced here, NOT redefined:

```rust
// DEFINED in M9p1 (tidal/src/governance/provenance.rs) -- shown for reference only.
//
// `Copy` where possible: AgentId is interned to a u16 on the ledger hot path
// (the full AgentId string lives in an intern table), so the on-entry record
// stays small and cache-friendly.
#[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
pub struct SignalProvenance {
    /// Interned id of the agent that wrote this signal.
    pub writer_agent_id: WriterAgentId,
    /// Shard that originated the write.
    pub origin_shard: ShardId,
    /// Sharing scope at write time. Default `Local`.
    pub scope: SignalScope,
    /// Version of the `SharePolicy` in force when written.
    pub share_policy_version: u16,
    /// Membership epoch in force when written (0 for non-community scope).
    pub membership_epoch: u32,
    /// HLC timestamp of the write (causal ordering for tombstone dominance).
    pub hlc: HlcTimestamp,
}
```

The ledger gains a parallel map keyed identically to `entries`, so provenance
lookup is the same O(1) `DashMap` access already used for signal state:

```rust
// tidal/src/signals/ledger/core.rs (MODIFIED)
pub struct SignalLedger {
    pub(crate) entries: DashMap<(EntityId, SignalTypeId), EntitySignalEntry>,
    /// Provenance for the most recent ranking-affecting write per
    /// (entity, signal_type). Parallel to `entries`; same key, same shard
    /// count so the two maps stay cache-aligned during scoring.
    pub(crate) provenance: DashMap<(EntityId, SignalTypeId), SignalProvenance>,
    wal: Box<dyn WalWriter>,
    schema: Schema,
    signal_name_to_id: HashMap<String, SignalTypeId>,
    signal_lambdas: HashMap<SignalTypeId, Vec<f64>>,
}

impl SignalLedger {
    /// Record a signal with explicit provenance (community/agent/session writes).
    ///
    /// Backward-compatible: `record_signal` delegates here with
    /// `SignalProvenance::local_default(entity_id)`, so existing call sites
    /// keep working and every entry still gets a provenance record.
    ///
    /// # Errors
    /// - `TidalError::Schema` if `signal_type_name` is unknown
    /// - `TidalError::Durability` if the WAL write fails
    pub fn record_signal_with_provenance(
        &self,
        signal_type_name: &str,
        entity_id: EntityId,
        weight: f64,
        timestamp: Timestamp,
        provenance: SignalProvenance,
    ) -> crate::Result<()>;

    /// Provenance for a ledger entry, if any signal has been recorded.
    #[must_use]
    pub fn provenance_for(
        &self,
        entity_id: EntityId,
        signal_type_id: SignalTypeId,
    ) -> Option<SignalProvenance>;
}
```

`apply_wal_event` (replay path) is widened to accept and store provenance so a
rebuilt ledger is provenance-identical to the live one:

```rust
// tidal/src/signals/ledger/core.rs (MODIFIED signature)
pub(crate) fn apply_wal_event(
    &self,
    signal_type_id: SignalTypeId,
    entity_id: EntityId,
    weight: f64,
    timestamp: Timestamp,
    provenance: SignalProvenance, // NEW: decoded from WAL V3 per-event metadata
);
```

### Provenance on the read surface

`Signal` (the per-result snapshot) gains the provenance record; `ScoredCandidate`
carries it through the scoring pipeline:

```rust
// tidal/src/query/retrieve/types.rs (MODIFIED)
#[derive(Debug, Clone)]
pub struct Signal {
    pub name: String,
    pub value: f64,
    pub source: String,
    /// Provenance of the contributing signal: who wrote it, in what scope,
    /// under which policy/membership version. Defaults to `Local` provenance
    /// for signals written before M10p3 or via the legacy path.
    pub provenance: SignalProvenance,
}

// tidal/src/ranking/executor/context.rs (MODIFIED)
#[derive(Debug, Clone)]
pub struct ScoredCandidate {
    pub entity_id: EntityId,
    pub score: f64,
    pub signal_snapshot: Vec<(String, f64)>,
    /// Provenance for each entry in `signal_snapshot`, index-aligned.
    pub signal_provenance: Vec<(String, SignalProvenance)>,
    pub creator_id: Option<EntityId>,
    pub format: Option<String>,
}
```

### Remove-by-scope

```rust
// tidal/src/db/remove_scope.rs (NEW)

use crate::governance::{CommunityId, SignalScope};
use crate::schema::EntityId;
use crate::session::types::{AgentId, SessionId};

/// The scope of a `remove_from_personalization` call.
///
/// Each variant removes exactly the contributions matching its scope and
/// nothing else. There is intentionally NO `All`/global variant: removal is
/// always surgical so the user-owned local profile is never collateral damage.
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
pub enum RemoveScope {
    /// Remove every contribution written by this agent (matched on
    /// `SignalProvenance.writer_agent_id`).
    Agent(AgentId),
    /// Remove this user's contributions made under a community overlay.
    Community { user_id: EntityId, community_id: CommunityId },
    /// Remove contributions written in a specific session.
    Session(SessionId),
    /// Remove the user's local-scope contributions (explicit, user-initiated
    /// only -- community/agent ops can never reach this variant).
    Local { user_id: EntityId },
}

impl RemoveScope {
    /// The `SignalScope` family this removal targets, used to short-circuit
    /// rematerialization so untouched scopes are never replayed.
    #[must_use]
    pub fn signal_scope_filter(&self) -> ScopeFilter;
}

/// Outcome of a remove-by-scope operation.
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
#[must_use]
pub struct RemoveOutcome {
    /// Tombstone watermark seqno written for this removal.
    pub watermark_seqno: u64,
    /// Number of ledger entries rematerialized.
    pub entries_rematerialized: usize,
    /// HLC stamped on the scope tombstone (dominates later replays).
    pub tombstone_hlc: HlcTimestamp,
}

impl TidalDb {
    /// Remove contributions from future ranking, precisely by scope.
    ///
    /// Writes a `ScopeTombstone` (durable, in `Tag::PurgeTombstone`) and runs
    /// the M9p3 deterministic rematerialization engine filtered to the matching
    /// scope: community/warm/hot tiers for the affected entities are recomputed
    /// from the WAL with the tombstoned contributions excluded. Local-scope
    /// state is never touched unless `RemoveScope::Local` is explicitly passed.
    ///
    /// Deterministic and idempotent: replaying the WAL (restart/failover) or
    /// merging across nodes reproduces the identical removed state because the
    /// tombstone HLC dominates any pre-removal contribution.
    ///
    /// # Errors
    /// - `TidalError::NotFound` if the targeted scope has no contributions
    /// - `TidalError::Durability` if the tombstone write fails
    pub fn remove_from_personalization(
        &self,
        scope: RemoveScope,
    ) -> crate::Result<RemoveOutcome>;
}
```

### Explainability

```rust
// tidal/src/query/explain.rs (NEW)

/// Per-query attribution graph: why each top item ranked where it did,
/// restricted to signals the governing `GovernancePolicy` allows.
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct ExplainabilityContext {
    /// One attribution per returned item, in result order.
    pub items: Vec<ItemAttribution>,
    /// The governing policy version applied to this query (from M10p1).
    pub policy_version: u32,
}

#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct ItemAttribution {
    pub entity_id: EntityId,
    pub rank: usize,
    pub score: f64,
    /// Only policy-allowed signals appear here; excluded intents are dropped.
    pub signals: Vec<SignalAttribution>,
}

#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct SignalAttribution {
    pub name: String,
    /// Resolved writer agent id (de-interned for human-readable output).
    pub writer: String,
    /// Governance policy version under which this signal was admitted.
    pub policy_version: u32,
    /// The weight this signal contributed to the final score.
    pub applied_weight: f64,
}

/// Build the attribution graph from scored results + ledger provenance,
/// filtering out signals excluded by the active governance policy.
#[must_use]
pub fn explain_results(
    results: &Results,
    policy: &GovernancePolicy,
) -> ExplainabilityContext;
```

The HTTP surface reuses the existing `db/http.rs` path-routing in
`handle_connection`, adding one arm (feature = `metrics`):

```rust
// tidal/src/db/http.rs (MODIFIED handle_connection match)
let (status, content_type, body) = match path {
    "/healthz" => ("200 OK", "application/json", state.render_healthz()),
    "/metrics" => ("200 OK", "text/plain; version=0.0.4; charset=utf-8",
                   state.render_prometheus()),
    p if p.starts_with("/explain") => (
        "200 OK",
        "application/json",
        // Parses ?user=&profile=&limit=, runs the query, attributes via
        // explain_results(&results, &active_policy), serializes to JSON.
        state.render_explain(p),
    ),
    _ => ("404 Not Found", "application/json", r#"{"error":"not found"}"#.into()),
};
```

### Scope tombstones in the CRDT merge

`CrdtSignalState` gains a `scope_tombstones` field; tombstone application is
max-HLC-wins, so it is order-independent and preserves the merge laws already
documented on the struct (commutative / associative / idempotent):

```rust
// tidal/src/replication/crdt/signal_state.rs (MODIFIED)
pub struct CrdtSignalState {
    node_decay_scores: HashMap<ShardId, f64>,
    node_last_update_ns: HashMap<ShardId, u64>,
    // ...existing fields...
    /// Scope removals that dominate contributions older than their HLC.
    /// Keyed by the removed scope; value is the tombstone HLC. A node's
    /// contribution is dropped at merge/decay time iff its last_update HLC
    /// is dominated by a matching tombstone HLC. Max-HLC-wins, so merge stays
    /// commutative/associative/idempotent.
    scope_tombstones: BTreeMap<ScopeKey, HlcTimestamp>,
}

impl CrdtSignalState {
    pub fn merge(&mut self, other: &Self) {
        // ...existing per-node LWW merge...
        // Tombstones: keep the max HLC per scope key (idempotent union).
        for (scope, &ts) in &other.scope_tombstones {
            let entry = self.scope_tombstones.entry(scope.clone()).or_insert(ts);
            if ts > *entry { *entry = ts; }
        }
        // After merging, drop any node contribution dominated by a tombstone.
        self.apply_scope_tombstones();
    }
}
```

`apply_replication_unhide` gains a tombstone guard so a stale unhide cannot
revive a removed hard negative after failover:

```rust
// tidal/src/entities/hard_neg.rs (MODIFIED)
/// An unhide is rejected if a scope tombstone with a dominating HLC removed
/// the contributing scope -- this closes the hard-negative leak on replay.
#[must_use]
pub fn apply_replication_unhide(
    &self,
    user_id: u64,
    item_id: u32,
    ts: HlcTimestamp,
    scope_tombstone_hlc: Option<HlcTimestamp>, // NEW
) -> bool;
```

## Notes

### Local-profile-intact is the load-bearing invariant (brief 4.5)

`RemoveScope` has NO global variant by construction. `Community`, `Agent`, and
`Session` removals filter strictly by `SignalProvenance`; a routing bug that
touches `SignalScope::Local` is silent data loss. Every UAT step asserts the
user's local decay scores and windowed counts are bit-identical across
join / leave / purge / revoke / remove-by-scope. `RemoveScope::Local` is the
only path that can affect local state and is reachable only by an explicit,
user-initiated call.

### <1s gates are synchronous atomic reads, never the sweeper (brief 4.4)

Stop-forward (M9p2) and capability revocation (M10p2) gates are O(1) atomic
`stop_forward_at_ns` / `revoked_at_ns` reads on the write path. Remove-by-scope
is a heavier rematerialization job and is NOT on that <1s path -- it is the
"optional retroactive removal" with its own SLA in the UAT. Do not route
remove-by-scope through the 60s sweeper (`db/sweeper.rs`) and do not couple it
to the revocation gate; they are distinct latency classes.

### Determinism + idempotence (brief 4.2, 4.3)

Remove-by-scope reuses the M9p3 rematerialization engine, which sorts replay by
`(hlc, seqno)` and applies forward-decay identically each run. The acceptance
test runs the removal twice and BLAKE3-hashes the resulting checkpoint payloads,
asserting equality. The CRDT `scope_tombstones` are max-HLC-wins, so partition
-> heal converges regardless of merge order, and replaying the same segment
twice is a no-op.

### Hard-negative leak on replay/failover (brief 4.6)

The explicit UAT requirement: hard negatives from a removed scope must not leak
back after replay. `apply_replication_unhide` already uses LWW with HLC gating;
the new tombstone guard ensures a late pre-removal unhide (whose HLC predates
the tombstone) cannot revive a tombstoned item. The tombstone HLC must dominate.

### Backward compatibility (non-negotiable)

WAL V3 (from M9p1) already carries per-event scope/writer/policy/epoch metadata;
m10p3 reads it to populate provenance during replay. V1/V2 events zero-fill the
provenance fields and decode as `SignalProvenance::local_default`, so older WAL
segments and checkpoints remain readable and every entry still has a provenance
record. The provenance `DashMap` is checkpointed alongside `entries` in the V3
checkpoint entry; restoring a pre-V3 checkpoint synthesizes local-default
provenance so no migration step is required.

### `AgentId` is not `Copy`

`AgentId` is a validated `String` newtype (`session/types.rs`), so it is not
`Copy`. `SignalProvenance` stores an interned `WriterAgentId(u16)` on the hot
path to stay `Copy` and cache-friendly; the de-interned agent string is resolved
only when building human-readable explainability output, never during scoring.

## Done When

A user with two agents can revoke an agent's community scope and call
`remove_from_personalization(RemoveScope::Agent(a_trusted))` to surgically erase
that agent's contributions from community ranking while their local profile
stays bit-identical; `GET /explain` attributes each top item to its
policy-allowed signals with `(writer, policy_version, applied_weight)`; and a
multi-node partition -> heal cycle leaves the removed contributions removed on
every node with no hard-negative leak -- all verified by `tidal/tests/m10_uat.rs`
with `cargo fmt`, `cargo clippy -p tidaldb -D warnings`, and the full lib + UAT
suites passing fast.