tidaldb/docs/planning/milestone-10/phase-3/OVERVIEW.md

m10p3: Provenance, Explainability, and Remove-by-Scope

Delivers

The read-side audit and revocation surface that closes the M9/M10 arc: every ranking-affecting signal carries a SignalProvenance record (writer, scope, policy_version, membership_epoch), users can remove contributions by scope (agent / community / session / local) with surgical precision (never globally), the explainability endpoint attributes each top-ranked item to its policy-allowed contributing signals, and all of these outcomes survive replay and failover deterministically via the same tombstone + CRDT machinery built in M9p3. This is the "make influence attributable and revocable" phase -- no new write primitives, but every ranking output becomes explainable and every contribution becomes removable by scope without destroying local history.

Deliverables:

• SignalProvenance (from M9p1) attached to every ledger entry via a parallel DashMap<(EntityId, SignalTypeId), SignalProvenance> in SignalLedger
• RetrieveResult.signals / Signal carry provenance so callers can audit per-signal (writer, scope, policy_version, membership_epoch)
• TidalDb::remove_from_personalization(scope: RemoveScope) -- precise, non-global deletion reusing the M9p3 rematerialization engine filtered by scope
• Explainability endpoint (GET /explain) attributing top items to policy-allowed signals with (writer, policy_version, applied_weight)
• Results.explainability_context carrying the per-item attribution graph
• Remove-by-scope writes ScopeTombstones; CRDT scope_tombstones make merge idempotent so partition -> heal preserves removals (hard-negative leak guard)
• Multi-node UAT (tidal/tests/m10_uat.rs): partition -> heal -> assert removed agent contributions stay removed and local profile is untouched

Dependencies

• Requires: M10p2 complete (CapabilityToken, revoke_capability, PolicyEvaluator capability phase). M10p1 complete (GovernancePolicy, policy_version threaded into results). M9p3 complete (PurgeTombstone, governance/rematerialize.rs deterministic replay engine, CrdtSignalState scope_tombstones extension point). M9p1 complete (the six shared primitives: SignalScope, SignalProvenance, Membership, SharePolicy, CapabilityToken, GovernancePolicy).
• Files modified:
  • tidal/src/signals/ledger/core.rs -- parallel provenance DashMap; record_signal / apply_wal_event thread provenance
  • tidal/src/query/retrieve/types.rs -- Signal gains provenance; Results gains explainability_context
  • tidal/src/ranking/executor/mod.rs -- carry provenance through scoring
  • tidal/src/ranking/executor/context.rs -- ScoredCandidate provenance slot
  • tidal/src/db/http.rs -- GET /explain route (feature = metrics)
  • tidal/src/replication/reconcile.rs -- scope-tombstone gating in plan/apply
  • tidal/src/replication/crdt/signal_state.rs -- scope_tombstones in merge
  • tidal/src/entities/hard_neg.rs -- apply_replication_unhide honors scope tombstone
  • tidal/src/db/mod.rs -- wire the remove-by-scope entry point
• Files created:
  • tidal/src/db/remove_scope.rs -- RemoveScope, remove_from_personalization
  • tidal/src/query/explain.rs -- ExplainabilityContext, ItemAttribution, SignalAttribution, explain_results
  • tidal/tests/m10_uat.rs -- M10 end-to-end UAT (governance + remove-by-scope)

Research References

• docs/research/tidaldb_signal_ledger.md -- ledger entry layout, the parallel-map pattern for per-entry metadata
• docs/research/tidaldb_wal.md -- tombstone replay ordering (hlc, seqno)
• thoughts.md -- Part II.1 (WAL convergence), Part V.14 (post-scoring passes), hard-negative leak invariants
• /tmp/m9m10_brief.md -- sections 1.2 (SignalProvenance), 4.5 (local-profile-intact), 4.6 (hard-negative leak on replay/failover)

Acceptance Criteria (Phase Level)

• Every ledger entry that affects ranking has a SignalProvenance reachable in O(1) via the parallel DashMap<(EntityId, SignalTypeId), SignalProvenance>; a write with no explicit provenance defaults to SignalScope::Local, writer_agent_id = the system agent, current membership_epoch
• RetrieveResult.signals[i] exposes provenance: SignalProvenance so a caller can read (writer_agent_id, scope, share_policy_version, membership_epoch) for each contributing signal
• remove_from_personalization(RemoveScope::Agent(agent_id)) removes only that agent's contributions; an integration test asserts the user's local-scope (SignalScope::Local) decay scores and windowed counts are bit-identical before and after (local-profile-intact)
• RemoveScope covers Agent(AgentId), Community(CommunityId), Session(SessionId), and Local; each routes through the M9p3 rematerialization engine with a scope-typed filter (no global wipe path)
• GET /explain?user=<id>&profile=<name>&limit=<n> returns, for each top-ranked item, the contributing signals with (writer, policy_version, applied_weight); signals excluded by GovernancePolicy are omitted from the attribution (only policy-allowed signals are explained)
• Remove-by-scope writes ScopeTombstones to Tag::PurgeTombstone; a double-invocation produces a BLAKE3-identical checkpoint payload (deterministic, idempotent)
• Multi-node UAT: partition -> remove agent scope on one node -> heal -> after convergence the removed agent's contributions are absent on all nodes AND no hard negative from the removed scope leaks back
• CRDT merge with scope_tombstones remains commutative, associative, and idempotent (proptest); a late pre-removal unhide cannot revive a tombstoned item (tombstone HLC dominates)
• cargo fmt clean, cargo clippy -p tidaldb -D warnings clean, all lib tests + m10_uat pass; tests are fast

Task Execution Order

Task 01: Provenance map in SignalLedger ───┐
  (parallel DashMap, default Local)         │
                                            ├──> Task 03: remove_from_personalization
Task 02: Provenance on read surface ───────┤        (db/remove_scope.rs, reuse M9p3
  (Signal + ScoredCandidate provenance,     │         rematerialize, scope filter)
   executor threading)                      │              │
                                            │              v
                                            └──> Task 04: Explainability endpoint
                                                     (query/explain.rs + GET /explain)
                                                          │
                                                          v
                                            Task 05: Scope tombstones + CRDT merge
                                              (reconcile + signal_state + hard_neg)
                                                          │
                                                          v
                                            Task 06: M10 UAT (single + multi-node)

Tasks 01 and 02 are parallelizable (write-side map vs read-side surface), but 02 reads the map 01 introduces, so commit 01 first. Task 03 depends on 01 (the provenance is the deletion key) and the M9p3 rematerialization engine. Task 04 depends on 01+02 (provenance is what it attributes). Task 05 makes 03's removals durable across replay. Task 06 verifies the whole arc end-to-end.

Module Location

File	Status	Contains
tidal/src/signals/ledger/core.rs	MODIFIED	provenance: DashMap<(EntityId, SignalTypeId), SignalProvenance>; record_signal_with_provenance, apply_wal_event provenance threading, provenance_for(entity, type)
tidal/src/query/retrieve/types.rs	MODIFIED	Signal.provenance: SignalProvenance; Results.explainability_context: Option<ExplainabilityContext>
tidal/src/ranking/executor/context.rs	MODIFIED	ScoredCandidate.signal_provenance: Vec<(String, SignalProvenance)>
tidal/src/ranking/executor/mod.rs	MODIFIED	Thread provenance from ledger -> ScoredCandidate -> RetrieveResult
tidal/src/db/remove_scope.rs	NEW	RemoveScope enum, TidalDb::remove_from_personalization, scope-filtered tombstone writes
tidal/src/query/explain.rs	NEW	ExplainabilityContext, ItemAttribution, SignalAttribution, explain_results
tidal/src/db/http.rs	MODIFIED	GET /explain route + JSON renderer (feature = metrics)
tidal/src/replication/reconcile.rs	MODIFIED	StateSnapshot carries ScopeTombstones; plan/apply gate by tombstone HLC
tidal/src/replication/crdt/signal_state.rs	MODIFIED	scope_tombstones field; merge applies max-HLC tombstone, drops dominated contributions
tidal/src/entities/hard_neg.rs	MODIFIED	apply_replication_unhide rejects unhide dominated by a scope tombstone
tidal/src/db/mod.rs	MODIFIED	Re-export RemoveScope; wire remove_from_personalization
tidal/tests/m10_uat.rs	NEW	M10 UAT: trusted/experimental agent, policy reweight, revoke + remove-by-scope, three views, partition/heal

Technical Design

Provenance attachment in SignalLedger

SignalProvenance is the M9p1 primitive (tidal/src/governance/provenance.rs), referenced here, NOT redefined:

// DEFINED in M9p1 (tidal/src/governance/provenance.rs) -- shown for reference only.
//
// `Copy` where possible: AgentId is interned to a u16 on the ledger hot path
// (the full AgentId string lives in an intern table), so the on-entry record
// stays small and cache-friendly.
#[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
pub struct SignalProvenance {
    /// Interned id of the agent that wrote this signal.
    pub writer_agent_id: WriterAgentId,
    /// Shard that originated the write.
    pub origin_shard: ShardId,
    /// Sharing scope at write time. Default `Local`.
    pub scope: SignalScope,
    /// Version of the `SharePolicy` in force when written.
    pub share_policy_version: u16,
    /// Membership epoch in force when written (0 for non-community scope).
    pub membership_epoch: u32,
    /// HLC timestamp of the write (causal ordering for tombstone dominance).
    pub hlc: HlcTimestamp,
}

The ledger gains a parallel map keyed identically to entries, so provenance lookup is the same O(1) DashMap access already used for signal state:

// tidal/src/signals/ledger/core.rs (MODIFIED)
pub struct SignalLedger {
    pub(crate) entries: DashMap<(EntityId, SignalTypeId), EntitySignalEntry>,
    /// Provenance for the most recent ranking-affecting write per
    /// (entity, signal_type). Parallel to `entries`; same key, same shard
    /// count so the two maps stay cache-aligned during scoring.
    pub(crate) provenance: DashMap<(EntityId, SignalTypeId), SignalProvenance>,
    wal: Box<dyn WalWriter>,
    schema: Schema,
    signal_name_to_id: HashMap<String, SignalTypeId>,
    signal_lambdas: HashMap<SignalTypeId, Vec<f64>>,
}

impl SignalLedger {
    /// Record a signal with explicit provenance (community/agent/session writes).
    ///
    /// Backward-compatible: `record_signal` delegates here with
    /// `SignalProvenance::local_default(entity_id)`, so existing call sites
    /// keep working and every entry still gets a provenance record.
    ///
    /// # Errors
    /// - `TidalError::Schema` if `signal_type_name` is unknown
    /// - `TidalError::Durability` if the WAL write fails
    pub fn record_signal_with_provenance(
        &self,
        signal_type_name: &str,
        entity_id: EntityId,
        weight: f64,
        timestamp: Timestamp,
        provenance: SignalProvenance,
    ) -> crate::Result<()>;

    /// Provenance for a ledger entry, if any signal has been recorded.
    #[must_use]
    pub fn provenance_for(
        &self,
        entity_id: EntityId,
        signal_type_id: SignalTypeId,
    ) -> Option<SignalProvenance>;
}

apply_wal_event (replay path) is widened to accept and store provenance so a rebuilt ledger is provenance-identical to the live one:

// tidal/src/signals/ledger/core.rs (MODIFIED signature)
pub(crate) fn apply_wal_event(
    &self,
    signal_type_id: SignalTypeId,
    entity_id: EntityId,
    weight: f64,
    timestamp: Timestamp,
    provenance: SignalProvenance, // NEW: decoded from WAL V3 per-event metadata
);

Provenance on the read surface

Signal (the per-result snapshot) gains the provenance record; ScoredCandidate carries it through the scoring pipeline:

// tidal/src/query/retrieve/types.rs (MODIFIED)
#[derive(Debug, Clone)]
pub struct Signal {
    pub name: String,
    pub value: f64,
    pub source: String,
    /// Provenance of the contributing signal: who wrote it, in what scope,
    /// under which policy/membership version. Defaults to `Local` provenance
    /// for signals written before M10p3 or via the legacy path.
    pub provenance: SignalProvenance,
}

// tidal/src/ranking/executor/context.rs (MODIFIED)
#[derive(Debug, Clone)]
pub struct ScoredCandidate {
    pub entity_id: EntityId,
    pub score: f64,
    pub signal_snapshot: Vec<(String, f64)>,
    /// Provenance for each entry in `signal_snapshot`, index-aligned.
    pub signal_provenance: Vec<(String, SignalProvenance)>,
    pub creator_id: Option<EntityId>,
    pub format: Option<String>,
}

Remove-by-scope

// tidal/src/db/remove_scope.rs (NEW)

use crate::governance::{CommunityId, SignalScope};
use crate::schema::EntityId;
use crate::session::types::{AgentId, SessionId};

/// The scope of a `remove_from_personalization` call.
///
/// Each variant removes exactly the contributions matching its scope and
/// nothing else. There is intentionally NO `All`/global variant: removal is
/// always surgical so the user-owned local profile is never collateral damage.
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
pub enum RemoveScope {
    /// Remove every contribution written by this agent (matched on
    /// `SignalProvenance.writer_agent_id`).
    Agent(AgentId),
    /// Remove this user's contributions made under a community overlay.
    Community { user_id: EntityId, community_id: CommunityId },
    /// Remove contributions written in a specific session.
    Session(SessionId),
    /// Remove the user's local-scope contributions (explicit, user-initiated
    /// only -- community/agent ops can never reach this variant).
    Local { user_id: EntityId },
}

impl RemoveScope {
    /// The `SignalScope` family this removal targets, used to short-circuit
    /// rematerialization so untouched scopes are never replayed.
    #[must_use]
    pub fn signal_scope_filter(&self) -> ScopeFilter;
}

/// Outcome of a remove-by-scope operation.
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
#[must_use]
pub struct RemoveOutcome {
    /// Tombstone watermark seqno written for this removal.
    pub watermark_seqno: u64,
    /// Number of ledger entries rematerialized.
    pub entries_rematerialized: usize,
    /// HLC stamped on the scope tombstone (dominates later replays).
    pub tombstone_hlc: HlcTimestamp,
}

impl TidalDb {
    /// Remove contributions from future ranking, precisely by scope.
    ///
    /// Writes a `ScopeTombstone` (durable, in `Tag::PurgeTombstone`) and runs
    /// the M9p3 deterministic rematerialization engine filtered to the matching
    /// scope: community/warm/hot tiers for the affected entities are recomputed
    /// from the WAL with the tombstoned contributions excluded. Local-scope
    /// state is never touched unless `RemoveScope::Local` is explicitly passed.
    ///
    /// Deterministic and idempotent: replaying the WAL (restart/failover) or
    /// merging across nodes reproduces the identical removed state because the
    /// tombstone HLC dominates any pre-removal contribution.
    ///
    /// # Errors
    /// - `TidalError::NotFound` if the targeted scope has no contributions
    /// - `TidalError::Durability` if the tombstone write fails
    pub fn remove_from_personalization(
        &self,
        scope: RemoveScope,
    ) -> crate::Result<RemoveOutcome>;
}

Explainability

// tidal/src/query/explain.rs (NEW)

/// Per-query attribution graph: why each top item ranked where it did,
/// restricted to signals the governing `GovernancePolicy` allows.
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct ExplainabilityContext {
    /// One attribution per returned item, in result order.
    pub items: Vec<ItemAttribution>,
    /// The governing policy version applied to this query (from M10p1).
    pub policy_version: u32,
}

#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct ItemAttribution {
    pub entity_id: EntityId,
    pub rank: usize,
    pub score: f64,
    /// Only policy-allowed signals appear here; excluded intents are dropped.
    pub signals: Vec<SignalAttribution>,
}

#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct SignalAttribution {
    pub name: String,
    /// Resolved writer agent id (de-interned for human-readable output).
    pub writer: String,
    /// Governance policy version under which this signal was admitted.
    pub policy_version: u32,
    /// The weight this signal contributed to the final score.
    pub applied_weight: f64,
}

/// Build the attribution graph from scored results + ledger provenance,
/// filtering out signals excluded by the active governance policy.
#[must_use]
pub fn explain_results(
    results: &Results,
    policy: &GovernancePolicy,
) -> ExplainabilityContext;

The HTTP surface reuses the existing db/http.rs path-routing in handle_connection, adding one arm (feature = metrics):

// tidal/src/db/http.rs (MODIFIED handle_connection match)
let (status, content_type, body) = match path {
    "/healthz" => ("200 OK", "application/json", state.render_healthz()),
    "/metrics" => ("200 OK", "text/plain; version=0.0.4; charset=utf-8",
                   state.render_prometheus()),
    p if p.starts_with("/explain") => (
        "200 OK",
        "application/json",
        // Parses ?user=&profile=&limit=, runs the query, attributes via
        // explain_results(&results, &active_policy), serializes to JSON.
        state.render_explain(p),
    ),
    _ => ("404 Not Found", "application/json", r#"{"error":"not found"}"#.into()),
};

Scope tombstones in the CRDT merge

CrdtSignalState gains a scope_tombstones field; tombstone application is max-HLC-wins, so it is order-independent and preserves the merge laws already documented on the struct (commutative / associative / idempotent):

// tidal/src/replication/crdt/signal_state.rs (MODIFIED)
pub struct CrdtSignalState {
    node_decay_scores: HashMap<ShardId, f64>,
    node_last_update_ns: HashMap<ShardId, u64>,
    // ...existing fields...
    /// Scope removals that dominate contributions older than their HLC.
    /// Keyed by the removed scope; value is the tombstone HLC. A node's
    /// contribution is dropped at merge/decay time iff its last_update HLC
    /// is dominated by a matching tombstone HLC. Max-HLC-wins, so merge stays
    /// commutative/associative/idempotent.
    scope_tombstones: BTreeMap<ScopeKey, HlcTimestamp>,
}

impl CrdtSignalState {
    pub fn merge(&mut self, other: &Self) {
        // ...existing per-node LWW merge...
        // Tombstones: keep the max HLC per scope key (idempotent union).
        for (scope, &ts) in &other.scope_tombstones {
            let entry = self.scope_tombstones.entry(scope.clone()).or_insert(ts);
            if ts > *entry { *entry = ts; }
        }
        // After merging, drop any node contribution dominated by a tombstone.
        self.apply_scope_tombstones();
    }
}

apply_replication_unhide gains a tombstone guard so a stale unhide cannot revive a removed hard negative after failover:

// tidal/src/entities/hard_neg.rs (MODIFIED)
/// An unhide is rejected if a scope tombstone with a dominating HLC removed
/// the contributing scope -- this closes the hard-negative leak on replay.
#[must_use]
pub fn apply_replication_unhide(
    &self,
    user_id: u64,
    item_id: u32,
    ts: HlcTimestamp,
    scope_tombstone_hlc: Option<HlcTimestamp>, // NEW
) -> bool;

Notes

Local-profile-intact is the load-bearing invariant (brief 4.5)

RemoveScope has NO global variant by construction. Community, Agent, and Session removals filter strictly by SignalProvenance; a routing bug that touches SignalScope::Local is silent data loss. Every UAT step asserts the user's local decay scores and windowed counts are bit-identical across join / leave / purge / revoke / remove-by-scope. RemoveScope::Local is the only path that can affect local state and is reachable only by an explicit, user-initiated call.

<1s gates are synchronous atomic reads, never the sweeper (brief 4.4)

Stop-forward (M9p2) and capability revocation (M10p2) gates are O(1) atomic stop_forward_at_ns / revoked_at_ns reads on the write path. Remove-by-scope is a heavier rematerialization job and is NOT on that <1s path -- it is the "optional retroactive removal" with its own SLA in the UAT. Do not route remove-by-scope through the 60s sweeper (db/sweeper.rs) and do not couple it to the revocation gate; they are distinct latency classes.

Determinism + idempotence (brief 4.2, 4.3)

Remove-by-scope reuses the M9p3 rematerialization engine, which sorts replay by (hlc, seqno) and applies forward-decay identically each run. The acceptance test runs the removal twice and BLAKE3-hashes the resulting checkpoint payloads, asserting equality. The CRDT scope_tombstones are max-HLC-wins, so partition -> heal converges regardless of merge order, and replaying the same segment twice is a no-op.

Hard-negative leak on replay/failover (brief 4.6)

The explicit UAT requirement: hard negatives from a removed scope must not leak back after replay. apply_replication_unhide already uses LWW with HLC gating; the new tombstone guard ensures a late pre-removal unhide (whose HLC predates the tombstone) cannot revive a tombstoned item. The tombstone HLC must dominate.

Backward compatibility (non-negotiable)

WAL V3 (from M9p1) already carries per-event scope/writer/policy/epoch metadata; m10p3 reads it to populate provenance during replay. V1/V2 events zero-fill the provenance fields and decode as SignalProvenance::local_default, so older WAL segments and checkpoints remain readable and every entry still has a provenance record. The provenance DashMap is checkpointed alongside entries in the V3 checkpoint entry; restoring a pre-V3 checkpoint synthesizes local-default provenance so no migration step is required.

AgentId is not Copy

AgentId is a validated String newtype (session/types.rs), so it is not Copy. SignalProvenance stores an interned WriterAgentId(u16) on the hot path to stay Copy and cache-friendly; the de-interned agent string is resolved only when building human-readable explainability output, never during scoring.

Done When

A user with two agents can revoke an agent's community scope and call remove_from_personalization(RemoveScope::Agent(a_trusted)) to surgically erase that agent's contributions from community ranking while their local profile stays bit-identical; GET /explain attributes each top item to its policy-allowed signals with (writer, policy_version, applied_weight); and a multi-node partition -> heal cycle leaves the removed contributions removed on every node with no hard-negative leak -- all verified by tidal/tests/m10_uat.rs with cargo fmt, cargo clippy -p tidaldb -D warnings, and the full lib + UAT suites passing fast.