3bcfb3c576
- Add BUILD.bazel across tidal, tidal-net, tidal-server, tidalctl for bzlmod build - Add tidal/ crate docs (README, CHANGELOG, CONTRIBUTING, AGENTS, CLAUDE, API, ARCHITECTURE) and ai-lookup reference - Add docker standalone/cluster/deploy images, compose, and prometheus config - Harden WAL (batch format, writer, dedup, diagnostics), text syncer/collectors, and vector registry - Expand tidalctl CLI and tests; restructure WAL/visibility integration test suites - Refine tidal-net transport/client/server and tidal-server cluster/scatter-gather
534 lines
26 KiB
Markdown
534 lines
26 KiB
Markdown
# m9p1: Signal Scope and Share Contract
|
||
|
||
## Delivers
|
||
|
||
The shared governance atoms that every subsequent M9/M10 phase extends rather
|
||
than redefines, plus the WAL V3 wire format that carries scope and share
|
||
metadata per event. After this phase, every signal has an explicit scope
|
||
(`Local` by default), the WAL event envelope records who wrote it under which
|
||
share policy and membership epoch, and community replication only ships
|
||
share-eligible events. This is the "build the atoms right" phase (mirroring
|
||
m8p1): a new `tidal/src/governance/` module owns the canonical definitions of
|
||
`SignalScope`, `SignalProvenance`, `SharePolicy`, and `CommunityId`; later
|
||
phases reference these exact shapes. No community lifecycle yet — that is m9p2.
|
||
|
||
Deliverables:
|
||
- New module root `tidal/src/governance/mod.rs`, wired into `lib.rs` with `pub mod governance;` (parallels `replication`)
|
||
- `SignalScope { Local, Community(CommunityId), Session, Agent }` with a stable u8 wire discriminant; `CommunityId(u64)` newtype
|
||
- `SignalProvenance { writer_agent, origin_shard, scope, share_policy_version, membership_epoch, hlc }` — the read-side audit anchor
|
||
- `SharePolicy { version, allowed_intents, mode }` with `ShareMode { LocalOnly, CommunityShare }` and per-intent filters
|
||
- WAL batch format **V3**: `EventRecord` widened from 21B → 32B, adding `scope`, `writer_agent` (interned u16), `share_policy_version`, `membership_epoch`, and a per-event idempotency tag; `FORMAT_VERSION` bumped to 3
|
||
- Default-local enforcement: `signal()`/`signal_with_context` default `SignalScope::Local`; the shipper drops non-share-eligible events before counting
|
||
- `tidalctl scope-stats` offline subcommand reporting scope distribution and share eligibility over a persisted directory
|
||
- All existing M0–M8 tests pass unchanged (V1/V2 batches decode as `Local` scope; single-node defaults to local-only)
|
||
|
||
## Dependencies
|
||
|
||
- **Requires:** M8 complete (WAL format V2 with `BatchHeader`/`EventRecord` at `tidal/src/wal/format/batch.rs`, `ShardId`/`RegionId` at `tidal/src/replication/shard.rs`, `Hlc`/`HlcTimestamp` at `tidal/src/replication/crdt/hlc.rs`, `AgentId` at `tidal/src/session/types.rs`, `WalShipper`/`SegmentReceiver` at `tidal/src/replication/{shipper,receiver}.rs`, signal ledger `apply_wal_event` at `tidal/src/signals/ledger/core.rs:245`)
|
||
- **Files modified:**
|
||
- `tidal/src/wal/format/batch.rs` — `EventRecord` V3 widening (21B → 32B), `FORMAT_VERSION` → 3, `EVENT_SIZE`, `encode_batch_with_shard`, `decode_batch`, `event_content_hash`
|
||
- `tidal/src/wal/format/batch_tests.rs` — V3 roundtrip + V1/V2/V3 backward-compat decode + proptest
|
||
- `tidal/src/db/signals.rs` — default `SignalScope::Local` in `signal`/`signal_with_context`; thread scope into the write path
|
||
- `tidal/src/replication/shipper.rs` — share-eligibility filter before `count_events_in_segment`
|
||
- `tidal/src/replication/receiver.rs` — receiver honors scope on `apply_payload`
|
||
- `tidal/src/signals/ledger/core.rs` — `apply_wal_event` threads scope/provenance metadata
|
||
- `tidal/src/signals/checkpoint/format.rs` — checkpoint entry V3 (carry scope-aware fields; V1/V2 decode clean)
|
||
- `tidalctl/src/main.rs` — `scope-stats` subcommand
|
||
- `tidal/src/lib.rs` — add `pub mod governance;`
|
||
- **Files created:**
|
||
- `tidal/src/governance/mod.rs` — module root, re-exports
|
||
- `tidal/src/governance/scope.rs` — `SignalScope`, `CommunityId`
|
||
- `tidal/src/governance/provenance.rs` — `SignalProvenance`
|
||
- `tidal/src/governance/share_policy.rs` — `SharePolicy`, `ShareMode`, `ShareIntent`
|
||
|
||
## Research References
|
||
|
||
- `docs/research/tidaldb_wal.md` — WAL segment format, batch header layout, BLAKE3 checksum, dedup window
|
||
- `docs/planning/milestone-8/phase-1/OVERVIEW.md` — the m8p1 "atoms right" template this phase mirrors; WAL V2 byte layout (shard/region at bytes 28-31 of the header)
|
||
- `thoughts.md` — Part V.12 (subject-prefix key encoding); Part II.1 (WAL convergence / replay determinism)
|
||
- `ai-lookup/services/signals.md` — signal intents (`not_for_me`, `low_quality`, `hide/mute/block`, `skip_for_now`)
|
||
- M9→M10 architecture brief — section 1 (canonical shared primitives), section 4 (WAL compat + local-profile-intact invariants), section 5 (test strategy)
|
||
|
||
## Acceptance Criteria (Phase Level)
|
||
|
||
- [ ] `SignalScope` is `Copy + Clone + Debug + PartialEq + Eq + Hash + Serialize + Deserialize` with a stable `as_u8()`/`from_u8()` round-trip: `Local`=0, `Community`=1, `Session`=2, `Agent`=3; unknown byte → `WalError::Corruption`
|
||
- [ ] `SignalScope::default()` returns `SignalScope::Local` (the local-profile-intact default)
|
||
- [ ] `CommunityId(u64)` newtype derives `Copy + Clone + Debug + Eq + Hash + Ord + Serialize + Deserialize` with `Display` producing `"c{n}"`
|
||
- [ ] `SignalProvenance` carries `writer_agent`, `origin_shard: ShardId`, `scope: SignalScope`, `share_policy_version: u16`, `membership_epoch: u32`, `hlc: HlcTimestamp`; derives `Clone + Debug + PartialEq + Serialize + Deserialize`
|
||
- [ ] `SharePolicy { version: u16, allowed_intents: BTreeSet<ShareIntent>, mode: ShareMode }`; `ShareMode { LocalOnly (default), CommunityShare }`; `SharePolicy::local_only()` constructor produces a policy that ships nothing
|
||
- [ ] `SharePolicy::is_share_eligible(scope, intent) -> bool` returns `false` for any `SignalScope::Local` event regardless of policy (load-bearing local-never-ships rule), and `true` only when `mode == CommunityShare` AND `intent ∈ allowed_intents`
|
||
- [ ] `EventRecord` V3 wire format is exactly 32 bytes: existing `entity_id`(8) + `signal_type`(1) + `weight`(4) + `timestamp_nanos`(8) = 21B, plus `scope`(1) + `writer_agent`(2) + `share_policy_version`(2) + `membership_epoch`(4) = 30B, plus 2B reserved/idempotency-tag padding to a clean 32B
|
||
- [ ] `FORMAT_VERSION` bumped to 3 (`FORMAT_VERSION_V3 = 3`); `encode_batch_with_shard` writes V3 events; `decode_batch` accepts V1, V2, and V3
|
||
- [ ] V1/V2 batches decode under V3 code with `scope = Local`, `writer_agent = 0`, `share_policy_version = 0`, `membership_epoch = 0` (zero-fill new fields) — verified by explicit per-version roundtrip tests in `batch_tests.rs`
|
||
- [ ] `event_content_hash` incorporates the new V3 fields so dedup never false-positives across scopes, AND a V1/V2 event hashes identically under V3 decode (back-compat dedup window unbroken)
|
||
- [ ] `signal()` and `signal_with_context()` write `SignalScope::Local` by default; a local signal written then queried on a follower's ledger is NOT present (shipper drops it)
|
||
- [ ] Shipper filter drops every `SignalScope::Local` event and every non-share-eligible community event before shipping; share-eligible community events ship intact
|
||
- [ ] Per-intent filter: a `low_quality` community signal ships when `low_quality ∈ allowed_intents`; a `view` signal does NOT ship when `view ∉ allowed_intents` — asserted on the receiver ledger
|
||
- [ ] `tidalctl scope-stats --path <dir>` performs an offline read-only WAL scan and emits JSON with per-scope event counts and a `share_eligible` count; exit code 0 on a seeded directory
|
||
- [ ] All existing M0–M8 lib + integration tests pass without modification; `cargo fmt` clean; `cargo clippy -p tidaldb -D warnings` clean
|
||
|
||
## Task Execution Order
|
||
|
||
```
|
||
Task 01: Governance atoms (scope, provenance) ──┐
|
||
├──> Task 03: EventRecord V3 wire format
|
||
Task 02: SharePolicy + ShareIntent ─────────────┤ │
|
||
│ ├──> Task 04: Write-path default-local
|
||
│ │ │
|
||
│ │ ├──> Task 05: Shipper share filter
|
||
│ │ │
|
||
│ └──────────┤
|
||
│ v
|
||
└──> Task 06: Checkpoint V3 + ledger metadata
|
||
│
|
||
v
|
||
Task 07: tidalctl scope-stats
|
||
```
|
||
|
||
Tasks 01 and 02 are fully parallelizable pure-type work (they unblock everything; do them first). Task 03 (the highest-blast-radius change) depends on Task 01. Task 04 (write path) depends on 01+03. Task 05 (shipper) depends on 02+04. Task 06 (checkpoint/ledger) depends on 01+03. Task 07 (CLI) depends on 03 (it scans the V3 wire format).
|
||
|
||
## Module Location
|
||
|
||
| File | Status | Contains |
|
||
|------|--------|----------|
|
||
| `tidal/src/governance/mod.rs` | NEW | Module root, re-exports of all governance atoms |
|
||
| `tidal/src/governance/scope.rs` | NEW | `SignalScope`, `CommunityId`, u8 wire codec |
|
||
| `tidal/src/governance/provenance.rs` | NEW | `SignalProvenance` (read-side audit anchor) |
|
||
| `tidal/src/governance/share_policy.rs` | NEW | `SharePolicy`, `ShareMode`, `ShareIntent`, eligibility logic |
|
||
| `tidal/src/wal/format/batch.rs` | MODIFIED | `EventRecord` V3 (32B), `FORMAT_VERSION_V3`, encode/decode/dedup |
|
||
| `tidal/src/wal/format/batch_tests.rs` | MODIFIED | V3 roundtrip + V1/V2/V3 back-compat + proptest |
|
||
| `tidal/src/db/signals.rs` | MODIFIED | Default `SignalScope::Local`; thread scope into write path |
|
||
| `tidal/src/replication/shipper.rs` | MODIFIED | Share-eligibility filter before ship |
|
||
| `tidal/src/replication/receiver.rs` | MODIFIED | Scope-aware `apply_payload` |
|
||
| `tidal/src/signals/ledger/core.rs` | MODIFIED | `apply_wal_event` threads scope/provenance |
|
||
| `tidal/src/signals/checkpoint/format.rs` | MODIFIED | Checkpoint entry V3 (scope-aware; V1/V2 decode clean) |
|
||
| `tidalctl/src/main.rs` | MODIFIED | `scope-stats` offline subcommand |
|
||
| `tidal/src/lib.rs` | MODIFIED | `pub mod governance;` |
|
||
|
||
## Technical Design
|
||
|
||
### SignalScope + CommunityId — `tidal/src/governance/scope.rs`
|
||
|
||
```rust
|
||
use std::fmt;
|
||
|
||
use crate::wal::error::WalError;
|
||
|
||
/// Identifies a community personalization overlay.
|
||
///
|
||
/// `CommunityId(0)` is reserved (no community). Communities are opt-in overlays
|
||
/// on top of a user's local profile; see `SignalScope::Community`.
|
||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord,
|
||
serde::Serialize, serde::Deserialize)]
|
||
pub struct CommunityId(pub u64);
|
||
|
||
impl CommunityId {
|
||
/// Sentinel meaning "no community".
|
||
pub const NONE: Self = Self(0);
|
||
|
||
/// Return the underlying `u64`.
|
||
#[must_use]
|
||
pub const fn as_u64(self) -> u64 {
|
||
self.0
|
||
}
|
||
}
|
||
|
||
impl fmt::Display for CommunityId {
|
||
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||
write!(f, "c{}", self.0)
|
||
}
|
||
}
|
||
|
||
/// The scope of a signal: where it lives and whether it may ever ship.
|
||
///
|
||
/// `Local` is the default and the single most load-bearing invariant of M9:
|
||
/// local-scope signals NEVER ship to a community and are NEVER purged by a
|
||
/// community operation. The wire discriminant is stable across versions.
|
||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default,
|
||
serde::Serialize, serde::Deserialize)]
|
||
pub enum SignalScope {
|
||
/// On-device personalization. Never shipped, never community-purged.
|
||
#[default]
|
||
Local,
|
||
/// Contributed to a community overlay (subject to `SharePolicy`).
|
||
Community(CommunityId),
|
||
/// Bound to a single session's lifetime.
|
||
Session,
|
||
/// Written by an agent on the user's behalf.
|
||
Agent,
|
||
}
|
||
|
||
impl SignalScope {
|
||
/// Stable wire discriminant. The `Community` payload is encoded separately
|
||
/// (the discriminant alone occupies one byte in the WAL event record).
|
||
#[must_use]
|
||
pub const fn discriminant(self) -> u8 {
|
||
match self {
|
||
SignalScope::Local => 0,
|
||
SignalScope::Community(_) => 1,
|
||
SignalScope::Session => 2,
|
||
SignalScope::Agent => 3,
|
||
}
|
||
}
|
||
|
||
/// Reconstruct a scope from its wire discriminant. `Community` is rebuilt
|
||
/// with `CommunityId::NONE`; the caller overlays the real id from the
|
||
/// membership-epoch context (the community id is not stored inline in the
|
||
/// per-event record — `membership_epoch` keys it).
|
||
///
|
||
/// # Errors
|
||
///
|
||
/// Returns `WalError::Corruption` for any discriminant outside `0..=3`.
|
||
pub fn from_discriminant(b: u8) -> Result<Self, WalError> {
|
||
match b {
|
||
0 => Ok(SignalScope::Local),
|
||
1 => Ok(SignalScope::Community(CommunityId::NONE)),
|
||
2 => Ok(SignalScope::Session),
|
||
3 => Ok(SignalScope::Agent),
|
||
other => Err(WalError::Corruption {
|
||
message: format!("invalid SignalScope discriminant: {other}"),
|
||
}),
|
||
}
|
||
}
|
||
|
||
/// Whether this scope may ever leave the local node. `Local` never ships.
|
||
#[must_use]
|
||
pub const fn is_shippable(self) -> bool {
|
||
!matches!(self, SignalScope::Local)
|
||
}
|
||
}
|
||
```
|
||
|
||
### SignalProvenance — `tidal/src/governance/provenance.rs`
|
||
|
||
```rust
|
||
use crate::governance::scope::SignalScope;
|
||
use crate::replication::{crdt::hlc::HlcTimestamp, shard::ShardId};
|
||
|
||
/// The audit anchor for a signal: who wrote it, in what scope, under which
|
||
/// share-policy version and membership epoch, ordered by HLC.
|
||
///
|
||
/// `writer_agent` is an interned agent id (0 = no agent / direct user write);
|
||
/// the intern table maps `AgentId` <-> `u16`. M10p3 builds the provenance graph
|
||
/// and explainability surface on top of this struct.
|
||
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||
pub struct SignalProvenance {
|
||
/// Interned writer agent id; 0 means a direct (non-agent) user write.
|
||
pub writer_agent: u16,
|
||
/// Shard that produced the originating WAL event.
|
||
pub origin_shard: ShardId,
|
||
/// Scope under which the signal was written.
|
||
pub scope: SignalScope,
|
||
/// Share-policy version in effect at write time.
|
||
pub share_policy_version: u16,
|
||
/// Membership epoch in effect at write time (0 = local / no membership).
|
||
pub membership_epoch: u32,
|
||
/// Hybrid logical clock stamp for deterministic ordering on replay.
|
||
pub hlc: HlcTimestamp,
|
||
}
|
||
|
||
impl SignalProvenance {
|
||
/// Provenance for a default local write (no agent, no community).
|
||
#[must_use]
|
||
pub fn local(origin_shard: ShardId, hlc: HlcTimestamp) -> Self {
|
||
Self {
|
||
writer_agent: 0,
|
||
origin_shard,
|
||
scope: SignalScope::Local,
|
||
share_policy_version: 0,
|
||
membership_epoch: 0,
|
||
hlc,
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
### SharePolicy + ShareIntent — `tidal/src/governance/share_policy.rs`
|
||
|
||
```rust
|
||
use std::collections::BTreeSet;
|
||
|
||
use crate::governance::scope::SignalScope;
|
||
|
||
/// A signal intent that may be selectively allowed to ship to a community.
|
||
///
|
||
/// Mirrors the engagement-feedback vocabulary surfaced in
|
||
/// `ai-lookup/services/signals.md`. New intents append to the enum; the
|
||
/// `BTreeSet` ordering keeps `allowed_intents` serialization deterministic.
|
||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord,
|
||
serde::Serialize, serde::Deserialize)]
|
||
pub enum ShareIntent {
|
||
SkipForNow,
|
||
NotForMe,
|
||
LowQuality,
|
||
Hide,
|
||
Mute,
|
||
Block,
|
||
Save,
|
||
View,
|
||
Like,
|
||
}
|
||
|
||
/// Whether a share policy keeps everything local or contributes to a community.
|
||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default,
|
||
serde::Serialize, serde::Deserialize)]
|
||
pub enum ShareMode {
|
||
/// Nothing ships; the local profile is the only sink.
|
||
#[default]
|
||
LocalOnly,
|
||
/// Share-eligible intents contribute to a community overlay.
|
||
CommunityShare,
|
||
}
|
||
|
||
/// Versioned per-user share contract: which intents may ship and in what mode.
|
||
///
|
||
/// The canonical definition referenced by m9p2 (membership) and m10p1
|
||
/// (governance policy). Versions are monotonic; the in-effect version is
|
||
/// stamped into every shipped `EventRecord` and `SignalProvenance`.
|
||
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||
pub struct SharePolicy {
|
||
/// Monotonic version; stamped into provenance and the WAL event.
|
||
pub version: u16,
|
||
/// Intents the user has opted to share. Empty in `LocalOnly`.
|
||
pub allowed_intents: BTreeSet<ShareIntent>,
|
||
/// Overall sharing mode.
|
||
pub mode: ShareMode,
|
||
}
|
||
|
||
impl SharePolicy {
|
||
/// A policy that ships nothing — the secure default.
|
||
#[must_use]
|
||
pub fn local_only() -> Self {
|
||
Self {
|
||
version: 0,
|
||
allowed_intents: BTreeSet::new(),
|
||
mode: ShareMode::LocalOnly,
|
||
}
|
||
}
|
||
|
||
/// Whether an event of `scope` carrying `intent` is eligible to ship.
|
||
///
|
||
/// `Local` scope is NEVER eligible regardless of policy (the
|
||
/// local-profile-intact guarantee). Otherwise eligibility requires
|
||
/// `CommunityShare` mode AND the intent in `allowed_intents`.
|
||
#[must_use]
|
||
pub fn is_share_eligible(&self, scope: SignalScope, intent: ShareIntent) -> bool {
|
||
if !scope.is_shippable() {
|
||
return false;
|
||
}
|
||
matches!(self.mode, ShareMode::CommunityShare) && self.allowed_intents.contains(&intent)
|
||
}
|
||
}
|
||
|
||
impl Default for SharePolicy {
|
||
fn default() -> Self {
|
||
Self::local_only()
|
||
}
|
||
}
|
||
```
|
||
|
||
### EventRecord V3 — `tidal/src/wal/format/batch.rs` (MODIFIED)
|
||
|
||
```rust
|
||
/// Wire format version 3: per-event scope and share metadata.
|
||
///
|
||
/// Widens `EventRecord` from 21 to 32 bytes, adding `scope` (u8),
|
||
/// `writer_agent` (interned u16 LE), `share_policy_version` (u16 LE), and
|
||
/// `membership_epoch` (u32 LE), plus 2 reserved bytes. Backward compatible:
|
||
/// V1/V2 events (21 bytes) decode with `scope = Local` and zeroed metadata.
|
||
/// The version byte at header offset 4 disambiguates event width.
|
||
pub const FORMAT_VERSION_V3: u8 = 3;
|
||
|
||
/// Current wire format version. All new batches are encoded as v3.
|
||
pub const FORMAT_VERSION: u8 = FORMAT_VERSION_V3;
|
||
|
||
/// Size of a V1/V2 event record in bytes.
|
||
pub const EVENT_SIZE_V2: usize = 21;
|
||
|
||
/// Size of a V3 event record in bytes (21 base + 11 governance/padding).
|
||
pub const EVENT_SIZE_V3: usize = 32;
|
||
|
||
/// A single signal event record in wire format.
|
||
///
|
||
/// V3 adds the governance fields. When decoded from a V1/V2 batch, the
|
||
/// governance fields default to a local, agent-less, epoch-0 event.
|
||
#[derive(Debug, Clone, PartialEq)]
|
||
pub struct EventRecord {
|
||
pub entity_id: u64,
|
||
pub signal_type: u8,
|
||
pub weight: f32,
|
||
pub timestamp_nanos: u64,
|
||
/// V3: scope discriminant (see `SignalScope::discriminant`).
|
||
pub scope: u8,
|
||
/// V3: interned writer agent id (0 = direct user write).
|
||
pub writer_agent: u16,
|
||
/// V3: share-policy version in effect at write time.
|
||
pub share_policy_version: u16,
|
||
/// V3: membership epoch in effect at write time (0 = local).
|
||
pub membership_epoch: u32,
|
||
}
|
||
|
||
impl EventRecord {
|
||
/// Serialize this event into the 32-byte V3 wire format.
|
||
#[must_use]
|
||
pub fn to_bytes_v3(&self) -> [u8; EVENT_SIZE_V3] {
|
||
let mut buf = [0u8; EVENT_SIZE_V3];
|
||
buf[0..8].copy_from_slice(&self.entity_id.to_le_bytes());
|
||
buf[8] = self.signal_type;
|
||
buf[9..13].copy_from_slice(&self.weight.to_le_bytes());
|
||
buf[13..21].copy_from_slice(&self.timestamp_nanos.to_le_bytes());
|
||
buf[21] = self.scope;
|
||
buf[22..24].copy_from_slice(&self.writer_agent.to_le_bytes());
|
||
buf[24..26].copy_from_slice(&self.share_policy_version.to_le_bytes());
|
||
buf[26..30].copy_from_slice(&self.membership_epoch.to_le_bytes());
|
||
// buf[30..32] reserved (zeroed)
|
||
buf
|
||
}
|
||
|
||
/// Deserialize a V3 (32-byte) or legacy V1/V2 (21-byte) event by width.
|
||
///
|
||
/// # Errors
|
||
///
|
||
/// Returns `WalError::Corruption` if the slice is neither 21 nor 32 bytes,
|
||
/// or the scope discriminant is invalid.
|
||
pub fn from_bytes_sized(bytes: &[u8]) -> Result<Self, WalError> {
|
||
match bytes.len() {
|
||
EVENT_SIZE_V2 => {
|
||
// Legacy: zero-fill governance fields, scope = Local.
|
||
let base = Self::decode_base(bytes)?;
|
||
Ok(Self { scope: 0, writer_agent: 0, share_policy_version: 0,
|
||
membership_epoch: 0, ..base })
|
||
}
|
||
EVENT_SIZE_V3 => {
|
||
let mut e = Self::decode_base(&bytes[0..EVENT_SIZE_V2])?;
|
||
e.scope = bytes[21];
|
||
// Validate the discriminant on the decode path.
|
||
crate::governance::scope::SignalScope::from_discriminant(e.scope)?;
|
||
e.writer_agent = u16::from_le_bytes(bytes[22..24].try_into().unwrap());
|
||
e.share_policy_version = u16::from_le_bytes(bytes[24..26].try_into().unwrap());
|
||
e.membership_epoch = u32::from_le_bytes(bytes[26..30].try_into().unwrap());
|
||
Ok(e)
|
||
}
|
||
n => Err(WalError::Corruption {
|
||
message: format!("event record: expected 21 or 32 bytes, got {n}"),
|
||
}),
|
||
}
|
||
}
|
||
}
|
||
|
||
/// Decode-time event width selected by batch `version`.
|
||
#[must_use]
|
||
pub const fn event_size_for_version(version: u8) -> usize {
|
||
if version >= FORMAT_VERSION_V3 { EVENT_SIZE_V3 } else { EVENT_SIZE_V2 }
|
||
}
|
||
|
||
/// Per-event content hash for dedup, computed over the full V3 byte image so
|
||
/// that scope/agent/epoch differences never collide, while a V1/V2 event
|
||
/// (which canonicalizes to scope=Local, zeroed metadata) hashes identically
|
||
/// whether read as 21 or 32 bytes.
|
||
#[must_use]
|
||
pub fn event_content_hash(event: &EventRecord) -> u128 {
|
||
let bytes = event.to_bytes_v3();
|
||
let hash = blake3::hash(&bytes);
|
||
let hash_bytes: &[u8; 32] = hash.as_bytes();
|
||
u128::from_le_bytes(hash_bytes[..16].try_into().expect("32-byte hash"))
|
||
}
|
||
```
|
||
|
||
`decode_batch` reads the header `version` byte, selects `event_size_for_version`,
|
||
and validates `payload_len == event_count * event_size`. The BLAKE3 batch
|
||
checksum (header[0..32] || event_bytes) is unchanged in mechanism but now
|
||
covers the wider event payload. `encode_batch_with_shard` writes V3 events via
|
||
`to_bytes_v3`.
|
||
|
||
### Write-path default — `tidal/src/db/signals.rs` (MODIFIED)
|
||
|
||
```rust
|
||
// signal() and signal_with_context() default to SignalScope::Local. The base
|
||
// signal stays exactly as today; scope is attached at the ledger/WAL boundary.
|
||
//
|
||
// record_signal gains a scope-aware sibling; the public `signal()` keeps its
|
||
// existing arity and passes SignalScope::Local. A new scoped entry point lands
|
||
// in m9p2 (community dispatch); m9p1 only guarantees the default.
|
||
let scope = SignalScope::Local; // default; community scope arrives in m9p2
|
||
```
|
||
|
||
### Shipper share filter — `tidal/src/replication/shipper.rs` (MODIFIED)
|
||
|
||
```rust
|
||
// Before count_events_in_segment / send_segment, decode the segment, drop every
|
||
// event whose scope is Local (never ships) and every community event that is
|
||
// not share-eligible under the active SharePolicy, then re-encode the filtered
|
||
// batch. Only share-eligible events leave the node.
|
||
fn filter_share_eligible(segment_bytes: &[u8], policy: &SharePolicy) -> Vec<u8> {
|
||
// decode_batch -> retain events where policy.is_share_eligible(scope, intent)
|
||
// -> encode_batch_with_shard(filtered, ...)
|
||
// A segment that filters down to zero events is skipped entirely.
|
||
unimplemented!("see task-05")
|
||
}
|
||
```
|
||
|
||
## Notes
|
||
|
||
### Backward compatibility is non-negotiable (brief section 4.1)
|
||
|
||
`FORMAT_VERSION` → 3. The 64-byte header is already full (shard/region live at
|
||
bytes **28-31** in V2 — verified against `batch.rs:193`), so V3 per-event
|
||
metadata goes in the EVENT payload by widening `EVENT_SIZE` 21B → 32B, NOT in
|
||
the header. The header `version` byte (offset 4) is the sole disambiguator of
|
||
event width. V1/V2 batches must decode under V3 code with `scope = Local` and
|
||
zeroed governance fields; this is safe because legacy events are 21 bytes and
|
||
`from_bytes_sized` branches on length. Explicit V1/V2/V3 roundtrip tests in
|
||
`batch_tests.rs` are required, plus a proptest over random V3 batches.
|
||
|
||
### Dedup window must not break (brief section 4.1)
|
||
|
||
`event_content_hash` now hashes the 32-byte V3 image. The canonicalization rule
|
||
is: a legacy event read as V1/V2 produces the same `EventRecord`
|
||
(scope=Local, zeroed metadata) it would produce written as V3, so its content
|
||
hash is identical across versions — the dedup window does not false-positive nor
|
||
false-negative across a format upgrade. Two community events that differ only in
|
||
scope/agent/epoch correctly hash differently.
|
||
|
||
### Local-profile-intact guarantee (brief section 5, the single load-bearing rule)
|
||
|
||
`SignalScope::Local` events NEVER ship and are NEVER purged by community
|
||
operations. This is enforced in two independent places: `SharePolicy::is_share_eligible`
|
||
short-circuits to `false` for local scope, and the shipper filter drops local
|
||
events before counting. Every M9 UAT asserts the local feed is unchanged across
|
||
join/share/leave/purge. A routing bug that ships or purges local scope is silent
|
||
data loss — treat any test that observes a local event on a follower as a
|
||
BLOCKER.
|
||
|
||
### Community id is keyed by epoch, not stored inline
|
||
|
||
To keep the per-event record at a clean 32 bytes, the `Community(CommunityId)`
|
||
payload is NOT serialized inline. The event stores the `scope` discriminant and
|
||
the `membership_epoch`; the membership registry (m9p2) maps
|
||
`(user, membership_epoch) -> CommunityId`. `from_discriminant` therefore returns
|
||
`Community(CommunityId::NONE)` and the caller overlays the real id. This keeps
|
||
m9p1 self-contained while leaving the join hook for m9p2.
|
||
|
||
### <1s gates are NOT in this phase
|
||
|
||
The synchronous atomic stop-forward / revocation gates are m9p2/m10p2. m9p1
|
||
only establishes the scope/share atoms and the WAL envelope. Do not route any
|
||
filtering through the 60s sweeper.
|
||
|
||
### tidalctl is offline and read-only
|
||
|
||
`scope-stats` opens no DB; it scans WAL segment files directly (the same
|
||
`decode_batch` path), tallies per-scope counts and share eligibility, and emits
|
||
JSON. Mirrors the existing `status`/`diagnostics` offline-scan pattern in
|
||
`tidalctl/src/main.rs`.
|
||
|
||
## Done When
|
||
|
||
A developer can write signals that default to `SignalScope::Local`, observe
|
||
that local signals never appear on a follower's ledger, opt specific intents
|
||
into a `SharePolicy` so only those community events ship, decode a mixed
|
||
V1/V2/V3 WAL directory without corruption, and run `tidalctl scope-stats` to
|
||
see the per-scope event distribution and share-eligible count. All existing
|
||
M0–M8 tests pass unchanged, and the WAL V3 roundtrip + backward-compat proptest
|
||
is green.
|