tidaldb/docs/planning/milestone-9/phase-2/OVERVIEW.md

m9p2: Membership Lifecycle and Stop-Forward

Delivers

The join / leave / rejoin lifecycle for community personalization overlays, built on the scope and provenance primitives delivered in m9p1. After this phase, a user can join a community with a SharePolicy, leave it with a stop_forward guarantee that blocks new community contributions in under a second, and rejoin into a fresh membership epoch so that replay never crosses epoch boundaries ambiguously. The stop_forward gate is an O(1) atomic read on the synchronous signal write path -- it never depends on the 60s session sweeper. Membership is durable (its own Tag::Membership keyspace), survives restart, and the active epoch is surfaced in query result metadata for debugging and explainability.

This is the lifecycle phase: m9p1 made scope-aware events shippable; m9p2 decides whose community-scoped events are eligible to ship right now and under which epoch they are stamped. No retroactive purge yet (that is m9p3) -- this phase only governs forward contributions.

Deliverables:

• MembershipState enum (Joined, LeavingStopForward, Left, Rejoined) and Membership struct with an atomic stop_forward_at_ns gate
• MembershipRegistry: in-memory DashMap<(UserId, CommunityId), Membership> source of truth, backed by a durable Tag::Membership checkpoint
• Public lifecycle API on TidalDb in a new db/communities.rs: join_community, leave_community, rejoin_community, membership_epoch
• A synchronous stop-forward gate in signal_with_context that drops community-scoped contributions for a member in LeavingStopForward/Left state without touching the local profile
• membership_epoch per community advanced on every rejoin, stamped into the m9p1 SignalProvenance.membership_epoch field and the WAL event
• membership_epoch surfaced in Results via policy_metadata, plus the active epoch added to QueryStats
• All existing M0-M8 tests pass unchanged; single-node default has no memberships and a zero-cost gate

Dependencies

• Requires: m9p1 complete -- SignalScope, CommunityId, SignalProvenance, SharePolicy/ShareMode, the WAL V3 EventRecord carrying scope + membership_epoch, and the governance module root wired into lib.rs. Also requires M8 (ReplicationState, shard identity) and M4 (AgentId, session policy) which m9p1 already extends.
• Files modified:
  • tidal/src/db/signals.rs -- stop-forward gate + epoch stamping inside signal_with_context community dispatch (signals.rs:219); reject path for community contributions when membership is in a stop-forward state
  • tidal/src/db/mod.rs -- add membership_registry field to TidalDb, declare mod communities;, restore the registry in open()
  • tidal/src/replication/state.rs -- add per-community membership_epoch tracking alongside the existing per-shard high-water-mark
  • tidal/src/query/retrieve/types.rs -- add policy_metadata: PolicyMetadata to Results (types.rs:434) carrying the active membership_epoch
  • tidal/src/query/stats.rs -- add membership_epoch: Option<u32> to QueryStats
  • tidal/src/governance/mod.rs -- re-export membership types
  • tidal/src/storage/keys.rs -- add Tag::Membership = 0x0E
• Files created:
  • tidal/src/governance/membership.rs -- MembershipState, Membership, MembershipRegistry, LeaveMode, UserId newtype
  • tidal/src/db/communities.rs -- join_community, leave_community, rejoin_community, membership_epoch on TidalDb

Research References

• /tmp/m9m10_brief.md -- section 1.3 (Membership Epoch primitive), section 2 (M9 Phase 2 acceptance->work table), section 4 invariants 4 (<1s gates are synchronous atomic reads, never the 60s sweeper) and 5 (local-profile-intact)
• docs/research/tidaldb_wal.md -- WAL event envelope, epoch stamping on the event payload
• thoughts.md -- Part II.1 (WAL convergence / replay determinism), Part V.12 (subject-prefix key encoding for the membership keyspace)
• docs/planning/milestone-8/phase-1/ -- ReplicationState extension pattern, identity-newtype conventions

Acceptance Criteria (Phase Level)

• MembershipState enum has exactly Joined, LeavingStopForward, Left, Rejoined; derives Copy + Clone + Debug + PartialEq + Eq + Serialize + Deserialize; round-trips through serde; has a u8 wire discriminant
• Membership holds user_id, community_id, state, epoch: u32, share_policy_version: u16, joined_at_ns: u64, and an atomic stop_forward_at_ns: AtomicU64 (0 = not stop-forwarded)
• TidalDb::join_community(user, community, SharePolicy) creates a Joined membership at epoch = 0 (or Rejoined at the next epoch if a prior Left membership exists), persists it to Tag::Membership, and is idempotent if already joined at the current epoch
• TidalDb::leave_community(user, community, LeaveMode::StopForward) transitions to LeavingStopForward, sets stop_forward_at_ns to the call wall-clock in a single atomic store, then settles to Left; the membership row persists
• After leave_community(StopForward), a community-scoped signal_with_context for that (user, community) records zero new community contributions; the gate is verified to be an O(1) atomic read (no sweeper, no scan) -- measured p99 latency for the gated path < 1s, in practice sub-microsecond
• The same post-leave signal still records the user's local profile update unchanged (local-profile-intact: SignalScope::Local side effects fire regardless of membership state)
• TidalDb::rejoin_community(...) increments the per-community epoch by exactly 1, sets state Rejoined, clears stop_forward_at_ns to 0, and the new epoch is stamped into SignalProvenance.membership_epoch and the WAL event for subsequent community contributions
• A pre-rejoin (old-epoch) WAL event replayed after a leave->rejoin does not revive community contributions under the new epoch: apply_wal_event / CRDT merge segregates events by membership_epoch
• TidalDb::membership_epoch(user, community) -> Option<u32> returns the active epoch; Results.policy_metadata.membership_epoch and QueryStats.membership_epoch expose it on community-scoped queries
• Membership state survives close/reopen: registry rebuilt from Tag::Membership checkpoint on open(); epochs and stop-forward state preserved
• All existing M0-M8 tests pass unchanged; a DB with no memberships pays a single DashMap::is_empty() check on the community dispatch path
• cargo fmt clean, cargo clippy -p tidaldb -D warnings clean, all cargo test -p tidaldb --lib pass

Task Execution Order

Task 01: Membership Types ───────────┬──> Task 03: Lifecycle API (db/communities.rs)
  (membership.rs, UserId, LeaveMode)  │           │
                                      │           v
Task 02: Membership Registry ────────┘     Task 04: Stop-Forward Gate
  (DashMap + Tag::Membership persist)             (signal_with_context)
                                                  │
                                                  v
                                      Task 05: Epoch Stamping + Cross-Epoch Replay
                                        (provenance.membership_epoch, apply_wal_event)
                                                  │
                                                  v
                                      Task 06: Epoch in Query Results
                                        (policy_metadata, QueryStats, ReplicationState)

Task 01 and Task 02 are parallelizable (types vs. persistence wiring). Task 03 depends on both. Task 04 depends on 03 (needs the registry lookup in the write path). Task 05 depends on 04 (epoch must be readable to stamp). Task 06 depends on 05 (epoch must be authoritative before it is surfaced).

Module Location

File	Status	Contains
tidal/src/governance/membership.rs	NEW	MembershipState, Membership, MembershipRegistry, LeaveMode, UserId
tidal/src/db/communities.rs	NEW	join_community, leave_community, rejoin_community, membership_epoch on TidalDb
tidal/src/governance/mod.rs	MODIFIED	Re-export membership types (pub mod membership;)
tidal/src/db/mod.rs	MODIFIED	membership_registry field; mod communities;; restore in open()
tidal/src/db/signals.rs	MODIFIED	Stop-forward gate + epoch stamping in signal_with_context community dispatch
tidal/src/replication/state.rs	MODIFIED	Per-community membership_epoch tracking
tidal/src/query/retrieve/types.rs	MODIFIED	PolicyMetadata + Results.policy_metadata
tidal/src/query/stats.rs	MODIFIED	QueryStats.membership_epoch
tidal/src/storage/keys.rs	MODIFIED	Tag::Membership = 0x0E

Technical Design

Membership state and the atomic stop-forward gate

Membership is the per-(user, community) lifecycle row. The single load-bearing field is stop_forward_at_ns: an AtomicU64 that the write path reads with Ordering::Acquire. Zero means "contributions flow"; non-zero means "stop-forwarded at this wall-clock ns". Because the gate is one atomic load, the <1s p99 requirement is met by construction -- there is no sweeper, no scan, no lock on the read side.

// tidal/src/governance/membership.rs

use std::sync::atomic::{AtomicU64, Ordering};

use serde::{Deserialize, Serialize};

use crate::governance::scope::CommunityId; // from m9p1

/// Identifies a user entity participating in a community overlay.
///
/// A thin newtype over the entity id so membership keys never collide with
/// raw `EntityId` usage elsewhere. `UserId(0)` is reserved (no user).
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord, Serialize, Deserialize)]
pub struct UserId(pub u64);

/// The lifecycle state of a `(user, community)` membership.
///
/// Transitions: `Joined -> LeavingStopForward -> Left -> Rejoined -> ...`.
/// Each `Rejoined` opens a new epoch; replay never crosses epochs.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[repr(u8)]
pub enum MembershipState {
    /// Active member; community-scoped contributions flow.
    Joined = 0,
    /// Leave initiated; `stop_forward_at_ns` is set and new community
    /// contributions are gated off, but no purge has occurred.
    LeavingStopForward = 1,
    /// Settled "left" state; contributions stay gated until a rejoin.
    Left = 2,
    /// Re-entered the community under a new epoch.
    Rejoined = 3,
}

/// How a leave is performed.
///
/// `StopForward` is the m9p2 guarantee: block new contributions immediately,
/// leave existing aggregates untouched. `Purge` (a superset, m9p3) additionally
/// removes prior contributions and is declared here so the API surface is
/// stable, but is rejected with `TidalError::Unsupported` until m9p3 lands.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum LeaveMode {
    /// Block new contributions; keep prior aggregates (m9p2).
    StopForward,
    /// Stop-forward **and** retroactively purge prior contributions (m9p3).
    Purge,
}

/// A durable per-`(user, community)` membership row.
///
/// The source of truth lives in `MembershipRegistry`; this struct is what
/// gets checkpointed under `Tag::Membership`. `stop_forward_at_ns` is the
/// only mutable hot-path field and is an atomic so the signal write path can
/// gate without taking a lock.
#[derive(Debug, Serialize, Deserialize)]
pub struct Membership {
    /// The participating user.
    pub user_id: UserId,
    /// The community overlay being joined.
    pub community_id: CommunityId,
    /// Current lifecycle state.
    pub state: MembershipState,
    /// Monotonic epoch; increments by 1 on each rejoin. First join = 0.
    pub epoch: u32,
    /// The `SharePolicy` version in force for this membership (m9p1).
    pub share_policy_version: u16,
    /// Wall-clock ns at first join (never changes across epochs).
    pub joined_at_ns: u64,
    /// Wall-clock ns at which contributions were stop-forwarded.
    /// `0` means "not stop-forwarded -- contributions flow". Read on the
    /// synchronous signal write path with `Ordering::Acquire`.
    #[serde(with = "atomic_u64_serde")]
    pub stop_forward_at_ns: AtomicU64,
}

impl Membership {
    /// Whether community-scoped contributions are currently gated off.
    ///
    /// `O(1)` atomic load -- this is the <1s stop-forward gate. Never routes
    /// through the sweeper.
    #[must_use]
    pub fn is_stop_forwarded(&self) -> bool {
        self.stop_forward_at_ns.load(Ordering::Acquire) != 0
    }

    /// Engage the stop-forward gate at `now_ns`. Idempotent: the first
    /// non-zero store wins; later calls do not move the timestamp backward.
    pub fn engage_stop_forward(&self, now_ns: u64) {
        let _ = self.stop_forward_at_ns.compare_exchange(
            0,
            now_ns,
            Ordering::AcqRel,
            Ordering::Acquire,
        );
    }

    /// Clear the gate (used on rejoin so the new epoch flows).
    pub fn clear_stop_forward(&self) {
        self.stop_forward_at_ns.store(0, Ordering::Release);
    }
}

Membership registry: in-memory source of truth + durable checkpoint

The registry mirrors the CohortRegistry/ReplicationState conventions: a DashMap for lock-free reads on the write path, a JSON checkpoint under Tag::Membership at a sentinel entity id for durability, and an is_empty() fast path so a DB with no communities pays nothing.

// tidal/src/governance/membership.rs (continued)

use dashmap::DashMap;

/// In-memory registry of all `(user, community)` memberships.
///
/// Source of truth for the lifecycle gate. Reads on the signal write path go
/// through `DashMap::get` (sharded, lock-free for distinct keys). Persisted to
/// `Tag::Membership` on checkpoint and rebuilt on `open()`.
#[derive(Debug, Default)]
pub struct MembershipRegistry {
    rows: DashMap<(UserId, CommunityId), Membership>,
}

impl MembershipRegistry {
    #[must_use]
    pub fn new() -> Self {
        Self { rows: DashMap::new() }
    }

    /// Fast path used on the signal write path before any community work.
    #[must_use]
    pub fn is_empty(&self) -> bool {
        self.rows.is_empty()
    }

    /// The active epoch for `(user, community)`, if a membership exists.
    #[must_use]
    pub fn epoch(&self, user: UserId, community: CommunityId) -> Option<u32> {
        self.rows.get(&(user, community)).map(|m| m.epoch)
    }

    /// Whether a community-scoped contribution for `(user, community)` is
    /// currently gated off (stop-forwarded, left, or no membership).
    ///
    /// Returns `true` (gated) when no membership exists -- a non-member cannot
    /// contribute to a community. `O(1)`.
    #[must_use]
    pub fn is_contribution_gated(&self, user: UserId, community: CommunityId) -> bool {
        match self.rows.get(&(user, community)) {
            Some(m) => m.is_stop_forwarded()
                || matches!(m.state, MembershipState::Left | MembershipState::LeavingStopForward),
            None => true,
        }
    }

    /// Serialize all rows for checkpoint persistence (`Tag::Membership`).
    #[must_use]
    pub fn to_checkpoint_bytes(&self) -> Vec<u8> { /* serde_json over a Vec<MembershipSnapshot> */ }

    /// Rebuild from checkpoint bytes; malformed bytes yield an empty registry.
    #[must_use]
    pub fn from_checkpoint_bytes(bytes: &[u8]) -> Self { /* ... */ }
}

Lifecycle API on TidalDb

The lifecycle methods live in the new db/communities.rs, mirroring how db/cohorts.rs and db/sessions.rs extend impl TidalDb. Each method mutates the registry, persists the row under Tag::Membership, and advances the per-community epoch counter in ReplicationState.

// tidal/src/db/communities.rs

use crate::{
    governance::membership::{LeaveMode, Membership, MembershipState, UserId},
    governance::scope::CommunityId,
    schema::{TidalError, Timestamp},
};

impl super::TidalDb {
    /// Join a community overlay under an explicit `SharePolicy`.
    ///
    /// Idempotent if already `Joined`/`Rejoined` at the current epoch. If a
    /// prior membership is `Left`, this is equivalent to `rejoin_community`
    /// (next epoch). The new row is persisted to `Tag::Membership`.
    ///
    /// # Errors
    /// - `TidalError::Internal` if no storage backend is wired.
    /// - `TidalError::Storage` on persistence failure.
    pub fn join_community(
        &self,
        user: UserId,
        community: CommunityId,
        policy: crate::governance::share_policy::SharePolicy, // from m9p1
    ) -> crate::Result<u32> /* returns active epoch */ { /* ... */ }

    /// Leave a community. `LeaveMode::StopForward` engages the atomic gate
    /// (<1s, synchronous) and settles the row to `Left`. `LeaveMode::Purge`
    /// returns `TidalError::Unsupported` until m9p3.
    ///
    /// # Errors
    /// - `TidalError::NotFound` if no membership exists.
    /// - `TidalError::Unsupported` for `LeaveMode::Purge` (m9p3).
    pub fn leave_community(
        &self,
        user: UserId,
        community: CommunityId,
        mode: LeaveMode,
    ) -> crate::Result<()> { /* ... */ }

    /// Rejoin a community after leaving: opens a new epoch (`epoch + 1`),
    /// clears the stop-forward gate, and sets state `Rejoined`.
    ///
    /// # Errors
    /// - `TidalError::NotFound` if no prior membership exists.
    pub fn rejoin_community(
        &self,
        user: UserId,
        community: CommunityId,
        policy: crate::governance::share_policy::SharePolicy,
    ) -> crate::Result<u32> /* returns the new epoch */ { /* ... */ }

    /// The active membership epoch for `(user, community)`, or `None` if the
    /// user is not a member. Surfaced into query result metadata.
    #[must_use]
    pub fn membership_epoch(&self, user: UserId, community: CommunityId) -> Option<u32> {
        self.membership_registry.epoch(user, community)
    }
}

Stop-forward gate on the synchronous write path

The gate slots into signal_with_context (db/signals.rs:219) inside the community-scoped dispatch branch added in m9p1. It runs before the contribution is attributed to the community, and it leaves the local-profile side effects (steps 1-7 in the existing dispatch) entirely alone -- only the community attribution is gated. This is the local-profile-intact invariant in code.

// tidal/src/db/signals.rs — inside the community-scope branch of signal_with_context

// `scope` and `community` come from the m9p1 SignalScope dispatch.
if let crate::governance::scope::SignalScope::Community(community) = scope {
    let uid = crate::governance::membership::UserId(user_id);

    // <1s STOP-FORWARD GATE: single atomic Acquire load. Never the sweeper.
    if self.membership_registry.is_contribution_gated(uid, community) {
        // Local profile already updated above; community contribution dropped.
        return Ok(());
    }

    // Stamp the active epoch into provenance + the WAL community event.
    let epoch = self
        .membership_registry
        .epoch(uid, community)
        .unwrap_or(0);
    let provenance = crate::governance::provenance::SignalProvenance {
        membership_epoch: epoch,
        scope,
        ..provenance // writer_agent_id, origin_shard, share_policy_version, hlc from m9p1
    };
    self.attribute_community_contribution(community, entity_id, weight, timestamp, provenance);
}

Per-community epoch tracking in ReplicationState

ReplicationState already tracks per-shard high-water-marks with atomics; the membership epoch is added as a parallel monotonic-per-community counter so the epoch authority is the same place CRDT/replay machinery already consults.

// tidal/src/replication/state.rs — additive

use crate::governance::scope::CommunityId;

impl ReplicationState {
    /// The current membership epoch for a community (0 if never joined).
    /// Monotonic; advanced only by `advance_membership_epoch`.
    #[must_use]
    pub fn membership_epoch(&self, community: CommunityId) -> u32 { /* atomic load */ }

    /// Advance a community's membership epoch on rejoin. CAS loop enforces
    /// monotonicity exactly like `advance` does for seqnos.
    pub fn advance_membership_epoch(&self, community: CommunityId, epoch: u32) { /* ... */ }
}

Surfacing the active epoch in query results

Results gains a policy_metadata field (a small struct so m9p3/m10 can extend it with purge_watermark / policy_version without further widening Results). QueryStats carries the epoch for the operational-visibility path.

// tidal/src/query/retrieve/types.rs — additive

/// Governance metadata attached to a query response.
///
/// Carries the active membership epoch for community-scoped queries.
/// Extended by m9p3 (`purge_watermark`) and m10 (`policy_version`).
#[derive(Debug, Clone, Default)]
pub struct PolicyMetadata {
    /// Active membership epoch when the query targets a community overlay.
    /// `None` for purely local queries.
    pub membership_epoch: Option<u32>,
}

// added to `pub struct Results`:
//     /// Governance metadata (membership epoch, later purge/policy versions).
//     pub policy_metadata: PolicyMetadata,

// tidal/src/query/stats.rs — additive field on QueryStats
//     /// Active membership epoch for community-scoped queries (None otherwise).
//     pub membership_epoch: Option<u32>,

New storage tag

// tidal/src/storage/keys.rs — next free discriminant after SchemaFingerprint (0x0D)
//     /// Community membership lifecycle rows (M9p2).
//     Membership = 0x0E,
// plus the matching `0x0E => Some(Self::Membership)` arm in `Tag::from_byte`.

Notes

The stop-forward gate is an atomic read, never the sweeper

Invariant 4 from the brief: both the <1s revocation gate (m10p2) and this <1s stop-forward gate are O(1) atomic loads on the synchronous write path. The 60s db/sweeper.rs is explicitly not in this path. Routing stop-forward through the sweeper would blow the p99 SLA by ~60x and is a hard "no". The gate returns early from signal_with_context after the local-profile update, before community attribution -- so the cost when a member is active is a single DashMap::get + atomic load, and the cost when there are no communities at all is one is_empty() check.

Local profile is sacred (local-profile-intact)

Invariant 5: a stop-forwarded or Left membership gates only the community contribution. Every SignalScope::Local side effect -- seen tracking, hard negatives, preference-vector blend, interaction weights -- still fires. A routing bug that lets the gate suppress local side effects is silent data loss for the user's own profile. Tests must assert the local feed/score is unchanged across join/leave/rejoin.

Rejoin = new epoch; replay never crosses epochs ambiguously

Invariant 3 from the brief: epoch += 1 on every rejoin, stamped into both SignalProvenance.membership_epoch and the WAL community event. On replay, apply_wal_event and CRDT merge must segregate by epoch so a late, pre-rejoin (old-epoch) event cannot revive contributions under the new epoch. The epoch is a u32 carried in the V3 EventRecord from m9p1 -- this phase only populates and reads it; it does not change the wire format.

LeaveMode::Purge is declared but deferred

leave_community(.., LeaveMode::Purge) is part of the stable API surface but returns TidalError::Unsupported until m9p3 wires the rematerialization engine. Declaring it now avoids a breaking signature change in m9p3.

Backward compatibility

No WAL or checkpoint format change in this phase -- m9p1 already bumped FORMAT_VERSION to 3 and added membership_epoch to the event. The only new durable artifact is the Tag::Membership keyspace, which is additive: a pre-m9p2 database simply has zero membership rows and the gate is a no-op. A DB opened by m9p2 code that never joins a community is byte-identical in behavior to M8.

Idempotency

join_community is idempotent at the current epoch; engage_stop_forward uses a compare_exchange(0, now) so concurrent leaves settle on the first timestamp. Membership mutations are persisted under a deterministic key so checkpoint replay is order-independent for distinct (user, community) pairs.

Done When

A user can join_community with a SharePolicy, contribute community-scoped signals that blend into the community feed, then leave_community(StopForward) and observe that (a) zero new community contributions are recorded while their local profile keeps updating, (b) the gate is a single atomic read with sub-microsecond p99 (never the sweeper), and (c) after rejoin_community the active epoch has incremented by one, old-epoch replayed events do not revive contributions, and the active epoch is visible in Results.policy_metadata and QueryStats. Membership survives close/reopen, and all existing M0-M8 tests pass unchanged.