tidaldb/docs/planning/milestone-8/phase-10/task-01-engine-relay-hlc-snapshot.md

Task 01: Engine relay module, HLC clock offset, StateSnapshot wire serde

Delivers

Three engine-side prerequisites for one-region-per-process cluster mode, all in the tidaldb crate:

1. replication::relay — a public, production (NOT testing-gated) module owning the signal-relay primitives that SimulatedCluster currently hides inside testing::cluster_transport, plus a new SignalRelay type that encapsulates one leader's replication stream (seqno counter, batch log, eager ship, redelivery) so the server's multi-process node and the simulated harness share ONE implementation of the load-bearing invariants.
2. HLC clock offset — Hlc accepts a wall-clock offset (ms, signed) applied inside wall_ms_now, plumbed from TidalDbBuilder::with_hlc_offset_ms(i64) through to the Hlc::for_shard construction in take_crdt_snapshot. This lets a whole process run with a genuinely skewed HLC for the m8p10 clock-skew UAT (the spec's "inject clock offset via HLC's max(wall_clock, last_seen + 1) mechanism").
3. StateSnapshot wire serde — StateSnapshot (and any contained type still missing it) becomes serializable/deserializable so CRDT snapshots can be exchanged between processes over HTTP for cross-process reconciliation.

Complexity: L

Dependencies

None (first task of the phase). Existing invariant tests in tidal/src/testing/cluster_transport.rs define the contract the moved code must keep.

Technical Design

1. tidal/src/replication/relay.rs

Move from testing::cluster_transport (and make pub):

• BatchEntry { source_shard, seqno, bytes } (fields pub)
• single_event_payload(source_shard, seqno, bytes) -> WalSegmentPayload
• redeliver_missed(transport, db, log) — keep doc comments and BOTH unit tests (single_event_payload_holds_replication_invariants, redeliver_ships_to_local_shard_with_source_payload) verbatim; they lock the m8p9 wrong-destination regression.

testing::cluster_transport keeps only the channel transports and re-exports the moved items so SimulatedCluster is otherwise untouched (pub(super) re-export is fine).

New type in the same module:

/// One leader's replication stream: monotonic seqno, batch log, eager ship,
/// idempotent redelivery. Single source of truth for the write/ship atomicity
/// invariants shared by `SimulatedCluster` and `tidal-server`'s multi-process
/// region node.
pub struct SignalRelay {
    seqno: Mutex<u64>,
    batch_log: Mutex<Vec<BatchEntry>>,
}

impl SignalRelay {
    pub fn new() -> Self;

    /// Atomically: bump seqno, encode the one-event batch, apply the write to
    /// the local db (all under the seqno lock, rolling the seqno back on
    /// encode/apply failure — exactly `SimulatedCluster::write_signal`'s
    /// contract), then best-effort eager-ship to every peer in `peers`
    /// (skipping `skip`, e.g. partitioned regions) and record the batch in the
    /// log. Returns the committed seqno.
    pub fn write_and_ship(
        &self,
        db: &TidalDb,
        source_shard: ShardId,
        signal_type_id: u8,
        signal_type: &str,
        entity_id: EntityId,
        weight: f64,
        transport: &dyn Transport,
        peers: &[ShardId],
        skip: &HashSet<ShardId>,
    ) -> crate::Result<u64>;

    /// Re-deliver unapplied batches to ONE peer (heal / convergence path).
    /// Destination is the peer's shard; payload keeps the source shard.
    pub fn redeliver_to(&self, transport: &dyn Transport, dest: ShardId, db_applied: impl Fn(ShardId) -> u64);

    pub fn last_seq(&self) -> u64;
    pub fn log_len(&self) -> usize;
}

Signature details are the implementer's to refine (e.g. redeliver_to may take the follower's applied-seqno lookup as a closure or take &TidalDb when local, as redeliver_missed does today — but in the multi-process case the sender does NOT have the follower's TidalDb, so redelivery must consult a remotely-reported applied seqno; design the API so both call shapes work). The non-negotiable invariants, documented on the type:

1. seqno bump + encode + local apply are atomic under one lock with rollback on failure (no burned seqnos, no leader-ahead-of-log states);
2. payloads are built ONLY via single_event_payload;
3. eager ship is best-effort (WARN on failure, never fails the write);
4. redelivery is idempotent and per-source FIFO (the seqno > applied gate).

SimulatedCluster::write_signal / await_convergence / heal_region are refactored to use SignalRelay (one per leader region — the existing leader_seqnos: HashMap becomes HashMap<RegionId, SignalRelay> or equivalent). Behavior must be bit-for-bit identical; the existing tier-2 suites are the regression net.

Note: SimulatedCluster resolves signal_type_ids (name → u8) up front; SignalRelay takes the resolved id so the relay stays schema-agnostic. Keep the u8-overflow guard where the map is built.

2. HLC offset

// hlc.rs
pub struct Hlc {
    node_id: u16,
    /// Signed wall-clock offset (ms) applied in wall_ms_now(); 0 in production.
    wall_offset_ms: i64,
    packed: AtomicU64,
}

impl Hlc {
    pub const fn new(node_id: u16) -> Self;                       // offset 0
    pub const fn with_offset(node_id: u16, offset_ms: i64) -> Self;
    pub const fn for_shard(shard: ShardId) -> Self;               // offset 0
    pub const fn for_shard_with_offset(shard: ShardId, offset_ms: i64) -> Self;

    fn wall_ms_now(&self) -> u64 {
        // SystemTime::now() ± offset, saturating both directions.
    }
}

wall_ms_now becomes an instance method (&self); update both call sites (now, update). Builder plumbing: TidalDbBuilder::with_hlc_offset_ms(i64) stores the offset on the open TidalDb (alongside existing replication config); take_crdt_snapshot uses Hlc::for_shard_with_offset(local_shard, self.hlc_offset_ms). Default 0 everywhere — no behavior change unless explicitly configured. Timestamp::now() (signal decay) is deliberately NOT touched: the spec scopes skew injection to the HLC.

3. StateSnapshot serde

StateSnapshot's maps are tuple-keyed ((EntityId, SignalTypeId) / (EntityId, EntityId)), which JSON cannot key. Add a wire representation:

#[derive(serde::Serialize, serde::Deserialize)]
struct StateSnapshotWire {
    signal_states: Vec<(u64, u16, CrdtSignalState)>,
    hardneg_registers: Vec<(u64, u64, LWWRegister<HardNegAction>)>,
}

with #[serde(from = "StateSnapshotWire", into = "StateSnapshotWire")] on StateSnapshot (it is already Clone). Add Serialize/Deserialize derives to CrdtSignalState, LWWRegister<T>, HlcTimestamp, and any other contained type that lacks them. Use the newtypes' inner values (EntityId::as_u64, SignalTypeId::as_u16) on the wire; reconstruct via their constructors.

Test Strategy

• Moved relay tests pass unchanged in their new home.
• New SignalRelay unit tests: seqno rollback on encode failure and on apply failure (inject a failing db write via unknown signal type), eager-ship skip set honored, redelivery idempotence (double redeliver → applied once — use a recording transport).
• Property test: any interleaving of write_and_ship + redeliver_to preserves per-source FIFO and never double-applies (model the receiver as monotonic max).
• HLC: with_offset(+500) produces timestamps ≥ 500ms ahead of new(); negative offset behind; update() from a skewed-ahead remote still yields strictly-greater local timestamps (causal consistency under skew); offset 0 byte-identical to today.
• Snapshot serde: JSON roundtrip preserves equality of merge results — ReconciliationEngine::plan(local, remote) == plan(local, roundtrip(remote)) (deterministic plan equality), plus a proptest over random snapshots.
• Full existing suites stay green: cargo test -p tidaldb, cargo test -p tidal-net, cargo test -p tidal-server.

Acceptance Criteria

• replication::relay is public, not feature-gated; BatchEntry, single_event_payload, redeliver_missed live there with their invariant tests
• SignalRelay encapsulates seqno/log/ship/redeliver; SimulatedCluster consumes it (no duplicated write/ship/redeliver logic remains in testing::)
• Hlc offset constructors exist; TidalDbBuilder::with_hlc_offset_ms plumbs to take_crdt_snapshot's HLC; default-0 behavior unchanged
• StateSnapshot serializes/deserializes (JSON) with reconcile-determinism proven across a roundtrip
• All workspace tests pass; cargo clippy -p tidaldb -- -D warnings clean; fmt clean