# Task 01: Engine relay module, HLC clock offset, StateSnapshot wire serde ## Delivers Three engine-side prerequisites for one-region-per-process cluster mode, all in the `tidaldb` crate: 1. **`replication::relay`** — a public, production (NOT `testing`-gated) module owning the signal-relay primitives that `SimulatedCluster` currently hides inside `testing::cluster_transport`, plus a new `SignalRelay` type that encapsulates one leader's replication stream (seqno counter, batch log, eager ship, redelivery) so the server's multi-process node and the simulated harness share ONE implementation of the load-bearing invariants. 2. **HLC clock offset** — `Hlc` accepts a wall-clock offset (ms, signed) applied inside `wall_ms_now`, plumbed from `TidalDbBuilder::with_hlc_offset_ms(i64)` through to the `Hlc::for_shard` construction in `take_crdt_snapshot`. This lets a whole process run with a genuinely skewed HLC for the m8p10 clock-skew UAT (the spec's "inject clock offset via HLC's max(wall_clock, last_seen + 1) mechanism"). 3. **`StateSnapshot` wire serde** — `StateSnapshot` (and any contained type still missing it) becomes serializable/deserializable so CRDT snapshots can be exchanged between processes over HTTP for cross-process reconciliation. ## Complexity: L ## Dependencies None (first task of the phase). Existing invariant tests in `tidal/src/testing/cluster_transport.rs` define the contract the moved code must keep. ## Technical Design ### 1. `tidal/src/replication/relay.rs` Move from `testing::cluster_transport` (and make `pub`): - `BatchEntry { source_shard, seqno, bytes }` (fields pub) - `single_event_payload(source_shard, seqno, bytes) -> WalSegmentPayload` - `redeliver_missed(transport, db, log)` — keep doc comments and BOTH unit tests (`single_event_payload_holds_replication_invariants`, `redeliver_ships_to_local_shard_with_source_payload`) verbatim; they lock the m8p9 wrong-destination regression. `testing::cluster_transport` keeps only the channel transports and re-exports the moved items so `SimulatedCluster` is otherwise untouched (`pub(super)` re-export is fine). New type in the same module: ```rust /// One leader's replication stream: monotonic seqno, batch log, eager ship, /// idempotent redelivery. Single source of truth for the write/ship atomicity /// invariants shared by `SimulatedCluster` and `tidal-server`'s multi-process /// region node. pub struct SignalRelay { seqno: Mutex, batch_log: Mutex>, } impl SignalRelay { pub fn new() -> Self; /// Atomically: bump seqno, encode the one-event batch, apply the write to /// the local db (all under the seqno lock, rolling the seqno back on /// encode/apply failure — exactly `SimulatedCluster::write_signal`'s /// contract), then best-effort eager-ship to every peer in `peers` /// (skipping `skip`, e.g. partitioned regions) and record the batch in the /// log. Returns the committed seqno. pub fn write_and_ship( &self, db: &TidalDb, source_shard: ShardId, signal_type_id: u8, signal_type: &str, entity_id: EntityId, weight: f64, transport: &dyn Transport, peers: &[ShardId], skip: &HashSet, ) -> crate::Result; /// Re-deliver unapplied batches to ONE peer (heal / convergence path). /// Destination is the peer's shard; payload keeps the source shard. pub fn redeliver_to(&self, transport: &dyn Transport, dest: ShardId, db_applied: impl Fn(ShardId) -> u64); pub fn last_seq(&self) -> u64; pub fn log_len(&self) -> usize; } ``` Signature details are the implementer's to refine (e.g. `redeliver_to` may take the follower's applied-seqno lookup as a closure or take `&TidalDb` when local, as `redeliver_missed` does today — but in the multi-process case the sender does NOT have the follower's `TidalDb`, so redelivery must consult a remotely-reported applied seqno; design the API so both call shapes work). The non-negotiable invariants, documented on the type: 1. seqno bump + encode + local apply are atomic under one lock with rollback on failure (no burned seqnos, no leader-ahead-of-log states); 2. payloads are built ONLY via `single_event_payload`; 3. eager ship is best-effort (WARN on failure, never fails the write); 4. redelivery is idempotent and per-source FIFO (the `seqno > applied` gate). `SimulatedCluster::write_signal` / `await_convergence` / `heal_region` are refactored to use `SignalRelay` (one per leader region — the existing `leader_seqnos: HashMap` becomes `HashMap` or equivalent). Behavior must be bit-for-bit identical; the existing tier-2 suites are the regression net. Note: `SimulatedCluster` resolves `signal_type_ids` (name → u8) up front; `SignalRelay` takes the resolved id so the relay stays schema-agnostic. Keep the u8-overflow guard where the map is built. ### 2. HLC offset ```rust // hlc.rs pub struct Hlc { node_id: u16, /// Signed wall-clock offset (ms) applied in wall_ms_now(); 0 in production. wall_offset_ms: i64, packed: AtomicU64, } impl Hlc { pub const fn new(node_id: u16) -> Self; // offset 0 pub const fn with_offset(node_id: u16, offset_ms: i64) -> Self; pub const fn for_shard(shard: ShardId) -> Self; // offset 0 pub const fn for_shard_with_offset(shard: ShardId, offset_ms: i64) -> Self; fn wall_ms_now(&self) -> u64 { // SystemTime::now() ± offset, saturating both directions. } } ``` `wall_ms_now` becomes an instance method (`&self`); update both call sites (`now`, `update`). Builder plumbing: `TidalDbBuilder::with_hlc_offset_ms(i64)` stores the offset on the open `TidalDb` (alongside existing replication config); `take_crdt_snapshot` uses `Hlc::for_shard_with_offset(local_shard, self.hlc_offset_ms)`. Default 0 everywhere — no behavior change unless explicitly configured. `Timestamp::now()` (signal decay) is deliberately NOT touched: the spec scopes skew injection to the HLC. ### 3. StateSnapshot serde `StateSnapshot`'s maps are tuple-keyed (`(EntityId, SignalTypeId)` / `(EntityId, EntityId)`), which JSON cannot key. Add a wire representation: ```rust #[derive(serde::Serialize, serde::Deserialize)] struct StateSnapshotWire { signal_states: Vec<(u64, u16, CrdtSignalState)>, hardneg_registers: Vec<(u64, u64, LWWRegister)>, } ``` with `#[serde(from = "StateSnapshotWire", into = "StateSnapshotWire")]` on `StateSnapshot` (it is already `Clone`). Add `Serialize`/`Deserialize` derives to `CrdtSignalState`, `LWWRegister`, `HlcTimestamp`, and any other contained type that lacks them. Use the newtypes' inner values (`EntityId::as_u64`, `SignalTypeId::as_u16`) on the wire; reconstruct via their constructors. ## Test Strategy - Moved relay tests pass unchanged in their new home. - New `SignalRelay` unit tests: seqno rollback on encode failure and on apply failure (inject a failing db write via unknown signal type), eager-ship skip set honored, redelivery idempotence (double redeliver → applied once — use a recording transport). - Property test: any interleaving of `write_and_ship` + `redeliver_to` preserves per-source FIFO and never double-applies (model the receiver as monotonic max). - HLC: `with_offset(+500)` produces timestamps ≥ 500ms ahead of `new()`; negative offset behind; `update()` from a skewed-ahead remote still yields strictly-greater local timestamps (causal consistency under skew); offset 0 byte-identical to today. - Snapshot serde: JSON roundtrip preserves equality of merge results — `ReconciliationEngine::plan(local, remote)` == `plan(local, roundtrip(remote))` (deterministic plan equality), plus a proptest over random snapshots. - Full existing suites stay green: `cargo test -p tidaldb`, `cargo test -p tidal-net`, `cargo test -p tidal-server`. ## Acceptance Criteria - [ ] `replication::relay` is public, not feature-gated; `BatchEntry`, `single_event_payload`, `redeliver_missed` live there with their invariant tests - [ ] `SignalRelay` encapsulates seqno/log/ship/redeliver; `SimulatedCluster` consumes it (no duplicated write/ship/redeliver logic remains in `testing::`) - [ ] `Hlc` offset constructors exist; `TidalDbBuilder::with_hlc_offset_ms` plumbs to `take_crdt_snapshot`'s HLC; default-0 behavior unchanged - [ ] `StateSnapshot` serializes/deserializes (JSON) with reconcile-determinism proven across a roundtrip - [ ] All workspace tests pass; `cargo clippy -p tidaldb -- -D warnings` clean; fmt clean