tidaldb/docs/planning/milestone-8/phase-10/task-01-engine-relay-hlc-snapshot.md

# Task 01: Engine relay module, HLC clock offset, StateSnapshot wire serde

## Delivers

Three engine-side prerequisites for one-region-per-process cluster mode, all in the
`tidaldb` crate:

1. **`replication::relay`** — a public, production (NOT `testing`-gated) module owning the
   signal-relay primitives that `SimulatedCluster` currently hides inside
   `testing::cluster_transport`, plus a new `SignalRelay` type that encapsulates one
   leader's replication stream (seqno counter, batch log, eager ship, redelivery) so the
   server's multi-process node and the simulated harness share ONE implementation of the
   load-bearing invariants.
2. **HLC clock offset** — `Hlc` accepts a wall-clock offset (ms, signed) applied inside
   `wall_ms_now`, plumbed from `TidalDbBuilder::with_hlc_offset_ms(i64)` through to the
   `Hlc::for_shard` construction in `take_crdt_snapshot`. This lets a whole process run
   with a genuinely skewed HLC for the m8p10 clock-skew UAT (the spec's "inject clock
   offset via HLC's max(wall_clock, last_seen + 1) mechanism").
3. **`StateSnapshot` wire serde** — `StateSnapshot` (and any contained type still missing
   it) becomes serializable/deserializable so CRDT snapshots can be exchanged between
   processes over HTTP for cross-process reconciliation.

## Complexity: L

## Dependencies

None (first task of the phase). Existing invariant tests in
`tidal/src/testing/cluster_transport.rs` define the contract the moved code must keep.

## Technical Design

### 1. `tidal/src/replication/relay.rs`

Move from `testing::cluster_transport` (and make `pub`):

- `BatchEntry { source_shard, seqno, bytes }` (fields pub)
- `single_event_payload(source_shard, seqno, bytes) -> WalSegmentPayload`
- `redeliver_missed(transport, db, log)` — keep doc comments and BOTH unit tests
  (`single_event_payload_holds_replication_invariants`,
  `redeliver_ships_to_local_shard_with_source_payload`) verbatim; they lock the m8p9
  wrong-destination regression.

`testing::cluster_transport` keeps only the channel transports and re-exports the moved
items so `SimulatedCluster` is otherwise untouched (`pub(super)` re-export is fine).

New type in the same module:

```rust
/// One leader's replication stream: monotonic seqno, batch log, eager ship,
/// idempotent redelivery. Single source of truth for the write/ship atomicity
/// invariants shared by `SimulatedCluster` and `tidal-server`'s multi-process
/// region node.
pub struct SignalRelay {
    seqno: Mutex<u64>,
    batch_log: Mutex<Vec<BatchEntry>>,
}

impl SignalRelay {
    pub fn new() -> Self;

    /// Atomically: bump seqno, encode the one-event batch, apply the write to
    /// the local db (all under the seqno lock, rolling the seqno back on
    /// encode/apply failure — exactly `SimulatedCluster::write_signal`'s
    /// contract), then best-effort eager-ship to every peer in `peers`
    /// (skipping `skip`, e.g. partitioned regions) and record the batch in the
    /// log. Returns the committed seqno.
    pub fn write_and_ship(
        &self,
        db: &TidalDb,
        source_shard: ShardId,
        signal_type_id: u8,
        signal_type: &str,
        entity_id: EntityId,
        weight: f64,
        transport: &dyn Transport,
        peers: &[ShardId],
        skip: &HashSet<ShardId>,
    ) -> crate::Result<u64>;

    /// Re-deliver unapplied batches to ONE peer (heal / convergence path).
    /// Destination is the peer's shard; payload keeps the source shard.
    pub fn redeliver_to(&self, transport: &dyn Transport, dest: ShardId, db_applied: impl Fn(ShardId) -> u64);

    pub fn last_seq(&self) -> u64;
    pub fn log_len(&self) -> usize;
}
```

Signature details are the implementer's to refine (e.g. `redeliver_to` may take the
follower's applied-seqno lookup as a closure or take `&TidalDb` when local, as
`redeliver_missed` does today — but in the multi-process case the sender does NOT have the
follower's `TidalDb`, so redelivery must consult a remotely-reported applied seqno; design
the API so both call shapes work). The non-negotiable invariants, documented on the type:

1. seqno bump + encode + local apply are atomic under one lock with rollback on failure
   (no burned seqnos, no leader-ahead-of-log states);
2. payloads are built ONLY via `single_event_payload`;
3. eager ship is best-effort (WARN on failure, never fails the write);
4. redelivery is idempotent and per-source FIFO (the `seqno > applied` gate).

`SimulatedCluster::write_signal` / `await_convergence` / `heal_region` are refactored to
use `SignalRelay` (one per leader region — the existing `leader_seqnos: HashMap` becomes
`HashMap<RegionId, SignalRelay>` or equivalent). Behavior must be bit-for-bit identical;
the existing tier-2 suites are the regression net.

Note: `SimulatedCluster` resolves `signal_type_ids` (name → u8) up front; `SignalRelay`
takes the resolved id so the relay stays schema-agnostic. Keep the u8-overflow guard where
the map is built.

### 2. HLC offset

```rust
// hlc.rs
pub struct Hlc {
    node_id: u16,
    /// Signed wall-clock offset (ms) applied in wall_ms_now(); 0 in production.
    wall_offset_ms: i64,
    packed: AtomicU64,
}

impl Hlc {
    pub const fn new(node_id: u16) -> Self;                       // offset 0
    pub const fn with_offset(node_id: u16, offset_ms: i64) -> Self;
    pub const fn for_shard(shard: ShardId) -> Self;               // offset 0
    pub const fn for_shard_with_offset(shard: ShardId, offset_ms: i64) -> Self;

    fn wall_ms_now(&self) -> u64 {
        // SystemTime::now() ± offset, saturating both directions.
    }
}
```

`wall_ms_now` becomes an instance method (`&self`); update both call sites (`now`,
`update`). Builder plumbing: `TidalDbBuilder::with_hlc_offset_ms(i64)` stores the offset
on the open `TidalDb` (alongside existing replication config); `take_crdt_snapshot` uses
`Hlc::for_shard_with_offset(local_shard, self.hlc_offset_ms)`. Default 0 everywhere — no
behavior change unless explicitly configured. `Timestamp::now()` (signal decay) is
deliberately NOT touched: the spec scopes skew injection to the HLC.

### 3. StateSnapshot serde

`StateSnapshot`'s maps are tuple-keyed (`(EntityId, SignalTypeId)` / `(EntityId,
EntityId)`), which JSON cannot key. Add a wire representation:

```rust
#[derive(serde::Serialize, serde::Deserialize)]
struct StateSnapshotWire {
    signal_states: Vec<(u64, u16, CrdtSignalState)>,
    hardneg_registers: Vec<(u64, u64, LWWRegister<HardNegAction>)>,
}
```

with `#[serde(from = "StateSnapshotWire", into = "StateSnapshotWire")]` on
`StateSnapshot` (it is already `Clone`). Add `Serialize`/`Deserialize` derives to
`CrdtSignalState`, `LWWRegister<T>`, `HlcTimestamp`, and any other contained type that
lacks them. Use the newtypes' inner values (`EntityId::as_u64`, `SignalTypeId::as_u16`)
on the wire; reconstruct via their constructors.

## Test Strategy

- Moved relay tests pass unchanged in their new home.
- New `SignalRelay` unit tests: seqno rollback on encode failure and on apply failure
  (inject a failing db write via unknown signal type), eager-ship skip set honored,
  redelivery idempotence (double redeliver → applied once — use a recording transport).
- Property test: any interleaving of `write_and_ship` + `redeliver_to` preserves
  per-source FIFO and never double-applies (model the receiver as monotonic max).
- HLC: `with_offset(+500)` produces timestamps ≥ 500ms ahead of `new()`; negative offset
  behind; `update()` from a skewed-ahead remote still yields strictly-greater local
  timestamps (causal consistency under skew); offset 0 byte-identical to today.
- Snapshot serde: JSON roundtrip preserves equality of merge results —
  `ReconciliationEngine::plan(local, remote)` == `plan(local, roundtrip(remote))`
  (deterministic plan equality), plus a proptest over random snapshots.
- Full existing suites stay green: `cargo test -p tidaldb`, `cargo test -p tidal-net`,
  `cargo test -p tidal-server`.

## Acceptance Criteria

- [ ] `replication::relay` is public, not feature-gated; `BatchEntry`,
      `single_event_payload`, `redeliver_missed` live there with their invariant tests
- [ ] `SignalRelay` encapsulates seqno/log/ship/redeliver; `SimulatedCluster` consumes it
      (no duplicated write/ship/redeliver logic remains in `testing::`)
- [ ] `Hlc` offset constructors exist; `TidalDbBuilder::with_hlc_offset_ms` plumbs to
      `take_crdt_snapshot`'s HLC; default-0 behavior unchanged
- [ ] `StateSnapshot` serializes/deserializes (JSON) with reconcile-determinism proven
      across a roundtrip
- [ ] All workspace tests pass; `cargo clippy -p tidaldb -- -D warnings` clean; fmt clean