tidaldb/docs/planning/milestone-11/phase-2.md

# m11p2 — One Replicated Log (COMPLETE — 2026-06-11)

Phase spec and exit gate: [docs/roadmap-to-cluster.md §4/m11p2](../../roadmap-to-cluster.md).
Predecessor: [phase-1.md](phase-1.md) (staged writes, ShipQueue, durable-frontier shipping).

**Goal:** every replicated mutation rides ONE log; the HTTP item/embedding
broadcast side channel (source of both 2026-06-10 live bugs) is deleted; heal
becomes pure log catch-up; `StreamSegments` is implemented; the relay's
in-memory `Vec` log dependency for recovery goes away.

## Design (as adopted)

### 1. The WAL is the replicated log — relay seqnos become WAL seqnos

m11p1 left two parallel streams on the leader: the WAL (its own seqnos,
assigned by the writer thread at group-commit drain) and the relay log (its
own seqnos, assigned at stage time, entries duplicated in memory). p2 unifies
them:

- **Ship payloads ARE the WAL's flushed batches.** The group-commit writer
  already encodes every batch (header: `first_seq`, `event_count`, BLAKE3)
  and fsyncs it. After a successful flush it hands `(encoded_bytes, first,
  last)` to the ship feed. Shipping is durable-by-construction (a batch is
  only handed over after its fsync), batching comes free (the group-commit
  batch IS the ship batch — no re-encode, byte-identical on leader disk,
  wire, and follower disk), and the m11p1 durable-frontier machinery
  simplifies to "the WAL flush frontier".
- **Seqnos survive restarts.** WAL seqnos continue from recovery, killing the
  m8p10-era hazard where a restarted leader's relay reset to 0 and followers
  silently ignored its new stream. (Terms/fencing remain m11p4.)
- **Catch-up reads the durable WAL,** not an in-memory Vec: a follower's
  reported applied seqno selects segments via `list_segments` +
  `scan_segment_readonly`, streamed in order. The ShipQueue keeps a bounded
  in-memory tail of recent flushed batches for the hot path; anything older
  falls back to segment read-back. The unbounded relay `Vec` dies.

### 2. Item-metadata + embedding records: batch-level kind (header `flags`)

The 64-byte batch header's reserved `flags` byte (byte 5) becomes the **batch
kind**: `0` = signal events (existing layout, fully back-compatible — v3
decoders that predate kinds only ever see kind 0 from old logs), `1` = item
metadata, `2` = item embedding. Kind 1/2 batches carry ONE record as a
length-delimited blob payload (`event_count = 1`, consuming one seqno):

- kind 1: `entity_id (u64 LE) | metadata` (the same serialization
  `persist_item_metadata` uses).
- kind 2: `entity_id (u64 LE) | dim (u32 LE) | f32×dim LE`.

Framing, the recovery scanner, BLAKE3 coverage, and segment rotation are
untouched (they treat the payload as opaque bytes); only encode/decode gain a
kind branch. The 64 MiB payload ceiling is ample for 1536-dim vectors (6 KiB).

### 3. Apply semantics

- **Leader:** `/items` and `/embeddings` apply to fjall storage (unchanged,
  write-through) AND append a kind-1/2 WAL record (replicated + restart
  replay). The HTTP broadcast fan-out and heal's O(items) re-broadcast are
  DELETED (`broadcast_to_peers` on the data routes, `rebroadcast_items_to_peer`,
  `post_marked_blocking`). The `x-tidal-internal` marker remains for
  forwarding only.
- **Follower receiver:** decodes per-batch kind. Signal batches fold into the
  ledger exactly as m11p1 (coalesced staged group-commit). Item/embedding
  batches apply as idempotent upserts via the same engine entry points the
  HTTP handlers use, AND append to the follower's own WAL (WAL-first, same
  invariant as signals) so follower recovery rebuilds them.
- **Replay on open (both roles):** kind-1/2 records re-apply to storage as
  idempotent upserts (storage already has them on a clean shutdown — replay
  double-apply is harmless by construction).
- **Hard negatives stay CRDT** (user-scoped, commutative; `/cluster/reconcile`)
  — the documented split: log = totally-ordered global data; CRDT = per-user
  convergent data.

### 4. StreamSegments (tidal-net)

The declared-but-unimplemented server-streaming RPC becomes the catch-up path:
request carries `from_seqno`; the leader streams encoded batches (the same
wire bytes) from its WAL tail/segments in seqno order. The ShipQueue's
catch-up path and `/cluster/heal`'s redelivery both ride it; heal of a
10-minute-down follower with 100k items completes via the stream with zero
O(items) HTTP traffic.

## Exit gate (from the roadmap)

- Chaos tests for both 2026-06-10 bug classes pass **by construction** (no
  marker-gated fan-out left to regress; no authed side POSTs left to 401).
- An item written anywhere is readable everywhere ≤ 2s p99.
- Heal of a 10-minute-down follower with 100k items completes via the stream
  without O(items) HTTP traffic.

## Status

- [x] WAL batch kinds (encode/decode + writer plumbing for kind-1/2 appends)
- [x] Flushed-batch ship feed (writer → ShipQueue tail) + WAL seqno unification
- [x] Segment read-back catch-up + StreamSegments RPC
- [x] Follower kind-aware apply (+ follower WAL-first for items/embeddings)
- [x] Delete HTTP broadcast + heal O(items) backfill; rewire handlers
- [x] Replay-on-open for kind-1/2; harness + chaos tests; docs

## As built (deltas from the design above)

- **Stream identity.** WAL seqnos ARE the stream seqnos and survive restarts
  (`WalShipFeed::initialize` continues from recovery; the m8p10 relay-reset
  hazard is dead). The m11p1 `SignalRelay`/`StagedRelayWrite`/durable-frontier
  machinery left the server entirely; `/signals` stages straight through
  `TidalDb::signal_staged` (the relay type remains for the in-process
  `SimulatedCluster` harness, which now implements `ShipSource`).
- **Bounded tail + pull, not leader read-back.** The ship queue pushes only
  the feed's bounded in-memory tail (`ShipCollect::Rotated` skips ahead);
  history is FOLLOWER-PULLED via `StreamSegments` over the leader's durable
  segments (`Transport::request_catchup`, single-flight + 2s rate limit per
  source). Pulled chunks enter the same inbound channel as live ships — one
  apply path. The pull triggers on: a detected gap after any apply round,
  follower boot, and the leader's heal nudge (`POST /cluster/catchup`,
  internal, carries the operator's own bearer credential — no unauthed side
  POSTs).
- **Promote grew a stream baseline.** A promoted leader's stream starts at
  its promote-time flushed frontier (persisted in `data_dir/stream_baseline`);
  the fan-out carries it so peers jump their frontier for the new shard, and
  every catch-up chunk announces it so a peer that missed the fan-out
  self-corrects instead of parking on pre-stream history. The ship queue is
  leadership-gated (`activate_from`/`deactivate`) — a follower's queue parks,
  so its replicated applies never echo.
- **Acks piggyback the follower's applied seqno** (`ShipSegmentResponse.
  applied_seqno`, served by an `AppliedSource` over the node's replication
  state); the ship queue folds it into its acked frontier and prunes retries
  of data the follower already holds. Heal = `resume_from(reported applied)`
  + the catch-up nudge — no redelivery scan, no O(items) anything.
- **Cluster mode now requires `--data-dir`** (validated at startup): the
  durable WAL is the replication stream, so an ephemeral node has nothing to
  ship or serve catch-up from. The tier-3 harness already provisioned
  per-node dirs; the in-process suites now do too.
- **Defensive receiver guards.** Partial-overlap ranges are refused (WARN +
  skip; the boundary-aligned pull re-delivers) instead of silently
  double-folding a prefix; a blob batch arriving with no applier wired halts
  the receiver loudly.
- **Single-node is untouched**: blob journaling and the ship feed are gated
  on cluster peers, so standalone item writes keep fjall-only durability and
  pay zero extra fsyncs.

### Test-infra finding (recorded for future debugging)

A fjall 3.0.2 keyspace whose data dir is deleted while the node still runs
wedges FOREVER: the flush worker crashes on `NotFound`, the sealed memtable
never drains, and `rotate_memtable_and_wait` polls indefinitely (the node's
checkpoint thread is the visible casualty). In tests, `TempDir` guards must
therefore be declared BEFORE anything that can hold the node — bindings in
one `let (a, b)` pattern drop in reverse order, which is exactly how the
in-process suites originally hit this. Production exposure is limited to an
operator deleting a live data dir.

## Exit-gate evidence

- `mp_items_ride_the_log_and_catchup_stream` (tier-3, real OS processes):
  items + embeddings written on the leader AND through a follower gateway
  converge to every node via the log (feed parity 1e-6); a follower stopped
  through 4 item+embedding+signal writes restarts and converges via its
  boot-time `StreamSegments` pull with NO heal verb and NO HTTP item traffic.
- Both 2026-06-10 bug classes are impossible by construction: the
  marker-gated item fan-out and the authed heal side-POSTs no longer exist
  (the one inter-node POST left, the heal nudge, forwards the operator's own
  credential).
- The "100k items / 10-minute-down follower" scale figure remains a Ref-A
  run (same infra blocker as the p1 ≤50ms p99 sub-gate: the k3s cluster is
  not reachable from this environment); the mechanism it measures is the one
  the tier-3 test drives end-to-end.