tidaldb/docs/planning/milestone-11/phase-2.md

m11p2 — One Replicated Log (COMPLETE — 2026-06-11)

Phase spec and exit gate: docs/roadmap-to-cluster.md §4/m11p2. Predecessor: phase-1.md (staged writes, ShipQueue, durable-frontier shipping).

Goal: every replicated mutation rides ONE log; the HTTP item/embedding broadcast side channel (source of both 2026-06-10 live bugs) is deleted; heal becomes pure log catch-up; StreamSegments is implemented; the relay's in-memory Vec log dependency for recovery goes away.

Design (as adopted)

1. The WAL is the replicated log — relay seqnos become WAL seqnos

m11p1 left two parallel streams on the leader: the WAL (its own seqnos, assigned by the writer thread at group-commit drain) and the relay log (its own seqnos, assigned at stage time, entries duplicated in memory). p2 unifies them:

• Ship payloads ARE the WAL's flushed batches. The group-commit writer already encodes every batch (header: first_seq, event_count, BLAKE3) and fsyncs it. After a successful flush it hands (encoded_bytes, first, last) to the ship feed. Shipping is durable-by-construction (a batch is only handed over after its fsync), batching comes free (the group-commit batch IS the ship batch — no re-encode, byte-identical on leader disk, wire, and follower disk), and the m11p1 durable-frontier machinery simplifies to "the WAL flush frontier".
• Seqnos survive restarts. WAL seqnos continue from recovery, killing the m8p10-era hazard where a restarted leader's relay reset to 0 and followers silently ignored its new stream. (Terms/fencing remain m11p4.)
• Catch-up reads the durable WAL, not an in-memory Vec: a follower's reported applied seqno selects segments via list_segments + scan_segment_readonly, streamed in order. The ShipQueue keeps a bounded in-memory tail of recent flushed batches for the hot path; anything older falls back to segment read-back. The unbounded relay Vec dies.

2. Item-metadata + embedding records: batch-level kind (header flags)

The 64-byte batch header's reserved flags byte (byte 5) becomes the batch kind: 0 = signal events (existing layout, fully back-compatible — v3 decoders that predate kinds only ever see kind 0 from old logs), 1 = item metadata, 2 = item embedding. Kind 1/2 batches carry ONE record as a length-delimited blob payload (event_count = 1, consuming one seqno):

• kind 1: entity_id (u64 LE) | metadata (the same serialization persist_item_metadata uses).
• kind 2: entity_id (u64 LE) | dim (u32 LE) | f32×dim LE.

Framing, the recovery scanner, BLAKE3 coverage, and segment rotation are untouched (they treat the payload as opaque bytes); only encode/decode gain a kind branch. The 64 MiB payload ceiling is ample for 1536-dim vectors (6 KiB).

3. Apply semantics

• Leader: /items and /embeddings apply to fjall storage (unchanged, write-through) AND append a kind-1/2 WAL record (replicated + restart replay). The HTTP broadcast fan-out and heal's O(items) re-broadcast are DELETED (broadcast_to_peers on the data routes, rebroadcast_items_to_peer, post_marked_blocking). The x-tidal-internal marker remains for forwarding only.
• Follower receiver: decodes per-batch kind. Signal batches fold into the ledger exactly as m11p1 (coalesced staged group-commit). Item/embedding batches apply as idempotent upserts via the same engine entry points the HTTP handlers use, AND append to the follower's own WAL (WAL-first, same invariant as signals) so follower recovery rebuilds them.
• Replay on open (both roles): kind-1/2 records re-apply to storage as idempotent upserts (storage already has them on a clean shutdown — replay double-apply is harmless by construction).
• Hard negatives stay CRDT (user-scoped, commutative; /cluster/reconcile) — the documented split: log = totally-ordered global data; CRDT = per-user convergent data.

4. StreamSegments (tidal-net)

The declared-but-unimplemented server-streaming RPC becomes the catch-up path: request carries from_seqno; the leader streams encoded batches (the same wire bytes) from its WAL tail/segments in seqno order. The ShipQueue's catch-up path and /cluster/heal's redelivery both ride it; heal of a 10-minute-down follower with 100k items completes via the stream with zero O(items) HTTP traffic.

Exit gate (from the roadmap)

• Chaos tests for both 2026-06-10 bug classes pass by construction (no marker-gated fan-out left to regress; no authed side POSTs left to 401).
• An item written anywhere is readable everywhere ≤ 2s p99.
• Heal of a 10-minute-down follower with 100k items completes via the stream without O(items) HTTP traffic.

Status

• WAL batch kinds (encode/decode + writer plumbing for kind-1/2 appends)
• Flushed-batch ship feed (writer → ShipQueue tail) + WAL seqno unification
• Segment read-back catch-up + StreamSegments RPC
• Follower kind-aware apply (+ follower WAL-first for items/embeddings)
• Delete HTTP broadcast + heal O(items) backfill; rewire handlers
• Replay-on-open for kind-1/2; harness + chaos tests; docs

As built (deltas from the design above)

• Stream identity. WAL seqnos ARE the stream seqnos and survive restarts (WalShipFeed::initialize continues from recovery; the m8p10 relay-reset hazard is dead). The m11p1 SignalRelay/StagedRelayWrite/durable-frontier machinery left the server entirely; /signals stages straight through TidalDb::signal_staged (the relay type remains for the in-process SimulatedCluster harness, which now implements ShipSource).
• Bounded tail + pull, not leader read-back. The ship queue pushes only the feed's bounded in-memory tail (ShipCollect::Rotated skips ahead); history is FOLLOWER-PULLED via StreamSegments over the leader's durable segments (Transport::request_catchup, single-flight + 2s rate limit per source). Pulled chunks enter the same inbound channel as live ships — one apply path. The pull triggers on: a detected gap after any apply round, follower boot, and the leader's heal nudge (POST /cluster/catchup, internal, carries the operator's own bearer credential — no unauthed side POSTs).
• Promote grew a stream baseline. A promoted leader's stream starts at its promote-time flushed frontier (persisted in data_dir/stream_baseline); the fan-out carries it so peers jump their frontier for the new shard, and every catch-up chunk announces it so a peer that missed the fan-out self-corrects instead of parking on pre-stream history. The ship queue is leadership-gated (activate_from/deactivate) — a follower's queue parks, so its replicated applies never echo.
• Acks piggyback the follower's applied seqno (ShipSegmentResponse. applied_seqno, served by an AppliedSource over the node's replication state); the ship queue folds it into its acked frontier and prunes retries of data the follower already holds. Heal = resume_from(reported applied)
  • the catch-up nudge — no redelivery scan, no O(items) anything.
• Cluster mode now requires --data-dir (validated at startup): the durable WAL is the replication stream, so an ephemeral node has nothing to ship or serve catch-up from. The tier-3 harness already provisioned per-node dirs; the in-process suites now do too.
• Defensive receiver guards. Partial-overlap ranges are refused (WARN + skip; the boundary-aligned pull re-delivers) instead of silently double-folding a prefix; a blob batch arriving with no applier wired halts the receiver loudly.
• Single-node is untouched: blob journaling and the ship feed are gated on cluster peers, so standalone item writes keep fjall-only durability and pay zero extra fsyncs.

Test-infra finding (recorded for future debugging)

A fjall 3.0.2 keyspace whose data dir is deleted while the node still runs wedges FOREVER: the flush worker crashes on NotFound, the sealed memtable never drains, and rotate_memtable_and_wait polls indefinitely (the node's checkpoint thread is the visible casualty). In tests, TempDir guards must therefore be declared BEFORE anything that can hold the node — bindings in one let (a, b) pattern drop in reverse order, which is exactly how the in-process suites originally hit this. Production exposure is limited to an operator deleting a live data dir.

Exit-gate evidence

• mp_items_ride_the_log_and_catchup_stream (tier-3, real OS processes): items + embeddings written on the leader AND through a follower gateway converge to every node via the log (feed parity 1e-6); a follower stopped through 4 item+embedding+signal writes restarts and converges via its boot-time StreamSegments pull with NO heal verb and NO HTTP item traffic.
• Both 2026-06-10 bug classes are impossible by construction: the marker-gated item fan-out and the authed heal side-POSTs no longer exist (the one inter-node POST left, the heal nudge, forwards the operator's own credential).
• The "100k items / 10-minute-down follower" scale figure remains a Ref-A run (same infra blocker as the p1 ≤50ms p99 sub-gate: the k3s cluster is not reachable from this environment); the mechanism it measures is the one the tier-3 test drives end-to-end.