Kind-3 term markers in the WAL stream, STREAM-relative vote frontiers,
heartbeat-only divergence detection + quarantine, and fenced promote.
Elections converge in 0.6–1.0s; zero acked-write loss across all kill points.
Closes G5 (leaderless recovery) from the v0.9 wave.
m11p1 — decoupled ack/ship path: staged writes (seqno+WAL+relay-push,
microseconds) separate from group-commit fsync; ShipQueue batches+windows
outbound segments; receiver coalesces inbound chunks before applying.
Adds first tidaldb_cluster_* metrics.
m11p2 — leader WAL is now THE replicated log: fsynced batches feed a
bounded WalShipFeed and ship byte-identical to followers; WAL seqnos
survive restarts (relay-reset hazard gone). Item metadata and embeddings
journal kind-1/2 blob records on the same stream as signals; the m8p10
HTTP broadcast is deleted. StreamSegments catch-up is follower-pulled via
server-streaming RPC, triggered on gap detection, follower boot, and
leader heal nudge. Promote carries a stream baseline so peers skip
pre-stream history.
Two correctness bugs in multi-process cluster mode (m8p10), both found live on a
real 3-pod k3s cluster while validating the deployment, both invisible to the
existing in-process / unauthenticated test suites:
1. Forwarded item/embedding writes dropped on the floor. create_item /
write_embedding gated the leader's peer broadcast on `if internal { return }`.
A write to a NON-leader gateway is forwarded to the leader with the internal
marker set (loop-prevention), so it hit that branch and terminated WITHOUT
broadcasting — the item landed only on the leader. Signals were unaffected
(the WAL relay ships regardless of the marker), which masked it. Fix: gate on
leadership, not the marker — a follower applying a marked broadcast/heal
terminates; the leader (external OR forwarded) always fans out. Forwarded
writes now return the {replicated_to,failed} report instead of a bodyless null.
2. Heal item/embedding backfill 401'd whenever TIDAL_API_KEY is set.
post_marked_blocking sent the internal marker but no Authorization header. The
marker is a trust signal, not an auth bypass (the bearer middleware runs
first), so every backfill POST was rejected 401 — a region that missed an item
while down stayed permanently inconsistent at lag 0. Unauthenticated tests
never caught it. Fix: thread TIDAL_API_KEY into RegionClusterState and attach
it (same key on every region) to the backfill POSTs.
Verified live: forwarded writes via follower gateways converge to all 3 regions;
a region scaled to 0 during an item write backfills on heal (item_failures=0).
Follow-up: add multiproc regression tests with auth for both paths.
Splits monolithic cluster.rs into tidal-server/src/cluster/ modules. Adds redeliver-missed
relay, bounded HLC drift, lag tracking, and reconcile idempotence. Five new tier-3 test suites
(chaos, lifecycle, multiproc, region, routes, runbook) all green. Docs, CHANGELOG, and ROADMAP
updated with G4/G5/G6 known gaps.
