tidaldb/docs/planning/milestone-11/phase-1.md

m11p1 — Replication Performance Floor (✅ COMPLETE 2026-06-11)

Phase spec and exit gate: docs/roadmap-to-cluster.md §4/m11p1. Status summary: ROADMAP Milestone 11. Changelog: CHANGELOG.md.

What shipped

1. Staged two-phase replicated writes (SignalRelay::stage_write / complete_write): seqno bump + WAL submission + relay-log push stay atomic under the seqno lock (microseconds, rollback on staging failure); the group-commit fsync wait happens outside it, so concurrent writers share fsyncs. The old path held the lock across the fsync, serializing every writer onto a solo batch_timeout + fsync — the measured ~90 writes/s ceiling and ~178ms latency (2–8 pool workers × ~11ms, 16-deep queue).
2. Per-peer windowed batch shipping (ShipQueue): dedicated sender threads per peer coalesce contiguous relay-log runs into multi-event batches (≤ replication.batch_max_events), replication.window in flight per peer, parked-run retry every replication.retry_ms. Senders ship only the leader's contiguous durable frontier; an fsync failure after staging poisons the relay (writes rejected, frontier frozen) instead of risking leader/follower divergence.
3. Follower group-commit coalescing: the segment receiver drains its inbound backlog (Transport::try_recv_segment) and applies it through one shared group commit (SignalLedger::apply_replicated_events) in range-disjoint groups (duplicate/subset re-ships cannot double-fold). Without this, each segment paid a solo follower fsync: ~1.8k events/s apply ceiling measured, unbounded lag under the new leader rates.
4. First cluster observability: per-region topology metrics_addr wires the engine's /metrics listener (cluster mode previously had none); tidaldb_cluster_* series for ship RTT/batch size (per-peer labeled), WAL fsync latency + group-commit fill, write-pool depth/rejections, and relay committed/durable frontiers. WAL group-commit knobs became config (wal.batch_size, wal.batch_timeout_ms).

Exit gate evidence (local Ref-A-equivalent: 3 real processes, release build)

Gate	Target	Measured
Replicated signal throughput	≥ 2,000/s	4,534/s within SLO on the ramp (knee ~5.5k/s); 2,739/s sustained 10 min (1.65M writes)
Error rate	< 1%	0.35% (10-min sustain), zero client shed
Replication lag	p99 ≤ 2s	≤ 103 events (~40ms) across the 10-min sustain
Replication metrics in Prometheus	visible	tidaldb_cluster_* live on the per-region listener
Signal write p99	≤ 50ms	p50 21ms / p90 35ms / p99 ~199ms locally — the macOS F_FULLFSYNC floor (7.4ms mean, 10–50ms tail per tidaldb_cluster_wal_fsync_us; 0% under 1ms). Validate on Ref-A (Linux fdatasync), where the fsync floor is far lower. Ref-A re-run pending infra access.

Tier-3 suites (multiproc 5, chaos 3, lifecycle 2, runbook 9, e2e 2) green over real OS processes with real TCP-relay partition injection.

Incidental fixes

• Ship-sender failure logging is transition-based (1st + every 50th consecutive failure at WARN, recovery at INFO): the per-retry WARN flood filled the test harness's undrained stderr pipe and froze every logging thread in the leader (spurious 408s). The harness now uses Stdio::null() for node logs (TIDAL_TEST_NODE_LOGS=inherit to stream them while debugging).

Carried hazards (tracked for later phases)

• The relay log remains an unbounded in-memory Vec (now RelayEvent, ~48B per entry vs ~150B before) — m11p2 replaces it with durable-WAL read-back (StreamSegments).
• Relay seqno resets on leader restart while followers keep their applied high-water marks — pre-existing; m11p4's durable terms/fencing owns it.
• Follower crash between WAL append and frontier advance can double-fold on redelivery after the dedup window rotates — pre-existing crash window; m11p3's no-acked-loss ledger checker will surface and bound it.