# m11p3 — Quorum-Acked Writes (COMPLETE — 2026-06-11) Phase spec and exit gate: [docs/roadmap-to-cluster.md §4/m11p3](../../roadmap-to-cluster.md). Closes ROADMAP gap **G4** ("no quorum durability — a leader-acked write can die with the leader"). Predecessors: [phase-1.md](phase-1.md) (ship queue), [phase-2.md](phase-2.md) (one replicated log; stream seqnos = WAL seqnos). **Goal:** an opt-in durability contract a system of record can sit on: `ack=quorum` writes succeed only once a **majority of the replica set durably holds them**, with pipelined batch-level acks, honest timeout semantics (retryable 503 naming the laggards), and a ledger checker proving zero acknowledged loss under leader SIGKILL. ## Design (as adopted) ### 1. Durable frontier reports (the foundation) m11p2's ship ack piggybacked the follower's applied seqno read **before** enqueue — a monotonic floor. The key observation: that floor is already **durable-true** (a follower's frontier provably advances only after storage upserts AND its own WAL group-commit fsync — `apply_replicated_events` waits every staged append before any in-memory fold; blob applies are engine-WAL-first). What m11p3 adds is **freshness**, pushed the other way: - The segment receiver calls the new `Transport::notify_applied(shard, applied)` once per **apply round**; the gRPC transport fires a `ReportApplied` RPC (fire-and-forget, deduped on unchanged frontiers) at the stream's source. One report covers a whole pipelined window of segments — batch-level, never per-write. - The leader's report handler folds the mark into its hint map AND the quorum commit index (via a late-bound `AppliedSink` — the ship queue that owns the index is built after the transport). - Ship acks stay instant (m11p2 semantics). Reports flow even when ships DON'T: a follower converging by catch-up pull, a quiet leader, a healed peer — all keep the commit index honest with zero leader work. **The design that did NOT ship** (recorded because the failure is instructive): holding each ship ack open until its segment's durable apply. Functionally correct, and it passed every targeted test — but under open-loop load it couples the leader's ship cadence to the follower's apply latency: one gap-parked follower throttles its own feed (window × ack-wait), falls further behind, and the cluster spirals into total quorum collapse (measured: 100% write failure at 1,000 rps). The report push removes the coupling entirely. ### 2. Commit index = k-th largest durable mark `CommitIndex` (engine, `replication/commit.rs`) tracks per-peer durable marks fed by frontier reports (`AppliedSink`), ship-ack hint folds (`record_success`), and heal resumes (the follower's own status report) — all three durable-true. With `n = peers + 1` replicas, a seqno is **committed** once `floor(n/2)` peers report marks at or past it (the leader is the remaining majority member; nothing unfsynced can ship by construction). The index is leadership-scoped and owned by the ship queue: `activate_from(baseline)` resets marks to the promote baseline, `deactivate()`/`shutdown()` fail every waiter with `Demoted`, and an epoch counter catches activate→deactivate→activate cycles mid-wait. `PeerState.acked` (transport-accept, retry pruning) is deliberately NOT a quorum input. ### 3. `ack=leader|quorum` on the write path - **Deployment default**: topology `replication.ack` (default `leader` — the m0–m11p2 contract, byte-for-byte). **Per-request override**: `x-tidal-ack` header (invalid value → 400), forwarded verbatim by follower gateways so the leader honors the CALLER's choice. - `/signals`, `/items`, `/embeddings` (the replicated mutations) gate `ack=quorum` AFTER leader durability by awaiting a **watch-channel mirror** of the commit index (one bridge thread publishes; any number of handlers await for free). Fully async by necessity, not taste: the first implementation parked one blocking-pool thread per waiter, and at 1k rps open-loop the 512-thread pool filled with parked waiters while the completions that advance the index queued behind them — total collapse. Budget: `replication.quorum_timeout_ms` (default 2000 — the cross-region SLO). - Every cluster write's response now carries **`x-tidal-seq`** (its replicated-log seqno; relayed through forwards), giving clients an exact durability cursor — and giving the ledger checker its ledger. - Timeout → **retryable 503** naming the laggard regions, the commit index, and the confirmed count. The write IS in the leader's log and MAY still commit: retries are at-least-once (see the idempotency decision below). Demotion mid-wait → the NotLeader 503 (never a false quorum claim). - Hard negatives stay CRDT (per-user convergent data, `/cluster/reconcile`) — the ack knob does not apply. The `/sharded/*` surface keeps leader-ack semantics this phase (quorum × sharding lands with m11p6). ### 4. Follower blob applies are group-committed (found by the gate) The quorum throughput gate exposed an m11p2 flaw invisible to convergence tests: replicated items/embeddings applied ONE record at a time — each paying a solo follower WAL fsync (`wal_blob_first` waits per record, the writer synced per blob batch). Item apply capped at the fsync floor (~100/s on macOS `F_FULLFSYNC`), so any item burst left followers with a multi-second frontier stall that every subsequent `ack=quorum` write then measured honestly as a 503. Two-sided fix, mirroring m11p1's signal coalescing: `ReplicatedBlobApplier` became batched (`apply_blobs`: the engine validates ALL records, stages ALL WAL appends, waits all — shared group syncs — then upserts storage in order), and the WAL writer flushes its queued blobs under ONE group fsync per drain window. Corpus seeding of 2,000 items + embeddings: 39.3s → **1.8s** (22×). ### 5. Observability - `tidaldb_cluster_relay_durable_seq` is repurposed: it now reports the **commit index** (the m11p2 value it replaced had become equal to `relay_last_seq` by construction). `relay_last_seq − relay_durable_seq` is the cluster's quorum lag. - New counter `tidaldb_cluster_quorum_timeouts_total`. - `/cluster/status/local` gains `commit_index` and `ack` (the node's deployment default). ### Idempotency open question — DECIDED (in-phase, per the roadmap) **No new idempotency-key machinery in m11p3.** Findings: the WAL's content-hash dedup window cannot dedup client retries (the server stamps `Timestamp::now()` per request, so retried bodies hash differently), and even when it fires it is log-only (the in-memory aggregate still folds). Items/embeddings retries are idempotent upserts — retrying a quorum-timeout 503 on `/items`//`/embeddings` is always safe. Signal retries are at-least-once: a retried 503 whose original DID commit double-counts that signal's weight — bounded, decays, and self-corrects relative to exact counting over time. Documented in runbook §8; client-supplied idempotency keys ride on the session-write precedent (`WalCommand::SessionSignal` already carries one) if a future phase needs exact-once accounting. ## Exit gate (from the roadmap) - SIGKILL the leader under `ack=quorum` load → ledger checker proves zero acknowledged loss across 100 random kill-points. - Quorum throughput ≥ 50% of p1's leader-ack number (≥1,000/s Ref-A). ## Status - [x] Durable frontier reports (`ReportApplied`, `notify_applied`, `AppliedSink`) - [x] Follower blob group-commit (`apply_blobs` + writer group fsync) - [x] `CommitIndex` (k-th-largest marks, leadership-scoped, laggard naming) - [x] Ship-queue wiring (durable-mark folds, activate/deactivate/shutdown) - [x] Seqno surfacing (`x-tidal-seq` through engine → handlers → forwards) - [x] `ack=leader|quorum`: topology default + header override + 503 body - [x] Metrics + status surfaces - [x] In-process gRPC tests (gate/503/heal; forwarded quorum; blob writes) - [x] Tier-3 suite: partition semantics + the ledger checker - [x] 100-kill-point gate run + throughput run (recorded below) - [x] Docs (runbook §8 rewrite, topology table, monitoring) + CHANGELOG ## Exit-gate evidence (local; release builds, real OS processes) | Gate | Target | Measured | |------|--------|----------| | Zero acknowledged loss under leader SIGKILL | 100 random kill-points | **167/167 kill-points, zero acked loss** on the final design (batches of 64 + 91 + 12; each round = fresh 3-process cluster, concurrent `ack=quorum` item+signal writers via leader AND follower gateways, SIGKILL at timings spread 120–598ms, then BOTH proofs on the promoted max-applied survivor: `max(acked seq) <= survivor applied_events` and per-item `/search` content probes). An earlier 100/100 run validated the superseded ack-holding design before it was replaced. | | Quorum throughput ≥ 50% of p1's leader-ack number | ≥ 2,267/s local-equivalent | **3,600 quorum signal-writes/s within SLO** (writes mix, `wal.batch_timeout_ms: 2`, knee not reached; p50 ~45ms at 1.5k, ~370ms at 3.6k from concurrency-limit queueing; zero errors; replication lag ≤3 events at ramp end) = **79%** of p1's 4,534/s. Same-day leader-ack baseline on this machine: 5,971/s (10ms batch timeout). Ref-A (Linux) runs remain blocked on k3s access — same as p1/p2. | Tier-3 partition semantics (real TCP severs): one follower down → quorum commits via the other; both down → fast 503 naming the laggards while `ack=leader` writes flow uninterrupted; healed → quorum recovers within the breaker window. (`mp_quorum_writes_gate_and_recover_under_partition`.) ### Performance findings the gate forced (all shipped) 1. **Thread-per-quorum-wait collapses at 1k rps**: parked Condvar waiters exhausted tokio's 512-thread blocking pool; the group-commit completions that advance the index queued BEHIND them → 0 OK writes, 40s latencies. Fix: async watch-channel bridge (one publisher thread, zero per-waiter threads). 2. **Holding ship acks until durable apply death-spirals**: ack-wait × window throttles ships to a gap-parked follower → it lags further → all quorum writes time out. Fix: follower-pushed `ReportApplied`, ship acks stay instant. 3. **Per-record blob fsyncs cap item apply at ~100/s** (m11p2 flaw): fixed with batched blob apply + writer group fsync → seeding 22× faster, item bursts no longer stall the quorum frontier.