tidaldb/docs/planning/milestone-11/phase-3.md

m11p3 — Quorum-Acked Writes (COMPLETE — 2026-06-11)

Phase spec and exit gate: docs/roadmap-to-cluster.md §4/m11p3. Closes ROADMAP gap G4 ("no quorum durability — a leader-acked write can die with the leader"). Predecessors: phase-1.md (ship queue), phase-2.md (one replicated log; stream seqnos = WAL seqnos).

Goal: an opt-in durability contract a system of record can sit on: ack=quorum writes succeed only once a majority of the replica set durably holds them, with pipelined batch-level acks, honest timeout semantics (retryable 503 naming the laggards), and a ledger checker proving zero acknowledged loss under leader SIGKILL.

Design (as adopted)

1. Durable frontier reports (the foundation)

m11p2's ship ack piggybacked the follower's applied seqno read before enqueue — a monotonic floor. The key observation: that floor is already durable-true (a follower's frontier provably advances only after storage upserts AND its own WAL group-commit fsync — apply_replicated_events waits every staged append before any in-memory fold; blob applies are engine-WAL-first). What m11p3 adds is freshness, pushed the other way:

• The segment receiver calls the new Transport::notify_applied(shard, applied) once per apply round; the gRPC transport fires a ReportApplied RPC (fire-and-forget, deduped on unchanged frontiers) at the stream's source. One report covers a whole pipelined window of segments — batch-level, never per-write.
• The leader's report handler folds the mark into its hint map AND the quorum commit index (via a late-bound AppliedSink — the ship queue that owns the index is built after the transport).
• Ship acks stay instant (m11p2 semantics). Reports flow even when ships DON'T: a follower converging by catch-up pull, a quiet leader, a healed peer — all keep the commit index honest with zero leader work.

The design that did NOT ship (recorded because the failure is instructive): holding each ship ack open until its segment's durable apply. Functionally correct, and it passed every targeted test — but under open-loop load it couples the leader's ship cadence to the follower's apply latency: one gap-parked follower throttles its own feed (window × ack-wait), falls further behind, and the cluster spirals into total quorum collapse (measured: 100% write failure at 1,000 rps). The report push removes the coupling entirely.

2. Commit index = k-th largest durable mark

CommitIndex (engine, replication/commit.rs) tracks per-peer durable marks fed by frontier reports (AppliedSink), ship-ack hint folds (record_success), and heal resumes (the follower's own status report) — all three durable-true. With n = peers + 1 replicas, a seqno is committed once floor(n/2) peers report marks at or past it (the leader is the remaining majority member; nothing unfsynced can ship by construction). The index is leadership-scoped and owned by the ship queue: activate_from(baseline) resets marks to the promote baseline, deactivate()/shutdown() fail every waiter with Demoted, and an epoch counter catches activate→deactivate→activate cycles mid-wait. PeerState.acked (transport-accept, retry pruning) is deliberately NOT a quorum input.

3. ack=leader|quorum on the write path

• Deployment default: topology replication.ack (default leader — the m0–m11p2 contract, byte-for-byte). Per-request override: x-tidal-ack header (invalid value → 400), forwarded verbatim by follower gateways so the leader honors the CALLER's choice.
• /signals, /items, /embeddings (the replicated mutations) gate ack=quorum AFTER leader durability by awaiting a watch-channel mirror of the commit index (one bridge thread publishes; any number of handlers await for free). Fully async by necessity, not taste: the first implementation parked one blocking-pool thread per waiter, and at 1k rps open-loop the 512-thread pool filled with parked waiters while the completions that advance the index queued behind them — total collapse. Budget: replication.quorum_timeout_ms (default 2000 — the cross-region SLO).
• Every cluster write's response now carries x-tidal-seq (its replicated-log seqno; relayed through forwards), giving clients an exact durability cursor — and giving the ledger checker its ledger.
• Timeout → retryable 503 naming the laggard regions, the commit index, and the confirmed count. The write IS in the leader's log and MAY still commit: retries are at-least-once (see the idempotency decision below). Demotion mid-wait → the NotLeader 503 (never a false quorum claim).
• Hard negatives stay CRDT (per-user convergent data, /cluster/reconcile) — the ack knob does not apply. The /sharded/* surface keeps leader-ack semantics this phase (quorum × sharding lands with m11p6).

4. Follower blob applies are group-committed (found by the gate)

The quorum throughput gate exposed an m11p2 flaw invisible to convergence tests: replicated items/embeddings applied ONE record at a time — each paying a solo follower WAL fsync (wal_blob_first waits per record, the writer synced per blob batch). Item apply capped at the fsync floor (~100/s on macOS F_FULLFSYNC), so any item burst left followers with a multi-second frontier stall that every subsequent ack=quorum write then measured honestly as a 503. Two-sided fix, mirroring m11p1's signal coalescing: ReplicatedBlobApplier became batched (apply_blobs: the engine validates ALL records, stages ALL WAL appends, waits all — shared group syncs — then upserts storage in order), and the WAL writer flushes its queued blobs under ONE group fsync per drain window. Corpus seeding of 2,000 items + embeddings: 39.3s → 1.8s (22×).

5. Observability

• tidaldb_cluster_relay_durable_seq is repurposed: it now reports the commit index (the m11p2 value it replaced had become equal to relay_last_seq by construction). relay_last_seq − relay_durable_seq is the cluster's quorum lag.
• New counter tidaldb_cluster_quorum_timeouts_total.
• /cluster/status/local gains commit_index and ack (the node's deployment default).

Idempotency open question — DECIDED (in-phase, per the roadmap)

No new idempotency-key machinery in m11p3. Findings: the WAL's content-hash dedup window cannot dedup client retries (the server stamps Timestamp::now() per request, so retried bodies hash differently), and even when it fires it is log-only (the in-memory aggregate still folds). Items/embeddings retries are idempotent upserts — retrying a quorum-timeout 503 on /items///embeddings is always safe. Signal retries are at-least-once: a retried 503 whose original DID commit double-counts that signal's weight — bounded, decays, and self-corrects relative to exact counting over time. Documented in runbook §8; client-supplied idempotency keys ride on the session-write precedent (WalCommand::SessionSignal already carries one) if a future phase needs exact-once accounting.

Exit gate (from the roadmap)

• SIGKILL the leader under ack=quorum load → ledger checker proves zero acknowledged loss across 100 random kill-points.
• Quorum throughput ≥ 50% of p1's leader-ack number (≥1,000/s Ref-A).

Status

• Durable frontier reports (ReportApplied, notify_applied, AppliedSink)
• Follower blob group-commit (apply_blobs + writer group fsync)
• CommitIndex (k-th-largest marks, leadership-scoped, laggard naming)
• Ship-queue wiring (durable-mark folds, activate/deactivate/shutdown)
• Seqno surfacing (x-tidal-seq through engine → handlers → forwards)
• ack=leader|quorum: topology default + header override + 503 body
• Metrics + status surfaces
• In-process gRPC tests (gate/503/heal; forwarded quorum; blob writes)
• Tier-3 suite: partition semantics + the ledger checker
• 100-kill-point gate run + throughput run (recorded below)
• Docs (runbook §8 rewrite, topology table, monitoring) + CHANGELOG

Exit-gate evidence (local; release builds, real OS processes)

Gate	Target	Measured
Zero acknowledged loss under leader SIGKILL	100 random kill-points	167/167 kill-points, zero acked loss on the final design (batches of 64 + 91 + 12; each round = fresh 3-process cluster, concurrent ack=quorum item+signal writers via leader AND follower gateways, SIGKILL at timings spread 120–598ms, then BOTH proofs on the promoted max-applied survivor: max(acked seq) <= survivor applied_events and per-item /search content probes). An earlier 100/100 run validated the superseded ack-holding design before it was replaced.
Quorum throughput ≥ 50% of p1's leader-ack number	≥ 2,267/s local-equivalent	3,600 quorum signal-writes/s within SLO (writes mix, wal.batch_timeout_ms: 2, knee not reached; p50 ~45ms at 1.5k, ~370ms at 3.6k from concurrency-limit queueing; zero errors; replication lag ≤3 events at ramp end) = 79% of p1's 4,534/s. Same-day leader-ack baseline on this machine: 5,971/s (10ms batch timeout). Ref-A (Linux) runs remain blocked on k3s access — same as p1/p2.

Tier-3 partition semantics (real TCP severs): one follower down → quorum commits via the other; both down → fast 503 naming the laggards while ack=leader writes flow uninterrupted; healed → quorum recovers within the breaker window. (mp_quorum_writes_gate_and_recover_under_partition.)

Performance findings the gate forced (all shipped)

1. Thread-per-quorum-wait collapses at 1k rps: parked Condvar waiters exhausted tokio's 512-thread blocking pool; the group-commit completions that advance the index queued BEHIND them → 0 OK writes, 40s latencies. Fix: async watch-channel bridge (one publisher thread, zero per-waiter threads).
2. Holding ship acks until durable apply death-spirals: ack-wait × window throttles ships to a gap-parked follower → it lags further → all quorum writes time out. Fix: follower-pushed ReportApplied, ship acks stay instant.
3. Per-record blob fsyncs cap item apply at ~100/s (m11p2 flaw): fixed with batched blob apply + writer group fsync → seeding 22× faster, item bursts no longer stall the quorum frontier.