5ed2edb211
ack=quorum gates replicated writes on a majority of the replica set durably holding them: followers push their durably-applied frontier (ReportApplied, once per apply round, decoupled from ship acks), the leader folds frontier reports + ship-ack hints + heal resumes into a leadership-scoped CommitIndex (k-th-largest durable mark), and handlers await it through an async watch-channel bridge (zero parked threads per waiter). Honest timeouts: retryable 503 naming the laggards; x-tidal-seq on every cluster write. Follower blob applies are batched under group-commit fsyncs (22x seeding). Exit gate: 167/167 leader-SIGKILL kill points, zero acked-write loss. Seven-dimension review pass (all confirmed findings fixed): - WAL blob drain now ABORTS on the first write failure instead of reusing the failed seqno mid-drain (a torn record buried mid-segment would truncate every later acked record on replay) - apply_replicated_blobs waits every staged append even after a mid-batch failure, parses metadata once, and moves records into Arcs shared with the WAL writer (no deep clone per record on the follower apply path) - CommitIndex: zero-peer fast path now respects demotion (active checked under lock before the single-replica return), k-th-largest uses select_nth over a reused scratch buffer - await_quorum: re-reads the index once after the deadline fires (no false 503 for a write that committed in the race window), warns when the commit-watch bridge dies outside shutdown, zero-peer path checks active - notify_applied report failures: WARN on the first failure of a streak, INFO on recovery (a silently stalling frontier reads as unexplained quorum 503s); receiver skips re-notifying unadvanced frontiers - x-tidal-deduplicated: 1 marks dedup-suppressed signal writes (relayed through forwards) so durability cursors can tell dedup from no-seqno - docs: 167/167 kill-point record corrected in CHANGELOG; rolling-upgrade order (leader first — a pre-m11p3 leader silently downgrades quorum requests to leader-ack) in CHANGELOG + runbook §8; monitoring note for report-loss diagnosis on the quorum-timeout alert Verified: workspace clippy -D warnings (incl. cluster-e2e targets), full tidaldb/tidal-net/tidal-server/tidalctl suites green, tier-3 multi-process quorum suite green (8/8 kill points, zero acked loss, partition gate/recover).
173 lines
10 KiB
Markdown
173 lines
10 KiB
Markdown
# m11p3 — Quorum-Acked Writes (COMPLETE — 2026-06-11)
|
||
|
||
Phase spec and exit gate: [docs/roadmap-to-cluster.md §4/m11p3](../../roadmap-to-cluster.md).
|
||
Closes ROADMAP gap **G4** ("no quorum durability — a leader-acked write can
|
||
die with the leader").
|
||
Predecessors: [phase-1.md](phase-1.md) (ship queue), [phase-2.md](phase-2.md)
|
||
(one replicated log; stream seqnos = WAL seqnos).
|
||
|
||
**Goal:** an opt-in durability contract a system of record can sit on:
|
||
`ack=quorum` writes succeed only once a **majority of the replica set
|
||
durably holds them**, with pipelined batch-level acks, honest timeout
|
||
semantics (retryable 503 naming the laggards), and a ledger checker proving
|
||
zero acknowledged loss under leader SIGKILL.
|
||
|
||
## Design (as adopted)
|
||
|
||
### 1. Durable frontier reports (the foundation)
|
||
|
||
m11p2's ship ack piggybacked the follower's applied seqno read **before**
|
||
enqueue — a monotonic floor. The key observation: that floor is already
|
||
**durable-true** (a follower's frontier provably advances only after storage
|
||
upserts AND its own WAL group-commit fsync — `apply_replicated_events` waits
|
||
every staged append before any in-memory fold; blob applies are
|
||
engine-WAL-first). What m11p3 adds is **freshness**, pushed the other way:
|
||
|
||
- The segment receiver calls the new `Transport::notify_applied(shard,
|
||
applied)` once per **apply round**; the gRPC transport fires a
|
||
`ReportApplied` RPC (fire-and-forget, deduped on unchanged frontiers) at
|
||
the stream's source. One report covers a whole pipelined window of
|
||
segments — batch-level, never per-write.
|
||
- The leader's report handler folds the mark into its hint map AND the
|
||
quorum commit index (via a late-bound `AppliedSink` — the ship queue that
|
||
owns the index is built after the transport).
|
||
- Ship acks stay instant (m11p2 semantics). Reports flow even when ships
|
||
DON'T: a follower converging by catch-up pull, a quiet leader, a healed
|
||
peer — all keep the commit index honest with zero leader work.
|
||
|
||
**The design that did NOT ship** (recorded because the failure is
|
||
instructive): holding each ship ack open until its segment's durable apply.
|
||
Functionally correct, and it passed every targeted test — but under
|
||
open-loop load it couples the leader's ship cadence to the follower's apply
|
||
latency: one gap-parked follower throttles its own feed (window × ack-wait),
|
||
falls further behind, and the cluster spirals into total quorum collapse
|
||
(measured: 100% write failure at 1,000 rps). The report push removes the
|
||
coupling entirely.
|
||
|
||
### 2. Commit index = k-th largest durable mark
|
||
|
||
`CommitIndex` (engine, `replication/commit.rs`) tracks per-peer durable
|
||
marks fed by frontier reports (`AppliedSink`), ship-ack hint folds
|
||
(`record_success`), and heal resumes (the follower's own status report) —
|
||
all three durable-true. With `n = peers + 1` replicas,
|
||
a seqno is **committed** once `floor(n/2)` peers report marks at or past it
|
||
(the leader is the remaining majority member; nothing unfsynced can ship by
|
||
construction). The index is leadership-scoped and owned by the ship queue:
|
||
`activate_from(baseline)` resets marks to the promote baseline,
|
||
`deactivate()`/`shutdown()` fail every waiter with `Demoted`, and an
|
||
epoch counter catches activate→deactivate→activate cycles mid-wait.
|
||
`PeerState.acked` (transport-accept, retry pruning) is deliberately NOT a
|
||
quorum input.
|
||
|
||
### 3. `ack=leader|quorum` on the write path
|
||
|
||
- **Deployment default**: topology `replication.ack` (default `leader` —
|
||
the m0–m11p2 contract, byte-for-byte). **Per-request override**:
|
||
`x-tidal-ack` header (invalid value → 400), forwarded verbatim by
|
||
follower gateways so the leader honors the CALLER's choice.
|
||
- `/signals`, `/items`, `/embeddings` (the replicated mutations) gate
|
||
`ack=quorum` AFTER leader durability by awaiting a **watch-channel mirror**
|
||
of the commit index (one bridge thread publishes; any number of handlers
|
||
await for free). Fully async by necessity, not taste: the first
|
||
implementation parked one blocking-pool thread per waiter, and at 1k rps
|
||
open-loop the 512-thread pool filled with parked waiters while the
|
||
completions that advance the index queued behind them — total collapse.
|
||
Budget: `replication.quorum_timeout_ms` (default 2000 — the cross-region
|
||
SLO).
|
||
- Every cluster write's response now carries **`x-tidal-seq`** (its
|
||
replicated-log seqno; relayed through forwards), giving clients an exact
|
||
durability cursor — and giving the ledger checker its ledger.
|
||
- Timeout → **retryable 503** naming the laggard regions, the commit index,
|
||
and the confirmed count. The write IS in the leader's log and MAY still
|
||
commit: retries are at-least-once (see the idempotency decision below).
|
||
Demotion mid-wait → the NotLeader 503 (never a false quorum claim).
|
||
- Hard negatives stay CRDT (per-user convergent data, `/cluster/reconcile`)
|
||
— the ack knob does not apply. The `/sharded/*` surface keeps leader-ack
|
||
semantics this phase (quorum × sharding lands with m11p6).
|
||
|
||
### 4. Follower blob applies are group-committed (found by the gate)
|
||
|
||
The quorum throughput gate exposed an m11p2 flaw invisible to convergence
|
||
tests: replicated items/embeddings applied ONE record at a time — each
|
||
paying a solo follower WAL fsync (`wal_blob_first` waits per record, the
|
||
writer synced per blob batch). Item apply capped at the fsync floor (~100/s
|
||
on macOS `F_FULLFSYNC`), so any item burst left followers with a
|
||
multi-second frontier stall that every subsequent `ack=quorum` write then
|
||
measured honestly as a 503. Two-sided fix, mirroring m11p1's signal
|
||
coalescing: `ReplicatedBlobApplier` became batched (`apply_blobs`: the
|
||
engine validates ALL records, stages ALL WAL appends, waits all — shared
|
||
group syncs — then upserts storage in order), and the WAL writer flushes
|
||
its queued blobs under ONE group fsync per drain window. Corpus seeding of
|
||
2,000 items + embeddings: 39.3s → **1.8s** (22×).
|
||
|
||
### 5. Observability
|
||
|
||
- `tidaldb_cluster_relay_durable_seq` is repurposed: it now reports the
|
||
**commit index** (the m11p2 value it replaced had become equal to
|
||
`relay_last_seq` by construction). `relay_last_seq − relay_durable_seq`
|
||
is the cluster's quorum lag.
|
||
- New counter `tidaldb_cluster_quorum_timeouts_total`.
|
||
- `/cluster/status/local` gains `commit_index` and `ack` (the node's
|
||
deployment default).
|
||
|
||
### Idempotency open question — DECIDED (in-phase, per the roadmap)
|
||
|
||
**No new idempotency-key machinery in m11p3.** Findings: the WAL's
|
||
content-hash dedup window cannot dedup client retries (the server stamps
|
||
`Timestamp::now()` per request, so retried bodies hash differently), and
|
||
even when it fires it is log-only (the in-memory aggregate still folds).
|
||
Items/embeddings retries are idempotent upserts — retrying a quorum-timeout
|
||
503 on `/items`//`/embeddings` is always safe. Signal retries are
|
||
at-least-once: a retried 503 whose original DID commit double-counts that
|
||
signal's weight — bounded, decays, and self-corrects relative to exact
|
||
counting over time. Documented in runbook §8; client-supplied idempotency
|
||
keys ride on the session-write precedent (`WalCommand::SessionSignal`
|
||
already carries one) if a future phase needs exact-once accounting.
|
||
|
||
## Exit gate (from the roadmap)
|
||
|
||
- SIGKILL the leader under `ack=quorum` load → ledger checker proves zero
|
||
acknowledged loss across 100 random kill-points.
|
||
- Quorum throughput ≥ 50% of p1's leader-ack number (≥1,000/s Ref-A).
|
||
|
||
## Status
|
||
|
||
- [x] Durable frontier reports (`ReportApplied`, `notify_applied`, `AppliedSink`)
|
||
- [x] Follower blob group-commit (`apply_blobs` + writer group fsync)
|
||
- [x] `CommitIndex` (k-th-largest marks, leadership-scoped, laggard naming)
|
||
- [x] Ship-queue wiring (durable-mark folds, activate/deactivate/shutdown)
|
||
- [x] Seqno surfacing (`x-tidal-seq` through engine → handlers → forwards)
|
||
- [x] `ack=leader|quorum`: topology default + header override + 503 body
|
||
- [x] Metrics + status surfaces
|
||
- [x] In-process gRPC tests (gate/503/heal; forwarded quorum; blob writes)
|
||
- [x] Tier-3 suite: partition semantics + the ledger checker
|
||
- [x] 100-kill-point gate run + throughput run (recorded below)
|
||
- [x] Docs (runbook §8 rewrite, topology table, monitoring) + CHANGELOG
|
||
|
||
## Exit-gate evidence (local; release builds, real OS processes)
|
||
|
||
| Gate | Target | Measured |
|
||
|------|--------|----------|
|
||
| Zero acknowledged loss under leader SIGKILL | 100 random kill-points | **167/167 kill-points, zero acked loss** on the final design (batches of 64 + 91 + 12; each round = fresh 3-process cluster, concurrent `ack=quorum` item+signal writers via leader AND follower gateways, SIGKILL at timings spread 120–598ms, then BOTH proofs on the promoted max-applied survivor: `max(acked seq) <= survivor applied_events` and per-item `/search` content probes). An earlier 100/100 run validated the superseded ack-holding design before it was replaced. |
|
||
| Quorum throughput ≥ 50% of p1's leader-ack number | ≥ 2,267/s local-equivalent | **3,600 quorum signal-writes/s within SLO** (writes mix, `wal.batch_timeout_ms: 2`, knee not reached; p50 ~45ms at 1.5k, ~370ms at 3.6k from concurrency-limit queueing; zero errors; replication lag ≤3 events at ramp end) = **79%** of p1's 4,534/s. Same-day leader-ack baseline on this machine: 5,971/s (10ms batch timeout). Ref-A (Linux) runs remain blocked on k3s access — same as p1/p2. |
|
||
|
||
Tier-3 partition semantics (real TCP severs): one follower down → quorum
|
||
commits via the other; both down → fast 503 naming the laggards while
|
||
`ack=leader` writes flow uninterrupted; healed → quorum recovers within the
|
||
breaker window. (`mp_quorum_writes_gate_and_recover_under_partition`.)
|
||
|
||
### Performance findings the gate forced (all shipped)
|
||
|
||
1. **Thread-per-quorum-wait collapses at 1k rps**: parked Condvar waiters
|
||
exhausted tokio's 512-thread blocking pool; the group-commit completions
|
||
that advance the index queued BEHIND them → 0 OK writes, 40s latencies.
|
||
Fix: async watch-channel bridge (one publisher thread, zero per-waiter
|
||
threads).
|
||
2. **Holding ship acks until durable apply death-spirals**: ack-wait ×
|
||
window throttles ships to a gap-parked follower → it lags further → all
|
||
quorum writes time out. Fix: follower-pushed `ReportApplied`, ship acks
|
||
stay instant.
|
||
3. **Per-record blob fsyncs cap item apply at ~100/s** (m11p2 flaw): fixed
|
||
with batched blob apply + writer group fsync → seeding 22× faster, item
|
||
bursts no longer stall the quorum frontier.
|