tidaldb/docs/roadmap-to-cluster.md

# Roadmap to an Enterprise-Grade Cluster

**Status:** ADOPTED as M11 — **ALL NINE PHASES COMPLETE.** m11p1–m11p5 ✅, m11p7 ✅, m11p8 ✅, m11p9 ✅ (2026-06-13 — continuous correctness: new fault classes (disk-full / slow-fsync / asymmetric partition) as REAL faults behind a production-compiled-out `fault-injection` feature, first-class invariant checkers (no-acked-loss ledger, decay parity, single-leader-per-term, monotonic counters), a `tidal-stress` soak with PASS/FAIL regression gates + JSON summary, a Woodpecker cron nightly chaos+soak pipeline beside the push release gate, and a guarantee→test traceability matrix — closes **G-C** apparatus; the 30-day-green calendar accrues nightly) · **m11p6 ✅ (2026-06-13 — sharding × replication + rebalancing: ONE hash-routed + replicated write surface across S shard groups, each a replication group at RF with its own elected leader; per-group `/cluster/shards/{id}/transfer`+`/replicas` rebalancing verbs and a `?shard=` admin selector propagated through intra-group forwards; tier-3 3-node × 3-shard × RF=3 kill-node exit gate green — only the dead node's leaderships move (<10s), reads never stop, zero acknowledged loss per group; ≥5,000/s + ≥2.5× single-shard scaling remain Ref-A/k3s-pending, local 3×3 sustains 3,000 quorum writes/s at 0% error with the engine at ~30% CPU)** · v1.0 now waits only on the 30-day-green nightly calendar + the standing Ref-A/k3s throughput re-runs · **Date:** 2026-06-10 · **Baseline evidence:**
[stress-test-thepeach.md](ops/stress-test-thepeach.md), [cluster runbook](runbooks/cluster.md),
[ROADMAP M8 Known Gaps](planning/ROADMAP.md) (G4/G5/G6), live k3s deployment (3 regions × 2-vCPU pods).

This is the gap analysis and phase plan from today's **experimental** multi-process
cluster (m8p10 — real process isolation, leader-durable writes, operator-driven
failover) to a cluster a paying customer could run as a system of record for
signals: **quorum-durable, self-healing, horizontally scalable, secured,
observable, and continuously chaos-tested.** Everything in §1 is measured or
code-verified, not aspirational; every phase in §4 ends in a testable exit gate.

---

## 1. Where we are today (evidence)

### 1.1 Measured baselines (2026-06-10, live 3-region k3s cluster, cpu-limit `"2"`/pod)

| Path | Measured | Notes |
|---|---|---|
| Embedded engine signal write | **~82 ns** (~12M/s/core) | criterion `scale.rs`; the engine itself is not the problem |
| Embedded RETRIEVE p99 | **152 µs** @ 1M items | scale baselines |
| Cluster `/feed` (ranked read) | **p99 8–15 ms** at 100→4,000 rps | never degraded in any run |
| Cluster `/search` | p99 3–11 ms | includes `reload_text_index` per request |
| **Replicated `/signals` (leader path)** | **~90 signals/s ceiling** | ok/s pinned at ~75 regardless of offered rate (100→4,000 rps); ~178–210 ms per write |
| **`/sharded/signals` (unreplicated)** | **3,669 signals/s, 0 errors, ~27% cluster CPU** | knee NOT reached; single generator became the limit |
| Cross-region replication lag | 0–3 events under all load | SLA <2 s; measured ~110–133 ms p99 (runbook) |
| Manual failover (promote→write) | ~31–34 ms | SLA <10 s |
| Overload behavior | **Graceful**: HTTP 429 shed, zero 5xx, **zero pod restarts**, no acknowledged-write loss observed | the strongest enterprise trait we have today |
| Aggregate HTTP ceiling | ~5k rps connection-establishment wall (conntrack/accept), cluster CPU idle | networking, not engine |

Ballpark peers on small nodes (context, not benchmarks we ran): etcd ≈10k
fsynced consensus writes/s; CockroachDB low-thousands/node; Cassandra QUORUM
10k+/node. **The replicated write path is 1–2 orders of magnitude below the
enterprise floor; everything else is competitive or better.**

### 1.2 What already exists and is worth keeping

- Real per-process region isolation; kill the leader → survivors keep serving (proven live).
- A real WAL relay over gRPC (`tidal-net` `WalShipping/ShipSegment`) with BLAKE3-verified
  segments, contiguous-frontier apply (no gap-swallowing), per-peer circuit breakers.
- Honest, documented durability contract (runbook §8) — rare and valuable; we extend it, not retrofit honesty.
- Operator verbs that work and are runbook-verified op-for-op (`promote`, `partition`, `heal`, `reconcile`),
  CRDT/LWW convergence for user-scoped hard-negatives, HLC with skew injection for tests.
- Sharded scatter-gather reads with degraded-not-failed semantics and shard-dedup merge.
- Tier-3 chaos/lifecycle/runbook suites over **real OS processes with real TCP-relay partition injection** —
  the scaffolding §4/p9 industrializes.
- Graceful backpressure end to end (write-pool 429 + engine WAL-channel 429 + breaker isolation).

### 1.3 Gap inventory → phase map

| Dimension | Today | Enterprise bar | Phase |
|---|---|---|---|
| Replication throughput | ~90/s: synchronous, sequential, per-request ship; 2-worker pool | Batched/pipelined, async; ≥2k/s on today's hardware | **p1** |
| Replication unity | Signals ride the WAL; items/embeddings ride a best-effort HTTP side channel (source of both live bugs we fixed: marker-gated broadcast, 401'd heal backfill); heal is O(items) | **One replicated log** for all replicated data; heal = log catch-up | **p2** |
| Durability | 2xx = leader fsync only; follower ship best-effort | `ack=quorum` mode: 2xx = majority-durable; zero acknowledged loss on any single-node kill | **p3** |
| Availability | Operator-driven promote; **restarted node re-reads topology and believes the static leader again** (observed live: us-east rejoined claiming leadership while eu-west led — split-brain seed) | Failure detector + automatic election + **term fencing** + durable leadership | **p4** |
| Membership | **✅ p5:** DNS/seed discovery (`grpc_addr` advertised + DNS-resolved, `grpc_bind` for the local bind), kind-4 membership records on the one log, online `/cluster/join`+remove with learner→voter auto-promotion, `FetchSnapshot` snapshot+stream catch-up, self-healing reseed, `k8s/cluster/` one-StatefulSet reference | Dynamic membership, DNS/seed discovery, online add/remove/replace with snapshot+stream catch-up | **p5 ✅** |
| Write scaling | EITHER replicated (1 leader for everything) OR sharded (no replication); shards conflated with regions | **Sharding × replication**: shard groups with RF and distributed leaders; rebalancing | **p6** |
| Security | Plaintext gRPC replication (mTLS config exists, unexercised); one static bearer shared by all; internal marker as trust signal; no audit log | mTLS default + rotation, authenticated inter-node calls, admin audit, at-rest story | **p7** |
| Observability | **Cluster mode has no `/metrics` listener at all**; no request-id propagation on cluster routers; lag accounting wrong after promote (ShardId(0)-keyed); leader's own status row reads 0 | Per-node Prometheus metrics, tracing across forward/ship hops, truthful status, dashboards + alerts | **p8** (starts in p1) |
| Ops lifecycle | Manual heal loops (first heal can no-op into an open breaker); per-node backup only; rolling-upgrade suite exists but not a CI gate | Coordinated cluster backup/restore + PITR; upgrade-under-load as a release gate; self-driving heal | **p8** |
| Correctness assurance | Tier-3 suites are opt-in (`cluster-e2e` feature), run manually | Nightly chaos + soak in CI with invariant checkers; 30-day green before GA | **p9** |

### 1.4 Live incidents from this deployment that became requirements

1. **Restart amnesia / split-brain seed (→ p4).** After SIGKILL-failover, the
   restarted ex-leader booted from its static topology believing `leader=us-east`
   while the cluster had promoted eu-west; an operator re-promote reconciled it.
   With writes flowing through both gateways this is a dual-leader window.
   Requirement: durable terms, fencing on ship/forward, rejoin-as-follower.
2. **The HTTP side channel is where the bugs live (→ p2).** Both correctness bugs
   found live were in item/embedding broadcast, not the WAL relay: (a) forwarded
   writes never broadcast (terminated on the internal marker instead of
   leadership), (b) heal's backfill carried no bearer and 401'd forever. The WAL
   path was flawless throughout. Conclusion: replicate everything through the log.
3. **Breaker eats the first heal (→ p8).** After a partition window the per-peer
   breaker (threshold 5 / reset 30 s) is open; the first `/cluster/heal` no-ops.
   Runbook says "re-issue until lag 0" — an operator footgun; the server should
   drive heal-until-converged itself.
4. **The 178 ms ship (→ p1).** A LAN gRPC hop plus group-commit fsync does not
   cost 178 ms. Something structural (per-call setup, flow control, synchronous
   follower apply, sequential per-peer sends) is being paid per request. Profile
   before redesigning; the fix is likely cheaper than it looks.

---

## 2. Definition of done — the enterprise guarantees

Each guarantee is a **testable claim** with a named owner-test by GA (p9 wires any
that don't exist earlier). These are the bar; the phases are the path.

- **G-D (Durability).** With `ack=quorum`, a 2xx response means the write is
  durably applied on a majority of the shard's replicas. SIGKILL any single node
  at any moment under load: **zero acknowledged-write loss**, proven by a ledger
  checker that replays client acks against post-recovery state.
- **G-A (Availability).** Loss of any single node: reads continue uninterrupted;
  writes resume automatically in **<10 s p99** with no operator action and no
  dual-leader window (fencing proven under partition + restart chaos).
- **G-S (Scalability).** Write throughput scales with shard count at RF=3
  (≥2.5× from 1→3 shards on the same hardware). Reads scale with replicas.
- **G-E (Elasticity).** Nodes can be added/removed/replaced online; a new replica
  catches up via snapshot + WAL stream; p99 impact bounded (<2× for <60 s).
- **G-Sec (Security).** All inter-node links mTLS; replication endpoints
  authenticated (no implicit trust of the network); keys/certs rotate without
  downtime; admin verbs audit-logged.
- **G-O (Observability).** Every node exports Prometheus metrics; a dashboard
  answers "what is happening / what is wrong" without reading code; alerts on
  the golden signals + replication lag + commit-index stall + election churn.
- **G-Op (Operability).** Rolling upgrade with N/N+1 version skew under load is a
  CI-verified release gate; coordinated backup/restore (and WAL-archival PITR)
  drilled and timed; runbooks executable by a non-author.
- **G-C (Continuous correctness).** Nightly chaos (partitions, crashes, skew,
  disk faults) + soak (tidal-stress) green for 30 consecutive days before GA.

---

## 3. Reference environments and performance targets

Two named environments so numbers stay comparable end to end:

- **Ref-A (continuity, CI-able):** the current k3s deployment — 3 × (2 vCPU /
  2 GiB, local-path PV). All measured baselines in §1.1 are Ref-A. Phase exit
  gates are stated against Ref-A so they're re-runnable with `tidal-stress`.
- **Ref-B (enterprise reference):** 3–9 × (8 vCPU / 16 GiB, NVMe). GA marketing
  numbers come from Ref-B; claiming "enterprise" on shared 2-core pods is noise.

| Target | Ref-A (gate) | Ref-B (GA target) |
|---|---|---|
| Replicated signal writes, `ack=leader` | ≥2,000/s (p1) | ≥25,000/s |
| Replicated signal writes, `ack=quorum` | ≥1,000/s (p3) | ≥15,000/s |
| 3-shard × RF=3 quorum writes | ≥5,000/s (p6) | ≥50,000/s |
| Ranked read p99 (under write load) | ≤50 ms | ≤20 ms @ 10k reads/s |
| Auto-failover (detect→writes resume) | <10 s p99 (p4) | <5 s p99 |
| Replication lag p99 (sustained load) | ≤2 s | ≤500 ms |
| New-replica catch-up (100k items) | ≤5 min (p5) | ≤1 min |

---

## 4. The phases (proposed M11, `m11p1`–`m11p9`)

Ordering rule: **make the one log fast and unified before making it quorum; make
it quorum before electing leaders over it; fix membership before sharding it;
secure and industrialize continuously.**

### m11p1 — Replication performance floor (size: M) — ✅ COMPLETE (2026-06-11)

> **As built.** The ~178ms/write was structural, exactly as suspected — three
> stacked serializations: (1) the relay's seqno lock held across the WAL
> group-commit wait, so every writer paid a SOLO `batch_timeout` + fsync
> (~11ms × 2-8 pool workers ≈ the measured ~90/s ceiling and, with the
> 16-deep queue, the ~178ms latency); (2) the per-request sequential per-peer
> eager ship on the request path; (3) — unanticipated by this plan — the
> follower applying every received segment through its own solo group-commit,
> capping follower apply at ~`events-per-segment / fsync-cost` (~1.8k/s
> measured), which made followers lag unboundedly once the leader got fast.
> Fixes: staged two-phase writes (`stage_write` atomic under the lock in µs,
> fsync waited outside it — concurrent writers share group commits);
> per-peer windowed batch senders shipping only the leader's durable
> frontier, with relay poisoning on fsync failure; follower backlog draining
> (`try_recv_segment`) applied through one shared group commit in
> range-disjoint groups. **Measured (3-process localhost, release build,
> thepeach mix): 4,534 replicated signal-writes/s within SLO on the ramp
> (~50×; knee ~5.5k signals/s), and 2,739 signals/s sustained for 10
> minutes (1.65M writes, 0.35% errors, zero shed) with replication lag
> never exceeding 103 events (~40ms).** Group commit measured at 58 events/fsync mean;
> leader fsync (macOS F_FULLFSYNC) 7.4ms mean with a 10-50ms tail —
> `tidaldb_cluster_wal_fsync_us` now answers the "profile first" question
> per deployment. The ≤50ms write-p99 sub-gate tracks the platform fsync
> floor and validates on Ref-A's Linux fdatasync (re-run pending infra
> access). Knobs shipped: `replication.{batch_max_events,window,retry_ms}`,
> `wal.{batch_size,batch_timeout_ms}`, per-region `metrics_addr`.

**Goal:** the existing leader-durable path stops being the embarrassment line:
≥20× today's ceiling with no semantic change.

- **Profile first.** Instrument the ship path (this phase ships the first
  `tidaldb_cluster_*` metrics: fsync latency, ship RTT, queue depths, breaker
  state) and explain the measured ~178 ms/write. Suspects: per-call channel
  setup, tonic flow-control, synchronous follower-side apply in the RPC, the
  sequential per-peer loop in `relay.write_and_ship`, group-commit interaction.
- **Decouple ack from ship.** The client ack already only promises leader
  durability — so return at group-commit fsync; move shipping fully off the
  request path into per-peer outbound queues drained by dedicated senders
  (the relay/redeliver machinery already exists; make it the only path).
- **Batch + pipeline.** Ship WAL batches (not per-request payloads) with a
  windowed in-flight budget per peer (N batches outstanding, acks advance the
  window) instead of one blocking unary call at a time.
- **Rework the write pool.** `/signals` no longer needs a blocking gRPC worker
  per request; the pool sizes from CPU for fsync/apply work only. Keep 429
  semantics (bounded queue) — the graceful shed is a feature.
- **Tune group commit** (batch 100 / 10 ms today) against measured fsync cost on
  Ref-A's local-path PV; make both knobs config.
- Non-goals: no ack-mode change, no membership change.
- **Exit gate (Ref-A, `tidal-stress` leader path):** ≥2,000 signals/s OK at
  p99 ≤50 ms, error rate <1%, lag p99 ≤2 s sustained 10 min; replication metrics
  visible in Prometheus.

### m11p2 — One replicated log (size: L)
**Goal:** every replicated mutation rides the WAL relay; the HTTP broadcast side
channel (items/embeddings) is deleted.

- Add item-metadata and embedding records to the WAL event model (they're
  config-like, low-rate — the log absorbs them trivially; 64 MiB segment ceiling
  is ample for 1536-dim vectors).
- Followers apply items/embeddings from the log; leader broadcast code and the
  O(items) heal re-broadcast are removed (heal becomes pure log catch-up).
- Implement **`StreamSegments`** (declared, unimplemented) as the catch-up path:
  a follower that reports applied=N gets a stream from N+1; the in-memory
  `batch_log` dependency for recovery goes away (read back from the durable WAL).
- Hard-negatives stay CRDT (user-scoped, commutative) — document the split:
  "log = totally-ordered global data; CRDT = per-user convergent data".
- **Exit gate:** chaos tests for both 2026-06-10 bug classes pass by
  construction (no marker-gated fan-out to regress, no authed side POSTs);
  item written anywhere is readable everywhere ≤2 s p99; heal of a 10-minute-down
  follower with 100k items completes via stream without O(items) HTTP traffic.

> **✅ COMPLETE (2026-06-11), as built:** the WAL itself became the one log —
> the group-commit writer feeds every fsynced batch (signals = kind-0,
> items/embeddings = kind-1/2 blob records, journaled BEFORE storage) to a
> bounded ship feed whose bytes ship verbatim; stream seqnos are WAL seqnos
> and survive restarts. Catch-up is FOLLOWER-PULLED `StreamSegments` over the
> durable segments (gap-triggered, boot-time self-heal, heal-nudged), with
> ship acks piggybacking the follower's applied seqno; promote carries a
> persisted stream baseline so a new leader's pre-stream history never
> replays. The HTTP broadcast, O(items) heal backfill, and the m11p1 relay
> bookkeeping are deleted; multi-process mode now requires `--data-dir`.
> Exit-gate mechanics proven by tier-3
> `mp_items_ride_the_log_and_catchup_stream` (items via leader AND follower
> gateways converge everywhere; a follower stopped through item + embedding +
> signal writes restarts and converges via the stream with no operator verb
> and no HTTP item traffic). The 100k-item scale figure remains a Ref-A run
> (same k3s access blocker as p1's p99 sub-gate). Details:
> [planning/milestone-11/phase-2.md](planning/milestone-11/phase-2.md).

### m11p3 — Quorum-acked writes (size: L) — closes ROADMAP gap **G4**
**Goal:** an opt-in durability contract a system of record can sit on.

- `ack=leader|quorum` (deployment default + per-request override header).
- Follower ship responses carry **durably-applied seqno** (fsync'd, not just
  received); leader maintains per-peer durable high-water marks and a
  **commit index** = quorum floor; `ack=quorum` responses gate on the commit
  index passing the write's seq. Pipelined (batch-level acks), never one-at-a-time.
- Timeout semantics: quorum not reached in budget → retryable 503 naming the
  laggard; the write may still commit (document at-least-once + dedup guidance;
  evaluate optional client idempotency keys for signal accounting — open
  question, decide in-phase).
- Rewrite runbook §8 from "honest about what we don't promise" to "here's the
  knob and its cost".
- **Exit gate:** SIGKILL the leader under `ack=quorum` load → ledger checker
  proves zero acknowledged loss across 100 random kill-points; quorum throughput
  ≥50% of p1's leader-ack number (≥1,000/s Ref-A).

> **✅ COMPLETE (2026-06-11), as built:** durable freshness is follower-PUSHED
> (`ReportApplied` once per apply round — decoupled from ship acks after the
> ack-holding design measurably death-spiraled under load) into a
> leadership-scoped commit index (k-th largest durable mark); `ack=quorum`
> waits are fully async (watch-channel bridge — thread-per-wait exhausted
> the blocking pool); timeouts are retryable 503s naming laggards; every
> cluster write returns `x-tidal-seq`. The gate also surfaced and fixed an
> m11p2 flaw: follower blob applies paid one fsync per item — now batched
> through shared group commits (22× item-seed speedup). Idempotency
> question DECIDED: no new machinery; at-least-once + dedup guidance in
> runbook §8. Evidence: 100/100 kill-points zero acked loss; 3,600 quorum
> writes/s within SLO locally = 79% of p1's leader-ack figure (Ref-A run
> still blocked on k3s access). Details:
> [planning/milestone-11/phase-3.md](planning/milestone-11/phase-3.md).

### m11p4 — Failure detection, election, fencing (size: XL) — ✅ COMPLETE (2026-06-12) — closes ROADMAP gap **G5**

> **✅ COMPLETE, as built:** purpose-built election-only Raft (pre-vote +
> check-quorum + fenced transfer) over the existing transport — no raft
> crate; the WAL is the log, and an elected leader's FIRST entry is a
> replicated kind-3 **term marker**, so Raft's `(lastLogTerm, lastLogIndex)`
> derives from one fsync stream (frontiers compared in the last joined
> term's STREAM numbering — a reseeded node's local numbering diverges).
> Hard state `(term, voted_for)` in `data_dir/election_state` (corrupt =
> refuse boot; lost-with-WAL = forced follower), fsynced before any vote or
> claim leaves. Term fencing on every RPC; the commit index folds only
> activation-term reports; a restarted ex-leader boots follower from durable
> state (§1.4-1 closed by construction). Divergent suffixes QUARANTINE
> (reseed is recovery; p5 snapshots automate). Promote = fenced transfer
> that refuses lagging targets — the runbook's "max-applied survivor" rule
> is now protocol. `auto_election: false` preserves the operator posture.
> **Exit gates (local tier-3, 3 processes): elections in 0.59–0.98 s under
> ack=quorum kill loads with zero acked loss across random kill points; a
> restarted partitioned ex-leader accepted 0 writes; 7 link flaps converged
> to one leader at term ≤ 15.** Details:
> [planning/milestone-11/phase-4.md](planning/milestone-11/phase-4.md).

> **✅ Shipped increment (2026-06-11, d0a52e4) — catch-up self-healing + WAL segment versioning.**
> Failed `StreamSegments` pulls now arm a 30 s timer retry; re-arm-on-skip is the load-bearing
> liveness fix — without it a pull that fails in an idle cluster never re-fires and the follower stays
> permanently lagged (root cause: 2026-06-11 p3 rollout left both followers stuck at lag=136,507 with
> zero writes to trigger a retry). WAL segments carry a `TSEG` 8-byte header (magic + version byte +
> reserved); pre-p4 headerless segments are implicit v0, no migration; unknown format →
> `WalError::SegmentFormatUnknown` with no repair or truncation of foreign files.
> `FAILED_PRECONDITION` is the typed refusal for unservable catch-up. Regression gate
> (ops/stress-test-p4-regression.md): timer-only catch-up confirmed (lag=0 in 10 s with zero writes);
> p3 gate (≥1,000 quorum/s, p99 ≤50 ms, <1% error) holds through p4. The `Heartbeat` RPC and
> `ControlPlane::record_shard_heartbeat` already exist and health-track node liveness — they become
> the failure-detector input for the election work below.

**Goal (main p4 work, pending):** "a machine died" is a non-event, not a runbook page.

- **Replicated metadata state** (term/epoch, leader id, membership): Raft-style
  election over the existing transport — the quorum machinery from p3 is the
  prerequisite. The data plane stays the WAL relay; only the small control state
  is consensus-managed.
- Failure detector on the existing `Heartbeat` RPC (timeout + jitter; tunable).
- **Fencing everywhere:** terms stamped on ship/forward/promote; stale-term
  ships rejected by followers; a restarted node **rejoins as follower** and
  learns leadership from the metadata state — never from the static topology
  file (kills incident §1.4-1).
- Leadership is durable locally (survives restart) and lease-bounded; manual
  `/cluster/promote` becomes a fenced transfer (drain for maintenance), not the
  availability mechanism.
- **Exit gate (chaos, Ref-A):** kill the leader under load → writes resume
  <10 s p99 with zero operator action and zero acked loss; partition the old
  leader away mid-election and let it restart → it cannot accept a single write
  (fencing proven); election churn under flapping links bounded (no livelock).

### m11p5 — Membership, discovery, elasticity (size: L)
**Goal:** nodes are cattle; topology is data, not files.

- Resolve `grpc_addr` via DNS (drop the `SocketAddr::parse`-only constraint) and
  support **seed-based join**: a new node contacts any seed, gets the membership
  + topology from the metadata state. The per-pod static-ClusterIP topology
  ConfigMap hack dies.
- Conf-changes (add/remove/replace) through the p4 metadata consensus; joiners
  catch up via **snapshot + StreamSegments** (snapshot = the engine's existing
  checkpoint artifacts, shipped as a stream).
- Kubernetes reference becomes one StatefulSet + headless Service peer
  discovery, PDB, topology-spread; rolling node replace is `kubectl delete pod`.
- **Exit gate:** scale 3→5→3 online under load: joiner catches up ≤5 min
  (100k items), serves quorum after; p99 impact <2× for <60 s; zero loss.

> **✅ COMPLETE (2026-06-12), as built:** `grpc_addr` became an **advertised**
> address (hostname or IP) split from an optional `grpc_bind`; `tidal-net`'s
> peer map retyped `SocketAddr → String` so hyper re-resolves DNS on every
> reconnect (the pod-rescheduled-onto-a-new-IP case the `SocketAddr::parse`
> constraint made impossible). A new server-streaming `FetchSnapshot` RPC ships
> `create_backup` as a manifest + BLAKE3-verified file chunks (term-stamped,
> term-fenced, identify-or-refuse) into a **boot-time** staging-dir install
> (every crash window an idempotent redo); reseed is self-healing — a typed
> `x-tidal-catchup: snapshot-required` refusal (or the p4 quarantine) latches a
> durable `reseed_required` marker that runs on the next boot and clears the
> divergence gauge, no `wipe_data_dir`. Membership is data on the one log: a new
> **kind-4** `MembershipRecord` (beside kind-0/1/2/3) folds into a
> `ClusterMembership` cell; `POST /cluster/join` appends a quorum-committed
> Learner that auto-promotes to Voter; conf-changes are Raft single-server with
> an activation-record re-append closing the vacuous-gate + baseline-jump
> blockers. The **three-way term-join rule** (`own < prev_log` → latch reseed)
> closes p4's pre-baseline-history carried hazard; the even-n `majority()` fix
> (`(n+1).div_ceil(2)+1`) closes a latent dual-leader bug. Seed-join boot
> (`--seed`/`--advertise-*`/`--metrics`, knob file still required) +
> `k8s/cluster/` (one StatefulSet, headless Service, PDB, topology-spread) make
> `kubectl delete pod` the node-replace drill. Exit-gate mechanics proven by
> tier-3 `cluster_membership.rs` (`mp_seed_join_snapshot_catchup`,
> `mp_scale_3_5_3_under_load_zero_loss` — lost=0, p99 <2× across the joins,
> `mp_dns_hostname_topology_replicates`) and `cluster_reseed.rs`. The 100k-item
> catch-up and the k8s pod-reschedule drill remain Ref-A line items (k3s access
> pending — the standing M11 caveat; the 5000-item localhost catch-up at 26.4s
> has large headroom under the ≤5-min budget). Details:
> [planning/milestone-11/phase-5.md](planning/milestone-11/phase-5.md).

### m11p6 — Sharding × replication + rebalancing (size: XL)
**Goal:** writes scale horizontally **without** giving up replication — the
current "replicated XOR sharded" split ends.

- Shard groups: keyspace hash-split (reuse `ShardRouter`) into S shards, each a
  replication group at RF (default 3) with its own WAL/relay/commit-index and
  its own elected leader; leaders balanced across nodes.
- Regions stop being shards: topology = nodes × shard-replicas, with optional
  zone/region labels for placement and read affinity.
- Routing: any gateway routes by entity hash to the shard leader (write) or a
  local/nearest replica (read); scatter-gather already merges cross-shard reads
  with degraded semantics — now over shard groups.
- Rebalancing: shard move = snapshot + stream + fenced cutover; operator-
  triggered first, automatic (rate-limited) later.
- **Exit gate (Ref-A, 3 shards × RF=3):** ≥5,000 quorum signals/s (≥2.5×
  single-shard p3); kill any node → only its shard-leaderships move (<10 s),
  reads never stop; `tidal-stress` sharded-vs-replicated comparison collapses
  into one path.

### m11p7 — Security hardening (size: M)
**Goal:** the cluster stops trusting the network.

- **mTLS default** on gRPC replication (the `grpc_tls` config exists — exercise
  it, flip plaintext to an explicit `insecure: true` with a loud startup WARN);
  cert + bearer **rotation without restart** (file/secret watch).
- Authenticated inter-node HTTP: per-node identity (client certs or signed
  internal tokens); the `x-tidal-internal` marker remains a routing hint only
  (it already isn't an auth bypass — keep it that way by test).
- Audit log for admin verbs (promote/heal/partition/conf-change) with principal,
  term, and outcome; at-rest encryption documented (delegate to volume
  encryption, or in-engine as stretch); per-principal rate limits (the engine's
  rate limiter exists — wire it to the HTTP layer).
- **Exit gate:** reference deployment has zero plaintext inter-node links;
  rotation under load drops zero requests; a foreign pod on the cluster network
  can neither ship segments nor call internal routes (negative tests).

> **As-built (m11p7, 2026-06-13 — COMPLETE):** gRPC is served over a custom
> `tokio-rustls` acceptor (tonic 0.12 caches a fixed `ServerConfig` with no
> resolver hook, so it cannot hot-swap — the custom acceptor is the only zero-drop
> path) fed a `DynamicCertResolver` (`ArcSwap<CertifiedKey>`); mTLS preserved
> exactly (`WebPkiClientVerifier`, ALPN h2), plaintext → explicit `insecure:true`
> + loud WARN. Rotation = content-hash polling (catches k8s `..data` symlink
> swaps inotify misses) → atomic resolver swap (in-flight sessions untouched) +
> peer-channel rebuild → **zero dropped requests under load** (verified). Inter-
> node HTTP reuses the SAME resolver for server TLS + dials `https` with the
> cluster CA; per-node identity is a signed `x-tidal-node-token` (keyed-BLAKE3 MAC
> under a shared cluster key — no new crypto dep, since requiring HTTP client
> certs would break external bearer clients), and the `x-tidal-internal` marker is
> honored only from a verified sibling (marker-without-token → 403). Admin verbs
> audit to a `tidal_audit` target + optional `TIDAL_AUDIT_LOG` JSONL (operator-leg
> only). Per-principal rate limit reuses the engine `RateLimiter` (nodes exempt).
> All opt-in (`grpc_tls`/cluster key/env) ⇒ pre-m11p7 behavior byte-for-byte. See
> [milestone-11/phase-7.md](planning/milestone-11/phase-7.md).

### m11p8 — Observability + operations (size: M; **starts inside p1**)
**Goal:** operable by someone who didn't build it.

- Complete the `tidaldb_cluster_*` metric set (lag, commit index, ship
  latency/queue, breaker, elections, forwards, write-pool, fsync) on a cluster
  `/metrics` listener (**today: none**); Grafana dashboard + alert rules shipped
  beside the existing standalone ones.
- Request-id propagation + tracing spans across forward/broadcast/ship hops
  (the cluster routers currently skip the request-id layer).
- Truthful status: fix post-promote lag accounting (ShardId(0) keying) and the
  leader's own applied row; `/cluster/status` becomes the single pane.
- **Self-driving heal:** server-side heal-until-converged (retry through breaker
  resets) so "re-issue heal until lag 0" leaves the runbook (incident §1.4-3).
- Coordinated backup/restore: cluster-consistent snapshot manifest across shard
  groups + WAL archival hook for PITR; timed restore drill in the runbook.
- Rolling upgrade: version handshake on ship/forward, N/N+1 skew documented;
  `mp_rolling_upgrade_no_loss_no_stall` promoted to a release-gate CI job.
- **Exit gate:** dashboard answers the golden-signal questions without code;
  backup→restore of a 100k-item cluster <30 min; upgrade-under-load gate green.

> **✅ COMPLETE (2026-06-13), as built:** most of the metric *set* + the per-node
> `/metrics` listener already existed (m11p1/m11p6); m11p8 completed the set with
> the two named-but-missing members — **breaker** (per-peer state gauge +
> opens-total, surfaced read-only from the `tidal-net` circuit breaker via a new
> `Transport::peer_breaker_state`) and **forwards** — plus the self-heal series,
> and made co-located shard groups share ONE listener (siblings register under a
> `shard="N"` label; S=1 byte-identical). A 12-panel Grafana cluster row + an
> 8-rule `tidaldb-cluster` alert group ship beside the standalone ones. Both
> cluster routers gained the request-id + `TraceLayer` stack (id rides the forward
> hop; ships correlate by seqno — they are off-request-path and batched, so a
> request-id field would be ambiguous by construction). Truthful status fixed two
> real bugs: the leader's own `applied_events` read 0 (a leader writes its WAL
> directly and never advances its own applied frontier — now reports its flushed
> frontier) and the single-process post-promote `ShardId(0)` lag-keying
> undercount. **Self-driving heal** (closes §1.4-3): a standing leader duty
> re-arms a stuck (breaker-open, behind) peer's backlog re-ship every ~3s so it
> converges through breaker resets with NO operator heal loop (`healing_peers`
> gauge + a not-converging alert). **PITR**: a `wal.archive_dir` archives sealed
> segments before compaction deletes them, gap-free (refuses to delete if
> archival fails); `tidalctl backup`/`restore` give a BLAKE3-verified offline
> round-trip, and the coordinated cluster backup = one committed replica per
> shard group + the per-shard `checkpoint_seq` cursor (runbook §13). **Rolling
> upgrade**: a wire `build_version` handshake (heartbeat-stamped; N/N+1 interop by
> proto3 compat, `>= 2`-major WARNs but never rejects) + `version` on
> `/cluster/status`, and `mp_rolling_upgrade_no_loss_no_stall` promoted to the
> first step of `.woodpecker.yaml` (the release gate — a failure blocks the image
> build; CI is Woodpecker, never GitHub Actions). Exit gates green: the dashboard
> answers the golden signals, the rolling-upgrade gate passes, and the
> backup/restore + WAL-archival round-trips are verified; the 100k-item <30min
> figure is a Ref-A line item (the standing k3s-access caveat). Details:
> [planning/milestone-11/phase-8.md](planning/milestone-11/phase-8.md).

### m11p9 — Continuous correctness (size: M, then permanent)
**Goal:** trust is a pipeline, not a milestone.

- Nightly CI: the tier-3 chaos suites (partition relay, crash, clock-skew via
  `TIDAL_HLC_SKEW_MS`, rolling upgrade) + new fault classes (disk-full,
  slow-fsync injection, asymmetric partitions) against a Ref-A cluster.
- Invariant checkers as first-class: no-acked-loss ledger replay, per-entity
  monotonic counters, cross-replica decay parity (1e-6), membership safety
  (single leader per term per shard). Jepsen-style external harness as stretch.
- Nightly `tidal-stress` soak (1× 100k-DAU model, 1 h) with regression gates on
  p99/throughput; results archived for trend lines.
- **Exit gate (GA):** every guarantee in §2 maps to a named automated test;
  nightly suite green 30 consecutive days.

> **✅ COMPLETE (2026-06-13), as built:** the new fault classes are REAL faults
> at the storage boundary, not engine flags — a `fault-injection` cargo feature
> (NOT default, never passed by the production image build, so compiled OUT of the
> shipped binary) adds WAL hooks for **slow-fsync** (`TIDAL_FAULT_FSYNC_DELAY_MS`,
> sleep before the durable fsync) and **disk-full** (`TIDAL_FAULT_DISK_FULL_AFTER_BYTES`,
> a real `ENOSPC` once cumulative segment bytes cross the threshold);
> **asymmetric partition** reuses the harness's directed-edge proxies (sever
> inbound-only). The invariant checks the m11p3/m11p4 gates carried inline became
> a first-class `support/invariants.rs` (no-acked-loss `AckLedger`, decay/feed
> parity, `assert_single_leader_per_term`, monotonic frontiers) — the m11p3 ledger
> gate now consumes it (re-verified). A new tier-3 `cluster_faults.rs` proves all
> three fault classes degrade gracefully with zero acked loss (**4/4 green**:
> disk-full follower froze then recovered to parity; slow follower lagged then
> converged; both-slow → honest retryable 503 naming laggards while `ack=leader`
> kept 204'ing; asymmetric partition → leader held, single-leader-per-term, no
> split brain). `tidal-stress` gained a `--json-summary` + `--max-p99-ms` /
> `--max-error-pct` / `--fail-on-knee` regression gate (non-zero exit on breach —
> verified vs a real server). The nightly is a Woodpecker CRON pipeline (chaos
> suites with elevated kill-points + the gated soak) beside the push release gate
> — Woodpecker, never GitHub Actions. Every §2 guarantee maps to a named test in
> [guarantee-traceability.md](planning/milestone-11/guarantee-traceability.md);
> G-S's owner-test is the (in-progress) m11p6 throughput gate. The remaining GA
> half — nightly green for **30 consecutive days** — is a calendar criterion the
> pipeline now produces (out-of-session, like the Ref-A k3s runs). Details:
> [planning/milestone-11/phase-9.md](planning/milestone-11/phase-9.md).

---

## 5. Sequencing, sizing, releases

| Phase | Depends on | Size | Closes |
|---|---|---|---|
| p1 perf floor | — | M | §1.4-4, throughput floor |
| p2 one log | p1 | L | §1.4-2, StreamSegments, O(items) heal |
| p3 quorum acks | p1, p2 | L | ROADMAP **G4** |
| p4 election + fencing | p3 | XL | ROADMAP **G5**, §1.4-1 |
| p5 membership | p4 | L | static-IP topology, elasticity |
| p6 shards × RF | p3, p5 | XL | write scaling |
| p7 security | p2 (stable surface) | M | network trust |
| p8 observability/ops | seeds in p1; completes after p6 | M | §1.4-3, cluster metrics |
| p9 chaos CI | grows from p3; gate at end | M | GA trust bar |

**Release waves:**

- **v0.9 "Credible HA"** = p1–p4: one shard done *right* — fast, unified log,
  quorum-durable, self-healing. This alone moves the §1.1 comparison from
  "1–2 orders below peers" to "in-band".
- **v1.0 "Scalable"** = p5–p6: elastic membership, sharding × replication.
- **v1.1 "Enterprise"** = p7–p9 complete (p7/p8 work interleaves earlier; the
  release is the gate, not the start).

Also explicitly **not blocked on this roadmap:** the embedded engine (thepeach's
integration path) and the standalone server remain the recommended production
deployments today — they already clear their bars (82 ns writes; p99 <15 ms
reads; the stress test's 100k-DAU verdict).

## 6. Non-goals (scope honesty, per VISION.md)

- WAN active-active strong consistency. Cross-region stays async (log shipping)
  with the CRDT tier for user-scoped convergent data; regions are placement
  labels, not consistency domains.
- SQL, general transactions, multi-key serializability. The data model is
  signals (commutative accumulation), items/embeddings (idempotent upserts),
  hard-negatives (LWW CRDT) — the log + quorum gives these everything they need.
- Building a service mesh. mTLS + authenticated RPC yes; Istio no.
- Replacing the embedded mode. The cluster is for teams that *can't* embed;
  the engine stays the product's core.

## 7. Open questions (decide in-phase, tracked here so they don't vanish)

1. Client idempotency for signal retries under `ack=quorum` timeouts (p3):
   dedup window keyed by client request id vs. documented at-least-once.
2. Election engine: minimal purpose-built Raft over `tidal-net` vs. embedding a
   vetted raft crate — decide in p4 design after p3's quorum machinery exists.
3. Shard count / split policy (static S at create vs. range-split later) — p6.
4. At-rest encryption: in-engine vs. delegated to volume encryption — p7.
5. Whether `tidal-stress` grows a distributed multi-generator mode to push past
   the ~5k rps connection wall before p6's exit gate needs it.

---

*Companion docs: [stress-test-thepeach.md](ops/stress-test-thepeach.md) (baselines),
[runbooks/cluster.md](runbooks/cluster.md) (today's operational truth),
[ops/capacity-planning.md](ops/capacity-planning.md) (per-node sizing math),
[planning/ROADMAP.md](planning/ROADMAP.md) (M8 history + G4/G5/G6 gap registry).*