tidaldb/docs/roadmap-to-cluster.md

Roadmap to an Enterprise-Grade Cluster

Status: ADOPTED as M11 — m11p1 ✅, m11p2 ✅, m11p3 ✅, m11p4 ✅, m11p5 ✅, m11p7 ✅ complete (m11p7 2026-06-13 — security hardening: gRPC mTLS default + zero-drop cert rotation, inter-node HTTP TLS + per-node signed tokens, admin audit log, per-principal rate limit; the v1.1 "Enterprise" network-trust leg) · m11p6 (sharding × replication) data plane in progress · p8–p9 planned · Date: 2026-06-10 · Baseline evidence: stress-test-thepeach.md, cluster runbook, ROADMAP M8 Known Gaps (G4/G5/G6), live k3s deployment (3 regions × 2-vCPU pods).

This is the gap analysis and phase plan from today's experimental multi-process cluster (m8p10 — real process isolation, leader-durable writes, operator-driven failover) to a cluster a paying customer could run as a system of record for signals: quorum-durable, self-healing, horizontally scalable, secured, observable, and continuously chaos-tested. Everything in §1 is measured or code-verified, not aspirational; every phase in §4 ends in a testable exit gate.

---

1. Where we are today (evidence)

1.1 Measured baselines (2026-06-10, live 3-region k3s cluster, cpu-limit "2"/pod)

Path	Measured	Notes
Embedded engine signal write	~82 ns (~12M/s/core)	criterion scale.rs; the engine itself is not the problem
Embedded RETRIEVE p99	152 µs @ 1M items	scale baselines
Cluster /feed (ranked read)	p99 8–15 ms at 100→4,000 rps	never degraded in any run
Cluster /search	p99 3–11 ms	includes reload_text_index per request
Replicated /signals (leader path)	~90 signals/s ceiling	ok/s pinned at ~75 regardless of offered rate (100→4,000 rps); ~178–210 ms per write
/sharded/signals (unreplicated)	3,669 signals/s, 0 errors, ~27% cluster CPU	knee NOT reached; single generator became the limit
Cross-region replication lag	0–3 events under all load	SLA <2 s; measured ~110–133 ms p99 (runbook)
Manual failover (promote→write)	~31–34 ms	SLA <10 s
Overload behavior	Graceful: HTTP 429 shed, zero 5xx, zero pod restarts, no acknowledged-write loss observed	the strongest enterprise trait we have today
Aggregate HTTP ceiling	~5k rps connection-establishment wall (conntrack/accept), cluster CPU idle	networking, not engine

Ballpark peers on small nodes (context, not benchmarks we ran): etcd ≈10k fsynced consensus writes/s; CockroachDB low-thousands/node; Cassandra QUORUM 10k+/node. The replicated write path is 1–2 orders of magnitude below the enterprise floor; everything else is competitive or better.

1.2 What already exists and is worth keeping

• Real per-process region isolation; kill the leader → survivors keep serving (proven live).
• A real WAL relay over gRPC (tidal-net WalShipping/ShipSegment) with BLAKE3-verified segments, contiguous-frontier apply (no gap-swallowing), per-peer circuit breakers.
• Honest, documented durability contract (runbook §8) — rare and valuable; we extend it, not retrofit honesty.
• Operator verbs that work and are runbook-verified op-for-op (promote, partition, heal, reconcile), CRDT/LWW convergence for user-scoped hard-negatives, HLC with skew injection for tests.
• Sharded scatter-gather reads with degraded-not-failed semantics and shard-dedup merge.
• Tier-3 chaos/lifecycle/runbook suites over real OS processes with real TCP-relay partition injection — the scaffolding §4/p9 industrializes.
• Graceful backpressure end to end (write-pool 429 + engine WAL-channel 429 + breaker isolation).

1.3 Gap inventory → phase map

Dimension	Today	Enterprise bar	Phase
Replication throughput	~90/s: synchronous, sequential, per-request ship; 2-worker pool	Batched/pipelined, async; ≥2k/s on today's hardware	p1
Replication unity	Signals ride the WAL; items/embeddings ride a best-effort HTTP side channel (source of both live bugs we fixed: marker-gated broadcast, 401'd heal backfill); heal is O(items)	One replicated log for all replicated data; heal = log catch-up	p2
Durability	2xx = leader fsync only; follower ship best-effort	ack=quorum mode: 2xx = majority-durable; zero acknowledged loss on any single-node kill	p3
Availability	Operator-driven promote; restarted node re-reads topology and believes the static leader again (observed live: us-east rejoined claiming leadership while eu-west led — split-brain seed)	Failure detector + automatic election + term fencing + durable leadership	p4
Membership	✅ p5: DNS/seed discovery (grpc_addr advertised + DNS-resolved, grpc_bind for the local bind), kind-4 membership records on the one log, online /cluster/join+remove with learner→voter auto-promotion, FetchSnapshot snapshot+stream catch-up, self-healing reseed, k8s/cluster/ one-StatefulSet reference	Dynamic membership, DNS/seed discovery, online add/remove/replace with snapshot+stream catch-up	p5 ✅
Write scaling	EITHER replicated (1 leader for everything) OR sharded (no replication); shards conflated with regions	Sharding × replication: shard groups with RF and distributed leaders; rebalancing	p6
Security	Plaintext gRPC replication (mTLS config exists, unexercised); one static bearer shared by all; internal marker as trust signal; no audit log	mTLS default + rotation, authenticated inter-node calls, admin audit, at-rest story	p7
Observability	Cluster mode has no /metrics listener at all; no request-id propagation on cluster routers; lag accounting wrong after promote (ShardId(0)-keyed); leader's own status row reads 0	Per-node Prometheus metrics, tracing across forward/ship hops, truthful status, dashboards + alerts	p8 (starts in p1)
Ops lifecycle	Manual heal loops (first heal can no-op into an open breaker); per-node backup only; rolling-upgrade suite exists but not a CI gate	Coordinated cluster backup/restore + PITR; upgrade-under-load as a release gate; self-driving heal	p8
Correctness assurance	Tier-3 suites are opt-in (cluster-e2e feature), run manually	Nightly chaos + soak in CI with invariant checkers; 30-day green before GA	p9

1.4 Live incidents from this deployment that became requirements

1. Restart amnesia / split-brain seed (→ p4). After SIGKILL-failover, the restarted ex-leader booted from its static topology believing leader=us-east while the cluster had promoted eu-west; an operator re-promote reconciled it. With writes flowing through both gateways this is a dual-leader window. Requirement: durable terms, fencing on ship/forward, rejoin-as-follower.
2. The HTTP side channel is where the bugs live (→ p2). Both correctness bugs found live were in item/embedding broadcast, not the WAL relay: (a) forwarded writes never broadcast (terminated on the internal marker instead of leadership), (b) heal's backfill carried no bearer and 401'd forever. The WAL path was flawless throughout. Conclusion: replicate everything through the log.
3. Breaker eats the first heal (→ p8). After a partition window the per-peer breaker (threshold 5 / reset 30 s) is open; the first /cluster/heal no-ops. Runbook says "re-issue until lag 0" — an operator footgun; the server should drive heal-until-converged itself.
4. The 178 ms ship (→ p1). A LAN gRPC hop plus group-commit fsync does not cost 178 ms. Something structural (per-call setup, flow control, synchronous follower apply, sequential per-peer sends) is being paid per request. Profile before redesigning; the fix is likely cheaper than it looks.

---

2. Definition of done — the enterprise guarantees

Each guarantee is a testable claim with a named owner-test by GA (p9 wires any that don't exist earlier). These are the bar; the phases are the path.

• G-D (Durability). With ack=quorum, a 2xx response means the write is durably applied on a majority of the shard's replicas. SIGKILL any single node at any moment under load: zero acknowledged-write loss, proven by a ledger checker that replays client acks against post-recovery state.
• G-A (Availability). Loss of any single node: reads continue uninterrupted; writes resume automatically in <10 s p99 with no operator action and no dual-leader window (fencing proven under partition + restart chaos).
• G-S (Scalability). Write throughput scales with shard count at RF=3 (≥2.5× from 1→3 shards on the same hardware). Reads scale with replicas.
• G-E (Elasticity). Nodes can be added/removed/replaced online; a new replica catches up via snapshot + WAL stream; p99 impact bounded (<2× for <60 s).
• G-Sec (Security). All inter-node links mTLS; replication endpoints authenticated (no implicit trust of the network); keys/certs rotate without downtime; admin verbs audit-logged.
• G-O (Observability). Every node exports Prometheus metrics; a dashboard answers "what is happening / what is wrong" without reading code; alerts on the golden signals + replication lag + commit-index stall + election churn.
• G-Op (Operability). Rolling upgrade with N/N+1 version skew under load is a CI-verified release gate; coordinated backup/restore (and WAL-archival PITR) drilled and timed; runbooks executable by a non-author.
• G-C (Continuous correctness). Nightly chaos (partitions, crashes, skew, disk faults) + soak (tidal-stress) green for 30 consecutive days before GA.

---

3. Reference environments and performance targets

Two named environments so numbers stay comparable end to end:

• Ref-A (continuity, CI-able): the current k3s deployment — 3 × (2 vCPU / 2 GiB, local-path PV). All measured baselines in §1.1 are Ref-A. Phase exit gates are stated against Ref-A so they're re-runnable with tidal-stress.
• Ref-B (enterprise reference): 3–9 × (8 vCPU / 16 GiB, NVMe). GA marketing numbers come from Ref-B; claiming "enterprise" on shared 2-core pods is noise.

Target	Ref-A (gate)	Ref-B (GA target)
Replicated signal writes, ack=leader	≥2,000/s (p1)	≥25,000/s
Replicated signal writes, ack=quorum	≥1,000/s (p3)	≥15,000/s
3-shard × RF=3 quorum writes	≥5,000/s (p6)	≥50,000/s
Ranked read p99 (under write load)	≤50 ms	≤20 ms @ 10k reads/s
Auto-failover (detect→writes resume)	<10 s p99 (p4)	<5 s p99
Replication lag p99 (sustained load)	≤2 s	≤500 ms
New-replica catch-up (100k items)	≤5 min (p5)	≤1 min

---

4. The phases (proposed M11, m11p1–m11p9)

Ordering rule: make the one log fast and unified before making it quorum; make it quorum before electing leaders over it; fix membership before sharding it; secure and industrialize continuously.

m11p1 — Replication performance floor (size: M) — ✅ COMPLETE (2026-06-11)

As built. The ~178ms/write was structural, exactly as suspected — three stacked serializations: (1) the relay's seqno lock held across the WAL group-commit wait, so every writer paid a SOLO batch_timeout + fsync (~11ms × 2-8 pool workers ≈ the measured ~90/s ceiling and, with the 16-deep queue, the ~178ms latency); (2) the per-request sequential per-peer eager ship on the request path; (3) — unanticipated by this plan — the follower applying every received segment through its own solo group-commit, capping follower apply at ~events-per-segment / fsync-cost (~1.8k/s measured), which made followers lag unboundedly once the leader got fast. Fixes: staged two-phase writes (stage_write atomic under the lock in µs, fsync waited outside it — concurrent writers share group commits); per-peer windowed batch senders shipping only the leader's durable frontier, with relay poisoning on fsync failure; follower backlog draining (try_recv_segment) applied through one shared group commit in range-disjoint groups. Measured (3-process localhost, release build, thepeach mix): 4,534 replicated signal-writes/s within SLO on the ramp (~50×; knee ~5.5k signals/s), and 2,739 signals/s sustained for 10 minutes (1.65M writes, 0.35% errors, zero shed) with replication lag never exceeding 103 events (~40ms). Group commit measured at 58 events/fsync mean; leader fsync (macOS F_FULLFSYNC) 7.4ms mean with a 10-50ms tail — tidaldb_cluster_wal_fsync_us now answers the "profile first" question per deployment. The ≤50ms write-p99 sub-gate tracks the platform fsync floor and validates on Ref-A's Linux fdatasync (re-run pending infra access). Knobs shipped: replication.{batch_max_events,window,retry_ms}, wal.{batch_size,batch_timeout_ms}, per-region metrics_addr.

Goal: the existing leader-durable path stops being the embarrassment line: ≥20× today's ceiling with no semantic change.

• Profile first. Instrument the ship path (this phase ships the first tidaldb_cluster_* metrics: fsync latency, ship RTT, queue depths, breaker state) and explain the measured ~178 ms/write. Suspects: per-call channel setup, tonic flow-control, synchronous follower-side apply in the RPC, the sequential per-peer loop in relay.write_and_ship, group-commit interaction.
• Decouple ack from ship. The client ack already only promises leader durability — so return at group-commit fsync; move shipping fully off the request path into per-peer outbound queues drained by dedicated senders (the relay/redeliver machinery already exists; make it the only path).
• Batch + pipeline. Ship WAL batches (not per-request payloads) with a windowed in-flight budget per peer (N batches outstanding, acks advance the window) instead of one blocking unary call at a time.
• Rework the write pool. /signals no longer needs a blocking gRPC worker per request; the pool sizes from CPU for fsync/apply work only. Keep 429 semantics (bounded queue) — the graceful shed is a feature.
• Tune group commit (batch 100 / 10 ms today) against measured fsync cost on Ref-A's local-path PV; make both knobs config.
• Non-goals: no ack-mode change, no membership change.
• Exit gate (Ref-A, tidal-stress leader path): ≥2,000 signals/s OK at p99 ≤50 ms, error rate <1%, lag p99 ≤2 s sustained 10 min; replication metrics visible in Prometheus.

m11p2 — One replicated log (size: L)

Goal: every replicated mutation rides the WAL relay; the HTTP broadcast side channel (items/embeddings) is deleted.

• Add item-metadata and embedding records to the WAL event model (they're config-like, low-rate — the log absorbs them trivially; 64 MiB segment ceiling is ample for 1536-dim vectors).
• Followers apply items/embeddings from the log; leader broadcast code and the O(items) heal re-broadcast are removed (heal becomes pure log catch-up).
• Implement StreamSegments (declared, unimplemented) as the catch-up path: a follower that reports applied=N gets a stream from N+1; the in-memory batch_log dependency for recovery goes away (read back from the durable WAL).
• Hard-negatives stay CRDT (user-scoped, commutative) — document the split: "log = totally-ordered global data; CRDT = per-user convergent data".
• Exit gate: chaos tests for both 2026-06-10 bug classes pass by construction (no marker-gated fan-out to regress, no authed side POSTs); item written anywhere is readable everywhere ≤2 s p99; heal of a 10-minute-down follower with 100k items completes via stream without O(items) HTTP traffic.

✅ COMPLETE (2026-06-11), as built: the WAL itself became the one log — the group-commit writer feeds every fsynced batch (signals = kind-0, items/embeddings = kind-1/2 blob records, journaled BEFORE storage) to a bounded ship feed whose bytes ship verbatim; stream seqnos are WAL seqnos and survive restarts. Catch-up is FOLLOWER-PULLED StreamSegments over the durable segments (gap-triggered, boot-time self-heal, heal-nudged), with ship acks piggybacking the follower's applied seqno; promote carries a persisted stream baseline so a new leader's pre-stream history never replays. The HTTP broadcast, O(items) heal backfill, and the m11p1 relay bookkeeping are deleted; multi-process mode now requires --data-dir. Exit-gate mechanics proven by tier-3 mp_items_ride_the_log_and_catchup_stream (items via leader AND follower gateways converge everywhere; a follower stopped through item + embedding + signal writes restarts and converges via the stream with no operator verb and no HTTP item traffic). The 100k-item scale figure remains a Ref-A run (same k3s access blocker as p1's p99 sub-gate). Details: planning/milestone-11/phase-2.md.

m11p3 — Quorum-acked writes (size: L) — closes ROADMAP gap G4

Goal: an opt-in durability contract a system of record can sit on.

• ack=leader|quorum (deployment default + per-request override header).
• Follower ship responses carry durably-applied seqno (fsync'd, not just received); leader maintains per-peer durable high-water marks and a commit index = quorum floor; ack=quorum responses gate on the commit index passing the write's seq. Pipelined (batch-level acks), never one-at-a-time.
• Timeout semantics: quorum not reached in budget → retryable 503 naming the laggard; the write may still commit (document at-least-once + dedup guidance; evaluate optional client idempotency keys for signal accounting — open question, decide in-phase).
• Rewrite runbook §8 from "honest about what we don't promise" to "here's the knob and its cost".
• Exit gate: SIGKILL the leader under ack=quorum load → ledger checker proves zero acknowledged loss across 100 random kill-points; quorum throughput ≥50% of p1's leader-ack number (≥1,000/s Ref-A).

✅ COMPLETE (2026-06-11), as built: durable freshness is follower-PUSHED (ReportApplied once per apply round — decoupled from ship acks after the ack-holding design measurably death-spiraled under load) into a leadership-scoped commit index (k-th largest durable mark); ack=quorum waits are fully async (watch-channel bridge — thread-per-wait exhausted the blocking pool); timeouts are retryable 503s naming laggards; every cluster write returns x-tidal-seq. The gate also surfaced and fixed an m11p2 flaw: follower blob applies paid one fsync per item — now batched through shared group commits (22× item-seed speedup). Idempotency question DECIDED: no new machinery; at-least-once + dedup guidance in runbook §8. Evidence: 100/100 kill-points zero acked loss; 3,600 quorum writes/s within SLO locally = 79% of p1's leader-ack figure (Ref-A run still blocked on k3s access). Details: planning/milestone-11/phase-3.md.

m11p4 — Failure detection, election, fencing (size: XL) — ✅ COMPLETE (2026-06-12) — closes ROADMAP gap G5

✅ COMPLETE, as built: purpose-built election-only Raft (pre-vote + check-quorum + fenced transfer) over the existing transport — no raft crate; the WAL is the log, and an elected leader's FIRST entry is a replicated kind-3 term marker, so Raft's (lastLogTerm, lastLogIndex) derives from one fsync stream (frontiers compared in the last joined term's STREAM numbering — a reseeded node's local numbering diverges). Hard state (term, voted_for) in data_dir/election_state (corrupt = refuse boot; lost-with-WAL = forced follower), fsynced before any vote or claim leaves. Term fencing on every RPC; the commit index folds only activation-term reports; a restarted ex-leader boots follower from durable state (§1.4-1 closed by construction). Divergent suffixes QUARANTINE (reseed is recovery; p5 snapshots automate). Promote = fenced transfer that refuses lagging targets — the runbook's "max-applied survivor" rule is now protocol. auto_election: false preserves the operator posture. Exit gates (local tier-3, 3 processes): elections in 0.59–0.98 s under ack=quorum kill loads with zero acked loss across random kill points; a restarted partitioned ex-leader accepted 0 writes; 7 link flaps converged to one leader at term ≤ 15. Details: planning/milestone-11/phase-4.md.

✅ Shipped increment (2026-06-11, d0a52e4) — catch-up self-healing + WAL segment versioning. Failed StreamSegments pulls now arm a 30 s timer retry; re-arm-on-skip is the load-bearing liveness fix — without it a pull that fails in an idle cluster never re-fires and the follower stays permanently lagged (root cause: 2026-06-11 p3 rollout left both followers stuck at lag=136,507 with zero writes to trigger a retry). WAL segments carry a TSEG 8-byte header (magic + version byte + reserved); pre-p4 headerless segments are implicit v0, no migration; unknown format → WalError::SegmentFormatUnknown with no repair or truncation of foreign files. FAILED_PRECONDITION is the typed refusal for unservable catch-up. Regression gate (ops/stress-test-p4-regression.md): timer-only catch-up confirmed (lag=0 in 10 s with zero writes); p3 gate (≥1,000 quorum/s, p99 ≤50 ms, <1% error) holds through p4. The Heartbeat RPC and ControlPlane::record_shard_heartbeat already exist and health-track node liveness — they become the failure-detector input for the election work below.

Goal (main p4 work, pending): "a machine died" is a non-event, not a runbook page.

• Replicated metadata state (term/epoch, leader id, membership): Raft-style election over the existing transport — the quorum machinery from p3 is the prerequisite. The data plane stays the WAL relay; only the small control state is consensus-managed.
• Failure detector on the existing Heartbeat RPC (timeout + jitter; tunable).
• Fencing everywhere: terms stamped on ship/forward/promote; stale-term ships rejected by followers; a restarted node rejoins as follower and learns leadership from the metadata state — never from the static topology file (kills incident §1.4-1).
• Leadership is durable locally (survives restart) and lease-bounded; manual /cluster/promote becomes a fenced transfer (drain for maintenance), not the availability mechanism.
• Exit gate (chaos, Ref-A): kill the leader under load → writes resume <10 s p99 with zero operator action and zero acked loss; partition the old leader away mid-election and let it restart → it cannot accept a single write (fencing proven); election churn under flapping links bounded (no livelock).

m11p5 — Membership, discovery, elasticity (size: L)

Goal: nodes are cattle; topology is data, not files.

• Resolve grpc_addr via DNS (drop the SocketAddr::parse-only constraint) and support seed-based join: a new node contacts any seed, gets the membership
  • topology from the metadata state. The per-pod static-ClusterIP topology ConfigMap hack dies.
• Conf-changes (add/remove/replace) through the p4 metadata consensus; joiners catch up via snapshot + StreamSegments (snapshot = the engine's existing checkpoint artifacts, shipped as a stream).
• Kubernetes reference becomes one StatefulSet + headless Service peer discovery, PDB, topology-spread; rolling node replace is kubectl delete pod.
• Exit gate: scale 3→5→3 online under load: joiner catches up ≤5 min (100k items), serves quorum after; p99 impact <2× for <60 s; zero loss.

✅ COMPLETE (2026-06-12), as built: grpc_addr became an advertised address (hostname or IP) split from an optional grpc_bind; tidal-net's peer map retyped SocketAddr → String so hyper re-resolves DNS on every reconnect (the pod-rescheduled-onto-a-new-IP case the SocketAddr::parse constraint made impossible). A new server-streaming FetchSnapshot RPC ships create_backup as a manifest + BLAKE3-verified file chunks (term-stamped, term-fenced, identify-or-refuse) into a boot-time staging-dir install (every crash window an idempotent redo); reseed is self-healing — a typed x-tidal-catchup: snapshot-required refusal (or the p4 quarantine) latches a durable reseed_required marker that runs on the next boot and clears the divergence gauge, no wipe_data_dir. Membership is data on the one log: a new kind-4 MembershipRecord (beside kind-0/1/2/3) folds into a ClusterMembership cell; POST /cluster/join appends a quorum-committed Learner that auto-promotes to Voter; conf-changes are Raft single-server with an activation-record re-append closing the vacuous-gate + baseline-jump blockers. The three-way term-join rule (own < prev_log → latch reseed) closes p4's pre-baseline-history carried hazard; the even-n majority() fix ((n+1).div_ceil(2)+1) closes a latent dual-leader bug. Seed-join boot (--seed/--advertise-*/--metrics, knob file still required) + k8s/cluster/ (one StatefulSet, headless Service, PDB, topology-spread) make kubectl delete pod the node-replace drill. Exit-gate mechanics proven by tier-3 cluster_membership.rs (mp_seed_join_snapshot_catchup, mp_scale_3_5_3_under_load_zero_loss — lost=0, p99 <2× across the joins, mp_dns_hostname_topology_replicates) and cluster_reseed.rs. The 100k-item catch-up and the k8s pod-reschedule drill remain Ref-A line items (k3s access pending — the standing M11 caveat; the 5000-item localhost catch-up at 26.4s has large headroom under the ≤5-min budget). Details: planning/milestone-11/phase-5.md.

m11p6 — Sharding × replication + rebalancing (size: XL)

Goal: writes scale horizontally without giving up replication — the current "replicated XOR sharded" split ends.

• Shard groups: keyspace hash-split (reuse ShardRouter) into S shards, each a replication group at RF (default 3) with its own WAL/relay/commit-index and its own elected leader; leaders balanced across nodes.
• Regions stop being shards: topology = nodes × shard-replicas, with optional zone/region labels for placement and read affinity.
• Routing: any gateway routes by entity hash to the shard leader (write) or a local/nearest replica (read); scatter-gather already merges cross-shard reads with degraded semantics — now over shard groups.
• Rebalancing: shard move = snapshot + stream + fenced cutover; operator- triggered first, automatic (rate-limited) later.
• Exit gate (Ref-A, 3 shards × RF=3): ≥5,000 quorum signals/s (≥2.5× single-shard p3); kill any node → only its shard-leaderships move (<10 s), reads never stop; tidal-stress sharded-vs-replicated comparison collapses into one path.

m11p7 — Security hardening (size: M)

Goal: the cluster stops trusting the network.

• mTLS default on gRPC replication (the grpc_tls config exists — exercise it, flip plaintext to an explicit insecure: true with a loud startup WARN); cert + bearer rotation without restart (file/secret watch).
• Authenticated inter-node HTTP: per-node identity (client certs or signed internal tokens); the x-tidal-internal marker remains a routing hint only (it already isn't an auth bypass — keep it that way by test).
• Audit log for admin verbs (promote/heal/partition/conf-change) with principal, term, and outcome; at-rest encryption documented (delegate to volume encryption, or in-engine as stretch); per-principal rate limits (the engine's rate limiter exists — wire it to the HTTP layer).
• Exit gate: reference deployment has zero plaintext inter-node links; rotation under load drops zero requests; a foreign pod on the cluster network can neither ship segments nor call internal routes (negative tests).

As-built (m11p7, 2026-06-13 — COMPLETE): gRPC is served over a custom tokio-rustls acceptor (tonic 0.12 caches a fixed ServerConfig with no resolver hook, so it cannot hot-swap — the custom acceptor is the only zero-drop path) fed a DynamicCertResolver (ArcSwap<CertifiedKey>); mTLS preserved exactly (WebPkiClientVerifier, ALPN h2), plaintext → explicit insecure:true

• loud WARN. Rotation = content-hash polling (catches k8s ..data symlink swaps inotify misses) → atomic resolver swap (in-flight sessions untouched) + peer-channel rebuild → zero dropped requests under load (verified). Inter- node HTTP reuses the SAME resolver for server TLS + dials https with the cluster CA; per-node identity is a signed x-tidal-node-token (keyed-BLAKE3 MAC under a shared cluster key — no new crypto dep, since requiring HTTP client certs would break external bearer clients), and the x-tidal-internal marker is honored only from a verified sibling (marker-without-token → 403). Admin verbs audit to a tidal_audit target + optional TIDAL_AUDIT_LOG JSONL (operator-leg only). Per-principal rate limit reuses the engine RateLimiter (nodes exempt). All opt-in (grpc_tls/cluster key/env) ⇒ pre-m11p7 behavior byte-for-byte. See milestone-11/phase-7.md.

m11p8 — Observability + operations (size: M; starts inside p1)

Goal: operable by someone who didn't build it.

• Complete the tidaldb_cluster_* metric set (lag, commit index, ship latency/queue, breaker, elections, forwards, write-pool, fsync) on a cluster /metrics listener (today: none); Grafana dashboard + alert rules shipped beside the existing standalone ones.
• Request-id propagation + tracing spans across forward/broadcast/ship hops (the cluster routers currently skip the request-id layer).
• Truthful status: fix post-promote lag accounting (ShardId(0) keying) and the leader's own applied row; /cluster/status becomes the single pane.
• Self-driving heal: server-side heal-until-converged (retry through breaker resets) so "re-issue heal until lag 0" leaves the runbook (incident §1.4-3).
• Coordinated backup/restore: cluster-consistent snapshot manifest across shard groups + WAL archival hook for PITR; timed restore drill in the runbook.
• Rolling upgrade: version handshake on ship/forward, N/N+1 skew documented; mp_rolling_upgrade_no_loss_no_stall promoted to a release-gate CI job.
• Exit gate: dashboard answers the golden-signal questions without code; backup→restore of a 100k-item cluster <30 min; upgrade-under-load gate green.

m11p9 — Continuous correctness (size: M, then permanent)

Goal: trust is a pipeline, not a milestone.

• Nightly CI: the tier-3 chaos suites (partition relay, crash, clock-skew via TIDAL_HLC_SKEW_MS, rolling upgrade) + new fault classes (disk-full, slow-fsync injection, asymmetric partitions) against a Ref-A cluster.
• Invariant checkers as first-class: no-acked-loss ledger replay, per-entity monotonic counters, cross-replica decay parity (1e-6), membership safety (single leader per term per shard). Jepsen-style external harness as stretch.
• Nightly tidal-stress soak (1× 100k-DAU model, 1 h) with regression gates on p99/throughput; results archived for trend lines.
• Exit gate (GA): every guarantee in §2 maps to a named automated test; nightly suite green 30 consecutive days.

---

5. Sequencing, sizing, releases

Phase	Depends on	Size	Closes
p1 perf floor	—	M	§1.4-4, throughput floor
p2 one log	p1	L	§1.4-2, StreamSegments, O(items) heal
p3 quorum acks	p1, p2	L	ROADMAP G4
p4 election + fencing	p3	XL	ROADMAP G5, §1.4-1
p5 membership	p4	L	static-IP topology, elasticity
p6 shards × RF	p3, p5	XL	write scaling
p7 security	p2 (stable surface)	M	network trust
p8 observability/ops	seeds in p1; completes after p6	M	§1.4-3, cluster metrics
p9 chaos CI	grows from p3; gate at end	M	GA trust bar

Release waves:

• v0.9 "Credible HA" = p1–p4: one shard done right — fast, unified log, quorum-durable, self-healing. This alone moves the §1.1 comparison from "1–2 orders below peers" to "in-band".
• v1.0 "Scalable" = p5–p6: elastic membership, sharding × replication.
• v1.1 "Enterprise" = p7–p9 complete (p7/p8 work interleaves earlier; the release is the gate, not the start).

Also explicitly not blocked on this roadmap: the embedded engine (thepeach's integration path) and the standalone server remain the recommended production deployments today — they already clear their bars (82 ns writes; p99 <15 ms reads; the stress test's 100k-DAU verdict).

6. Non-goals (scope honesty, per VISION.md)

• WAN active-active strong consistency. Cross-region stays async (log shipping) with the CRDT tier for user-scoped convergent data; regions are placement labels, not consistency domains.
• SQL, general transactions, multi-key serializability. The data model is signals (commutative accumulation), items/embeddings (idempotent upserts), hard-negatives (LWW CRDT) — the log + quorum gives these everything they need.
• Building a service mesh. mTLS + authenticated RPC yes; Istio no.
• Replacing the embedded mode. The cluster is for teams that can't embed; the engine stays the product's core.

7. Open questions (decide in-phase, tracked here so they don't vanish)

1. Client idempotency for signal retries under ack=quorum timeouts (p3): dedup window keyed by client request id vs. documented at-least-once.
2. Election engine: minimal purpose-built Raft over tidal-net vs. embedding a vetted raft crate — decide in p4 design after p3's quorum machinery exists.
3. Shard count / split policy (static S at create vs. range-split later) — p6.
4. At-rest encryption: in-engine vs. delegated to volume encryption — p7.
5. Whether tidal-stress grows a distributed multi-generator mode to push past the ~5k rps connection wall before p6's exit gate needs it.

---

Companion docs: stress-test-thepeach.md (baselines), runbooks/cluster.md (today's operational truth), ops/capacity-planning.md (per-node sizing math), planning/ROADMAP.md (M8 history + G4/G5/G6 gap registry).