# Roadmap to an Enterprise-Grade Cluster

**Status:** Proposed (M11 candidate) · **Date:** 2026-06-10 · **Baseline evidence:**
[stress-test-thepeach.md](ops/stress-test-thepeach.md), [cluster runbook](runbooks/cluster.md),
[ROADMAP M8 Known Gaps](planning/ROADMAP.md) (G4/G5/G6), live k3s deployment (3 regions × 2-vCPU pods).

This is the gap analysis and phase plan from today's **experimental** multi-process
cluster (m8p10 — real process isolation, leader-durable writes, operator-driven
failover) to a cluster a paying customer could run as a system of record for
signals: **quorum-durable, self-healing, horizontally scalable, secured,
observable, and continuously chaos-tested.** Everything in §1 is measured or
code-verified, not aspirational; every phase in §4 ends in a testable exit gate.

---

## 1. Where we are today (evidence)

### 1.1 Measured baselines (2026-06-10, live 3-region k3s cluster, cpu-limit `"2"`/pod)

| Path | Measured | Notes |
|---|---|---|
| Embedded engine signal write | **~82 ns** (~12M/s/core) | criterion `scale.rs`; the engine itself is not the problem |
| Embedded RETRIEVE p99 | **152 µs** @ 1M items | scale baselines |
| Cluster `/feed` (ranked read) | **p99 8–15 ms** at 100→4,000 rps | never degraded in any run |
| Cluster `/search` | p99 3–11 ms | includes `reload_text_index` per request |
| **Replicated `/signals` (leader path)** | **~90 signals/s ceiling** | ok/s pinned at ~75 regardless of offered rate (100→4,000 rps); ~178–210 ms per write |
| **`/sharded/signals` (unreplicated)** | **3,669 signals/s, 0 errors, ~27% cluster CPU** | knee NOT reached; single generator became the limit |
| Cross-region replication lag | 0–3 events under all load | SLA <2 s; measured ~110–133 ms p99 (runbook) |
| Manual failover (promote→write) | ~31–34 ms | SLA <10 s |
| Overload behavior | **Graceful**: HTTP 429 shed, zero 5xx, **zero pod restarts**, no acknowledged-write loss observed | the strongest enterprise trait we have today |
| Aggregate HTTP ceiling | ~5k rps connection-establishment wall (conntrack/accept), cluster CPU idle | networking, not engine |

Ballpark peers on small nodes (context, not benchmarks we ran): etcd ≈10k
fsynced consensus writes/s; CockroachDB low-thousands/node; Cassandra QUORUM
10k+/node. **The replicated write path is 1–2 orders of magnitude below the
enterprise floor; everything else is competitive or better.**

### 1.2 What already exists and is worth keeping

- Real per-process region isolation; kill the leader → survivors keep serving (proven live).
- A real WAL relay over gRPC (`tidal-net` `WalShipping/ShipSegment`) with BLAKE3-verified
  segments, contiguous-frontier apply (no gap-swallowing), per-peer circuit breakers.
- Honest, documented durability contract (runbook §8) — rare and valuable; we extend it, not retrofit honesty.
- Operator verbs that work and are runbook-verified op-for-op (`promote`, `partition`, `heal`, `reconcile`),
  CRDT/LWW convergence for user-scoped hard-negatives, HLC with skew injection for tests.
- Sharded scatter-gather reads with degraded-not-failed semantics and shard-dedup merge.
- Tier-3 chaos/lifecycle/runbook suites over **real OS processes with real TCP-relay partition injection** —
  the scaffolding §4/p9 industrializes.
- Graceful backpressure end to end (write-pool 429 + engine WAL-channel 429 + breaker isolation).

### 1.3 Gap inventory → phase map

| Dimension | Today | Enterprise bar | Phase |
|---|---|---|---|
| Replication throughput | ~90/s: synchronous, sequential, per-request ship; 2-worker pool | Batched/pipelined, async; ≥2k/s on today's hardware | **p1** |
| Replication unity | Signals ride the WAL; items/embeddings ride a best-effort HTTP side channel (source of both live bugs we fixed: marker-gated broadcast, 401'd heal backfill); heal is O(items) | **One replicated log** for all replicated data; heal = log catch-up | **p2** |
| Durability | 2xx = leader fsync only; follower ship best-effort | `ack=quorum` mode: 2xx = majority-durable; zero acknowledged loss on any single-node kill | **p3** |
| Availability | Operator-driven promote; **restarted node re-reads topology and believes the static leader again** (observed live: us-east rejoined claiming leadership while eu-west led — split-brain seed) | Failure detector + automatic election + **term fencing** + durable leadership | **p4** |
| Membership | Static per-pod topology files with **literal IPs** (`grpc_addr` is `SocketAddr::parse` — no DNS); add/remove node = redeploy everything | Dynamic membership, DNS/seed discovery, online add/remove/replace with snapshot+stream catch-up (`StreamSegments` is declared but unimplemented) | **p5** |
| Write scaling | EITHER replicated (1 leader for everything) OR sharded (no replication); shards conflated with regions | **Sharding × replication**: shard groups with RF and distributed leaders; rebalancing | **p6** |
| Security | Plaintext gRPC replication (mTLS config exists, unexercised); one static bearer shared by all; internal marker as trust signal; no audit log | mTLS default + rotation, authenticated inter-node calls, admin audit, at-rest story | **p7** |
| Observability | **Cluster mode has no `/metrics` listener at all**; no request-id propagation on cluster routers; lag accounting wrong after promote (ShardId(0)-keyed); leader's own status row reads 0 | Per-node Prometheus metrics, tracing across forward/ship hops, truthful status, dashboards + alerts | **p8** (starts in p1) |
| Ops lifecycle | Manual heal loops (first heal can no-op into an open breaker); per-node backup only; rolling-upgrade suite exists but not a CI gate | Coordinated cluster backup/restore + PITR; upgrade-under-load as a release gate; self-driving heal | **p8** |
| Correctness assurance | Tier-3 suites are opt-in (`cluster-e2e` feature), run manually | Nightly chaos + soak in CI with invariant checkers; 30-day green before GA | **p9** |

### 1.4 Live incidents from this deployment that became requirements

1. **Restart amnesia / split-brain seed (→ p4).** After SIGKILL-failover, the
   restarted ex-leader booted from its static topology believing `leader=us-east`
   while the cluster had promoted eu-west; an operator re-promote reconciled it.
   With writes flowing through both gateways this is a dual-leader window.
   Requirement: durable terms, fencing on ship/forward, rejoin-as-follower.
2. **The HTTP side channel is where the bugs live (→ p2).** Both correctness bugs
   found live were in item/embedding broadcast, not the WAL relay: (a) forwarded
   writes never broadcast (terminated on the internal marker instead of
   leadership), (b) heal's backfill carried no bearer and 401'd forever. The WAL
   path was flawless throughout. Conclusion: replicate everything through the log.
3. **Breaker eats the first heal (→ p8).** After a partition window the per-peer
   breaker (threshold 5 / reset 30 s) is open; the first `/cluster/heal` no-ops.
   Runbook says "re-issue until lag 0" — an operator footgun; the server should
   drive heal-until-converged itself.
4. **The 178 ms ship (→ p1).** A LAN gRPC hop plus group-commit fsync does not
   cost 178 ms. Something structural (per-call setup, flow control, synchronous
   follower apply, sequential per-peer sends) is being paid per request. Profile
   before redesigning; the fix is likely cheaper than it looks.

---

## 2. Definition of done — the enterprise guarantees

Each guarantee is a **testable claim** with a named owner-test by GA (p9 wires any
that don't exist earlier). These are the bar; the phases are the path.

- **G-D (Durability).** With `ack=quorum`, a 2xx response means the write is
  durably applied on a majority of the shard's replicas. SIGKILL any single node
  at any moment under load: **zero acknowledged-write loss**, proven by a ledger
  checker that replays client acks against post-recovery state.
- **G-A (Availability).** Loss of any single node: reads continue uninterrupted;
  writes resume automatically in **<10 s p99** with no operator action and no
  dual-leader window (fencing proven under partition + restart chaos).
- **G-S (Scalability).** Write throughput scales with shard count at RF=3
  (≥2.5× from 1→3 shards on the same hardware). Reads scale with replicas.
- **G-E (Elasticity).** Nodes can be added/removed/replaced online; a new replica
  catches up via snapshot + WAL stream; p99 impact bounded (<2× for <60 s).
- **G-Sec (Security).** All inter-node links mTLS; replication endpoints
  authenticated (no implicit trust of the network); keys/certs rotate without
  downtime; admin verbs audit-logged.
- **G-O (Observability).** Every node exports Prometheus metrics; a dashboard
  answers "what is happening / what is wrong" without reading code; alerts on
  the golden signals + replication lag + commit-index stall + election churn.
- **G-Op (Operability).** Rolling upgrade with N/N+1 version skew under load is a
  CI-verified release gate; coordinated backup/restore (and WAL-archival PITR)
  drilled and timed; runbooks executable by a non-author.
- **G-C (Continuous correctness).** Nightly chaos (partitions, crashes, skew,
  disk faults) + soak (tidal-stress) green for 30 consecutive days before GA.

---

## 3. Reference environments and performance targets

Two named environments so numbers stay comparable end to end:

- **Ref-A (continuity, CI-able):** the current k3s deployment — 3 × (2 vCPU /
  2 GiB, local-path PV). All measured baselines in §1.1 are Ref-A. Phase exit
  gates are stated against Ref-A so they're re-runnable with `tidal-stress`.
- **Ref-B (enterprise reference):** 3–9 × (8 vCPU / 16 GiB, NVMe). GA marketing
  numbers come from Ref-B; claiming "enterprise" on shared 2-core pods is noise.

| Target | Ref-A (gate) | Ref-B (GA target) |
|---|---|---|
| Replicated signal writes, `ack=leader` | ≥2,000/s (p1) | ≥25,000/s |
| Replicated signal writes, `ack=quorum` | ≥1,000/s (p3) | ≥15,000/s |
| 3-shard × RF=3 quorum writes | ≥5,000/s (p6) | ≥50,000/s |
| Ranked read p99 (under write load) | ≤50 ms | ≤20 ms @ 10k reads/s |
| Auto-failover (detect→writes resume) | <10 s p99 (p4) | <5 s p99 |
| Replication lag p99 (sustained load) | ≤2 s | ≤500 ms |
| New-replica catch-up (100k items) | ≤5 min (p5) | ≤1 min |

---

## 4. The phases (proposed M11, `m11p1`–`m11p9`)

Ordering rule: **make the one log fast and unified before making it quorum; make
it quorum before electing leaders over it; fix membership before sharding it;
secure and industrialize continuously.**

### m11p1 — Replication performance floor (size: M)
**Goal:** the existing leader-durable path stops being the embarrassment line:
≥20× today's ceiling with no semantic change.

- **Profile first.** Instrument the ship path (this phase ships the first
  `tidaldb_cluster_*` metrics: fsync latency, ship RTT, queue depths, breaker
  state) and explain the measured ~178 ms/write. Suspects: per-call channel
  setup, tonic flow-control, synchronous follower-side apply in the RPC, the
  sequential per-peer loop in `relay.write_and_ship`, group-commit interaction.
- **Decouple ack from ship.** The client ack already only promises leader
  durability — so return at group-commit fsync; move shipping fully off the
  request path into per-peer outbound queues drained by dedicated senders
  (the relay/redeliver machinery already exists; make it the only path).
- **Batch + pipeline.** Ship WAL batches (not per-request payloads) with a
  windowed in-flight budget per peer (N batches outstanding, acks advance the
  window) instead of one blocking unary call at a time.
- **Rework the write pool.** `/signals` no longer needs a blocking gRPC worker
  per request; the pool sizes from CPU for fsync/apply work only. Keep 429
  semantics (bounded queue) — the graceful shed is a feature.
- **Tune group commit** (batch 100 / 10 ms today) against measured fsync cost on
  Ref-A's local-path PV; make both knobs config.
- Non-goals: no ack-mode change, no membership change.
- **Exit gate (Ref-A, `tidal-stress` leader path):** ≥2,000 signals/s OK at
  p99 ≤50 ms, error rate <1%, lag p99 ≤2 s sustained 10 min; replication metrics
  visible in Prometheus.

### m11p2 — One replicated log (size: L)
**Goal:** every replicated mutation rides the WAL relay; the HTTP broadcast side
channel (items/embeddings) is deleted.

- Add item-metadata and embedding records to the WAL event model (they're
  config-like, low-rate — the log absorbs them trivially; 64 MiB segment ceiling
  is ample for 1536-dim vectors).
- Followers apply items/embeddings from the log; leader broadcast code and the
  O(items) heal re-broadcast are removed (heal becomes pure log catch-up).
- Implement **`StreamSegments`** (declared, unimplemented) as the catch-up path:
  a follower that reports applied=N gets a stream from N+1; the in-memory
  `batch_log` dependency for recovery goes away (read back from the durable WAL).
- Hard-negatives stay CRDT (user-scoped, commutative) — document the split:
  "log = totally-ordered global data; CRDT = per-user convergent data".
- **Exit gate:** chaos tests for both 2026-06-10 bug classes pass by
  construction (no marker-gated fan-out to regress, no authed side POSTs);
  item written anywhere is readable everywhere ≤2 s p99; heal of a 10-minute-down
  follower with 100k items completes via stream without O(items) HTTP traffic.

### m11p3 — Quorum-acked writes (size: L) — closes ROADMAP gap **G4**
**Goal:** an opt-in durability contract a system of record can sit on.

- `ack=leader|quorum` (deployment default + per-request override header).
- Follower ship responses carry **durably-applied seqno** (fsync'd, not just
  received); leader maintains per-peer durable high-water marks and a
  **commit index** = quorum floor; `ack=quorum` responses gate on the commit
  index passing the write's seq. Pipelined (batch-level acks), never one-at-a-time.
- Timeout semantics: quorum not reached in budget → retryable 503 naming the
  laggard; the write may still commit (document at-least-once + dedup guidance;
  evaluate optional client idempotency keys for signal accounting — open
  question, decide in-phase).
- Rewrite runbook §8 from "honest about what we don't promise" to "here's the
  knob and its cost".
- **Exit gate:** SIGKILL the leader under `ack=quorum` load → ledger checker
  proves zero acknowledged loss across 100 random kill-points; quorum throughput
  ≥50% of p1's leader-ack number (≥1,000/s Ref-A).

### m11p4 — Failure detection, election, fencing (size: XL) — closes ROADMAP gap **G5**
**Goal:** "a machine died" is a non-event, not a runbook page.

- **Replicated metadata state** (term/epoch, leader id, membership): Raft-style
  election over the existing transport — the quorum machinery from p3 is the
  prerequisite. The data plane stays the WAL relay; only the small control state
  is consensus-managed.
- Failure detector on the existing `Heartbeat` RPC (timeout + jitter; tunable).
- **Fencing everywhere:** terms stamped on ship/forward/promote; stale-term
  ships rejected by followers; a restarted node **rejoins as follower** and
  learns leadership from the metadata state — never from the static topology
  file (kills incident §1.4-1).
- Leadership is durable locally (survives restart) and lease-bounded; manual
  `/cluster/promote` becomes a fenced transfer (drain for maintenance), not the
  availability mechanism.
- **Exit gate (chaos, Ref-A):** kill the leader under load → writes resume
  <10 s p99 with zero operator action and zero acked loss; partition the old
  leader away mid-election and let it restart → it cannot accept a single write
  (fencing proven); election churn under flapping links bounded (no livelock).

### m11p5 — Membership, discovery, elasticity (size: L)
**Goal:** nodes are cattle; topology is data, not files.

- Resolve `grpc_addr` via DNS (drop the `SocketAddr::parse`-only constraint) and
  support **seed-based join**: a new node contacts any seed, gets the membership
  + topology from the metadata state. The per-pod static-ClusterIP topology
  ConfigMap hack dies.
- Conf-changes (add/remove/replace) through the p4 metadata consensus; joiners
  catch up via **snapshot + StreamSegments** (snapshot = the engine's existing
  checkpoint artifacts, shipped as a stream).
- Kubernetes reference becomes one StatefulSet + headless Service peer
  discovery, PDB, topology-spread; rolling node replace is `kubectl delete pod`.
- **Exit gate:** scale 3→5→3 online under load: joiner catches up ≤5 min
  (100k items), serves quorum after; p99 impact <2× for <60 s; zero loss.

### m11p6 — Sharding × replication + rebalancing (size: XL)
**Goal:** writes scale horizontally **without** giving up replication — the
current "replicated XOR sharded" split ends.

- Shard groups: keyspace hash-split (reuse `ShardRouter`) into S shards, each a
  replication group at RF (default 3) with its own WAL/relay/commit-index and
  its own elected leader; leaders balanced across nodes.
- Regions stop being shards: topology = nodes × shard-replicas, with optional
  zone/region labels for placement and read affinity.
- Routing: any gateway routes by entity hash to the shard leader (write) or a
  local/nearest replica (read); scatter-gather already merges cross-shard reads
  with degraded semantics — now over shard groups.
- Rebalancing: shard move = snapshot + stream + fenced cutover; operator-
  triggered first, automatic (rate-limited) later.
- **Exit gate (Ref-A, 3 shards × RF=3):** ≥5,000 quorum signals/s (≥2.5×
  single-shard p3); kill any node → only its shard-leaderships move (<10 s),
  reads never stop; `tidal-stress` sharded-vs-replicated comparison collapses
  into one path.

### m11p7 — Security hardening (size: M)
**Goal:** the cluster stops trusting the network.

- **mTLS default** on gRPC replication (the `grpc_tls` config exists — exercise
  it, flip plaintext to an explicit `insecure: true` with a loud startup WARN);
  cert + bearer **rotation without restart** (file/secret watch).
- Authenticated inter-node HTTP: per-node identity (client certs or signed
  internal tokens); the `x-tidal-internal` marker remains a routing hint only
  (it already isn't an auth bypass — keep it that way by test).
- Audit log for admin verbs (promote/heal/partition/conf-change) with principal,
  term, and outcome; at-rest encryption documented (delegate to volume
  encryption, or in-engine as stretch); per-principal rate limits (the engine's
  rate limiter exists — wire it to the HTTP layer).
- **Exit gate:** reference deployment has zero plaintext inter-node links;
  rotation under load drops zero requests; a foreign pod on the cluster network
  can neither ship segments nor call internal routes (negative tests).

### m11p8 — Observability + operations (size: M; **starts inside p1**)
**Goal:** operable by someone who didn't build it.

- Complete the `tidaldb_cluster_*` metric set (lag, commit index, ship
  latency/queue, breaker, elections, forwards, write-pool, fsync) on a cluster
  `/metrics` listener (**today: none**); Grafana dashboard + alert rules shipped
  beside the existing standalone ones.
- Request-id propagation + tracing spans across forward/broadcast/ship hops
  (the cluster routers currently skip the request-id layer).
- Truthful status: fix post-promote lag accounting (ShardId(0) keying) and the
  leader's own applied row; `/cluster/status` becomes the single pane.
- **Self-driving heal:** server-side heal-until-converged (retry through breaker
  resets) so "re-issue heal until lag 0" leaves the runbook (incident §1.4-3).
- Coordinated backup/restore: cluster-consistent snapshot manifest across shard
  groups + WAL archival hook for PITR; timed restore drill in the runbook.
- Rolling upgrade: version handshake on ship/forward, N/N+1 skew documented;
  `mp_rolling_upgrade_no_loss_no_stall` promoted to a release-gate CI job.
- **Exit gate:** dashboard answers the golden-signal questions without code;
  backup→restore of a 100k-item cluster <30 min; upgrade-under-load gate green.

### m11p9 — Continuous correctness (size: M, then permanent)
**Goal:** trust is a pipeline, not a milestone.

- Nightly CI: the tier-3 chaos suites (partition relay, crash, clock-skew via
  `TIDAL_HLC_SKEW_MS`, rolling upgrade) + new fault classes (disk-full,
  slow-fsync injection, asymmetric partitions) against a Ref-A cluster.
- Invariant checkers as first-class: no-acked-loss ledger replay, per-entity
  monotonic counters, cross-replica decay parity (1e-6), membership safety
  (single leader per term per shard). Jepsen-style external harness as stretch.
- Nightly `tidal-stress` soak (1× 100k-DAU model, 1 h) with regression gates on
  p99/throughput; results archived for trend lines.
- **Exit gate (GA):** every guarantee in §2 maps to a named automated test;
  nightly suite green 30 consecutive days.

---

## 5. Sequencing, sizing, releases

| Phase | Depends on | Size | Closes |
|---|---|---|---|
| p1 perf floor | — | M | §1.4-4, throughput floor |
| p2 one log | p1 | L | §1.4-2, StreamSegments, O(items) heal |
| p3 quorum acks | p1, p2 | L | ROADMAP **G4** |
| p4 election + fencing | p3 | XL | ROADMAP **G5**, §1.4-1 |
| p5 membership | p4 | L | static-IP topology, elasticity |
| p6 shards × RF | p3, p5 | XL | write scaling |
| p7 security | p2 (stable surface) | M | network trust |
| p8 observability/ops | seeds in p1; completes after p6 | M | §1.4-3, cluster metrics |
| p9 chaos CI | grows from p3; gate at end | M | GA trust bar |

**Release waves:**

- **v0.9 "Credible HA"** = p1–p4: one shard done *right* — fast, unified log,
  quorum-durable, self-healing. This alone moves the §1.1 comparison from
  "1–2 orders below peers" to "in-band".
- **v1.0 "Scalable"** = p5–p6: elastic membership, sharding × replication.
- **v1.1 "Enterprise"** = p7–p9 complete (p7/p8 work interleaves earlier; the
  release is the gate, not the start).

Also explicitly **not blocked on this roadmap:** the embedded engine (thepeach's
integration path) and the standalone server remain the recommended production
deployments today — they already clear their bars (82 ns writes; p99 <15 ms
reads; the stress test's 100k-DAU verdict).

## 6. Non-goals (scope honesty, per VISION.md)

- WAN active-active strong consistency. Cross-region stays async (log shipping)
  with the CRDT tier for user-scoped convergent data; regions are placement
  labels, not consistency domains.
- SQL, general transactions, multi-key serializability. The data model is
  signals (commutative accumulation), items/embeddings (idempotent upserts),
  hard-negatives (LWW CRDT) — the log + quorum gives these everything they need.
- Building a service mesh. mTLS + authenticated RPC yes; Istio no.
- Replacing the embedded mode. The cluster is for teams that *can't* embed;
  the engine stays the product's core.

## 7. Open questions (decide in-phase, tracked here so they don't vanish)

1. Client idempotency for signal retries under `ack=quorum` timeouts (p3):
   dedup window keyed by client request id vs. documented at-least-once.
2. Election engine: minimal purpose-built Raft over `tidal-net` vs. embedding a
   vetted raft crate — decide in p4 design after p3's quorum machinery exists.
3. Shard count / split policy (static S at create vs. range-split later) — p6.
4. At-rest encryption: in-engine vs. delegated to volume encryption — p7.
5. Whether `tidal-stress` grows a distributed multi-generator mode to push past
   the ~5k rps connection wall before p6's exit gate needs it.

---

*Companion docs: [stress-test-thepeach.md](ops/stress-test-thepeach.md) (baselines),
[runbooks/cluster.md](runbooks/cluster.md) (today's operational truth),
[ops/capacity-planning.md](ops/capacity-planning.md) (per-node sizing math),
[planning/ROADMAP.md](planning/ROADMAP.md) (M8 history + G4/G5/G6 gap registry).*