diff --git a/docs/README.md b/docs/README.md index 921214e..0d590ee 100644 --- a/docs/README.md +++ b/docs/README.md @@ -25,6 +25,7 @@ The authoritative component specifications (status: Implemented, M0–M8). ## Planning — `planning/` - [**ROADMAP.md**](planning/ROADMAP.md) — milestones M0–M10, phase status, known gaps +- [**roadmap-to-cluster.md**](roadmap-to-cluster.md) — gap analysis + phase plan (M11 candidate) from the experimental multi-process cluster to an enterprise-grade one, grounded in the 2026-06-10 live stress-test baselines - [PRODUCT_ROADMAP.md](planning/PRODUCT_ROADMAP.md) · [architecture-review.md](planning/architecture-review.md) · [roadmap-cohort-analysis.md](planning/roadmap-cohort-analysis.md) · [site-cohort-analysis.md](planning/site-cohort-analysis.md) - Per-milestone phase/task archive: `planning/milestone-0,1,2,3,5,7,8,9,10,p/` diff --git a/docs/planning/ROADMAP.md b/docs/planning/ROADMAP.md index b8569f3..6b3db1e 100644 --- a/docs/planning/ROADMAP.md +++ b/docs/planning/ROADMAP.md @@ -2895,6 +2895,12 @@ These phases take the proven in-process primitives and deliver actual multi-node ### Post-M8 Follow-ups (tracked, out of M8 scope) +> **Phase plan:** G4 and G5 (plus the full enterprise-cluster gap inventory — +> replication throughput, unified log, membership/discovery, sharding × RF, +> security, observability, chaos CI) are sequenced with measured baselines and +> testable exit gates in [docs/roadmap-to-cluster.md](../roadmap-to-cluster.md) +> (M11 candidate, drafted 2026-06-10 from live stress-test evidence). + | Gap | Origin | Severity | Description | |-----|--------|----------|-------------| | G4 | post-M8 | Medium | **Quorum-ack write contract.** A `204` from `/signals` (and the data writes) asserts leader durability only (storage + WAL fsync); the follower ship is best-effort and no follower acknowledgement is asserted. A write contract that blocks for N-follower acks is future work — explicitly NOT part of m8p10. (Runbook §8 documents the current leader-durable contract.) | diff --git a/docs/roadmap-to-cluster.md b/docs/roadmap-to-cluster.md new file mode 100644 index 0000000..5178574 --- /dev/null +++ b/docs/roadmap-to-cluster.md @@ -0,0 +1,373 @@ +# Roadmap to an Enterprise-Grade Cluster + +**Status:** Proposed (M11 candidate) · **Date:** 2026-06-10 · **Baseline evidence:** +[stress-test-thepeach.md](ops/stress-test-thepeach.md), [cluster runbook](runbooks/cluster.md), +[ROADMAP M8 Known Gaps](planning/ROADMAP.md) (G4/G5/G6), live k3s deployment (3 regions × 2-vCPU pods). + +This is the gap analysis and phase plan from today's **experimental** multi-process +cluster (m8p10 — real process isolation, leader-durable writes, operator-driven +failover) to a cluster a paying customer could run as a system of record for +signals: **quorum-durable, self-healing, horizontally scalable, secured, +observable, and continuously chaos-tested.** Everything in §1 is measured or +code-verified, not aspirational; every phase in §4 ends in a testable exit gate. + +--- + +## 1. Where we are today (evidence) + +### 1.1 Measured baselines (2026-06-10, live 3-region k3s cluster, cpu-limit `"2"`/pod) + +| Path | Measured | Notes | +|---|---|---| +| Embedded engine signal write | **~82 ns** (~12M/s/core) | criterion `scale.rs`; the engine itself is not the problem | +| Embedded RETRIEVE p99 | **152 µs** @ 1M items | scale baselines | +| Cluster `/feed` (ranked read) | **p99 8–15 ms** at 100→4,000 rps | never degraded in any run | +| Cluster `/search` | p99 3–11 ms | includes `reload_text_index` per request | +| **Replicated `/signals` (leader path)** | **~90 signals/s ceiling** | ok/s pinned at ~75 regardless of offered rate (100→4,000 rps); ~178–210 ms per write | +| **`/sharded/signals` (unreplicated)** | **3,669 signals/s, 0 errors, ~27% cluster CPU** | knee NOT reached; single generator became the limit | +| Cross-region replication lag | 0–3 events under all load | SLA <2 s; measured ~110–133 ms p99 (runbook) | +| Manual failover (promote→write) | ~31–34 ms | SLA <10 s | +| Overload behavior | **Graceful**: HTTP 429 shed, zero 5xx, **zero pod restarts**, no acknowledged-write loss observed | the strongest enterprise trait we have today | +| Aggregate HTTP ceiling | ~5k rps connection-establishment wall (conntrack/accept), cluster CPU idle | networking, not engine | + +Ballpark peers on small nodes (context, not benchmarks we ran): etcd ≈10k +fsynced consensus writes/s; CockroachDB low-thousands/node; Cassandra QUORUM +10k+/node. **The replicated write path is 1–2 orders of magnitude below the +enterprise floor; everything else is competitive or better.** + +### 1.2 What already exists and is worth keeping + +- Real per-process region isolation; kill the leader → survivors keep serving (proven live). +- A real WAL relay over gRPC (`tidal-net` `WalShipping/ShipSegment`) with BLAKE3-verified + segments, contiguous-frontier apply (no gap-swallowing), per-peer circuit breakers. +- Honest, documented durability contract (runbook §8) — rare and valuable; we extend it, not retrofit honesty. +- Operator verbs that work and are runbook-verified op-for-op (`promote`, `partition`, `heal`, `reconcile`), + CRDT/LWW convergence for user-scoped hard-negatives, HLC with skew injection for tests. +- Sharded scatter-gather reads with degraded-not-failed semantics and shard-dedup merge. +- Tier-3 chaos/lifecycle/runbook suites over **real OS processes with real TCP-relay partition injection** — + the scaffolding §4/p9 industrializes. +- Graceful backpressure end to end (write-pool 429 + engine WAL-channel 429 + breaker isolation). + +### 1.3 Gap inventory → phase map + +| Dimension | Today | Enterprise bar | Phase | +|---|---|---|---| +| Replication throughput | ~90/s: synchronous, sequential, per-request ship; 2-worker pool | Batched/pipelined, async; ≥2k/s on today's hardware | **p1** | +| Replication unity | Signals ride the WAL; items/embeddings ride a best-effort HTTP side channel (source of both live bugs we fixed: marker-gated broadcast, 401'd heal backfill); heal is O(items) | **One replicated log** for all replicated data; heal = log catch-up | **p2** | +| Durability | 2xx = leader fsync only; follower ship best-effort | `ack=quorum` mode: 2xx = majority-durable; zero acknowledged loss on any single-node kill | **p3** | +| Availability | Operator-driven promote; **restarted node re-reads topology and believes the static leader again** (observed live: us-east rejoined claiming leadership while eu-west led — split-brain seed) | Failure detector + automatic election + **term fencing** + durable leadership | **p4** | +| Membership | Static per-pod topology files with **literal IPs** (`grpc_addr` is `SocketAddr::parse` — no DNS); add/remove node = redeploy everything | Dynamic membership, DNS/seed discovery, online add/remove/replace with snapshot+stream catch-up (`StreamSegments` is declared but unimplemented) | **p5** | +| Write scaling | EITHER replicated (1 leader for everything) OR sharded (no replication); shards conflated with regions | **Sharding × replication**: shard groups with RF and distributed leaders; rebalancing | **p6** | +| Security | Plaintext gRPC replication (mTLS config exists, unexercised); one static bearer shared by all; internal marker as trust signal; no audit log | mTLS default + rotation, authenticated inter-node calls, admin audit, at-rest story | **p7** | +| Observability | **Cluster mode has no `/metrics` listener at all**; no request-id propagation on cluster routers; lag accounting wrong after promote (ShardId(0)-keyed); leader's own status row reads 0 | Per-node Prometheus metrics, tracing across forward/ship hops, truthful status, dashboards + alerts | **p8** (starts in p1) | +| Ops lifecycle | Manual heal loops (first heal can no-op into an open breaker); per-node backup only; rolling-upgrade suite exists but not a CI gate | Coordinated cluster backup/restore + PITR; upgrade-under-load as a release gate; self-driving heal | **p8** | +| Correctness assurance | Tier-3 suites are opt-in (`cluster-e2e` feature), run manually | Nightly chaos + soak in CI with invariant checkers; 30-day green before GA | **p9** | + +### 1.4 Live incidents from this deployment that became requirements + +1. **Restart amnesia / split-brain seed (→ p4).** After SIGKILL-failover, the + restarted ex-leader booted from its static topology believing `leader=us-east` + while the cluster had promoted eu-west; an operator re-promote reconciled it. + With writes flowing through both gateways this is a dual-leader window. + Requirement: durable terms, fencing on ship/forward, rejoin-as-follower. +2. **The HTTP side channel is where the bugs live (→ p2).** Both correctness bugs + found live were in item/embedding broadcast, not the WAL relay: (a) forwarded + writes never broadcast (terminated on the internal marker instead of + leadership), (b) heal's backfill carried no bearer and 401'd forever. The WAL + path was flawless throughout. Conclusion: replicate everything through the log. +3. **Breaker eats the first heal (→ p8).** After a partition window the per-peer + breaker (threshold 5 / reset 30 s) is open; the first `/cluster/heal` no-ops. + Runbook says "re-issue until lag 0" — an operator footgun; the server should + drive heal-until-converged itself. +4. **The 178 ms ship (→ p1).** A LAN gRPC hop plus group-commit fsync does not + cost 178 ms. Something structural (per-call setup, flow control, synchronous + follower apply, sequential per-peer sends) is being paid per request. Profile + before redesigning; the fix is likely cheaper than it looks. + +--- + +## 2. Definition of done — the enterprise guarantees + +Each guarantee is a **testable claim** with a named owner-test by GA (p9 wires any +that don't exist earlier). These are the bar; the phases are the path. + +- **G-D (Durability).** With `ack=quorum`, a 2xx response means the write is + durably applied on a majority of the shard's replicas. SIGKILL any single node + at any moment under load: **zero acknowledged-write loss**, proven by a ledger + checker that replays client acks against post-recovery state. +- **G-A (Availability).** Loss of any single node: reads continue uninterrupted; + writes resume automatically in **<10 s p99** with no operator action and no + dual-leader window (fencing proven under partition + restart chaos). +- **G-S (Scalability).** Write throughput scales with shard count at RF=3 + (≥2.5× from 1→3 shards on the same hardware). Reads scale with replicas. +- **G-E (Elasticity).** Nodes can be added/removed/replaced online; a new replica + catches up via snapshot + WAL stream; p99 impact bounded (<2× for <60 s). +- **G-Sec (Security).** All inter-node links mTLS; replication endpoints + authenticated (no implicit trust of the network); keys/certs rotate without + downtime; admin verbs audit-logged. +- **G-O (Observability).** Every node exports Prometheus metrics; a dashboard + answers "what is happening / what is wrong" without reading code; alerts on + the golden signals + replication lag + commit-index stall + election churn. +- **G-Op (Operability).** Rolling upgrade with N/N+1 version skew under load is a + CI-verified release gate; coordinated backup/restore (and WAL-archival PITR) + drilled and timed; runbooks executable by a non-author. +- **G-C (Continuous correctness).** Nightly chaos (partitions, crashes, skew, + disk faults) + soak (tidal-stress) green for 30 consecutive days before GA. + +--- + +## 3. Reference environments and performance targets + +Two named environments so numbers stay comparable end to end: + +- **Ref-A (continuity, CI-able):** the current k3s deployment — 3 × (2 vCPU / + 2 GiB, local-path PV). All measured baselines in §1.1 are Ref-A. Phase exit + gates are stated against Ref-A so they're re-runnable with `tidal-stress`. +- **Ref-B (enterprise reference):** 3–9 × (8 vCPU / 16 GiB, NVMe). GA marketing + numbers come from Ref-B; claiming "enterprise" on shared 2-core pods is noise. + +| Target | Ref-A (gate) | Ref-B (GA target) | +|---|---|---| +| Replicated signal writes, `ack=leader` | ≥2,000/s (p1) | ≥25,000/s | +| Replicated signal writes, `ack=quorum` | ≥1,000/s (p3) | ≥15,000/s | +| 3-shard × RF=3 quorum writes | ≥5,000/s (p6) | ≥50,000/s | +| Ranked read p99 (under write load) | ≤50 ms | ≤20 ms @ 10k reads/s | +| Auto-failover (detect→writes resume) | <10 s p99 (p4) | <5 s p99 | +| Replication lag p99 (sustained load) | ≤2 s | ≤500 ms | +| New-replica catch-up (100k items) | ≤5 min (p5) | ≤1 min | + +--- + +## 4. The phases (proposed M11, `m11p1`–`m11p9`) + +Ordering rule: **make the one log fast and unified before making it quorum; make +it quorum before electing leaders over it; fix membership before sharding it; +secure and industrialize continuously.** + +### m11p1 — Replication performance floor (size: M) +**Goal:** the existing leader-durable path stops being the embarrassment line: +≥20× today's ceiling with no semantic change. + +- **Profile first.** Instrument the ship path (this phase ships the first + `tidaldb_cluster_*` metrics: fsync latency, ship RTT, queue depths, breaker + state) and explain the measured ~178 ms/write. Suspects: per-call channel + setup, tonic flow-control, synchronous follower-side apply in the RPC, the + sequential per-peer loop in `relay.write_and_ship`, group-commit interaction. +- **Decouple ack from ship.** The client ack already only promises leader + durability — so return at group-commit fsync; move shipping fully off the + request path into per-peer outbound queues drained by dedicated senders + (the relay/redeliver machinery already exists; make it the only path). +- **Batch + pipeline.** Ship WAL batches (not per-request payloads) with a + windowed in-flight budget per peer (N batches outstanding, acks advance the + window) instead of one blocking unary call at a time. +- **Rework the write pool.** `/signals` no longer needs a blocking gRPC worker + per request; the pool sizes from CPU for fsync/apply work only. Keep 429 + semantics (bounded queue) — the graceful shed is a feature. +- **Tune group commit** (batch 100 / 10 ms today) against measured fsync cost on + Ref-A's local-path PV; make both knobs config. +- Non-goals: no ack-mode change, no membership change. +- **Exit gate (Ref-A, `tidal-stress` leader path):** ≥2,000 signals/s OK at + p99 ≤50 ms, error rate <1%, lag p99 ≤2 s sustained 10 min; replication metrics + visible in Prometheus. + +### m11p2 — One replicated log (size: L) +**Goal:** every replicated mutation rides the WAL relay; the HTTP broadcast side +channel (items/embeddings) is deleted. + +- Add item-metadata and embedding records to the WAL event model (they're + config-like, low-rate — the log absorbs them trivially; 64 MiB segment ceiling + is ample for 1536-dim vectors). +- Followers apply items/embeddings from the log; leader broadcast code and the + O(items) heal re-broadcast are removed (heal becomes pure log catch-up). +- Implement **`StreamSegments`** (declared, unimplemented) as the catch-up path: + a follower that reports applied=N gets a stream from N+1; the in-memory + `batch_log` dependency for recovery goes away (read back from the durable WAL). +- Hard-negatives stay CRDT (user-scoped, commutative) — document the split: + "log = totally-ordered global data; CRDT = per-user convergent data". +- **Exit gate:** chaos tests for both 2026-06-10 bug classes pass by + construction (no marker-gated fan-out to regress, no authed side POSTs); + item written anywhere is readable everywhere ≤2 s p99; heal of a 10-minute-down + follower with 100k items completes via stream without O(items) HTTP traffic. + +### m11p3 — Quorum-acked writes (size: L) — closes ROADMAP gap **G4** +**Goal:** an opt-in durability contract a system of record can sit on. + +- `ack=leader|quorum` (deployment default + per-request override header). +- Follower ship responses carry **durably-applied seqno** (fsync'd, not just + received); leader maintains per-peer durable high-water marks and a + **commit index** = quorum floor; `ack=quorum` responses gate on the commit + index passing the write's seq. Pipelined (batch-level acks), never one-at-a-time. +- Timeout semantics: quorum not reached in budget → retryable 503 naming the + laggard; the write may still commit (document at-least-once + dedup guidance; + evaluate optional client idempotency keys for signal accounting — open + question, decide in-phase). +- Rewrite runbook §8 from "honest about what we don't promise" to "here's the + knob and its cost". +- **Exit gate:** SIGKILL the leader under `ack=quorum` load → ledger checker + proves zero acknowledged loss across 100 random kill-points; quorum throughput + ≥50% of p1's leader-ack number (≥1,000/s Ref-A). + +### m11p4 — Failure detection, election, fencing (size: XL) — closes ROADMAP gap **G5** +**Goal:** "a machine died" is a non-event, not a runbook page. + +- **Replicated metadata state** (term/epoch, leader id, membership): Raft-style + election over the existing transport — the quorum machinery from p3 is the + prerequisite. The data plane stays the WAL relay; only the small control state + is consensus-managed. +- Failure detector on the existing `Heartbeat` RPC (timeout + jitter; tunable). +- **Fencing everywhere:** terms stamped on ship/forward/promote; stale-term + ships rejected by followers; a restarted node **rejoins as follower** and + learns leadership from the metadata state — never from the static topology + file (kills incident §1.4-1). +- Leadership is durable locally (survives restart) and lease-bounded; manual + `/cluster/promote` becomes a fenced transfer (drain for maintenance), not the + availability mechanism. +- **Exit gate (chaos, Ref-A):** kill the leader under load → writes resume + <10 s p99 with zero operator action and zero acked loss; partition the old + leader away mid-election and let it restart → it cannot accept a single write + (fencing proven); election churn under flapping links bounded (no livelock). + +### m11p5 — Membership, discovery, elasticity (size: L) +**Goal:** nodes are cattle; topology is data, not files. + +- Resolve `grpc_addr` via DNS (drop the `SocketAddr::parse`-only constraint) and + support **seed-based join**: a new node contacts any seed, gets the membership + + topology from the metadata state. The per-pod static-ClusterIP topology + ConfigMap hack dies. +- Conf-changes (add/remove/replace) through the p4 metadata consensus; joiners + catch up via **snapshot + StreamSegments** (snapshot = the engine's existing + checkpoint artifacts, shipped as a stream). +- Kubernetes reference becomes one StatefulSet + headless Service peer + discovery, PDB, topology-spread; rolling node replace is `kubectl delete pod`. +- **Exit gate:** scale 3→5→3 online under load: joiner catches up ≤5 min + (100k items), serves quorum after; p99 impact <2× for <60 s; zero loss. + +### m11p6 — Sharding × replication + rebalancing (size: XL) +**Goal:** writes scale horizontally **without** giving up replication — the +current "replicated XOR sharded" split ends. + +- Shard groups: keyspace hash-split (reuse `ShardRouter`) into S shards, each a + replication group at RF (default 3) with its own WAL/relay/commit-index and + its own elected leader; leaders balanced across nodes. +- Regions stop being shards: topology = nodes × shard-replicas, with optional + zone/region labels for placement and read affinity. +- Routing: any gateway routes by entity hash to the shard leader (write) or a + local/nearest replica (read); scatter-gather already merges cross-shard reads + with degraded semantics — now over shard groups. +- Rebalancing: shard move = snapshot + stream + fenced cutover; operator- + triggered first, automatic (rate-limited) later. +- **Exit gate (Ref-A, 3 shards × RF=3):** ≥5,000 quorum signals/s (≥2.5× + single-shard p3); kill any node → only its shard-leaderships move (<10 s), + reads never stop; `tidal-stress` sharded-vs-replicated comparison collapses + into one path. + +### m11p7 — Security hardening (size: M) +**Goal:** the cluster stops trusting the network. + +- **mTLS default** on gRPC replication (the `grpc_tls` config exists — exercise + it, flip plaintext to an explicit `insecure: true` with a loud startup WARN); + cert + bearer **rotation without restart** (file/secret watch). +- Authenticated inter-node HTTP: per-node identity (client certs or signed + internal tokens); the `x-tidal-internal` marker remains a routing hint only + (it already isn't an auth bypass — keep it that way by test). +- Audit log for admin verbs (promote/heal/partition/conf-change) with principal, + term, and outcome; at-rest encryption documented (delegate to volume + encryption, or in-engine as stretch); per-principal rate limits (the engine's + rate limiter exists — wire it to the HTTP layer). +- **Exit gate:** reference deployment has zero plaintext inter-node links; + rotation under load drops zero requests; a foreign pod on the cluster network + can neither ship segments nor call internal routes (negative tests). + +### m11p8 — Observability + operations (size: M; **starts inside p1**) +**Goal:** operable by someone who didn't build it. + +- Complete the `tidaldb_cluster_*` metric set (lag, commit index, ship + latency/queue, breaker, elections, forwards, write-pool, fsync) on a cluster + `/metrics` listener (**today: none**); Grafana dashboard + alert rules shipped + beside the existing standalone ones. +- Request-id propagation + tracing spans across forward/broadcast/ship hops + (the cluster routers currently skip the request-id layer). +- Truthful status: fix post-promote lag accounting (ShardId(0) keying) and the + leader's own applied row; `/cluster/status` becomes the single pane. +- **Self-driving heal:** server-side heal-until-converged (retry through breaker + resets) so "re-issue heal until lag 0" leaves the runbook (incident §1.4-3). +- Coordinated backup/restore: cluster-consistent snapshot manifest across shard + groups + WAL archival hook for PITR; timed restore drill in the runbook. +- Rolling upgrade: version handshake on ship/forward, N/N+1 skew documented; + `mp_rolling_upgrade_no_loss_no_stall` promoted to a release-gate CI job. +- **Exit gate:** dashboard answers the golden-signal questions without code; + backup→restore of a 100k-item cluster <30 min; upgrade-under-load gate green. + +### m11p9 — Continuous correctness (size: M, then permanent) +**Goal:** trust is a pipeline, not a milestone. + +- Nightly CI: the tier-3 chaos suites (partition relay, crash, clock-skew via + `TIDAL_HLC_SKEW_MS`, rolling upgrade) + new fault classes (disk-full, + slow-fsync injection, asymmetric partitions) against a Ref-A cluster. +- Invariant checkers as first-class: no-acked-loss ledger replay, per-entity + monotonic counters, cross-replica decay parity (1e-6), membership safety + (single leader per term per shard). Jepsen-style external harness as stretch. +- Nightly `tidal-stress` soak (1× 100k-DAU model, 1 h) with regression gates on + p99/throughput; results archived for trend lines. +- **Exit gate (GA):** every guarantee in §2 maps to a named automated test; + nightly suite green 30 consecutive days. + +--- + +## 5. Sequencing, sizing, releases + +| Phase | Depends on | Size | Closes | +|---|---|---|---| +| p1 perf floor | — | M | §1.4-4, throughput floor | +| p2 one log | p1 | L | §1.4-2, StreamSegments, O(items) heal | +| p3 quorum acks | p1, p2 | L | ROADMAP **G4** | +| p4 election + fencing | p3 | XL | ROADMAP **G5**, §1.4-1 | +| p5 membership | p4 | L | static-IP topology, elasticity | +| p6 shards × RF | p3, p5 | XL | write scaling | +| p7 security | p2 (stable surface) | M | network trust | +| p8 observability/ops | seeds in p1; completes after p6 | M | §1.4-3, cluster metrics | +| p9 chaos CI | grows from p3; gate at end | M | GA trust bar | + +**Release waves:** + +- **v0.9 "Credible HA"** = p1–p4: one shard done *right* — fast, unified log, + quorum-durable, self-healing. This alone moves the §1.1 comparison from + "1–2 orders below peers" to "in-band". +- **v1.0 "Scalable"** = p5–p6: elastic membership, sharding × replication. +- **v1.1 "Enterprise"** = p7–p9 complete (p7/p8 work interleaves earlier; the + release is the gate, not the start). + +Also explicitly **not blocked on this roadmap:** the embedded engine (thepeach's +integration path) and the standalone server remain the recommended production +deployments today — they already clear their bars (82 ns writes; p99 <15 ms +reads; the stress test's 100k-DAU verdict). + +## 6. Non-goals (scope honesty, per VISION.md) + +- WAN active-active strong consistency. Cross-region stays async (log shipping) + with the CRDT tier for user-scoped convergent data; regions are placement + labels, not consistency domains. +- SQL, general transactions, multi-key serializability. The data model is + signals (commutative accumulation), items/embeddings (idempotent upserts), + hard-negatives (LWW CRDT) — the log + quorum gives these everything they need. +- Building a service mesh. mTLS + authenticated RPC yes; Istio no. +- Replacing the embedded mode. The cluster is for teams that *can't* embed; + the engine stays the product's core. + +## 7. Open questions (decide in-phase, tracked here so they don't vanish) + +1. Client idempotency for signal retries under `ack=quorum` timeouts (p3): + dedup window keyed by client request id vs. documented at-least-once. +2. Election engine: minimal purpose-built Raft over `tidal-net` vs. embedding a + vetted raft crate — decide in p4 design after p3's quorum machinery exists. +3. Shard count / split policy (static S at create vs. range-split later) — p6. +4. At-rest encryption: in-engine vs. delegated to volume encryption — p7. +5. Whether `tidal-stress` grows a distributed multi-generator mode to push past + the ~5k rps connection wall before p6's exit gate needs it. + +--- + +*Companion docs: [stress-test-thepeach.md](ops/stress-test-thepeach.md) (baselines), +[runbooks/cluster.md](runbooks/cluster.md) (today's operational truth), +[ops/capacity-planning.md](ops/capacity-planning.md) (per-node sizing math), +[planning/ROADMAP.md](planning/ROADMAP.md) (M8 history + G4/G5/G6 gap registry).*