jordan/tidaldb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: roadmap to an enterprise-grade cluster (M11 candidate)

Browse Source 
Gap analysis + 9-phase plan (m11p1-p9) from the experimental m8p10 cluster to an
enterprise-grade one, grounded in the 2026-06-10 live stress-test baselines:
replicated /signals ~90/s vs sharded 3,669/s vs embedded ~82ns, plus four
live-observed incidents (restart leadership amnesia, side-channel broadcast bugs,
breaker-eaten heal, the 178ms ship) promoted to requirements. Phases: perf floor,
one replicated log, quorum acks (G4), election+fencing (G5), membership/discovery,
sharding x RF, security, observability/ops, chaos CI — each with a testable
Ref-A exit gate; release waves v0.9 credible / v1.0 scalable / v1.1 enterprise.
Indexed in docs/README.md; pointer from ROADMAP's Post-M8 Follow-ups.
This commit is contained in:
jx12n 2026-06-10 22:20:15 -06:00
parent f640764d89
commit 7e8b7b013a
3 changed files with 380 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
docs/README.md
View File

@ -25,6 +25,7 @@ The authoritative component specifications (status: Implemented, M0–M8).
## Planning — `planning/`
- [**ROADMAP.md**](planning/ROADMAP.md) — milestones M0–M10, phase status, known gaps
- [**roadmap-to-cluster.md**](roadmap-to-cluster.md) — gap analysis + phase plan (M11 candidate) from the experimental multi-process cluster to an enterprise-grade one, grounded in the 2026-06-10 live stress-test baselines
- [PRODUCT_ROADMAP.md](planning/PRODUCT_ROADMAP.md) · [architecture-review.md](planning/architecture-review.md) · [roadmap-cohort-analysis.md](planning/roadmap-cohort-analysis.md) · [site-cohort-analysis.md](planning/site-cohort-analysis.md)
- Per-milestone phase/task archive: `planning/milestone-0,1,2,3,5,7,8,9,10,p/`

6
docs/planning/ROADMAP.md
View File

@ -2895,6 +2895,12 @@ These phases take the proven in-process primitives and deliver actual multi-node
### Post-M8 Follow-ups (tracked, out of M8 scope)
> **Phase plan:** G4 and G5 (plus the full enterprise-cluster gap inventory —
> replication throughput, unified log, membership/discovery, sharding × RF,
> security, observability, chaos CI) are sequenced with measured baselines and
> testable exit gates in [docs/roadmap-to-cluster.md](../roadmap-to-cluster.md)
> (M11 candidate, drafted 2026-06-10 from live stress-test evidence).
| Gap | Origin | Severity | Description |
|-----|--------|----------|-------------|
| G4 | post-M8 | Medium | **Quorum-ack write contract.** A `204` from `/signals` (and the data writes) asserts leader durability only (storage + WAL fsync); the follower ship is best-effort and no follower acknowledgement is asserted. A write contract that blocks for N-follower acks is future work — explicitly NOT part of m8p10. (Runbook §8 documents the current leader-durable contract.) |

373
docs/roadmap-to-cluster.md Normal file
View File

@ -0,0 +1,373 @@
# Roadmap to an Enterprise-Grade Cluster
**Status:** Proposed (M11 candidate) · **Date:** 2026-06-10 · **Baseline evidence:**
[stress-test-thepeach.md](ops/stress-test-thepeach.md), [cluster runbook](runbooks/cluster.md),
[ROADMAP M8 Known Gaps](planning/ROADMAP.md) (G4/G5/G6), live k3s deployment (3 regions × 2-vCPU pods).
This is the gap analysis and phase plan from today's **experimental** multi-process
cluster (m8p10 — real process isolation, leader-durable writes, operator-driven
failover) to a cluster a paying customer could run as a system of record for
signals: **quorum-durable, self-healing, horizontally scalable, secured,
observable, and continuously chaos-tested.** Everything in §1 is measured or
code-verified, not aspirational; every phase in §4 ends in a testable exit gate.
---
## 1. Where we are today (evidence)
### 1.1 Measured baselines (2026-06-10, live 3-region k3s cluster, cpu-limit `"2"`/pod)
| Path | Measured | Notes |
|---|---|---|
| Embedded engine signal write | **~82 ns** (~12M/s/core) | criterion `scale.rs`; the engine itself is not the problem |
| Embedded RETRIEVE p99 | **152 µs** @ 1M items | scale baselines |
| Cluster `/feed` (ranked read) | **p99 8–15 ms** at 100→4,000 rps | never degraded in any run |
| Cluster `/search` | p99 3–11 ms | includes `reload_text_index` per request |
| **Replicated `/signals` (leader path)** | **~90 signals/s ceiling** | ok/s pinned at ~75 regardless of offered rate (100→4,000 rps); ~178–210 ms per write |
| **`/sharded/signals` (unreplicated)** | **3,669 signals/s, 0 errors, ~27% cluster CPU** | knee NOT reached; single generator became the limit |
| Cross-region replication lag | 0–3 events under all load | SLA <2 s; measured ~110–133 ms p99 (runbook) |
| Manual failover (promote→write) | ~31–34 ms | SLA <10 s |
| Overload behavior | **Graceful**: HTTP 429 shed, zero 5xx, **zero pod restarts**, no acknowledged-write loss observed | the strongest enterprise trait we have today |
| Aggregate HTTP ceiling | ~5k rps connection-establishment wall (conntrack/accept), cluster CPU idle | networking, not engine |
Ballpark peers on small nodes (context, not benchmarks we ran): etcd ≈10k
fsynced consensus writes/s; CockroachDB low-thousands/node; Cassandra QUORUM
10k+/node. **The replicated write path is 1–2 orders of magnitude below the
enterprise floor; everything else is competitive or better.**
### 1.2 What already exists and is worth keeping
- Real per-process region isolation; kill the leader → survivors keep serving (proven live).
- A real WAL relay over gRPC (`tidal-net` `WalShipping/ShipSegment`) with BLAKE3-verified
segments, contiguous-frontier apply (no gap-swallowing), per-peer circuit breakers.
- Honest, documented durability contract (runbook §8) — rare and valuable; we extend it, not retrofit honesty.
- Operator verbs that work and are runbook-verified op-for-op (`promote`, `partition`, `heal`, `reconcile`),
CRDT/LWW convergence for user-scoped hard-negatives, HLC with skew injection for tests.
- Sharded scatter-gather reads with degraded-not-failed semantics and shard-dedup merge.
- Tier-3 chaos/lifecycle/runbook suites over **real OS processes with real TCP-relay partition injection** —
the scaffolding §4/p9 industrializes.
- Graceful backpressure end to end (write-pool 429 + engine WAL-channel 429 + breaker isolation).
### 1.3 Gap inventory → phase map
| Dimension | Today | Enterprise bar | Phase |
|---|---|---|---|
| Replication throughput | ~90/s: synchronous, sequential, per-request ship; 2-worker pool | Batched/pipelined, async; ≥2k/s on today's hardware | **p1** |
| Replication unity | Signals ride the WAL; items/embeddings ride a best-effort HTTP side channel (source of both live bugs we fixed: marker-gated broadcast, 401'd heal backfill); heal is O(items) | **One replicated log** for all replicated data; heal = log catch-up | **p2** |
| Durability | 2xx = leader fsync only; follower ship best-effort | `ack=quorum` mode: 2xx = majority-durable; zero acknowledged loss on any single-node kill | **p3** |
| Availability | Operator-driven promote; **restarted node re-reads topology and believes the static leader again** (observed live: us-east rejoined claiming leadership while eu-west led — split-brain seed) | Failure detector + automatic election + **term fencing** + durable leadership | **p4** |
| Membership | Static per-pod topology files with **literal IPs** (`grpc_addr` is `SocketAddr::parse` — no DNS); add/remove node = redeploy everything | Dynamic membership, DNS/seed discovery, online add/remove/replace with snapshot+stream catch-up (`StreamSegments` is declared but unimplemented) | **p5** |
| Write scaling | EITHER replicated (1 leader for everything) OR sharded (no replication); shards conflated with regions | **Sharding × replication**: shard groups with RF and distributed leaders; rebalancing | **p6** |
| Security | Plaintext gRPC replication (mTLS config exists, unexercised); one static bearer shared by all; internal marker as trust signal; no audit log | mTLS default + rotation, authenticated inter-node calls, admin audit, at-rest story | **p7** |
| Observability | **Cluster mode has no `/metrics` listener at all**; no request-id propagation on cluster routers; lag accounting wrong after promote (ShardId(0)-keyed); leader's own status row reads 0 | Per-node Prometheus metrics, tracing across forward/ship hops, truthful status, dashboards + alerts | **p8** (starts in p1) |
| Ops lifecycle | Manual heal loops (first heal can no-op into an open breaker); per-node backup only; rolling-upgrade suite exists but not a CI gate | Coordinated cluster backup/restore + PITR; upgrade-under-load as a release gate; self-driving heal | **p8** |
| Correctness assurance | Tier-3 suites are opt-in (`cluster-e2e` feature), run manually | Nightly chaos + soak in CI with invariant checkers; 30-day green before GA | **p9** |
### 1.4 Live incidents from this deployment that became requirements
1. **Restart amnesia / split-brain seed (→ p4).** After SIGKILL-failover, the
restarted ex-leader booted from its static topology believing `leader=us-east`
while the cluster had promoted eu-west; an operator re-promote reconciled it.
With writes flowing through both gateways this is a dual-leader window.
Requirement: durable terms, fencing on ship/forward, rejoin-as-follower.
2. **The HTTP side channel is where the bugs live (→ p2).** Both correctness bugs
found live were in item/embedding broadcast, not the WAL relay: (a) forwarded
writes never broadcast (terminated on the internal marker instead of
leadership), (b) heal's backfill carried no bearer and 401'd forever. The WAL
path was flawless throughout. Conclusion: replicate everything through the log.
3. **Breaker eats the first heal (→ p8).** After a partition window the per-peer
breaker (threshold 5 / reset 30 s) is open; the first `/cluster/heal` no-ops.
Runbook says "re-issue until lag 0" — an operator footgun; the server should
drive heal-until-converged itself.
4. **The 178 ms ship (→ p1).** A LAN gRPC hop plus group-commit fsync does not
cost 178 ms. Something structural (per-call setup, flow control, synchronous
follower apply, sequential per-peer sends) is being paid per request. Profile
before redesigning; the fix is likely cheaper than it looks.
---
## 2. Definition of done — the enterprise guarantees
Each guarantee is a **testable claim** with a named owner-test by GA (p9 wires any
that don't exist earlier). These are the bar; the phases are the path.
- **G-D (Durability).** With `ack=quorum`, a 2xx response means the write is
durably applied on a majority of the shard's replicas. SIGKILL any single node
at any moment under load: **zero acknowledged-write loss**, proven by a ledger
checker that replays client acks against post-recovery state.
- **G-A (Availability).** Loss of any single node: reads continue uninterrupted;
writes resume automatically in **<10 s p99** with no operator action and no
dual-leader window (fencing proven under partition + restart chaos).
- **G-S (Scalability).** Write throughput scales with shard count at RF=3
(≥2.5× from 1→3 shards on the same hardware). Reads scale with replicas.
- **G-E (Elasticity).** Nodes can be added/removed/replaced online; a new replica
catches up via snapshot + WAL stream; p99 impact bounded (<2× for <60 s).
- **G-Sec (Security).** All inter-node links mTLS; replication endpoints
authenticated (no implicit trust of the network); keys/certs rotate without
downtime; admin verbs audit-logged.
- **G-O (Observability).** Every node exports Prometheus metrics; a dashboard
answers "what is happening / what is wrong" without reading code; alerts on
the golden signals + replication lag + commit-index stall + election churn.
- **G-Op (Operability).** Rolling upgrade with N/N+1 version skew under load is a
CI-verified release gate; coordinated backup/restore (and WAL-archival PITR)
drilled and timed; runbooks executable by a non-author.
- **G-C (Continuous correctness).** Nightly chaos (partitions, crashes, skew,
disk faults) + soak (tidal-stress) green for 30 consecutive days before GA.
---
## 3. Reference environments and performance targets
Two named environments so numbers stay comparable end to end:
- **Ref-A (continuity, CI-able):** the current k3s deployment — 3 × (2 vCPU /
2 GiB, local-path PV). All measured baselines in §1.1 are Ref-A. Phase exit
gates are stated against Ref-A so they're re-runnable with `tidal-stress`.
- **Ref-B (enterprise reference):** 3–9 × (8 vCPU / 16 GiB, NVMe). GA marketing
numbers come from Ref-B; claiming "enterprise" on shared 2-core pods is noise.
| Target | Ref-A (gate) | Ref-B (GA target) |
|---|---|---|
| Replicated signal writes, `ack=leader` | ≥2,000/s (p1) | ≥25,000/s |
| Replicated signal writes, `ack=quorum` | ≥1,000/s (p3) | ≥15,000/s |
| 3-shard × RF=3 quorum writes | ≥5,000/s (p6) | ≥50,000/s |
| Ranked read p99 (under write load) | ≤50 ms | ≤20 ms @ 10k reads/s |
| Auto-failover (detect→writes resume) | <10 s p99 (p4) | <5 s p99 |
| Replication lag p99 (sustained load) | ≤2 s | ≤500 ms |
| New-replica catch-up (100k items) | ≤5 min (p5) | ≤1 min |
---
## 4. The phases (proposed M11, `m11p1`–`m11p9`)
Ordering rule: **make the one log fast and unified before making it quorum; make
it quorum before electing leaders over it; fix membership before sharding it;
secure and industrialize continuously.**
### m11p1 — Replication performance floor (size: M)
**Goal:** the existing leader-durable path stops being the embarrassment line:
≥20× today's ceiling with no semantic change.
- **Profile first.** Instrument the ship path (this phase ships the first
`tidaldb_cluster_*` metrics: fsync latency, ship RTT, queue depths, breaker
state) and explain the measured ~178 ms/write. Suspects: per-call channel
setup, tonic flow-control, synchronous follower-side apply in the RPC, the
sequential per-peer loop in `relay.write_and_ship`, group-commit interaction.
- **Decouple ack from ship.** The client ack already only promises leader
durability — so return at group-commit fsync; move shipping fully off the
request path into per-peer outbound queues drained by dedicated senders
(the relay/redeliver machinery already exists; make it the only path).
- **Batch + pipeline.** Ship WAL batches (not per-request payloads) with a
windowed in-flight budget per peer (N batches outstanding, acks advance the
window) instead of one blocking unary call at a time.
- **Rework the write pool.** `/signals` no longer needs a blocking gRPC worker
per request; the pool sizes from CPU for fsync/apply work only. Keep 429
semantics (bounded queue) — the graceful shed is a feature.
- **Tune group commit** (batch 100 / 10 ms today) against measured fsync cost on
Ref-A's local-path PV; make both knobs config.
- Non-goals: no ack-mode change, no membership change.
- **Exit gate (Ref-A, `tidal-stress` leader path):** ≥2,000 signals/s OK at
p99 ≤50 ms, error rate <1%, lag p99 ≤2 s sustained 10 min; replication metrics
visible in Prometheus.
### m11p2 — One replicated log (size: L)
**Goal:** every replicated mutation rides the WAL relay; the HTTP broadcast side
channel (items/embeddings) is deleted.
- Add item-metadata and embedding records to the WAL event model (they're
config-like, low-rate — the log absorbs them trivially; 64 MiB segment ceiling
is ample for 1536-dim vectors).
- Followers apply items/embeddings from the log; leader broadcast code and the
O(items) heal re-broadcast are removed (heal becomes pure log catch-up).
- Implement **`StreamSegments`** (declared, unimplemented) as the catch-up path:
a follower that reports applied=N gets a stream from N+1; the in-memory
`batch_log` dependency for recovery goes away (read back from the durable WAL).
- Hard-negatives stay CRDT (user-scoped, commutative) — document the split:
"log = totally-ordered global data; CRDT = per-user convergent data".
- **Exit gate:** chaos tests for both 2026-06-10 bug classes pass by
construction (no marker-gated fan-out to regress, no authed side POSTs);
item written anywhere is readable everywhere ≤2 s p99; heal of a 10-minute-down
follower with 100k items completes via stream without O(items) HTTP traffic.
### m11p3 — Quorum-acked writes (size: L) — closes ROADMAP gap **G4**
**Goal:** an opt-in durability contract a system of record can sit on.
- `ack=leader|quorum` (deployment default + per-request override header).
- Follower ship responses carry **durably-applied seqno** (fsync'd, not just
received); leader maintains per-peer durable high-water marks and a
**commit index** = quorum floor; `ack=quorum` responses gate on the commit
index passing the write's seq. Pipelined (batch-level acks), never one-at-a-time.
- Timeout semantics: quorum not reached in budget → retryable 503 naming the
laggard; the write may still commit (document at-least-once + dedup guidance;
evaluate optional client idempotency keys for signal accounting — open
question, decide in-phase).
- Rewrite runbook §8 from "honest about what we don't promise" to "here's the
knob and its cost".
- **Exit gate:** SIGKILL the leader under `ack=quorum` load → ledger checker
proves zero acknowledged loss across 100 random kill-points; quorum throughput
≥50% of p1's leader-ack number (≥1,000/s Ref-A).
### m11p4 — Failure detection, election, fencing (size: XL) — closes ROADMAP gap **G5**
**Goal:** "a machine died" is a non-event, not a runbook page.
- **Replicated metadata state** (term/epoch, leader id, membership): Raft-style
election over the existing transport — the quorum machinery from p3 is the
prerequisite. The data plane stays the WAL relay; only the small control state
is consensus-managed.
- Failure detector on the existing `Heartbeat` RPC (timeout + jitter; tunable).
- **Fencing everywhere:** terms stamped on ship/forward/promote; stale-term
ships rejected by followers; a restarted node **rejoins as follower** and
learns leadership from the metadata state — never from the static topology
file (kills incident §1.4-1).
- Leadership is durable locally (survives restart) and lease-bounded; manual
`/cluster/promote` becomes a fenced transfer (drain for maintenance), not the
availability mechanism.
- **Exit gate (chaos, Ref-A):** kill the leader under load → writes resume
<10 s p99 with zero operator action and zero acked loss; partition the old
leader away mid-election and let it restart → it cannot accept a single write
(fencing proven); election churn under flapping links bounded (no livelock).
### m11p5 — Membership, discovery, elasticity (size: L)
**Goal:** nodes are cattle; topology is data, not files.
- Resolve `grpc_addr` via DNS (drop the `SocketAddr::parse`-only constraint) and
support **seed-based join**: a new node contacts any seed, gets the membership
+ topology from the metadata state. The per-pod static-ClusterIP topology
ConfigMap hack dies.
- Conf-changes (add/remove/replace) through the p4 metadata consensus; joiners
catch up via **snapshot + StreamSegments** (snapshot = the engine's existing
checkpoint artifacts, shipped as a stream).
- Kubernetes reference becomes one StatefulSet + headless Service peer
discovery, PDB, topology-spread; rolling node replace is `kubectl delete pod`.
- **Exit gate:** scale 3→5→3 online under load: joiner catches up ≤5 min
(100k items), serves quorum after; p99 impact <2× for <60 s; zero loss.
### m11p6 — Sharding × replication + rebalancing (size: XL)
**Goal:** writes scale horizontally **without** giving up replication — the
current "replicated XOR sharded" split ends.
- Shard groups: keyspace hash-split (reuse `ShardRouter`) into S shards, each a
replication group at RF (default 3) with its own WAL/relay/commit-index and
its own elected leader; leaders balanced across nodes.
- Regions stop being shards: topology = nodes × shard-replicas, with optional
zone/region labels for placement and read affinity.
- Routing: any gateway routes by entity hash to the shard leader (write) or a
local/nearest replica (read); scatter-gather already merges cross-shard reads
with degraded semantics — now over shard groups.
- Rebalancing: shard move = snapshot + stream + fenced cutover; operator-
triggered first, automatic (rate-limited) later.
- **Exit gate (Ref-A, 3 shards × RF=3):** ≥5,000 quorum signals/s (≥2.5×
single-shard p3); kill any node → only its shard-leaderships move (<10 s),
reads never stop; `tidal-stress` sharded-vs-replicated comparison collapses
into one path.
### m11p7 — Security hardening (size: M)
**Goal:** the cluster stops trusting the network.
- **mTLS default** on gRPC replication (the `grpc_tls` config exists — exercise
it, flip plaintext to an explicit `insecure: true` with a loud startup WARN);
cert + bearer **rotation without restart** (file/secret watch).
- Authenticated inter-node HTTP: per-node identity (client certs or signed
internal tokens); the `x-tidal-internal` marker remains a routing hint only
(it already isn't an auth bypass — keep it that way by test).
- Audit log for admin verbs (promote/heal/partition/conf-change) with principal,
term, and outcome; at-rest encryption documented (delegate to volume
encryption, or in-engine as stretch); per-principal rate limits (the engine's
rate limiter exists — wire it to the HTTP layer).
- **Exit gate:** reference deployment has zero plaintext inter-node links;
rotation under load drops zero requests; a foreign pod on the cluster network
can neither ship segments nor call internal routes (negative tests).
### m11p8 — Observability + operations (size: M; **starts inside p1**)
**Goal:** operable by someone who didn't build it.
- Complete the `tidaldb_cluster_*` metric set (lag, commit index, ship
latency/queue, breaker, elections, forwards, write-pool, fsync) on a cluster
`/metrics` listener (**today: none**); Grafana dashboard + alert rules shipped
beside the existing standalone ones.
- Request-id propagation + tracing spans across forward/broadcast/ship hops
(the cluster routers currently skip the request-id layer).
- Truthful status: fix post-promote lag accounting (ShardId(0) keying) and the
leader's own applied row; `/cluster/status` becomes the single pane.
- **Self-driving heal:** server-side heal-until-converged (retry through breaker
resets) so "re-issue heal until lag 0" leaves the runbook (incident §1.4-3).
- Coordinated backup/restore: cluster-consistent snapshot manifest across shard
groups + WAL archival hook for PITR; timed restore drill in the runbook.
- Rolling upgrade: version handshake on ship/forward, N/N+1 skew documented;
`mp_rolling_upgrade_no_loss_no_stall` promoted to a release-gate CI job.
- **Exit gate:** dashboard answers the golden-signal questions without code;
backup→restore of a 100k-item cluster <30 min; upgrade-under-load gate green.
### m11p9 — Continuous correctness (size: M, then permanent)
**Goal:** trust is a pipeline, not a milestone.
- Nightly CI: the tier-3 chaos suites (partition relay, crash, clock-skew via
`TIDAL_HLC_SKEW_MS`, rolling upgrade) + new fault classes (disk-full,
slow-fsync injection, asymmetric partitions) against a Ref-A cluster.
- Invariant checkers as first-class: no-acked-loss ledger replay, per-entity
monotonic counters, cross-replica decay parity (1e-6), membership safety
(single leader per term per shard). Jepsen-style external harness as stretch.
- Nightly `tidal-stress` soak (1× 100k-DAU model, 1 h) with regression gates on
p99/throughput; results archived for trend lines.
- **Exit gate (GA):** every guarantee in §2 maps to a named automated test;
nightly suite green 30 consecutive days.
---
## 5. Sequencing, sizing, releases
| Phase | Depends on | Size | Closes |
|---|---|---|---|
| p1 perf floor | — | M | §1.4-4, throughput floor |
| p2 one log | p1 | L | §1.4-2, StreamSegments, O(items) heal |
| p3 quorum acks | p1, p2 | L | ROADMAP **G4** |
| p4 election + fencing | p3 | XL | ROADMAP **G5**, §1.4-1 |
| p5 membership | p4 | L | static-IP topology, elasticity |
| p6 shards × RF | p3, p5 | XL | write scaling |
| p7 security | p2 (stable surface) | M | network trust |
| p8 observability/ops | seeds in p1; completes after p6 | M | §1.4-3, cluster metrics |
| p9 chaos CI | grows from p3; gate at end | M | GA trust bar |
**Release waves:**
- **v0.9 "Credible HA"** = p1–p4: one shard done *right* — fast, unified log,
quorum-durable, self-healing. This alone moves the §1.1 comparison from
"1–2 orders below peers" to "in-band".
- **v1.0 "Scalable"** = p5–p6: elastic membership, sharding × replication.
- **v1.1 "Enterprise"** = p7–p9 complete (p7/p8 work interleaves earlier; the
release is the gate, not the start).
Also explicitly **not blocked on this roadmap:** the embedded engine (thepeach's
integration path) and the standalone server remain the recommended production
deployments today — they already clear their bars (82 ns writes; p99 <15 ms
reads; the stress test's 100k-DAU verdict).
## 6. Non-goals (scope honesty, per VISION.md)
- WAN active-active strong consistency. Cross-region stays async (log shipping)
with the CRDT tier for user-scoped convergent data; regions are placement
labels, not consistency domains.
- SQL, general transactions, multi-key serializability. The data model is
signals (commutative accumulation), items/embeddings (idempotent upserts),
hard-negatives (LWW CRDT) — the log + quorum gives these everything they need.
- Building a service mesh. mTLS + authenticated RPC yes; Istio no.
- Replacing the embedded mode. The cluster is for teams that *can't* embed;
the engine stays the product's core.
## 7. Open questions (decide in-phase, tracked here so they don't vanish)
1. Client idempotency for signal retries under `ack=quorum` timeouts (p3):
dedup window keyed by client request id vs. documented at-least-once.
2. Election engine: minimal purpose-built Raft over `tidal-net` vs. embedding a
vetted raft crate — decide in p4 design after p3's quorum machinery exists.
3. Shard count / split policy (static S at create vs. range-split later) — p6.
4. At-rest encryption: in-engine vs. delegated to volume encryption — p7.
5. Whether `tidal-stress` grows a distributed multi-generator mode to push past
the ~5k rps connection wall before p6's exit gate needs it.
---
*Companion docs: [stress-test-thepeach.md](ops/stress-test-thepeach.md) (baselines),
[runbooks/cluster.md](runbooks/cluster.md) (today's operational truth),
[ops/capacity-planning.md](ops/capacity-planning.md) (per-node sizing math),
[planning/ROADMAP.md](planning/ROADMAP.md) (M8 history + G4/G5/G6 gap registry).*