# tidalDB Cluster Runbook

Operating the multi-region `tidal-server cluster` surface: launch (single-process
dev fabric **and** the multi-process region nodes), the operational API,
replication transport facts, failover and partition drills, and the honest
write-durability contract.

> ## STATUS: EXPERIMENTAL — TWO MODES, NEITHER IS QUORUM-ACKED HA YET
>
> Cluster mode has **two shapes**, both behind the same experimental opt-in:
>
> **1. Multi-process (`--region`) — real process isolation.** Each
> `tidal-server cluster --region <name>` process owns **exactly one region**: one
> `TidalDb`, one [`GrpcTransport`](#4-grpc-replication-transport-tidal-net) whose
> server binds *this* region's `grpc_addr` and whose peers are every **sibling
> region's real `grpc_addr`**. Processes peer over real gRPC and forward over real
> HTTP, so a crash of one region's process takes down **only that region** — the
> survivors keep serving. This is genuine process (and, across hosts, host)
> isolation. It is verified end-to-end by the tier-3 suites
> (`cluster_multiproc`, `cluster_chaos`, `cluster_lifecycle`, `cluster_runbook`)
> over real OS processes with real network-partition injection.
>
> **2. Single-process (no `--region`) — the dev/demo default.** Every region runs
> inside **one** process. Replication still traverses the **real `tidal-net` gRPC
> transport** on loopback (faithful multi-region semantics over a real wire), but
> there is **no process isolation**: a crash, OOM, or host failure takes the whole
> "cluster" down at once. This is a development / staging / demo fabric and a
> correctness harness for the replication paths — **not** production HA. It remains
> the default because it needs no per-region topology addresses and no process
> orchestration.
>
> **Honest remaining limits (both modes):**
> * **Quorum durability is opt-in.** The default `204` is leader-durable
>   (storage + WAL fsync; follower ship off the request path). Since m11p3,
>   `ack=quorum` — topology default or per-request `x-tidal-ack` header —
>   gates success on a **majority of the replica set durably holding the
>   write**, surviving permanent leader loss (see
>   [§8](#8-write-durability-contract-the-ack-knob-and-its-cost-m11p3)).
> * **Leadership is automatic** (m11p4). Every node runs a failure detector and
>   a Raft-style election (pre-vote + vote + check-quorum + fenced transfer):
>   kill the leader and the survivors elect a successor — typically under a
>   second with the defaults — with **zero operator verbs** and zero
>   acknowledged-write loss (see [§9.1](#91-automatic-failover-m11p4--the-default)).
>   `/cluster/promote` is now a **fenced transfer** for maintenance/override, not
>   the availability mechanism. `election.auto_election: false` preserves the
>   pre-m11p4 operator-driven posture.
> * **Membership is elastic and addresses are DNS names** (m11p5). `grpc_addr`
>   is an advertised hostname or IP (DNS-resolved on every reconnect); nodes join
>   online via `--seed` and catch up via `FetchSnapshot` + the `StreamSegments`
>   stream; add/remove ride kind-4 membership records on the replicated log
>   (see [§1b](#1b-multi-process-one-process-per-region),
>   [§3](#3-topology-yaml), [§6](#6-cluster-management-api),
>   [§9.1](#91-automatic-failover-m11p4--the-default)).
>
> **For a production deployment today**, run a **single `tidal-server standalone`**
> node backed by host-level redundancy and disk durability (see
> [kubernetes.md](kubernetes.md) and [server-deployment.md](../guides/server-deployment.md)),
> and reach for multi-process cluster mode for read-scale / multi-region
> deployments whose writes need `ack=quorum`'s failover-survivable contract.
>
> **Both modes refuse to start** unless you explicitly opt in with either the
> `--experimental-cluster` flag or the `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1`
> environment variable. On start each emits a loud, **mode-specific** `WARN`
> restating exactly what it does and does not provide. Do not wire either into a
> production load balancer.

## Scope: what cluster mode does and does not own

tidalDB owns **retrieval and ranking only**. Cluster mode adds multi-region
replication of that, nothing more. It does **not** generate embeddings, store
video/blobs, run a CDN or transcoder, do auth/moderation/payments, or provide a
distributed consensus log. Bring your vectors; tidalDB retrieves and ranks over
them. See [VISION.md](../../VISION.md) for the scope boundary.

## Prerequisites

- Rust toolchain ≥ 1.91 if running directly (the Docker build pins
  `rust:1.91-bookworm`).
- `protobuf-compiler` (`protoc`) and a C++ toolchain (`g++`) on the build host —
  `tidal-net`'s build script compiles the WAL-shipping `.proto`, and USearch's
  HNSW core is C++. The Docker image installs both.
- Docker 25+ if running via container.
- For **single-process** mode: one HTTP port (default `9500`); each follower
  region also binds an **OS-assigned loopback port** for its gRPC replication
  server unless you pin one in the topology (see [§3](#3-topology-yaml)).
- For **multi-process** mode: per-region `grpc_addr` **and** `http_addr` declared
  in the topology, and the ports they name available on each host
  (see [§3](#3-topology-yaml)).

## 1. Launch the cluster locally

Cluster mode is gated. Pass `--experimental-cluster` (or set
`TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1`) or the server exits with a mode-specific
error explaining why.

### 1a. Single-process (dev/demo default)

```bash
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
cargo run -p tidal-server -- \
  cluster \
  --listen 127.0.0.1:9500 \
  --schema tidal-server/config/default-schema.yaml \
  --topology tidal-server/config/default-cluster.yaml \
  --experimental-cluster
```

The default topology spins up three regions (`us-east`, `eu-west`, `ap-south`)
with `us-east` as leader, all inside one process. On a clean start you will see a
`WARN` line stating this is experimental and single-process, an `info` line per
follower (`follower gRPC transport ready`), and finally
`listening on http://127.0.0.1:9500`.

### 1b. Multi-process (one process per region)

Pass `--region <name>` (or set `TIDAL_REGION`) to run **only that region** in this
process. The topology must declare a per-region `grpc_addr` **and** `http_addr` for
every region (see [§3](#3-topology-yaml)); siblings reach each other over those.
Launch one process per region — typically one per host, each with its own
`--data-dir`. **`--data-dir` is REQUIRED in multi-process mode** (m11p2): the
durable WAL is the replicated log itself — it is what ships to peers and what
serves their catch-up streams — so a node without one is rejected at startup:

```bash
# Region us-east (the initial leader) on host A
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
tidal-server cluster --experimental-cluster \
  --region us-east \
  --listen 0.0.0.0:9501 \
  --schema /etc/tidal/schema.yaml \
  --topology /etc/tidal/topology.yaml \
  --data-dir /var/lib/tidal/us-east

# Region eu-west on host B
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
tidal-server cluster --experimental-cluster \
  --region eu-west --listen 0.0.0.0:9502 \
  --schema /etc/tidal/schema.yaml --topology /etc/tidal/topology.yaml \
  --data-dir /var/lib/tidal/eu-west

# Region ap-south on host C
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
tidal-server cluster --experimental-cluster \
  --region ap-south --listen 0.0.0.0:9503 \
  --schema /etc/tidal/schema.yaml --topology /etc/tidal/topology.yaml \
  --data-dir /var/lib/tidal/ap-south
```

Every process parses the **same** topology file; `RegionId`s are assigned by
declaration order, so all processes agree on the region→id mapping. Each process
binds its own region's `grpc_addr`/`http_addr` and dials its siblings' addresses.
The `--listen` HTTP address is the public gateway for that region.

#### Seed-join boot (m11p5 — adding a node without editing the topology)

A node can join an existing cluster **online**, without appearing in any
declared topology, by contacting a running peer as a **seed**:

```bash
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
tidal-server cluster --experimental-cluster \
  --region eu-2 \
  --listen 0.0.0.0:9504 \
  --schema /etc/tidal/schema.yaml \
  --topology /etc/tidal/topology.yaml \
  --data-dir /var/lib/tidal/eu-2 \
  --seed http://10.0.1.10:9501 \
  --seed http://10.0.2.10:9502 \
  --advertise-grpc eu-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9600 \
  --advertise-http  eu-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9504 \
  --metrics 0.0.0.0:9091
```

| Flag | Meaning |
|------|---------|
| `--seed <url>` (repeatable) | One or more seed HTTP base URLs. The joiner polls each for the current leader, then `POST /cluster/join`s through it. Any reachable seed works; list a few for resilience. |
| `--advertise-grpc <host:port>` | This node's **advertised** gRPC address — what siblings dial. Required with `--seed` (the joiner has no topology entry of its own). DNS-capable. |
| `--advertise-http <host:port>` | This node's advertised HTTP gateway — what peers forward writes/status to. Required with `--seed`. |
| `--metrics <addr>` | Prometheus `/metrics` bind (the joiner has no topology `metrics_addr`). |

What happens: the joiner skips the topology's "every region declared" gate,
learns its **roster + assigned id + current term** from the seed's join
response, appends a **Learner** record to the replicated log (the leader answers
only after it is quorum-committed), persists the roster to a durable membership
cache and the term to `election_state` (persist-before-act), installs a
[snapshot](#8-write-durability-contract-the-ack-knob-and-its-cost-m11p3) when it
is behind the leader's retained WAL, then streams the live tail. The leader's
**auto-promotion** duty flips it Learner → Voter once it is within
`replication.learner_promote_lag` (default 1024) of the flushed frontier. A
**restart** boots from the cache without the seed.

> **A `--seed` boot still requires the local topology/config file** for the
> behavioral knob blocks (`replication:`, `wal:`, `election:`, `timeouts:`,
> `grpc_tls`) — its `regions:` list is ignored for the roster (the join response
> is the roster), but a bare `--seed` with neither `--topology` nor
> `TIDAL_CONFIG` **refuses to boot** naming the rule, so a joiner never silently
> inherits the compiled-in defaults' wrong ack mode, quorum timeout, or election
> timing. The k8s manifests mount the shared bootstrap ConfigMap on every pod
> including N≥3, so this costs nothing there.

> **Capability gate (m11p5 mixed-version safety).** The leader **refuses
> `/cluster/join` and every conf-change until all current voters report kind-4
> capability** — see [§8](#8-write-durability-contract-the-ack-knob-and-its-cost-m11p3)'s
> downgrade rule. Complete the binary upgrade before adding or removing nodes.

**Auth:** set `TIDAL_API_KEY=<secret>` to require `Authorization: Bearer <secret>`
on the data and `/cluster/*` mutation routes. If it is **unset the server runs
UNAUTHENTICATED and logs a WARN** — never expose an unauthenticated cluster
beyond loopback / a trusted VPC. Health probes and `/openapi.json` are always
unauthenticated. In multi-process mode set the **same** key on every process: a
forwarded/broadcast request passes the caller's `Authorization` through verbatim,
and the internal-propagation marker (`x-tidal-internal: 1`) is an inter-sibling
trust signal, **not** an auth bypass (the bearer middleware still runs first).

Useful environment variables:

| Var | Mode | Effect |
|-----|------|--------|
| `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1` | both | Opt in to cluster mode (alternative to `--experimental-cluster`). |
| `TIDAL_REGION` | multi-process | Selects the single region this process owns (alternative to `--region`). |
| `TIDAL_API_KEY` | both | Bearer token for protected routes. Unset ⇒ unauthenticated + WARN. Set the SAME key on every region in multi-process mode. |
| `TIDAL_HLC_SKEW_MS` | multi-process | Signed ms offset applied to THIS process's HLC. Affects reconcile-time LWW stamping ONLY (not signal-decay timestamps). A test/ops escape hatch for verifying causal convergence under clock skew — do not set it in normal operation. |
| `TIDAL_CONFIG` | both | Config dir holding `default-schema.yaml` / `default-cluster.yaml` (used when `--schema` / `--topology` omitted). |
| `PORT` | both | Listen address. A bare port (`9500`) normalises to `0.0.0.0:9500`. |
| `TIDAL_SERVER_LOG` | both | `tracing` filter (default `info`). |

## 2. Launch via Docker

```bash
# Build the image once (build context is the repo root).
docker build -f docker/cluster/Dockerfile -t tidaldb:cluster .

# Run (press Ctrl+C to stop). The image already sets
# TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 so cluster mode starts; it still logs the
# loud experimental WARN. The default CMD is SINGLE-process cluster mode.
docker run --rm -p 9500:9500 tidaldb:cluster
```

The image bakes the default schema/topology under `/etc/tidal-server` and uses
`ENTRYPOINT ["tidal-server"]` + a `cluster …` `CMD`, so you can override the
subcommand (e.g. `docker run tidaldb:cluster standalone …`), pass `--region` for a
multi-process container, or supply your own config files:

```bash
docker run --rm -p 9500:9500 \
  -e TIDAL_API_KEY=changeme \
  -v "$PWD/configs/my-schema.yaml:/srv/schema.yaml:ro" \
  -v "$PWD/configs/my-topology.yaml:/srv/topology.yaml:ro" \
  tidaldb:cluster \
  cluster \
    --listen 0.0.0.0:9500 \
    --schema /srv/schema.yaml \
    --topology /srv/topology.yaml
```

The container ships a `HEALTHCHECK` that curls `/health`. For Kubernetes/Compose
deployment patterns see [Cross-references](#cross-references).

## 3. Topology YAML

The topology file declares regions and the leader.

**Single-process minimal default** (`tidal-server/config/default-cluster.yaml`):

```yaml
regions:
  - name: us-east
  - name: eu-west
  - name: ap-south
leader: us-east
# Optional: OS worker threads serving cluster write/heal requests (gRPC ship).
# Defaults to available parallelism when omitted.
# write_workers: 4
```

**Multi-process** requires a per-region `grpc_addr` **and** `http_addr` (validated
at startup — a missing or syntactically invalid `host:port` is a hard error naming
the region; reachability is deliberately not probed, since siblings boot in any
order):

`grpc_addr` is an **advertised** address — a literal `host:port` OR a DNS
hostname (m11p5). A new optional per-region `grpc_bind` controls the local bind
independently of what siblings dial; the example below shows the DNS shape (one
shared file names every region by its per-pod DNS name while each pod binds
`0.0.0.0`):

```yaml
regions:
  - name: us-east
    grpc_addr: "tidaldb-0.tidaldb-peers.svc.cluster.local:9601"  # ADVERTISED (siblings dial; DNS re-resolved on reconnect)
    grpc_bind: "0.0.0.0:9601"      # LOCAL bind (optional; see derivation rule below)
    http_addr: "10.0.1.10:9501"    # this region's public HTTP gateway (forwarding + status)
    metrics_addr: "10.0.1.10:9091" # optional Prometheus /metrics listener (set it in production)
  - name: eu-west
    grpc_addr: "tidaldb-1.tidaldb-peers.svc.cluster.local:9602"
    grpc_bind: "0.0.0.0:9602"
    http_addr: "10.0.2.10:9502"
    metrics_addr: "10.0.2.10:9091"
  - name: ap-south
    grpc_addr: "tidaldb-2.tidaldb-peers.svc.cluster.local:9603"
    grpc_bind: "0.0.0.0:9603"
    http_addr: "10.0.3.10:9503"
    metrics_addr: "10.0.3.10:9091"
leader: us-east
# Optional m11p1 tuning blocks (engine defaults shown):
# replication:
#   batch_max_events: 256   # events coalesced per shipped batch (1-256)
#   window: 4               # in-flight batches per peer (1-64)
#   retry_ms: 100           # backoff before a transiently-failed batch retries
#   catchup_retry_ms: 30000           # backoff before a FAILED catch-up pull re-pulls (m11p4)
#   snapshot_artifact_ttl_ms: 600000  # staged-snapshot reuse window (m11p5; never unpins an active consumer)
#   learner_promote_lag: 1024         # learner→voter promotion distance + readiness hysteresis (m11p5)
#   reseed_self_restart: false        # drain+exit(0) when reseed_required latches (m11p5; k8s sets true)
# wal:
#   batch_size: 100         # events per group-commit fsync (1-256)
#   batch_timeout_ms: 10    # max wait before a partial batch flushes
# election:                 # m11p4 failure-detector / election timers (defaults shown)
#   heartbeat_interval_ms: 300
#   election_timeout_min_ms: 1500     # validated: leader_lease_ms + heartbeat_interval_ms < this
#   election_timeout_max_ms: 3000
#   leader_lease_ms: 900
#   auto_election: true               # false → pre-m11p4 operator-driven failover
```

**`grpc_bind` derivation (m11p5 §1):** `grpc_bind` present → bind it. Absent +
`grpc_addr` parses as a literal SocketAddr → bind that (today's behavior,
byte-for-byte — every existing IP topology keeps working). Absent + `grpc_addr`
is a hostname → bind `0.0.0.0:<port from grpc_addr>`. Because the peer-dial path
no longer parses `grpc_addr` as a SocketAddr, hyper re-resolves the hostname on
every reconnect — a pod rescheduled onto a new IP is reached with no peer
restarts. **TLS note:** SNI follows the URI host, so DNS peer names require
DNS-SAN certs (see `grpc_tls` below); no code change.

Fields:

| Field | Single-process | Multi-process | Meaning |
|-------|----------------|---------------|---------|
| `regions[].name` | required | required | Region name used everywhere in the HTTP API (`?region=`, `/cluster/promote`, etc.). Must be unique. |
| `regions[].grpc_addr` | optional | **required** | This region's **advertised** gRPC replication address — what siblings dial. Since m11p5 it may be a literal `host:port` OR a **DNS hostname** (re-resolved on every reconnect). In single-process mode **omit it** — the server allocates a free loopback port and self-heals a bind race by retrying on a fresh port. In multi-process mode it is required and is tried exactly once. |
| `regions[].grpc_bind` | unused | optional | This region's **local** gRPC bind `host:port` (m11p5), independent of the advertised `grpc_addr`. Omitted: a literal `grpc_addr` binds itself; a hostname `grpc_addr` binds `0.0.0.0:<its port>`. Set it to bind a specific interface while advertising a DNS name. |
| `regions[].http_addr` | unused | **required** | This region's public HTTP gateway address, used by siblings for write/read forwarding and status aggregation. Unused in single-process mode. |
| `regions[].grpc_tls` | unused | optional | TLS material for this region's gRPC transport: `ca_cert`, `server_cert`, `server_key` (PEM paths), plus optional `client_cert`/`client_key` for mTLS. Omitted ⇒ plaintext, the right posture for loopback/VPC topologies. **DNS `grpc_addr` requires DNS-SAN certs** (SNI follows the dialed hostname). |
| `regions[].metrics_addr` | unused | optional | Bind address for this region's Prometheus `/metrics` listener (m11p1; cluster mode previously had none). Omitted ⇒ no metrics endpoint. Set it in every production topology, and bind it internally — the endpoint is unauthenticated. |
| `leader` | required | required | Must name one of the declared regions. The initial write leader. |
| `write_workers` | optional | optional | Size of the runtime-free OS-thread pool that admission-controls cluster writes (`/signals` staging) and runs `/cluster/heal`'s blocking redelivery. Bounded queue ⇒ 429 backpressure. Must be ≥ 1 when given. |
| `timeouts.broadcast_peer_secs` | unused | optional | Per-peer budget (seconds) for the `/cluster/promote` fan-out (the only remaining peer fan-out — the m11p2 log replaced the item/embedding broadcast). Default 2s — right for loopback/VPC; raise it for WAN topologies where a distant region cannot answer in 2s. Must be ≥ 1 when given. |
| `replication.batch_max_events` | unused | optional | Max relay events coalesced into one shipped batch (1–256, the WAL wire-format ceiling). Default 256. |
| `replication.window` | unused | optional | In-flight batches per peer (1–64). 1 = strictly in-order shipping; higher pipelines across the peer RTT (out-of-order arrivals park gap-aware on the receiver). Default 4. |
| `replication.retry_ms` | unused | optional | Backoff (ms) before a transiently-failed batch ship retries. Default 100. |
| `replication.ack` | unused | optional | Deployment-default write acknowledgment: `leader` (default) or `quorum` (majority-durable — see [§8](#8-write-durability-contract-the-ack-knob-and-its-cost-m11p3)). Per-request override: the `x-tidal-ack` header. |
| `replication.quorum_timeout_ms` | unused | optional | Budget an `ack=quorum` write waits for the commit index before the retryable 503 naming the laggards. Default 2000. |
| `replication.catchup_retry_ms` | unused | optional | Backoff (ms) before a FAILED catch-up pull re-pulls on a timer (m11p4) — lets an idle cluster self-heal a follower whose pull failed during a rolling restart. Default 30000. Must be ≥ 1 when given. |
| `replication.snapshot_artifact_ttl_ms` | unused | optional | Staged-snapshot reuse window (m11p5), counted from the last fetch's completion. Governs artifact REUSE only — **never** unpins the WAL retention of an active consumer; a hard cap (4×) force-drops a never-releasing pin (`tidaldb_cluster_snapshot_pin_force_drops_total`). Default 600000 (10 min). Must be ≥ 1. |
| `replication.learner_promote_lag` | unused | optional | Learner→voter promotion distance AND the readiness-convergence hysteresis threshold (m11p5), in events. Default 1024. Must be ≥ 1. |
| `replication.reseed_self_restart` | unused | optional | Drain + clean-exit(0) once the durable `reseed_required` marker latches (m11p5; k8s sets it `true`). **Refused** when the remaining voters can't sustain quorum without this node. Default `false`. |
| `election.heartbeat_interval_ms` | unused | optional | Leader heartbeat interval (m11p4). Default 300. |
| `election.election_timeout_min_ms` / `..max_ms` | unused | optional | Randomized follower election timeout window (m11p4). Defaults 1500 / 3000. Validated: `leader_lease_ms + heartbeat_interval_ms < election_timeout_min_ms`. |
| `election.leader_lease_ms` | unused | optional | Leader freshness lease — a leader that loses majority contact steps down within it (m11p4). Default 900. |
| `election.auto_election` | unused | optional | `true` (default) = automatic failover; `false` = pre-m11p4 operator-driven posture (no auto elections, no check-quorum step-down). |
| `wal.batch_size` | optional | optional | Events per WAL group-commit fsync (1–256). Default 100. |
| `wal.batch_timeout_ms` | optional | optional | Max ms a partial group-commit batch waits before flushing. Default 10. Tune against the measured `tidaldb_cluster_wal_fsync_us` on the deployment's volume. |

The schema YAML (signals / text_fields / embedding_slots / profiles) is the same
format the standalone server loads — see [API.md](../../API.md) and
[QUICKSTART.md](../../QUICKSTART.md) for the full schema/profile grammar and the
built-in ranking profiles. Topology only adds the region layout above.

> **Personalization correctness note.** Preference-vector personalization (the
> taste vector that powers `for_you`) only updates for signals declared
> `positive_engagement: true` in the schema. The cluster `/signals` route writes
> **global** signals (no `user_id` / `creator_id` context), so it never folds
> item embeddings into a user's taste vector regardless of the
> `positive_engagement` flag. Personalized writes go through the embedded engine
> (`signal_with_context`), not this HTTP route. The exception is `/hardnegs`,
> which records a **user-scoped** hide that converges across regions via
> [`/cluster/reconcile`](#6-cluster-management-api).

## 4. gRPC replication transport (tidal-net)

Replication between regions is the real `tidal-net` `WalShipping` gRPC service,
not in-process channels. The facts you need to operate and firewall it:

| Property | Value | Notes |
|----------|-------|-------|
| Service | `WalShipping` (proto `tidal.replication.v1`) | RPCs: **`ShipSegment`** (unary; the live-tail push path), **`StreamSegments`** (server-streaming; the follower-pulled catch-up path since m11p2), **`FetchSnapshot`** (server-streaming; the m11p5 snapshot transfer for joiners + reseeds), **`JoinCluster`** (m11p5 conf-change), and **`Heartbeat`** (ControlPlane health). |
| Bind address | per-region `grpc_addr` (multi-process); an **auto-allocated** loopback port (single-process) | Default dev band `59520–59529` if you pin one. |
| Transport security | mTLS via `TlsConfig` (`ca_cert`, `server_cert`, `server_key`, optional `client_cert` / `client_key`) | `insecure = true` (plaintext) is the loopback/VPC default in this phase. A peer with no TLS config **must** set `insecure`. |
| Max payload | **64 MiB** | Both encoder and decoder codec limits are raised to this; pinned by a compile-time assert to the engine's `InProcessTransport` limit so both ends agree. WAL segments default to 16 MiB. |
| Circuit breaker | per-peer, **threshold 5**, **reset 30s** | After 5 consecutive ship failures the breaker opens; it stays open for 30s, then the next attempt probes (HalfOpen → Closed on success, Open again on failure). Backpressure (channel full) does **not** trip it. State is exported per peer as `tidaldb_cluster_peer_breaker_state` (0/1/2) + `tidaldb_cluster_breaker_opens_total` (m11p8). **Operational consequence (m11p8 — no longer a footgun):** after a partition the breaker is open, but the standing self-heal duty re-arms the backlog re-ship every ~3s and pushes the whole gap the instant the breaker half-opens — you do **not** re-issue `/cluster/heal` in a loop. Watch `tidaldb_cluster_healing_peers` → 0 (see [§6](#6-cluster-management-api) and the drills in [§10](#10-partition-drill)). |
| Timeouts (defaults) | connect 5s, request 10s, keep-alive PING every 10s / 5s ACK | A blackholed peer fails fast instead of stalling the single-threaded shipper. |

`ShipSegment` carries the WAL segment id, BLAKE3-validated payload bytes, event
count, and the leader's authoritative `leader_last_seq` (so a follower can
advance its replication-lag high-water-mark even for an all-local segment that
filters to empty). The follower's segment-receiver thread drains and applies it
on arrival. The engine's applied high-water-mark is a **contiguous frontier with a
bounded ahead-buffer**, so out-of-order eager ships can never swallow a sequence
gap (a silent-data-loss class fixed in m8p10).

> **Operational guard:** the gRPC replication ports and the Prometheus `/metrics`
> endpoint are **UNAUTHENTICATED**. Bind them to loopback or a cluster-internal
> network only — never the public interface.

## 5. Core HTTP API

All routes are JSON unless noted. Examples assume `BASE=http://localhost:9501`
(any region's gateway in multi-process mode; the single `--listen` address in
single-process mode) and, when `TIDAL_API_KEY` is set,
`AUTH='-H "Authorization: Bearer $TIDAL_API_KEY"'`. Drop the `-H` header when
running unauthenticated.

Middleware mirrors standalone: 30s request timeout (408), 100 max in-flight
(429), 2 MB body limit (413), and an `x-request-id` on every response. Health
probes sit **outside** the load-shedding stack so liveness/readiness are never
queued or timed out under saturation.

> **Multi-process routing (one coherent surface).** Any region's gateway accepts
> any operation and routes it to the node that owns it:
> * **Writes** (`/signals`, `/items`, `/embeddings`, `/hardnegs`) on a non-leader
>   **forward to the leader** (the caller's `Authorization` passes through); the
>   client sees the leader's status/body. A leader that is unreachable degrades to
>   a `503` naming the leader, never a hang.
> * **Reads** (`/feed`, `/search`) default to the **LOCAL** region. This is the
>   key difference from single-process mode, whose default read is the leader. A
>   `?region=<other>` read **forwards** to that region's process.
> * `/cluster/promote`, `/cluster/partition`, `/cluster/heal` route to / fan out
>   from the leader as documented in [§6](#6-cluster-management-api).

### Health

```bash
curl "$BASE/health"          # 200 ok / 503 while draining; reports mode, region, leader
curl "$BASE/health/startup"  # always 200
curl "$BASE/health/live"     # always 200
curl "$BASE/openapi.json"    # served OpenAPI 3.1, UNAUTHENTICATED — canonical HTTP reference
```

`/health` returns `{ "ok": true, "service": "tidaldb", "mode": "cluster",
"region": "us-east", "leader": "us-east", ... }`. The `/openapi.json` document is
the machine-readable source of truth for the **data**, **`/cluster/*`**,
**`/sharded/*`**, and **`/hardnegs`** request/response shapes (the health probes
are intentionally outside the documented API surface).

### Register items & embeddings (one replicated log — m11p2)

Items and embeddings ride the **same replicated WAL stream as signals**: the
leader journals each write as a kind-1 (item metadata) or kind-2 (embedding)
WAL record BEFORE touching storage, and followers apply it from the log —
live pushes for the hot tail, the `StreamSegments` catch-up stream for
history. There is **no HTTP broadcast** anymore (the m8p10 side channel and
its bug classes were deleted in m11p2), so the responses are plain statuses:

```bash
curl -X POST "$BASE/items" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "metadata": { "title": "Jazz Piano", "category": "music" } }'
# → 201 Created (item durably journaled into the replicated log + applied)

curl -X POST "$BASE/embeddings" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "values": [0.1, 0.2, 0.3, 0.4] }'
# → 204 No Content
```

The `201`/`204` asserts **leader durability** and carries **`x-tidal-seq`**
(the record's seqno in the replicated log); `x-tidal-ack: quorum` upgrades it
to a majority-durable ack (see
[§8](#8-write-durability-contract-the-ack-knob-and-its-cost-m11p3)). The
record is fsynced into the stream every follower receives (live push, or
pull-based catch-up after downtime — see
[§6 heal](#6-cluster-management-api)). A write to a non-leader forwards to
the leader transparently (the ack header and seq header travel through the
forward). A down/partitioned peer needs no backfill bookkeeping: it
converges from the log when it returns.

Embeddings: tidalDB does **not** generate vectors — the caller brings them. The
write L2-normalizes and inserts into the HNSW index. Dimensions are **strict**:
they must equal the slot's declared dimensions (min 2, max 4096) or the insert is
**rejected with a 500** (the engine surfaces a dimension mismatch as an internal
error, not a 400 — a known wart, tracked post-M8); **zero-norm vectors are also
rejected (500)**. `RETRIEVE` / `SEARCH` route through the **first declared
embedding slot only**; multi-modal apps must fuse offline or use separate entity
kinds.

### Record signals (cluster `/signals` = global only)

```bash
curl -X POST "$BASE/signals" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "signal": "view", "weight": 1.0 }'
# → 204 No Content (see the durability contract in §8)
```

The signal name must be declared in the schema (an undeclared name returns **400**
naming it). The 204 carries **`x-tidal-seq`** (the write's replicated-log seqno),
and `x-tidal-ack: quorum` upgrades it to a majority-durable ack — see
[§8](#8-write-durability-contract-the-ack-knob-and-its-cost-m11p3). This route
records a **global** signal on the leader and ships it to
followers over the replicated WAL stream; it does not personalize (see the
personalization note in [§3](#3-topology-yaml)). On a non-leader gateway it
forwards to the leader transparently and still returns `204`.

### Hard negatives (user-scoped hides)

```bash
curl -X POST "$BASE/hardnegs" \
  -H 'Content-Type: application/json' \
  -d '{ "user_id": 42, "item_id": 7 }'
# → 204 No Content
```

Records a user-scoped hide on the leader (a non-leader gateway forwards). The item
is filtered from that user's `/feed?user_id=42`. Hard negatives converge across
regions via the LWW-resolved [`/cluster/reconcile`](#6-cluster-management-api) CRDT
path — NOT the signal WAL relay and NOT a broadcast.

### Retrieve and search (region-pinned reads)

```bash
# Default read region is the LOCAL region (multi-process mode).
curl "$BASE/feed?user_id=42&profile=for_you&limit=20"
curl "$BASE/search?query=jazz%20piano&user_id=42&limit=5"

# Pin a read to a specific region. The gateway forwards to that region's process.
# Followers may lag the leader (and lag jumps during a partition) — use this for
# canary reads and lag verification.
curl "$BASE/feed?profile=trending&region=eu-west"
```

`?region=` accepts any declared region name; an unknown name returns **400**. Omit
it and the read serves from the **LOCAL** region (multi-process) / the current
leader (single-process). `limit` is clamped to 1000 at the trust boundary (a
larger value cannot amplify memory unboundedly). The `for_you` profile applies the
built-in diversity defaults (`max_per_creator=2`, `format_mix_max_fraction=0.4`,
`exploration=0.1`).

## 6. Cluster management API

### Check cluster status

Two views. `/cluster/status/local` reports THIS node's own replication/leadership
state (no peer calls); `/cluster/status` aggregates EVERY region (the gateway calls
each peer's `/cluster/status/local` concurrently with a tight per-peer budget).

```bash
curl "$BASE/cluster/status/local" | jq
```

```json
{
  "region": "ap-south",
  "is_leader": false,
  "leader": "us-east",
  "last_seq": 0,
  "applied_events": 124,
  "lag_events": 1,
  "partitioned": [],
  "reachable": true
}
```

```bash
curl "$BASE/cluster/status" | jq
```

```json
{
  "leader": "us-east",
  "relay_log_len": 125,
  "regions": [
    { "name": "us-east",  "applied_events": 125, "lag_events": 0, "partitioned": false, "reachable": true },
    { "name": "eu-west",  "applied_events": 125, "lag_events": 0, "partitioned": false, "reachable": true },
    { "name": "ap-south", "applied_events": 124, "lag_events": 1, "partitioned": false, "reachable": true }
  ]
}
```

`relay_log_len` is the leader's high-water-mark (`last_seq`); each region's
`lag_events` is `relay_log_len − applied_events` (saturating). Since m11p3
`/cluster/status/local` also reports the node's `ack` default and, on the
leader, `commit_index` — the highest seqno a majority of the replica set
durably holds (`last_seq − commit_index` is the quorum lag). A region the
gateway **cannot reach** within the per-peer budget is reported honestly as
`reachable: false`, `partitioned: true`, `applied_events: 0`, and worst-case lag
(`lag_events == relay_log_len`). A non-zero, *growing* lag on a reachable region is
the signal that it is partitioned (ship-skip) or its segment-receiver is wedged.

### Promote a new leader

```bash
curl -X POST "$BASE/cluster/promote" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "eu-west" }'
# → { "ok": true, "leader": "eu-west", "acked": ["eu-west","ap-south"], "failed": [] }
```

The gateway first resolves the TARGET's **stream baseline** — the new leader's
WAL flushed frontier at promotion, persisted in its `data_dir/stream_baseline`
— then fans the promote out to every peer carrying it (internal marker, so
each applies locally and does not re-fan). Peers jump their applied frontier
for the new leader's shard to the baseline: everything at or below it is
pre-stream history (replicated applies of the OLD stream), not data. The
response carries `baseline` plus the `acked`/`failed` fan-out report; a peer
in `failed` (e.g. a dead old leader) self-corrects — its first parked batch
from the new stream triggers a catch-up pull whose chunks announce the
baseline. After promotion `/cluster/status` reports the new leader; new
writes route there; the new leader's ship queue activates and the demoted
node's deactivates. An unknown region returns **400**. Since m11p4 `promote` is
a **fenced transfer** (it drains the target, then sanctions an election), not
the availability mechanism — automatic failover handles a dead leader with no
operator action ([§9.1](#91-automatic-failover-m11p4--the-default)); this verb
is for maintenance and deliberate successor choice
([§9.2](#92-manual-promote-a-fenced-transfer-maintenance--override)).

### Simulate a partition & heal

There are **two** ways to partition a region; the runbook drills demonstrate both
([§10](#10-partition-drill)):

1. **Simulated ship-skip flag** — `/cluster/partition` tells the leader to stop
   shipping to the named region (no sockets touched). Good for a controlled,
   reversible lag demo.
2. **Real network partition** — sever the actual TCP path between processes
   (firewall / proxy). The chaos suite uses a root-free in-harness TCP relay;
   operators can use `iptables`/`pfctl` (see [§10](#10-partition-drill)).

```bash
# Simulated: isolate ap-south — leader ships skip this follower, so its lag climbs.
curl -X POST "$BASE/cluster/partition" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "ap-south" }'
# → { "ok": true, "partitioned": "ap-south" }

# Heal: resume shipping past everything the follower reports applied, and
# nudge it to PULL any history that rotated out of the leader's ship tail
# (the single recovery verb — log catch-up IS the full heal since m11p2).
curl -X POST "$BASE/cluster/heal" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "ap-south" }'
# → { "ok": true, "healed": "ap-south" }
```

`/cluster/heal` is the **single recovery verb**, and since m11p2 it is pure
log catch-up: it clears the partition flag, **resumes** the ship queue past
the follower's reported applied seqno (retries of data the follower already
holds prune automatically — every ship ack piggybacks the follower's applied
seqno), and **nudges** the follower to pull anything older than the leader's
in-memory ship tail via its `StreamSegments` catch-up stream over the
leader's durable WAL segments (`POST /cluster/catchup`, internal; the nudge
forwards the healing operator's own bearer credential). Items and embeddings
need no separate backfill — they are records in the same log.

> **Convergence is self-driving (m11p8 — the operator no longer loops heal).**
> The per-peer ship senders retry parked batches every `replication.retry_ms`
> (default 100ms); a follower that detects a gap pulls the catch-up stream
> itself (also on boot, so a restarted node converges with no operator action
> at all — the tier-3 `mp_items_ride_the_log_and_catchup_stream` proves it).
> On top of that, a **standing leader heal duty** runs every ~3s: for any
> non-partitioned peer whose ship breaker is open AND that trails the leader,
> it re-arms the backlog re-ship from the peer's durable mark, so the moment the
> breaker half-opens (threshold 5, reset 30s — see
> [§4](#4-grpc-replication-transport-tidal-net)) the leader pushes the WHOLE
> gap. This closes the old footgun: you **no longer re-issue `/cluster/heal`
> until lag 0** — the server drives it. Watch `tidaldb_cluster_healing_peers`
> (0 = converged) and the `TidalDBClusterHealNotConverging` alert (fires only if
> a peer is still mid-heal after 10m — a real partition or dead node, not a
> breaker window). `/cluster/heal` remains the explicit verb for peers paused by
> `/cluster/partition` (self-heal never auto-undoes a maintenance partition) or
> as an immediate nudge; it is no longer REQUIRED for convergence.

### Reconcile (cross-region CRDT convergence)

```bash
curl -X POST "$BASE/cluster/reconcile" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "ap-south" }'
# → { "ok": true, "region": "ap-south",
#     "local_elapsed_ms": 0, "remote_elapsed_ms": 1, "ops_applied": 3 }
```

`/cluster/reconcile` exchanges a CRDT state snapshot with the target region: this
node ships its snapshot into the target's merge AND applies the target's pre-merge
snapshot back, so **both sides converge to identical state** by deterministic LWW.
This is how **hard negatives** (recorded via `/hardnegs`) converge across regions —
they do not ride the WAL relay. `local_elapsed_ms` / `remote_elapsed_ms` are the
merge+apply times each side measured (not the HTTP round-trip); both are typically
0–1ms. Reconcile is **idempotent**: a repeat reconcile of already-converged regions
is an exact no-op on scores (no drift). `/cluster/reconcile/snapshot` is the
INTERNAL snapshot-exchange leg this verb drives (marker required); operators never
call it directly.

### Membership verbs (m11p5 — online add / remove / inspect / reseed)

Membership is **data on the replicated log** (kind-4 records, latest wins,
folded into a `ClusterMembership` cell). All four verbs route to / are answered
by the leader and are **quorum-commit-gated**, one change at a time. The leader
**refuses every conf-change until all current voters report kind-4 capability**
(the mixed-version safety gate — complete the binary upgrade first).

```bash
# Add a node (idempotent by name): forwards to the leader, which assigns the
# next id, appends a Learner record, and answers AFTER quorum-commit.
curl -X POST "$BASE/cluster/join" \
  -H 'Content-Type: application/json' \
  -d '{ "name": "eu-2",
        "grpc_addr": "eu-2.tidaldb-peers.svc.cluster.local:9600",
        "http_addr": "eu-2.tidaldb-peers.svc.cluster.local:9504" }'
# → { "id": 4, "role": "learner", "term": 7, "leader": "us-east", "members": [ … ] }
```

The joiner normally seed-joins itself ([§1b](#1b-multi-process-one-process-per-region));
this verb is the same conf-change for tooling. A re-join from a known name
returns its existing id and current role and appends nothing. The leader's
**auto-promotion** duty (a standing duty, re-armed on every activation and
membership apply — it survives the joining-era leader's death) flips the learner
to Voter once it is within `replication.learner_promote_lag` of the flushed
frontier; `promotion_pending (lag=N)` in `/cluster/status/local` makes a stuck
scale-up diagnosable.

```bash
# Inspect the applied roster (ids, names, addresses, roles, conf version).
curl "$BASE/cluster/members" | jq
```

```bash
# Remove a node: appends a Removed tombstone (quorum-commit-gated). The peer's
# ship cell is retired only AFTER the record is delivered-to/acked-by the removed
# peer (bounded give-up → tidaldb_cluster_remove_delivery_giveups_total); the
# removed node's readiness flips to 503 and it stops campaigning. Its id is BURNED
# (never renumbered, never reused).
curl -X POST "$BASE/cluster/members/remove" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "eu-2" }'
# → { "removed": "eu-2", "membership_version": 9 }
```

```bash
# Force a reseed on demand: latches the durable reseed_required marker. The node
# keeps serving degraded (voting enabled) and reseeds via snapshot on its NEXT
# boot (or self-restarts if replication.reseed_self_restart is true and quorum
# can be sustained without it). See §9.1.
curl -X POST "$BASE/cluster/reseed"
# → { "reseed_required": true }
```

**Scale-down order (decommission):** call `/cluster/members/remove` **first**
(so the cluster's quorum math shrinks before the node disappears), wait for the
record to quorum-commit, then stop / delete the node — lowest ordinal last under
a StatefulSet.

## 7. Sharded scatter-gather API

The `/sharded/*` routes hash-partition entities across regions
(`hash(entity_id) % num_shards`, using the engine's own `ShardRouter` so writes
and reads never disagree). Writes route to the single owning region; reads fan out
to **all** regions and K-way merge by score.

Write routes (each routes to the owning region; a non-owner gateway forwards):

```bash
curl -X POST "$BASE/sharded/items"      -d '{ "entity_id": 7, "metadata": { "title": "..." } }'  # 201
curl -X POST "$BASE/sharded/embeddings" -d '{ "entity_id": 7, "values": [0.1,0.2,0.3,0.4] }'      # 204
curl -X POST "$BASE/sharded/signals"    -d '{ "entity_id": 7, "signal": "view", "weight": 1.0 }'  # 204
```

Read routes (scatter-gather across all regions):

```bash
curl "$BASE/sharded/feed?profile=for_you&limit=20&deadline_ms=50"
curl "$BASE/sharded/search?query=jazz&limit=10&deadline_ms=100"
```

Each sharded read accepts an optional `deadline_ms` — the **total** scatter budget
(default 50ms, server-clamped to 10s). The per-shard deadline is `deadline_ms − 5ms`
network overhead. The response includes a `scatter_gather` block:

```json
{
  "items": [ /* merged, deduped, diversity-enforced, re-ranked */ ],
  "total_candidates": 4210,
  "scatter_gather": {
    "degraded": false,
    "shards_queried": 3,
    "elapsed_ms": 12,
    "shard_deadline_ms": 45
  }
}
```

Degraded semantics: a shard that is partitioned, errors, or misses the deadline is
reported in `unavailable_shards` (a name list) and flips `degraded: true` — it is
**never silently dropped**, and the read **still returns 200** with the live
shards' results. The merge dedups replicated copies of an entity (keeping the
best-scoring copy), reconciles `total_candidates` so replicated shards are not
counted multiple times, and re-enforces `max_per_creator` across the merged set.

## 8. Write-durability contract: the ack knob and its cost (m11p3)

Every replicated write (`/signals`, `/items`, `/embeddings`) runs under one of
two acknowledgment modes. Pick the deployment default with the topology's
`replication.ack`; any caller overrides per request with the **`x-tidal-ack`
header** (`leader` or `quorum`; anything else is a 400).

| | `ack=leader` (default) | `ack=quorum` |
|---|---|---|
| Success means | Durable on the **leader** (storage + WAL group-commit fsync) | Durable on a **majority of the replica set** (leader + `floor(n/2)` followers, each storage-applied + own-WAL-fsynced) |
| Survives | Any follower failure; leader restart (WAL replay) | **Any single node's permanent loss, including the leader's** — promote the max-applied survivor and every acked write is there (the 167-kill-point ledger gate proves it) |
| Does NOT survive | Permanent leader loss before followers caught up (the un-shipped tail dies with it) | Simultaneous majority loss |
| Latency cost | One group-commit fsync (~ms-scale; the macOS F_FULLFSYNC tail is the local floor) | + one ship RTT + the follower's group-commit fsync, **pipelined**: acks are batch-level, so concurrent writers share the round trip the same way they share fsyncs |
| Failure mode | 5xx only for local faults | Additionally a **retryable 503** when the quorum budget (`replication.quorum_timeout_ms`, default 2000) expires — the JSON body names the `laggards`, the `commit_index`, and `needed`/`confirmed` counts |
| Availability | Unaffected by follower outages | **Blocks when a majority is unreachable.** In a 2-region cluster quorum = leader + THE follower: one follower outage stops all quorum writes (by design — that is what the contract says). 3+ regions tolerate `floor((n-1)/2)` follower outages |

Mechanics, in one paragraph: there is **one replicated log** (the leader's
WAL — m11p2); peers only ever receive **fsynced batches** by construction.
Since m11p3 every follower **pushes its durably-applied frontier** back to
the leader once per apply round (`ReportApplied` — batch-level, decoupled
from ship acks, flowing even when the follower converges by catch-up pull or
the leader is quiet); ship acks additionally carry the same frontier as an
instant floor hint. The leader folds both into per-peer durable marks; the
**commit index** is the k-th largest mark (k = `floor(n/2)`), and
`ack=quorum` responses gate on it passing the write's seqno — awaited
asynchronously, so quorum waiters never hold threads or starve completions.
The index is leadership-scoped: promote resets it to the new stream
baseline, and a demoted leader fails its in-flight quorum waiters (it must
never claim quorum for a stream it no longer owns).

Every cluster write's success response carries **`x-tidal-seq`** — the
write's seqno in the replicated log (relayed through gateway forwards).
Persist it if you need an exact durability cursor: `commit_index >= seq` on
`/cluster/status/local` is "this write is majority-durable", regardless of
which mode acked it. (The rare dedup-suppressed signal write — an identical
event within the WAL's ~60s content-hash window is already durably logged,
so no new record exists to track — carries **`x-tidal-deduplicated: 1`**
instead, relayed through forwards like the seq header.)

**Retry semantics under `ack=quorum` — read this twice.** A quorum-timeout
503 means *not confirmed in budget*, not *not written*: the write is in the
leader's log and usually commits moments later. Retries are therefore
**at-least-once**:

- `/items` and `/embeddings` retries are **always safe** — idempotent
  upserts keyed by `entity_id`.
- `/signals` retries can **double-count** the signal's weight when the
  original did commit (the server stamps each request's timestamp, so the
  WAL's content-hash dedup window cannot identify a client retry). The
  distortion is one extra decaying signal per retried timeout — bounded by
  your retry rate (`tidaldb_cluster_quorum_timeouts_total` is exactly that
  budget). Accounting that cannot tolerate it should route through session
  writes (which carry idempotency keys) or dedup client-side on its own key.
- The laggard names in the 503 are your runbook pointer: a persistent
  laggard is a down/partitioned region — heal it ([§6](#6-cluster-management-api))
  or accept leader-ack for the duration (`x-tidal-ack: leader`).

**Rolling upgrades into m11p3 — upgrade the leader first.** A pre-m11p3
leader neither serves the `ReportApplied` RPC nor recognizes `x-tidal-ack`:
it silently applies **leader-ack semantics to a `quorum` request** — a
durability downgrade the caller cannot see. Upgrade order:

1. Promote leadership off the leader node if needed, upgrade it, promote it
   back (or simply upgrade the standing leader per [§9](#9-failover-multi-process)'s
   restart procedure). From this moment `ack=quorum` is honored: the commit
   index rides the m11p2 ship-ack floor hints from not-yet-upgraded
   followers (correct, just laggier).
2. Upgrade followers one at a time. Each upgraded follower starts pushing
   `ReportApplied` and quorum freshness returns to batch-level. (An upgraded
   follower reporting to a still-old leader is harmless — the report is
   refused and logged, replication and heal are unaffected.)

Until step 1 completes, treat the cluster as `ack=leader`-only — do not
point `ack=quorum` traffic at it expecting majority durability.

Other facts unchanged from m11p1/p2: the success code never waits on
shipping for `ack=leader` (sender threads push the WAL flush feed's tail off
the request path; `replication.batch_max_events`/`window` tune it); a failed
fsync errors that write and nothing unfsynced can ship; long-outage data is
follower-pulled via `StreamSegments` (or, when it has rotated past the leader's
retained WAL, via the m11p5 `FetchSnapshot` snapshot transfer); a leader crash
is an **automatic failover** since m11p4 (the survivors elect the
up-to-date successor — see [§9.1](#91-automatic-failover-m11p4--the-default);
the vote restriction only elects a node whose log covers every quorum-acked
write, which is what makes the zero-acked-loss guarantee hold).

In short: **`ack=leader` = leader durability. `ack=quorum` = failover-survivable
durability, priced at one pipelined replication round trip and majority
availability.**

**WAL segment format across upgrades (m11p4).** Segment files carry an
8-byte version header (`TSEG` + version byte; headerless pre-m11p4 files
stay readable — no migration). Three behaviors follow:

- **Unreadable segments fail the boot, loudly.** A node whose WAL dir holds
  segments written by an incompatible tidalDB version (or stray/foreign
  `.seg` files) refuses to start with `WAL segment format unknown: <path>`
  instead of booting with the data invisibly absent (`segments=0` — the
  2026-06-11 p3 rollout failure mode). Remedy: run a compatible binary, or
  reseed the node (delete its PVC and let it pull from the leader).
- **Unservable catch-up resolves via snapshot transfer (m11p5).** A leader
  that cannot serve a follower's requested range from its retained WAL answers
  the `StreamSegments` pull with `FAILED_PRECONDITION` — `"segments not
  available from seq N; snapshot required"`, carrying the typed trailer
  `x-tidal-catchup: snapshot-required`. Since m11p5 the follower **latches a
  durable `reseed_required` marker** (only on that typed trailer — an ordinary
  election term-mismatch never latches it fleet-wide) and reseeds itself via the
  `FetchSnapshot` snapshot stream on its next boot (or self-restarts if
  `replication.reseed_self_restart` is set) — no operator verb, no
  `wipe_data_dir`. See [§9.1](#91-automatic-failover-m11p4--the-default). The
  marker is surfaced in `/cluster/status/local` (`reseed_required: true`) and the
  `tidaldb_cluster_reseed_required` gauge.
- **Failed pulls self-heal on a timer.** A catch-up pull that fails (e.g.
  the leader's gRPC server not yet ready during a rolling restart) retries
  every `replication.catchup_retry_ms` (default 30000) without waiting for
  a write to re-expose the gap — an idle cluster no longer strands lagged
  followers.

**Downgrade hazard (m11p4):** a pre-m11p4 binary reading a header-bearing
segment treats the header as a torn tail and may truncate the final segment.
Downgrading across the m11p4 boundary requires reseeding the node's WAL.

**Membership records + capability gate (m11p5).** A **kind-4 membership
record** is a new WAL blob kind. A kind-4 record shipped to a **pre-p5
follower** is an unknown batch kind → `WalError::Corruption` → that follower's
torn-state receiver halt, **permanent across restarts** (boot self-heal
re-pulls the same record). Both followers halted = a quorum-write outage. To
make this structurally impossible, `HeartbeatResponse`/`ReportApplied` carry a
`capabilities` bit-field (proto3 zero-default = pre-p5 = incapable) and the
leader **refuses `/cluster/join` and every conf-change until all current voters
report kind-4 capability** — so the first conf-change cannot fire mid-upgrade.
**Downgrade rule (kind-3 precedent verbatim): once any kind-4 record is in a
node's WAL, downgrading it below p5 requires a reseed.** Complete the binary
upgrade across all voters before adding, removing, or replacing a node.

## 9. Failover (multi-process)

### 9.1 Automatic failover (m11p4 — the default)

**"A machine died" is a non-event.** Every node runs a failure detector
(leader heartbeats every `election.heartbeat_interval_ms`, default 300) and a
Raft-style election (pre-vote + vote, randomized
`election.election_timeout_{min,max}_ms`, default 1500–3000). Kill the
leader and the survivors elect a successor — typically in **under one
second** with the defaults, bounded well inside 10s — with **zero operator
verbs** and zero acknowledged-write loss (the vote restriction only elects a
node whose log covers every quorum-acked write; the tier-3
`cluster_election.rs::mp_auto_failover_writes_resume_zero_acked_loss` gate
proves it across repeated random kill points under `ack=quorum` load).

What the operator sees:

- `/cluster/status/local` carries `term` (the election term, 0 = the
  pre-election "topology era"), `role` (`leader`/`follower`/`pre-candidate`/
  `candidate`) and `quarantined`.
- During the brief leaderless window, writes return a retryable 503 naming
  the election (`leader: "none (election in progress)"` plus the responding
  node's `term`); clients retry and land on the new leader.
- A **restarted ex-leader can never re-claim leadership from its topology
  file**: its durable election state (`data_dir/election_state`) boots it as
  a follower, and every replication RPC is term-fenced — a deposed leader's
  ships, heartbeats and frontier reports are rejected until it rejoins the
  current term (the §1.4-1 split-brain incident is closed by construction;
  proven by `mp_fenced_ex_leader_restart_cannot_write`).
- A leader that loses contact with a majority **steps down within
  `election.leader_lease_ms`** (default 900) and stops accepting writes:
  `ack=leader` writes during a minority partition are bounded by the lease,
  and `ack=quorum` writes were never at risk.

**Divergent suffix / quarantine → automated reseed (m11p5).** A node that
held leader-acked (never quorum-acked) writes when it died can rejoin into a
cluster that elected past them. It detects this at term-join and
**quarantines**: it serves status (`quarantined: true`, metric
`tidaldb_cluster_divergence_quarantined`) and keeps voting, but refuses the
data plane. Since m11p5 recovery is **automatic and full-history** — no
`wipe_data_dir`: the quarantine latches the durable `reseed_required` marker,
the node reseeds via the `FetchSnapshot` snapshot stream on its next boot (or
self-restarts if `replication.reseed_self_restart` is set and quorum can be
sustained without it), rejoins clean, and the divergence gauge clears. This is
strictly leader-ack-only data, within the documented `ack=leader` crash
contract (§8). The tier-3 `cluster_reseed.rs` proves the full quarantine →
marker → restart → reseeded → gauges-cleared loop.

**The three-way term-join rule (m11p5).** The divergent-suffix check above is
one of three outcomes a node reaches when it joins a new term, comparing its own
stream position against the leader's election-time position (`prev_log`, carried
on the heartbeat):

- `own > prev_log` → **divergent suffix** → quarantine (above);
- `own < prev_log` → the node is genuinely **missing committed-era history**
  that the new stream's baseline jump would silently skip (the p4 carried
  hazard) → it latches `reseed_required` and reseeds via snapshot (no silent
  gap);
- `own == prev_log` (or a within-term rejoin) → **clean**, catch-up via the
  stream.

A snapshot-installed node always joins clean by construction (its WAL is the
leader's copy, so its tail term equals the leader's).

To run the pre-m11p4 posture (operator-driven failover, no automatic
elections, no check-quorum step-down), set in the topology:

```yaml
election:
  auto_election: false
```

### 9.2 Manual promote: a FENCED TRANSFER (maintenance / override)

`POST /cluster/promote { "region": "eu-west" }` remains the maintenance verb
— but it is now a **fenced leadership transfer**, not a view flip:

1. With a live leader: the leader waits for the target to hold the full
   flushed prefix (the catch-up wait IS the drain), then sanctions an
   immediate election (`TimeoutNow`); the target wins term+1 and the old
   leader steps down on first higher-term contact. Response:
   `{ ok, leader, term, transfer: "elected" }`.
2. With a dead leader: the target campaigns among the survivors directly —
   the manual override of the automatic path (and the way to *choose* the
   successor).
3. The election can **refuse a target that lags** (its log loses the
   up-to-date comparison — e.g. the survivor you sampled fell behind between
   your status read and the vote). The verb 503s naming the cause; promote
   the other survivor. **You can no longer accidentally promote a node that
   would discard acked data** — the m11p3 "max-applied survivor" operator
   rule is now enforced by the protocol.
4. Promoting the node that already leads is a no-op 200.

The legacy term-0 fan-out promote survives only for clusters that have never
elected (mixed-version rollouts mid-upgrade, and the deliberate
isolated-node override used by the chaos drills); the first joined election
permanently retires it on each node.

> Rolling upgrade m11p3 → m11p4: upgrade ALL binaries before relying on
> auto-failover (pre-m11p4 peers answer vote RPCs with `Unimplemented`, so
> no election can reach quorum until a majority is upgraded — the cluster
> simply keeps its m11p3 behavior until then). The first ELECTED leader
> journals a kind-3 term-marker WAL record; pre-m11p4 binaries cannot decode
> it, so do not downgrade a node after the first election without reseeding.

## 10. Partition drill (multi-process)

The rewritten drill demonstrates **both** partition mechanisms, exactly as
`cluster_runbook.rs::runbook_s10_partition_drill` scripts them.

1. **Baseline.** `GET /cluster/status`; all `lag_events: 0`, `partitioned: false`,
   `reachable: true`.
2. **Inject.** Use **one** of:
   - **Real network partition** — sever the TCP path peers use to reach the region.
     The chaos suite (`cluster_chaos.rs`) uses a root-free in-harness TCP relay
     proxy (the ROADMAP-sanctioned toxiproxy-style alternative). On a real host you
     can instead use `iptables` (Linux) or `pfctl` (macOS), e.g.
     `iptables -A INPUT -p tcp --dport 9603 -j DROP` to blackhole ap-south's gRPC
     port from a peer. A real cut shows `reachable: false` in the aggregate status.
   - **Simulated ship-skip flag** — `POST /cluster/partition { "region": "ap-south" }`
     (`→ { ok, partitioned: "ap-south" }`). The leader stops shipping to ap-south
     without touching sockets; the aggregate status shows `partitioned: true`.
3. **Write through it.** Send several `POST /signals`. Each must still `204` (the
   leader-durable contract holds). Watch ap-south's `lag_events` climb while its
   `applied_events` stalls — the leader's ships to it are dropped.
4. **Read the stale follower.** `GET /feed?region=ap-south` (or read ap-south's
   gateway directly) returns the pre-partition view — eventual, not strong, read
   consistency. The operator console survives a real partition because clients talk
   to each region's gateway directly.
5. **Scatter-gather degradation.** While partitioned, `GET /sharded/feed` returns
   **200** with `degraded: true` and `ap-south` in `unavailable_shards` — never an
   error, and the live shards' items are still returned.
6. **Heal.** Clear the cut (heal the proxy / drop the firewall rule, or just call
   heal for the simulated flag), then `POST /cluster/heal { "region": "ap-south" }`.
   **Re-issue heal until `/cluster/status` shows ap-south at `lag_events: 0`** — the
   gRPC circuit breaker opened during the partition (threshold 5, reset 30s), so the
   first heal may ship into an open breaker and no-op. This is genuine production
   behavior, not a flag.
7. **Verify convergence.** `GET /cluster/status` shows ap-south `lag_events: 0`,
   `partitioned: false`, `reachable: true`; `/sharded/feed` is no longer degraded;
   feed scores on ap-south match the leader to within float tolerance (no loss, no
   duplication).

## 11. Shutdown

Send `SIGTERM` (or `SIGINT` / Ctrl+C, or stop the container). The server flips
readiness to **503** (so a load balancer stops routing to it), **drains** in-flight
requests, then drops the region's `TidalDb` shutdown path: checkpoint in-memory
signal state → flush storage → write the WAL checkpoint marker + **fsync** → join
the WAL, sweeper, checkpoint, text-syncer, and replication-receiver threads. The
drop is idempotent; the process exits 0. You will see
`region cluster node shutdown: database closed (checkpoint + WAL fsync)`
(multi-process) / `cluster shutdown: closing all nodes (checkpoint + WAL fsync)`
(single-process).

On **restart with the same `--data-dir`**, the region recovers its pre-shutdown
state from the WAL (verified by `cluster_runbook.rs::runbook_s11_shutdown_and_wal_recovery`:
SIGTERM → exit 0 → restart → the same items are served). In multi-process mode, the
restarted node rejoins the cluster and pulls anything it missed while down via
its boot-time `StreamSegments` catch-up request — no operator verb needed
(m11p2; `/cluster/heal` remains the explicit lever for paused peers). This is the per-node step of a
**rolling upgrade** (promote leadership off the node, SIGTERM, restart on the same
data dir with the new binary, heal) — see `cluster_lifecycle.rs::mp_rolling_upgrade_no_loss_no_stall`,
which proves zero acknowledged-write loss across a full rolling upgrade under load.

### Node-replace drill (m11p5)

Replacing a dead or recycled node is `kubectl delete pod` — no topology edit, no
operator verb:

- **PVC retained** (the data dir survives the pod): the replacement boots on the
  same data dir and converges via its boot-time `StreamSegments` catch-up — the
  ordinary restart path above. If it rotated past the leader's retained WAL while
  down, it reseeds via `FetchSnapshot` automatically (the `reseed_required`
  marker + snapshot path, [§9.1](#91-automatic-failover-m11p4--the-default)).
- **PVC deleted + pod deleted** (fresh node): the replacement comes up empty and
  **seed-joins** ([§1b](#1b-multi-process-one-process-per-region)) — it `--seed`s
  a survivor, `FetchSnapshot`s the current state, and the leader auto-promotes it
  back to Voter. Under the `k8s/cluster/` StatefulSet this is the default: the
  pod's args carry `--seed` against the headless Service, so a recreated pod
  rejoins with no human in the loop.

**Add a node:** scale the StatefulSet up (pod N≥3 seed-joins as a learner and
auto-promotes). **Remove a node:** `POST /cluster/members/remove` **first**
([§6](#6-cluster-management-api)), wait for the record to quorum-commit, then
scale down (lowest ordinal last).

## 12. Security (m11p7): mTLS, rotation, identity, audit, rate limits

The cluster does not trust the network. Everything here is **opt-in** — absent
the `grpc_tls` block and the cluster key, the cluster behaves exactly as pre-m11p7
(plaintext, hint-only marker, no audit/limit). A reference (k8s) deployment turns
it all on.

### 12.1 mTLS (gRPC replication) — the default posture

- Configure the `grpc_tls` block per region (`ca_cert`, `server_cert`,
  `server_key`, `client_cert`, `client_key`). The gRPC server then REQUIRES a
  client cert chained to the cluster CA (mutual TLS): a foreign pod with no cert,
  a cert from another CA, or a plaintext probe fails the TLS handshake and never
  reaches an RPC.
- **No `grpc_tls` ⇒ plaintext, with a loud startup WARN** on both the server and
  the client. Acceptable only on a trusted single-host / loopback topology. To
  serve plaintext intentionally there is nothing else to set — the WARN is the
  signal that you are on the insecure path.
- The inter-node HTTP plane (forwards, broadcasts, scatter, status, seed-join)
  is served over TLS with the SAME cert and dials `https://` with the cluster CA
  whenever `grpc_tls` is set, so enabling it gives **zero plaintext inter-node
  links** on both planes at once.

### 12.2 Cert + bearer rotation WITHOUT restart

- A background poller (`TIDAL_ROTATION_POLL_MS`, default 30000) content-hashes the
  cert files and the credential files; on a change it **atomically swaps the
  served cert** (in-flight TLS sessions keep their negotiated keys — zero dropped
  requests) and rebuilds the outbound peer channels.
- **Procedure:** issue a new cert under the same CA (cert-manager renewal, or
  re-run `scripts/gen-cluster-certs.sh` and re-apply the Secret) — the files
  change in place, the poller swaps within one interval, no pod restart. Verify
  with `tidaldb_cluster_*` logs (`TLS material rotated…`) or the `tidal_audit`
  /tracing stream.
- The bearer (`TIDAL_API_KEY_FILE`) and the cluster key (`TIDAL_CLUSTER_KEY_FILE`)
  rotate the same way. Use FILE mounts (not inline env) so a Secret rotation is
  picked up live. During a CA roll, keep both old and new CAs trusted for one
  cycle (CA-overlap) so in-flight connections complete.

### 12.3 Per-node identity + the marker

- Set a shared **cluster key** (`TIDAL_CLUSTER_KEY` / `TIDAL_CLUSTER_KEY_FILE`,
  any random string — BLAKE3-derived to the MAC key). Each node then mints a
  signed `x-tidal-node-token` on every forward/broadcast; the receiver verifies
  it. This gives inter-node calls a verifiable node identity and is the
  defense-in-depth layer beyond the shared bearer.
- With a cluster key configured, the `x-tidal-internal` marker is honored ONLY
  from a verified sibling: a request that sets the marker without a valid node
  token is rejected **403** (the marker is a routing hint, never an auth bypass).
  Never hand the cluster key to external clients.

### 12.4 Admin audit log

- promote / partition / heal / join / member-remove / reseed each emit one
  structured record: `{principal, verb, target, term, outcome}`. The principal is
  the verified node (`node:<id>`) for inter-node calls or `external` for an
  operator with the bearer.
- Sinks: a `tidal_audit` **tracing target** (always — capture it in your log
  pipeline), plus an **append-only JSONL file** when `TIDAL_AUDIT_LOG=<path>` is
  set. Recorded on the operator-originated leg only (no double-record on a
  forwarded re-apply).
- **At-rest encryption** of the JSONL file is delegated to the volume — mount
  `TIDAL_AUDIT_LOG` on an encrypted PV (or a `gVisor`/LUKS-backed volume); the
  server does not encrypt it in-engine.

### 12.5 Per-principal rate limits

- `TIDAL_RATE_LIMIT_RPS` (+ optional `TIDAL_RATE_LIMIT_BURST`, default 2×) caps
  per-principal request rate; a deny is **429 + `Retry-After`**. Off by default.
- Verified sibling nodes are EXEMPT — replication/forward traffic is never
  throttled by the external-client budget. Today external callers share one
  bucket (the shared bearer); a future multi-key registry gives per-key buckets.

### 12.6 Foreign-pod / negative behavior (what an attacker on the network sees)

| Attempt | Result |
|---------|--------|
| Ship a gRPC segment without a cluster client cert | TLS handshake fails — no RPC dispatched |
| Call an internal HTTP route without trusting the cluster CA | TLS handshake fails — no route reached |
| Set `x-tidal-internal` without a valid node token (key configured) | 403 — marker honored only from a verified sibling |
| Call a protected route without the bearer | 401 (unchanged) |

## 13. Coordinated backup / restore + point-in-time recovery (m11p8)

The building blocks: the engine's crash-consistent `create_backup`, the WAL
**archive** (`wal.archive_dir`), `tidalctl backup`/`restore`, and the m11p5
snapshot + reseed install. Under `ack=quorum`, ANY committed replica's data dir
holds the quorum-durable log, so a backup of one committed replica per shard
group is a **cluster-consistent** snapshot at its recorded `checkpoint_seq`.

### 13.1 Enable the WAL archive (point-in-time recovery)

Set `wal.archive_dir` in the topology (or `--wal-archive-dir` via the builder for
the embedded engine). Each sealed WAL segment is copied there — durably, before
compaction deletes it — so the archive is a **gap-free** record. Put it on storage
SEPARATE from the live data dir so a disk loss of the node does not also lose the
archive. Segment filenames encode `shard + first_seq`, so co-located groups share
one archive dir without collision.

```yaml
wal:
  archive_dir: "/archive/tidaldb"   # durable, off-node storage
```

### 13.2 Coordinated backup drill

1. Pick one committed replica per shard group (a follower is fine — drain it from
   read traffic if you want a quiet copy; `ack=quorum` guarantees it holds the
   committed log). Stop it (or snapshot its volume).
2. `tidalctl backup --path /data/<node> --out /backups/<cluster>-<ts>/shard-<id>`
   — writes a recursive copy + `BACKUP_MANIFEST.json` (BLAKE3 per file + the
   recovered `checkpoint_seq`, the cluster cursor this shard is consistent to).
3. Repeat per shard group. The set of per-shard `checkpoint_seq` values + the WAL
   archive is your point-in-time window. Restart the replica; self-heal/catch-up
   reconverges it.

### 13.3 Restore drill (timed)

1. `tidalctl restore --from /backups/<cluster>-<ts>/shard-<id> --path /data/<new-node>`
   — verifies EVERY file's BLAKE3 against the manifest BEFORE writing, and refuses
   a non-empty target (it never overwrites a live data dir).
2. Point a stopped node at the restored dir and boot it. Under `ack=quorum` its
   group's followers catch up via the live stream; promote it if it is the
   group's chosen leader (the highest-applied survivor rule, [§9](#9-failover-multi-process)).
3. **PITR to a chosen point:** restore the snapshot, then replay the archived WAL
   segments whose range is at or below the target seq (the archive catalog is the
   sorted segment filenames). Replay stops at the target — events above it are not
   applied.

Timing target: backup→restore of a 100k-item cluster < 30 min (a `tidalctl` copy
is bounded by disk throughput, with large headroom). The Ref-A timed figure is a
k3s line item (the standing M11 access caveat).

## 14. Rolling upgrade + version skew (m11p8)

Nodes carry a build version on the wire (`HeartbeatRequest.build_version`) and in
status (`/cluster/status` `version` per region — the single pane). Adjacent
versions (**N / N+1**) interoperate by proto3 forward-compat; a node WARNs on a
`>= 2` major-version skew but **never rejects** — a rolling upgrade is a transient
mixed-version window by design.

**Procedure (one node at a time):**

1. `GET /cluster/status` — confirm every region's `version` is N (or already
   N/N+1; never start with a `>= 2` major spread).
2. Graceful SIGTERM one follower → it drains (readiness 503 → checkpoint).
3. Restart it on N+1 (same ports, same data dir). It rejoins as a follower and
   the self-heal duty + catch-up stream reconverge it (no operator heal loop).
4. Repeat for each follower. Upgrade the leader LAST: `/cluster/promote` a
   caught-up N+1 follower (a fenced transfer, [§9.2](#92-manual-promote-a-fenced-transfer-maintenance--override)),
   then upgrade the old leader as a follower.
5. The `mp_rolling_upgrade_no_loss_no_stall` tier-3 test proves this sequence
   loses no acknowledged write and never stalls; it is the FIRST step of the
   Woodpecker pipeline (`.woodpecker.yaml`) — a failure blocks the image build.

> Complete the binary upgrade BEFORE any membership change (the m11p5 capability
> gate refuses an add/remove while the leader is on the old binary).

## Performance (measured over real localhost processes)

| Operation | SLA | Measured (p99 / typical) |
|-----------|-----|--------------------------|
| Replicated `/signals` throughput (m11p1, 3 nodes, release build) | ≥ 2,000/s | **4,534 signal-writes/s** within SLO on the ramp (knee ~5.5k/s); **2,739/s sustained 10 min** (1.65M writes, 0.35% errors); was ~90/s pre-m11p1 |
| Replication lag under that load (m11p1) | < 2s | ≤ 103 events (~40ms) across the 10-min sustain; ≤ 377 events on the 5k/s ramp (follower group-commit coalescing) |
| Cross-region replication (write → follower applied) | < 2s | ~110–133ms p99 (m8p10) |
| Failover (`/cluster/promote` → first successful write) | < 10s | ~31–34ms |
| CRDT reconcile (merge+apply, each side) | < 100ms | 0–1ms |

Write-latency note: signal p50 ≈ 17–25ms with a p99 tail of 160–220ms on
macOS, where `F_FULLFSYNC` averages ~7.4ms with a 10–50ms tail
(`tidaldb_cluster_wal_fsync_us` — 0% complete under 1ms). On Linux
`fdatasync` volumes the same pipeline's fsync floor is far lower; validate
the p99 gate on the reference environment, and tune `wal.batch_timeout_ms`
against the measured fsync histogram.

## Cross-references

- **Kubernetes deployment** — [docs/runbooks/kubernetes.md](kubernetes.md)
  (the hardened standalone single-replica set in [`k8s/`](../../k8s/), and the
  multi-region cluster reference in [`k8s/cluster/`](../../k8s/cluster/) — one
  StatefulSet + headless Service peer discovery + PDB, with `--seed`-based scale
  and `kubectl delete pod` node-replace, shipped in m11p5).
- **Server deployment guide** — [docs/guides/server-deployment.md](../guides/server-deployment.md)
  (standalone and cluster launch, config, env, health probes).
- **Monitoring & alerts** — [docs/ops/monitoring.md](../ops/monitoring.md)
  (Prometheus scrape config, replication-lag and WAL metrics, recommended alerts;
  per-region replication lag is observable via `/cluster/status` `lag_events`;
  remember the `/metrics` endpoint is unauthenticated — bind it internally).
- **Roadmap / cluster status & known gaps** —
  [docs/planning/ROADMAP.md](../planning/ROADMAP.md) and
  [docs/roadmap-to-cluster.md](../roadmap-to-cluster.md) for the M11 cluster
  status — quorum-ack writes (m11p3), automatic failover (m11p4), and
  membership/discovery/elasticity (m11p5) all shipped; sharding × replication
  (p6), security hardening (p7), and continuous correctness (p9) remain.
- **API & schema reference** — [API.md](../../API.md),
  [QUICKSTART.md](../../QUICKSTART.md), and the live `/openapi.json` document.
- **Scope & vision** — [VISION.md](../../VISION.md).