# tidalDB Cluster Runbook

Operating the multi-region `tidal-server cluster` surface: launch (single-process
dev fabric **and** the multi-process region nodes), the operational API,
replication transport facts, failover and partition drills, and the honest
write-durability contract.

> ## STATUS: EXPERIMENTAL — TWO MODES, NEITHER IS QUORUM-ACKED HA YET
>
> Cluster mode has **two shapes**, both behind the same experimental opt-in:
>
> **1. Multi-process (`--region`) — real process isolation.** Each
> `tidal-server cluster --region <name>` process owns **exactly one region**: one
> `TidalDb`, one [`GrpcTransport`](#4-grpc-replication-transport-tidal-net) whose
> server binds *this* region's `grpc_addr` and whose peers are every **sibling
> region's real `grpc_addr`**. Processes peer over real gRPC and forward over real
> HTTP, so a crash of one region's process takes down **only that region** — the
> survivors keep serving. This is genuine process (and, across hosts, host)
> isolation. It is verified end-to-end by the tier-3 suites
> (`cluster_multiproc`, `cluster_chaos`, `cluster_lifecycle`, `cluster_runbook`)
> over real OS processes with real network-partition injection.
>
> **2. Single-process (no `--region`) — the dev/demo default.** Every region runs
> inside **one** process. Replication still traverses the **real `tidal-net` gRPC
> transport** on loopback (faithful multi-region semantics over a real wire), but
> there is **no process isolation**: a crash, OOM, or host failure takes the whole
> "cluster" down at once. This is a development / staging / demo fabric and a
> correctness harness for the replication paths — **not** production HA. It remains
> the default because it needs no per-region topology addresses and no process
> orchestration.
>
> **Honest remaining limits (both modes):**
> * **Quorum durability is opt-in.** The default `204` is leader-durable
>   (storage + WAL fsync; follower ship off the request path). Since m11p3,
>   `ack=quorum` — topology default or per-request `x-tidal-ack` header —
>   gates success on a **majority of the replica set durably holding the
>   write**, surviving permanent leader loss (see
>   [§8](#8-write-durability-contract-the-ack-knob-and-its-cost-m11p3)).
> * **Leadership is automatic** (m11p4). Every node runs a failure detector and
>   a Raft-style election (pre-vote + vote + check-quorum + fenced transfer):
>   kill the leader and the survivors elect a successor — typically under a
>   second with the defaults — with **zero operator verbs** and zero
>   acknowledged-write loss (see [§9.1](#91-automatic-failover-m11p4--the-default)).
>   `/cluster/promote` is now a **fenced transfer** for maintenance/override, not
>   the availability mechanism. `election.auto_election: false` preserves the
>   pre-m11p4 operator-driven posture.
> * **Membership is elastic and addresses are DNS names** (m11p5). `grpc_addr`
>   is an advertised hostname or IP (DNS-resolved on every reconnect); nodes join
>   online via `--seed` and catch up via `FetchSnapshot` + the `StreamSegments`
>   stream; add/remove ride kind-4 membership records on the replicated log
>   (see [§1b](#1b-multi-process-one-process-per-region),
>   [§3](#3-topology-yaml), [§6](#6-cluster-management-api),
>   [§9.1](#91-automatic-failover-m11p4--the-default)).
>
> **For a production deployment today**, run a **single `tidal-server standalone`**
> node backed by host-level redundancy and disk durability (see
> [kubernetes.md](kubernetes.md) and [server-deployment.md](../guides/server-deployment.md)),
> and reach for multi-process cluster mode for read-scale / multi-region
> deployments whose writes need `ack=quorum`'s failover-survivable contract.
>
> **Both modes refuse to start** unless you explicitly opt in with either the
> `--experimental-cluster` flag or the `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1`
> environment variable. On start each emits a loud, **mode-specific** `WARN`
> restating exactly what it does and does not provide. Do not wire either into a
> production load balancer.

## Scope: what cluster mode does and does not own

tidalDB owns **retrieval and ranking only**. Cluster mode adds multi-region
replication of that, nothing more. It does **not** generate embeddings, store
video/blobs, run a CDN or transcoder, do auth/moderation/payments, or provide a
distributed consensus log. Bring your vectors; tidalDB retrieves and ranks over
them. See [VISION.md](../../VISION.md) for the scope boundary.

## Prerequisites

- Rust toolchain ≥ 1.91 if running directly (the Docker build pins
  `rust:1.91-bookworm`).
- `protobuf-compiler` (`protoc`) and a C++ toolchain (`g++`) on the build host —
  `tidal-net`'s build script compiles the WAL-shipping `.proto`, and USearch's
  HNSW core is C++. The Docker image installs both.
- Docker 25+ if running via container.
- For **single-process** mode: one HTTP port (default `9500`); each follower
  region also binds an **OS-assigned loopback port** for its gRPC replication
  server unless you pin one in the topology (see [§3](#3-topology-yaml)).
- For **multi-process** mode: per-region `grpc_addr` **and** `http_addr` declared
  in the topology, and the ports they name available on each host
  (see [§3](#3-topology-yaml)).

## 1. Launch the cluster locally

Cluster mode is gated. Pass `--experimental-cluster` (or set
`TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1`) or the server exits with a mode-specific
error explaining why.

### 1a. Single-process (dev/demo default)

```bash
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
cargo run -p tidal-server -- \
  cluster \
  --listen 127.0.0.1:9500 \
  --schema tidal-server/config/default-schema.yaml \
  --topology tidal-server/config/default-cluster.yaml \
  --experimental-cluster
```

The default topology spins up three regions (`us-east`, `eu-west`, `ap-south`)
with `us-east` as leader, all inside one process. On a clean start you will see a
`WARN` line stating this is experimental and single-process, an `info` line per
follower (`follower gRPC transport ready`), and finally
`listening on http://127.0.0.1:9500`.

### 1b. Multi-process (one process per region)

Pass `--region <name>` (or set `TIDAL_REGION`) to run **only that region** in this
process. The topology must declare a per-region `grpc_addr` **and** `http_addr` for
every region (see [§3](#3-topology-yaml)); siblings reach each other over those.
Launch one process per region — typically one per host, each with its own
`--data-dir`. **`--data-dir` is REQUIRED in multi-process mode** (m11p2): the
durable WAL is the replicated log itself — it is what ships to peers and what
serves their catch-up streams — so a node without one is rejected at startup:

```bash
# Region us-east (the initial leader) on host A
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
tidal-server cluster --experimental-cluster \
  --region us-east \
  --listen 0.0.0.0:9501 \
  --schema /etc/tidal/schema.yaml \
  --topology /etc/tidal/topology.yaml \
  --data-dir /var/lib/tidal/us-east

# Region eu-west on host B
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
tidal-server cluster --experimental-cluster \
  --region eu-west --listen 0.0.0.0:9502 \
  --schema /etc/tidal/schema.yaml --topology /etc/tidal/topology.yaml \
  --data-dir /var/lib/tidal/eu-west

# Region ap-south on host C
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
tidal-server cluster --experimental-cluster \
  --region ap-south --listen 0.0.0.0:9503 \
  --schema /etc/tidal/schema.yaml --topology /etc/tidal/topology.yaml \
  --data-dir /var/lib/tidal/ap-south
```

Every process parses the **same** topology file; `RegionId`s are assigned by
declaration order, so all processes agree on the region→id mapping. Each process
binds its own region's `grpc_addr`/`http_addr` and dials its siblings' addresses.
The `--listen` HTTP address is the public gateway for that region.

#### Seed-join boot (m11p5 — adding a node without editing the topology)

A node can join an existing cluster **online**, without appearing in any
declared topology, by contacting a running peer as a **seed**:

```bash
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
tidal-server cluster --experimental-cluster \
  --region eu-2 \
  --listen 0.0.0.0:9504 \
  --schema /etc/tidal/schema.yaml \
  --topology /etc/tidal/topology.yaml \
  --data-dir /var/lib/tidal/eu-2 \
  --seed http://10.0.1.10:9501 \
  --seed http://10.0.2.10:9502 \
  --advertise-grpc eu-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9600 \
  --advertise-http  eu-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9504 \
  --metrics 0.0.0.0:9091
```

| Flag | Meaning |
|------|---------|
| `--seed <url>` (repeatable) | One or more seed HTTP base URLs. The joiner polls each for the current leader, then `POST /cluster/join`s through it. Any reachable seed works; list a few for resilience. |
| `--advertise-grpc <host:port>` | This node's **advertised** gRPC address — what siblings dial. Required with `--seed` (the joiner has no topology entry of its own). DNS-capable. |
| `--advertise-http <host:port>` | This node's advertised HTTP gateway — what peers forward writes/status to. Required with `--seed`. |
| `--metrics <addr>` | Prometheus `/metrics` bind (the joiner has no topology `metrics_addr`). |

What happens: the joiner skips the topology's "every region declared" gate,
learns its **roster + assigned id + current term** from the seed's join
response, appends a **Learner** record to the replicated log (the leader answers
only after it is quorum-committed), persists the roster to a durable membership
cache and the term to `election_state` (persist-before-act), installs a
[snapshot](#8-write-durability-contract-the-ack-knob-and-its-cost-m11p3) when it
is behind the leader's retained WAL, then streams the live tail. The leader's
**auto-promotion** duty flips it Learner → Voter once it is within
`replication.learner_promote_lag` (default 1024) of the flushed frontier. A
**restart** boots from the cache without the seed.

> **A `--seed` boot still requires the local topology/config file** for the
> behavioral knob blocks (`replication:`, `wal:`, `election:`, `timeouts:`,
> `grpc_tls`) — its `regions:` list is ignored for the roster (the join response
> is the roster), but a bare `--seed` with neither `--topology` nor
> `TIDAL_CONFIG` **refuses to boot** naming the rule, so a joiner never silently
> inherits the compiled-in defaults' wrong ack mode, quorum timeout, or election
> timing. The k8s manifests mount the shared bootstrap ConfigMap on every pod
> including N≥3, so this costs nothing there.

> **Capability gate (m11p5 mixed-version safety).** The leader **refuses
> `/cluster/join` and every conf-change until all current voters report kind-4
> capability** — see [§8](#8-write-durability-contract-the-ack-knob-and-its-cost-m11p3)'s
> downgrade rule. Complete the binary upgrade before adding or removing nodes.

**Auth:** set `TIDAL_API_KEY=<secret>` to require `Authorization: Bearer <secret>`
on the data and `/cluster/*` mutation routes. If it is **unset the server runs
UNAUTHENTICATED and logs a WARN** — never expose an unauthenticated cluster
beyond loopback / a trusted VPC. Health probes and `/openapi.json` are always
unauthenticated. In multi-process mode set the **same** key on every process: a
forwarded/broadcast request passes the caller's `Authorization` through verbatim,
and the internal-propagation marker (`x-tidal-internal: 1`) is an inter-sibling
trust signal, **not** an auth bypass (the bearer middleware still runs first).

Useful environment variables:

| Var | Mode | Effect |
|-----|------|--------|
| `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1` | both | Opt in to cluster mode (alternative to `--experimental-cluster`). |
| `TIDAL_REGION` | multi-process | Selects the single region this process owns (alternative to `--region`). |
| `TIDAL_API_KEY` | both | Bearer token for protected routes. Unset ⇒ unauthenticated + WARN. Set the SAME key on every region in multi-process mode. |
| `TIDAL_HLC_SKEW_MS` | multi-process | Signed ms offset applied to THIS process's HLC. Affects reconcile-time LWW stamping ONLY (not signal-decay timestamps). A test/ops escape hatch for verifying causal convergence under clock skew — do not set it in normal operation. |
| `TIDAL_CONFIG` | both | Config dir holding `default-schema.yaml` / `default-cluster.yaml` (used when `--schema` / `--topology` omitted). |
| `PORT` | both | Listen address. A bare port (`9500`) normalises to `0.0.0.0:9500`. |
| `TIDAL_SERVER_LOG` | both | `tracing` filter (default `info`). |

## 2. Launch via Docker

```bash
# Build the image once (build context is the repo root).
docker build -f docker/cluster/Dockerfile -t tidaldb:cluster .

# Run (press Ctrl+C to stop). The image already sets
# TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 so cluster mode starts; it still logs the
# loud experimental WARN. The default CMD is SINGLE-process cluster mode.
docker run --rm -p 9500:9500 tidaldb:cluster
```

The image bakes the default schema/topology under `/etc/tidal-server` and uses
`ENTRYPOINT ["tidal-server"]` + a `cluster …` `CMD`, so you can override the
subcommand (e.g. `docker run tidaldb:cluster standalone …`), pass `--region` for a
multi-process container, or supply your own config files:

```bash
docker run --rm -p 9500:9500 \
  -e TIDAL_API_KEY=changeme \
  -v "$PWD/configs/my-schema.yaml:/srv/schema.yaml:ro" \
  -v "$PWD/configs/my-topology.yaml:/srv/topology.yaml:ro" \
  tidaldb:cluster \
  cluster \
    --listen 0.0.0.0:9500 \
    --schema /srv/schema.yaml \
    --topology /srv/topology.yaml
```

The container ships a `HEALTHCHECK` that curls `/health`. For Kubernetes/Compose
deployment patterns see [Cross-references](#cross-references).

## 3. Topology YAML

The topology file declares regions and the leader.

**Single-process minimal default** (`tidal-server/config/default-cluster.yaml`):

```yaml
regions:
  - name: us-east
  - name: eu-west
  - name: ap-south
leader: us-east
# Optional: OS worker threads serving cluster write/heal requests (gRPC ship).
# Defaults to available parallelism when omitted.
# write_workers: 4
```

**Multi-process** requires a per-region `grpc_addr` **and** `http_addr` (validated
at startup — a missing or syntactically invalid `host:port` is a hard error naming
the region; reachability is deliberately not probed, since siblings boot in any
order):

`grpc_addr` is an **advertised** address — a literal `host:port` OR a DNS
hostname (m11p5). A new optional per-region `grpc_bind` controls the local bind
independently of what siblings dial; the example below shows the DNS shape (one
shared file names every region by its per-pod DNS name while each pod binds
`0.0.0.0`):

```yaml
regions:
  - name: us-east
    grpc_addr: "tidaldb-0.tidaldb-peers.svc.cluster.local:9601"  # ADVERTISED (siblings dial; DNS re-resolved on reconnect)
    grpc_bind: "0.0.0.0:9601"      # LOCAL bind (optional; see derivation rule below)
    http_addr: "10.0.1.10:9501"    # this region's public HTTP gateway (forwarding + status)
    metrics_addr: "10.0.1.10:9091" # optional Prometheus /metrics listener (set it in production)
  - name: eu-west
    grpc_addr: "tidaldb-1.tidaldb-peers.svc.cluster.local:9602"
    grpc_bind: "0.0.0.0:9602"
    http_addr: "10.0.2.10:9502"
    metrics_addr: "10.0.2.10:9091"
  - name: ap-south
    grpc_addr: "tidaldb-2.tidaldb-peers.svc.cluster.local:9603"
    grpc_bind: "0.0.0.0:9603"
    http_addr: "10.0.3.10:9503"
    metrics_addr: "10.0.3.10:9091"
leader: us-east
# Optional m11p1 tuning blocks (engine defaults shown):
# replication:
#   batch_max_events: 256   # events coalesced per shipped batch (1-256)
#   window: 4               # in-flight batches per peer (1-64)
#   retry_ms: 100           # backoff before a transiently-failed batch retries
#   catchup_retry_ms: 30000           # backoff before a FAILED catch-up pull re-pulls (m11p4)
#   snapshot_artifact_ttl_ms: 600000  # staged-snapshot reuse window (m11p5; never unpins an active consumer)
#   learner_promote_lag: 1024         # learner→voter promotion distance + readiness hysteresis (m11p5)
#   reseed_self_restart: false        # drain+exit(0) when reseed_required latches (m11p5; k8s sets true)
# wal:
#   batch_size: 100         # events per group-commit fsync (1-256)
#   batch_timeout_ms: 10    # max wait before a partial batch flushes
# election:                 # m11p4 failure-detector / election timers (defaults shown)
#   heartbeat_interval_ms: 300
#   election_timeout_min_ms: 1500     # validated: leader_lease_ms + heartbeat_interval_ms < this
#   election_timeout_max_ms: 3000
#   leader_lease_ms: 900
#   auto_election: true               # false → pre-m11p4 operator-driven failover
```

**`grpc_bind` derivation (m11p5 §1):** `grpc_bind` present → bind it. Absent +
`grpc_addr` parses as a literal SocketAddr → bind that (today's behavior,
byte-for-byte — every existing IP topology keeps working). Absent + `grpc_addr`
is a hostname → bind `0.0.0.0:<port from grpc_addr>`. Because the peer-dial path
no longer parses `grpc_addr` as a SocketAddr, hyper re-resolves the hostname on
every reconnect — a pod rescheduled onto a new IP is reached with no peer
restarts. **TLS note:** SNI follows the URI host, so DNS peer names require
DNS-SAN certs (see `grpc_tls` below); no code change.

Fields:

| Field | Single-process | Multi-process | Meaning |
|-------|----------------|---------------|---------|
| `regions[].name` | required | required | Region name used everywhere in the HTTP API (`?region=`, `/cluster/promote`, etc.). Must be unique. |
| `regions[].grpc_addr` | optional | **required** | This region's **advertised** gRPC replication address — what siblings dial. Since m11p5 it may be a literal `host:port` OR a **DNS hostname** (re-resolved on every reconnect). In single-process mode **omit it** — the server allocates a free loopback port and self-heals a bind race by retrying on a fresh port. In multi-process mode it is required and is tried exactly once. |
| `regions[].grpc_bind` | unused | optional | This region's **local** gRPC bind `host:port` (m11p5), independent of the advertised `grpc_addr`. Omitted: a literal `grpc_addr` binds itself; a hostname `grpc_addr` binds `0.0.0.0:<its port>`. Set it to bind a specific interface while advertising a DNS name. |
| `regions[].http_addr` | unused | **required** | This region's public HTTP gateway address, used by siblings for write/read forwarding and status aggregation. Unused in single-process mode. |
| `regions[].grpc_tls` | unused | optional | TLS material for this region's gRPC transport: `ca_cert`, `server_cert`, `server_key` (PEM paths), plus optional `client_cert`/`client_key` for mTLS. Omitted ⇒ plaintext, the right posture for loopback/VPC topologies. **DNS `grpc_addr` requires DNS-SAN certs** (SNI follows the dialed hostname). |
| `regions[].metrics_addr` | unused | optional | Bind address for this region's Prometheus `/metrics` listener (m11p1; cluster mode previously had none). Omitted ⇒ no metrics endpoint. Set it in every production topology, and bind it internally — the endpoint is unauthenticated. |
| `leader` | required | required | Must name one of the declared regions. The initial write leader. |
| `write_workers` | optional | optional | Size of the runtime-free OS-thread pool that admission-controls cluster writes (`/signals` staging) and runs `/cluster/heal`'s blocking redelivery. Bounded queue ⇒ 429 backpressure. Must be ≥ 1 when given. |
| `timeouts.broadcast_peer_secs` | unused | optional | Per-peer budget (seconds) for the `/cluster/promote` fan-out (the only remaining peer fan-out — the m11p2 log replaced the item/embedding broadcast). Default 2s — right for loopback/VPC; raise it for WAN topologies where a distant region cannot answer in 2s. Must be ≥ 1 when given. |
| `replication.batch_max_events` | unused | optional | Max relay events coalesced into one shipped batch (1–256, the WAL wire-format ceiling). Default 256. |
| `replication.window` | unused | optional | In-flight batches per peer (1–64). 1 = strictly in-order shipping; higher pipelines across the peer RTT (out-of-order arrivals park gap-aware on the receiver). Default 4. |
| `replication.retry_ms` | unused | optional | Backoff (ms) before a transiently-failed batch ship retries. Default 100. |
| `replication.ack` | unused | optional | Deployment-default write acknowledgment: `leader` (default) or `quorum` (majority-durable — see [§8](#8-write-durability-contract-the-ack-knob-and-its-cost-m11p3)). Per-request override: the `x-tidal-ack` header. |
| `replication.quorum_timeout_ms` | unused | optional | Budget an `ack=quorum` write waits for the commit index before the retryable 503 naming the laggards. Default 2000. |
| `replication.catchup_retry_ms` | unused | optional | Backoff (ms) before a FAILED catch-up pull re-pulls on a timer (m11p4) — lets an idle cluster self-heal a follower whose pull failed during a rolling restart. Default 30000. Must be ≥ 1 when given. |
| `replication.snapshot_artifact_ttl_ms` | unused | optional | Staged-snapshot reuse window (m11p5), counted from the last fetch's completion. Governs artifact REUSE only — **never** unpins the WAL retention of an active consumer; a hard cap (4×) force-drops a never-releasing pin (`tidaldb_cluster_snapshot_pin_force_drops_total`). Default 600000 (10 min). Must be ≥ 1. |
| `replication.learner_promote_lag` | unused | optional | Learner→voter promotion distance AND the readiness-convergence hysteresis threshold (m11p5), in events. Default 1024. Must be ≥ 1. |
| `replication.reseed_self_restart` | unused | optional | Drain + clean-exit(0) once the durable `reseed_required` marker latches (m11p5; k8s sets it `true`). **Refused** when the remaining voters can't sustain quorum without this node. Default `false`. |
| `election.heartbeat_interval_ms` | unused | optional | Leader heartbeat interval (m11p4). Default 300. |
| `election.election_timeout_min_ms` / `..max_ms` | unused | optional | Randomized follower election timeout window (m11p4). Defaults 1500 / 3000. Validated: `leader_lease_ms + heartbeat_interval_ms < election_timeout_min_ms`. |
| `election.leader_lease_ms` | unused | optional | Leader freshness lease — a leader that loses majority contact steps down within it (m11p4). Default 900. |
| `election.auto_election` | unused | optional | `true` (default) = automatic failover; `false` = pre-m11p4 operator-driven posture (no auto elections, no check-quorum step-down). |
| `wal.batch_size` | optional | optional | Events per WAL group-commit fsync (1–256). Default 100. |
| `wal.batch_timeout_ms` | optional | optional | Max ms a partial group-commit batch waits before flushing. Default 10. Tune against the measured `tidaldb_cluster_wal_fsync_us` on the deployment's volume. |

The schema YAML (signals / text_fields / embedding_slots / profiles) is the same
format the standalone server loads — see [API.md](../../API.md) and
[QUICKSTART.md](../../QUICKSTART.md) for the full schema/profile grammar and the
built-in ranking profiles. Topology only adds the region layout above.

> **Personalization correctness note.** Preference-vector personalization (the
> taste vector that powers `for_you`) only updates for signals declared
> `positive_engagement: true` in the schema. The cluster `/signals` route writes
> **global** signals (no `user_id` / `creator_id` context), so it never folds
> item embeddings into a user's taste vector regardless of the
> `positive_engagement` flag. Personalized writes go through the embedded engine
> (`signal_with_context`), not this HTTP route. The exception is `/hardnegs`,
> which records a **user-scoped** hide that converges across regions via
> [`/cluster/reconcile`](#6-cluster-management-api).

## 4. gRPC replication transport (tidal-net)

Replication between regions is the real `tidal-net` `WalShipping` gRPC service,
not in-process channels. The facts you need to operate and firewall it:

| Property | Value | Notes |
|----------|-------|-------|
| Service | `WalShipping` (proto `tidal.replication.v1`) | RPCs: **`ShipSegment`** (unary; the live-tail push path), **`StreamSegments`** (server-streaming; the follower-pulled catch-up path since m11p2), **`FetchSnapshot`** (server-streaming; the m11p5 snapshot transfer for joiners + reseeds), **`JoinCluster`** (m11p5 conf-change), and **`Heartbeat`** (ControlPlane health). |
| Bind address | per-region `grpc_addr` (multi-process); an **auto-allocated** loopback port (single-process) | Default dev band `59520–59529` if you pin one. |
| Transport security | mTLS via `TlsConfig` (`ca_cert`, `server_cert`, `server_key`, optional `client_cert` / `client_key`) | `insecure = true` (plaintext) is the loopback/VPC default in this phase. A peer with no TLS config **must** set `insecure`. |
| Max payload | **64 MiB** | Both encoder and decoder codec limits are raised to this; pinned by a compile-time assert to the engine's `InProcessTransport` limit so both ends agree. WAL segments default to 16 MiB. |
| Circuit breaker | per-peer, **threshold 5**, **reset 30s** | After 5 consecutive ship failures the breaker opens; it stays open for 30s, then the next attempt probes (HalfOpen → Closed on success, Open again on failure). Backpressure (channel full) does **not** trip it. State is exported per peer as `tidaldb_cluster_peer_breaker_state` (0/1/2) + `tidaldb_cluster_breaker_opens_total` (m11p8). **Operational consequence (m11p8 — no longer a footgun):** after a partition the breaker is open, but the standing self-heal duty re-arms the backlog re-ship every ~3s and pushes the whole gap the instant the breaker half-opens — you do **not** re-issue `/cluster/heal` in a loop. Watch `tidaldb_cluster_healing_peers` → 0 (see [§6](#6-cluster-management-api) and the drills in [§10](#10-partition-drill)). |
| Timeouts (defaults) | connect 5s, request 10s, keep-alive PING every 10s / 5s ACK | A blackholed peer fails fast instead of stalling the single-threaded shipper. |

`ShipSegment` carries the WAL segment id, BLAKE3-validated payload bytes, event
count, and the leader's authoritative `leader_last_seq` (so a follower can
advance its replication-lag high-water-mark even for an all-local segment that
filters to empty). The follower's segment-receiver thread drains and applies it
on arrival. The engine's applied high-water-mark is a **contiguous frontier with a
bounded ahead-buffer**, so out-of-order eager ships can never swallow a sequence
gap (a silent-data-loss class fixed in m8p10).

> **Operational guard:** the gRPC replication ports and the Prometheus `/metrics`
> endpoint are **UNAUTHENTICATED**. Bind them to loopback or a cluster-internal
> network only — never the public interface.

## 5. Core HTTP API

All routes are JSON unless noted. Examples assume `BASE=http://localhost:9501`
(any region's gateway in multi-process mode; the single `--listen` address in
single-process mode) and, when `TIDAL_API_KEY` is set,
`AUTH='-H "Authorization: Bearer $TIDAL_API_KEY"'`. Drop the `-H` header when
running unauthenticated.

Middleware mirrors standalone: 30s request timeout (408), 100 max in-flight
(429), 2 MB body limit (413), and an `x-request-id` on every response. Health
probes sit **outside** the load-shedding stack so liveness/readiness are never
queued or timed out under saturation.

> **Multi-process routing (one coherent surface).** Any region's gateway accepts
> any operation and routes it to the node that owns it:
> * **Writes** (`/signals`, `/items`, `/embeddings`, `/hardnegs`) on a non-leader
>   **forward to the leader** (the caller's `Authorization` passes through); the
>   client sees the leader's status/body. A leader that is unreachable degrades to
>   a `503` naming the leader, never a hang.
> * **Reads** (`/feed`, `/search`) default to the **LOCAL** region. This is the
>   key difference from single-process mode, whose default read is the leader. A
>   `?region=<other>` read **forwards** to that region's process.
> * `/cluster/promote`, `/cluster/partition`, `/cluster/heal` route to / fan out
>   from the leader as documented in [§6](#6-cluster-management-api).

### Health

```bash
curl "$BASE/health"          # 200 ok / 503 while draining; reports mode, region, leader
curl "$BASE/health/startup"  # always 200
curl "$BASE/health/live"     # always 200
curl "$BASE/openapi.json"    # served OpenAPI 3.1, UNAUTHENTICATED — canonical HTTP reference
```

`/health` returns `{ "ok": true, "service": "tidaldb", "mode": "cluster",
"region": "us-east", "leader": "us-east", ... }`. The `/openapi.json` document is
the machine-readable source of truth for the **data**, **`/cluster/*`**,
**`/sharded/*`**, and **`/hardnegs`** request/response shapes (the health probes
are intentionally outside the documented API surface).

### Register items & embeddings (one replicated log — m11p2)

Items and embeddings ride the **same replicated WAL stream as signals**: the
leader journals each write as a kind-1 (item metadata) or kind-2 (embedding)
WAL record BEFORE touching storage, and followers apply it from the log —
live pushes for the hot tail, the `StreamSegments` catch-up stream for
history. There is **no HTTP broadcast** anymore (the m8p10 side channel and
its bug classes were deleted in m11p2), so the responses are plain statuses:

```bash
curl -X POST "$BASE/items" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "metadata": { "title": "Jazz Piano", "category": "music" } }'
# → 201 Created (item durably journaled into the replicated log + applied)

curl -X POST "$BASE/embeddings" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "values": [0.1, 0.2, 0.3, 0.4] }'
# → 204 No Content
```

The `201`/`204` asserts **leader durability** and carries **`x-tidal-seq`**
(the record's seqno in the replicated log); `x-tidal-ack: quorum` upgrades it
to a majority-durable ack (see
[§8](#8-write-durability-contract-the-ack-knob-and-its-cost-m11p3)). The
record is fsynced into the stream every follower receives (live push, or
pull-based catch-up after downtime — see
[§6 heal](#6-cluster-management-api)). A write to a non-leader forwards to
the leader transparently (the ack header and seq header travel through the
forward). A down/partitioned peer needs no backfill bookkeeping: it
converges from the log when it returns.

Embeddings: tidalDB does **not** generate vectors — the caller brings them. The
write L2-normalizes and inserts into the HNSW index. Dimensions are **strict**:
they must equal the slot's declared dimensions (min 2, max 4096) or the insert is
**rejected with a 500** (the engine surfaces a dimension mismatch as an internal
error, not a 400 — a known wart, tracked post-M8); **zero-norm vectors are also
rejected (500)**. `RETRIEVE` / `SEARCH` route through the **first declared
embedding slot only**; multi-modal apps must fuse offline or use separate entity
kinds.

### Record signals (cluster `/signals` = global only)

```bash
curl -X POST "$BASE/signals" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "signal": "view", "weight": 1.0 }'
# → 204 No Content (see the durability contract in §8)
```

The signal name must be declared in the schema (an undeclared name returns **400**
naming it). The 204 carries **`x-tidal-seq`** (the write's replicated-log seqno),
and `x-tidal-ack: quorum` upgrades it to a majority-durable ack — see
[§8](#8-write-durability-contract-the-ack-knob-and-its-cost-m11p3). This route
records a **global** signal on the leader and ships it to
followers over the replicated WAL stream; it does not personalize (see the
personalization note in [§3](#3-topology-yaml)). On a non-leader gateway it
forwards to the leader transparently and still returns `204`.

### Hard negatives (user-scoped hides)

```bash
curl -X POST "$BASE/hardnegs" \
  -H 'Content-Type: application/json' \
  -d '{ "user_id": 42, "item_id": 7 }'
# → 204 No Content
```

Records a user-scoped hide on the leader (a non-leader gateway forwards). The item
is filtered from that user's `/feed?user_id=42`. Hard negatives converge across
regions via the LWW-resolved [`/cluster/reconcile`](#6-cluster-management-api) CRDT
path — NOT the signal WAL relay and NOT a broadcast.

### Retrieve and search (region-pinned reads)

```bash
# Default read region is the LOCAL region (multi-process mode).
curl "$BASE/feed?user_id=42&profile=for_you&limit=20"
curl "$BASE/search?query=jazz%20piano&user_id=42&limit=5"

# Pin a read to a specific region. The gateway forwards to that region's process.
# Followers may lag the leader (and lag jumps during a partition) — use this for
# canary reads and lag verification.
curl "$BASE/feed?profile=trending&region=eu-west"
```

`?region=` accepts any declared region name; an unknown name returns **400**. Omit
it and the read serves from the **LOCAL** region (multi-process) / the current
leader (single-process). `limit` is clamped to 1000 at the trust boundary (a
larger value cannot amplify memory unboundedly). The `for_you` profile applies the
built-in diversity defaults (`max_per_creator=2`, `format_mix_max_fraction=0.4`,
`exploration=0.1`).

## 6. Cluster management API

### Check cluster status

Two views. `/cluster/status/local` reports THIS node's own replication/leadership
state (no peer calls); `/cluster/status` aggregates EVERY region (the gateway calls
each peer's `/cluster/status/local` concurrently with a tight per-peer budget).

```bash
curl "$BASE/cluster/status/local" | jq
```

```json
{
  "region": "ap-south",
  "is_leader": false,
  "leader": "us-east",
  "last_seq": 0,
  "applied_events": 124,
  "lag_events": 1,
  "partitioned": [],
  "reachable": true
}
```

```bash
curl "$BASE/cluster/status" | jq
```

```json
{
  "leader": "us-east",
  "relay_log_len": 125,
  "regions": [
    { "name": "us-east",  "applied_events": 125, "lag_events": 0, "partitioned": false, "reachable": true },
    { "name": "eu-west",  "applied_events": 125, "lag_events": 0, "partitioned": false, "reachable": true },
    { "name": "ap-south", "applied_events": 124, "lag_events": 1, "partitioned": false, "reachable": true }
  ]
}
```

`relay_log_len` is the leader's high-water-mark (`last_seq`); each region's
`lag_events` is `relay_log_len − applied_events` (saturating). Since m11p3
`/cluster/status/local` also reports the node's `ack` default and, on the
leader, `commit_index` — the highest seqno a majority of the replica set
durably holds (`last_seq − commit_index` is the quorum lag). A region the
gateway **cannot reach** within the per-peer budget is reported honestly as
`reachable: false`, `partitioned: true`, `applied_events: 0`, and worst-case lag
(`lag_events == relay_log_len`). A non-zero, *growing* lag on a reachable region is
the signal that it is partitioned (ship-skip) or its segment-receiver is wedged.

### Promote a new leader

```bash
curl -X POST "$BASE/cluster/promote" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "eu-west" }'
# → { "ok": true, "leader": "eu-west", "acked": ["eu-west","ap-south"], "failed": [] }
```

The gateway first resolves the TARGET's **stream baseline** — the new leader's
WAL flushed frontier at promotion, persisted in its `data_dir/stream_baseline`
— then fans the promote out to every peer carrying it (internal marker, so
each applies locally and does not re-fan). Peers jump their applied frontier
for the new leader's shard to the baseline: everything at or below it is
pre-stream history (replicated applies of the OLD stream), not data. The
response carries `baseline` plus the `acked`/`failed` fan-out report; a peer
in `failed` (e.g. a dead old leader) self-corrects — its first parked batch
from the new stream triggers a catch-up pull whose chunks announce the
baseline. After promotion `/cluster/status` reports the new leader; new
writes route there; the new leader's ship queue activates and the demoted
node's deactivates. An unknown region returns **400**. Since m11p4 `promote` is
a **fenced transfer** (it drains the target, then sanctions an election), not
the availability mechanism — automatic failover handles a dead leader with no
operator action ([§9.1](#91-automatic-failover-m11p4--the-default)); this verb
is for maintenance and deliberate successor choice
([§9.2](#92-manual-promote-a-fenced-transfer-maintenance--override)).

### Simulate a partition & heal

There are **two** ways to partition a region; the runbook drills demonstrate both
([§10](#10-partition-drill)):

1. **Simulated ship-skip flag** — `/cluster/partition` tells the leader to stop
   shipping to the named region (no sockets touched). Good for a controlled,
   reversible lag demo.
2. **Real network partition** — sever the actual TCP path between processes
   (firewall / proxy). The chaos suite uses a root-free in-harness TCP relay;
   operators can use `iptables`/`pfctl` (see [§10](#10-partition-drill)).

```bash
# Simulated: isolate ap-south — leader ships skip this follower, so its lag climbs.
curl -X POST "$BASE/cluster/partition" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "ap-south" }'
# → { "ok": true, "partitioned": "ap-south" }

# Heal: resume shipping past everything the follower reports applied, and
# nudge it to PULL any history that rotated out of the leader's ship tail
# (the single recovery verb — log catch-up IS the full heal since m11p2).
curl -X POST "$BASE/cluster/heal" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "ap-south" }'
# → { "ok": true, "healed": "ap-south" }
```

`/cluster/heal` is the **single recovery verb**, and since m11p2 it is pure
log catch-up: it clears the partition flag, **resumes** the ship queue past
the follower's reported applied seqno (retries of data the follower already
holds prune automatically — every ship ack piggybacks the follower's applied
seqno), and **nudges** the follower to pull anything older than the leader's
in-memory ship tail via its `StreamSegments` catch-up stream over the
leader's durable WAL segments (`POST /cluster/catchup`, internal; the nudge
forwards the healing operator's own bearer credential). Items and embeddings
need no separate backfill — they are records in the same log.

> **Convergence is self-driving (m11p8 — the operator no longer loops heal).**
> The per-peer ship senders retry parked batches every `replication.retry_ms`
> (default 100ms); a follower that detects a gap pulls the catch-up stream
> itself (also on boot, so a restarted node converges with no operator action
> at all — the tier-3 `mp_items_ride_the_log_and_catchup_stream` proves it).
> On top of that, a **standing leader heal duty** runs every ~3s: for any
> non-partitioned peer whose ship breaker is open AND that trails the leader,
> it re-arms the backlog re-ship from the peer's durable mark, so the moment the
> breaker half-opens (threshold 5, reset 30s — see
> [§4](#4-grpc-replication-transport-tidal-net)) the leader pushes the WHOLE
> gap. This closes the old footgun: you **no longer re-issue `/cluster/heal`
> until lag 0** — the server drives it. Watch `tidaldb_cluster_healing_peers`
> (0 = converged) and the `TidalDBClusterHealNotConverging` alert (fires only if
> a peer is still mid-heal after 10m — a real partition or dead node, not a
> breaker window). `/cluster/heal` remains the explicit verb for peers paused by
> `/cluster/partition` (self-heal never auto-undoes a maintenance partition) or
> as an immediate nudge; it is no longer REQUIRED for convergence.

### Reconcile (cross-region CRDT convergence)

```bash
curl -X POST "$BASE/cluster/reconcile" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "ap-south" }'
# → { "ok": true, "region": "ap-south",
#     "local_elapsed_ms": 0, "remote_elapsed_ms": 1, "ops_applied": 3 }
```

`/cluster/reconcile` exchanges a CRDT state snapshot with the target region: this
node ships its snapshot into the target's merge AND applies the target's pre-merge
snapshot back, so **both sides converge to identical state** by deterministic LWW.
This is how **hard negatives** (recorded via `/hardnegs`) converge across regions —
they do not ride the WAL relay. `local_elapsed_ms` / `remote_elapsed_ms` are the
merge+apply times each side measured (not the HTTP round-trip); both are typically
0–1ms. Reconcile is **idempotent**: a repeat reconcile of already-converged regions
is an exact no-op on scores (no drift). `/cluster/reconcile/snapshot` is the
INTERNAL snapshot-exchange leg this verb drives (marker required); operators never
call it directly.

### Membership verbs (m11p5 — online add / remove / inspect / reseed)

Membership is **data on the replicated log** (kind-4 records, latest wins,
folded into a `ClusterMembership` cell). All four verbs route to / are answered
by the leader and are **quorum-commit-gated**, one change at a time. The leader
**refuses every conf-change until all current voters report kind-4 capability**
(the mixed-version safety gate — complete the binary upgrade first).

```bash
# Add a node (idempotent by name): forwards to the leader, which assigns the
# next id, appends a Learner record, and answers AFTER quorum-commit.
curl -X POST "$BASE/cluster/join" \
  -H 'Content-Type: application/json' \
  -d '{ "name": "eu-2",
        "grpc_addr": "eu-2.tidaldb-peers.svc.cluster.local:9600",
        "http_addr": "eu-2.tidaldb-peers.svc.cluster.local:9504" }'
# → { "id": 4, "role": "learner", "term": 7, "leader": "us-east", "members": [ … ] }
```

The joiner normally seed-joins itself ([§1b](#1b-multi-process-one-process-per-region));
this verb is the same conf-change for tooling. A re-join from a known name
returns its existing id and current role and appends nothing. The leader's
**auto-promotion** duty (a standing duty, re-armed on every activation and
membership apply — it survives the joining-era leader's death) flips the learner
to Voter once it is within `replication.learner_promote_lag` of the flushed
frontier; `promotion_pending (lag=N)` in `/cluster/status/local` makes a stuck
scale-up diagnosable.

```bash
# Inspect the applied roster (ids, names, addresses, roles, conf version).
curl "$BASE/cluster/members" | jq
```

```bash
# Remove a node: appends a Removed tombstone (quorum-commit-gated). The peer's
# ship cell is retired only AFTER the record is delivered-to/acked-by the removed
# peer (bounded give-up → tidaldb_cluster_remove_delivery_giveups_total); the
# removed node's readiness flips to 503 and it stops campaigning. Its id is BURNED
# (never renumbered, never reused).
curl -X POST "$BASE/cluster/members/remove" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "eu-2" }'
# → { "removed": "eu-2", "membership_version": 9 }
```

```bash
# Force a reseed on demand: latches the durable reseed_required marker. The node
# keeps serving degraded (voting enabled) and reseeds via snapshot on its NEXT
# boot (or self-restarts if replication.reseed_self_restart is true and quorum
# can be sustained without it). See §9.1.
curl -X POST "$BASE/cluster/reseed"
# → { "reseed_required": true }
```

**Scale-down order (decommission):** call `/cluster/members/remove` **first**
(so the cluster's quorum math shrinks before the node disappears), wait for the
record to quorum-commit, then stop / delete the node — lowest ordinal last under
a StatefulSet.

## 7. Sharded scatter-gather API

The `/sharded/*` routes hash-partition entities across regions
(`hash(entity_id) % num_shards`, using the engine's own `ShardRouter` so writes
and reads never disagree). Writes route to the single owning region; reads fan out
to **all** regions and K-way merge by score.

Write routes (each routes to the owning region; a non-owner gateway forwards):

```bash
curl -X POST "$BASE/sharded/items"      -d '{ "entity_id": 7, "metadata": { "title": "..." } }'  # 201
curl -X POST "$BASE/sharded/embeddings" -d '{ "entity_id": 7, "values": [0.1,0.2,0.3,0.4] }'      # 204
curl -X POST "$BASE/sharded/signals"    -d '{ "entity_id": 7, "signal": "view", "weight": 1.0 }'  # 204
```

Read routes (scatter-gather across all regions):

```bash
curl "$BASE/sharded/feed?profile=for_you&limit=20&deadline_ms=50"
curl "$BASE/sharded/search?query=jazz&limit=10&deadline_ms=100"
```

Each sharded read accepts an optional `deadline_ms` — the **total** scatter budget
(default 50ms, server-clamped to 10s). The per-shard deadline is `deadline_ms − 5ms`
network overhead. The response includes a `scatter_gather` block:

```json
{
  "items": [ /* merged, deduped, diversity-enforced, re-ranked */ ],
  "total_candidates": 4210,
  "scatter_gather": {
    "degraded": false,
    "shards_queried": 3,
    "elapsed_ms": 12,
    "shard_deadline_ms": 45
  }
}
```

Degraded semantics: a shard that is partitioned, errors, or misses the deadline is
reported in `unavailable_shards` (a name list) and flips `degraded: true` — it is
**never silently dropped**, and the read **still returns 200** with the live
shards' results. The merge dedups replicated copies of an entity (keeping the
best-scoring copy), reconciles `total_candidates` so replicated shards are not
counted multiple times, and re-enforces `max_per_creator` across the merged set.

## 8. Write-durability contract: the ack knob and its cost (m11p3)

Every replicated write (`/signals`, `/items`, `/embeddings`) runs under one of
two acknowledgment modes. Pick the deployment default with the topology's
`replication.ack`; any caller overrides per request with the **`x-tidal-ack`
header** (`leader` or `quorum`; anything else is a 400).

| | `ack=leader` (default) | `ack=quorum` |
|---|---|---|
| Success means | Durable on the **leader** (storage + WAL group-commit fsync) | Durable on a **majority of the replica set** (leader + `floor(n/2)` followers, each storage-applied + own-WAL-fsynced) |
| Survives | Any follower failure; leader restart (WAL replay) | **Any single node's permanent loss, including the leader's** — promote the max-applied survivor and every acked write is there (the 167-kill-point ledger gate proves it) |
| Does NOT survive | Permanent leader loss before followers caught up (the un-shipped tail dies with it) | Simultaneous majority loss |
| Latency cost | One group-commit fsync (~ms-scale; the macOS F_FULLFSYNC tail is the local floor) | + one ship RTT + the follower's group-commit fsync, **pipelined**: acks are batch-level, so concurrent writers share the round trip the same way they share fsyncs |
| Failure mode | 5xx only for local faults | Additionally a **retryable 503** when the quorum budget (`replication.quorum_timeout_ms`, default 2000) expires — the JSON body names the `laggards`, the `commit_index`, and `needed`/`confirmed` counts |
| Availability | Unaffected by follower outages | **Blocks when a majority is unreachable.** In a 2-region cluster quorum = leader + THE follower: one follower outage stops all quorum writes (by design — that is what the contract says). 3+ regions tolerate `floor((n-1)/2)` follower outages |

Mechanics, in one paragraph: there is **one replicated log** (the leader's
WAL — m11p2); peers only ever receive **fsynced batches** by construction.
Since m11p3 every follower **pushes its durably-applied frontier** back to
the leader once per apply round (`ReportApplied` — batch-level, decoupled
from ship acks, flowing even when the follower converges by catch-up pull or
the leader is quiet); ship acks additionally carry the same frontier as an
instant floor hint. The leader folds both into per-peer durable marks; the
**commit index** is the k-th largest mark (k = `floor(n/2)`), and
`ack=quorum` responses gate on it passing the write's seqno — awaited
asynchronously, so quorum waiters never hold threads or starve completions.
The index is leadership-scoped: promote resets it to the new stream
baseline, and a demoted leader fails its in-flight quorum waiters (it must
never claim quorum for a stream it no longer owns).

Every cluster write's success response carries **`x-tidal-seq`** — the
write's seqno in the replicated log (relayed through gateway forwards).
Persist it if you need an exact durability cursor: `commit_index >= seq` on
`/cluster/status/local` is "this write is majority-durable", regardless of
which mode acked it. (The rare dedup-suppressed signal write — an identical
event within the WAL's ~60s content-hash window is already durably logged,
so no new record exists to track — carries **`x-tidal-deduplicated: 1`**
instead, relayed through forwards like the seq header.)

**Retry semantics under `ack=quorum` — read this twice.** A quorum-timeout
503 means *not confirmed in budget*, not *not written*: the write is in the
leader's log and usually commits moments later. Retries are therefore
**at-least-once**:

- `/items` and `/embeddings` retries are **always safe** — idempotent
  upserts keyed by `entity_id`.
- `/signals` retries can **double-count** the signal's weight when the
  original did commit (the server stamps each request's timestamp, so the
  WAL's content-hash dedup window cannot identify a client retry). The
  distortion is one extra decaying signal per retried timeout — bounded by
  your retry rate (`tidaldb_cluster_quorum_timeouts_total` is exactly that
  budget). Accounting that cannot tolerate it should route through session
  writes (which carry idempotency keys) or dedup client-side on its own key.
- The laggard names in the 503 are your runbook pointer: a persistent
  laggard is a down/partitioned region — heal it ([§6](#6-cluster-management-api))
  or accept leader-ack for the duration (`x-tidal-ack: leader`).

**Rolling upgrades into m11p3 — upgrade the leader first.** A pre-m11p3
leader neither serves the `ReportApplied` RPC nor recognizes `x-tidal-ack`:
it silently applies **leader-ack semantics to a `quorum` request** — a
durability downgrade the caller cannot see. Upgrade order:

1. Promote leadership off the leader node if needed, upgrade it, promote it
   back (or simply upgrade the standing leader per [§9](#9-failover-multi-process)'s
   restart procedure). From this moment `ack=quorum` is honored: the commit
   index rides the m11p2 ship-ack floor hints from not-yet-upgraded
   followers (correct, just laggier).
2. Upgrade followers one at a time. Each upgraded follower starts pushing
   `ReportApplied` and quorum freshness returns to batch-level. (An upgraded
   follower reporting to a still-old leader is harmless — the report is
   refused and logged, replication and heal are unaffected.)

Until step 1 completes, treat the cluster as `ack=leader`-only — do not
point `ack=quorum` traffic at it expecting majority durability.

Other facts unchanged from m11p1/p2: the success code never waits on
shipping for `ack=leader` (sender threads push the WAL flush feed's tail off
the request path; `replication.batch_max_events`/`window` tune it); a failed
fsync errors that write and nothing unfsynced can ship; long-outage data is
follower-pulled via `StreamSegments` (or, when it has rotated past the leader's
retained WAL, via the m11p5 `FetchSnapshot` snapshot transfer); a leader crash
is an **automatic failover** since m11p4 (the survivors elect the
up-to-date successor — see [§9.1](#91-automatic-failover-m11p4--the-default);
the vote restriction only elects a node whose log covers every quorum-acked
write, which is what makes the zero-acked-loss guarantee hold).

In short: **`ack=leader` = leader durability. `ack=quorum` = failover-survivable
durability, priced at one pipelined replication round trip and majority
availability.**

**WAL segment format across upgrades (m11p4).** Segment files carry an
8-byte version header (`TSEG` + version byte; headerless pre-m11p4 files
stay readable — no migration). Three behaviors follow:

- **Unreadable segments fail the boot, loudly.** A node whose WAL dir holds
  segments written by an incompatible tidalDB version (or stray/foreign
  `.seg` files) refuses to start with `WAL segment format unknown: <path>`
  instead of booting with the data invisibly absent (`segments=0` — the
  2026-06-11 p3 rollout failure mode). Remedy: run a compatible binary, or
  reseed the node (delete its PVC and let it pull from the leader).
- **Unservable catch-up resolves via snapshot transfer (m11p5).** A leader
  that cannot serve a follower's requested range from its retained WAL answers
  the `StreamSegments` pull with `FAILED_PRECONDITION` — `"segments not
  available from seq N; snapshot required"`, carrying the typed trailer
  `x-tidal-catchup: snapshot-required`. Since m11p5 the follower **latches a
  durable `reseed_required` marker** (only on that typed trailer — an ordinary
  election term-mismatch never latches it fleet-wide) and reseeds itself via the
  `FetchSnapshot` snapshot stream on its next boot (or self-restarts if
  `replication.reseed_self_restart` is set) — no operator verb, no
  `wipe_data_dir`. See [§9.1](#91-automatic-failover-m11p4--the-default). The
  marker is surfaced in `/cluster/status/local` (`reseed_required: true`) and the
  `tidaldb_cluster_reseed_required` gauge.
- **Failed pulls self-heal on a timer.** A catch-up pull that fails (e.g.
  the leader's gRPC server not yet ready during a rolling restart) retries
  every `replication.catchup_retry_ms` (default 30000) without waiting for
  a write to re-expose the gap — an idle cluster no longer strands lagged
  followers.

**Downgrade hazard (m11p4):** a pre-m11p4 binary reading a header-bearing
segment treats the header as a torn tail and may truncate the final segment.
Downgrading across the m11p4 boundary requires reseeding the node's WAL.

**Membership records + capability gate (m11p5).** A **kind-4 membership
record** is a new WAL blob kind. A kind-4 record shipped to a **pre-p5
follower** is an unknown batch kind → `WalError::Corruption` → that follower's
torn-state receiver halt, **permanent across restarts** (boot self-heal
re-pulls the same record). Both followers halted = a quorum-write outage. To
make this structurally impossible, `HeartbeatResponse`/`ReportApplied` carry a
`capabilities` bit-field (proto3 zero-default = pre-p5 = incapable) and the
leader **refuses `/cluster/join` and every conf-change until all current voters
report kind-4 capability** — so the first conf-change cannot fire mid-upgrade.
**Downgrade rule (kind-3 precedent verbatim): once any kind-4 record is in a
node's WAL, downgrading it below p5 requires a reseed.** Complete the binary
upgrade across all voters before adding, removing, or replacing a node.

## 9. Failover (multi-process)

### 9.1 Automatic failover (m11p4 — the default)

**"A machine died" is a non-event.** Every node runs a failure detector
(leader heartbeats every `election.heartbeat_interval_ms`, default 300) and a
Raft-style election (pre-vote + vote, randomized
`election.election_timeout_{min,max}_ms`, default 1500–3000). Kill the
leader and the survivors elect a successor — typically in **under one
second** with the defaults, bounded well inside 10s — with **zero operator
verbs** and zero acknowledged-write loss (the vote restriction only elects a
node whose log covers every quorum-acked write; the tier-3
`cluster_election.rs::mp_auto_failover_writes_resume_zero_acked_loss` gate
proves it across repeated random kill points under `ack=quorum` load).

What the operator sees:

- `/cluster/status/local` carries `term` (the election term, 0 = the
  pre-election "topology era"), `role` (`leader`/`follower`/`pre-candidate`/
  `candidate`) and `quarantined`.
- During the brief leaderless window, writes return a retryable 503 naming
  the election (`leader: "none (election in progress)"` plus the responding
  node's `term`); clients retry and land on the new leader.
- A **restarted ex-leader can never re-claim leadership from its topology
  file**: its durable election state (`data_dir/election_state`) boots it as
  a follower, and every replication RPC is term-fenced — a deposed leader's
  ships, heartbeats and frontier reports are rejected until it rejoins the
  current term (the §1.4-1 split-brain incident is closed by construction;
  proven by `mp_fenced_ex_leader_restart_cannot_write`).
- A leader that loses contact with a majority **steps down within
  `election.leader_lease_ms`** (default 900) and stops accepting writes:
  `ack=leader` writes during a minority partition are bounded by the lease,
  and `ack=quorum` writes were never at risk.

**Divergent suffix / quarantine → automated reseed (m11p5).** A node that
held leader-acked (never quorum-acked) writes when it died can rejoin into a
cluster that elected past them. It detects this at term-join and
**quarantines**: it serves status (`quarantined: true`, metric
`tidaldb_cluster_divergence_quarantined`) and keeps voting, but refuses the
data plane. Since m11p5 recovery is **automatic and full-history** — no
`wipe_data_dir`: the quarantine latches the durable `reseed_required` marker,
the node reseeds via the `FetchSnapshot` snapshot stream on its next boot (or
self-restarts if `replication.reseed_self_restart` is set and quorum can be
sustained without it), rejoins clean, and the divergence gauge clears. This is
strictly leader-ack-only data, within the documented `ack=leader` crash
contract (§8). The tier-3 `cluster_reseed.rs` proves the full quarantine →
marker → restart → reseeded → gauges-cleared loop.

**The three-way term-join rule (m11p5).** The divergent-suffix check above is
one of three outcomes a node reaches when it joins a new term, comparing its own
stream position against the leader's election-time position (`prev_log`, carried
on the heartbeat):

- `own > prev_log` → **divergent suffix** → quarantine (above);
- `own < prev_log` → the node is genuinely **missing committed-era history**
  that the new stream's baseline jump would silently skip (the p4 carried
  hazard) → it latches `reseed_required` and reseeds via snapshot (no silent
  gap);
- `own == prev_log` (or a within-term rejoin) → **clean**, catch-up via the
  stream.

A snapshot-installed node always joins clean by construction (its WAL is the
leader's copy, so its tail term equals the leader's).

To run the pre-m11p4 posture (operator-driven failover, no automatic
elections, no check-quorum step-down), set in the topology:

```yaml
election:
  auto_election: false
```

### 9.2 Manual promote: a FENCED TRANSFER (maintenance / override)

`POST /cluster/promote { "region": "eu-west" }` remains the maintenance verb
— but it is now a **fenced leadership transfer**, not a view flip:

1. With a live leader: the leader waits for the target to hold the full
   flushed prefix (the catch-up wait IS the drain), then sanctions an
   immediate election (`TimeoutNow`); the target wins term+1 and the old
   leader steps down on first higher-term contact. Response:
   `{ ok, leader, term, transfer: "elected" }`.
2. With a dead leader: the target campaigns among the survivors directly —
   the manual override of the automatic path (and the way to *choose* the
   successor).
3. The election can **refuse a target that lags** (its log loses the
   up-to-date comparison — e.g. the survivor you sampled fell behind between
   your status read and the vote). The verb 503s naming the cause; promote
   the other survivor. **You can no longer accidentally promote a node that
   would discard acked data** — the m11p3 "max-applied survivor" operator
   rule is now enforced by the protocol.
4. Promoting the node that already leads is a no-op 200.

The legacy term-0 fan-out promote survives only for clusters that have never
elected (mixed-version rollouts mid-upgrade, and the deliberate
isolated-node override used by the chaos drills); the first joined election
permanently retires it on each node.

> Rolling upgrade m11p3 → m11p4: upgrade ALL binaries before relying on
> auto-failover (pre-m11p4 peers answer vote RPCs with `Unimplemented`, so
> no election can reach quorum until a majority is upgraded — the cluster
> simply keeps its m11p3 behavior until then). The first ELECTED leader
> journals a kind-3 term-marker WAL record; pre-m11p4 binaries cannot decode
> it, so do not downgrade a node after the first election without reseeding.

## 10. Partition drill (multi-process)

The rewritten drill demonstrates **both** partition mechanisms, exactly as
`cluster_runbook.rs::runbook_s10_partition_drill` scripts them.

1. **Baseline.** `GET /cluster/status`; all `lag_events: 0`, `partitioned: false`,
   `reachable: true`.
2. **Inject.** Use **one** of:
   - **Real network partition** — sever the TCP path peers use to reach the region.
     The chaos suite (`cluster_chaos.rs`) uses a root-free in-harness TCP relay
     proxy (the ROADMAP-sanctioned toxiproxy-style alternative). On a real host you
     can instead use `iptables` (Linux) or `pfctl` (macOS), e.g.
     `iptables -A INPUT -p tcp --dport 9603 -j DROP` to blackhole ap-south's gRPC
     port from a peer. A real cut shows `reachable: false` in the aggregate status.
   - **Simulated ship-skip flag** — `POST /cluster/partition { "region": "ap-south" }`
     (`→ { ok, partitioned: "ap-south" }`). The leader stops shipping to ap-south
     without touching sockets; the aggregate status shows `partitioned: true`.
3. **Write through it.** Send several `POST /signals`. Each must still `204` (the
   leader-durable contract holds). Watch ap-south's `lag_events` climb while its
   `applied_events` stalls — the leader's ships to it are dropped.
4. **Read the stale follower.** `GET /feed?region=ap-south` (or read ap-south's
   gateway directly) returns the pre-partition view — eventual, not strong, read
   consistency. The operator console survives a real partition because clients talk
   to each region's gateway directly.
5. **Scatter-gather degradation.** While partitioned, `GET /sharded/feed` returns
   **200** with `degraded: true` and `ap-south` in `unavailable_shards` — never an
   error, and the live shards' items are still returned.
6. **Heal.** Clear the cut (heal the proxy / drop the firewall rule, or just call
   heal for the simulated flag), then `POST /cluster/heal { "region": "ap-south" }`.
   **Re-issue heal until `/cluster/status` shows ap-south at `lag_events: 0`** — the
   gRPC circuit breaker opened during the partition (threshold 5, reset 30s), so the
   first heal may ship into an open breaker and no-op. This is genuine production
   behavior, not a flag.
7. **Verify convergence.** `GET /cluster/status` shows ap-south `lag_events: 0`,
   `partitioned: false`, `reachable: true`; `/sharded/feed` is no longer degraded;
   feed scores on ap-south match the leader to within float tolerance (no loss, no
   duplication).

## 11. Shutdown

Send `SIGTERM` (or `SIGINT` / Ctrl+C, or stop the container). The server flips
readiness to **503** (so a load balancer stops routing to it), **drains** in-flight
requests, then drops the region's `TidalDb` shutdown path: checkpoint in-memory
signal state → flush storage → write the WAL checkpoint marker + **fsync** → join
the WAL, sweeper, checkpoint, text-syncer, and replication-receiver threads. The
drop is idempotent; the process exits 0. You will see
`region cluster node shutdown: database closed (checkpoint + WAL fsync)`
(multi-process) / `cluster shutdown: closing all nodes (checkpoint + WAL fsync)`
(single-process).

On **restart with the same `--data-dir`**, the region recovers its pre-shutdown
state from the WAL (verified by `cluster_runbook.rs::runbook_s11_shutdown_and_wal_recovery`:
SIGTERM → exit 0 → restart → the same items are served). In multi-process mode, the
restarted node rejoins the cluster and pulls anything it missed while down via
its boot-time `StreamSegments` catch-up request — no operator verb needed
(m11p2; `/cluster/heal` remains the explicit lever for paused peers). This is the per-node step of a
**rolling upgrade** (promote leadership off the node, SIGTERM, restart on the same
data dir with the new binary, heal) — see `cluster_lifecycle.rs::mp_rolling_upgrade_no_loss_no_stall`,
which proves zero acknowledged-write loss across a full rolling upgrade under load.

### Node-replace drill (m11p5)

Replacing a dead or recycled node is `kubectl delete pod` — no topology edit, no
operator verb:

- **PVC retained** (the data dir survives the pod): the replacement boots on the
  same data dir and converges via its boot-time `StreamSegments` catch-up — the
  ordinary restart path above. If it rotated past the leader's retained WAL while
  down, it reseeds via `FetchSnapshot` automatically (the `reseed_required`
  marker + snapshot path, [§9.1](#91-automatic-failover-m11p4--the-default)).
- **PVC deleted + pod deleted** (fresh node): the replacement comes up empty and
  **seed-joins** ([§1b](#1b-multi-process-one-process-per-region)) — it `--seed`s
  a survivor, `FetchSnapshot`s the current state, and the leader auto-promotes it
  back to Voter. Under the `k8s/cluster/` StatefulSet this is the default: the
  pod's args carry `--seed` against the headless Service, so a recreated pod
  rejoins with no human in the loop.

**Add a node:** scale the StatefulSet up (pod N≥3 seed-joins as a learner and
auto-promotes). **Remove a node:** `POST /cluster/members/remove` **first**
([§6](#6-cluster-management-api)), wait for the record to quorum-commit, then
scale down (lowest ordinal last).

## 12. Security (m11p7): mTLS, rotation, identity, audit, rate limits

The cluster does not trust the network. Everything here is **opt-in** — absent
the `grpc_tls` block and the cluster key, the cluster behaves exactly as pre-m11p7
(plaintext, hint-only marker, no audit/limit). A reference (k8s) deployment turns
it all on.

### 12.1 mTLS (gRPC replication) — the default posture

- Configure the `grpc_tls` block per region (`ca_cert`, `server_cert`,
  `server_key`, `client_cert`, `client_key`). The gRPC server then REQUIRES a
  client cert chained to the cluster CA (mutual TLS): a foreign pod with no cert,
  a cert from another CA, or a plaintext probe fails the TLS handshake and never
  reaches an RPC.
- **No `grpc_tls` ⇒ plaintext, with a loud startup WARN** on both the server and
  the client. Acceptable only on a trusted single-host / loopback topology. To
  serve plaintext intentionally there is nothing else to set — the WARN is the
  signal that you are on the insecure path.
- The inter-node HTTP plane (forwards, broadcasts, scatter, status, seed-join)
  is served over TLS with the SAME cert and dials `https://` with the cluster CA
  whenever `grpc_tls` is set, so enabling it gives **zero plaintext inter-node
  links** on both planes at once.

### 12.2 Cert + bearer rotation WITHOUT restart

- A background poller (`TIDAL_ROTATION_POLL_MS`, default 30000) content-hashes the
  cert files and the credential files; on a change it **atomically swaps the
  served cert** (in-flight TLS sessions keep their negotiated keys — zero dropped
  requests) and rebuilds the outbound peer channels.
- **Procedure:** issue a new cert under the same CA (cert-manager renewal, or
  re-run `scripts/gen-cluster-certs.sh` and re-apply the Secret) — the files
  change in place, the poller swaps within one interval, no pod restart. Verify
  with `tidaldb_cluster_*` logs (`TLS material rotated…`) or the `tidal_audit`
  /tracing stream.
- The bearer (`TIDAL_API_KEY_FILE`) and the cluster key (`TIDAL_CLUSTER_KEY_FILE`)
  rotate the same way. Use FILE mounts (not inline env) so a Secret rotation is
  picked up live. During a CA roll, keep both old and new CAs trusted for one
  cycle (CA-overlap) so in-flight connections complete.

### 12.3 Per-node identity + the marker

- Set a shared **cluster key** (`TIDAL_CLUSTER_KEY` / `TIDAL_CLUSTER_KEY_FILE`,
  any random string — BLAKE3-derived to the MAC key). Each node then mints a
  signed `x-tidal-node-token` on every forward/broadcast; the receiver verifies
  it. This gives inter-node calls a verifiable node identity and is the
  defense-in-depth layer beyond the shared bearer.
- With a cluster key configured, the `x-tidal-internal` marker is honored ONLY
  from a verified sibling: a request that sets the marker without a valid node
  token is rejected **403** (the marker is a routing hint, never an auth bypass).
  Never hand the cluster key to external clients.

### 12.4 Admin audit log

- promote / partition / heal / join / member-remove / reseed each emit one
  structured record: `{principal, verb, target, term, outcome}`. The principal is
  the verified node (`node:<id>`) for inter-node calls or `external` for an
  operator with the bearer.
- Sinks: a `tidal_audit` **tracing target** (always — capture it in your log
  pipeline), plus an **append-only JSONL file** when `TIDAL_AUDIT_LOG=<path>` is
  set. Recorded on the operator-originated leg only (no double-record on a
  forwarded re-apply).
- **At-rest encryption** of the JSONL file is delegated to the volume — mount
  `TIDAL_AUDIT_LOG` on an encrypted PV (or a `gVisor`/LUKS-backed volume); the
  server does not encrypt it in-engine.

### 12.5 Per-principal rate limits

- `TIDAL_RATE_LIMIT_RPS` (+ optional `TIDAL_RATE_LIMIT_BURST`, default 2×) caps
  per-principal request rate; a deny is **429 + `Retry-After`**. Off by default.
- Verified sibling nodes are EXEMPT — replication/forward traffic is never
  throttled by the external-client budget. Today external callers share ONE
  bucket: `TIDAL_RATE_LIMIT_RPS` is an AGGREGATE cap across all external clients,
  not a per-client budget — set it to your total external ceiling, not a
  per-client one (one noisy client can consume it). A future multi-key registry
  adds per-key buckets.

### 12.6 Foreign-pod / negative behavior (what an attacker on the network sees)

| Attempt | Result |
|---------|--------|
| Ship a gRPC segment without a cluster client cert | TLS handshake fails — no RPC dispatched |
| Call an internal HTTP route without trusting the cluster CA | TLS handshake fails — no route reached |
| Set `x-tidal-internal` without a valid node token (key configured) | 403 — marker honored only from a verified sibling |
| Call a protected route without the bearer | 401 (unchanged) |

## 13. Coordinated backup / restore + point-in-time recovery (m11p8)

The building blocks: the engine's crash-consistent `create_backup`, the WAL
**archive** (`wal.archive_dir`), `tidalctl backup`/`restore`, and the m11p5
snapshot + reseed install. Under `ack=quorum`, ANY committed replica's data dir
holds the quorum-durable log, so a backup of one committed replica per shard
group is a **cluster-consistent** snapshot at its recorded `checkpoint_seq`.

### 13.1 Enable the WAL archive (point-in-time recovery)

Set `wal.archive_dir` in the topology (or `TidalDb::builder().wal_archive_dir(path)`
for the embedded engine). Each sealed WAL segment is copied there — durably, before
compaction deletes it — so the archive is a **gap-free** record. Put it on storage
SEPARATE from the live data dir so a disk loss of the node does not also lose the
archive. Segment filenames encode `shard + first_seq`, so co-located groups share
one archive dir without collision.

```yaml
wal:
  archive_dir: "/archive/tidaldb"   # durable, off-node storage
```

### 13.2 Coordinated backup drill

1. Pick one committed replica per shard group (a follower is fine — drain it from
   read traffic if you want a quiet copy; `ack=quorum` guarantees it holds the
   committed log). Stop it (or snapshot its volume).
2. `tidalctl backup --path /data/<node> --out /backups/<cluster>-<ts>/shard-<id>`
   — writes a recursive copy + `BACKUP_MANIFEST.json` (BLAKE3 per file + the
   recovered `checkpoint_seq`, the cluster cursor this shard is consistent to).
3. Repeat per shard group. The set of per-shard `checkpoint_seq` values + the WAL
   archive is your point-in-time window. Restart the replica; self-heal/catch-up
   reconverges it.

### 13.3 Restore drill (timed)

1. `tidalctl restore --from /backups/<cluster>-<ts>/shard-<id> --path /data/<new-node>`
   — verifies EVERY file's BLAKE3 against the manifest BEFORE writing, and refuses
   a non-empty target (it never overwrites a live data dir).
2. Point a stopped node at the restored dir and boot it. Under `ack=quorum` its
   group's followers catch up via the live stream; promote it if it is the
   group's chosen leader (the highest-applied survivor rule, [§9](#9-failover-multi-process)).

**Recovery granularity today is full-snapshot-to-frontier, NOT arbitrary
point-in-time.** A restore recovers a node to the backup's `checkpoint_seq`, then
the live stream reconverges it to the cluster's CURRENT frontier. The WAL archive
(`wal.archive_dir`) is the durable, gap-free **primitive** that *backs* a future
point-in-time replay — but it is write-only today: no tool replays archived
segments up to a chosen target seq. `tidalctl recover` is verify-only
(`--verify-only`; an in-place replay mode is reserved for a future release), and
`tidalctl restore` copies a full snapshot with no seq bound. So the inputs you
retain for a future PITR are the per-shard `checkpoint_seq` set + the archive;
treat them as the **window**, not a one-command restore-to-instant. Do not plan an
incident around seq-bounded replay until a `tidalctl replay --until <seq>` verb
ships.

Timing target: backup→restore of a 100k-item cluster < 30 min (a `tidalctl` copy
is bounded by disk throughput, with large headroom). The Ref-A timed figure is a
k3s line item (the standing M11 access caveat).

## 14. Rolling upgrade + version skew (m11p8)

Nodes carry a build version on the wire (`HeartbeatRequest.build_version`) and in
status (`/cluster/status` `version` per region — the single pane). Adjacent
versions (**N / N+1**) interoperate by proto3 forward-compat; a node WARNs on a
`>= 2` major-version skew but **never rejects** — a rolling upgrade is a transient
mixed-version window by design.

**Procedure (one node at a time):**

1. `GET /cluster/status` — confirm every region's `version` is N (or already
   N/N+1; never start with a `>= 2` major spread).
2. Graceful SIGTERM one follower → it drains (readiness 503 → checkpoint).
3. Restart it on N+1 (same ports, same data dir). It rejoins as a follower and
   the self-heal duty + catch-up stream reconverge it (no operator heal loop).
4. Repeat for each follower. Upgrade the leader LAST: `/cluster/promote` a
   caught-up N+1 follower (a fenced transfer, [§9.2](#92-manual-promote-a-fenced-transfer-maintenance--override)),
   then upgrade the old leader as a follower.
5. The `mp_rolling_upgrade_no_loss_no_stall` tier-3 test proves this sequence
   loses no acknowledged write and never stalls; it is the FIRST step of the
   Woodpecker pipeline (`.woodpecker.yaml`) — a failure blocks the image build.

> Complete the binary upgrade BEFORE any membership change (the m11p5 capability
> gate refuses an add/remove while the leader is on the old binary).

## Performance (measured over real localhost processes)

| Operation | SLA | Measured (p99 / typical) |
|-----------|-----|--------------------------|
| Replicated `/signals` throughput (m11p1, 3 nodes, release build) | ≥ 2,000/s | **4,534 signal-writes/s** within SLO on the ramp (knee ~5.5k/s); **2,739/s sustained 10 min** (1.65M writes, 0.35% errors); was ~90/s pre-m11p1 |
| Replication lag under that load (m11p1) | < 2s | ≤ 103 events (~40ms) across the 10-min sustain; ≤ 377 events on the 5k/s ramp (follower group-commit coalescing) |
| Cross-region replication (write → follower applied) | < 2s | ~110–133ms p99 (m8p10) |
| Failover (`/cluster/promote` → first successful write) | < 10s | ~31–34ms |
| CRDT reconcile (merge+apply, each side) | < 100ms | 0–1ms |

Write-latency note: signal p50 ≈ 17–25ms with a p99 tail of 160–220ms on
macOS, where `F_FULLFSYNC` averages ~7.4ms with a 10–50ms tail
(`tidaldb_cluster_wal_fsync_us` — 0% complete under 1ms). On Linux
`fdatasync` volumes the same pipeline's fsync floor is far lower; validate
the p99 gate on the reference environment, and tune `wal.batch_timeout_ms`
against the measured fsync histogram.

## Cross-references

- **Kubernetes deployment** — [docs/runbooks/kubernetes.md](kubernetes.md)
  (the hardened standalone single-replica set in [`k8s/`](../../k8s/), and the
  multi-region cluster reference in [`k8s/cluster/`](../../k8s/cluster/) — one
  StatefulSet + headless Service peer discovery + PDB, with `--seed`-based scale
  and `kubectl delete pod` node-replace, shipped in m11p5).
- **Server deployment guide** — [docs/guides/server-deployment.md](../guides/server-deployment.md)
  (standalone and cluster launch, config, env, health probes).
- **Monitoring & alerts** — [docs/ops/monitoring.md](../ops/monitoring.md)
  (Prometheus scrape config, replication-lag and WAL metrics, recommended alerts;
  per-region replication lag is observable via `/cluster/status` `lag_events`;
  remember the `/metrics` endpoint is unauthenticated — bind it internally).
- **Roadmap / cluster status & known gaps** —
  [docs/planning/ROADMAP.md](../planning/ROADMAP.md) and
  [docs/roadmap-to-cluster.md](../roadmap-to-cluster.md) for the M11 cluster
  status — quorum-ack writes (m11p3), automatic failover (m11p4), and
  membership/discovery/elasticity (m11p5) all shipped; sharding × replication
  (p6), security hardening (p7), and continuous correctness (p9) remain.
- **API & schema reference** — [API.md](../../API.md),
  [QUICKSTART.md](../../QUICKSTART.md), and the live `/openapi.json` document.
- **Scope & vision** — [VISION.md](../../VISION.md).