tidaldb/docs/runbooks/cluster.md

# tidalDB Cluster Runbook

Operating the multi-region `tidal-server cluster` surface: launch (single-process
dev fabric **and** the multi-process region nodes), the operational API,
replication transport facts, failover and partition drills, and the honest
write-durability contract.

> ## STATUS: EXPERIMENTAL — TWO MODES, NEITHER IS QUORUM-ACKED HA YET
>
> Cluster mode has **two shapes**, both behind the same experimental opt-in:
>
> **1. Multi-process (`--region`) — real process isolation.** Each
> `tidal-server cluster --region <name>` process owns **exactly one region**: one
> `TidalDb`, one [`GrpcTransport`](#4-grpc-replication-transport-tidal-net) whose
> server binds *this* region's `grpc_addr` and whose peers are every **sibling
> region's real `grpc_addr`**. Processes peer over real gRPC and forward over real
> HTTP, so a crash of one region's process takes down **only that region** — the
> survivors keep serving. This is genuine process (and, across hosts, host)
> isolation. It is verified end-to-end by the tier-3 suites
> (`cluster_multiproc`, `cluster_chaos`, `cluster_lifecycle`, `cluster_runbook`)
> over real OS processes with real network-partition injection.
>
> **2. Single-process (no `--region`) — the dev/demo default.** Every region runs
> inside **one** process. Replication still traverses the **real `tidal-net` gRPC
> transport** on loopback (faithful multi-region semantics over a real wire), but
> there is **no process isolation**: a crash, OOM, or host failure takes the whole
> "cluster" down at once. This is a development / staging / demo fabric and a
> correctness harness for the replication paths — **not** production HA. It remains
> the default because it needs no per-region topology addresses and no process
> orchestration.
>
> **Honest remaining limits (both modes):**
> * **Writes are leader-durable, NOT quorum-acked.** A `204` means the leader
>   durably applied the write (storage + WAL fsync). The follower ship is
>   best-effort; no follower acknowledgement is asserted. A quorum-ack write
>   contract is **post-M8 follow-up** (see [§8](#8-write-durability-contract-read-this-before-trusting-a-204)).
> * **Leadership is operator-driven, not automatic.** There is **no automatic
>   failure detector and no automatic leader election.** `/cluster/promote` moves
>   leadership and fans the new view out to peers; a node that misses the fan-out
>   self-corrects on its next forwarded write / status poll. "Survive a machine
>   dying" is an operator runbook step (detect → promote), not an automatic
>   failover.
>
> **For a production deployment today**, run a **single `tidal-server standalone`**
> node backed by host-level redundancy and disk durability (see
> [kubernetes.md](kubernetes.md) and [server-deployment.md](../guides/server-deployment.md)),
> and reach for multi-process cluster mode for read-scale / multi-region
> experiments — not as a quorum-HA story.
>
> **Both modes refuse to start** unless you explicitly opt in with either the
> `--experimental-cluster` flag or the `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1`
> environment variable. On start each emits a loud, **mode-specific** `WARN`
> restating exactly what it does and does not provide. Do not wire either into a
> production load balancer.

## Scope: what cluster mode does and does not own

tidalDB owns **retrieval and ranking only**. Cluster mode adds multi-region
replication of that, nothing more. It does **not** generate embeddings, store
video/blobs, run a CDN or transcoder, do auth/moderation/payments, or provide a
distributed consensus log. Bring your vectors; tidalDB retrieves and ranks over
them. See [VISION.md](../../VISION.md) for the scope boundary.

## Prerequisites

- Rust toolchain ≥ 1.91 if running directly (the Docker build pins
  `rust:1.91-bookworm`).
- `protobuf-compiler` (`protoc`) and a C++ toolchain (`g++`) on the build host —
  `tidal-net`'s build script compiles the WAL-shipping `.proto`, and USearch's
  HNSW core is C++. The Docker image installs both.
- Docker 25+ if running via container.
- For **single-process** mode: one HTTP port (default `9500`); each follower
  region also binds an **OS-assigned loopback port** for its gRPC replication
  server unless you pin one in the topology (see [§3](#3-topology-yaml)).
- For **multi-process** mode: per-region `grpc_addr` **and** `http_addr` declared
  in the topology, and the ports they name available on each host
  (see [§3](#3-topology-yaml)).

## 1. Launch the cluster locally

Cluster mode is gated. Pass `--experimental-cluster` (or set
`TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1`) or the server exits with a mode-specific
error explaining why.

### 1a. Single-process (dev/demo default)

```bash
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
cargo run -p tidal-server -- \
  cluster \
  --listen 127.0.0.1:9500 \
  --schema tidal-server/config/default-schema.yaml \
  --topology tidal-server/config/default-cluster.yaml \
  --experimental-cluster
```

The default topology spins up three regions (`us-east`, `eu-west`, `ap-south`)
with `us-east` as leader, all inside one process. On a clean start you will see a
`WARN` line stating this is experimental and single-process, an `info` line per
follower (`follower gRPC transport ready`), and finally
`listening on http://127.0.0.1:9500`.

### 1b. Multi-process (one process per region)

Pass `--region <name>` (or set `TIDAL_REGION`) to run **only that region** in this
process. The topology must declare a per-region `grpc_addr` **and** `http_addr` for
every region (see [§3](#3-topology-yaml)); siblings reach each other over those.
Launch one process per region — typically one per host, each with its own
`--data-dir`:

```bash
# Region us-east (the initial leader) on host A
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
tidal-server cluster --experimental-cluster \
  --region us-east \
  --listen 0.0.0.0:9501 \
  --schema /etc/tidal/schema.yaml \
  --topology /etc/tidal/topology.yaml \
  --data-dir /var/lib/tidal/us-east

# Region eu-west on host B
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
tidal-server cluster --experimental-cluster \
  --region eu-west --listen 0.0.0.0:9502 \
  --schema /etc/tidal/schema.yaml --topology /etc/tidal/topology.yaml \
  --data-dir /var/lib/tidal/eu-west

# Region ap-south on host C
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
tidal-server cluster --experimental-cluster \
  --region ap-south --listen 0.0.0.0:9503 \
  --schema /etc/tidal/schema.yaml --topology /etc/tidal/topology.yaml \
  --data-dir /var/lib/tidal/ap-south
```

Every process parses the **same** topology file; `RegionId`s are assigned by
declaration order, so all processes agree on the region→id mapping. Each process
binds its own region's `grpc_addr`/`http_addr` and dials its siblings' addresses.
The `--listen` HTTP address is the public gateway for that region.

**Auth:** set `TIDAL_API_KEY=<secret>` to require `Authorization: Bearer <secret>`
on the data and `/cluster/*` mutation routes. If it is **unset the server runs
UNAUTHENTICATED and logs a WARN** — never expose an unauthenticated cluster
beyond loopback / a trusted VPC. Health probes and `/openapi.json` are always
unauthenticated. In multi-process mode set the **same** key on every process: a
forwarded/broadcast request passes the caller's `Authorization` through verbatim,
and the internal-propagation marker (`x-tidal-internal: 1`) is an inter-sibling
trust signal, **not** an auth bypass (the bearer middleware still runs first).

Useful environment variables:

| Var | Mode | Effect |
|-----|------|--------|
| `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1` | both | Opt in to cluster mode (alternative to `--experimental-cluster`). |
| `TIDAL_REGION` | multi-process | Selects the single region this process owns (alternative to `--region`). |
| `TIDAL_API_KEY` | both | Bearer token for protected routes. Unset ⇒ unauthenticated + WARN. Set the SAME key on every region in multi-process mode. |
| `TIDAL_HLC_SKEW_MS` | multi-process | Signed ms offset applied to THIS process's HLC. Affects reconcile-time LWW stamping ONLY (not signal-decay timestamps). A test/ops escape hatch for verifying causal convergence under clock skew — do not set it in normal operation. |
| `TIDAL_CONFIG` | both | Config dir holding `default-schema.yaml` / `default-cluster.yaml` (used when `--schema` / `--topology` omitted). |
| `PORT` | both | Listen address. A bare port (`9500`) normalises to `0.0.0.0:9500`. |
| `TIDAL_SERVER_LOG` | both | `tracing` filter (default `info`). |

## 2. Launch via Docker

```bash
# Build the image once (build context is the repo root).
docker build -f docker/cluster/Dockerfile -t tidaldb:cluster .

# Run (press Ctrl+C to stop). The image already sets
# TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 so cluster mode starts; it still logs the
# loud experimental WARN. The default CMD is SINGLE-process cluster mode.
docker run --rm -p 9500:9500 tidaldb:cluster
```

The image bakes the default schema/topology under `/etc/tidal-server` and uses
`ENTRYPOINT ["tidal-server"]` + a `cluster …` `CMD`, so you can override the
subcommand (e.g. `docker run tidaldb:cluster standalone …`), pass `--region` for a
multi-process container, or supply your own config files:

```bash
docker run --rm -p 9500:9500 \
  -e TIDAL_API_KEY=changeme \
  -v "$PWD/configs/my-schema.yaml:/srv/schema.yaml:ro" \
  -v "$PWD/configs/my-topology.yaml:/srv/topology.yaml:ro" \
  tidaldb:cluster \
  cluster \
    --listen 0.0.0.0:9500 \
    --schema /srv/schema.yaml \
    --topology /srv/topology.yaml
```

The container ships a `HEALTHCHECK` that curls `/health`. For Kubernetes/Compose
deployment patterns see [Cross-references](#cross-references).

## 3. Topology YAML

The topology file declares regions and the leader.

**Single-process minimal default** (`tidal-server/config/default-cluster.yaml`):

```yaml
regions:
  - name: us-east
  - name: eu-west
  - name: ap-south
leader: us-east
# Optional: OS worker threads serving cluster write/heal requests (gRPC ship).
# Defaults to available parallelism when omitted.
# write_workers: 4
```

**Multi-process** requires a per-region `grpc_addr` **and** `http_addr` (validated
at startup — a missing or syntactically invalid `host:port` is a hard error naming
the region; reachability is deliberately not probed, since siblings boot in any
order):

```yaml
regions:
  - name: us-east
    grpc_addr: "10.0.1.10:9601"   # this region's gRPC replication bind / dial addr
    http_addr: "10.0.1.10:9501"   # this region's public HTTP gateway (forwarding + status)
  - name: eu-west
    grpc_addr: "10.0.2.10:9602"
    http_addr: "10.0.2.10:9502"
  - name: ap-south
    grpc_addr: "10.0.3.10:9603"
    http_addr: "10.0.3.10:9503"
leader: us-east
```

Fields:

| Field | Single-process | Multi-process | Meaning |
|-------|----------------|---------------|---------|
| `regions[].name` | required | required | Region name used everywhere in the HTTP API (`?region=`, `/cluster/promote`, etc.). Must be unique. |
| `regions[].grpc_addr` | optional | **required** | This region's gRPC replication address. In single-process mode **omit it** — the server allocates a free loopback port and self-heals a bind race by retrying on a fresh port. In multi-process mode it is required (siblings dial it) and is tried exactly once. |
| `regions[].http_addr` | unused | **required** | This region's public HTTP gateway address, used by siblings for write/read forwarding and status aggregation. Unused in single-process mode. |
| `regions[].grpc_tls` | unused | optional | TLS material for this region's gRPC transport: `ca_cert`, `server_cert`, `server_key` (PEM paths), plus optional `client_cert`/`client_key` for mTLS. Omitted ⇒ plaintext, the right posture for loopback/VPC topologies. |
| `leader` | required | required | Must name one of the declared regions. The initial write leader. |
| `write_workers` | optional | optional | Size of the runtime-free OS-thread pool that runs the blocking gRPC segment-ship on `/signals` and `/cluster/heal`. Bounds write concurrency on the hottest path. Must be ≥ 1 when given. |
| `timeouts.broadcast_peer_secs` | unused | optional | Per-peer budget (seconds) for the item/embedding leader broadcast and the promote fan-out. Default 2s — right for loopback/VPC; raise it for WAN topologies where a distant region cannot answer in 2s. Must be ≥ 1 when given. |

The schema YAML (signals / text_fields / embedding_slots / profiles) is the same
format the standalone server loads — see [API.md](../../API.md) and
[QUICKSTART.md](../../QUICKSTART.md) for the full schema/profile grammar and the
built-in ranking profiles. Topology only adds the region layout above.

> **Personalization correctness note.** Preference-vector personalization (the
> taste vector that powers `for_you`) only updates for signals declared
> `positive_engagement: true` in the schema. The cluster `/signals` route writes
> **global** signals (no `user_id` / `creator_id` context), so it never folds
> item embeddings into a user's taste vector regardless of the
> `positive_engagement` flag. Personalized writes go through the embedded engine
> (`signal_with_context`), not this HTTP route. The exception is `/hardnegs`,
> which records a **user-scoped** hide that converges across regions via
> [`/cluster/reconcile`](#6-cluster-management-api).

## 4. gRPC replication transport (tidal-net)

Replication between regions is the real `tidal-net` `WalShipping` gRPC service,
not in-process channels. The facts you need to operate and firewall it:

| Property | Value | Notes |
|----------|-------|-------|
| Service | `WalShipping` (proto `tidal.replication.v1`) | RPCs: **`ShipSegment`** (unary; the production replication path) and **`Heartbeat`** (ControlPlane health). `StreamSegments` is declared but **not implemented** — segment delivery is the unary `ShipSegment`. |
| Bind address | per-region `grpc_addr` (multi-process); an **auto-allocated** loopback port (single-process) | Default dev band `59520–59529` if you pin one. |
| Transport security | mTLS via `TlsConfig` (`ca_cert`, `server_cert`, `server_key`, optional `client_cert` / `client_key`) | `insecure = true` (plaintext) is the loopback/VPC default in this phase. A peer with no TLS config **must** set `insecure`. |
| Max payload | **64 MiB** | Both encoder and decoder codec limits are raised to this; pinned by a compile-time assert to the engine's `InProcessTransport` limit so both ends agree. WAL segments default to 16 MiB. |
| Circuit breaker | per-peer, **threshold 5**, **reset 30s** | After 5 consecutive ship failures the breaker opens; it stays open for 30s, then the next attempt probes (HalfOpen → Closed on success, Open again on failure). Backpressure (channel full) does **not** trip it. **Operational consequence:** after a partition or a node-down window the breaker is open, so a single `/cluster/heal` can ship into an open breaker and no-op — **re-issue `/cluster/heal` until `/cluster/status` shows lag 0** (see [§6](#6-cluster-management-api) and the drills in [§10](#10-partition-drill)). |
| Timeouts (defaults) | connect 5s, request 10s, keep-alive PING every 10s / 5s ACK | A blackholed peer fails fast instead of stalling the single-threaded shipper. |

`ShipSegment` carries the WAL segment id, BLAKE3-validated payload bytes, event
count, and the leader's authoritative `leader_last_seq` (so a follower can
advance its replication-lag high-water-mark even for an all-local segment that
filters to empty). The follower's segment-receiver thread drains and applies it
on arrival. The engine's applied high-water-mark is a **contiguous frontier with a
bounded ahead-buffer**, so out-of-order eager ships can never swallow a sequence
gap (a silent-data-loss class fixed in m8p10).

> **Operational guard:** the gRPC replication ports and the Prometheus `/metrics`
> endpoint are **UNAUTHENTICATED**. Bind them to loopback or a cluster-internal
> network only — never the public interface.

## 5. Core HTTP API

All routes are JSON unless noted. Examples assume `BASE=http://localhost:9501`
(any region's gateway in multi-process mode; the single `--listen` address in
single-process mode) and, when `TIDAL_API_KEY` is set,
`AUTH='-H "Authorization: Bearer $TIDAL_API_KEY"'`. Drop the `-H` header when
running unauthenticated.

Middleware mirrors standalone: 30s request timeout (408), 100 max in-flight
(429), 2 MB body limit (413), and an `x-request-id` on every response. Health
probes sit **outside** the load-shedding stack so liveness/readiness are never
queued or timed out under saturation.

> **Multi-process routing (one coherent surface).** Any region's gateway accepts
> any operation and routes it to the node that owns it:
> * **Writes** (`/signals`, `/items`, `/embeddings`, `/hardnegs`) on a non-leader
>   **forward to the leader** (the caller's `Authorization` passes through); the
>   client sees the leader's status/body. A leader that is unreachable degrades to
>   a `503` naming the leader, never a hang.
> * **Reads** (`/feed`, `/search`) default to the **LOCAL** region. This is the
>   key difference from single-process mode, whose default read is the leader. A
>   `?region=<other>` read **forwards** to that region's process.
> * `/cluster/promote`, `/cluster/partition`, `/cluster/heal` route to / fan out
>   from the leader as documented in [§6](#6-cluster-management-api).

### Health

```bash
curl "$BASE/health"          # 200 ok / 503 while draining; reports mode, region, leader
curl "$BASE/health/startup"  # always 200
curl "$BASE/health/live"     # always 200
curl "$BASE/openapi.json"    # served OpenAPI 3.1, UNAUTHENTICATED — canonical HTTP reference
```

`/health` returns `{ "ok": true, "service": "tidaldb", "mode": "cluster",
"region": "us-east", "leader": "us-east", ... }`. The `/openapi.json` document is
the machine-readable source of truth for the **data**, **`/cluster/*`**,
**`/sharded/*`**, and **`/hardnegs`** request/response shapes (the health probes
are intentionally outside the documented API surface).

### Register items & embeddings (leader-applied + broadcast)

Items and embeddings are **config-like data**: they are applied on the leader and
**HTTP-broadcast to every peer** (they do NOT ride the signal WAL relay — that is
the `/signals` stream). The leader's response carries a per-peer broadcast report.

```bash
curl -X POST "$BASE/items" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "metadata": { "title": "Jazz Piano", "category": "music" } }'
# → 201 Created
#   { "replicated_to": 2, "failed": [] }

curl -X POST "$BASE/embeddings" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "values": [0.1, 0.2, 0.3, 0.4] }'
# → 200 OK  (NOT 204 — a 204 cannot carry the report)
#   { "replicated_to": 2, "failed": [] }
```

`replicated_to` is the count of peers that acknowledged; `failed` names the peers
that did not (a partitioned/down peer lands here — it is backfilled on the next
[`/cluster/heal`](#6-cluster-management-api), which re-broadcasts item metadata +
embeddings to the healed region). The `201`/`200` still asserts the **local**
write succeeded. A write to a non-leader forwards to the leader; the
**forwarded/internal** path returns the bodyless `201`/`204` (only the leader's
external path carries the broadcast report).

Embeddings: tidalDB does **not** generate vectors — the caller brings them. The
write L2-normalizes and inserts into the HNSW index. Dimensions are **strict**:
they must equal the slot's declared dimensions (min 2, max 4096) or the insert is
**rejected with a 500** (the engine surfaces a dimension mismatch as an internal
error, not a 400 — a known wart, tracked post-M8); **zero-norm vectors are also
rejected (500)**. `RETRIEVE` / `SEARCH` route through the **first declared
embedding slot only**; multi-modal apps must fuse offline or use separate entity
kinds.

### Record signals (cluster `/signals` = global only)

```bash
curl -X POST "$BASE/signals" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "signal": "view", "weight": 1.0 }'
# → 204 No Content (see the durability contract in §8)
```

The signal name must be declared in the schema (an undeclared name returns **400**
naming it). This route records a **global** signal on the leader and ships it to
followers over the gRPC WAL relay; it does not personalize (see the
personalization note in [§3](#3-topology-yaml)). On a non-leader gateway it
forwards to the leader transparently and still returns `204`.

### Hard negatives (user-scoped hides)

```bash
curl -X POST "$BASE/hardnegs" \
  -H 'Content-Type: application/json' \
  -d '{ "user_id": 42, "item_id": 7 }'
# → 204 No Content
```

Records a user-scoped hide on the leader (a non-leader gateway forwards). The item
is filtered from that user's `/feed?user_id=42`. Hard negatives converge across
regions via the LWW-resolved [`/cluster/reconcile`](#6-cluster-management-api) CRDT
path — NOT the signal WAL relay and NOT a broadcast.

### Retrieve and search (region-pinned reads)

```bash
# Default read region is the LOCAL region (multi-process mode).
curl "$BASE/feed?user_id=42&profile=for_you&limit=20"
curl "$BASE/search?query=jazz%20piano&user_id=42&limit=5"

# Pin a read to a specific region. The gateway forwards to that region's process.
# Followers may lag the leader (and lag jumps during a partition) — use this for
# canary reads and lag verification.
curl "$BASE/feed?profile=trending&region=eu-west"
```

`?region=` accepts any declared region name; an unknown name returns **400**. Omit
it and the read serves from the **LOCAL** region (multi-process) / the current
leader (single-process). `limit` is clamped to 1000 at the trust boundary (a
larger value cannot amplify memory unboundedly). The `for_you` profile applies the
built-in diversity defaults (`max_per_creator=2`, `format_mix_max_fraction=0.4`,
`exploration=0.1`).

## 6. Cluster management API

### Check cluster status

Two views. `/cluster/status/local` reports THIS node's own replication/leadership
state (no peer calls); `/cluster/status` aggregates EVERY region (the gateway calls
each peer's `/cluster/status/local` concurrently with a tight per-peer budget).

```bash
curl "$BASE/cluster/status/local" | jq
```

```json
{
  "region": "ap-south",
  "is_leader": false,
  "leader": "us-east",
  "last_seq": 0,
  "applied_events": 124,
  "lag_events": 1,
  "partitioned": [],
  "reachable": true
}
```

```bash
curl "$BASE/cluster/status" | jq
```

```json
{
  "leader": "us-east",
  "relay_log_len": 125,
  "regions": [
    { "name": "us-east",  "applied_events": 125, "lag_events": 0, "partitioned": false, "reachable": true },
    { "name": "eu-west",  "applied_events": 125, "lag_events": 0, "partitioned": false, "reachable": true },
    { "name": "ap-south", "applied_events": 124, "lag_events": 1, "partitioned": false, "reachable": true }
  ]
}
```

`relay_log_len` is the leader's high-water-mark (`last_seq`); each region's
`lag_events` is `relay_log_len − applied_events` (saturating). A region the
gateway **cannot reach** within the per-peer budget is reported honestly as
`reachable: false`, `partitioned: true`, `applied_events: 0`, and worst-case lag
(`lag_events == relay_log_len`). A non-zero, *growing* lag on a reachable region is
the signal that it is partitioned (ship-skip) or its segment-receiver is wedged.

### Promote a new leader

```bash
curl -X POST "$BASE/cluster/promote" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "eu-west" }'
# → { "ok": true, "leader": "eu-west", "acked": ["eu-west","ap-south"], "failed": [] }
```

The gateway applies the new leadership view locally, then fans the promote out to
every peer (with the internal marker, so each applies it locally and does not
re-fan). `acked`/`failed` report the fan-out; a peer in `failed` (e.g. a dead old
leader) self-corrects on its next forwarded write or status poll, so a partial
fan-out is **not** an error. After promotion `/cluster/status` reports the new
leader; new writes route there. An unknown region returns **400**. There is no
automatic election — this is the operator's failover lever.

### Simulate a partition & heal

There are **two** ways to partition a region; the runbook drills demonstrate both
([§10](#10-partition-drill)):

1. **Simulated ship-skip flag** — `/cluster/partition` tells the leader to stop
   shipping to the named region (no sockets touched). Good for a controlled,
   reversible lag demo.
2. **Real network partition** — sever the actual TCP path between processes
   (firewall / proxy). The chaos suite uses a root-free in-harness TCP relay;
   operators can use `iptables`/`pfctl` (see [§10](#10-partition-drill)).

```bash
# Simulated: isolate ap-south — leader ships skip this follower, so its lag climbs.
curl -X POST "$BASE/cluster/partition" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "ap-south" }'
# → { "ok": true, "partitioned": "ap-south" }

# Heal: redeliver missed signal segments AND re-broadcast item metadata/embeddings
# to the healed region (the single recovery verb). Blocking work is offloaded to
# the write pool — a saturated pool degrades to 429, not 500.
curl -X POST "$BASE/cluster/heal" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "ap-south" }'
# → { "ok": true, "healed": "ap-south" }
```

`/cluster/heal` is the **single recovery verb**. It (a) redelivers the signal WAL
segments the follower missed — gap-aware, so re-shipping is idempotent (the
receiver's monotonic frontier drops already-applied batches) — and (b)
re-broadcasts every item's metadata and embedding to the healed region (idempotent
upserts; per-item failures are WARN-logged and retried on the next heal). Item
metadata is HTTP-broadcast, not WAL-relayed, so a node that was down during a
broadcast would otherwise be missing items forever even at lag 0 — heal closes
that gap.

> **Re-issue heal until lag is 0.** After a partition/down window the leader's
> per-peer gRPC circuit breaker is **open** (threshold 5, reset 30s — see
> [§4](#4-grpc-replication-transport-tidal-net)). A single `/cluster/heal` issued
> while the breaker is open ships into it and **no-ops**. Re-issue
> `POST /cluster/heal` (or let a new write fire the eager-ship probe) until
> `GET /cluster/status` shows the region at `lag_events: 0`. The chaos suite's
> `heal_until_converged` does exactly this.

### Reconcile (cross-region CRDT convergence)

```bash
curl -X POST "$BASE/cluster/reconcile" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "ap-south" }'
# → { "ok": true, "region": "ap-south",
#     "local_elapsed_ms": 0, "remote_elapsed_ms": 1, "ops_applied": 3 }
```

`/cluster/reconcile` exchanges a CRDT state snapshot with the target region: this
node ships its snapshot into the target's merge AND applies the target's pre-merge
snapshot back, so **both sides converge to identical state** by deterministic LWW.
This is how **hard negatives** (recorded via `/hardnegs`) converge across regions —
they do not ride the WAL relay. `local_elapsed_ms` / `remote_elapsed_ms` are the
merge+apply times each side measured (not the HTTP round-trip); both are typically
0–1ms. Reconcile is **idempotent**: a repeat reconcile of already-converged regions
is an exact no-op on scores (no drift). `/cluster/reconcile/snapshot` is the
INTERNAL snapshot-exchange leg this verb drives (marker required); operators never
call it directly.

## 7. Sharded scatter-gather API

The `/sharded/*` routes hash-partition entities across regions
(`hash(entity_id) % num_shards`, using the engine's own `ShardRouter` so writes
and reads never disagree). Writes route to the single owning region; reads fan out
to **all** regions and K-way merge by score.

Write routes (each routes to the owning region; a non-owner gateway forwards):

```bash
curl -X POST "$BASE/sharded/items"      -d '{ "entity_id": 7, "metadata": { "title": "..." } }'  # 201
curl -X POST "$BASE/sharded/embeddings" -d '{ "entity_id": 7, "values": [0.1,0.2,0.3,0.4] }'      # 204
curl -X POST "$BASE/sharded/signals"    -d '{ "entity_id": 7, "signal": "view", "weight": 1.0 }'  # 204
```

Read routes (scatter-gather across all regions):

```bash
curl "$BASE/sharded/feed?profile=for_you&limit=20&deadline_ms=50"
curl "$BASE/sharded/search?query=jazz&limit=10&deadline_ms=100"
```

Each sharded read accepts an optional `deadline_ms` — the **total** scatter budget
(default 50ms, server-clamped to 10s). The per-shard deadline is `deadline_ms − 5ms`
network overhead. The response includes a `scatter_gather` block:

```json
{
  "items": [ /* merged, deduped, diversity-enforced, re-ranked */ ],
  "total_candidates": 4210,
  "scatter_gather": {
    "degraded": false,
    "shards_queried": 3,
    "elapsed_ms": 12,
    "shard_deadline_ms": 45
  }
}
```

Degraded semantics: a shard that is partitioned, errors, or misses the deadline is
reported in `unavailable_shards` (a name list) and flips `degraded: true` — it is
**never silently dropped**, and the read **still returns 200** with the live
shards' results. The merge dedups replicated copies of an entity (keeping the
best-scoring copy), reconciles `total_candidates` so replicated shards are not
counted multiple times, and re-enforces `max_per_creator` across the merged set.

## 8. Write-durability contract (read this before trusting a 204)

A `204 No Content` from `/signals` (and the data writes' success codes) means
**the write is durably applied on the leader** — storage updated plus **WAL
fsync** — and nothing more.

- The follower ship is **best-effort**. A ship that fails is logged engine-side
  and queued for re-delivery via the heal / convergence path. The 204 does **NOT**
  assert quorum and does **NOT** assert any follower acknowledged the write.
- In **multi-process** mode, if the leader's process crashes after the 204 but
  before followers caught up, the leader's WAL still has the write (it survives
  restart from disk); the followers reconcile on the next ship/heal. The other
  regions' processes keep serving — this is real process isolation, but it is
  **not** an automatic failover (an operator promotes a survivor — see
  [§9](#9-failover-drill)).
- In **single-process** mode there is only one process, so a crash is total
  downtime, not a failover.

A **quorum-ack write contract** (a 204 that asserts N followers acknowledged) is
explicitly **post-M8 follow-up** work, tracked in
[docs/planning/ROADMAP.md](../planning/ROADMAP.md) M8 Known Gaps. It is **not** part
of m8p10.

In short: **204 = leader durability, not cluster durability.** Design your client
retries accordingly (the writes are idempotent on `entity_id` + signal).

## 9. Failover drill (multi-process)

Move the write leader to another region. Scripted exactly as the runbook-verification
suite (`cluster_runbook.rs::runbook_s9_failover_drill`) executes it:

1. **Baseline.** `GET /cluster/status`; confirm the expected leader and
   `lag_events: 0` on every region.
2. **Pre-seed reads.** Issue a region-pinned read against the target region
   (`?region=eu-west`) to confirm it is serving and roughly caught up.
3. **Promote.** `POST /cluster/promote { "region": "eu-west" }`. Confirm the
   `{ ok, leader, acked, failed }` response. If the OLD leader is dead, expect it
   in `failed` — that is fine.
4. **Verify.** `GET /cluster/status` now reports `eu-west` as leader. Send a write
   (`POST /signals`) to the new leader and confirm `relay_log_len` advances and the
   other regions' `applied_events` follow within a heartbeat.
5. **Cut over traffic.** Point your client's writes at **any** region gateway — a
   write to a non-leader forwards to the new leader transparently (204), no client
   change needed.

**Crash failover** is the same drill triggered by a real outage: a region's process
dies (its gateway stops answering), you detect it (monitoring on `/health` /
`/cluster/status` `reachable`), and you `POST /cluster/promote` a survivor via
**another** survivor's gateway. The tier-3 `mp_uat_step2_leader_crash_failover_under_10s`
test SIGKILLs the leader and proves promote→first-successful-write < 10s with zero
data loss. There is **no automatic detector/election** — promotion is the operator
step.

> This is a *leadership move*, not a quorum hand-off. Use it for "move the write
> region during maintenance" and for "a region died — promote a survivor."

## 10. Partition drill (multi-process)

The rewritten drill demonstrates **both** partition mechanisms, exactly as
`cluster_runbook.rs::runbook_s10_partition_drill` scripts them.

1. **Baseline.** `GET /cluster/status`; all `lag_events: 0`, `partitioned: false`,
   `reachable: true`.
2. **Inject.** Use **one** of:
   - **Real network partition** — sever the TCP path peers use to reach the region.
     The chaos suite (`cluster_chaos.rs`) uses a root-free in-harness TCP relay
     proxy (the ROADMAP-sanctioned toxiproxy-style alternative). On a real host you
     can instead use `iptables` (Linux) or `pfctl` (macOS), e.g.
     `iptables -A INPUT -p tcp --dport 9603 -j DROP` to blackhole ap-south's gRPC
     port from a peer. A real cut shows `reachable: false` in the aggregate status.
   - **Simulated ship-skip flag** — `POST /cluster/partition { "region": "ap-south" }`
     (`→ { ok, partitioned: "ap-south" }`). The leader stops shipping to ap-south
     without touching sockets; the aggregate status shows `partitioned: true`.
3. **Write through it.** Send several `POST /signals`. Each must still `204` (the
   leader-durable contract holds). Watch ap-south's `lag_events` climb while its
   `applied_events` stalls — the leader's ships to it are dropped.
4. **Read the stale follower.** `GET /feed?region=ap-south` (or read ap-south's
   gateway directly) returns the pre-partition view — eventual, not strong, read
   consistency. The operator console survives a real partition because clients talk
   to each region's gateway directly.
5. **Scatter-gather degradation.** While partitioned, `GET /sharded/feed` returns
   **200** with `degraded: true` and `ap-south` in `unavailable_shards` — never an
   error, and the live shards' items are still returned.
6. **Heal.** Clear the cut (heal the proxy / drop the firewall rule, or just call
   heal for the simulated flag), then `POST /cluster/heal { "region": "ap-south" }`.
   **Re-issue heal until `/cluster/status` shows ap-south at `lag_events: 0`** — the
   gRPC circuit breaker opened during the partition (threshold 5, reset 30s), so the
   first heal may ship into an open breaker and no-op. This is genuine production
   behavior, not a flag.
7. **Verify convergence.** `GET /cluster/status` shows ap-south `lag_events: 0`,
   `partitioned: false`, `reachable: true`; `/sharded/feed` is no longer degraded;
   feed scores on ap-south match the leader to within float tolerance (no loss, no
   duplication).

## 11. Shutdown

Send `SIGTERM` (or `SIGINT` / Ctrl+C, or stop the container). The server flips
readiness to **503** (so a load balancer stops routing to it), **drains** in-flight
requests, then drops the region's `TidalDb` shutdown path: checkpoint in-memory
signal state → flush storage → write the WAL checkpoint marker + **fsync** → join
the WAL, sweeper, checkpoint, text-syncer, and replication-receiver threads. The
drop is idempotent; the process exits 0. You will see
`region cluster node shutdown: database closed (checkpoint + WAL fsync)`
(multi-process) / `cluster shutdown: closing all nodes (checkpoint + WAL fsync)`
(single-process).

On **restart with the same `--data-dir`**, the region recovers its pre-shutdown
state from the WAL (verified by `cluster_runbook.rs::runbook_s11_shutdown_and_wal_recovery`:
SIGTERM → exit 0 → restart → the same items are served). In multi-process mode, the
restarted node rejoins the cluster and the leader backfills anything it missed while
down via `/cluster/heal` (re-issue until lag 0). This is the per-node step of a
**rolling upgrade** (promote leadership off the node, SIGTERM, restart on the same
data dir with the new binary, heal) — see `cluster_lifecycle.rs::mp_rolling_upgrade_no_loss_no_stall`,
which proves zero acknowledged-write loss across a full rolling upgrade under load.

## Performance (measured over real localhost processes, m8p10)

| Operation | SLA | Measured (p99 / typical) |
|-----------|-----|--------------------------|
| Cross-region replication (write → follower applied) | < 2s | ~110–133ms p99 |
| Failover (`/cluster/promote` → first successful write) | < 10s | ~31–34ms |
| CRDT reconcile (merge+apply, each side) | < 100ms | 0–1ms |

## Cross-references

- **Kubernetes deployment** — [docs/runbooks/kubernetes.md](kubernetes.md)
  (standalone single-replica is the recommended production deployment until
  quorum-ack / auto-failover land; an experimental StatefulSet-per-region sketch
  for multi-process cluster mode is noted there).
- **Server deployment guide** — [docs/guides/server-deployment.md](../guides/server-deployment.md)
  (standalone and cluster launch, config, env, health probes).
- **Monitoring & alerts** — [docs/ops/monitoring.md](../ops/monitoring.md)
  (Prometheus scrape config, replication-lag and WAL metrics, recommended alerts;
  per-region replication lag is observable via `/cluster/status` `lag_events`;
  remember the `/metrics` endpoint is unauthenticated — bind it internally).
- **Roadmap / M8 status & known gaps** —
  [docs/planning/ROADMAP.md](../planning/ROADMAP.md) for the distributed-fabric
  status (M8 COMPLETE), the m8p1–m8p10 phase history, and the post-M8 follow-ups
  (quorum-ack writes, automatic failure detection / leader election).
- **API & schema reference** — [API.md](../../API.md),
  [QUICKSTART.md](../../QUICKSTART.md), and the live `/openapi.json` document.
- **Scope & vision** — [VISION.md](../../VISION.md).