tidaldb/docs/runbooks/cluster.md

# tidalDB Cluster Runbook

Operating the multi-region `tidal-server cluster` surface: launch, the
operational API, replication transport facts, failover and partition drills,
and the honest write-durability contract.

> ## STATUS: EXPERIMENTAL — SINGLE-PROCESS, NOT PRODUCTION HA
>
> Cluster mode runs **every region inside one `tidal-server` process**. The
> regions replicate over the **real `tidal-net` gRPC transport** (`ShipSegment`
> RPC, serialization, circuit breaker, HTTP/2 — see [§4](#4-grpc-replication-transport-tidal-net)),
> but that transport is a **loopback self-loop**: a follower's gRPC server and
> its only peer entry are the same loopback address. So you get faithful
> multi-region replication *semantics* over a real wire, with **no host or
> process isolation**.
>
> **A process crash, OOM, or host failure takes the entire "cluster" down at
> once.** There is no second machine to fail over to. This is a development /
> staging / demo fabric and a correctness harness for the replication paths —
> **it is NOT production high availability.**
>
> True multi-process, multi-host HA (process isolation, real network between
> regions, quorum-acked writes) is tracked as **m8p10** in
> [docs/planning/ROADMAP.md](../planning/ROADMAP.md). Until then, for a real
> production deployment run a **single `tidal-server standalone`** node and back
> it with host-level redundancy and disk durability.
>
> **Cluster mode refuses to start** unless you explicitly opt in with either the
> `--experimental-cluster` flag or the `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1`
> environment variable. On start it emits a loud `WARN` restating exactly the
> above. Do not wire it into a production load balancer.

## Scope: what cluster mode does and does not own

tidalDB owns **retrieval and ranking only**. Cluster mode adds multi-region
replication of that, nothing more. It does **not** generate embeddings, store
video/blobs, run a CDN or transcoder, do auth/moderation/payments, or provide a
distributed consensus log. Bring your vectors; tidalDB retrieves and ranks over
them. See [VISION.md](../../VISION.md) for the scope boundary.

## Prerequisites

- Rust toolchain ≥ 1.91 if running directly (the Docker build pins
  `rust:1.91-bookworm`).
- `protobuf-compiler` (`protoc`) and a C++ toolchain (`g++`) on the build host —
  `tidal-net`'s build script compiles the WAL-shipping `.proto`, and USearch's
  HNSW core is C++. The Docker image installs both.
- Docker 25+ if running via container.
- Port `9500` available (default cluster HTTP listener). Each follower region
  also binds an **OS-assigned loopback port** for its gRPC replication server
  unless you pin one in the topology (see [§3](#3-topology-yaml)).

## 1. Launch the cluster locally

Cluster mode is gated. Pass `--experimental-cluster` (or set
`TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1`) or the server exits with an error
explaining why.

```bash
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
cargo run -p tidal-server -- \
  cluster \
  --listen 127.0.0.1:9500 \
  --schema tidal-server/config/default-schema.yaml \
  --topology tidal-server/config/default-cluster.yaml \
  --experimental-cluster
```

The default topology spins up three regions (`us-east`, `eu-west`, `ap-south`)
with `us-east` as leader. On a clean start you will see a `WARN` line stating
this is experimental and single-process, an `info` line per follower
(`follower gRPC transport ready`), and finally `listening on http://127.0.0.1:9500`.

**Auth:** set `TIDAL_API_KEY=<secret>` to require `Authorization: Bearer <secret>`
on the data and `/cluster/*` mutation routes. If it is **unset the server runs
UNAUTHENTICATED and logs a WARN** — never expose an unauthenticated cluster
beyond loopback. Health probes and `/openapi.json` are always unauthenticated.

Useful environment variables (shared with standalone):

| Var | Effect |
|-----|--------|
| `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1` | Opt in to cluster mode (alternative to `--experimental-cluster`). |
| `TIDAL_API_KEY` | Bearer token for protected routes. Unset ⇒ unauthenticated + WARN. |
| `TIDAL_CONFIG` | Config dir holding `default-schema.yaml` / `default-cluster.yaml` (used when `--schema` / `--topology` omitted). |
| `PORT` | Listen address. A bare port (`9500`) normalises to `0.0.0.0:9500`. |
| `TIDAL_SERVER_LOG` | `tracing` filter (default `info`). |

## 2. Launch via Docker

```bash
# Build the image once (build context is the repo root).
docker build -f docker/cluster/Dockerfile -t tidaldb:cluster .

# Run (press Ctrl+C to stop). The image already sets
# TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 so cluster mode starts; it still logs the
# loud experimental WARN.
docker run --rm -p 9500:9500 tidaldb:cluster
```

The image bakes the default schema/topology under `/etc/tidal-server` and uses
`ENTRYPOINT ["tidal-server"]` + a `cluster …` `CMD`, so you can override the
subcommand (e.g. `docker run tidaldb:cluster standalone …`) or supply your own
config files:

```bash
docker run --rm -p 9500:9500 \
  -e TIDAL_API_KEY=changeme \
  -v "$PWD/configs/my-schema.yaml:/srv/schema.yaml:ro" \
  -v "$PWD/configs/my-topology.yaml:/srv/topology.yaml:ro" \
  tidaldb:cluster \
  cluster \
    --listen 0.0.0.0:9500 \
    --schema /srv/schema.yaml \
    --topology /srv/topology.yaml
```

The container ships a `HEALTHCHECK` that curls `/health`. For Kubernetes/Compose
deployment patterns see [Cross-references](#cross-references).

## 3. Topology YAML

The topology file declares regions and the leader. Minimal default
(`tidal-server/config/default-cluster.yaml`):

```yaml
regions:
  - name: us-east
  - name: eu-west
  - name: ap-south
leader: us-east
# Optional: OS worker threads serving cluster write/heal requests (gRPC ship).
# Defaults to available parallelism when omitted.
# write_workers: 4
```

Fields:

| Field | Required | Meaning |
|-------|----------|---------|
| `regions[].name` | yes | Region name used everywhere in the HTTP API (`?region=`, `/cluster/promote`, etc.). Must be unique. |
| `regions[].grpc_addr` | no | This region's gRPC replication bind address, e.g. `"127.0.0.1:9601"`. **Omit it** for the single-process default — the server allocates a free loopback port and self-heals a bind race by retrying on a fresh port. An explicit address is only needed for the future multi-process deployment (m8p10) and is tried exactly once. |
| `leader` | yes | Must name one of the declared regions. Writes route here. |
| `write_workers` | no | Size of the runtime-free OS-thread pool that runs the blocking gRPC segment-ship on `/signals` and `/cluster/heal`. Bounds write concurrency on the hottest path. |

The schema YAML (signals / text_fields / embedding_slots / profiles) is the same
format the standalone server loads — see [API.md](../../API.md) and
[QUICKSTART.md](../../QUICKSTART.md) for the full schema/profile grammar and the
25 built-in ranking profiles. Topology only adds the region layout above.

> **Personalization correctness note.** Preference-vector personalization (the
> taste vector that powers `for_you`) only updates for signals declared
> `positive_engagement: true` in the schema. The cluster `/signals` route writes
> **global** signals (no `user_id` / `creator_id` context), so it never folds
> item embeddings into a user's taste vector regardless of the
> `positive_engagement` flag. Personalized writes go through the embedded engine
> (`signal_with_context`), not this HTTP route.

## 4. gRPC replication transport (tidal-net)

Replication between regions is the real `tidal-net` `WalShipping` gRPC service,
not in-process channels. The facts you need to operate and firewall it:

| Property | Value | Notes |
|----------|-------|-------|
| Service | `WalShipping` (proto `tidal.replication.v1`) | RPCs: **`ShipSegment`** (unary; the production replication path) and **`Heartbeat`** (ControlPlane health). `StreamSegments` is declared but **not implemented** — segment delivery is the unary `ShipSegment`. |
| Default port | `59529` | In tidalDB's reserved dev band `59520–59529`. In single-process cluster mode each follower binds an **auto-allocated** loopback port instead (or the topology's `grpc_addr`). |
| Transport security | mTLS via `TlsConfig` (`ca_cert`, `server_cert`, `server_key`, optional `client_cert` / `client_key`) | `insecure = true` (plaintext) is the single-process loopback default. A peer with no TLS config **must** set `insecure`. |
| Max payload | **64 MiB** | Both encoder and decoder codec limits are raised to this; pinned by a compile-time assert to the engine's `InProcessTransport` limit so both ends agree. WAL segments default to 16 MiB. |
| Circuit breaker | per-peer, **threshold 5**, **reset 30s** | After 5 consecutive ship failures the breaker opens; it stays open for 30s, then the next attempt probes (HalfOpen → Closed on success, Open again on failure). Backpressure (channel full) does **not** trip it. |
| Timeouts (defaults) | connect 5s, request 10s, keep-alive PING every 10s / 5s ACK | A blackholed peer fails fast instead of stalling the single-threaded shipper. |

`ShipSegment` carries the WAL segment id, BLAKE3-validated payload bytes, event
count, and the leader's authoritative `leader_last_seq` (so a follower can
advance its replication-lag high-water-mark even for an all-local segment that
filters to empty). The follower's segment-receiver thread drains and applies it
on arrival.

> **Operational guard:** the gRPC replication ports and the Prometheus `/metrics`
> endpoint are **UNAUTHENTICATED**. Bind them to loopback or a cluster-internal
> network only — never the public interface.

## 5. Core HTTP API

All routes are JSON unless noted. Examples assume `BASE=http://localhost:9500`
and, when `TIDAL_API_KEY` is set, `AUTH='-H "Authorization: Bearer $TIDAL_API_KEY"'`.
Drop the `-H` header when running unauthenticated.

Middleware mirrors standalone: 30s request timeout (408), 100 max in-flight
(429), 2 MB body limit (413), and an `x-request-id` on every response. Health
probes sit **outside** the load-shedding stack so liveness/readiness are never
queued or timed out under saturation.

### Health

```bash
curl "$BASE/health"          # 200 ok / 503 while draining; reports leader + item count
curl "$BASE/health/startup"  # always 200
curl "$BASE/health/live"     # always 200
curl "$BASE/openapi.json"    # served OpenAPI 3.1, UNAUTHENTICATED — canonical HTTP reference
```

The cluster `/openapi.json` is a superset that documents the `/cluster/*` and
`/sharded/*` routes. It is the machine-readable source of truth for request /
response shapes.

### Register items & embeddings (leader-routed)

```bash
curl -X POST "$BASE/items" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "metadata": { "title": "Jazz Piano", "category": "music" } }'
# → 201 Created

curl -X POST "$BASE/embeddings" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "values": [0.1, 0.2, 0.3, 0.4] }'
# → 204 No Content
```

Embeddings: tidalDB does **not** generate vectors — the caller brings them. The
write L2-normalizes and inserts into the HNSW index. Dimensions are **strict**:
they must equal the slot's declared dimensions (min 2, max 4096) or the insert
fails, and **zero-norm vectors are rejected**. `RETRIEVE` / `SEARCH` route
through the **first declared embedding slot only**; multi-modal apps must fuse
offline or use separate entity kinds.

### Record signals (cluster `/signals` = global only)

```bash
curl -X POST "$BASE/signals" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "signal": "view", "weight": 1.0 }'
# → 204 No Content (see the durability contract in §8)
```

The signal name must be declared in the schema. This route records a **global**
signal on the leader and ships it to followers; it does not personalize (see the
personalization note in [§3](#3-topology-yaml)).

### Retrieve and search (region-pinned reads)

```bash
# Default read region is the leader.
curl "$BASE/feed?user_id=42&profile=for_you&limit=20"
curl "$BASE/search?query=jazz%20piano&user_id=42&limit=5"

# Pin a read to a specific region. Followers may lag the leader (and lag jumps
# during a partition) — use this for canary reads and lag verification.
curl "$BASE/feed?profile=trending&region=eu-west"
```

`?region=` accepts any declared region name; an unknown name returns 400. Omit
it and the read serves from the current leader. `limit` is clamped to 1000 at the
trust boundary. The `for_you` profile applies the built-in diversity defaults
(`max_per_creator=2`, `format_mix_max_fraction=0.4`, `exploration=0.1`).

## 6. Cluster management API

### Check cluster status

```bash
curl "$BASE/cluster/status" | jq
```

```json
{
  "leader": "us-east",
  "relay_log_len": 125,
  "regions": [
    { "name": "us-east", "applied_events": 125, "lag_events": 0, "partitioned": false },
    { "name": "eu-west", "applied_events": 125, "lag_events": 0, "partitioned": false },
    { "name": "ap-south", "applied_events": 124, "lag_events": 1, "partitioned": false }
  ]
}
```

`lag_events` is `relay_log_len − applied_events` (saturating). A non-zero,
*growing* lag on a region is the signal that it is partitioned or its
segment-receiver is wedged.

### Promote a new leader

```bash
curl -X POST "$BASE/cluster/promote" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "eu-west" }'
# → { "ok": true, "leader": "eu-west" }
```

After promotion `/cluster/status` reports the new leader; new writes route there
and replay to the other regions. An unknown region returns 400.

### Simulate a partition & heal

```bash
# Isolate ap-south: leader ships skip this follower, so its lag climbs.
curl -X POST "$BASE/cluster/partition" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "ap-south" }'
# → { "ok": true, "partitioned": "ap-south" }

# Heal: missed segments are re-shipped over gRPC (blocking, offloaded to the
# write pool — a saturated pool degrades to 429, not 500).
curl -X POST "$BASE/cluster/heal" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "ap-south" }'
# → { "ok": true, "healed": "ap-south" }
```

## 7. Sharded scatter-gather API

The `/sharded/*` routes hash-partition entities across regions
(`hash(entity_id) % num_shards`, using the engine's own `ShardRouter` so writes
and reads never disagree) and fan a query out to **all** shards, K-way merging
the per-shard results by score.

Write routes (each writes to the single owning shard):

```bash
curl -X POST "$BASE/sharded/items"      -d '{ "entity_id": 7, "metadata": { "title": "..." } }'  # 201
curl -X POST "$BASE/sharded/embeddings" -d '{ "entity_id": 7, "values": [0.1,0.2,0.3,0.4] }'      # 204
curl -X POST "$BASE/sharded/signals"    -d '{ "entity_id": 7, "signal": "view", "weight": 1.0 }'  # 204
```

Read routes (scatter-gather across all shards):

```bash
curl "$BASE/sharded/feed?profile=for_you&limit=20&deadline_ms=50"
curl "$BASE/sharded/search?query=jazz&limit=10&deadline_ms=100"
```

Each sharded read accepts an optional `deadline_ms` — the **total** scatter
budget (default 50ms, server-clamped to 10s). The per-shard deadline is
`deadline_ms − 5ms` network overhead. The response includes a `scatter_gather`
block:

```json
{
  "items": [ /* merged, deduped, diversity-enforced, re-ranked */ ],
  "total_candidates": 4210,
  "scatter_gather": {
    "degraded": false,
    "shards_queried": 3,
    "elapsed_ms": 12,
    "shard_deadline_ms": 45
  }
}
```

Degraded semantics: a shard that is partitioned, errors, or misses the deadline
is reported in `unavailable_shards` (a name list) and flips `degraded: true` —
it is **never silently dropped**. The merge dedups replicated copies of an entity
(keeping the best-scoring copy), reconciles `total_candidates` so replicated
shards are not counted multiple times, and re-enforces `max_per_creator` across
the merged set.

## 8. Write-durability contract (read this before trusting a 204)

A `204 No Content` from `/signals` (and the data writes) means **the write is
durably applied on the leader** — storage updated plus **WAL fsync** — and
nothing more.

- The follower ship is **best-effort**. A ship that fails is logged engine-side
  and queued for re-delivery via the heal / convergence path. The 204 does
  **NOT** assert quorum and does **NOT** assert any follower acknowledged the
  write.
- If the single process crashes after the 204 but before followers caught up,
  the leader's WAL still has the write (it survives restart from disk); the
  followers will reconcile on the next ship/heal. But because there is **only
  one process**, a crash is total downtime, not a failover — see the status box.
- A quorum-ack write contract is explicitly future work, tracked with the
  multi-process cluster effort (**m8p10**).

In short: **204 = leader durability, not cluster durability.** Design your
client retries accordingly (the writes are idempotent on `entity_id` + signal).

## 9. Failover drill

1. **Baseline.** `GET /cluster/status`; confirm the expected leader and
   `lag_events: 0` on every region.
2. **Pre-seed reads.** Issue a region-pinned read against the target region
   (`?region=eu-west`) to confirm it is serving and roughly caught up.
3. **Promote.** `POST /cluster/promote { "region": "eu-west" }`.
4. **Verify.** `GET /cluster/status` now reports `eu-west` as leader. Send a
   write (`POST /signals`) and confirm `relay_log_len` advances and the other
   regions' `applied_events` follow within a heartbeat.
5. **Cut over traffic.** Point your client's writes at the same `$BASE` (the
   single surface routes writes to whoever is leader) — no client change needed.

> Reminder: this is a *leadership move within one process*, not a host failover.
> It is the right drill for "move the write region during maintenance," not for
> "survive a machine dying."

## 10. Partition drill

1. **Baseline.** `GET /cluster/status`; all `lag_events: 0`, all
   `partitioned: false`.
2. **Inject.** `POST /cluster/partition { "region": "ap-south" }`. Confirm
   `partitioned: true` for that region.
3. **Write through it.** Send several `POST /signals`. Watch `ap-south`'s
   `lag_events` climb while `applied_events` stalls — the leader's ships to it
   are dropped.
4. **Read the stale follower.** `GET /feed?region=ap-south` should return the
   pre-partition view (demonstrating eventual, not strong, read consistency).
5. **Heal.** `POST /cluster/heal { "region": "ap-south" }`. Missed segments
   re-ship over gRPC.
6. **Verify convergence.** Poll `GET /cluster/status` until `ap-south`'s
   `lag_events` returns to 0 and `partitioned: false`.
7. **Scatter-gather degradation.** While partitioned, a `GET /sharded/feed`
   should report `degraded: true` with `ap-south` in `unavailable_shards` — not
   an error. Confirm it clears after heal.

## 11. Shutdown

Send `SIGTERM` (or `SIGINT` / Ctrl+C, or stop the container). The server flips
readiness to 503, drains in-flight requests, then drops the cluster — which runs
each region's `TidalDb` shutdown path: checkpoint in-memory signal state → flush
storage → write the WAL checkpoint marker + **fsync** → join the WAL, sweeper,
checkpoint, and text-syncer threads. The drop is idempotent. You will see
`cluster shutdown: closing all nodes (checkpoint + WAL fsync)`.

## Cross-references

- **Kubernetes deployment** — `docs/runbooks/kubernetes.md` *(planned for the
  m8p10 multi-process work; not yet written — until then deploy a single
  standalone node per the production guidance in the status box, and use the
  Docker `HEALTHCHECK` / `/health` probe as the readiness gate).*
- **Server deployment guide** — `docs/guides/server-deployment.md` *(planned;
  not yet written — see [API.md](../../API.md) and [QUICKSTART.md](../../QUICKSTART.md)
  for the current server config and route reference).*
- **Monitoring & alerts** — [docs/ops/monitoring.md](../ops/monitoring.md)
  (Prometheus scrape config, replication-lag and WAL metrics, recommended
  alerts; remember the `/metrics` endpoint is unauthenticated — bind it
  internally).
- **Roadmap / m8p10 (true multi-process HA)** —
  [docs/planning/ROADMAP.md](../planning/ROADMAP.md) for the distributed-fabric
  status, the m8p1–m8p9 completed phases, and the outstanding multi-process /
  tier-3 chaos work.
- **API & schema reference** — [API.md](../../API.md),
  [QUICKSTART.md](../../QUICKSTART.md), and the live `/openapi.json` document.
- **Scope & vision** — [VISION.md](../../VISION.md).
```