tidaldb/docs/runbooks/cluster.md

tidalDB Cluster Runbook

Operating the multi-region tidal-server cluster surface: launch, the operational API, replication transport facts, failover and partition drills, and the honest write-durability contract.

STATUS: EXPERIMENTAL — SINGLE-PROCESS, NOT PRODUCTION HA

Cluster mode runs every region inside one tidal-server process. The regions replicate over the real tidal-net gRPC transport (ShipSegment RPC, serialization, circuit breaker, HTTP/2 — see §4), but that transport is a loopback self-loop: a follower's gRPC server and its only peer entry are the same loopback address. So you get faithful multi-region replication semantics over a real wire, with no host or process isolation.

A process crash, OOM, or host failure takes the entire "cluster" down at once. There is no second machine to fail over to. This is a development / staging / demo fabric and a correctness harness for the replication paths — it is NOT production high availability.

True multi-process, multi-host HA (process isolation, real network between regions, quorum-acked writes) is tracked as m8p10 in docs/planning/ROADMAP.md. Until then, for a real production deployment run a single tidal-server standalone node and back it with host-level redundancy and disk durability.

Cluster mode refuses to start unless you explicitly opt in with either the --experimental-cluster flag or the TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 environment variable. On start it emits a loud WARN restating exactly the above. Do not wire it into a production load balancer.

Scope: what cluster mode does and does not own

tidalDB owns retrieval and ranking only. Cluster mode adds multi-region replication of that, nothing more. It does not generate embeddings, store video/blobs, run a CDN or transcoder, do auth/moderation/payments, or provide a distributed consensus log. Bring your vectors; tidalDB retrieves and ranks over them. See VISION.md for the scope boundary.

Prerequisites

• Rust toolchain ≥ 1.91 if running directly (the Docker build pins rust:1.91-bookworm).
• protobuf-compiler (protoc) and a C++ toolchain (g++) on the build host — tidal-net's build script compiles the WAL-shipping .proto, and USearch's HNSW core is C++. The Docker image installs both.
• Docker 25+ if running via container.
• Port 9500 available (default cluster HTTP listener). Each follower region also binds an OS-assigned loopback port for its gRPC replication server unless you pin one in the topology (see §3).

1. Launch the cluster locally

Cluster mode is gated. Pass --experimental-cluster (or set TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1) or the server exits with an error explaining why.

TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
cargo run -p tidal-server -- \
  cluster \
  --listen 127.0.0.1:9500 \
  --schema tidal-server/config/default-schema.yaml \
  --topology tidal-server/config/default-cluster.yaml \
  --experimental-cluster

The default topology spins up three regions (us-east, eu-west, ap-south) with us-east as leader. On a clean start you will see a WARN line stating this is experimental and single-process, an info line per follower (follower gRPC transport ready), and finally listening on http://127.0.0.1:9500.

Auth: set TIDAL_API_KEY=<secret> to require Authorization: Bearer <secret> on the data and /cluster/* mutation routes. If it is unset the server runs UNAUTHENTICATED and logs a WARN — never expose an unauthenticated cluster beyond loopback. Health probes and /openapi.json are always unauthenticated.

Useful environment variables (shared with standalone):

Var	Effect
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1	Opt in to cluster mode (alternative to --experimental-cluster).
TIDAL_API_KEY	Bearer token for protected routes. Unset ⇒ unauthenticated + WARN.
TIDAL_CONFIG	Config dir holding default-schema.yaml / default-cluster.yaml (used when --schema / --topology omitted).
PORT	Listen address. A bare port (9500) normalises to 0.0.0.0:9500.
TIDAL_SERVER_LOG	tracing filter (default info).

2. Launch via Docker

# Build the image once (build context is the repo root).
docker build -f docker/cluster/Dockerfile -t tidaldb:cluster .

# Run (press Ctrl+C to stop). The image already sets
# TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 so cluster mode starts; it still logs the
# loud experimental WARN.
docker run --rm -p 9500:9500 tidaldb:cluster

The image bakes the default schema/topology under /etc/tidal-server and uses ENTRYPOINT ["tidal-server"] + a cluster … CMD, so you can override the subcommand (e.g. docker run tidaldb:cluster standalone …) or supply your own config files:

docker run --rm -p 9500:9500 \
  -e TIDAL_API_KEY=changeme \
  -v "$PWD/configs/my-schema.yaml:/srv/schema.yaml:ro" \
  -v "$PWD/configs/my-topology.yaml:/srv/topology.yaml:ro" \
  tidaldb:cluster \
  cluster \
    --listen 0.0.0.0:9500 \
    --schema /srv/schema.yaml \
    --topology /srv/topology.yaml

The container ships a HEALTHCHECK that curls /health. For Kubernetes/Compose deployment patterns see Cross-references.

3. Topology YAML

The topology file declares regions and the leader. Minimal default (tidal-server/config/default-cluster.yaml):

regions:
  - name: us-east
  - name: eu-west
  - name: ap-south
leader: us-east
# Optional: OS worker threads serving cluster write/heal requests (gRPC ship).
# Defaults to available parallelism when omitted.
# write_workers: 4

Fields:

Field	Required	Meaning
regions[].name	yes	Region name used everywhere in the HTTP API (?region=, /cluster/promote, etc.). Must be unique.
regions[].grpc_addr	no	This region's gRPC replication bind address, e.g. "127.0.0.1:9601". Omit it for the single-process default — the server allocates a free loopback port and self-heals a bind race by retrying on a fresh port. An explicit address is only needed for the future multi-process deployment (m8p10) and is tried exactly once.
leader	yes	Must name one of the declared regions. Writes route here.
write_workers	no	Size of the runtime-free OS-thread pool that runs the blocking gRPC segment-ship on /signals and /cluster/heal. Bounds write concurrency on the hottest path.

The schema YAML (signals / text_fields / embedding_slots / profiles) is the same format the standalone server loads — see API.md and QUICKSTART.md for the full schema/profile grammar and the 25 built-in ranking profiles. Topology only adds the region layout above.

Personalization correctness note. Preference-vector personalization (the taste vector that powers for_you) only updates for signals declared positive_engagement: true in the schema. The cluster /signals route writes global signals (no user_id / creator_id context), so it never folds item embeddings into a user's taste vector regardless of the positive_engagement flag. Personalized writes go through the embedded engine (signal_with_context), not this HTTP route.

4. gRPC replication transport (tidal-net)

Replication between regions is the real tidal-net WalShipping gRPC service, not in-process channels. The facts you need to operate and firewall it:

Property	Value	Notes
Service	WalShipping (proto tidal.replication.v1)	RPCs: ShipSegment (unary; the production replication path) and Heartbeat (ControlPlane health). StreamSegments is declared but not implemented — segment delivery is the unary ShipSegment.
Default port	59529	In tidalDB's reserved dev band 59520–59529. In single-process cluster mode each follower binds an auto-allocated loopback port instead (or the topology's grpc_addr).
Transport security	mTLS via TlsConfig (ca_cert, server_cert, server_key, optional client_cert / client_key)	insecure = true (plaintext) is the single-process loopback default. A peer with no TLS config must set insecure.
Max payload	64 MiB	Both encoder and decoder codec limits are raised to this; pinned by a compile-time assert to the engine's InProcessTransport limit so both ends agree. WAL segments default to 16 MiB.
Circuit breaker	per-peer, threshold 5, reset 30s	After 5 consecutive ship failures the breaker opens; it stays open for 30s, then the next attempt probes (HalfOpen → Closed on success, Open again on failure). Backpressure (channel full) does not trip it.
Timeouts (defaults)	connect 5s, request 10s, keep-alive PING every 10s / 5s ACK	A blackholed peer fails fast instead of stalling the single-threaded shipper.

ShipSegment carries the WAL segment id, BLAKE3-validated payload bytes, event count, and the leader's authoritative leader_last_seq (so a follower can advance its replication-lag high-water-mark even for an all-local segment that filters to empty). The follower's segment-receiver thread drains and applies it on arrival.

Operational guard: the gRPC replication ports and the Prometheus /metrics endpoint are UNAUTHENTICATED. Bind them to loopback or a cluster-internal network only — never the public interface.

5. Core HTTP API

All routes are JSON unless noted. Examples assume BASE=http://localhost:9500 and, when TIDAL_API_KEY is set, AUTH='-H "Authorization: Bearer $TIDAL_API_KEY"'. Drop the -H header when running unauthenticated.

Middleware mirrors standalone: 30s request timeout (408), 100 max in-flight (429), 2 MB body limit (413), and an x-request-id on every response. Health probes sit outside the load-shedding stack so liveness/readiness are never queued or timed out under saturation.

Health

curl "$BASE/health"          # 200 ok / 503 while draining; reports leader + item count
curl "$BASE/health/startup"  # always 200
curl "$BASE/health/live"     # always 200
curl "$BASE/openapi.json"    # served OpenAPI 3.1, UNAUTHENTICATED — canonical HTTP reference

The cluster /openapi.json is a superset that documents the /cluster/* and /sharded/* routes. It is the machine-readable source of truth for request / response shapes.

Register items & embeddings (leader-routed)

curl -X POST "$BASE/items" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "metadata": { "title": "Jazz Piano", "category": "music" } }'
# → 201 Created

curl -X POST "$BASE/embeddings" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "values": [0.1, 0.2, 0.3, 0.4] }'
# → 204 No Content

Embeddings: tidalDB does not generate vectors — the caller brings them. The write L2-normalizes and inserts into the HNSW index. Dimensions are strict: they must equal the slot's declared dimensions (min 2, max 4096) or the insert fails, and zero-norm vectors are rejected. RETRIEVE / SEARCH route through the first declared embedding slot only; multi-modal apps must fuse offline or use separate entity kinds.

Record signals (cluster /signals = global only)

curl -X POST "$BASE/signals" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "signal": "view", "weight": 1.0 }'
# → 204 No Content (see the durability contract in §8)

The signal name must be declared in the schema. This route records a global signal on the leader and ships it to followers; it does not personalize (see the personalization note in §3).

Retrieve and search (region-pinned reads)

# Default read region is the leader.
curl "$BASE/feed?user_id=42&profile=for_you&limit=20"
curl "$BASE/search?query=jazz%20piano&user_id=42&limit=5"

# Pin a read to a specific region. Followers may lag the leader (and lag jumps
# during a partition) — use this for canary reads and lag verification.
curl "$BASE/feed?profile=trending&region=eu-west"

?region= accepts any declared region name; an unknown name returns 400. Omit it and the read serves from the current leader. limit is clamped to 1000 at the trust boundary. The for_you profile applies the built-in diversity defaults (max_per_creator=2, format_mix_max_fraction=0.4, exploration=0.1).

6. Cluster management API

Check cluster status

curl "$BASE/cluster/status" | jq

{
  "leader": "us-east",
  "relay_log_len": 125,
  "regions": [
    { "name": "us-east", "applied_events": 125, "lag_events": 0, "partitioned": false },
    { "name": "eu-west", "applied_events": 125, "lag_events": 0, "partitioned": false },
    { "name": "ap-south", "applied_events": 124, "lag_events": 1, "partitioned": false }
  ]
}

lag_events is relay_log_len − applied_events (saturating). A non-zero, growing lag on a region is the signal that it is partitioned or its segment-receiver is wedged.

Promote a new leader

curl -X POST "$BASE/cluster/promote" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "eu-west" }'
# → { "ok": true, "leader": "eu-west" }

After promotion /cluster/status reports the new leader; new writes route there and replay to the other regions. An unknown region returns 400.

Simulate a partition & heal

# Isolate ap-south: leader ships skip this follower, so its lag climbs.
curl -X POST "$BASE/cluster/partition" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "ap-south" }'
# → { "ok": true, "partitioned": "ap-south" }

# Heal: missed segments are re-shipped over gRPC (blocking, offloaded to the
# write pool — a saturated pool degrades to 429, not 500).
curl -X POST "$BASE/cluster/heal" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "ap-south" }'
# → { "ok": true, "healed": "ap-south" }

7. Sharded scatter-gather API

The /sharded/* routes hash-partition entities across regions (hash(entity_id) % num_shards, using the engine's own ShardRouter so writes and reads never disagree) and fan a query out to all shards, K-way merging the per-shard results by score.

Write routes (each writes to the single owning shard):

curl -X POST "$BASE/sharded/items"      -d '{ "entity_id": 7, "metadata": { "title": "..." } }'  # 201
curl -X POST "$BASE/sharded/embeddings" -d '{ "entity_id": 7, "values": [0.1,0.2,0.3,0.4] }'      # 204
curl -X POST "$BASE/sharded/signals"    -d '{ "entity_id": 7, "signal": "view", "weight": 1.0 }'  # 204

Read routes (scatter-gather across all shards):

curl "$BASE/sharded/feed?profile=for_you&limit=20&deadline_ms=50"
curl "$BASE/sharded/search?query=jazz&limit=10&deadline_ms=100"

Each sharded read accepts an optional deadline_ms — the total scatter budget (default 50ms, server-clamped to 10s). The per-shard deadline is deadline_ms − 5ms network overhead. The response includes a scatter_gather block:

{
  "items": [ /* merged, deduped, diversity-enforced, re-ranked */ ],
  "total_candidates": 4210,
  "scatter_gather": {
    "degraded": false,
    "shards_queried": 3,
    "elapsed_ms": 12,
    "shard_deadline_ms": 45
  }
}

Degraded semantics: a shard that is partitioned, errors, or misses the deadline is reported in unavailable_shards (a name list) and flips degraded: true — it is never silently dropped. The merge dedups replicated copies of an entity (keeping the best-scoring copy), reconciles total_candidates so replicated shards are not counted multiple times, and re-enforces max_per_creator across the merged set.

8. Write-durability contract (read this before trusting a 204)

A 204 No Content from /signals (and the data writes) means the write is durably applied on the leader — storage updated plus WAL fsync — and nothing more.

• The follower ship is best-effort. A ship that fails is logged engine-side and queued for re-delivery via the heal / convergence path. The 204 does NOT assert quorum and does NOT assert any follower acknowledged the write.
• If the single process crashes after the 204 but before followers caught up, the leader's WAL still has the write (it survives restart from disk); the followers will reconcile on the next ship/heal. But because there is only one process, a crash is total downtime, not a failover — see the status box.
• A quorum-ack write contract is explicitly future work, tracked with the multi-process cluster effort (m8p10).

In short: 204 = leader durability, not cluster durability. Design your client retries accordingly (the writes are idempotent on entity_id + signal).

9. Failover drill

1. Baseline. GET /cluster/status; confirm the expected leader and lag_events: 0 on every region.
2. Pre-seed reads. Issue a region-pinned read against the target region (?region=eu-west) to confirm it is serving and roughly caught up.
3. Promote. POST /cluster/promote { "region": "eu-west" }.
4. Verify. GET /cluster/status now reports eu-west as leader. Send a write (POST /signals) and confirm relay_log_len advances and the other regions' applied_events follow within a heartbeat.
5. Cut over traffic. Point your client's writes at the same $BASE (the single surface routes writes to whoever is leader) — no client change needed.

Reminder: this is a leadership move within one process, not a host failover. It is the right drill for "move the write region during maintenance," not for "survive a machine dying."

10. Partition drill

1. Baseline. GET /cluster/status; all lag_events: 0, all partitioned: false.
2. Inject. POST /cluster/partition { "region": "ap-south" }. Confirm partitioned: true for that region.
3. Write through it. Send several POST /signals. Watch ap-south's lag_events climb while applied_events stalls — the leader's ships to it are dropped.
4. Read the stale follower. GET /feed?region=ap-south should return the pre-partition view (demonstrating eventual, not strong, read consistency).
5. Heal. POST /cluster/heal { "region": "ap-south" }. Missed segments re-ship over gRPC.
6. Verify convergence. Poll GET /cluster/status until ap-south's lag_events returns to 0 and partitioned: false.
7. Scatter-gather degradation. While partitioned, a GET /sharded/feed should report degraded: true with ap-south in unavailable_shards — not an error. Confirm it clears after heal.

11. Shutdown

Send SIGTERM (or SIGINT / Ctrl+C, or stop the container). The server flips readiness to 503, drains in-flight requests, then drops the cluster — which runs each region's TidalDb shutdown path: checkpoint in-memory signal state → flush storage → write the WAL checkpoint marker + fsync → join the WAL, sweeper, checkpoint, and text-syncer threads. The drop is idempotent. You will see cluster shutdown: closing all nodes (checkpoint + WAL fsync).

Cross-references

• Kubernetes deployment — docs/runbooks/kubernetes.md (planned for the m8p10 multi-process work; not yet written — until then deploy a single standalone node per the production guidance in the status box, and use the Docker HEALTHCHECK / /health probe as the readiness gate).
• Server deployment guide — docs/guides/server-deployment.md (planned; not yet written — see API.md and QUICKSTART.md for the current server config and route reference).
• Monitoring & alerts — docs/ops/monitoring.md (Prometheus scrape config, replication-lag and WAL metrics, recommended alerts; remember the /metrics endpoint is unauthenticated — bind it internally).
• Roadmap / m8p10 (true multi-process HA) — docs/planning/ROADMAP.md for the distributed-fabric status, the m8p1–m8p9 completed phases, and the outstanding multi-process / tier-3 chaos work.
• API & schema reference — API.md, QUICKSTART.md, and the live /openapi.json document.
• Scope & vision — VISION.md.