tidaldb/docs/runbooks/cluster.md

tidalDB Cluster Runbook

Operating the multi-region tidal-server cluster surface: launch (single-process dev fabric and the multi-process region nodes), the operational API, replication transport facts, failover and partition drills, and the honest write-durability contract.

STATUS: EXPERIMENTAL — TWO MODES, NEITHER IS QUORUM-ACKED HA YET

Cluster mode has two shapes, both behind the same experimental opt-in:

1. Multi-process (--region) — real process isolation. Each tidal-server cluster --region <name> process owns exactly one region: one TidalDb, one GrpcTransport whose server binds this region's grpc_addr and whose peers are every sibling region's real grpc_addr. Processes peer over real gRPC and forward over real HTTP, so a crash of one region's process takes down only that region — the survivors keep serving. This is genuine process (and, across hosts, host) isolation. It is verified end-to-end by the tier-3 suites (cluster_multiproc, cluster_chaos, cluster_lifecycle, cluster_runbook) over real OS processes with real network-partition injection.

2. Single-process (no --region) — the dev/demo default. Every region runs inside one process. Replication still traverses the real tidal-net gRPC transport on loopback (faithful multi-region semantics over a real wire), but there is no process isolation: a crash, OOM, or host failure takes the whole "cluster" down at once. This is a development / staging / demo fabric and a correctness harness for the replication paths — not production HA. It remains the default because it needs no per-region topology addresses and no process orchestration.

Honest remaining limits (both modes):

• Writes are leader-durable, NOT quorum-acked. A 204 means the leader durably applied the write (storage + WAL fsync). The follower ship is best-effort; no follower acknowledgement is asserted. A quorum-ack write contract is post-M8 follow-up (see §8).
• Leadership is operator-driven, not automatic. There is no automatic failure detector and no automatic leader election. /cluster/promote moves leadership and fans the new view out to peers; a node that misses the fan-out self-corrects on its next forwarded write / status poll. "Survive a machine dying" is an operator runbook step (detect → promote), not an automatic failover.

For a production deployment today, run a single tidal-server standalone node backed by host-level redundancy and disk durability (see kubernetes.md and server-deployment.md), and reach for multi-process cluster mode for read-scale / multi-region experiments — not as a quorum-HA story.

Both modes refuse to start unless you explicitly opt in with either the --experimental-cluster flag or the TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 environment variable. On start each emits a loud, mode-specific WARN restating exactly what it does and does not provide. Do not wire either into a production load balancer.

Scope: what cluster mode does and does not own

tidalDB owns retrieval and ranking only. Cluster mode adds multi-region replication of that, nothing more. It does not generate embeddings, store video/blobs, run a CDN or transcoder, do auth/moderation/payments, or provide a distributed consensus log. Bring your vectors; tidalDB retrieves and ranks over them. See VISION.md for the scope boundary.

Prerequisites

• Rust toolchain ≥ 1.91 if running directly (the Docker build pins rust:1.91-bookworm).
• protobuf-compiler (protoc) and a C++ toolchain (g++) on the build host — tidal-net's build script compiles the WAL-shipping .proto, and USearch's HNSW core is C++. The Docker image installs both.
• Docker 25+ if running via container.
• For single-process mode: one HTTP port (default 9500); each follower region also binds an OS-assigned loopback port for its gRPC replication server unless you pin one in the topology (see §3).
• For multi-process mode: per-region grpc_addr and http_addr declared in the topology, and the ports they name available on each host (see §3).

1. Launch the cluster locally

Cluster mode is gated. Pass --experimental-cluster (or set TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1) or the server exits with a mode-specific error explaining why.

1a. Single-process (dev/demo default)

TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
cargo run -p tidal-server -- \
  cluster \
  --listen 127.0.0.1:9500 \
  --schema tidal-server/config/default-schema.yaml \
  --topology tidal-server/config/default-cluster.yaml \
  --experimental-cluster

The default topology spins up three regions (us-east, eu-west, ap-south) with us-east as leader, all inside one process. On a clean start you will see a WARN line stating this is experimental and single-process, an info line per follower (follower gRPC transport ready), and finally listening on http://127.0.0.1:9500.

1b. Multi-process (one process per region)

Pass --region <name> (or set TIDAL_REGION) to run only that region in this process. The topology must declare a per-region grpc_addr and http_addr for every region (see §3); siblings reach each other over those. Launch one process per region — typically one per host, each with its own --data-dir. --data-dir is REQUIRED in multi-process mode (m11p2): the durable WAL is the replicated log itself — it is what ships to peers and what serves their catch-up streams — so a node without one is rejected at startup:

# Region us-east (the initial leader) on host A
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
tidal-server cluster --experimental-cluster \
  --region us-east \
  --listen 0.0.0.0:9501 \
  --schema /etc/tidal/schema.yaml \
  --topology /etc/tidal/topology.yaml \
  --data-dir /var/lib/tidal/us-east

# Region eu-west on host B
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
tidal-server cluster --experimental-cluster \
  --region eu-west --listen 0.0.0.0:9502 \
  --schema /etc/tidal/schema.yaml --topology /etc/tidal/topology.yaml \
  --data-dir /var/lib/tidal/eu-west

# Region ap-south on host C
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
tidal-server cluster --experimental-cluster \
  --region ap-south --listen 0.0.0.0:9503 \
  --schema /etc/tidal/schema.yaml --topology /etc/tidal/topology.yaml \
  --data-dir /var/lib/tidal/ap-south

Every process parses the same topology file; RegionIds are assigned by declaration order, so all processes agree on the region→id mapping. Each process binds its own region's grpc_addr/http_addr and dials its siblings' addresses. The --listen HTTP address is the public gateway for that region.

Auth: set TIDAL_API_KEY=<secret> to require Authorization: Bearer <secret> on the data and /cluster/* mutation routes. If it is unset the server runs UNAUTHENTICATED and logs a WARN — never expose an unauthenticated cluster beyond loopback / a trusted VPC. Health probes and /openapi.json are always unauthenticated. In multi-process mode set the same key on every process: a forwarded/broadcast request passes the caller's Authorization through verbatim, and the internal-propagation marker (x-tidal-internal: 1) is an inter-sibling trust signal, not an auth bypass (the bearer middleware still runs first).

Useful environment variables:

Var	Mode	Effect
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1	both	Opt in to cluster mode (alternative to --experimental-cluster).
TIDAL_REGION	multi-process	Selects the single region this process owns (alternative to --region).
TIDAL_API_KEY	both	Bearer token for protected routes. Unset ⇒ unauthenticated + WARN. Set the SAME key on every region in multi-process mode.
TIDAL_HLC_SKEW_MS	multi-process	Signed ms offset applied to THIS process's HLC. Affects reconcile-time LWW stamping ONLY (not signal-decay timestamps). A test/ops escape hatch for verifying causal convergence under clock skew — do not set it in normal operation.
TIDAL_CONFIG	both	Config dir holding default-schema.yaml / default-cluster.yaml (used when --schema / --topology omitted).
PORT	both	Listen address. A bare port (9500) normalises to 0.0.0.0:9500.
TIDAL_SERVER_LOG	both	tracing filter (default info).

2. Launch via Docker

# Build the image once (build context is the repo root).
docker build -f docker/cluster/Dockerfile -t tidaldb:cluster .

# Run (press Ctrl+C to stop). The image already sets
# TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 so cluster mode starts; it still logs the
# loud experimental WARN. The default CMD is SINGLE-process cluster mode.
docker run --rm -p 9500:9500 tidaldb:cluster

The image bakes the default schema/topology under /etc/tidal-server and uses ENTRYPOINT ["tidal-server"] + a cluster … CMD, so you can override the subcommand (e.g. docker run tidaldb:cluster standalone …), pass --region for a multi-process container, or supply your own config files:

docker run --rm -p 9500:9500 \
  -e TIDAL_API_KEY=changeme \
  -v "$PWD/configs/my-schema.yaml:/srv/schema.yaml:ro" \
  -v "$PWD/configs/my-topology.yaml:/srv/topology.yaml:ro" \
  tidaldb:cluster \
  cluster \
    --listen 0.0.0.0:9500 \
    --schema /srv/schema.yaml \
    --topology /srv/topology.yaml

The container ships a HEALTHCHECK that curls /health. For Kubernetes/Compose deployment patterns see Cross-references.

3. Topology YAML

The topology file declares regions and the leader.

Single-process minimal default (tidal-server/config/default-cluster.yaml):

regions:
  - name: us-east
  - name: eu-west
  - name: ap-south
leader: us-east
# Optional: OS worker threads serving cluster write/heal requests (gRPC ship).
# Defaults to available parallelism when omitted.
# write_workers: 4

Multi-process requires a per-region grpc_addr and http_addr (validated at startup — a missing or syntactically invalid host:port is a hard error naming the region; reachability is deliberately not probed, since siblings boot in any order):

regions:
  - name: us-east
    grpc_addr: "10.0.1.10:9601"   # this region's gRPC replication bind / dial addr
    http_addr: "10.0.1.10:9501"   # this region's public HTTP gateway (forwarding + status)
    metrics_addr: "10.0.1.10:9091" # optional Prometheus /metrics listener (set it in production)
  - name: eu-west
    grpc_addr: "10.0.2.10:9602"
    http_addr: "10.0.2.10:9502"
    metrics_addr: "10.0.2.10:9091"
  - name: ap-south
    grpc_addr: "10.0.3.10:9603"
    http_addr: "10.0.3.10:9503"
    metrics_addr: "10.0.3.10:9091"
leader: us-east
# Optional m11p1 tuning blocks (engine defaults shown):
# replication:
#   batch_max_events: 256   # events coalesced per shipped batch (1-256)
#   window: 4               # in-flight batches per peer (1-64)
#   retry_ms: 100           # backoff before a transiently-failed batch retries
# wal:
#   batch_size: 100         # events per group-commit fsync (1-256)
#   batch_timeout_ms: 10    # max wait before a partial batch flushes

Fields:

Field	Single-process	Multi-process	Meaning
regions[].name	required	required	Region name used everywhere in the HTTP API (?region=, /cluster/promote, etc.). Must be unique.
regions[].grpc_addr	optional	required	This region's gRPC replication address. In single-process mode omit it — the server allocates a free loopback port and self-heals a bind race by retrying on a fresh port. In multi-process mode it is required (siblings dial it) and is tried exactly once.
regions[].http_addr	unused	required	This region's public HTTP gateway address, used by siblings for write/read forwarding and status aggregation. Unused in single-process mode.
regions[].grpc_tls	unused	optional	TLS material for this region's gRPC transport: ca_cert, server_cert, server_key (PEM paths), plus optional client_cert/client_key for mTLS. Omitted ⇒ plaintext, the right posture for loopback/VPC topologies.
regions[].metrics_addr	unused	optional	Bind address for this region's Prometheus /metrics listener (m11p1; cluster mode previously had none). Omitted ⇒ no metrics endpoint. Set it in every production topology, and bind it internally — the endpoint is unauthenticated.
leader	required	required	Must name one of the declared regions. The initial write leader.
write_workers	optional	optional	Size of the runtime-free OS-thread pool that admission-controls cluster writes (/signals staging) and runs /cluster/heal's blocking redelivery. Bounded queue ⇒ 429 backpressure. Must be ≥ 1 when given.
timeouts.broadcast_peer_secs	unused	optional	Per-peer budget (seconds) for the /cluster/promote fan-out (the only remaining peer fan-out — the m11p2 log replaced the item/embedding broadcast). Default 2s — right for loopback/VPC; raise it for WAN topologies where a distant region cannot answer in 2s. Must be ≥ 1 when given.
replication.batch_max_events	unused	optional	Max relay events coalesced into one shipped batch (1–256, the WAL wire-format ceiling). Default 256.
replication.window	unused	optional	In-flight batches per peer (1–64). 1 = strictly in-order shipping; higher pipelines across the peer RTT (out-of-order arrivals park gap-aware on the receiver). Default 4.
replication.retry_ms	unused	optional	Backoff (ms) before a transiently-failed batch ship retries. Default 100.
wal.batch_size	optional	optional	Events per WAL group-commit fsync (1–256). Default 100.
wal.batch_timeout_ms	optional	optional	Max ms a partial group-commit batch waits before flushing. Default 10. Tune against the measured tidaldb_cluster_wal_fsync_us on the deployment's volume.

The schema YAML (signals / text_fields / embedding_slots / profiles) is the same format the standalone server loads — see API.md and QUICKSTART.md for the full schema/profile grammar and the built-in ranking profiles. Topology only adds the region layout above.

Personalization correctness note. Preference-vector personalization (the taste vector that powers for_you) only updates for signals declared positive_engagement: true in the schema. The cluster /signals route writes global signals (no user_id / creator_id context), so it never folds item embeddings into a user's taste vector regardless of the positive_engagement flag. Personalized writes go through the embedded engine (signal_with_context), not this HTTP route. The exception is /hardnegs, which records a user-scoped hide that converges across regions via /cluster/reconcile.

4. gRPC replication transport (tidal-net)

Replication between regions is the real tidal-net WalShipping gRPC service, not in-process channels. The facts you need to operate and firewall it:

Property	Value	Notes
Service	WalShipping (proto tidal.replication.v1)	RPCs: ShipSegment (unary; the production replication path) and Heartbeat (ControlPlane health). StreamSegments is declared but not implemented — segment delivery is the unary ShipSegment.
Bind address	per-region grpc_addr (multi-process); an auto-allocated loopback port (single-process)	Default dev band 59520–59529 if you pin one.
Transport security	mTLS via TlsConfig (ca_cert, server_cert, server_key, optional client_cert / client_key)	insecure = true (plaintext) is the loopback/VPC default in this phase. A peer with no TLS config must set insecure.
Max payload	64 MiB	Both encoder and decoder codec limits are raised to this; pinned by a compile-time assert to the engine's InProcessTransport limit so both ends agree. WAL segments default to 16 MiB.
Circuit breaker	per-peer, threshold 5, reset 30s	After 5 consecutive ship failures the breaker opens; it stays open for 30s, then the next attempt probes (HalfOpen → Closed on success, Open again on failure). Backpressure (channel full) does not trip it. Operational consequence: after a partition or a node-down window the breaker is open, so a single /cluster/heal can ship into an open breaker and no-op — re-issue /cluster/heal until /cluster/status shows lag 0 (see §6 and the drills in §10).
Timeouts (defaults)	connect 5s, request 10s, keep-alive PING every 10s / 5s ACK	A blackholed peer fails fast instead of stalling the single-threaded shipper.

ShipSegment carries the WAL segment id, BLAKE3-validated payload bytes, event count, and the leader's authoritative leader_last_seq (so a follower can advance its replication-lag high-water-mark even for an all-local segment that filters to empty). The follower's segment-receiver thread drains and applies it on arrival. The engine's applied high-water-mark is a contiguous frontier with a bounded ahead-buffer, so out-of-order eager ships can never swallow a sequence gap (a silent-data-loss class fixed in m8p10).

Operational guard: the gRPC replication ports and the Prometheus /metrics endpoint are UNAUTHENTICATED. Bind them to loopback or a cluster-internal network only — never the public interface.

5. Core HTTP API

All routes are JSON unless noted. Examples assume BASE=http://localhost:9501 (any region's gateway in multi-process mode; the single --listen address in single-process mode) and, when TIDAL_API_KEY is set, AUTH='-H "Authorization: Bearer $TIDAL_API_KEY"'. Drop the -H header when running unauthenticated.

Middleware mirrors standalone: 30s request timeout (408), 100 max in-flight (429), 2 MB body limit (413), and an x-request-id on every response. Health probes sit outside the load-shedding stack so liveness/readiness are never queued or timed out under saturation.

Multi-process routing (one coherent surface). Any region's gateway accepts any operation and routes it to the node that owns it:

• Writes (/signals, /items, /embeddings, /hardnegs) on a non-leader forward to the leader (the caller's Authorization passes through); the client sees the leader's status/body. A leader that is unreachable degrades to a 503 naming the leader, never a hang.
• Reads (/feed, /search) default to the LOCAL region. This is the key difference from single-process mode, whose default read is the leader. A ?region=<other> read forwards to that region's process.
• /cluster/promote, /cluster/partition, /cluster/heal route to / fan out from the leader as documented in §6.

Health

curl "$BASE/health"          # 200 ok / 503 while draining; reports mode, region, leader
curl "$BASE/health/startup"  # always 200
curl "$BASE/health/live"     # always 200
curl "$BASE/openapi.json"    # served OpenAPI 3.1, UNAUTHENTICATED — canonical HTTP reference

/health returns { "ok": true, "service": "tidaldb", "mode": "cluster", "region": "us-east", "leader": "us-east", ... }. The /openapi.json document is the machine-readable source of truth for the data, /cluster/*, /sharded/*, and /hardnegs request/response shapes (the health probes are intentionally outside the documented API surface).

Register items & embeddings (one replicated log — m11p2)

Items and embeddings ride the same replicated WAL stream as signals: the leader journals each write as a kind-1 (item metadata) or kind-2 (embedding) WAL record BEFORE touching storage, and followers apply it from the log — live pushes for the hot tail, the StreamSegments catch-up stream for history. There is no HTTP broadcast anymore (the m8p10 side channel and its bug classes were deleted in m11p2), so the responses are plain statuses:

curl -X POST "$BASE/items" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "metadata": { "title": "Jazz Piano", "category": "music" } }'
# → 201 Created (item durably journaled into the replicated log + applied)

curl -X POST "$BASE/embeddings" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "values": [0.1, 0.2, 0.3, 0.4] }'
# → 204 No Content

The 201/204 asserts leader durability: the record is fsynced into the stream every follower receives (live push, or pull-based catch-up after downtime — see §6 heal). A write to a non-leader forwards to the leader transparently. A down/partitioned peer needs no backfill bookkeeping: it converges from the log when it returns.

Embeddings: tidalDB does not generate vectors — the caller brings them. The write L2-normalizes and inserts into the HNSW index. Dimensions are strict: they must equal the slot's declared dimensions (min 2, max 4096) or the insert is rejected with a 500 (the engine surfaces a dimension mismatch as an internal error, not a 400 — a known wart, tracked post-M8); zero-norm vectors are also rejected (500). RETRIEVE / SEARCH route through the first declared embedding slot only; multi-modal apps must fuse offline or use separate entity kinds.

Record signals (cluster /signals = global only)

curl -X POST "$BASE/signals" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "signal": "view", "weight": 1.0 }'
# → 204 No Content (see the durability contract in §8)

The signal name must be declared in the schema (an undeclared name returns 400 naming it). This route records a global signal on the leader and ships it to followers over the replicated WAL stream; it does not personalize (see the personalization note in §3). On a non-leader gateway it forwards to the leader transparently and still returns 204.

Hard negatives (user-scoped hides)

curl -X POST "$BASE/hardnegs" \
  -H 'Content-Type: application/json' \
  -d '{ "user_id": 42, "item_id": 7 }'
# → 204 No Content

Records a user-scoped hide on the leader (a non-leader gateway forwards). The item is filtered from that user's /feed?user_id=42. Hard negatives converge across regions via the LWW-resolved /cluster/reconcile CRDT path — NOT the signal WAL relay and NOT a broadcast.

Retrieve and search (region-pinned reads)

# Default read region is the LOCAL region (multi-process mode).
curl "$BASE/feed?user_id=42&profile=for_you&limit=20"
curl "$BASE/search?query=jazz%20piano&user_id=42&limit=5"

# Pin a read to a specific region. The gateway forwards to that region's process.
# Followers may lag the leader (and lag jumps during a partition) — use this for
# canary reads and lag verification.
curl "$BASE/feed?profile=trending&region=eu-west"

?region= accepts any declared region name; an unknown name returns 400. Omit it and the read serves from the LOCAL region (multi-process) / the current leader (single-process). limit is clamped to 1000 at the trust boundary (a larger value cannot amplify memory unboundedly). The for_you profile applies the built-in diversity defaults (max_per_creator=2, format_mix_max_fraction=0.4, exploration=0.1).

6. Cluster management API

Check cluster status

Two views. /cluster/status/local reports THIS node's own replication/leadership state (no peer calls); /cluster/status aggregates EVERY region (the gateway calls each peer's /cluster/status/local concurrently with a tight per-peer budget).

curl "$BASE/cluster/status/local" | jq

{
  "region": "ap-south",
  "is_leader": false,
  "leader": "us-east",
  "last_seq": 0,
  "applied_events": 124,
  "lag_events": 1,
  "partitioned": [],
  "reachable": true
}

curl "$BASE/cluster/status" | jq

{
  "leader": "us-east",
  "relay_log_len": 125,
  "regions": [
    { "name": "us-east",  "applied_events": 125, "lag_events": 0, "partitioned": false, "reachable": true },
    { "name": "eu-west",  "applied_events": 125, "lag_events": 0, "partitioned": false, "reachable": true },
    { "name": "ap-south", "applied_events": 124, "lag_events": 1, "partitioned": false, "reachable": true }
  ]
}

relay_log_len is the leader's high-water-mark (last_seq); each region's lag_events is relay_log_len − applied_events (saturating). A region the gateway cannot reach within the per-peer budget is reported honestly as reachable: false, partitioned: true, applied_events: 0, and worst-case lag (lag_events == relay_log_len). A non-zero, growing lag on a reachable region is the signal that it is partitioned (ship-skip) or its segment-receiver is wedged.

Promote a new leader

curl -X POST "$BASE/cluster/promote" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "eu-west" }'
# → { "ok": true, "leader": "eu-west", "acked": ["eu-west","ap-south"], "failed": [] }

The gateway first resolves the TARGET's stream baseline — the new leader's WAL flushed frontier at promotion, persisted in its data_dir/stream_baseline — then fans the promote out to every peer carrying it (internal marker, so each applies locally and does not re-fan). Peers jump their applied frontier for the new leader's shard to the baseline: everything at or below it is pre-stream history (replicated applies of the OLD stream), not data. The response carries baseline plus the acked/failed fan-out report; a peer in failed (e.g. a dead old leader) self-corrects — its first parked batch from the new stream triggers a catch-up pull whose chunks announce the baseline. After promotion /cluster/status reports the new leader; new writes route there; the new leader's ship queue activates and the demoted node's deactivates. An unknown region returns 400. There is no automatic election — this is the operator's failover lever (m11p4 adds elections).

Simulate a partition & heal

There are two ways to partition a region; the runbook drills demonstrate both (§10):

1. Simulated ship-skip flag — /cluster/partition tells the leader to stop shipping to the named region (no sockets touched). Good for a controlled, reversible lag demo.
2. Real network partition — sever the actual TCP path between processes (firewall / proxy). The chaos suite uses a root-free in-harness TCP relay; operators can use iptables/pfctl (see §10).

# Simulated: isolate ap-south — leader ships skip this follower, so its lag climbs.
curl -X POST "$BASE/cluster/partition" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "ap-south" }'
# → { "ok": true, "partitioned": "ap-south" }

# Heal: resume shipping past everything the follower reports applied, and
# nudge it to PULL any history that rotated out of the leader's ship tail
# (the single recovery verb — log catch-up IS the full heal since m11p2).
curl -X POST "$BASE/cluster/heal" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "ap-south" }'
# → { "ok": true, "healed": "ap-south" }

/cluster/heal is the single recovery verb, and since m11p2 it is pure log catch-up: it clears the partition flag, resumes the ship queue past the follower's reported applied seqno (retries of data the follower already holds prune automatically — every ship ack piggybacks the follower's applied seqno), and nudges the follower to pull anything older than the leader's in-memory ship tail via its StreamSegments catch-up stream over the leader's durable WAL segments (POST /cluster/catchup, internal; the nudge forwards the healing operator's own bearer credential). Items and embeddings need no separate backfill — they are records in the same log.

Convergence is self-driving. The per-peer ship senders retry parked batches every replication.retry_ms (default 100ms); a follower that detects a gap pulls the catch-up stream itself (also on boot, so a restarted node converges with no operator action at all — the tier-3 mp_items_ride_the_log_and_catchup_stream proves it). /cluster/heal remains the explicit verb for peers paused by /cluster/partition or a PERMANENT transport failure (TLS/auth/codec — those never self-resume by design). The per-peer gRPC circuit breaker (threshold 5, reset 30s — see §4) can still swallow the first post-heal ships, so if GET /cluster/status does not show lag_events: 0 within the breaker window, re-issue POST /cluster/heal — the chaos suite's heal_until_converged does exactly this.

Reconcile (cross-region CRDT convergence)

curl -X POST "$BASE/cluster/reconcile" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "ap-south" }'
# → { "ok": true, "region": "ap-south",
#     "local_elapsed_ms": 0, "remote_elapsed_ms": 1, "ops_applied": 3 }

/cluster/reconcile exchanges a CRDT state snapshot with the target region: this node ships its snapshot into the target's merge AND applies the target's pre-merge snapshot back, so both sides converge to identical state by deterministic LWW. This is how hard negatives (recorded via /hardnegs) converge across regions — they do not ride the WAL relay. local_elapsed_ms / remote_elapsed_ms are the merge+apply times each side measured (not the HTTP round-trip); both are typically 0–1ms. Reconcile is idempotent: a repeat reconcile of already-converged regions is an exact no-op on scores (no drift). /cluster/reconcile/snapshot is the INTERNAL snapshot-exchange leg this verb drives (marker required); operators never call it directly.

7. Sharded scatter-gather API

The /sharded/* routes hash-partition entities across regions (hash(entity_id) % num_shards, using the engine's own ShardRouter so writes and reads never disagree). Writes route to the single owning region; reads fan out to all regions and K-way merge by score.

Write routes (each routes to the owning region; a non-owner gateway forwards):

curl -X POST "$BASE/sharded/items"      -d '{ "entity_id": 7, "metadata": { "title": "..." } }'  # 201
curl -X POST "$BASE/sharded/embeddings" -d '{ "entity_id": 7, "values": [0.1,0.2,0.3,0.4] }'      # 204
curl -X POST "$BASE/sharded/signals"    -d '{ "entity_id": 7, "signal": "view", "weight": 1.0 }'  # 204

Read routes (scatter-gather across all regions):

curl "$BASE/sharded/feed?profile=for_you&limit=20&deadline_ms=50"
curl "$BASE/sharded/search?query=jazz&limit=10&deadline_ms=100"

Each sharded read accepts an optional deadline_ms — the total scatter budget (default 50ms, server-clamped to 10s). The per-shard deadline is deadline_ms − 5ms network overhead. The response includes a scatter_gather block:

{
  "items": [ /* merged, deduped, diversity-enforced, re-ranked */ ],
  "total_candidates": 4210,
  "scatter_gather": {
    "degraded": false,
    "shards_queried": 3,
    "elapsed_ms": 12,
    "shard_deadline_ms": 45
  }
}

Degraded semantics: a shard that is partitioned, errors, or misses the deadline is reported in unavailable_shards (a name list) and flips degraded: true — it is never silently dropped, and the read still returns 200 with the live shards' results. The merge dedups replicated copies of an entity (keeping the best-scoring copy), reconciles total_candidates so replicated shards are not counted multiple times, and re-enforces max_per_creator across the merged set.

8. Write-durability contract (read this before trusting a 204)

A 204 No Content from /signals (and the data writes' success codes) means the write is durably applied on the leader — storage updated plus WAL fsync — and nothing more.

Since m11p2 there is one replicated log: the leader's WAL. Every replicated mutation — signals (kind-0 batches) AND items/embeddings (kind-1/2 blob records, journaled BEFORE storage) — rides it, and what ships to peers is the WAL's own fsynced batches, byte-identical on disk and on the wire.

• The success code returns at leader group-commit fsync. Follower shipping happens entirely off the request path on per-peer sender threads that push the WAL flush feed's tail (replication.batch_max_events per run, replication.window runs in flight per peer). Concurrent writers share group-commit fsyncs instead of serializing one solo fsync each — this is what lifted the replicated ceiling from ~90/s to thousands/s (m11p1).
• Peers only ever receive fsynced batches by construction: nothing enters the ship feed before its fsync completes, so an event a follower holds but the leader could lose is impossible.
• If a write's fsync fails, that write errors (the WAL writer notifies every waiting caller and keeps serving; the failed seqno range is reused on retry) — nothing unfsynced can ship. Repeated fsync failures are a dying volume: restart the node or promote a survivor.
• The follower ship is best-effort. A ship that fails is parked and retried by its sender every replication.retry_ms (first failure and every 50th log at WARN with the running count; recovery logs at INFO). Data that rotates out of the in-memory ship tail during a long outage is follower-pulled via the StreamSegments catch-up stream over the durable segments. The success code does NOT assert quorum and does NOT assert any follower acknowledged the write (quorum acks are m11p3).
• In multi-process mode, if the leader's process crashes after the 204 but before followers caught up, the leader's WAL still has the write (it survives restart from disk); the followers reconcile on the next ship/heal. The other regions' processes keep serving — this is real process isolation, but it is not an automatic failover (an operator promotes a survivor — see §9).
• In single-process mode there is only one process, so a crash is total downtime, not a failover.

A quorum-ack write contract (a 204 that asserts N followers acknowledged) is explicitly post-M8 follow-up work, tracked in docs/planning/ROADMAP.md M8 Known Gaps. It is not part of m8p10.

In short: 204 = leader durability, not cluster durability. Design your client retries accordingly (the writes are idempotent on entity_id + signal).

9. Failover drill (multi-process)

Move the write leader to another region. Scripted exactly as the runbook-verification suite (cluster_runbook.rs::runbook_s9_failover_drill) executes it:

1. Baseline. GET /cluster/status; confirm the expected leader and lag_events: 0 on every region.
2. Pre-seed reads. Issue a region-pinned read against the target region (?region=eu-west) to confirm it is serving and roughly caught up.
3. Promote. POST /cluster/promote { "region": "eu-west" }. Confirm the { ok, leader, acked, failed } response. If the OLD leader is dead, expect it in failed — that is fine.
4. Verify. GET /cluster/status now reports eu-west as leader. Send a write (POST /signals) to the new leader and confirm relay_log_len advances and the other regions' applied_events follow within a heartbeat.
5. Cut over traffic. Point your client's writes at any region gateway — a write to a non-leader forwards to the new leader transparently (204), no client change needed.

Crash failover is the same drill triggered by a real outage: a region's process dies (its gateway stops answering), you detect it (monitoring on /health / /cluster/status reachable), and you POST /cluster/promote a survivor via another survivor's gateway. The tier-3 mp_uat_step2_leader_crash_failover_under_10s test SIGKILLs the leader and proves promote→first-successful-write < 10s with zero data loss. There is no automatic detector/election — promotion is the operator step.

This is a leadership move, not a quorum hand-off. Use it for "move the write region during maintenance" and for "a region died — promote a survivor."

10. Partition drill (multi-process)

The rewritten drill demonstrates both partition mechanisms, exactly as cluster_runbook.rs::runbook_s10_partition_drill scripts them.

1. Baseline. GET /cluster/status; all lag_events: 0, partitioned: false, reachable: true.
2. Inject. Use one of:
   • Real network partition — sever the TCP path peers use to reach the region. The chaos suite (cluster_chaos.rs) uses a root-free in-harness TCP relay proxy (the ROADMAP-sanctioned toxiproxy-style alternative). On a real host you can instead use iptables (Linux) or pfctl (macOS), e.g. iptables -A INPUT -p tcp --dport 9603 -j DROP to blackhole ap-south's gRPC port from a peer. A real cut shows reachable: false in the aggregate status.
   • Simulated ship-skip flag — POST /cluster/partition { "region": "ap-south" } (→ { ok, partitioned: "ap-south" }). The leader stops shipping to ap-south without touching sockets; the aggregate status shows partitioned: true.
3. Write through it. Send several POST /signals. Each must still 204 (the leader-durable contract holds). Watch ap-south's lag_events climb while its applied_events stalls — the leader's ships to it are dropped.
4. Read the stale follower. GET /feed?region=ap-south (or read ap-south's gateway directly) returns the pre-partition view — eventual, not strong, read consistency. The operator console survives a real partition because clients talk to each region's gateway directly.
5. Scatter-gather degradation. While partitioned, GET /sharded/feed returns 200 with degraded: true and ap-south in unavailable_shards — never an error, and the live shards' items are still returned.
6. Heal. Clear the cut (heal the proxy / drop the firewall rule, or just call heal for the simulated flag), then POST /cluster/heal { "region": "ap-south" }. Re-issue heal until /cluster/status shows ap-south at lag_events: 0 — the gRPC circuit breaker opened during the partition (threshold 5, reset 30s), so the first heal may ship into an open breaker and no-op. This is genuine production behavior, not a flag.
7. Verify convergence. GET /cluster/status shows ap-south lag_events: 0, partitioned: false, reachable: true; /sharded/feed is no longer degraded; feed scores on ap-south match the leader to within float tolerance (no loss, no duplication).

11. Shutdown

Send SIGTERM (or SIGINT / Ctrl+C, or stop the container). The server flips readiness to 503 (so a load balancer stops routing to it), drains in-flight requests, then drops the region's TidalDb shutdown path: checkpoint in-memory signal state → flush storage → write the WAL checkpoint marker + fsync → join the WAL, sweeper, checkpoint, text-syncer, and replication-receiver threads. The drop is idempotent; the process exits 0. You will see region cluster node shutdown: database closed (checkpoint + WAL fsync) (multi-process) / cluster shutdown: closing all nodes (checkpoint + WAL fsync) (single-process).

On restart with the same --data-dir, the region recovers its pre-shutdown state from the WAL (verified by cluster_runbook.rs::runbook_s11_shutdown_and_wal_recovery: SIGTERM → exit 0 → restart → the same items are served). In multi-process mode, the restarted node rejoins the cluster and pulls anything it missed while down via its boot-time StreamSegments catch-up request — no operator verb needed (m11p2; /cluster/heal remains the explicit lever for paused peers). This is the per-node step of a rolling upgrade (promote leadership off the node, SIGTERM, restart on the same data dir with the new binary, heal) — see cluster_lifecycle.rs::mp_rolling_upgrade_no_loss_no_stall, which proves zero acknowledged-write loss across a full rolling upgrade under load.

Performance (measured over real localhost processes)

Operation	SLA	Measured (p99 / typical)
Replicated /signals throughput (m11p1, 3 nodes, release build)	≥ 2,000/s	4,534 signal-writes/s within SLO on the ramp (knee ~5.5k/s); 2,739/s sustained 10 min (1.65M writes, 0.35% errors); was ~90/s pre-m11p1
Replication lag under that load (m11p1)	< 2s	≤ 103 events (~40ms) across the 10-min sustain; ≤ 377 events on the 5k/s ramp (follower group-commit coalescing)
Cross-region replication (write → follower applied)	< 2s	~110–133ms p99 (m8p10)
Failover (/cluster/promote → first successful write)	< 10s	~31–34ms
CRDT reconcile (merge+apply, each side)	< 100ms	0–1ms

Write-latency note: signal p50 ≈ 17–25ms with a p99 tail of 160–220ms on macOS, where F_FULLFSYNC averages ~7.4ms with a 10–50ms tail (tidaldb_cluster_wal_fsync_us — 0% complete under 1ms). On Linux fdatasync volumes the same pipeline's fsync floor is far lower; validate the p99 gate on the reference environment, and tune wal.batch_timeout_ms against the measured fsync histogram.

Cross-references

• Kubernetes deployment — docs/runbooks/kubernetes.md (standalone single-replica is the recommended production deployment until quorum-ack / auto-failover land; an experimental StatefulSet-per-region sketch for multi-process cluster mode is noted there).
• Server deployment guide — docs/guides/server-deployment.md (standalone and cluster launch, config, env, health probes).
• Monitoring & alerts — docs/ops/monitoring.md (Prometheus scrape config, replication-lag and WAL metrics, recommended alerts; per-region replication lag is observable via /cluster/status lag_events; remember the /metrics endpoint is unauthenticated — bind it internally).
• Roadmap / M8 status & known gaps — docs/planning/ROADMAP.md for the distributed-fabric status (M8 COMPLETE), the m8p1–m8p10 phase history, and the post-M8 follow-ups (quorum-ack writes, automatic failure detection / leader election).
• API & schema reference — API.md, QUICKSTART.md, and the live /openapi.json document.
• Scope & vision — VISION.md.