tidaldb/docs/runbooks/cluster.md

tidalDB Cluster Runbook

Operating the multi-region tidal-server cluster surface: launch (single-process dev fabric and the multi-process region nodes), the operational API, replication transport facts, failover and partition drills, and the honest write-durability contract.

STATUS: EXPERIMENTAL — TWO MODES, NEITHER IS QUORUM-ACKED HA YET

Cluster mode has two shapes, both behind the same experimental opt-in:

1. Multi-process (--region) — real process isolation. Each tidal-server cluster --region <name> process owns exactly one region: one TidalDb, one GrpcTransport whose server binds this region's grpc_addr and whose peers are every sibling region's real grpc_addr. Processes peer over real gRPC and forward over real HTTP, so a crash of one region's process takes down only that region — the survivors keep serving. This is genuine process (and, across hosts, host) isolation. It is verified end-to-end by the tier-3 suites (cluster_multiproc, cluster_chaos, cluster_lifecycle, cluster_runbook) over real OS processes with real network-partition injection.

2. Single-process (no --region) — the dev/demo default. Every region runs inside one process. Replication still traverses the real tidal-net gRPC transport on loopback (faithful multi-region semantics over a real wire), but there is no process isolation: a crash, OOM, or host failure takes the whole "cluster" down at once. This is a development / staging / demo fabric and a correctness harness for the replication paths — not production HA. It remains the default because it needs no per-region topology addresses and no process orchestration.

Honest remaining limits (both modes):

• Quorum durability is opt-in. The default 204 is leader-durable (storage + WAL fsync; follower ship off the request path). Since m11p3, ack=quorum — topology default or per-request x-tidal-ack header — gates success on a majority of the replica set durably holding the write, surviving permanent leader loss (see §8).
• Leadership is automatic (m11p4). Every node runs a failure detector and a Raft-style election (pre-vote + vote + check-quorum + fenced transfer): kill the leader and the survivors elect a successor — typically under a second with the defaults — with zero operator verbs and zero acknowledged-write loss (see §9.1). /cluster/promote is now a fenced transfer for maintenance/override, not the availability mechanism. election.auto_election: false preserves the pre-m11p4 operator-driven posture.
• Membership is elastic and addresses are DNS names (m11p5). grpc_addr is an advertised hostname or IP (DNS-resolved on every reconnect); nodes join online via --seed and catch up via FetchSnapshot + the StreamSegments stream; add/remove ride kind-4 membership records on the replicated log (see §1b, §3, §6, §9.1).

For a production deployment today, run a single tidal-server standalone node backed by host-level redundancy and disk durability (see kubernetes.md and server-deployment.md), and reach for multi-process cluster mode for read-scale / multi-region deployments whose writes need ack=quorum's failover-survivable contract.

Both modes refuse to start unless you explicitly opt in with either the --experimental-cluster flag or the TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 environment variable. On start each emits a loud, mode-specific WARN restating exactly what it does and does not provide. Do not wire either into a production load balancer.

Scope: what cluster mode does and does not own

tidalDB owns retrieval and ranking only. Cluster mode adds multi-region replication of that, nothing more. It does not generate embeddings, store video/blobs, run a CDN or transcoder, do auth/moderation/payments, or provide a distributed consensus log. Bring your vectors; tidalDB retrieves and ranks over them. See VISION.md for the scope boundary.

Prerequisites

• Rust toolchain ≥ 1.91 if running directly (the Docker build pins rust:1.91-bookworm).
• protobuf-compiler (protoc) and a C++ toolchain (g++) on the build host — tidal-net's build script compiles the WAL-shipping .proto, and USearch's HNSW core is C++. The Docker image installs both.
• Docker 25+ if running via container.
• For single-process mode: one HTTP port (default 9500); each follower region also binds an OS-assigned loopback port for its gRPC replication server unless you pin one in the topology (see §3).
• For multi-process mode: per-region grpc_addr and http_addr declared in the topology, and the ports they name available on each host (see §3).

1. Launch the cluster locally

Cluster mode is gated. Pass --experimental-cluster (or set TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1) or the server exits with a mode-specific error explaining why.

1a. Single-process (dev/demo default)

TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
cargo run -p tidal-server -- \
  cluster \
  --listen 127.0.0.1:9500 \
  --schema tidal-server/config/default-schema.yaml \
  --topology tidal-server/config/default-cluster.yaml \
  --experimental-cluster

The default topology spins up three regions (us-east, eu-west, ap-south) with us-east as leader, all inside one process. On a clean start you will see a WARN line stating this is experimental and single-process, an info line per follower (follower gRPC transport ready), and finally listening on http://127.0.0.1:9500.

1b. Multi-process (one process per region)

Pass --region <name> (or set TIDAL_REGION) to run only that region in this process. The topology must declare a per-region grpc_addr and http_addr for every region (see §3); siblings reach each other over those. Launch one process per region — typically one per host, each with its own --data-dir. --data-dir is REQUIRED in multi-process mode (m11p2): the durable WAL is the replicated log itself — it is what ships to peers and what serves their catch-up streams — so a node without one is rejected at startup:

# Region us-east (the initial leader) on host A
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
tidal-server cluster --experimental-cluster \
  --region us-east \
  --listen 0.0.0.0:9501 \
  --schema /etc/tidal/schema.yaml \
  --topology /etc/tidal/topology.yaml \
  --data-dir /var/lib/tidal/us-east

# Region eu-west on host B
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
tidal-server cluster --experimental-cluster \
  --region eu-west --listen 0.0.0.0:9502 \
  --schema /etc/tidal/schema.yaml --topology /etc/tidal/topology.yaml \
  --data-dir /var/lib/tidal/eu-west

# Region ap-south on host C
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
tidal-server cluster --experimental-cluster \
  --region ap-south --listen 0.0.0.0:9503 \
  --schema /etc/tidal/schema.yaml --topology /etc/tidal/topology.yaml \
  --data-dir /var/lib/tidal/ap-south

Every process parses the same topology file; RegionIds are assigned by declaration order, so all processes agree on the region→id mapping. Each process binds its own region's grpc_addr/http_addr and dials its siblings' addresses. The --listen HTTP address is the public gateway for that region.

Seed-join boot (m11p5 — adding a node without editing the topology)

A node can join an existing cluster online, without appearing in any declared topology, by contacting a running peer as a seed:

TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
tidal-server cluster --experimental-cluster \
  --region eu-2 \
  --listen 0.0.0.0:9504 \
  --schema /etc/tidal/schema.yaml \
  --topology /etc/tidal/topology.yaml \
  --data-dir /var/lib/tidal/eu-2 \
  --seed http://10.0.1.10:9501 \
  --seed http://10.0.2.10:9502 \
  --advertise-grpc eu-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9600 \
  --advertise-http  eu-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9504 \
  --metrics 0.0.0.0:9091

Flag	Meaning
--seed <url> (repeatable)	One or more seed HTTP base URLs. The joiner polls each for the current leader, then POST /cluster/joins through it. Any reachable seed works; list a few for resilience.
--advertise-grpc <host:port>	This node's advertised gRPC address — what siblings dial. Required with --seed (the joiner has no topology entry of its own). DNS-capable.
--advertise-http <host:port>	This node's advertised HTTP gateway — what peers forward writes/status to. Required with --seed.
--metrics <addr>	Prometheus /metrics bind (the joiner has no topology metrics_addr).

What happens: the joiner skips the topology's "every region declared" gate, learns its roster + assigned id + current term from the seed's join response, appends a Learner record to the replicated log (the leader answers only after it is quorum-committed), persists the roster to a durable membership cache and the term to election_state (persist-before-act), installs a snapshot when it is behind the leader's retained WAL, then streams the live tail. The leader's auto-promotion duty flips it Learner → Voter once it is within replication.learner_promote_lag (default 1024) of the flushed frontier. A restart boots from the cache without the seed.

A --seed boot still requires the local topology/config file for the behavioral knob blocks (replication:, wal:, election:, timeouts:, grpc_tls) — its regions: list is ignored for the roster (the join response is the roster), but a bare --seed with neither --topology nor TIDAL_CONFIG refuses to boot naming the rule, so a joiner never silently inherits the compiled-in defaults' wrong ack mode, quorum timeout, or election timing. The k8s manifests mount the shared bootstrap ConfigMap on every pod including N≥3, so this costs nothing there.

Capability gate (m11p5 mixed-version safety). The leader refuses /cluster/join and every conf-change until all current voters report kind-4 capability — see §8's downgrade rule. Complete the binary upgrade before adding or removing nodes.

Auth: set TIDAL_API_KEY=<secret> to require Authorization: Bearer <secret> on the data and /cluster/* mutation routes. If it is unset the server runs UNAUTHENTICATED and logs a WARN — never expose an unauthenticated cluster beyond loopback / a trusted VPC. Health probes and /openapi.json are always unauthenticated. In multi-process mode set the same key on every process: a forwarded/broadcast request passes the caller's Authorization through verbatim, and the internal-propagation marker (x-tidal-internal: 1) is an inter-sibling trust signal, not an auth bypass (the bearer middleware still runs first).

Useful environment variables:

Var	Mode	Effect
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1	both	Opt in to cluster mode (alternative to --experimental-cluster).
TIDAL_REGION	multi-process	Selects the single region this process owns (alternative to --region).
TIDAL_API_KEY	both	Bearer token for protected routes. Unset ⇒ unauthenticated + WARN. Set the SAME key on every region in multi-process mode.
TIDAL_HLC_SKEW_MS	multi-process	Signed ms offset applied to THIS process's HLC. Affects reconcile-time LWW stamping ONLY (not signal-decay timestamps). A test/ops escape hatch for verifying causal convergence under clock skew — do not set it in normal operation.
TIDAL_CONFIG	both	Config dir holding default-schema.yaml / default-cluster.yaml (used when --schema / --topology omitted).
PORT	both	Listen address. A bare port (9500) normalises to 0.0.0.0:9500.
TIDAL_SERVER_LOG	both	tracing filter (default info).

2. Launch via Docker

# Build the image once (build context is the repo root).
docker build -f docker/cluster/Dockerfile -t tidaldb:cluster .

# Run (press Ctrl+C to stop). The image already sets
# TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 so cluster mode starts; it still logs the
# loud experimental WARN. The default CMD is SINGLE-process cluster mode.
docker run --rm -p 9500:9500 tidaldb:cluster

The image bakes the default schema/topology under /etc/tidal-server and uses ENTRYPOINT ["tidal-server"] + a cluster … CMD, so you can override the subcommand (e.g. docker run tidaldb:cluster standalone …), pass --region for a multi-process container, or supply your own config files:

docker run --rm -p 9500:9500 \
  -e TIDAL_API_KEY=changeme \
  -v "$PWD/configs/my-schema.yaml:/srv/schema.yaml:ro" \
  -v "$PWD/configs/my-topology.yaml:/srv/topology.yaml:ro" \
  tidaldb:cluster \
  cluster \
    --listen 0.0.0.0:9500 \
    --schema /srv/schema.yaml \
    --topology /srv/topology.yaml

The container ships a HEALTHCHECK that curls /health. For Kubernetes/Compose deployment patterns see Cross-references.

3. Topology YAML

The topology file declares regions and the leader.

Single-process minimal default (tidal-server/config/default-cluster.yaml):

regions:
  - name: us-east
  - name: eu-west
  - name: ap-south
leader: us-east
# Optional: OS worker threads serving cluster write/heal requests (gRPC ship).
# Defaults to available parallelism when omitted.
# write_workers: 4

Multi-process requires a per-region grpc_addr and http_addr (validated at startup — a missing or syntactically invalid host:port is a hard error naming the region; reachability is deliberately not probed, since siblings boot in any order):

grpc_addr is an advertised address — a literal host:port OR a DNS hostname (m11p5). A new optional per-region grpc_bind controls the local bind independently of what siblings dial; the example below shows the DNS shape (one shared file names every region by its per-pod DNS name while each pod binds 0.0.0.0):

regions:
  - name: us-east
    grpc_addr: "tidaldb-0.tidaldb-peers.svc.cluster.local:9601"  # ADVERTISED (siblings dial; DNS re-resolved on reconnect)
    grpc_bind: "0.0.0.0:9601"      # LOCAL bind (optional; see derivation rule below)
    http_addr: "10.0.1.10:9501"    # this region's public HTTP gateway (forwarding + status)
    metrics_addr: "10.0.1.10:9091" # optional Prometheus /metrics listener (set it in production)
  - name: eu-west
    grpc_addr: "tidaldb-1.tidaldb-peers.svc.cluster.local:9602"
    grpc_bind: "0.0.0.0:9602"
    http_addr: "10.0.2.10:9502"
    metrics_addr: "10.0.2.10:9091"
  - name: ap-south
    grpc_addr: "tidaldb-2.tidaldb-peers.svc.cluster.local:9603"
    grpc_bind: "0.0.0.0:9603"
    http_addr: "10.0.3.10:9503"
    metrics_addr: "10.0.3.10:9091"
leader: us-east
# Optional m11p1 tuning blocks (engine defaults shown):
# replication:
#   batch_max_events: 256   # events coalesced per shipped batch (1-256)
#   window: 4               # in-flight batches per peer (1-64)
#   retry_ms: 100           # backoff before a transiently-failed batch retries
#   catchup_retry_ms: 30000           # backoff before a FAILED catch-up pull re-pulls (m11p4)
#   snapshot_artifact_ttl_ms: 600000  # staged-snapshot reuse window (m11p5; never unpins an active consumer)
#   learner_promote_lag: 1024         # learner→voter promotion distance + readiness hysteresis (m11p5)
#   reseed_self_restart: false        # drain+exit(0) when reseed_required latches (m11p5; k8s sets true)
# wal:
#   batch_size: 100         # events per group-commit fsync (1-256)
#   batch_timeout_ms: 10    # max wait before a partial batch flushes
# election:                 # m11p4 failure-detector / election timers (defaults shown)
#   heartbeat_interval_ms: 300
#   election_timeout_min_ms: 1500     # validated: leader_lease_ms + heartbeat_interval_ms < this
#   election_timeout_max_ms: 3000
#   leader_lease_ms: 900
#   auto_election: true               # false → pre-m11p4 operator-driven failover

grpc_bind derivation (m11p5 §1): grpc_bind present → bind it. Absent + grpc_addr parses as a literal SocketAddr → bind that (today's behavior, byte-for-byte — every existing IP topology keeps working). Absent + grpc_addr is a hostname → bind 0.0.0.0:<port from grpc_addr>. Because the peer-dial path no longer parses grpc_addr as a SocketAddr, hyper re-resolves the hostname on every reconnect — a pod rescheduled onto a new IP is reached with no peer restarts. TLS note: SNI follows the URI host, so DNS peer names require DNS-SAN certs (see grpc_tls below); no code change.

Fields:

Field	Single-process	Multi-process	Meaning
regions[].name	required	required	Region name used everywhere in the HTTP API (?region=, /cluster/promote, etc.). Must be unique.
regions[].grpc_addr	optional	required	This region's advertised gRPC replication address — what siblings dial. Since m11p5 it may be a literal host:port OR a DNS hostname (re-resolved on every reconnect). In single-process mode omit it — the server allocates a free loopback port and self-heals a bind race by retrying on a fresh port. In multi-process mode it is required and is tried exactly once.
regions[].grpc_bind	unused	optional	This region's local gRPC bind host:port (m11p5), independent of the advertised grpc_addr. Omitted: a literal grpc_addr binds itself; a hostname grpc_addr binds 0.0.0.0:<its port>. Set it to bind a specific interface while advertising a DNS name.
regions[].http_addr	unused	required	This region's public HTTP gateway address, used by siblings for write/read forwarding and status aggregation. Unused in single-process mode.
regions[].grpc_tls	unused	optional	TLS material for this region's gRPC transport: ca_cert, server_cert, server_key (PEM paths), plus optional client_cert/client_key for mTLS. Omitted ⇒ plaintext, the right posture for loopback/VPC topologies. DNS grpc_addr requires DNS-SAN certs (SNI follows the dialed hostname).
regions[].metrics_addr	unused	optional	Bind address for this region's Prometheus /metrics listener (m11p1; cluster mode previously had none). Omitted ⇒ no metrics endpoint. Set it in every production topology, and bind it internally — the endpoint is unauthenticated.
leader	required	required	Must name one of the declared regions. The initial write leader.
write_workers	optional	optional	Size of the runtime-free OS-thread pool that admission-controls cluster writes (/signals staging) and runs /cluster/heal's blocking redelivery. Bounded queue ⇒ 429 backpressure. Must be ≥ 1 when given.
timeouts.broadcast_peer_secs	unused	optional	Per-peer budget (seconds) for the /cluster/promote fan-out (the only remaining peer fan-out — the m11p2 log replaced the item/embedding broadcast). Default 2s — right for loopback/VPC; raise it for WAN topologies where a distant region cannot answer in 2s. Must be ≥ 1 when given.
replication.batch_max_events	unused	optional	Max relay events coalesced into one shipped batch (1–256, the WAL wire-format ceiling). Default 256.
replication.window	unused	optional	In-flight batches per peer (1–64). 1 = strictly in-order shipping; higher pipelines across the peer RTT (out-of-order arrivals park gap-aware on the receiver). Default 4.
replication.retry_ms	unused	optional	Backoff (ms) before a transiently-failed batch ship retries. Default 100.
replication.ack	unused	optional	Deployment-default write acknowledgment: leader (default) or quorum (majority-durable — see §8). Per-request override: the x-tidal-ack header.
replication.quorum_timeout_ms	unused	optional	Budget an ack=quorum write waits for the commit index before the retryable 503 naming the laggards. Default 2000.
replication.catchup_retry_ms	unused	optional	Backoff (ms) before a FAILED catch-up pull re-pulls on a timer (m11p4) — lets an idle cluster self-heal a follower whose pull failed during a rolling restart. Default 30000. Must be ≥ 1 when given.
replication.snapshot_artifact_ttl_ms	unused	optional	Staged-snapshot reuse window (m11p5), counted from the last fetch's completion. Governs artifact REUSE only — never unpins the WAL retention of an active consumer; a hard cap (4×) force-drops a never-releasing pin (tidaldb_cluster_snapshot_pin_force_drops_total). Default 600000 (10 min). Must be ≥ 1.
replication.learner_promote_lag	unused	optional	Learner→voter promotion distance AND the readiness-convergence hysteresis threshold (m11p5), in events. Default 1024. Must be ≥ 1.
replication.reseed_self_restart	unused	optional	Drain + clean-exit(0) once the durable reseed_required marker latches (m11p5; k8s sets it true). Refused when the remaining voters can't sustain quorum without this node. Default false.
election.heartbeat_interval_ms	unused	optional	Leader heartbeat interval (m11p4). Default 300.
election.election_timeout_min_ms / ..max_ms	unused	optional	Randomized follower election timeout window (m11p4). Defaults 1500 / 3000. Validated: leader_lease_ms + heartbeat_interval_ms < election_timeout_min_ms.
election.leader_lease_ms	unused	optional	Leader freshness lease — a leader that loses majority contact steps down within it (m11p4). Default 900.
election.auto_election	unused	optional	true (default) = automatic failover; false = pre-m11p4 operator-driven posture (no auto elections, no check-quorum step-down).
wal.batch_size	optional	optional	Events per WAL group-commit fsync (1–256). Default 100.
wal.batch_timeout_ms	optional	optional	Max ms a partial group-commit batch waits before flushing. Default 10. Tune against the measured tidaldb_cluster_wal_fsync_us on the deployment's volume.

The schema YAML (signals / text_fields / embedding_slots / profiles) is the same format the standalone server loads — see API.md and QUICKSTART.md for the full schema/profile grammar and the built-in ranking profiles. Topology only adds the region layout above.

Personalization correctness note. Preference-vector personalization (the taste vector that powers for_you) only updates for signals declared positive_engagement: true in the schema. The cluster /signals route writes global signals (no user_id / creator_id context), so it never folds item embeddings into a user's taste vector regardless of the positive_engagement flag. Personalized writes go through the embedded engine (signal_with_context), not this HTTP route. The exception is /hardnegs, which records a user-scoped hide that converges across regions via /cluster/reconcile.

4. gRPC replication transport (tidal-net)

Replication between regions is the real tidal-net WalShipping gRPC service, not in-process channels. The facts you need to operate and firewall it:

Property	Value	Notes
Service	WalShipping (proto tidal.replication.v1)	RPCs: ShipSegment (unary; the live-tail push path), StreamSegments (server-streaming; the follower-pulled catch-up path since m11p2), FetchSnapshot (server-streaming; the m11p5 snapshot transfer for joiners + reseeds), JoinCluster (m11p5 conf-change), and Heartbeat (ControlPlane health).
Bind address	per-region grpc_addr (multi-process); an auto-allocated loopback port (single-process)	Default dev band 59520–59529 if you pin one.
Transport security	mTLS via TlsConfig (ca_cert, server_cert, server_key, optional client_cert / client_key)	insecure = true (plaintext) is the loopback/VPC default in this phase. A peer with no TLS config must set insecure.
Max payload	64 MiB	Both encoder and decoder codec limits are raised to this; pinned by a compile-time assert to the engine's InProcessTransport limit so both ends agree. WAL segments default to 16 MiB.
Circuit breaker	per-peer, threshold 5, reset 30s	After 5 consecutive ship failures the breaker opens; it stays open for 30s, then the next attempt probes (HalfOpen → Closed on success, Open again on failure). Backpressure (channel full) does not trip it. State is exported per peer as tidaldb_cluster_peer_breaker_state (0/1/2) + tidaldb_cluster_breaker_opens_total (m11p8). Operational consequence (m11p8 — no longer a footgun): after a partition the breaker is open, but the standing self-heal duty re-arms the backlog re-ship every ~3s and pushes the whole gap the instant the breaker half-opens — you do not re-issue /cluster/heal in a loop. Watch tidaldb_cluster_healing_peers → 0 (see §6 and the drills in §10).
Timeouts (defaults)	connect 5s, request 10s, keep-alive PING every 10s / 5s ACK	A blackholed peer fails fast instead of stalling the single-threaded shipper.

ShipSegment carries the WAL segment id, BLAKE3-validated payload bytes, event count, and the leader's authoritative leader_last_seq (so a follower can advance its replication-lag high-water-mark even for an all-local segment that filters to empty). The follower's segment-receiver thread drains and applies it on arrival. The engine's applied high-water-mark is a contiguous frontier with a bounded ahead-buffer, so out-of-order eager ships can never swallow a sequence gap (a silent-data-loss class fixed in m8p10).

Operational guard: the gRPC replication ports and the Prometheus /metrics endpoint are UNAUTHENTICATED. Bind them to loopback or a cluster-internal network only — never the public interface.

5. Core HTTP API

All routes are JSON unless noted. Examples assume BASE=http://localhost:9501 (any region's gateway in multi-process mode; the single --listen address in single-process mode) and, when TIDAL_API_KEY is set, AUTH='-H "Authorization: Bearer $TIDAL_API_KEY"'. Drop the -H header when running unauthenticated.

Middleware mirrors standalone: 30s request timeout (408), 100 max in-flight (429), 2 MB body limit (413), and an x-request-id on every response. Health probes sit outside the load-shedding stack so liveness/readiness are never queued or timed out under saturation.

Multi-process routing (one coherent surface). Any region's gateway accepts any operation and routes it to the node that owns it:

• Writes (/signals, /items, /embeddings, /hardnegs) on a non-leader forward to the leader (the caller's Authorization passes through); the client sees the leader's status/body. A leader that is unreachable degrades to a 503 naming the leader, never a hang.
• Reads (/feed, /search) default to the LOCAL region. This is the key difference from single-process mode, whose default read is the leader. A ?region=<other> read forwards to that region's process.
• /cluster/promote, /cluster/partition, /cluster/heal route to / fan out from the leader as documented in §6.

Health

curl "$BASE/health"          # 200 ok / 503 while draining; reports mode, region, leader
curl "$BASE/health/startup"  # always 200
curl "$BASE/health/live"     # always 200
curl "$BASE/openapi.json"    # served OpenAPI 3.1, UNAUTHENTICATED — canonical HTTP reference

/health returns { "ok": true, "service": "tidaldb", "mode": "cluster", "region": "us-east", "leader": "us-east", ... }. The /openapi.json document is the machine-readable source of truth for the data, /cluster/*, /sharded/*, and /hardnegs request/response shapes (the health probes are intentionally outside the documented API surface).

Register items & embeddings (one replicated log — m11p2)

Items and embeddings ride the same replicated WAL stream as signals: the leader journals each write as a kind-1 (item metadata) or kind-2 (embedding) WAL record BEFORE touching storage, and followers apply it from the log — live pushes for the hot tail, the StreamSegments catch-up stream for history. There is no HTTP broadcast anymore (the m8p10 side channel and its bug classes were deleted in m11p2), so the responses are plain statuses:

curl -X POST "$BASE/items" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "metadata": { "title": "Jazz Piano", "category": "music" } }'
# → 201 Created (item durably journaled into the replicated log + applied)

curl -X POST "$BASE/embeddings" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "values": [0.1, 0.2, 0.3, 0.4] }'
# → 204 No Content

The 201/204 asserts leader durability and carries x-tidal-seq (the record's seqno in the replicated log); x-tidal-ack: quorum upgrades it to a majority-durable ack (see §8). The record is fsynced into the stream every follower receives (live push, or pull-based catch-up after downtime — see §6 heal). A write to a non-leader forwards to the leader transparently (the ack header and seq header travel through the forward). A down/partitioned peer needs no backfill bookkeeping: it converges from the log when it returns.

Embeddings: tidalDB does not generate vectors — the caller brings them. The write L2-normalizes and inserts into the HNSW index. Dimensions are strict: they must equal the slot's declared dimensions (min 2, max 4096) or the insert is rejected with a 500 (the engine surfaces a dimension mismatch as an internal error, not a 400 — a known wart, tracked post-M8); zero-norm vectors are also rejected (500). RETRIEVE / SEARCH route through the first declared embedding slot only; multi-modal apps must fuse offline or use separate entity kinds.

Record signals (cluster /signals = global only)

curl -X POST "$BASE/signals" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "signal": "view", "weight": 1.0 }'
# → 204 No Content (see the durability contract in §8)

The signal name must be declared in the schema (an undeclared name returns 400 naming it). The 204 carries x-tidal-seq (the write's replicated-log seqno), and x-tidal-ack: quorum upgrades it to a majority-durable ack — see §8. This route records a global signal on the leader and ships it to followers over the replicated WAL stream; it does not personalize (see the personalization note in §3). On a non-leader gateway it forwards to the leader transparently and still returns 204.

Hard negatives (user-scoped hides)

curl -X POST "$BASE/hardnegs" \
  -H 'Content-Type: application/json' \
  -d '{ "user_id": 42, "item_id": 7 }'
# → 204 No Content

Records a user-scoped hide on the leader (a non-leader gateway forwards). The item is filtered from that user's /feed?user_id=42. Hard negatives converge across regions via the LWW-resolved /cluster/reconcile CRDT path — NOT the signal WAL relay and NOT a broadcast.

Retrieve and search (region-pinned reads)

# Default read region is the LOCAL region (multi-process mode).
curl "$BASE/feed?user_id=42&profile=for_you&limit=20"
curl "$BASE/search?query=jazz%20piano&user_id=42&limit=5"

# Pin a read to a specific region. The gateway forwards to that region's process.
# Followers may lag the leader (and lag jumps during a partition) — use this for
# canary reads and lag verification.
curl "$BASE/feed?profile=trending&region=eu-west"

?region= accepts any declared region name; an unknown name returns 400. Omit it and the read serves from the LOCAL region (multi-process) / the current leader (single-process). limit is clamped to 1000 at the trust boundary (a larger value cannot amplify memory unboundedly). The for_you profile applies the built-in diversity defaults (max_per_creator=2, format_mix_max_fraction=0.4, exploration=0.1).

6. Cluster management API

Check cluster status

Two views. /cluster/status/local reports THIS node's own replication/leadership state (no peer calls); /cluster/status aggregates EVERY region (the gateway calls each peer's /cluster/status/local concurrently with a tight per-peer budget).

curl "$BASE/cluster/status/local" | jq

{
  "region": "ap-south",
  "is_leader": false,
  "leader": "us-east",
  "last_seq": 0,
  "applied_events": 124,
  "lag_events": 1,
  "partitioned": [],
  "reachable": true
}

curl "$BASE/cluster/status" | jq

{
  "leader": "us-east",
  "relay_log_len": 125,
  "regions": [
    { "name": "us-east",  "applied_events": 125, "lag_events": 0, "partitioned": false, "reachable": true },
    { "name": "eu-west",  "applied_events": 125, "lag_events": 0, "partitioned": false, "reachable": true },
    { "name": "ap-south", "applied_events": 124, "lag_events": 1, "partitioned": false, "reachable": true }
  ]
}

relay_log_len is the leader's high-water-mark (last_seq); each region's lag_events is relay_log_len − applied_events (saturating). Since m11p3 /cluster/status/local also reports the node's ack default and, on the leader, commit_index — the highest seqno a majority of the replica set durably holds (last_seq − commit_index is the quorum lag). A region the gateway cannot reach within the per-peer budget is reported honestly as reachable: false, partitioned: true, applied_events: 0, and worst-case lag (lag_events == relay_log_len). A non-zero, growing lag on a reachable region is the signal that it is partitioned (ship-skip) or its segment-receiver is wedged.

Promote a new leader

curl -X POST "$BASE/cluster/promote" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "eu-west" }'
# → { "ok": true, "leader": "eu-west", "acked": ["eu-west","ap-south"], "failed": [] }

The gateway first resolves the TARGET's stream baseline — the new leader's WAL flushed frontier at promotion, persisted in its data_dir/stream_baseline — then fans the promote out to every peer carrying it (internal marker, so each applies locally and does not re-fan). Peers jump their applied frontier for the new leader's shard to the baseline: everything at or below it is pre-stream history (replicated applies of the OLD stream), not data. The response carries baseline plus the acked/failed fan-out report; a peer in failed (e.g. a dead old leader) self-corrects — its first parked batch from the new stream triggers a catch-up pull whose chunks announce the baseline. After promotion /cluster/status reports the new leader; new writes route there; the new leader's ship queue activates and the demoted node's deactivates. An unknown region returns 400. Since m11p4 promote is a fenced transfer (it drains the target, then sanctions an election), not the availability mechanism — automatic failover handles a dead leader with no operator action (§9.1); this verb is for maintenance and deliberate successor choice (§9.2).

Simulate a partition & heal

There are two ways to partition a region; the runbook drills demonstrate both (§10):

1. Simulated ship-skip flag — /cluster/partition tells the leader to stop shipping to the named region (no sockets touched). Good for a controlled, reversible lag demo.
2. Real network partition — sever the actual TCP path between processes (firewall / proxy). The chaos suite uses a root-free in-harness TCP relay; operators can use iptables/pfctl (see §10).

# Simulated: isolate ap-south — leader ships skip this follower, so its lag climbs.
curl -X POST "$BASE/cluster/partition" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "ap-south" }'
# → { "ok": true, "partitioned": "ap-south" }

# Heal: resume shipping past everything the follower reports applied, and
# nudge it to PULL any history that rotated out of the leader's ship tail
# (the single recovery verb — log catch-up IS the full heal since m11p2).
curl -X POST "$BASE/cluster/heal" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "ap-south" }'
# → { "ok": true, "healed": "ap-south" }

/cluster/heal is the single recovery verb, and since m11p2 it is pure log catch-up: it clears the partition flag, resumes the ship queue past the follower's reported applied seqno (retries of data the follower already holds prune automatically — every ship ack piggybacks the follower's applied seqno), and nudges the follower to pull anything older than the leader's in-memory ship tail via its StreamSegments catch-up stream over the leader's durable WAL segments (POST /cluster/catchup, internal; the nudge forwards the healing operator's own bearer credential). Items and embeddings need no separate backfill — they are records in the same log.

Convergence is self-driving (m11p8 — the operator no longer loops heal). The per-peer ship senders retry parked batches every replication.retry_ms (default 100ms); a follower that detects a gap pulls the catch-up stream itself (also on boot, so a restarted node converges with no operator action at all — the tier-3 mp_items_ride_the_log_and_catchup_stream proves it). On top of that, a standing leader heal duty runs every ~3s: for any non-partitioned peer whose ship breaker is open AND that trails the leader, it re-arms the backlog re-ship from the peer's durable mark, so the moment the breaker half-opens (threshold 5, reset 30s — see §4) the leader pushes the WHOLE gap. This closes the old footgun: you no longer re-issue /cluster/heal until lag 0 — the server drives it. Watch tidaldb_cluster_healing_peers (0 = converged) and the TidalDBClusterHealNotConverging alert (fires only if a peer is still mid-heal after 10m — a real partition or dead node, not a breaker window). /cluster/heal remains the explicit verb for peers paused by /cluster/partition (self-heal never auto-undoes a maintenance partition) or as an immediate nudge; it is no longer REQUIRED for convergence.

Reconcile (cross-region CRDT convergence)

curl -X POST "$BASE/cluster/reconcile" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "ap-south" }'
# → { "ok": true, "region": "ap-south",
#     "local_elapsed_ms": 0, "remote_elapsed_ms": 1, "ops_applied": 3 }

/cluster/reconcile exchanges a CRDT state snapshot with the target region: this node ships its snapshot into the target's merge AND applies the target's pre-merge snapshot back, so both sides converge to identical state by deterministic LWW. This is how hard negatives (recorded via /hardnegs) converge across regions — they do not ride the WAL relay. local_elapsed_ms / remote_elapsed_ms are the merge+apply times each side measured (not the HTTP round-trip); both are typically 0–1ms. Reconcile is idempotent: a repeat reconcile of already-converged regions is an exact no-op on scores (no drift). /cluster/reconcile/snapshot is the INTERNAL snapshot-exchange leg this verb drives (marker required); operators never call it directly.

Membership verbs (m11p5 — online add / remove / inspect / reseed)

Membership is data on the replicated log (kind-4 records, latest wins, folded into a ClusterMembership cell). All four verbs route to / are answered by the leader and are quorum-commit-gated, one change at a time. The leader refuses every conf-change until all current voters report kind-4 capability (the mixed-version safety gate — complete the binary upgrade first).

# Add a node (idempotent by name): forwards to the leader, which assigns the
# next id, appends a Learner record, and answers AFTER quorum-commit.
curl -X POST "$BASE/cluster/join" \
  -H 'Content-Type: application/json' \
  -d '{ "name": "eu-2",
        "grpc_addr": "eu-2.tidaldb-peers.svc.cluster.local:9600",
        "http_addr": "eu-2.tidaldb-peers.svc.cluster.local:9504" }'
# → { "id": 4, "role": "learner", "term": 7, "leader": "us-east", "members": [ … ] }

The joiner normally seed-joins itself (§1b); this verb is the same conf-change for tooling. A re-join from a known name returns its existing id and current role and appends nothing. The leader's auto-promotion duty (a standing duty, re-armed on every activation and membership apply — it survives the joining-era leader's death) flips the learner to Voter once it is within replication.learner_promote_lag of the flushed frontier; promotion_pending (lag=N) in /cluster/status/local makes a stuck scale-up diagnosable.

# Inspect the applied roster (ids, names, addresses, roles, conf version).
curl "$BASE/cluster/members" | jq

# Remove a node: appends a Removed tombstone (quorum-commit-gated). The peer's
# ship cell is retired only AFTER the record is delivered-to/acked-by the removed
# peer (bounded give-up → tidaldb_cluster_remove_delivery_giveups_total); the
# removed node's readiness flips to 503 and it stops campaigning. Its id is BURNED
# (never renumbered, never reused).
curl -X POST "$BASE/cluster/members/remove" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "eu-2" }'
# → { "removed": "eu-2", "membership_version": 9 }

# Force a reseed on demand: latches the durable reseed_required marker. The node
# keeps serving degraded (voting enabled) and reseeds via snapshot on its NEXT
# boot (or self-restarts if replication.reseed_self_restart is true and quorum
# can be sustained without it). See §9.1.
curl -X POST "$BASE/cluster/reseed"
# → { "reseed_required": true }

Scale-down order (decommission): call /cluster/members/remove first (so the cluster's quorum math shrinks before the node disappears), wait for the record to quorum-commit, then stop / delete the node — lowest ordinal last under a StatefulSet.

7. Sharded scatter-gather API

The /sharded/* routes hash-partition entities across regions (hash(entity_id) % num_shards, using the engine's own ShardRouter so writes and reads never disagree). Writes route to the single owning region; reads fan out to all regions and K-way merge by score.

Write routes (each routes to the owning region; a non-owner gateway forwards):

curl -X POST "$BASE/sharded/items"      -d '{ "entity_id": 7, "metadata": { "title": "..." } }'  # 201
curl -X POST "$BASE/sharded/embeddings" -d '{ "entity_id": 7, "values": [0.1,0.2,0.3,0.4] }'      # 204
curl -X POST "$BASE/sharded/signals"    -d '{ "entity_id": 7, "signal": "view", "weight": 1.0 }'  # 204

Read routes (scatter-gather across all regions):

curl "$BASE/sharded/feed?profile=for_you&limit=20&deadline_ms=50"
curl "$BASE/sharded/search?query=jazz&limit=10&deadline_ms=100"

Each sharded read accepts an optional deadline_ms — the total scatter budget (default 50ms, server-clamped to 10s). The per-shard deadline is deadline_ms − 5ms network overhead. The response includes a scatter_gather block:

{
  "items": [ /* merged, deduped, diversity-enforced, re-ranked */ ],
  "total_candidates": 4210,
  "scatter_gather": {
    "degraded": false,
    "shards_queried": 3,
    "elapsed_ms": 12,
    "shard_deadline_ms": 45
  }
}

Degraded semantics: a shard that is partitioned, errors, or misses the deadline is reported in unavailable_shards (a name list) and flips degraded: true — it is never silently dropped, and the read still returns 200 with the live shards' results. The merge dedups replicated copies of an entity (keeping the best-scoring copy), reconciles total_candidates so replicated shards are not counted multiple times, and re-enforces max_per_creator across the merged set.

8. Write-durability contract: the ack knob and its cost (m11p3)

Every replicated write (/signals, /items, /embeddings) runs under one of two acknowledgment modes. Pick the deployment default with the topology's replication.ack; any caller overrides per request with the x-tidal-ack header (leader or quorum; anything else is a 400).

	ack=leader (default)	ack=quorum
Success means	Durable on the leader (storage + WAL group-commit fsync)	Durable on a majority of the replica set (leader + floor(n/2) followers, each storage-applied + own-WAL-fsynced)
Survives	Any follower failure; leader restart (WAL replay)	Any single node's permanent loss, including the leader's — promote the max-applied survivor and every acked write is there (the 167-kill-point ledger gate proves it)
Does NOT survive	Permanent leader loss before followers caught up (the un-shipped tail dies with it)	Simultaneous majority loss
Latency cost	One group-commit fsync (~ms-scale; the macOS F_FULLFSYNC tail is the local floor)	+ one ship RTT + the follower's group-commit fsync, pipelined: acks are batch-level, so concurrent writers share the round trip the same way they share fsyncs
Failure mode	5xx only for local faults	Additionally a retryable 503 when the quorum budget (replication.quorum_timeout_ms, default 2000) expires — the JSON body names the laggards, the commit_index, and needed/confirmed counts
Availability	Unaffected by follower outages	Blocks when a majority is unreachable. In a 2-region cluster quorum = leader + THE follower: one follower outage stops all quorum writes (by design — that is what the contract says). 3+ regions tolerate floor((n-1)/2) follower outages

Mechanics, in one paragraph: there is one replicated log (the leader's WAL — m11p2); peers only ever receive fsynced batches by construction. Since m11p3 every follower pushes its durably-applied frontier back to the leader once per apply round (ReportApplied — batch-level, decoupled from ship acks, flowing even when the follower converges by catch-up pull or the leader is quiet); ship acks additionally carry the same frontier as an instant floor hint. The leader folds both into per-peer durable marks; the commit index is the k-th largest mark (k = floor(n/2)), and ack=quorum responses gate on it passing the write's seqno — awaited asynchronously, so quorum waiters never hold threads or starve completions. The index is leadership-scoped: promote resets it to the new stream baseline, and a demoted leader fails its in-flight quorum waiters (it must never claim quorum for a stream it no longer owns).

Every cluster write's success response carries x-tidal-seq — the write's seqno in the replicated log (relayed through gateway forwards). Persist it if you need an exact durability cursor: commit_index >= seq on /cluster/status/local is "this write is majority-durable", regardless of which mode acked it. (The rare dedup-suppressed signal write — an identical event within the WAL's ~60s content-hash window is already durably logged, so no new record exists to track — carries x-tidal-deduplicated: 1 instead, relayed through forwards like the seq header.)

Retry semantics under ack=quorum — read this twice. A quorum-timeout 503 means not confirmed in budget, not not written: the write is in the leader's log and usually commits moments later. Retries are therefore at-least-once:

• /items and /embeddings retries are always safe — idempotent upserts keyed by entity_id.
• /signals retries can double-count the signal's weight when the original did commit (the server stamps each request's timestamp, so the WAL's content-hash dedup window cannot identify a client retry). The distortion is one extra decaying signal per retried timeout — bounded by your retry rate (tidaldb_cluster_quorum_timeouts_total is exactly that budget). Accounting that cannot tolerate it should route through session writes (which carry idempotency keys) or dedup client-side on its own key.
• The laggard names in the 503 are your runbook pointer: a persistent laggard is a down/partitioned region — heal it (§6) or accept leader-ack for the duration (x-tidal-ack: leader).

Rolling upgrades into m11p3 — upgrade the leader first. A pre-m11p3 leader neither serves the ReportApplied RPC nor recognizes x-tidal-ack: it silently applies leader-ack semantics to a quorum request — a durability downgrade the caller cannot see. Upgrade order:

1. Promote leadership off the leader node if needed, upgrade it, promote it back (or simply upgrade the standing leader per §9's restart procedure). From this moment ack=quorum is honored: the commit index rides the m11p2 ship-ack floor hints from not-yet-upgraded followers (correct, just laggier).
2. Upgrade followers one at a time. Each upgraded follower starts pushing ReportApplied and quorum freshness returns to batch-level. (An upgraded follower reporting to a still-old leader is harmless — the report is refused and logged, replication and heal are unaffected.)

Until step 1 completes, treat the cluster as ack=leader-only — do not point ack=quorum traffic at it expecting majority durability.

Other facts unchanged from m11p1/p2: the success code never waits on shipping for ack=leader (sender threads push the WAL flush feed's tail off the request path; replication.batch_max_events/window tune it); a failed fsync errors that write and nothing unfsynced can ship; long-outage data is follower-pulled via StreamSegments (or, when it has rotated past the leader's retained WAL, via the m11p5 FetchSnapshot snapshot transfer); a leader crash is an automatic failover since m11p4 (the survivors elect the up-to-date successor — see §9.1; the vote restriction only elects a node whose log covers every quorum-acked write, which is what makes the zero-acked-loss guarantee hold).

In short: ack=leader = leader durability. ack=quorum = failover-survivable durability, priced at one pipelined replication round trip and majority availability.

WAL segment format across upgrades (m11p4). Segment files carry an 8-byte version header (TSEG + version byte; headerless pre-m11p4 files stay readable — no migration). Three behaviors follow:

• Unreadable segments fail the boot, loudly. A node whose WAL dir holds segments written by an incompatible tidalDB version (or stray/foreign .seg files) refuses to start with WAL segment format unknown: <path> instead of booting with the data invisibly absent (segments=0 — the 2026-06-11 p3 rollout failure mode). Remedy: run a compatible binary, or reseed the node (delete its PVC and let it pull from the leader).
• Unservable catch-up resolves via snapshot transfer (m11p5). A leader that cannot serve a follower's requested range from its retained WAL answers the StreamSegments pull with FAILED_PRECONDITION — "segments not available from seq N; snapshot required", carrying the typed trailer x-tidal-catchup: snapshot-required. Since m11p5 the follower latches a durable reseed_required marker (only on that typed trailer — an ordinary election term-mismatch never latches it fleet-wide) and reseeds itself via the FetchSnapshot snapshot stream on its next boot (or self-restarts if replication.reseed_self_restart is set) — no operator verb, no wipe_data_dir. See §9.1. The marker is surfaced in /cluster/status/local (reseed_required: true) and the tidaldb_cluster_reseed_required gauge.
• Failed pulls self-heal on a timer. A catch-up pull that fails (e.g. the leader's gRPC server not yet ready during a rolling restart) retries every replication.catchup_retry_ms (default 30000) without waiting for a write to re-expose the gap — an idle cluster no longer strands lagged followers.

Downgrade hazard (m11p4): a pre-m11p4 binary reading a header-bearing segment treats the header as a torn tail and may truncate the final segment. Downgrading across the m11p4 boundary requires reseeding the node's WAL.

Membership records + capability gate (m11p5). A kind-4 membership record is a new WAL blob kind. A kind-4 record shipped to a pre-p5 follower is an unknown batch kind → WalError::Corruption → that follower's torn-state receiver halt, permanent across restarts (boot self-heal re-pulls the same record). Both followers halted = a quorum-write outage. To make this structurally impossible, HeartbeatResponse/ReportApplied carry a capabilities bit-field (proto3 zero-default = pre-p5 = incapable) and the leader refuses /cluster/join and every conf-change until all current voters report kind-4 capability — so the first conf-change cannot fire mid-upgrade. Downgrade rule (kind-3 precedent verbatim): once any kind-4 record is in a node's WAL, downgrading it below p5 requires a reseed. Complete the binary upgrade across all voters before adding, removing, or replacing a node.

9. Failover (multi-process)

9.1 Automatic failover (m11p4 — the default)

"A machine died" is a non-event. Every node runs a failure detector (leader heartbeats every election.heartbeat_interval_ms, default 300) and a Raft-style election (pre-vote + vote, randomized election.election_timeout_{min,max}_ms, default 1500–3000). Kill the leader and the survivors elect a successor — typically in under one second with the defaults, bounded well inside 10s — with zero operator verbs and zero acknowledged-write loss (the vote restriction only elects a node whose log covers every quorum-acked write; the tier-3 cluster_election.rs::mp_auto_failover_writes_resume_zero_acked_loss gate proves it across repeated random kill points under ack=quorum load).

What the operator sees:

• /cluster/status/local carries term (the election term, 0 = the pre-election "topology era"), role (leader/follower/pre-candidate/ candidate) and quarantined.
• During the brief leaderless window, writes return a retryable 503 naming the election (leader: "none (election in progress)" plus the responding node's term); clients retry and land on the new leader.
• A restarted ex-leader can never re-claim leadership from its topology file: its durable election state (data_dir/election_state) boots it as a follower, and every replication RPC is term-fenced — a deposed leader's ships, heartbeats and frontier reports are rejected until it rejoins the current term (the §1.4-1 split-brain incident is closed by construction; proven by mp_fenced_ex_leader_restart_cannot_write).
• A leader that loses contact with a majority steps down within election.leader_lease_ms (default 900) and stops accepting writes: ack=leader writes during a minority partition are bounded by the lease, and ack=quorum writes were never at risk.

Divergent suffix / quarantine → automated reseed (m11p5). A node that held leader-acked (never quorum-acked) writes when it died can rejoin into a cluster that elected past them. It detects this at term-join and quarantines: it serves status (quarantined: true, metric tidaldb_cluster_divergence_quarantined) and keeps voting, but refuses the data plane. Since m11p5 recovery is automatic and full-history — no wipe_data_dir: the quarantine latches the durable reseed_required marker, the node reseeds via the FetchSnapshot snapshot stream on its next boot (or self-restarts if replication.reseed_self_restart is set and quorum can be sustained without it), rejoins clean, and the divergence gauge clears. This is strictly leader-ack-only data, within the documented ack=leader crash contract (§8). The tier-3 cluster_reseed.rs proves the full quarantine → marker → restart → reseeded → gauges-cleared loop.

The three-way term-join rule (m11p5). The divergent-suffix check above is one of three outcomes a node reaches when it joins a new term, comparing its own stream position against the leader's election-time position (prev_log, carried on the heartbeat):

• own > prev_log → divergent suffix → quarantine (above);
• own < prev_log → the node is genuinely missing committed-era history that the new stream's baseline jump would silently skip (the p4 carried hazard) → it latches reseed_required and reseeds via snapshot (no silent gap);
• own == prev_log (or a within-term rejoin) → clean, catch-up via the stream.

A snapshot-installed node always joins clean by construction (its WAL is the leader's copy, so its tail term equals the leader's).

To run the pre-m11p4 posture (operator-driven failover, no automatic elections, no check-quorum step-down), set in the topology:

election:
  auto_election: false

9.2 Manual promote: a FENCED TRANSFER (maintenance / override)

POST /cluster/promote { "region": "eu-west" } remains the maintenance verb — but it is now a fenced leadership transfer, not a view flip:

1. With a live leader: the leader waits for the target to hold the full flushed prefix (the catch-up wait IS the drain), then sanctions an immediate election (TimeoutNow); the target wins term+1 and the old leader steps down on first higher-term contact. Response: { ok, leader, term, transfer: "elected" }.
2. With a dead leader: the target campaigns among the survivors directly — the manual override of the automatic path (and the way to choose the successor).
3. The election can refuse a target that lags (its log loses the up-to-date comparison — e.g. the survivor you sampled fell behind between your status read and the vote). The verb 503s naming the cause; promote the other survivor. You can no longer accidentally promote a node that would discard acked data — the m11p3 "max-applied survivor" operator rule is now enforced by the protocol.
4. Promoting the node that already leads is a no-op 200.

The legacy term-0 fan-out promote survives only for clusters that have never elected (mixed-version rollouts mid-upgrade, and the deliberate isolated-node override used by the chaos drills); the first joined election permanently retires it on each node.

Rolling upgrade m11p3 → m11p4: upgrade ALL binaries before relying on auto-failover (pre-m11p4 peers answer vote RPCs with Unimplemented, so no election can reach quorum until a majority is upgraded — the cluster simply keeps its m11p3 behavior until then). The first ELECTED leader journals a kind-3 term-marker WAL record; pre-m11p4 binaries cannot decode it, so do not downgrade a node after the first election without reseeding.

10. Partition drill (multi-process)

The rewritten drill demonstrates both partition mechanisms, exactly as cluster_runbook.rs::runbook_s10_partition_drill scripts them.

1. Baseline. GET /cluster/status; all lag_events: 0, partitioned: false, reachable: true.
2. Inject. Use one of:
   • Real network partition — sever the TCP path peers use to reach the region. The chaos suite (cluster_chaos.rs) uses a root-free in-harness TCP relay proxy (the ROADMAP-sanctioned toxiproxy-style alternative). On a real host you can instead use iptables (Linux) or pfctl (macOS), e.g. iptables -A INPUT -p tcp --dport 9603 -j DROP to blackhole ap-south's gRPC port from a peer. A real cut shows reachable: false in the aggregate status.
   • Simulated ship-skip flag — POST /cluster/partition { "region": "ap-south" } (→ { ok, partitioned: "ap-south" }). The leader stops shipping to ap-south without touching sockets; the aggregate status shows partitioned: true.
3. Write through it. Send several POST /signals. Each must still 204 (the leader-durable contract holds). Watch ap-south's lag_events climb while its applied_events stalls — the leader's ships to it are dropped.
4. Read the stale follower. GET /feed?region=ap-south (or read ap-south's gateway directly) returns the pre-partition view — eventual, not strong, read consistency. The operator console survives a real partition because clients talk to each region's gateway directly.
5. Scatter-gather degradation. While partitioned, GET /sharded/feed returns 200 with degraded: true and ap-south in unavailable_shards — never an error, and the live shards' items are still returned.
6. Heal. Clear the cut (heal the proxy / drop the firewall rule, or just call heal for the simulated flag), then POST /cluster/heal { "region": "ap-south" }. Re-issue heal until /cluster/status shows ap-south at lag_events: 0 — the gRPC circuit breaker opened during the partition (threshold 5, reset 30s), so the first heal may ship into an open breaker and no-op. This is genuine production behavior, not a flag.
7. Verify convergence. GET /cluster/status shows ap-south lag_events: 0, partitioned: false, reachable: true; /sharded/feed is no longer degraded; feed scores on ap-south match the leader to within float tolerance (no loss, no duplication).

11. Shutdown

Send SIGTERM (or SIGINT / Ctrl+C, or stop the container). The server flips readiness to 503 (so a load balancer stops routing to it), drains in-flight requests, then drops the region's TidalDb shutdown path: checkpoint in-memory signal state → flush storage → write the WAL checkpoint marker + fsync → join the WAL, sweeper, checkpoint, text-syncer, and replication-receiver threads. The drop is idempotent; the process exits 0. You will see region cluster node shutdown: database closed (checkpoint + WAL fsync) (multi-process) / cluster shutdown: closing all nodes (checkpoint + WAL fsync) (single-process).

On restart with the same --data-dir, the region recovers its pre-shutdown state from the WAL (verified by cluster_runbook.rs::runbook_s11_shutdown_and_wal_recovery: SIGTERM → exit 0 → restart → the same items are served). In multi-process mode, the restarted node rejoins the cluster and pulls anything it missed while down via its boot-time StreamSegments catch-up request — no operator verb needed (m11p2; /cluster/heal remains the explicit lever for paused peers). This is the per-node step of a rolling upgrade (promote leadership off the node, SIGTERM, restart on the same data dir with the new binary, heal) — see cluster_lifecycle.rs::mp_rolling_upgrade_no_loss_no_stall, which proves zero acknowledged-write loss across a full rolling upgrade under load.

Node-replace drill (m11p5)

Replacing a dead or recycled node is kubectl delete pod — no topology edit, no operator verb:

• PVC retained (the data dir survives the pod): the replacement boots on the same data dir and converges via its boot-time StreamSegments catch-up — the ordinary restart path above. If it rotated past the leader's retained WAL while down, it reseeds via FetchSnapshot automatically (the reseed_required marker + snapshot path, §9.1).
• PVC deleted + pod deleted (fresh node): the replacement comes up empty and seed-joins (§1b) — it --seeds a survivor, FetchSnapshots the current state, and the leader auto-promotes it back to Voter. Under the k8s/cluster/ StatefulSet this is the default: the pod's args carry --seed against the headless Service, so a recreated pod rejoins with no human in the loop.

Add a node: scale the StatefulSet up (pod N≥3 seed-joins as a learner and auto-promotes). Remove a node: POST /cluster/members/remove first (§6), wait for the record to quorum-commit, then scale down (lowest ordinal last).

12. Security (m11p7): mTLS, rotation, identity, audit, rate limits

The cluster does not trust the network. Everything here is opt-in — absent the grpc_tls block and the cluster key, the cluster behaves exactly as pre-m11p7 (plaintext, hint-only marker, no audit/limit). A reference (k8s) deployment turns it all on.

12.1 mTLS (gRPC replication) — the default posture

• Configure the grpc_tls block per region (ca_cert, server_cert, server_key, client_cert, client_key). The gRPC server then REQUIRES a client cert chained to the cluster CA (mutual TLS): a foreign pod with no cert, a cert from another CA, or a plaintext probe fails the TLS handshake and never reaches an RPC.
• No grpc_tls ⇒ plaintext, with a loud startup WARN on both the server and the client. Acceptable only on a trusted single-host / loopback topology. To serve plaintext intentionally there is nothing else to set — the WARN is the signal that you are on the insecure path.
• The inter-node HTTP plane (forwards, broadcasts, scatter, status, seed-join) is served over TLS with the SAME cert and dials https:// with the cluster CA whenever grpc_tls is set, so enabling it gives zero plaintext inter-node links on both planes at once.

12.2 Cert + bearer rotation WITHOUT restart

• A background poller (TIDAL_ROTATION_POLL_MS, default 30000) content-hashes the cert files and the credential files; on a change it atomically swaps the served cert (in-flight TLS sessions keep their negotiated keys — zero dropped requests) and rebuilds the outbound peer channels.
• Procedure: issue a new cert under the same CA (cert-manager renewal, or re-run scripts/gen-cluster-certs.sh and re-apply the Secret) — the files change in place, the poller swaps within one interval, no pod restart. Verify with tidaldb_cluster_* logs (TLS material rotated…) or the tidal_audit /tracing stream.
• The bearer (TIDAL_API_KEY_FILE) and the cluster key (TIDAL_CLUSTER_KEY_FILE) rotate the same way. Use FILE mounts (not inline env) so a Secret rotation is picked up live. During a CA roll, keep both old and new CAs trusted for one cycle (CA-overlap) so in-flight connections complete.

12.3 Per-node identity + the marker

• Set a shared cluster key (TIDAL_CLUSTER_KEY / TIDAL_CLUSTER_KEY_FILE, any random string — BLAKE3-derived to the MAC key). Each node then mints a signed x-tidal-node-token on every forward/broadcast; the receiver verifies it. This gives inter-node calls a verifiable node identity and is the defense-in-depth layer beyond the shared bearer.
• With a cluster key configured, the x-tidal-internal marker is honored ONLY from a verified sibling: a request that sets the marker without a valid node token is rejected 403 (the marker is a routing hint, never an auth bypass). Never hand the cluster key to external clients.

12.4 Admin audit log

• promote / partition / heal / join / member-remove / reseed each emit one structured record: {principal, verb, target, term, outcome}. The principal is the verified node (node:<id>) for inter-node calls or external for an operator with the bearer.
• Sinks: a tidal_audit tracing target (always — capture it in your log pipeline), plus an append-only JSONL file when TIDAL_AUDIT_LOG=<path> is set. Recorded on the operator-originated leg only (no double-record on a forwarded re-apply).
• At-rest encryption of the JSONL file is delegated to the volume — mount TIDAL_AUDIT_LOG on an encrypted PV (or a gVisor/LUKS-backed volume); the server does not encrypt it in-engine.

12.5 Per-principal rate limits

• TIDAL_RATE_LIMIT_RPS (+ optional TIDAL_RATE_LIMIT_BURST, default 2×) caps per-principal request rate; a deny is 429 + Retry-After. Off by default.
• Verified sibling nodes are EXEMPT — replication/forward traffic is never throttled by the external-client budget. Today external callers share one bucket (the shared bearer); a future multi-key registry gives per-key buckets.

12.6 Foreign-pod / negative behavior (what an attacker on the network sees)

Attempt	Result
Ship a gRPC segment without a cluster client cert	TLS handshake fails — no RPC dispatched
Call an internal HTTP route without trusting the cluster CA	TLS handshake fails — no route reached
Set x-tidal-internal without a valid node token (key configured)	403 — marker honored only from a verified sibling
Call a protected route without the bearer	401 (unchanged)

13. Coordinated backup / restore + point-in-time recovery (m11p8)

The building blocks: the engine's crash-consistent create_backup, the WAL archive (wal.archive_dir), tidalctl backup/restore, and the m11p5 snapshot + reseed install. Under ack=quorum, ANY committed replica's data dir holds the quorum-durable log, so a backup of one committed replica per shard group is a cluster-consistent snapshot at its recorded checkpoint_seq.

13.1 Enable the WAL archive (point-in-time recovery)

Set wal.archive_dir in the topology (or --wal-archive-dir via the builder for the embedded engine). Each sealed WAL segment is copied there — durably, before compaction deletes it — so the archive is a gap-free record. Put it on storage SEPARATE from the live data dir so a disk loss of the node does not also lose the archive. Segment filenames encode shard + first_seq, so co-located groups share one archive dir without collision.

wal:
  archive_dir: "/archive/tidaldb"   # durable, off-node storage

13.2 Coordinated backup drill

1. Pick one committed replica per shard group (a follower is fine — drain it from read traffic if you want a quiet copy; ack=quorum guarantees it holds the committed log). Stop it (or snapshot its volume).
2. tidalctl backup --path /data/<node> --out /backups/<cluster>-<ts>/shard-<id> — writes a recursive copy + BACKUP_MANIFEST.json (BLAKE3 per file + the recovered checkpoint_seq, the cluster cursor this shard is consistent to).
3. Repeat per shard group. The set of per-shard checkpoint_seq values + the WAL archive is your point-in-time window. Restart the replica; self-heal/catch-up reconverges it.

13.3 Restore drill (timed)

1. tidalctl restore --from /backups/<cluster>-<ts>/shard-<id> --path /data/<new-node> — verifies EVERY file's BLAKE3 against the manifest BEFORE writing, and refuses a non-empty target (it never overwrites a live data dir).
2. Point a stopped node at the restored dir and boot it. Under ack=quorum its group's followers catch up via the live stream; promote it if it is the group's chosen leader (the highest-applied survivor rule, §9).
3. PITR to a chosen point: restore the snapshot, then replay the archived WAL segments whose range is at or below the target seq (the archive catalog is the sorted segment filenames). Replay stops at the target — events above it are not applied.

Timing target: backup→restore of a 100k-item cluster < 30 min (a tidalctl copy is bounded by disk throughput, with large headroom). The Ref-A timed figure is a k3s line item (the standing M11 access caveat).

14. Rolling upgrade + version skew (m11p8)

Nodes carry a build version on the wire (HeartbeatRequest.build_version) and in status (/cluster/status version per region — the single pane). Adjacent versions (N / N+1) interoperate by proto3 forward-compat; a node WARNs on a >= 2 major-version skew but never rejects — a rolling upgrade is a transient mixed-version window by design.

Procedure (one node at a time):

1. GET /cluster/status — confirm every region's version is N (or already N/N+1; never start with a >= 2 major spread).
2. Graceful SIGTERM one follower → it drains (readiness 503 → checkpoint).
3. Restart it on N+1 (same ports, same data dir). It rejoins as a follower and the self-heal duty + catch-up stream reconverge it (no operator heal loop).
4. Repeat for each follower. Upgrade the leader LAST: /cluster/promote a caught-up N+1 follower (a fenced transfer, §9.2), then upgrade the old leader as a follower.
5. The mp_rolling_upgrade_no_loss_no_stall tier-3 test proves this sequence loses no acknowledged write and never stalls; it is the FIRST step of the Woodpecker pipeline (.woodpecker.yaml) — a failure blocks the image build.

Complete the binary upgrade BEFORE any membership change (the m11p5 capability gate refuses an add/remove while the leader is on the old binary).

Performance (measured over real localhost processes)

Operation	SLA	Measured (p99 / typical)
Replicated /signals throughput (m11p1, 3 nodes, release build)	≥ 2,000/s	4,534 signal-writes/s within SLO on the ramp (knee ~5.5k/s); 2,739/s sustained 10 min (1.65M writes, 0.35% errors); was ~90/s pre-m11p1
Replication lag under that load (m11p1)	< 2s	≤ 103 events (~40ms) across the 10-min sustain; ≤ 377 events on the 5k/s ramp (follower group-commit coalescing)
Cross-region replication (write → follower applied)	< 2s	~110–133ms p99 (m8p10)
Failover (/cluster/promote → first successful write)	< 10s	~31–34ms
CRDT reconcile (merge+apply, each side)	< 100ms	0–1ms

Write-latency note: signal p50 ≈ 17–25ms with a p99 tail of 160–220ms on macOS, where F_FULLFSYNC averages ~7.4ms with a 10–50ms tail (tidaldb_cluster_wal_fsync_us — 0% complete under 1ms). On Linux fdatasync volumes the same pipeline's fsync floor is far lower; validate the p99 gate on the reference environment, and tune wal.batch_timeout_ms against the measured fsync histogram.

Cross-references

• Kubernetes deployment — docs/runbooks/kubernetes.md (the hardened standalone single-replica set in k8s/, and the multi-region cluster reference in k8s/cluster/ — one StatefulSet + headless Service peer discovery + PDB, with --seed-based scale and kubectl delete pod node-replace, shipped in m11p5).
• Server deployment guide — docs/guides/server-deployment.md (standalone and cluster launch, config, env, health probes).
• Monitoring & alerts — docs/ops/monitoring.md (Prometheus scrape config, replication-lag and WAL metrics, recommended alerts; per-region replication lag is observable via /cluster/status lag_events; remember the /metrics endpoint is unauthenticated — bind it internally).
• Roadmap / cluster status & known gaps — docs/planning/ROADMAP.md and docs/roadmap-to-cluster.md for the M11 cluster status — quorum-ack writes (m11p3), automatic failover (m11p4), and membership/discovery/elasticity (m11p5) all shipped; sharding × replication (p6), security hardening (p7), and continuous correctness (p9) remain.
• API & schema reference — API.md, QUICKSTART.md, and the live /openapi.json document.
• Scope & vision — VISION.md.