tidaldb/docs/runbooks/cluster.md

tidalDB Cluster Runbook

Operating the multi-region tidal-server cluster surface: launch (single-process dev fabric and the multi-process region nodes), the operational API, replication transport facts, failover and partition drills, and the honest write-durability contract.

STATUS: QUORUM-ACKED HA IS LIVE — RUNNING IN PRODUCTION ON k3s

Quorum-ack writes (m11p3) and automatic election/failover (m11p4) are LIVE and deployed. The reference cluster runs in production on k3s as a single StatefulSet, full-placement RF3 deployment: ns tidaldb-cluster, replicas: 3, every pod a region (tidaldb-0/1/2) hosting all three shard groups, image m12-writeburst-rc7. ack=quorum is the cluster deployment default (topology replication.ack: quorum) and leader election + failover are automatic — kill the leader and the survivors elect a successor with zero operator verbs and zero acknowledged-write loss. See the live production topology in §1 and the deployed shape in §3a.

Honest caveats that still hold: cluster mode replicates global retrieval signals only (no per-user personalization on the /signals route — see the personalization note in §3); and because a standalone node is the right answer for most deployments, both launch modes refuse to start unless you explicitly opt in with --experimental-cluster or TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 (the k8s manifests set the env var) — a guard against standing up a multi-node fabric by accident, not a readiness warning. The two launch shapes below remain:

1. Multi-process (--region) — real process isolation (the production shape). Each tidal-server cluster --region <name> process owns exactly one region: one TidalDb, one GrpcTransport whose server binds this region's grpc_addr and whose peers are every sibling region's real grpc_addr. Processes peer over real gRPC and forward over real HTTP, so a crash of one region's process takes down only that region — the survivors keep serving. This is genuine process (and, across hosts, host) isolation. It is verified end-to-end by the tier-3 suites (cluster_multiproc, cluster_chaos, cluster_lifecycle, cluster_runbook) over real OS processes with real network-partition injection.

2. Single-process (no --region) — the dev/demo default. Every region runs inside one process. Replication still traverses the real tidal-net gRPC transport on loopback (faithful multi-region semantics over a real wire), but there is no process isolation: a crash, OOM, or host failure takes the whole "cluster" down at once. This is a development / staging / demo fabric and a correctness harness for the replication paths — not production HA. It remains the default because it needs no per-region topology addresses and no process orchestration.

Durability / leadership / membership facts (both modes):

• Quorum durability is the cluster default, per-request overridable. replication.ack: quorum is the topology default the k8s reference cluster ships, so a write succeeds only once a majority of the replica set durably holds it (m11p3), surviving permanent leader loss. A caller can downgrade a single write to leader-durable with x-tidal-ack: leader (storage + WAL fsync; follower ship off the request path). See §8.
• Leadership is automatic (m11p4). Every node runs a failure detector and a Raft-style election (pre-vote + vote + check-quorum + fenced transfer): kill the leader and the survivors elect a successor — typically under a second with the defaults — with zero operator verbs and zero acknowledged-write loss (see §9.1). /cluster/promote is now a fenced transfer for maintenance/override, not the availability mechanism. election.auto_election: false preserves the pre-m11p4 operator-driven posture.
• Membership is elastic and addresses are DNS names (m11p5). grpc_addr is an advertised hostname or IP (DNS-resolved on every reconnect); nodes join online via --seed and catch up via FetchSnapshot + the StreamSegments stream; add/remove ride kind-4 membership records on the replicated log (see §1b, §3, §6, §9.1).

Production deployment today is the multi-process cluster on k3s described above — the single-StatefulSet full-placement RF3 reference in k8s/cluster/ (see §1, §3a, and kubernetes.md). The single-process shape below remains the dev/demo fabric and replication-correctness harness, and a single tidal-server standalone node (see server-deployment.md) stays valid for deployments that do not need multi-region / ack=quorum.

Both modes refuse to start unless you explicitly opt in with either the --experimental-cluster flag or the TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 environment variable. On start each emits a loud, mode-specific WARN restating exactly what it does and does not provide. Do not wire either into a production load balancer.

Scope: what cluster mode does and does not own

tidalDB owns retrieval and ranking only. Cluster mode adds multi-region replication of that, nothing more. It does not generate embeddings, store video/blobs, run a CDN or transcoder, do auth/moderation/payments, or provide a distributed consensus log. Bring your vectors; tidalDB retrieves and ranks over them. See VISION.md for the scope boundary.

Prerequisites

• Rust toolchain ≥ 1.91 if running directly (the Docker build pins rust:1.91-bookworm).
• protobuf-compiler (protoc) and a C++ toolchain (g++) on the build host — tidal-net's build script compiles the WAL-shipping .proto, and USearch's HNSW core is C++. The Docker image installs both.
• Docker 25+ if running via container.
• For single-process mode: one HTTP port (default 9500); each follower region also binds an OS-assigned loopback port for its gRPC replication server unless you pin one in the topology (see §3).
• For multi-process mode: per-region grpc_addr and http_addr declared in the topology, and the ports they name available on each host (see §3).

1. Launch the cluster locally

Cluster mode is gated. Pass --experimental-cluster (or set TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1) or the server exits with a mode-specific error explaining why.

Live production topology (the deployed shape on k3s — 2026-06-19). The reference cluster is ONE StatefulSet tidaldb in namespace tidaldb-cluster, replicas: 3 = 3 pods = 3 regions = 3 voters (tidaldb-0/1/2), full-placement RF3: every pod hosts all three shard groups (the 3-group shards: block in §3a is the DEPLOYED shape, not optional). Each group's data lives under /data/db/shard-0000N on the pod's one PVC. Each pod binds one gRPC port per group — shard 0 → 9601, shard 1 → 9602, shard 2 → 9603 (derived node base port + shard id). The HTTP plane is :9500 over HTTPS with inter-node mTLS (m11p7 — every probe/curl uses https://); Prometheus /metrics is :9091. replication.ack: quorum is the deployment default and election/failover is automatic (election.auto_election: true). Live image m12-writeburst-rc7 (@sha256:171505745b801dcf231b531de6167dbc309a7182957811cbc2228f0a302572b1). Networking is a headless peer Service tidaldb-peers (per-pod DNS tidaldb-N.tidaldb-peers.tidaldb-cluster.svc.cluster.local) plus a ready-only client Service tidaldb (the seed-join discovery target). See k8s/cluster/statefulset.yaml and k8s/cluster/topology-configmap.yaml.

1a. Single-process (dev/demo default)

TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
cargo run -p tidal-server -- \
  cluster \
  --listen 127.0.0.1:9500 \
  --schema tidal-server/config/default-schema.yaml \
  --topology tidal-server/config/default-cluster.yaml \
  --experimental-cluster

The default topology spins up three regions (us-east, eu-west, ap-south) with us-east as leader, all inside one process. On a clean start you will see a WARN line stating this is experimental and single-process, an info line per follower (follower gRPC transport ready), and finally listening on http://127.0.0.1:9500.

1b. Multi-process (one process per region)

Pass --region <name> (or set TIDAL_REGION) to run only that region in this process. The topology must declare a per-region grpc_addr and http_addr for every region (see §3); siblings reach each other over those. Launch one process per region — typically one per host, each with its own --data-dir. --data-dir is REQUIRED in multi-process mode (m11p2): the durable WAL is the replicated log itself — it is what ships to peers and what serves their catch-up streams — so a node without one is rejected at startup:

# Region us-east (the initial leader) on host A
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
tidal-server cluster --experimental-cluster \
  --region us-east \
  --listen 0.0.0.0:9501 \
  --schema /etc/tidal/schema.yaml \
  --topology /etc/tidal/topology.yaml \
  --data-dir /var/lib/tidal/us-east

# Region eu-west on host B
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
tidal-server cluster --experimental-cluster \
  --region eu-west --listen 0.0.0.0:9502 \
  --schema /etc/tidal/schema.yaml --topology /etc/tidal/topology.yaml \
  --data-dir /var/lib/tidal/eu-west

# Region ap-south on host C
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
tidal-server cluster --experimental-cluster \
  --region ap-south --listen 0.0.0.0:9503 \
  --schema /etc/tidal/schema.yaml --topology /etc/tidal/topology.yaml \
  --data-dir /var/lib/tidal/ap-south

Every process parses the same topology file; RegionIds are assigned by declaration order, so all processes agree on the region→id mapping. Each process binds its own region's grpc_addr/http_addr and dials its siblings' addresses. The --listen HTTP address is the public gateway for that region.

Seed-join boot (m11p5 — adding a node without editing the topology)

A node can join an existing cluster online, without appearing in any declared topology, by contacting a running peer as a seed:

TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
tidal-server cluster --experimental-cluster \
  --region eu-2 \
  --listen 0.0.0.0:9504 \
  --schema /etc/tidal/schema.yaml \
  --topology /etc/tidal/topology.yaml \
  --data-dir /var/lib/tidal/eu-2 \
  --seed http://10.0.1.10:9501 \
  --seed http://10.0.2.10:9502 \
  --advertise-grpc eu-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9600 \
  --advertise-http  eu-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9504 \
  --metrics 0.0.0.0:9091

Flag	Meaning
--seed <url> (repeatable)	One or more seed HTTP base URLs. The joiner polls each for the current leader, then POST /cluster/joins through it. Any reachable seed works; list a few for resilience.
--advertise-grpc <host:port>	This node's advertised gRPC address — what siblings dial. Required with --seed (the joiner has no topology entry of its own). DNS-capable.
--advertise-http <host:port>	This node's advertised HTTP gateway — what peers forward writes/status to. Required with --seed.
--metrics <addr>	Prometheus /metrics bind (the joiner has no topology metrics_addr).

What happens: the joiner skips the topology's "every region declared" gate, learns its roster + assigned id + current term from the seed's join response, appends a Learner record to the replicated log (the leader answers only after it is quorum-committed), persists the roster to a durable membership cache and the term to election_state (persist-before-act), installs a snapshot when it is behind the leader's retained WAL, then streams the live tail. The leader's auto-promotion duty flips it Learner → Voter once it is within replication.learner_promote_lag (default 1024) of the flushed frontier. A restart boots from the cache without the seed.

A --seed boot still requires the local topology/config file for the behavioral knob blocks (replication:, wal:, election:, timeouts:, grpc_tls) — its regions: list is ignored for the roster (the join response is the roster), but a bare --seed with neither --topology nor TIDAL_CONFIG refuses to boot naming the rule, so a joiner never silently inherits the compiled-in defaults' wrong ack mode, quorum timeout, or election timing. The k8s manifests mount the shared bootstrap ConfigMap on every pod including N≥3, so this costs nothing there.

Capability gate (m11p5 mixed-version safety). The leader refuses /cluster/join and every conf-change until all current voters report kind-4 capability — see §8's downgrade rule. Complete the binary upgrade before adding or removing nodes.

Auth: set TIDAL_API_KEY=<secret> to require Authorization: Bearer <secret> on the data and /cluster/* mutation routes. If it is unset the server runs UNAUTHENTICATED and logs a WARN — never expose an unauthenticated cluster beyond loopback / a trusted VPC. Health probes and /openapi.json are always unauthenticated. In multi-process mode set the same key on every process: a forwarded/broadcast request passes the caller's Authorization through verbatim, and the internal-propagation marker (x-tidal-internal: 1) is an inter-sibling trust signal, not an auth bypass (the bearer middleware still runs first).

Useful environment variables:

Var	Mode	Effect
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1	both	Opt in to cluster mode (alternative to --experimental-cluster).
TIDAL_REGION	multi-process	Selects the single region this process owns (alternative to --region).
TIDAL_API_KEY	both	Bearer token for protected routes. Unset ⇒ unauthenticated + WARN. Set the SAME key on every region in multi-process mode.
TIDAL_HLC_SKEW_MS	multi-process	Signed ms offset applied to THIS process's HLC. Affects reconcile-time LWW stamping ONLY (not signal-decay timestamps). A test/ops escape hatch for verifying causal convergence under clock skew — do not set it in normal operation.
TIDAL_CONFIG	both	Config dir holding default-schema.yaml / default-cluster.yaml (used when --schema / --topology omitted).
TIDAL_SEED_STATUS_TIMEOUT_MS	seed-join	Per-status-poll HTTP timeout during seed-join leader discovery (default 5000). Raise on a TLS cluster under heavy CPU contention where a cold rustls handshake alone can blow a tighter budget — the joiner would then time out every poll and burn the whole 120s discovery window despite the peer being reachable. Lower only for fast loopback/test rigs.
PORT	both	Listen address. A bare port (9500) normalises to 0.0.0.0:9500.
TIDAL_SERVER_LOG	both	tracing filter (default info).

2. Launch via Docker

# Build the image once (build context is the repo root).
docker build -f docker/cluster/Dockerfile -t tidaldb:cluster .

# Run (press Ctrl+C to stop). The image already sets
# TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 so cluster mode starts; it still logs the
# loud experimental WARN. The default CMD is SINGLE-process cluster mode.
docker run --rm -p 9500:9500 tidaldb:cluster

The image bakes the default schema/topology under /etc/tidal-server and uses ENTRYPOINT ["tidal-server"] + a cluster … CMD, so you can override the subcommand (e.g. docker run tidaldb:cluster standalone …), pass --region for a multi-process container, or supply your own config files:

docker run --rm -p 9500:9500 \
  -e TIDAL_API_KEY=changeme \
  -v "$PWD/configs/my-schema.yaml:/srv/schema.yaml:ro" \
  -v "$PWD/configs/my-topology.yaml:/srv/topology.yaml:ro" \
  tidaldb:cluster \
  cluster \
    --listen 0.0.0.0:9500 \
    --schema /srv/schema.yaml \
    --topology /srv/topology.yaml

The container ships a HEALTHCHECK that curls /health. For Kubernetes/Compose deployment patterns see Cross-references.

3. Topology YAML

The topology file declares regions and the leader.

Single-process minimal default (tidal-server/config/default-cluster.yaml):

regions:
  - name: us-east
  - name: eu-west
  - name: ap-south
leader: us-east
# Optional: OS worker threads serving cluster write/heal requests (gRPC ship).
# Defaults to available parallelism when omitted.
# write_workers: 4

Multi-process requires a per-region grpc_addr and http_addr (validated at startup — a missing or syntactically invalid host:port is a hard error naming the region; reachability is deliberately not probed, since siblings boot in any order):

grpc_addr is an advertised address — a literal host:port OR a DNS hostname (m11p5). A new optional per-region grpc_bind controls the local bind independently of what siblings dial; the example below shows the DNS shape (one shared file names every region by its per-pod DNS name while each pod binds 0.0.0.0):

regions:
  - name: us-east
    grpc_addr: "tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9601"  # ADVERTISED (siblings dial; DNS re-resolved on reconnect)
    grpc_bind: "0.0.0.0:9601"      # LOCAL bind (optional; see derivation rule below)
    http_addr: "10.0.1.10:9501"    # this region's public HTTP gateway (forwarding + status)
    metrics_addr: "10.0.1.10:9091" # optional Prometheus /metrics listener (set it in production)
  - name: eu-west
    grpc_addr: "tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9602"
    grpc_bind: "0.0.0.0:9602"
    http_addr: "10.0.2.10:9502"
    metrics_addr: "10.0.2.10:9091"
  - name: ap-south
    grpc_addr: "tidaldb-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9603"
    grpc_bind: "0.0.0.0:9603"
    http_addr: "10.0.3.10:9503"
    metrics_addr: "10.0.3.10:9091"
leader: us-east
# Optional m11p1 tuning blocks (engine defaults shown):
# replication:
#   batch_max_events: 256   # events coalesced per shipped batch (1-256)
#   window: 4               # in-flight batches per peer (1-64)
#   retry_ms: 100           # backoff before a transiently-failed batch retries
#   catchup_retry_ms: 30000           # backoff before a FAILED catch-up pull re-pulls (m11p4)
#   snapshot_artifact_ttl_ms: 600000  # staged-snapshot reuse window (m11p5; never unpins an active consumer)
#   learner_promote_lag: 1024         # learner→voter promotion distance + readiness hysteresis (m11p5)
#   reseed_self_restart: false        # drain+exit(0) when reseed_required latches (m11p5; k8s sets true)
# wal:
#   batch_size: 100         # events per group-commit fsync (1-256)
#   batch_timeout_ms: 10    # max wait before a partial batch flushes
# election:                 # m11p4 failure-detector / election timers (defaults shown)
#   heartbeat_interval_ms: 300
#   election_timeout_min_ms: 1500     # validated: leader_lease_ms + heartbeat_interval_ms < this
#   election_timeout_max_ms: 3000
#   leader_lease_ms: 900
#   auto_election: true               # false → pre-m11p4 operator-driven failover

grpc_bind derivation (m11p5 §1): grpc_bind present → bind it. Absent + grpc_addr parses as a literal SocketAddr → bind that (today's behavior, byte-for-byte — every existing IP topology keeps working). Absent + grpc_addr is a hostname → bind 0.0.0.0:<port from grpc_addr>. Because the peer-dial path no longer parses grpc_addr as a SocketAddr, hyper re-resolves the hostname on every reconnect — a pod rescheduled onto a new IP is reached with no peer restarts. TLS note: SNI follows the URI host, so DNS peer names require DNS-SAN certs (see grpc_tls below); no code change.

3a. Sharding × replication (shards:, m11p6)

Absent shards:, the cluster is 1 shard × RF = all regions — one replicated log, one elected leader, byte-for-byte everything above. Add a shards: block to split the entity space into S groups, each a replication group at RF with its own elected leader, leaders balanced across nodes.

The k3s reference cluster runs this block enabled: S=3, full placement, RF3 (k8s/cluster/topology-configmap.yaml). Every pod replicates all three groups; tidaldb-0 leads shard 0, tidaldb-1 shard 1, tidaldb-2 shard 2, binding gRPC 9601/9602/9603 respectively. Full placement is the production shape, not a sharding experiment — a single pod loss never loses a group's quorum (2 of 3 survive per group) and any pod can serve a corpus-wide read locally.

Writes to /items///embeddings///signals hash-route (the engine's FNV-1a router) to the owning group's leader and replicate at RF; any gateway accepts any write.

regions:                                  # the NODE list (identity + addresses)
  - { name: us-east, grpc_addr: "10.0.1.10:9601", http_addr: "10.0.1.10:9501" }
  - { name: eu-west, grpc_addr: "10.0.2.10:9601", http_addr: "10.0.2.10:9501" }
  - { name: ap-south, grpc_addr: "10.0.3.10:9601", http_addr: "10.0.3.10:9501" }
shards:                                   # NEW — S groups (dense ids 0..S)
  - id: 0
    leader: us-east                       # term-0 / preferred leader (balance: group i → node i)
    replicas:                             # the RF nodes hosting this group
      - { node: us-east }                 # grpc_addr optional → derived node base port + shard id
      - { node: eu-west }
      - { node: ap-south }
  - { id: 1, leader: eu-west,  replicas: [ {node: us-east}, {node: eu-west}, {node: ap-south} ] }
  - { id: 2, leader: ap-south, replicas: [ {node: us-east}, {node: eu-west}, {node: ap-south} ] }
leader: us-east                           # legacy field, ignored once shards: is set

• Ports & dirs. A node hosting several groups binds one gRPC port PER group: set replicas[].grpc_addr explicitly, or omit it to derive node base port + shard id. Each group's data lives under <data_dir>/shard-{id:05}/; the single-group legacy layout (shards: absent) keeps <data_dir> verbatim, so existing clusters restart unchanged.
• Placement. replicas need not be every node (RF < N is valid). The exit gate and the simplest production shape are full placement (every node replicates every group); a strict-subset placement is supported for writes but see the read caveat in §7.
• Validation. Shard ids must be dense and unique; every leader/replica.node names a declared region; a group needs ≥1 replica; co-hosted groups must not collide on a derived gRPC port.

Per-shard admin (?shard=). With shards: set, every cluster admin verb takes an optional ?shard=<id> selecting which hosted group to act on; omitted = the node's lowest-id hosted group (and the only group when S=1, so S=1 URLs are unchanged). The selector is forwarded on intra-group hops, and NotLeader names the group. See §6a for the rebalancing verbs.

Fields:

Field	Single-process	Multi-process	Meaning
regions[].name	required	required	Region name used everywhere in the HTTP API (?region=, /cluster/promote, etc.). Must be unique.
regions[].grpc_addr	optional	required	This region's advertised gRPC replication address — what siblings dial. Since m11p5 it may be a literal host:port OR a DNS hostname (re-resolved on every reconnect). In single-process mode omit it — the server allocates a free loopback port and self-heals a bind race by retrying on a fresh port. In multi-process mode it is required and is tried exactly once.
regions[].grpc_bind	unused	optional	This region's local gRPC bind host:port (m11p5), independent of the advertised grpc_addr. Omitted: a literal grpc_addr binds itself; a hostname grpc_addr binds 0.0.0.0:<its port>. Set it to bind a specific interface while advertising a DNS name.
regions[].http_addr	unused	required	This region's public HTTP gateway address, used by siblings for write/read forwarding and status aggregation. Unused in single-process mode.
regions[].grpc_tls	unused	optional	TLS material for this region's gRPC transport: ca_cert, server_cert, server_key (PEM paths), plus optional client_cert/client_key for mTLS. Omitted ⇒ plaintext, the right posture for loopback/VPC topologies. DNS grpc_addr requires DNS-SAN certs (SNI follows the dialed hostname).
regions[].metrics_addr	unused	optional	Bind address for this region's Prometheus /metrics listener (m11p1; cluster mode previously had none). Omitted ⇒ no metrics endpoint. Set it in every production topology, and bind it internally — the endpoint is unauthenticated.
leader	required	required	Must name one of the declared regions. The initial write leader.
write_workers	optional	optional	Size of the runtime-free OS-thread pool that admission-controls cluster writes (/signals staging) and runs /cluster/heal's blocking redelivery. Bounded queue ⇒ 429 backpressure. Must be ≥ 1 when given.
timeouts.broadcast_peer_secs	unused	optional	Per-peer budget (seconds) for the /cluster/promote fan-out (the only remaining peer fan-out — the m11p2 log replaced the item/embedding broadcast). Default 2s — right for loopback/VPC; raise it for WAN topologies where a distant region cannot answer in 2s. Must be ≥ 1 when given.
replication.batch_max_events	unused	optional	Max relay events coalesced into one shipped batch (1–256, the WAL wire-format ceiling). Default 256.
replication.window	unused	optional	In-flight batches per peer (1–64). 1 = strictly in-order shipping; higher pipelines across the peer RTT (out-of-order arrivals park gap-aware on the receiver). Default 4.
replication.retry_ms	unused	optional	Backoff (ms) before a transiently-failed batch ship retries. Default 100.
replication.ack	unused	optional	Deployment-default write acknowledgment: leader (default) or quorum (majority-durable — see §8). Per-request override: the x-tidal-ack header.
replication.quorum_timeout_ms	unused	optional	Budget an ack=quorum write waits for the commit index before the retryable 503 naming the laggards. Default 2000.
replication.catchup_retry_ms	unused	optional	Backoff (ms) before a FAILED catch-up pull re-pulls on a timer (m11p4) — lets an idle cluster self-heal a follower whose pull failed during a rolling restart. Default 30000. Must be ≥ 1 when given.
replication.snapshot_artifact_ttl_ms	unused	optional	Staged-snapshot reuse window (m11p5), counted from the last fetch's completion. Governs artifact REUSE only — never unpins the WAL retention of an active consumer; a hard cap (4×) force-drops a never-releasing pin (tidaldb_cluster_snapshot_pin_force_drops_total). Default 600000 (10 min). Must be ≥ 1.
replication.learner_promote_lag	unused	optional	Learner→voter promotion distance AND the readiness-convergence hysteresis threshold (m11p5), in events. Default 1024. Must be ≥ 1.
replication.reseed_self_restart	unused	optional	Drain + clean-exit(0) once the durable reseed_required marker latches (m11p5; k8s sets it true). Refused when the remaining voters can't sustain quorum without this node. Default false.
election.heartbeat_interval_ms	unused	optional	Leader heartbeat interval (m11p4). Default 300.
election.election_timeout_min_ms / ..max_ms	unused	optional	Randomized follower election timeout window (m11p4). Defaults 1500 / 3000. Validated: leader_lease_ms + heartbeat_interval_ms < election_timeout_min_ms.
election.leader_lease_ms	unused	optional	Leader freshness lease — a leader that loses majority contact steps down within it (m11p4). Default 900.
election.auto_election	unused	optional	true (default) = automatic failover; false = pre-m11p4 operator-driven posture (no auto elections, no check-quorum step-down).
wal.batch_size	optional	optional	Events per WAL group-commit fsync (1–256). Default 100.
wal.batch_timeout_ms	optional	optional	Max ms a partial group-commit batch waits before flushing. Default 10. Tune against the measured tidaldb_cluster_wal_fsync_us on the deployment's volume.

The schema YAML (signals / text_fields / embedding_slots / profiles) is the same format the standalone server loads — see API.md and QUICKSTART.md for the full schema/profile grammar and the built-in ranking profiles. Topology only adds the region layout above.

Personalization correctness note. Preference-vector personalization (the taste vector that powers for_you) only updates for signals declared positive_engagement: true in the schema. The cluster /signals route writes global signals (no user_id / creator_id context), so it never folds item embeddings into a user's taste vector regardless of the positive_engagement flag. Personalized writes go through the embedded engine (signal_with_context), not this HTTP route. The exception is /hardnegs, which records a user-scoped hide that converges across regions via /cluster/reconcile.

4. gRPC replication transport (tidal-net)

Replication between regions is the real tidal-net WalShipping gRPC service, not in-process channels. The facts you need to operate and firewall it:

Property	Value	Notes
Service	WalShipping (proto tidal.replication.v1)	RPCs: ShipSegment (unary; the live-tail push path), StreamSegments (server-streaming; the follower-pulled catch-up path since m11p2), FetchSnapshot (server-streaming; the m11p5 snapshot transfer for joiners + reseeds), JoinCluster (m11p5 conf-change), and Heartbeat (ControlPlane health).
Bind address	per-region grpc_addr (multi-process); an auto-allocated loopback port (single-process)	Default dev band 59520–59529 if you pin one.
Transport security	mTLS via TlsConfig (ca_cert, server_cert, server_key, optional client_cert / client_key)	insecure = true (plaintext) is the loopback/VPC default in this phase. A peer with no TLS config must set insecure.
Max payload	64 MiB	Both encoder and decoder codec limits are raised to this; pinned by a compile-time assert to the engine's InProcessTransport limit so both ends agree. WAL segments default to 16 MiB.
Circuit breaker	per-peer, threshold 5, reset 30s	After 5 consecutive ship failures the breaker opens; it stays open for 30s, then the next attempt probes (HalfOpen → Closed on success, Open again on failure). Backpressure (channel full) does not trip it. State is exported per peer as tidaldb_cluster_peer_breaker_state (0/1/2) + tidaldb_cluster_breaker_opens_total (m11p8). Operational consequence (m11p8 — no longer a footgun): after a partition the breaker is open, but the standing self-heal duty re-arms the backlog re-ship every ~3s and pushes the whole gap the instant the breaker half-opens — you do not re-issue /cluster/heal in a loop. Watch tidaldb_cluster_healing_peers → 0 (see §6 and the drills in §10).
Timeouts (defaults)	connect 5s, request 10s, keep-alive PING every 10s / 5s ACK	A blackholed peer fails fast instead of stalling the single-threaded shipper.

ShipSegment carries the WAL segment id, BLAKE3-validated payload bytes, event count, and the leader's authoritative leader_last_seq (so a follower can advance its replication-lag high-water-mark even for an all-local segment that filters to empty). The follower's segment-receiver thread drains and applies it on arrival. The engine's applied high-water-mark is a contiguous frontier with a bounded ahead-buffer, so out-of-order eager ships can never swallow a sequence gap (a silent-data-loss class fixed in m8p10).

Operational guard: the gRPC replication ports and the Prometheus /metrics endpoint are UNAUTHENTICATED. Bind them to loopback or a cluster-internal network only — never the public interface.

5. Core HTTP API

All routes are JSON unless noted. Examples assume BASE=http://localhost:9501 (any region's gateway in multi-process mode; the single --listen address in single-process mode) and, when TIDAL_API_KEY is set, AUTH='-H "Authorization: Bearer $TIDAL_API_KEY"'. Drop the -H header when running unauthenticated.

Middleware mirrors standalone: 30s request timeout (408), 100 max in-flight (429), 2 MB body limit (413), and an x-request-id on every response. Health probes sit outside the load-shedding stack so liveness/readiness are never queued or timed out under saturation.

Multi-process routing (one coherent surface). Any region's gateway accepts any operation and routes it to the node that owns it:

• Writes (/signals, /items, /embeddings, /hardnegs) on a non-leader forward to the leader (the caller's Authorization passes through); the client sees the leader's status/body. A leader that is unreachable degrades to a 503 naming the leader, never a hang.
• Reads (/feed, /search) default to the LOCAL region. This is the key difference from single-process mode, whose default read is the leader. A ?region=<other> read forwards to that region's process.
• /cluster/promote, /cluster/partition, /cluster/heal route to / fan out from the leader as documented in §6.

Health

curl "$BASE/health"          # 200 ok / 503 while draining; reports mode, region, leader
curl "$BASE/health/startup"  # always 200
curl "$BASE/health/live"     # always 200
curl "$BASE/openapi.json"    # served OpenAPI 3.1, UNAUTHENTICATED — canonical HTTP reference

/health returns { "ok": true, "service": "tidaldb", "mode": "cluster", "region": "us-east", "leader": "us-east", ... }. The /openapi.json document is the machine-readable source of truth for the data, /cluster/*, /sharded/*, and /hardnegs request/response shapes (the health probes are intentionally outside the documented API surface).

Register items & embeddings (one replicated log — m11p2)

Items and embeddings ride the same replicated WAL stream as signals: the leader journals each write as a kind-1 (item metadata) or kind-2 (embedding) WAL record BEFORE touching storage, and followers apply it from the log — live pushes for the hot tail, the StreamSegments catch-up stream for history. There is no HTTP broadcast anymore (the m8p10 side channel and its bug classes were deleted in m11p2), so the responses are plain statuses:

curl -X POST "$BASE/items" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "metadata": { "title": "Jazz Piano", "category": "music" } }'
# → 201 Created (item durably journaled into the replicated log + applied)

curl -X POST "$BASE/embeddings" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "values": [0.1, 0.2, 0.3, 0.4] }'
# → 204 No Content

The 201/204 asserts leader durability and carries x-tidal-seq (the record's seqno in the replicated log); x-tidal-ack: quorum upgrades it to a majority-durable ack (see §8). The record is fsynced into the stream every follower receives (live push, or pull-based catch-up after downtime — see §6 heal). A write to a non-leader forwards to the leader transparently (the ack header and seq header travel through the forward). A down/partitioned peer needs no backfill bookkeeping: it converges from the log when it returns.

Embeddings: tidalDB does not generate vectors — the caller brings them. The write L2-normalizes and inserts into the HNSW index. Dimensions are strict: they must equal the slot's declared dimensions (min 2, max 4096) or the insert is rejected with a 500 (the engine surfaces a dimension mismatch as an internal error, not a 400 — a known wart, tracked post-M8); zero-norm vectors are also rejected (500). RETRIEVE / SEARCH route through the first declared embedding slot only; multi-modal apps must fuse offline or use separate entity kinds.

Record signals (cluster /signals = global only)

curl -X POST "$BASE/signals" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "signal": "view", "weight": 1.0 }'
# → 204 No Content (see the durability contract in §8)

The signal name must be declared in the schema (an undeclared name returns 400 naming it). The 204 carries x-tidal-seq (the write's replicated-log seqno), and x-tidal-ack: quorum upgrades it to a majority-durable ack — see §8. This route records a global signal on the leader and ships it to followers over the replicated WAL stream; it does not personalize (see the personalization note in §3). On a non-leader gateway it forwards to the leader transparently and still returns 204.

Hard negatives (user-scoped hides)

curl -X POST "$BASE/hardnegs" \
  -H 'Content-Type: application/json' \
  -d '{ "user_id": 42, "item_id": 7 }'
# → 204 No Content

Records a user-scoped hide on the leader (a non-leader gateway forwards). The item is filtered from that user's /feed?user_id=42. Hard negatives converge across regions via the LWW-resolved /cluster/reconcile CRDT path — NOT the signal WAL relay and NOT a broadcast.

Retrieve and search (region-pinned reads)

# Default read region is the LOCAL region (multi-process mode).
curl "$BASE/feed?user_id=42&profile=for_you&limit=20"
curl "$BASE/search?query=jazz%20piano&user_id=42&limit=5"

# Pin a read to a specific region. The gateway forwards to that region's process.
# Followers may lag the leader (and lag jumps during a partition) — use this for
# canary reads and lag verification.
curl "$BASE/feed?profile=trending&region=eu-west"

?region= accepts any declared region name; an unknown name returns 400. Omit it and the read serves from the LOCAL region (multi-process) / the current leader (single-process). limit is clamped to 1000 at the trust boundary (a larger value cannot amplify memory unboundedly). The for_you profile applies the built-in diversity defaults (max_per_creator=2, format_mix_max_fraction=0.4, exploration=0.1).

6. Cluster management API

Check cluster status

Two views. /cluster/status/local reports THIS node's own replication/leadership state (no peer calls); /cluster/status aggregates EVERY region (the gateway calls each peer's /cluster/status/local concurrently with a tight per-peer budget).

curl "$BASE/cluster/status/local" | jq

{
  "region": "ap-south",
  "is_leader": false,
  "leader": "us-east",
  "last_seq": 0,
  "applied_events": 124,
  "lag_events": 1,
  "partitioned": [],
  "reachable": true
}

curl "$BASE/cluster/status" | jq

{
  "leader": "us-east",
  "relay_log_len": 125,
  "regions": [
    { "name": "us-east",  "applied_events": 125, "lag_events": 0, "partitioned": false, "reachable": true },
    { "name": "eu-west",  "applied_events": 125, "lag_events": 0, "partitioned": false, "reachable": true },
    { "name": "ap-south", "applied_events": 124, "lag_events": 1, "partitioned": false, "reachable": true }
  ]
}

relay_log_len is the leader's high-water-mark (last_seq); each region's lag_events is relay_log_len − applied_events (saturating). Since m11p3 /cluster/status/local also reports the node's ack default and, on the leader, commit_index — the highest seqno a majority of the replica set durably holds (last_seq − commit_index is the quorum lag). A region the gateway cannot reach within the per-peer budget is reported honestly as reachable: false, partitioned: true, applied_events: 0, and worst-case lag (lag_events == relay_log_len). A non-zero, growing lag on a reachable region is the signal that it is partitioned (ship-skip) or its segment-receiver is wedged.

Promote a new leader

curl -X POST "$BASE/cluster/promote" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "eu-west" }'
# → { "ok": true, "leader": "eu-west", "acked": ["eu-west","ap-south"], "failed": [] }

The gateway first resolves the TARGET's stream baseline — the new leader's WAL flushed frontier at promotion, persisted in its data_dir/stream_baseline — then fans the promote out to every peer carrying it (internal marker, so each applies locally and does not re-fan). Peers jump their applied frontier for the new leader's shard to the baseline: everything at or below it is pre-stream history (replicated applies of the OLD stream), not data. The response carries baseline plus the acked/failed fan-out report; a peer in failed (e.g. a dead old leader) self-corrects — its first parked batch from the new stream triggers a catch-up pull whose chunks announce the baseline. After promotion /cluster/status reports the new leader; new writes route there; the new leader's ship queue activates and the demoted node's deactivates. An unknown region returns 400. Since m11p4 promote is a fenced transfer (it drains the target, then sanctions an election), not the availability mechanism — automatic failover handles a dead leader with no operator action (§9.1); this verb is for maintenance and deliberate successor choice (§9.2).

Simulate a partition & heal

There are two ways to partition a region; the runbook drills demonstrate both (§10):

1. Simulated ship-skip flag — /cluster/partition tells the leader to stop shipping to the named region (no sockets touched). Good for a controlled, reversible lag demo.
2. Real network partition — sever the actual TCP path between processes (firewall / proxy). The chaos suite uses a root-free in-harness TCP relay; operators can use iptables/pfctl (see §10).

# Simulated: isolate ap-south — leader ships skip this follower, so its lag climbs.
curl -X POST "$BASE/cluster/partition" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "ap-south" }'
# → { "ok": true, "partitioned": "ap-south" }

# Heal: resume shipping past everything the follower reports applied, and
# nudge it to PULL any history that rotated out of the leader's ship tail
# (the single recovery verb — log catch-up IS the full heal since m11p2).
curl -X POST "$BASE/cluster/heal" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "ap-south" }'
# → { "ok": true, "healed": "ap-south" }

/cluster/heal is the single recovery verb, and since m11p2 it is pure log catch-up: it clears the partition flag, resumes the ship queue past the follower's reported applied seqno (retries of data the follower already holds prune automatically — every ship ack piggybacks the follower's applied seqno), and nudges the follower to pull anything older than the leader's in-memory ship tail via its StreamSegments catch-up stream over the leader's durable WAL segments (POST /cluster/catchup, internal; the nudge forwards the healing operator's own bearer credential). Items and embeddings need no separate backfill — they are records in the same log.

Convergence is self-driving (m11p8 — the operator no longer loops heal). The per-peer ship senders retry parked batches every replication.retry_ms (default 100ms); a follower that detects a gap pulls the catch-up stream itself (also on boot, so a restarted node converges with no operator action at all — the tier-3 mp_items_ride_the_log_and_catchup_stream proves it). On top of that, a standing leader heal duty runs every ~3s: for any non-partitioned peer whose ship breaker is open AND that trails the leader, it re-arms the backlog re-ship from the peer's durable mark, so the moment the breaker half-opens (threshold 5, reset 30s — see §4) the leader pushes the WHOLE gap. This closes the old footgun: you no longer re-issue /cluster/heal until lag 0 — the server drives it. Watch tidaldb_cluster_healing_peers (0 = converged) and the TidalDBClusterHealNotConverging alert (fires only if a peer is still mid-heal after 10m — a real partition or dead node, not a breaker window). /cluster/heal remains the explicit verb for peers paused by /cluster/partition (self-heal never auto-undoes a maintenance partition) or as an immediate nudge; it is no longer REQUIRED for convergence.

Reconcile (cross-region CRDT convergence)

curl -X POST "$BASE/cluster/reconcile" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "ap-south" }'
# → { "ok": true, "region": "ap-south",
#     "local_elapsed_ms": 0, "remote_elapsed_ms": 1, "ops_applied": 3 }

/cluster/reconcile exchanges a CRDT state snapshot with the target region: this node ships its snapshot into the target's merge AND applies the target's pre-merge snapshot back, so both sides converge to identical state by deterministic LWW. This is how hard negatives (recorded via /hardnegs) converge across regions — they do not ride the WAL relay. local_elapsed_ms / remote_elapsed_ms are the merge+apply times each side measured (not the HTTP round-trip); both are typically 0–1ms. Reconcile is idempotent: a repeat reconcile of already-converged regions is an exact no-op on scores (no drift). /cluster/reconcile/snapshot is the INTERNAL snapshot-exchange leg this verb drives (marker required); operators never call it directly.

Membership verbs (m11p5 — online add / remove / inspect / reseed)

Membership is data on the replicated log (kind-4 records, latest wins, folded into a ClusterMembership cell). All four verbs route to / are answered by the leader and are quorum-commit-gated, one change at a time. The leader refuses every conf-change until all current voters report kind-4 capability (the mixed-version safety gate — complete the binary upgrade first).

# Add a node (idempotent by name): forwards to the leader, which assigns the
# next id, appends a Learner record, and answers AFTER quorum-commit.
curl -X POST "$BASE/cluster/join" \
  -H 'Content-Type: application/json' \
  -d '{ "name": "eu-2",
        "grpc_addr": "eu-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9600",
        "http_addr": "eu-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9504" }'
# → { "id": 4, "role": "learner", "term": 7, "leader": "us-east", "members": [ … ] }

The joiner normally seed-joins itself (§1b); this verb is the same conf-change for tooling. A re-join from a known name returns its existing id and current role and appends nothing. The leader's auto-promotion duty (a standing duty, re-armed on every activation and membership apply — it survives the joining-era leader's death) flips the learner to Voter once it is within replication.learner_promote_lag of the flushed frontier; promotion_pending (lag=N) in /cluster/status/local makes a stuck scale-up diagnosable.

# Inspect the applied roster (ids, names, addresses, roles, conf version).
curl "$BASE/cluster/members" | jq

# Remove a node: appends a Removed tombstone (quorum-commit-gated). The peer's
# ship cell is retired only AFTER the record is delivered-to/acked-by the removed
# peer (bounded give-up → tidaldb_cluster_remove_delivery_giveups_total); the
# removed node's readiness flips to 503 and it stops campaigning. Its id is BURNED
# (never renumbered, never reused).
curl -X POST "$BASE/cluster/members/remove" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "eu-2" }'
# → { "removed": "eu-2", "membership_version": 9 }

# Force a reseed on demand: latches the durable reseed_required marker. The node
# keeps serving degraded (voting enabled) and reseeds via snapshot on its NEXT
# boot (or self-restarts if replication.reseed_self_restart is true and quorum
# can be sustained without it). See §9.1.
curl -X POST "$BASE/cluster/reseed"
# → { "reseed_required": true }

Scale-down order (decommission): call /cluster/members/remove first (so the cluster's quorum math shrinks before the node disappears), wait for the record to quorum-commit, then stop / delete the node — lowest ordinal last under a StatefulSet.

6a. Rebalancing (m11p6)

With a shards: topology (§3a) the operator moves load between nodes per group. Automatic rate-limited rebalancing is explicitly later; m11p6 ships the manual verbs (each reuses the proven m11p4/m11p5 machinery, scoped to one group).

Move a group's leadership (the rebalance-back-to-preferred path) — a fenced transfer (catch-up drain → TimeoutNow → term+1 election), identical to /cluster/promote?shard=<id>:

curl -X POST "$BASE/cluster/shards/0/transfer" \
  -H 'content-type: application/json' \
  -d '{"region":"eu-west"}'
# only group 0's leader moves; groups 1,2 are untouched

Add / remove a replica of a group — add runs the m11p5 join (the named node joins group {id} as a Learner, catches up via snapshot+stream, auto-promotes to Voter); remove runs the m11p5 fenced removal on that group's log:

curl -X POST "$BASE/cluster/shards/2/replicas" \
  -H 'content-type: application/json' \
  -d '{"action":"add","name":"region-3","grpc_addr":"10.0.4.10:9603","http_addr":"10.0.4.10:9504"}'

Multi-group caveat (tracked follow-up). A node's readiness is currently node-global across its co-hosted groups, so removing a node from ONE group of a node that hosts several would wrongly flip the whole node's readiness. Use the replicas verbs today for nodes that host a SINGLE group (or a brand-new node joining one group); per-group-aware readiness is the elasticity follow-up.

Inspect / target one group. Every admin verb takes ?shard=<id>; /cluster/status/local always lists a per-group shards[] array (leader, term, role, applied, lag, commit index per group), and /cluster/members?shard=<id> returns that group's roster.

7. Sharded scatter-gather API

The /sharded/* routes hash-partition entities across regions (hash(entity_id) % num_shards, using the engine's own ShardRouter so writes and reads never disagree). Writes route to the single owning region; reads fan out to all regions and K-way merge by score.

Writes are SINGLE-COPY and require an explicit opt-in. A /sharded/* write is applied to the owning region's LOCAL store with no WAL append, so it does not ride the leader relay and has redundancy 1 regardless of the replication factor — RF3 with ack: quorum does not replicate it. That is the design (parallel write throughput across shard owners: §11 measured 3,669 signals/s vs ~90/s replicated), but the endpoint used to answer 201/204 with nothing saying so. It now requires x-tidal-ack: local and returns 400 without it, naming the header and the replicating alternative in the body.

Write routes (each routes to the owning region; a non-owner gateway forwards):

ACK='x-tidal-ack: local'   # the single-copy opt-in; without it every line below is 400
curl -X POST "$BASE/sharded/items"      -H "$ACK" -d '{ "entity_id": 7, "metadata": { "title": "..." } }'  # 201
curl -X POST "$BASE/sharded/embeddings" -H "$ACK" -d '{ "entity_id": 7, "values": [0.1,0.2,0.3,0.4] }'      # 204
curl -X POST "$BASE/sharded/signals"    -H "$ACK" -d '{ "entity_id": 7, "signal": "view", "weight": 1.0 }'  # 204

For a replicated write use POST /items / /embeddings / /signals (leader WAL relay, x-tidal-ack: leader|quorum). local is rejected on those routes.

Read routes (scatter-gather across all regions):

curl "$BASE/sharded/feed?profile=for_you&limit=20&deadline_ms=50"
curl "$BASE/sharded/search?query=jazz&limit=10&deadline_ms=100"

Each sharded read accepts an optional deadline_ms — the total scatter budget (default 50ms, server-clamped to 10s). The per-shard deadline is deadline_ms − 5ms network overhead. The response includes a scatter_gather block:

{
  "items": [ /* merged, deduped, diversity-enforced, re-ranked */ ],
  "total_candidates": 4210,
  "scatter_gather": {
    "degraded": false,
    "shards_queried": 3,
    "elapsed_ms": 12,
    "shard_deadline_ms": 45
  }
}

Degraded semantics: a shard that is partitioned, errors, or misses the deadline is reported in unavailable_shards (a name list) and flips degraded: true — it is never silently dropped, and the read still returns 200 with the live shards' results. The merge dedups replicated copies of an entity (keeping the best-scoring copy), reconciles total_candidates so replicated shards are not counted multiple times, and re-enforces max_per_creator across the merged set.

8. Write-durability contract: the ack knob and its cost (m11p3)

Every replicated write (/signals, /items, /embeddings) runs under one of two acknowledgment modes. Pick the deployment default with the topology's replication.ack; any caller overrides per request with the x-tidal-ack header (leader or quorum; anything else is a 400).

	ack=leader (default)	ack=quorum
Success means	Durable on the leader (storage + WAL group-commit fsync)	Durable on a majority of the replica set (leader + floor(n/2) followers, each storage-applied + own-WAL-fsynced)
Survives	Any follower failure; leader restart (WAL replay)	Any single node's permanent loss, including the leader's — promote the max-applied survivor and every acked write is there (the 167-kill-point ledger gate proves it)
Does NOT survive	Permanent leader loss before followers caught up (the un-shipped tail dies with it)	Simultaneous majority loss
Latency cost	One group-commit fsync (~ms-scale; the macOS F_FULLFSYNC tail is the local floor)	+ one ship RTT + the follower's group-commit fsync, pipelined: acks are batch-level, so concurrent writers share the round trip the same way they share fsyncs
Failure mode	5xx only for local faults	Additionally a retryable 503 when the quorum budget (replication.quorum_timeout_ms, default 2000) expires — the JSON body names the laggards, the commit_index, and needed/confirmed counts
Availability	Unaffected by follower outages	Blocks when a majority is unreachable. In a 2-region cluster quorum = leader + THE follower: one follower outage stops all quorum writes (by design — that is what the contract says). 3+ regions tolerate floor((n-1)/2) follower outages

Mechanics, in one paragraph: there is one replicated log (the leader's WAL — m11p2); peers only ever receive fsynced batches by construction. Since m11p3 every follower pushes its durably-applied frontier back to the leader once per apply round (ReportApplied — batch-level, decoupled from ship acks, flowing even when the follower converges by catch-up pull or the leader is quiet); ship acks additionally carry the same frontier as an instant floor hint. The leader folds both into per-peer durable marks; the commit index is the k-th largest mark (k = floor(n/2)), and ack=quorum responses gate on it passing the write's seqno — awaited asynchronously, so quorum waiters never hold threads or starve completions. The index is leadership-scoped: promote resets it to the new stream baseline, and a demoted leader fails its in-flight quorum waiters (it must never claim quorum for a stream it no longer owns).

Every cluster write's success response carries x-tidal-seq — the write's seqno in the replicated log (relayed through gateway forwards). Persist it if you need an exact durability cursor: commit_index >= seq on /cluster/status/local is "this write is majority-durable", regardless of which mode acked it. (The rare dedup-suppressed signal write — an identical event within the WAL's ~60s content-hash window is already durably logged, so no new record exists to track — carries x-tidal-deduplicated: 1 instead, relayed through forwards like the seq header.)

Retry semantics under ack=quorum — read this twice. A quorum-timeout 503 means not confirmed in budget, not not written: the write is in the leader's log and usually commits moments later. Retries are therefore at-least-once:

• /items and /embeddings retries are always safe — idempotent upserts keyed by entity_id.
• /signals retries can double-count the signal's weight when the original did commit (the server stamps each request's timestamp, so the WAL's content-hash dedup window cannot identify a client retry). The distortion is one extra decaying signal per retried timeout — bounded by your retry rate (tidaldb_cluster_quorum_timeouts_total is exactly that budget). Accounting that cannot tolerate it should route through session writes (which carry idempotency keys) or dedup client-side on its own key.
• The laggard names in the 503 are your runbook pointer: a persistent laggard is a down/partitioned region — heal it (§6) or accept leader-ack for the duration (x-tidal-ack: leader).

Rolling upgrades into m11p3 — upgrade the leader first. A pre-m11p3 leader neither serves the ReportApplied RPC nor recognizes x-tidal-ack: it silently applies leader-ack semantics to a quorum request — a durability downgrade the caller cannot see. Upgrade order:

1. Promote leadership off the leader node if needed, upgrade it, promote it back (or simply upgrade the standing leader per §9's restart procedure). From this moment ack=quorum is honored: the commit index rides the m11p2 ship-ack floor hints from not-yet-upgraded followers (correct, just laggier).
2. Upgrade followers one at a time. Each upgraded follower starts pushing ReportApplied and quorum freshness returns to batch-level. (An upgraded follower reporting to a still-old leader is harmless — the report is refused and logged, replication and heal are unaffected.)

Until step 1 completes, treat the cluster as ack=leader-only — do not point ack=quorum traffic at it expecting majority durability.

Other facts unchanged from m11p1/p2: the success code never waits on shipping for ack=leader (sender threads push the WAL flush feed's tail off the request path; replication.batch_max_events/window tune it); a failed fsync errors that write and nothing unfsynced can ship; long-outage data is follower-pulled via StreamSegments (or, when it has rotated past the leader's retained WAL, via the m11p5 FetchSnapshot snapshot transfer); a leader crash is an automatic failover since m11p4 (the survivors elect the up-to-date successor — see §9.1; the vote restriction only elects a node whose log covers every quorum-acked write, which is what makes the zero-acked-loss guarantee hold).

In short: ack=leader = leader durability. ack=quorum = failover-survivable durability, priced at one pipelined replication round trip and majority availability.

WAL segment format across upgrades (m11p4). Segment files carry an 8-byte version header (TSEG + version byte; headerless pre-m11p4 files stay readable — no migration). Three behaviors follow:

• Unreadable segments fail the boot, loudly. A node whose WAL dir holds segments written by an incompatible tidalDB version (or stray/foreign .seg files) refuses to start with WAL segment format unknown: <path> instead of booting with the data invisibly absent (segments=0 — the 2026-06-11 p3 rollout failure mode). Remedy: run a compatible binary, or reseed the node (delete its PVC and let it pull from the leader).
• Unservable catch-up resolves via snapshot transfer (m11p5). A leader that cannot serve a follower's requested range from its retained WAL answers the StreamSegments pull with FAILED_PRECONDITION — "segments not available from seq N; snapshot required", carrying the typed trailer x-tidal-catchup: snapshot-required. Since m11p5 the follower latches a durable reseed_required marker (only on that typed trailer — an ordinary election term-mismatch never latches it fleet-wide) and reseeds itself via the FetchSnapshot snapshot stream on its next boot (or self-restarts if replication.reseed_self_restart is set) — no operator verb, no wipe_data_dir. See §9.1. The marker is surfaced in /cluster/status/local (reseed_required: true) and the tidaldb_cluster_reseed_required gauge.
• Failed pulls self-heal on a timer. A catch-up pull that fails (e.g. the leader's gRPC server not yet ready during a rolling restart) retries every replication.catchup_retry_ms (default 30000) without waiting for a write to re-expose the gap — an idle cluster no longer strands lagged followers.

Downgrade hazard (m11p4): a pre-m11p4 binary reading a header-bearing segment treats the header as a torn tail and may truncate the final segment. Downgrading across the m11p4 boundary requires reseeding the node's WAL.

Membership records + capability gate (m11p5). A kind-4 membership record is a new WAL blob kind. A kind-4 record shipped to a pre-p5 follower is an unknown batch kind → WalError::Corruption → that follower's torn-state receiver halt, permanent across restarts (boot self-heal re-pulls the same record). Both followers halted = a quorum-write outage. To make this structurally impossible, HeartbeatResponse/ReportApplied carry a capabilities bit-field (proto3 zero-default = pre-p5 = incapable) and the leader refuses /cluster/join and every conf-change until all current voters report kind-4 capability — so the first conf-change cannot fire mid-upgrade. Downgrade rule (kind-3 precedent verbatim): once any kind-4 record is in a node's WAL, downgrading it below p5 requires a reseed. Complete the binary upgrade across all voters before adding, removing, or replacing a node.

9. Failover (multi-process)

9.1 Automatic failover (m11p4 — the default)

"A machine died" is a non-event. Every node runs a failure detector (leader heartbeats every election.heartbeat_interval_ms, default 300) and a Raft-style election (pre-vote + vote, randomized election.election_timeout_{min,max}_ms, default 1500–3000). Kill the leader and the survivors elect a successor — typically in under one second with the defaults, bounded well inside 10s — with zero operator verbs and zero acknowledged-write loss (the vote restriction only elects a node whose log covers every quorum-acked write; the tier-3 cluster_election.rs::mp_auto_failover_writes_resume_zero_acked_loss gate proves it across repeated random kill points under ack=quorum load).

What the operator sees:

• /cluster/status/local carries term (the election term, 0 = the pre-election "topology era"), role (leader/follower/pre-candidate/ candidate) and quarantined.
• During the brief leaderless window, writes return a retryable 503 naming the election (leader: "none (election in progress)" plus the responding node's term); clients retry and land on the new leader.
• A restarted ex-leader can never re-claim leadership from its topology file: its durable election state (data_dir/election_state) boots it as a follower, and every replication RPC is term-fenced — a deposed leader's ships, heartbeats and frontier reports are rejected until it rejoins the current term (the §1.4-1 split-brain incident is closed by construction; proven by mp_fenced_ex_leader_restart_cannot_write).
• A leader that loses contact with a majority steps down within election.leader_lease_ms (default 900) and stops accepting writes: ack=leader writes during a minority partition are bounded by the lease, and ack=quorum writes were never at risk.

Divergent suffix / quarantine → automated reseed (m11p5). A node that held leader-acked (never quorum-acked) writes when it died can rejoin into a cluster that elected past them. It detects this at term-join and quarantines: it serves status (quarantined: true, metric tidaldb_cluster_divergence_quarantined) and keeps voting, but refuses the data plane. Since m11p5 recovery is automatic and full-history — no wipe_data_dir: the quarantine latches the durable reseed_required marker, the node reseeds via the FetchSnapshot snapshot stream on its next boot (or self-restarts if replication.reseed_self_restart is set and quorum can be sustained without it), rejoins clean, and the divergence gauge clears. This is strictly leader-ack-only data, within the documented ack=leader crash contract (§8). The tier-3 cluster_reseed.rs proves the full quarantine → marker → restart → reseeded → gauges-cleared loop.

The three-way term-join rule (m11p5). The divergent-suffix check above is one of three outcomes a node reaches when it joins a new term, comparing its own stream position against the leader's election-time position (prev_log, carried on the heartbeat):

• own > prev_log → divergent suffix → quarantine (above);
• own < prev_log → the node is genuinely missing committed-era history that the new stream's baseline jump would silently skip (the p4 carried hazard) → it latches reseed_required and reseeds via snapshot (no silent gap);
• own == prev_log (or a within-term rejoin) → clean, catch-up via the stream.

A snapshot-installed node always joins clean by construction (its WAL is the leader's copy, so its tail term equals the leader's).

Failure mode: write-burst false-partition (the headline m12 fix — rc7).

• SYMPTOM. Under a sustained 1536-D ack=quorum ingest burst, ack=quorum writes start 503-storming and do not self-heal. The breaker gauge tidaldb_cluster_peer_breaker_state shows BOTH followers stuck at 1 (Open) while they are actually alive — still heartbeating, still applying. The commit index stalls because the leader believes it cannot ship to a majority.
• CAUSE. A follower's CPU-heavy HNSW apply (each apply an ef_construction=400 insert at 1536-D) momentarily starves its transport runtime, so a leader ship RPC misses the 10s request deadline. Pre-rc7 that tonic DeadlineExceeded was counted as a transport failure (record_failure) and opened the breaker even though the peer was alive and heartbeating — both followers' breakers latched Open, commit stalled, quorum writes 503-stormed with no self-heal.
• FIX. Shipped in m12-writeburst-rc7 (tidal-net record_timeout): a ship deadline opens the breaker only when there is no recent proof of life (last_contact stale ⇒ a genuine blackhole still opens it; DeadlineExceeded / Cancelled route through record_timeout, while a genuine Unavailable still opens immediately). Heuristic-only change — the commit / election / vote paths are untouched.
• RESPONSE if seen on an older image. Roll the StatefulSet to ≥ rc7 (@sha256:171505745b801dcf231b531de6167dbc309a7182957811cbc2228f0a302572b1). Confirm recovery by watching tidaldb_cluster_peer_breaker_state clear back to 0 once load eases and ack=quorum writes stop 503-ing.

To run the pre-m11p4 posture (operator-driven failover, no automatic elections, no check-quorum step-down), set in the topology:

election:
  auto_election: false

9.2 Manual promote: a FENCED TRANSFER (maintenance / override)

POST /cluster/promote { "region": "eu-west" } remains the maintenance verb — but it is now a fenced leadership transfer, not a view flip:

1. With a live leader: the leader waits for the target to hold the full flushed prefix (the catch-up wait IS the drain), then sanctions an immediate election (TimeoutNow); the target wins term+1 and the old leader steps down on first higher-term contact. Response: { ok, leader, term, transfer: "elected" }.
2. With a dead leader: the target campaigns among the survivors directly — the manual override of the automatic path (and the way to choose the successor).
3. The election can refuse a target that lags (its log loses the up-to-date comparison — e.g. the survivor you sampled fell behind between your status read and the vote). The verb 503s naming the cause; promote the other survivor. You can no longer accidentally promote a node that would discard acked data — the m11p3 "max-applied survivor" operator rule is now enforced by the protocol.
4. Promoting the node that already leads is a no-op 200.

The legacy term-0 fan-out promote survives only for clusters that have never elected (mixed-version rollouts mid-upgrade, and the deliberate isolated-node override used by the chaos drills); the first joined election permanently retires it on each node.

Rolling upgrade m11p3 → m11p4: upgrade ALL binaries before relying on auto-failover (pre-m11p4 peers answer vote RPCs with Unimplemented, so no election can reach quorum until a majority is upgraded — the cluster simply keeps its m11p3 behavior until then). The first ELECTED leader journals a kind-3 term-marker WAL record; pre-m11p4 binaries cannot decode it, so do not downgrade a node after the first election without reseeding.

10. Partition drill (multi-process)

The rewritten drill demonstrates both partition mechanisms, exactly as cluster_runbook.rs::runbook_s10_partition_drill scripts them.

1. Baseline. GET /cluster/status; all lag_events: 0, partitioned: false, reachable: true.
2. Inject. Use one of:
   • Real network partition — sever the TCP path peers use to reach the region. The chaos suite (cluster_chaos.rs) uses a root-free in-harness TCP relay proxy (the ROADMAP-sanctioned toxiproxy-style alternative). On a real host you can instead use iptables (Linux) or pfctl (macOS), e.g. iptables -A INPUT -p tcp --dport 9603 -j DROP to blackhole ap-south's gRPC port from a peer. A real cut shows reachable: false in the aggregate status.
   • Simulated ship-skip flag — POST /cluster/partition { "region": "ap-south" } (→ { ok, partitioned: "ap-south" }). The leader stops shipping to ap-south without touching sockets; the aggregate status shows partitioned: true.
3. Write through it. Send several POST /signals. Each must still 204 (the leader-durable contract holds). Watch ap-south's lag_events climb while its applied_events stalls — the leader's ships to it are dropped.
4. Read the stale follower. GET /feed?region=ap-south (or read ap-south's gateway directly) returns the pre-partition view — eventual, not strong, read consistency. The operator console survives a real partition because clients talk to each region's gateway directly.
5. Scatter-gather degradation. While partitioned, GET /sharded/feed returns 200 with degraded: true and ap-south in unavailable_shards — never an error, and the live shards' items are still returned.
6. Heal. Clear the cut (heal the proxy / drop the firewall rule, or just call heal for the simulated flag), then POST /cluster/heal { "region": "ap-south" }. Re-issue heal until /cluster/status shows ap-south at lag_events: 0 — the gRPC circuit breaker opened during the partition (threshold 5, reset 30s), so the first heal may ship into an open breaker and no-op. This is genuine production behavior, not a flag.
7. Verify convergence. GET /cluster/status shows ap-south lag_events: 0, partitioned: false, reachable: true; /sharded/feed is no longer degraded; feed scores on ap-south match the leader to within float tolerance (no loss, no duplication).

11. Shutdown

Send SIGTERM (or SIGINT / Ctrl+C, or stop the container). The server flips readiness to 503 (so a load balancer stops routing to it), drains in-flight requests, then drops the region's TidalDb shutdown path: checkpoint in-memory signal state → flush storage → write the WAL checkpoint marker + fsync → join the WAL, sweeper, checkpoint, text-syncer, and replication-receiver threads. The drop is idempotent; the process exits 0. You will see region cluster node shutdown: database closed (checkpoint + WAL fsync) (multi-process) / cluster shutdown: closing all nodes (checkpoint + WAL fsync) (single-process).

On restart with the same --data-dir, the region recovers its pre-shutdown state from the WAL (verified by cluster_runbook.rs::runbook_s11_shutdown_and_wal_recovery: SIGTERM → exit 0 → restart → the same items are served). In multi-process mode, the restarted node rejoins the cluster and pulls anything it missed while down via its boot-time StreamSegments catch-up request — no operator verb needed (m11p2; /cluster/heal remains the explicit lever for paused peers). This is the per-node step of a rolling upgrade (promote leadership off the node, SIGTERM, restart on the same data dir with the new binary, heal) — see cluster_lifecycle.rs::mp_rolling_upgrade_no_loss_no_stall, which proves zero acknowledged-write loss across a full rolling upgrade under load.

Node-replace drill (m11p5)

Replacing a dead or recycled node is kubectl delete pod — no topology edit, no operator verb:

• PVC retained (the data dir survives the pod): the replacement boots on the same data dir and converges via its boot-time StreamSegments catch-up — the ordinary restart path above. If it rotated past the leader's retained WAL while down, it reseeds via FetchSnapshot automatically (the reseed_required marker + snapshot path, §9.1).
• PVC deleted + pod deleted (fresh node): the replacement comes up empty and seed-joins (§1b) — it --seeds a survivor, FetchSnapshots the current state, and the leader auto-promotes it back to Voter. Under the k8s/cluster/ StatefulSet this is the default: the pod's args carry --seed against the headless Service, so a recreated pod rejoins with no human in the loop.

Add a node: scale the StatefulSet up (pod N≥3 seed-joins as a learner and auto-promotes). Remove a node: POST /cluster/members/remove first (§6), wait for the record to quorum-commit, then scale down (lowest ordinal last).

12. Security (m11p7): mTLS, rotation, identity, audit, rate limits

The cluster does not trust the network. Everything here is opt-in — absent the grpc_tls block and the cluster key, the cluster behaves exactly as pre-m11p7 (plaintext, hint-only marker, no audit/limit). A reference (k8s) deployment turns it all on.

12.1 mTLS (gRPC replication) — the default posture

• Configure the grpc_tls block per region (ca_cert, server_cert, server_key, client_cert, client_key). The gRPC server then REQUIRES a client cert chained to the cluster CA (mutual TLS): a foreign pod with no cert, a cert from another CA, or a plaintext probe fails the TLS handshake and never reaches an RPC.
• No grpc_tls ⇒ plaintext, with a loud startup WARN on both the server and the client. Acceptable only on a trusted single-host / loopback topology. To serve plaintext intentionally there is nothing else to set — the WARN is the signal that you are on the insecure path.
• The inter-node HTTP plane (forwards, broadcasts, scatter, status, seed-join) is served over TLS with the SAME cert and dials https:// with the cluster CA whenever grpc_tls is set, so enabling it gives zero plaintext inter-node links on both planes at once.

12.2 Cert + bearer rotation WITHOUT restart

• A background poller (TIDAL_ROTATION_POLL_MS, default 30000) content-hashes the cert files and the credential files; on a change it atomically swaps the served cert (in-flight TLS sessions keep their negotiated keys — zero dropped requests) and rebuilds the outbound peer channels.
• Procedure: issue a new cert under the same CA (cert-manager renewal, or re-run scripts/gen-cluster-certs.sh and re-apply the Secret) — the files change in place, the poller swaps within one interval, no pod restart. Verify with tidaldb_cluster_* logs (TLS material rotated…) or the tidal_audit /tracing stream.
• The bearer (TIDAL_API_KEY_FILE) and the cluster key (TIDAL_CLUSTER_KEY_FILE) rotate the same way. Use FILE mounts (not inline env) so a Secret rotation is picked up live. During a CA roll, keep both old and new CAs trusted for one cycle (CA-overlap) so in-flight connections complete.

12.3 Per-node identity + the marker

• Set a shared cluster key (TIDAL_CLUSTER_KEY / TIDAL_CLUSTER_KEY_FILE, any random string — BLAKE3-derived to the MAC key). Each node then mints a signed x-tidal-node-token on every forward/broadcast; the receiver verifies it. This gives inter-node calls a verifiable node identity and is the defense-in-depth layer beyond the shared bearer.
• With a cluster key configured, the x-tidal-internal marker is honored ONLY from a verified sibling: a request that sets the marker without a valid node token is rejected 403 (the marker is a routing hint, never an auth bypass). Never hand the cluster key to external clients.

12.4 Admin audit log

• promote / partition / heal / join / member-remove / reseed each emit one structured record: {principal, verb, target, term, outcome}. The principal is the verified node (node:<id>) for inter-node calls or external for an operator with the bearer.
• Sinks: a tidal_audit tracing target (always — capture it in your log pipeline), plus an append-only JSONL file when TIDAL_AUDIT_LOG=<path> is set. Recorded on the operator-originated leg only (no double-record on a forwarded re-apply).
• At-rest encryption of the JSONL file is delegated to the volume — mount TIDAL_AUDIT_LOG on an encrypted PV (or a gVisor/LUKS-backed volume); the server does not encrypt it in-engine.

12.5 Per-principal rate limits

• TIDAL_RATE_LIMIT_RPS (+ optional TIDAL_RATE_LIMIT_BURST, default 2×) caps per-principal request rate; a deny is 429 + Retry-After. Off by default.
• Verified sibling nodes are EXEMPT — replication/forward traffic is never throttled by the external-client budget. Today external callers share ONE bucket: TIDAL_RATE_LIMIT_RPS is an AGGREGATE cap across all external clients, not a per-client budget — set it to your total external ceiling, not a per-client one (one noisy client can consume it). A future multi-key registry adds per-key buckets.

12.6 Foreign-pod / negative behavior (what an attacker on the network sees)

Attempt	Result
Ship a gRPC segment without a cluster client cert	TLS handshake fails — no RPC dispatched
Call an internal HTTP route without trusting the cluster CA	TLS handshake fails — no route reached
Set x-tidal-internal without a valid node token (key configured)	403 — marker honored only from a verified sibling
Call a protected route without the bearer	401 (unchanged)

13. Coordinated backup / restore + point-in-time recovery (m11p8)

See also the dedicated DR runbook: docs/runbooks/disaster-recovery.md for full disaster-recovery procedures (region/cluster loss, restore drills, RPO/RTO).

The building blocks: the engine's crash-consistent create_backup, the WAL archive (wal.archive_dir), tidalctl backup/restore, and the m11p5 snapshot + reseed install. Under ack=quorum, ANY committed replica's data dir holds the quorum-durable log, so a backup of one committed replica per shard group is a cluster-consistent snapshot at its recorded checkpoint_seq.

13.1 Enable the WAL archive (point-in-time recovery)

Set wal.archive_dir in the topology (or TidalDb::builder().wal_archive_dir(path) for the embedded engine). Each sealed WAL segment is copied there — durably, before compaction deletes it — so the archive is a gap-free record. Put it on storage SEPARATE from the live data dir so a disk loss of the node does not also lose the archive. Segment filenames encode shard + first_seq, so co-located groups share one archive dir without collision.

wal:
  archive_dir: "/archive/tidaldb"   # durable, off-node storage

13.2 Coordinated backup drill

1. Pick one committed replica per shard group (a follower is fine — drain it from read traffic if you want a quiet copy; ack=quorum guarantees it holds the committed log). Stop it (or snapshot its volume).
2. tidalctl backup --path /data/<node> --out /backups/<cluster>-<ts>/shard-<id> — writes a recursive copy + BACKUP_MANIFEST.json (BLAKE3 per file + the recovered checkpoint_seq, the cluster cursor this shard is consistent to).
3. Repeat per shard group. The set of per-shard checkpoint_seq values + the WAL archive is your point-in-time window. Restart the replica; self-heal/catch-up reconverges it.

13.3 Restore drill (timed)

1. tidalctl restore --from /backups/<cluster>-<ts>/shard-<id> --path /data/<new-node> — verifies EVERY file's BLAKE3 against the manifest BEFORE writing, and refuses a non-empty target (it never overwrites a live data dir).
2. Point a stopped node at the restored dir and boot it. Under ack=quorum its group's followers catch up via the live stream; promote it if it is the group's chosen leader (the highest-applied survivor rule, §9).

Recovery granularity today is full-snapshot-to-frontier, NOT arbitrary point-in-time. A restore recovers a node to the backup's checkpoint_seq, then the live stream reconverges it to the cluster's CURRENT frontier. The WAL archive (wal.archive_dir) is the durable, gap-free primitive that backs a future point-in-time replay — but it is write-only today: no tool replays archived segments up to a chosen target seq. tidalctl recover is verify-only (--verify-only; an in-place replay mode is reserved for a future release), and tidalctl restore copies a full snapshot with no seq bound. So the inputs you retain for a future PITR are the per-shard checkpoint_seq set + the archive; treat them as the window, not a one-command restore-to-instant. Do not plan an incident around seq-bounded replay until a tidalctl replay --until <seq> verb ships.

Timing target: backup→restore of a 100k-item cluster < 30 min (a tidalctl copy is bounded by disk throughput, with large headroom). The Ref-A timed figure is a k3s line item (the standing M11 access caveat).

14. Rolling upgrade + version skew (m11p8)

Nodes carry a build version on the wire (HeartbeatRequest.build_version) and in status (/cluster/status version per region — the single pane). Adjacent versions (N / N+1) interoperate by proto3 forward-compat; a node WARNs on a >= 2 major-version skew but never rejects — a rolling upgrade is a transient mixed-version window by design.

Procedure (one node at a time):

1. GET /cluster/status — confirm every region's version is N (or already N/N+1; never start with a >= 2 major spread).
2. Graceful SIGTERM one follower → it drains (readiness 503 → checkpoint).
3. Restart it on N+1 (same ports, same data dir). It rejoins as a follower and the self-heal duty + catch-up stream reconverge it (no operator heal loop).
4. Repeat for each follower. Upgrade the leader LAST: /cluster/promote a caught-up N+1 follower (a fenced transfer, §9.2), then upgrade the old leader as a follower.
5. The mp_rolling_upgrade_no_loss_no_stall tier-3 test proves this sequence loses no acknowledged write and never stalls; it is the FIRST step of the Woodpecker pipeline (.woodpecker.yaml) — a failure blocks the image build.

Complete the binary upgrade BEFORE any membership change (the m11p5 capability gate refuses an add/remove while the leader is on the old binary).

Performance

Read this table by era. The /signals-throughput row below is the m11p1 signal-write benchmark (3-byte signal writes, small payloads, measured over real localhost processes) — it is NOT the m12 1536-D production shape and must not be cited as the live cluster's read/write ceiling. The m12 reality on the 3-node k3s fleet at the 1536-D production corpus (see docs/profiling/m12-cluster-deploy-findings.md):

• Reads: p99 7.97–11.47 ms at 100–500 rps; G1 (p99 ≤ 10 ms) MET at 100k with recall@10 0.9989 (clean index). Read ceiling ~1000 rps clean (~1500 rps saturated), CPU-bound — beyond it a node sheds/errors.
• Writes: the peach mix is write-heavy and the write knee is ~250 rps on this fleet (1536-D ack=quorum ingest; each apply is an HNSW insert). This is why the soak (§15) was re-scoped to 200 rps on 2026-06-19.

The legacy m11p1 signal-write benchmark (measured over real localhost processes):

Operation	SLA	Measured (p99 / typical) — m11p1 signal-write benchmark, pre-m12, NOT the 1536-D shape
Replicated /signals throughput (m11p1, 3 nodes, release build)	≥ 2,000/s	4,534 signal-writes/s within SLO on the ramp (knee ~5.5k/s); 2,739/s sustained 10 min (1.65M writes, 0.35% errors); was ~90/s pre-m11p1. Superseded for production sizing by the m12 1536-D figures above.
Replication lag under that load (m11p1)	< 2s	≤ 103 events (~40ms) across the 10-min sustain; ≤ 377 events on the 5k/s ramp (follower group-commit coalescing)
Cross-region replication (write → follower applied)	< 2s	~110–133ms p99 (m8p10)
Failover (/cluster/promote → first successful write)	< 10s	~31–34ms
CRDT reconcile (merge+apply, each side)	< 100ms	0–1ms

Write-latency note: signal p50 ≈ 17–25ms with a p99 tail of 160–220ms on macOS, where F_FULLFSYNC averages ~7.4ms with a 10–50ms tail (tidaldb_cluster_wal_fsync_us — 0% complete under 1ms). On Linux fdatasync volumes the same pipeline's fsync floor is far lower; validate the p99 gate on the reference environment, and tune wal.batch_timeout_ms against the measured fsync histogram.

15. Continuous correctness: chaos suites + soak (m11p9)

Correctness is a pipeline, not a one-time gate. The tier-3 suites run REAL OS processes; the new fault classes inject REAL faults.

Run the chaos suites locally (serial — they bind fixed ports and spawn processes, so suites must not overlap):

cargo test -p tidal-server --features "cluster-e2e fault-injection" \
  --test cluster_faults -- --test-threads 1 --nocapture

cluster_faults covers the fault classes the partition/crash/skew suites lacked:

Test	Fault	Asserts
mp_disk_full_follower_degrades_no_acked_loss	follower WAL ENOSPC	receiver halts (degraded, alive), healthy majority keeps acking quorum, zero acked loss, restart recovers to parity
mp_slow_fsync_follower_lags_but_quorum_holds	one follower's fsync slowed	fast follower supplies quorum, slow node lags then converges, no loss
mp_slow_fsync_both_followers_force_honest_quorum_timeout	both followers slowed below the budget	ack=quorum → retryable 503 naming laggards; ack=leader → 204; recover
mp_asymmetric_partition_no_split_brain_no_loss	inbound to one node severed (outbound up)	pre-vote + check-quorum hold the leader; single-leader-per-term; no loss

Fault knobs (behind the fault-injection feature — compiled OUT of the production image, inert until armed): TIDAL_FAULT_FSYNC_DELAY_MS=<ms> slows every durable WAL fsync; TIDAL_FAULT_DISK_FULL_AFTER_BYTES=<n> fails segment writes with ENOSPC after n cumulative bytes this process lifetime (arm it on a node at restart to fail after n bytes of post-restart writes). NEVER set these on a production node.

Fleet soak with regression gates (tidal-stress):

tidal-stress --target https://<cluster-node>:9500 \
  --ca-cert /etc/tidaldb/tls/ca.crt --skip-seed \
  --corpus 100000 --embedding-dim 1536 \
  --ramp "200:3600" --mix peach --json-summary soak.json \
  --max-error-pct 1 --max-p99-ms 150 --fail-on-knee

The Ref-A gate is 200 rps (re-scoped 2026-06-19). The peach mix is write-heavy and the measured write knee on this full-placement three-node fleet is about 250 rps. The one-hour gate therefore uses the measured 200-rps stage, not the retired pre-m12 3900:3600 figure. The :9500 plane is TLS.

--fail-on-knee, --max-p99-ms, and --max-error-pct make the load process exit non-zero on regression. The in-cluster tidal-soak-nightly CronJob adds the recovery half of the gate: one ledger row per UTC date, one-minute pod UID and restart-count evidence immediately before and after the exact load window, and an atomic streak.tsv. A date advances the streak only when the load passes, all three pods have fresh boundary samples, and no UID or counter changes. Duplicate dates, gaps, stale evidence, pod replacement, and load failure cannot inflate the 30-night streak.

Nightly CI (.woodpecker.yaml, cron nightly — Woodpecker, never GitHub Actions) remains the bounded code-regression signal: tier-3 chaos with elevated kill-points, then a local gated soak. It is not the Ref-A calendar gate unless TIDAL_SOAK_TARGET and its duration/rate are explicitly pointed at that fleet. The guarantee→test map is docs/planning/milestone-11/guarantee-traceability.md.

16. Vector index divergence

Target of the runbook_url on TidalDBClusterVectorIndexDiverged, TidalDBClusterBlobApplyFailing, and TidalDBClusterPeerShipFailing.

Symptom. Replicas of one shard group answer the same vector query differently: an entity is the top hit on one replica and absent on the others. Health surfaces stay green throughout — lag_events counts WAL apply, not blob apply, so a blob that never lands moves no lag number.

16.0 First: rule out the two benign causes

Both look exactly like a replication defect. On 2026-08-30 the first one consumed several rounds of investigation and produced a retracted durability incident report.

1. The write used /sharded/*. That surface hash-partitions and applies to the owning region's local store with no WAL append, so it does not replicate — by design (tidal-server/src/cluster/node.rs:8828-8829; sharded_write_embedding → ShardReplica::apply_embedding_local). An entity written that way living on exactly one node is the contract, not a fault. Only the non-sharded surface (/items, /embeddings, /signals) rides the leader WAL relay and replicates. Check: re-probe with /embeddings and see whether parity holds. It settles in seconds what measuring the sharded path more precisely never will.
2. A rolling deploy is in flight. A restarted pod rebuilds its index from store and legitimately reads a different count until it finishes. This is why the alert carries for: 15m.

Both are covered by tests: mp_embedding_is_searchable_on_every_replica_without_restart and mp_sharded_surface_writes_are_local_to_the_owner (tidal-server/tests/cluster_sharding.rs).

DO NOT restart a pod first

rebuild_from_store runs at open and re-derives the whole index from the durable store, which converges the counts and destroys the evidence. Capture every number below before touching a pod. The restart is a diagnostic step with a specific meaning (§16.3), not a remedy.

16.1 Read the counts per shard group

tidaldb_usearch_vector_count is per shard group, not per node: the node's metrics-owner group renders unlabeled and each co-located group renders shard="N". The owner is deterministically the same group on every node, so compare within a label set, never across:

kubectl -n observability port-forward svc/vmsingle 18428:8428 &
curl -s --get --data-urlencode \
  'query=tidaldb_usearch_vector_count{namespace="tidaldb-cluster"}' \
  localhost:18428/prometheus/api/v1/query | jq -r \
  '.data.result[] | "\(.metric.pod) shard=\(.metric.shard // "0(owner)") \(.value[1])"'

Replicas of one group MUST agree. A spread that persists past a rolling deploy is real.

16.2 Localise it with the blob ledger

# on the node that ACCEPTED the write:
#   tidaldb_cluster_blobs_originated_total{kind="embedding"}
# on every peer:
#   tidaldb_cluster_blobs_applied_total{kind="embedding"}
#   tidaldb_cluster_blobs_apply_failed_total{kind="embedding"}

With RF3 a healthy cluster shows applied ≈ (RF-1) × originated in aggregate — one apply per follower. Read it per (pod, kind), not as a fleet-wide subtraction.

reading	meaning	next step
apply_failed > 0	the receiver rejected a record and halted that stream rather than skipping it	read that pod's logs for the apply error; the stream is stalled, not silently lossy
peer's applied never advances while the writer's originated does	the record is not arriving — enqueue or ship	§16.4
applied advances but that group's usearch_vector_count lags	it arrived and was not indexed	index-gap; the store is intact, a restart recovers it

16.3 The decisive test (only after 16.1 and 16.2 are captured)

Confirm kubectl -n tidaldb-cluster get pdb tidaldb -o jsonpath='{.status.disruptionsAllowed}' is 1, run a quorum-write probe (POST /items with x-tidal-ack: quorum → expect 201), then restart one follower that lacks the entity — never the leader.

The probe MUST go through the replicating surface, and /sharded/* can never serve as one. This step used to say POST /sharded/items → expect 201. A /sharded/* write is applied to the owning region's local store with no WAL append, so it never enters the replication stream and no quorum is ever consulted — its 201 means "one region accepted a single-copy write", which is true whether quorum is intact, degraded, or gone. A green probe therefore carried no information about the property it was run to check.

Do not reintroduce it on the reasoning that it answers faster: speed is exactly what it buys by skipping the WAL append, and skipping the WAL append is precisely what makes it blind. Only a write that appends to the leader WAL and waits for a quorum ack can verify quorum, which is POST /items|/embeddings|/signals with x-tidal-ack: quorum. (Since the opt-in landed, /sharded/* also returns 400 without x-tidal-ack: local — see §7.)

This probe was used as the between-step safety check of a staged rolling deploy on 2026-08-30; that run's quorum verification must be treated as never having happened. See k3s-fleet/cluster-state.yaml.

• entity becomes retrievable after the rebuild ⇒ its durable store had it, only the live index was missing it. Data was safe.
• entity still absent ⇒ its store never had it. Before calling this a durability event, re-check §16.0: for a /sharded/* write this is the intended outcome and the store was never supposed to hold it. If the write went through the non-sharded surface and the entity is still absent after a rebuild, that is a durability event — the entity exists on fewer replicas than the replication factor claims. Record the affected count and escalate.

Re-probe the quorum write afterwards and confirm 201 again.

16.4 Check the ship path

kubectl -n tidaldb-cluster logs tidaldb-<leader> -c tidaldb | grep 'ship sender'

batch ship failing; retrying every 100ms … consecutive_failures=N means the leader cannot reach that peer. tidaldb_cluster_peer_breaker_state is not a reliable signal here — on 2026-08-30 it read 0 (closed/healthy) through 2697 consecutive failures. Trust tidaldb_cluster_peer_ship_failures_total's rate, which TidalDBClusterPeerShipFailing now alerts on.

Also note GET /cluster/status's regions[] block can report peers partitioned: true, reachable: false with full lag_events while the same response's shards[] and peer_acked_seqno show complete convergence. The shards[] block is authoritative; regions[] is a stale legacy view.

16.5 Expected convergence window

A write is not instantly visible on every replica, and that is not this bug. Ship and apply normally complete in well under a second on an in-cluster (low-RTT) deployment. "Still converging" vs "diverged" is decided by whether applied on the lagging peer is advancing: if it is, wait; if it is flat while the writer's originated climbs, it is the failure in §16.2. The alert's for: 15m exists so a rolling deploy's legitimate rebuild window never pages.

Cross-references

• Kubernetes deployment — docs/runbooks/kubernetes.md (the hardened standalone single-replica set in k8s/, and the multi-region cluster reference in k8s/cluster/ — one StatefulSet + headless Service peer discovery + PDB, with --seed-based scale and kubectl delete pod node-replace, shipped in m11p5).
• Disaster recovery — docs/runbooks/disaster-recovery.md (backup/restore/PITR procedures, region/cluster loss, RPO/RTO; the operational companion to §13).
• Server deployment guide — docs/guides/server-deployment.md (standalone and cluster launch, config, env, health probes).
• Monitoring & alerts — docs/ops/monitoring.md (Prometheus scrape config, replication-lag and WAL metrics, recommended alerts; per-region replication lag is observable via /cluster/status lag_events; remember the /metrics endpoint is unauthenticated — bind it internally).
• Roadmap / cluster status & known gaps — docs/planning/ROADMAP.md and docs/roadmap-to-cluster.md for the M11 cluster status — quorum-ack writes (m11p3), automatic failover (m11p4), membership/discovery/elasticity (m11p5), security hardening (m11p7), observability + operations (m11p8), and continuous correctness (m11p9) all shipped; sharding × replication (p6) data plane in progress.
• API & schema reference — API.md, QUICKSTART.md, and the live /openapi.json document.
• Scope & vision — VISION.md.