tidaldb/docs/runbooks/cluster.md

tidalDB Cluster Runbook

Operating the multi-region tidal-server cluster surface: launch (single-process dev fabric and the multi-process region nodes), the operational API, replication transport facts, failover and partition drills, and the honest write-durability contract.

STATUS: QUORUM-ACKED HA IS LIVE — RUNNING IN PRODUCTION ON k3s

Quorum-ack writes (m11p3) and automatic election/failover (m11p4) are LIVE and deployed. The reference cluster runs in production on k3s as a single StatefulSet, full-placement RF3 deployment: ns tidaldb-cluster, replicas: 3, every pod a region (tidaldb-0/1/2) hosting all three shard groups, image m12-writeburst-rc7. ack=quorum is the cluster deployment default (topology replication.ack: quorum) and leader election + failover are automatic — kill the leader and the survivors elect a successor with zero operator verbs and zero acknowledged-write loss. See the live production topology in §1 and the deployed shape in §3a.

Honest caveats that still hold: cluster mode replicates global retrieval signals only (no per-user personalization on the /signals route — see the personalization note in §3); and both launch modes still refuse to start unless you explicitly opt in with --experimental-cluster or TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 (the k8s manifests set the env var). The two launch shapes below remain:

1. Multi-process (--region) — real process isolation (the production shape). Each tidal-server cluster --region <name> process owns exactly one region: one TidalDb, one GrpcTransport whose server binds this region's grpc_addr and whose peers are every sibling region's real grpc_addr. Processes peer over real gRPC and forward over real HTTP, so a crash of one region's process takes down only that region — the survivors keep serving. This is genuine process (and, across hosts, host) isolation. It is verified end-to-end by the tier-3 suites (cluster_multiproc, cluster_chaos, cluster_lifecycle, cluster_runbook) over real OS processes with real network-partition injection.

2. Single-process (no --region) — the dev/demo default. Every region runs inside one process. Replication still traverses the real tidal-net gRPC transport on loopback (faithful multi-region semantics over a real wire), but there is no process isolation: a crash, OOM, or host failure takes the whole "cluster" down at once. This is a development / staging / demo fabric and a correctness harness for the replication paths — not production HA. It remains the default because it needs no per-region topology addresses and no process orchestration.

Durability / leadership / membership facts (both modes):

• Quorum durability is the cluster default, per-request overridable. replication.ack: quorum is the topology default the k8s reference cluster ships, so a write succeeds only once a majority of the replica set durably holds it (m11p3), surviving permanent leader loss. A caller can downgrade a single write to leader-durable with x-tidal-ack: leader (storage + WAL fsync; follower ship off the request path). See §8.
• Leadership is automatic (m11p4). Every node runs a failure detector and a Raft-style election (pre-vote + vote + check-quorum + fenced transfer): kill the leader and the survivors elect a successor — typically under a second with the defaults — with zero operator verbs and zero acknowledged-write loss (see §9.1). /cluster/promote is now a fenced transfer for maintenance/override, not the availability mechanism. election.auto_election: false preserves the pre-m11p4 operator-driven posture.
• Membership is elastic and addresses are DNS names (m11p5). grpc_addr is an advertised hostname or IP (DNS-resolved on every reconnect); nodes join online via --seed and catch up via FetchSnapshot + the StreamSegments stream; add/remove ride kind-4 membership records on the replicated log (see §1b, §3, §6, §9.1).

Production deployment today is the multi-process cluster on k3s described above — the single-StatefulSet full-placement RF3 reference in k8s/cluster/ (see §1, §3a, and kubernetes.md). The single-process shape below remains the dev/demo fabric and replication-correctness harness, and a single tidal-server standalone node (see server-deployment.md) stays valid for deployments that do not need multi-region / ack=quorum.

Both modes refuse to start unless you explicitly opt in with either the --experimental-cluster flag or the TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 environment variable. On start each emits a loud, mode-specific WARN restating exactly what it does and does not provide. Do not wire either into a production load balancer.

Scope: what cluster mode does and does not own

tidalDB owns retrieval and ranking only. Cluster mode adds multi-region replication of that, nothing more. It does not generate embeddings, store video/blobs, run a CDN or transcoder, do auth/moderation/payments, or provide a distributed consensus log. Bring your vectors; tidalDB retrieves and ranks over them. See VISION.md for the scope boundary.

Prerequisites

• Rust toolchain ≥ 1.91 if running directly (the Docker build pins rust:1.91-bookworm).
• protobuf-compiler (protoc) and a C++ toolchain (g++) on the build host — tidal-net's build script compiles the WAL-shipping .proto, and USearch's HNSW core is C++. The Docker image installs both.
• Docker 25+ if running via container.
• For single-process mode: one HTTP port (default 9500); each follower region also binds an OS-assigned loopback port for its gRPC replication server unless you pin one in the topology (see §3).
• For multi-process mode: per-region grpc_addr and http_addr declared in the topology, and the ports they name available on each host (see §3).

1. Launch the cluster locally

Cluster mode is gated. Pass --experimental-cluster (or set TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1) or the server exits with a mode-specific error explaining why.

Live production topology (the deployed shape on k3s — 2026-06-19). The reference cluster is ONE StatefulSet tidaldb in namespace tidaldb-cluster, replicas: 3 = 3 pods = 3 regions = 3 voters (tidaldb-0/1/2), full-placement RF3: every pod hosts all three shard groups (the 3-group shards: block in §3a is the DEPLOYED shape, not optional). Each group's data lives under /data/db/shard-0000N on the pod's one PVC. Each pod binds one gRPC port per group — shard 0 → 9601, shard 1 → 9602, shard 2 → 9603 (derived node base port + shard id). The HTTP plane is :9500 over HTTPS with inter-node mTLS (m11p7 — every probe/curl uses https://); Prometheus /metrics is :9091. replication.ack: quorum is the deployment default and election/failover is automatic (election.auto_election: true). Live image m12-writeburst-rc7 (@sha256:171505745b801dcf231b531de6167dbc309a7182957811cbc2228f0a302572b1). Networking is a headless peer Service tidaldb-peers (per-pod DNS tidaldb-N.tidaldb-peers.tidaldb-cluster.svc.cluster.local) plus a ready-only client Service tidaldb (the seed-join discovery target). See k8s/cluster/statefulset.yaml and k8s/cluster/topology-configmap.yaml.

1a. Single-process (dev/demo default)

TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
cargo run -p tidal-server -- \
  cluster \
  --listen 127.0.0.1:9500 \
  --schema tidal-server/config/default-schema.yaml \
  --topology tidal-server/config/default-cluster.yaml \
  --experimental-cluster

The default topology spins up three regions (us-east, eu-west, ap-south) with us-east as leader, all inside one process. On a clean start you will see a WARN line stating this is experimental and single-process, an info line per follower (follower gRPC transport ready), and finally listening on http://127.0.0.1:9500.

1b. Multi-process (one process per region)

Pass --region <name> (or set TIDAL_REGION) to run only that region in this process. The topology must declare a per-region grpc_addr and http_addr for every region (see §3); siblings reach each other over those. Launch one process per region — typically one per host, each with its own --data-dir. --data-dir is REQUIRED in multi-process mode (m11p2): the durable WAL is the replicated log itself — it is what ships to peers and what serves their catch-up streams — so a node without one is rejected at startup:

# Region us-east (the initial leader) on host A
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
tidal-server cluster --experimental-cluster \
  --region us-east \
  --listen 0.0.0.0:9501 \
  --schema /etc/tidal/schema.yaml \
  --topology /etc/tidal/topology.yaml \
  --data-dir /var/lib/tidal/us-east

# Region eu-west on host B
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
tidal-server cluster --experimental-cluster \
  --region eu-west --listen 0.0.0.0:9502 \
  --schema /etc/tidal/schema.yaml --topology /etc/tidal/topology.yaml \
  --data-dir /var/lib/tidal/eu-west

# Region ap-south on host C
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
tidal-server cluster --experimental-cluster \
  --region ap-south --listen 0.0.0.0:9503 \
  --schema /etc/tidal/schema.yaml --topology /etc/tidal/topology.yaml \
  --data-dir /var/lib/tidal/ap-south

Every process parses the same topology file; RegionIds are assigned by declaration order, so all processes agree on the region→id mapping. Each process binds its own region's grpc_addr/http_addr and dials its siblings' addresses. The --listen HTTP address is the public gateway for that region.

Seed-join boot (m11p5 — adding a node without editing the topology)

A node can join an existing cluster online, without appearing in any declared topology, by contacting a running peer as a seed:

TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
tidal-server cluster --experimental-cluster \
  --region eu-2 \
  --listen 0.0.0.0:9504 \
  --schema /etc/tidal/schema.yaml \
  --topology /etc/tidal/topology.yaml \
  --data-dir /var/lib/tidal/eu-2 \
  --seed http://10.0.1.10:9501 \
  --seed http://10.0.2.10:9502 \
  --advertise-grpc eu-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9600 \
  --advertise-http  eu-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9504 \
  --metrics 0.0.0.0:9091

Flag	Meaning
--seed <url> (repeatable)	One or more seed HTTP base URLs. The joiner polls each for the current leader, then POST /cluster/joins through it. Any reachable seed works; list a few for resilience.
--advertise-grpc <host:port>	This node's advertised gRPC address — what siblings dial. Required with --seed (the joiner has no topology entry of its own). DNS-capable.
--advertise-http <host:port>	This node's advertised HTTP gateway — what peers forward writes/status to. Required with --seed.
--metrics <addr>	Prometheus /metrics bind (the joiner has no topology metrics_addr).

What happens: the joiner skips the topology's "every region declared" gate, learns its roster + assigned id + current term from the seed's join response, appends a Learner record to the replicated log (the leader answers only after it is quorum-committed), persists the roster to a durable membership cache and the term to election_state (persist-before-act), installs a snapshot when it is behind the leader's retained WAL, then streams the live tail. The leader's auto-promotion duty flips it Learner → Voter once it is within replication.learner_promote_lag (default 1024) of the flushed frontier. A restart boots from the cache without the seed.

A --seed boot still requires the local topology/config file for the behavioral knob blocks (replication:, wal:, election:, timeouts:, grpc_tls) — its regions: list is ignored for the roster (the join response is the roster), but a bare --seed with neither --topology nor TIDAL_CONFIG refuses to boot naming the rule, so a joiner never silently inherits the compiled-in defaults' wrong ack mode, quorum timeout, or election timing. The k8s manifests mount the shared bootstrap ConfigMap on every pod including N≥3, so this costs nothing there.

Capability gate (m11p5 mixed-version safety). The leader refuses /cluster/join and every conf-change until all current voters report kind-4 capability — see §8's downgrade rule. Complete the binary upgrade before adding or removing nodes.

Auth: set TIDAL_API_KEY=<secret> to require Authorization: Bearer <secret> on the data and /cluster/* mutation routes. If it is unset the server runs UNAUTHENTICATED and logs a WARN — never expose an unauthenticated cluster beyond loopback / a trusted VPC. Health probes and /openapi.json are always unauthenticated. In multi-process mode set the same key on every process: a forwarded/broadcast request passes the caller's Authorization through verbatim, and the internal-propagation marker (x-tidal-internal: 1) is an inter-sibling trust signal, not an auth bypass (the bearer middleware still runs first).

Useful environment variables:

Var	Mode	Effect
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1	both	Opt in to cluster mode (alternative to --experimental-cluster).
TIDAL_REGION	multi-process	Selects the single region this process owns (alternative to --region).
TIDAL_API_KEY	both	Bearer token for protected routes. Unset ⇒ unauthenticated + WARN. Set the SAME key on every region in multi-process mode.
TIDAL_HLC_SKEW_MS	multi-process	Signed ms offset applied to THIS process's HLC. Affects reconcile-time LWW stamping ONLY (not signal-decay timestamps). A test/ops escape hatch for verifying causal convergence under clock skew — do not set it in normal operation.
TIDAL_CONFIG	both	Config dir holding default-schema.yaml / default-cluster.yaml (used when --schema / --topology omitted).
TIDAL_SEED_STATUS_TIMEOUT_MS	seed-join	Per-status-poll HTTP timeout during seed-join leader discovery (default 5000). Raise on a TLS cluster under heavy CPU contention where a cold rustls handshake alone can blow a tighter budget — the joiner would then time out every poll and burn the whole 120s discovery window despite the peer being reachable. Lower only for fast loopback/test rigs.
PORT	both	Listen address. A bare port (9500) normalises to 0.0.0.0:9500.
TIDAL_SERVER_LOG	both	tracing filter (default info).

2. Launch via Docker

# Build the image once (build context is the repo root).
docker build -f docker/cluster/Dockerfile -t tidaldb:cluster .

# Run (press Ctrl+C to stop). The image already sets
# TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 so cluster mode starts; it still logs the
# loud experimental WARN. The default CMD is SINGLE-process cluster mode.
docker run --rm -p 9500:9500 tidaldb:cluster

The image bakes the default schema/topology under /etc/tidal-server and uses ENTRYPOINT ["tidal-server"] + a cluster … CMD, so you can override the subcommand (e.g. docker run tidaldb:cluster standalone …), pass --region for a multi-process container, or supply your own config files:

docker run --rm -p 9500:9500 \
  -e TIDAL_API_KEY=changeme \
  -v "$PWD/configs/my-schema.yaml:/srv/schema.yaml:ro" \
  -v "$PWD/configs/my-topology.yaml:/srv/topology.yaml:ro" \
  tidaldb:cluster \
  cluster \
    --listen 0.0.0.0:9500 \
    --schema /srv/schema.yaml \
    --topology /srv/topology.yaml

The container ships a HEALTHCHECK that curls /health. For Kubernetes/Compose deployment patterns see Cross-references.

3. Topology YAML

The topology file declares regions and the leader.

Single-process minimal default (tidal-server/config/default-cluster.yaml):

regions:
  - name: us-east
  - name: eu-west
  - name: ap-south
leader: us-east
# Optional: OS worker threads serving cluster write/heal requests (gRPC ship).
# Defaults to available parallelism when omitted.
# write_workers: 4

Multi-process requires a per-region grpc_addr and http_addr (validated at startup — a missing or syntactically invalid host:port is a hard error naming the region; reachability is deliberately not probed, since siblings boot in any order):

grpc_addr is an advertised address — a literal host:port OR a DNS hostname (m11p5). A new optional per-region grpc_bind controls the local bind independently of what siblings dial; the example below shows the DNS shape (one shared file names every region by its per-pod DNS name while each pod binds 0.0.0.0):

regions:
  - name: us-east
    grpc_addr: "tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9601"  # ADVERTISED (siblings dial; DNS re-resolved on reconnect)
    grpc_bind: "0.0.0.0:9601"      # LOCAL bind (optional; see derivation rule below)
    http_addr: "10.0.1.10:9501"    # this region's public HTTP gateway (forwarding + status)
    metrics_addr: "10.0.1.10:9091" # optional Prometheus /metrics listener (set it in production)
  - name: eu-west
    grpc_addr: "tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9602"
    grpc_bind: "0.0.0.0:9602"
    http_addr: "10.0.2.10:9502"
    metrics_addr: "10.0.2.10:9091"
  - name: ap-south
    grpc_addr: "tidaldb-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9603"
    grpc_bind: "0.0.0.0:9603"
    http_addr: "10.0.3.10:9503"
    metrics_addr: "10.0.3.10:9091"
leader: us-east
# Optional m11p1 tuning blocks (engine defaults shown):
# replication:
#   batch_max_events: 256   # events coalesced per shipped batch (1-256)
#   window: 4               # in-flight batches per peer (1-64)
#   retry_ms: 100           # backoff before a transiently-failed batch retries
#   catchup_retry_ms: 30000           # backoff before a FAILED catch-up pull re-pulls (m11p4)
#   snapshot_artifact_ttl_ms: 600000  # staged-snapshot reuse window (m11p5; never unpins an active consumer)
#   learner_promote_lag: 1024         # learner→voter promotion distance + readiness hysteresis (m11p5)
#   reseed_self_restart: false        # drain+exit(0) when reseed_required latches (m11p5; k8s sets true)
# wal:
#   batch_size: 100         # events per group-commit fsync (1-256)
#   batch_timeout_ms: 10    # max wait before a partial batch flushes
# election:                 # m11p4 failure-detector / election timers (defaults shown)
#   heartbeat_interval_ms: 300
#   election_timeout_min_ms: 1500     # validated: leader_lease_ms + heartbeat_interval_ms < this
#   election_timeout_max_ms: 3000
#   leader_lease_ms: 900
#   auto_election: true               # false → pre-m11p4 operator-driven failover

grpc_bind derivation (m11p5 §1): grpc_bind present → bind it. Absent + grpc_addr parses as a literal SocketAddr → bind that (today's behavior, byte-for-byte — every existing IP topology keeps working). Absent + grpc_addr is a hostname → bind 0.0.0.0:<port from grpc_addr>. Because the peer-dial path no longer parses grpc_addr as a SocketAddr, hyper re-resolves the hostname on every reconnect — a pod rescheduled onto a new IP is reached with no peer restarts. TLS note: SNI follows the URI host, so DNS peer names require DNS-SAN certs (see grpc_tls below); no code change.

3a. Sharding × replication (shards:, m11p6)

Absent shards:, the cluster is 1 shard × RF = all regions — one replicated log, one elected leader, byte-for-byte everything above. Add a shards: block to split the entity space into S groups, each a replication group at RF with its own elected leader, leaders balanced across nodes.

The k3s reference cluster runs this block enabled: S=3, full placement, RF3 (k8s/cluster/topology-configmap.yaml). Every pod replicates all three groups; tidaldb-0 leads shard 0, tidaldb-1 shard 1, tidaldb-2 shard 2, binding gRPC 9601/9602/9603 respectively. Full placement is the production shape, not a sharding experiment — a single pod loss never loses a group's quorum (2 of 3 survive per group) and any pod can serve a corpus-wide read locally.

Writes to /items///embeddings///signals hash-route (the engine's FNV-1a router) to the owning group's leader and replicate at RF; any gateway accepts any write.

regions:                                  # the NODE list (identity + addresses)
  - { name: us-east, grpc_addr: "10.0.1.10:9601", http_addr: "10.0.1.10:9501" }
  - { name: eu-west, grpc_addr: "10.0.2.10:9601", http_addr: "10.0.2.10:9501" }
  - { name: ap-south, grpc_addr: "10.0.3.10:9601", http_addr: "10.0.3.10:9501" }
shards:                                   # NEW — S groups (dense ids 0..S)
  - id: 0
    leader: us-east                       # term-0 / preferred leader (balance: group i → node i)
    replicas:                             # the RF nodes hosting this group
      - { node: us-east }                 # grpc_addr optional → derived node base port + shard id
      - { node: eu-west }
      - { node: ap-south }
  - { id: 1, leader: eu-west,  replicas: [ {node: us-east}, {node: eu-west}, {node: ap-south} ] }
  - { id: 2, leader: ap-south, replicas: [ {node: us-east}, {node: eu-west}, {node: ap-south} ] }
leader: us-east                           # legacy field, ignored once shards: is set

• Ports & dirs. A node hosting several groups binds one gRPC port PER group: set replicas[].grpc_addr explicitly, or omit it to derive node base port + shard id. Each group's data lives under <data_dir>/shard-{id:05}/; the single-group legacy layout (shards: absent) keeps <data_dir> verbatim, so existing clusters restart unchanged.
• Placement. replicas need not be every node (RF < N is valid). The exit gate and the simplest production shape are full placement (every node replicates every group); a strict-subset placement is supported for writes but see the read caveat in §7.
• Validation. Shard ids must be dense and unique; every leader/replica.node names a declared region; a group needs ≥1 replica; co-hosted groups must not collide on a derived gRPC port.

Per-shard admin (?shard=). With shards: set, every cluster admin verb takes an optional ?shard=<id> selecting which hosted group to act on; omitted = the node's lowest-id hosted group (and the only group when S=1, so S=1 URLs are unchanged). The selector is forwarded on intra-group hops, and NotLeader names the group. See §6a for the rebalancing verbs.

Fields:

Field	Single-process	Multi-process	Meaning
regions[].name	required	required	Region name used everywhere in the HTTP API (?region=, /cluster/promote, etc.). Must be unique.
regions[].grpc_addr	optional	required	This region's advertised gRPC replication address — what siblings dial. Since m11p5 it may be a literal host:port OR a DNS hostname (re-resolved on every reconnect). In single-process mode omit it — the server allocates a free loopback port and self-heals a bind race by retrying on a fresh port. In multi-process mode it is required and is tried exactly once.
regions[].grpc_bind	unused	optional	This region's local gRPC bind host:port (m11p5), independent of the advertised grpc_addr. Omitted: a literal grpc_addr binds itself; a hostname grpc_addr binds 0.0.0.0:<its port>. Set it to bind a specific interface while advertising a DNS name.
regions[].http_addr	unused	required	This region's public HTTP gateway address, used by siblings for write/read forwarding and status aggregation. Unused in single-process mode.
regions[].grpc_tls	unused	optional	TLS material for this region's gRPC transport: ca_cert, server_cert, server_key (PEM paths), plus optional client_cert/client_key for mTLS. Omitted ⇒ plaintext, the right posture for loopback/VPC topologies. DNS grpc_addr requires DNS-SAN certs (SNI follows the dialed hostname).
regions[].metrics_addr	unused	optional	Bind address for this region's Prometheus /metrics listener (m11p1; cluster mode previously had none). Omitted ⇒ no metrics endpoint. Set it in every production topology, and bind it internally — the endpoint is unauthenticated.
leader	required	required	Must name one of the declared regions. The initial write leader.
write_workers	optional	optional	Size of the runtime-free OS-thread pool that admission-controls cluster writes (/signals staging) and runs /cluster/heal's blocking redelivery. Bounded queue ⇒ 429 backpressure. Must be ≥ 1 when given.
timeouts.broadcast_peer_secs	unused	optional	Per-peer budget (seconds) for the /cluster/promote fan-out (the only remaining peer fan-out — the m11p2 log replaced the item/embedding broadcast). Default 2s — right for loopback/VPC; raise it for WAN topologies where a distant region cannot answer in 2s. Must be ≥ 1 when given.
replication.batch_max_events	unused	optional	Max relay events coalesced into one shipped batch (1–256, the WAL wire-format ceiling). Default 256.
replication.window	unused	optional	In-flight batches per peer (1–64). 1 = strictly in-order shipping; higher pipelines across the peer RTT (out-of-order arrivals park gap-aware on the receiver). Default 4.
replication.retry_ms	unused	optional	Backoff (ms) before a transiently-failed batch ship retries. Default 100.
replication.ack	unused	optional	Deployment-default write acknowledgment: leader (default) or quorum (majority-durable — see §8). Per-request override: the x-tidal-ack header.
replication.quorum_timeout_ms	unused	optional	Budget an ack=quorum write waits for the commit index before the retryable 503 naming the laggards. Default 2000.
replication.catchup_retry_ms	unused	optional	Backoff (ms) before a FAILED catch-up pull re-pulls on a timer (m11p4) — lets an idle cluster self-heal a follower whose pull failed during a rolling restart. Default 30000. Must be ≥ 1 when given.
replication.snapshot_artifact_ttl_ms	unused	optional	Staged-snapshot reuse window (m11p5), counted from the last fetch's completion. Governs artifact REUSE only — never unpins the WAL retention of an active consumer; a hard cap (4×) force-drops a never-releasing pin (tidaldb_cluster_snapshot_pin_force_drops_total). Default 600000 (10 min). Must be ≥ 1.
replication.learner_promote_lag	unused	optional	Learner→voter promotion distance AND the readiness-convergence hysteresis threshold (m11p5), in events. Default 1024. Must be ≥ 1.
replication.reseed_self_restart	unused	optional	Drain + clean-exit(0) once the durable reseed_required marker latches (m11p5; k8s sets it true). Refused when the remaining voters can't sustain quorum without this node. Default false.
election.heartbeat_interval_ms	unused	optional	Leader heartbeat interval (m11p4). Default 300.
election.election_timeout_min_ms / ..max_ms	unused	optional	Randomized follower election timeout window (m11p4). Defaults 1500 / 3000. Validated: leader_lease_ms + heartbeat_interval_ms < election_timeout_min_ms.
election.leader_lease_ms	unused	optional	Leader freshness lease — a leader that loses majority contact steps down within it (m11p4). Default 900.
election.auto_election	unused	optional	true (default) = automatic failover; false = pre-m11p4 operator-driven posture (no auto elections, no check-quorum step-down).
wal.batch_size	optional	optional	Events per WAL group-commit fsync (1–256). Default 100.
wal.batch_timeout_ms	optional	optional	Max ms a partial group-commit batch waits before flushing. Default 10. Tune against the measured tidaldb_cluster_wal_fsync_us on the deployment's volume.

The schema YAML (signals / text_fields / embedding_slots / profiles) is the same format the standalone server loads — see API.md and QUICKSTART.md for the full schema/profile grammar and the built-in ranking profiles. Topology only adds the region layout above.

Personalization correctness note. Preference-vector personalization (the taste vector that powers for_you) only updates for signals declared positive_engagement: true in the schema. The cluster /signals route writes global signals (no user_id / creator_id context), so it never folds item embeddings into a user's taste vector regardless of the positive_engagement flag. Personalized writes go through the embedded engine (signal_with_context), not this HTTP route. The exception is /hardnegs, which records a user-scoped hide that converges across regions via /cluster/reconcile.

4. gRPC replication transport (tidal-net)

Replication between regions is the real tidal-net WalShipping gRPC service, not in-process channels. The facts you need to operate and firewall it:

Property	Value	Notes
Service	WalShipping (proto tidal.replication.v1)	RPCs: ShipSegment (unary; the live-tail push path), StreamSegments (server-streaming; the follower-pulled catch-up path since m11p2), FetchSnapshot (server-streaming; the m11p5 snapshot transfer for joiners + reseeds), JoinCluster (m11p5 conf-change), and Heartbeat (ControlPlane health).
Bind address	per-region grpc_addr (multi-process); an auto-allocated loopback port (single-process)	Default dev band 59520–59529 if you pin one.
Transport security	mTLS via TlsConfig (ca_cert, server_cert, server_key, optional client_cert / client_key)	insecure = true (plaintext) is the loopback/VPC default in this phase. A peer with no TLS config must set insecure.
Max payload	64 MiB	Both encoder and decoder codec limits are raised to this; pinned by a compile-time assert to the engine's InProcessTransport limit so both ends agree. WAL segments default to 16 MiB.
Circuit breaker	per-peer, threshold 5, reset 30s	After 5 consecutive ship failures the breaker opens; it stays open for 30s, then the next attempt probes (HalfOpen → Closed on success, Open again on failure). Backpressure (channel full) does not trip it. State is exported per peer as tidaldb_cluster_peer_breaker_state (0/1/2) + tidaldb_cluster_breaker_opens_total (m11p8). Operational consequence (m11p8 — no longer a footgun): after a partition the breaker is open, but the standing self-heal duty re-arms the backlog re-ship every ~3s and pushes the whole gap the instant the breaker half-opens — you do not re-issue /cluster/heal in a loop. Watch tidaldb_cluster_healing_peers → 0 (see §6 and the drills in §10).
Timeouts (defaults)	connect 5s, request 10s, keep-alive PING every 10s / 5s ACK	A blackholed peer fails fast instead of stalling the single-threaded shipper.

ShipSegment carries the WAL segment id, BLAKE3-validated payload bytes, event count, and the leader's authoritative leader_last_seq (so a follower can advance its replication-lag high-water-mark even for an all-local segment that filters to empty). The follower's segment-receiver thread drains and applies it on arrival. The engine's applied high-water-mark is a contiguous frontier with a bounded ahead-buffer, so out-of-order eager ships can never swallow a sequence gap (a silent-data-loss class fixed in m8p10).

Operational guard: the gRPC replication ports and the Prometheus /metrics endpoint are UNAUTHENTICATED. Bind them to loopback or a cluster-internal network only — never the public interface.

5. Core HTTP API

All routes are JSON unless noted. Examples assume BASE=http://localhost:9501 (any region's gateway in multi-process mode; the single --listen address in single-process mode) and, when TIDAL_API_KEY is set, AUTH='-H "Authorization: Bearer $TIDAL_API_KEY"'. Drop the -H header when running unauthenticated.

Middleware mirrors standalone: 30s request timeout (408), 100 max in-flight (429), 2 MB body limit (413), and an x-request-id on every response. Health probes sit outside the load-shedding stack so liveness/readiness are never queued or timed out under saturation.

Multi-process routing (one coherent surface). Any region's gateway accepts any operation and routes it to the node that owns it:

• Writes (/signals, /items, /embeddings, /hardnegs) on a non-leader forward to the leader (the caller's Authorization passes through); the client sees the leader's status/body. A leader that is unreachable degrades to a 503 naming the leader, never a hang.
• Reads (/feed, /search) default to the LOCAL region. This is the key difference from single-process mode, whose default read is the leader. A ?region=<other> read forwards to that region's process.
• /cluster/promote, /cluster/partition, /cluster/heal route to / fan out from the leader as documented in §6.

Health

curl "$BASE/health"          # 200 ok / 503 while draining; reports mode, region, leader
curl "$BASE/health/startup"  # always 200
curl "$BASE/health/live"     # always 200
curl "$BASE/openapi.json"    # served OpenAPI 3.1, UNAUTHENTICATED — canonical HTTP reference

/health returns { "ok": true, "service": "tidaldb", "mode": "cluster", "region": "us-east", "leader": "us-east", ... }. The /openapi.json document is the machine-readable source of truth for the data, /cluster/*, /sharded/*, and /hardnegs request/response shapes (the health probes are intentionally outside the documented API surface).

Register items & embeddings (one replicated log — m11p2)

Items and embeddings ride the same replicated WAL stream as signals: the leader journals each write as a kind-1 (item metadata) or kind-2 (embedding) WAL record BEFORE touching storage, and followers apply it from the log — live pushes for the hot tail, the StreamSegments catch-up stream for history. There is no HTTP broadcast anymore (the m8p10 side channel and its bug classes were deleted in m11p2), so the responses are plain statuses:

curl -X POST "$BASE/items" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "metadata": { "title": "Jazz Piano", "category": "music" } }'
# → 201 Created (item durably journaled into the replicated log + applied)

curl -X POST "$BASE/embeddings" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "values": [0.1, 0.2, 0.3, 0.4] }'
# → 204 No Content

The 201/204 asserts leader durability and carries x-tidal-seq (the record's seqno in the replicated log); x-tidal-ack: quorum upgrades it to a majority-durable ack (see §8). The record is fsynced into the stream every follower receives (live push, or pull-based catch-up after downtime — see §6 heal). A write to a non-leader forwards to the leader transparently (the ack header and seq header travel through the forward). A down/partitioned peer needs no backfill bookkeeping: it converges from the log when it returns.

Embeddings: tidalDB does not generate vectors — the caller brings them. The write L2-normalizes and inserts into the HNSW index. Dimensions are strict: they must equal the slot's declared dimensions (min 2, max 4096) or the insert is rejected with a 500 (the engine surfaces a dimension mismatch as an internal error, not a 400 — a known wart, tracked post-M8); zero-norm vectors are also rejected (500). RETRIEVE / SEARCH route through the first declared embedding slot only; multi-modal apps must fuse offline or use separate entity kinds.

Record signals (cluster /signals = global only)

curl -X POST "$BASE/signals" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1, "signal": "view", "weight": 1.0 }'
# → 204 No Content (see the durability contract in §8)

The signal name must be declared in the schema (an undeclared name returns 400 naming it). The 204 carries x-tidal-seq (the write's replicated-log seqno), and x-tidal-ack: quorum upgrades it to a majority-durable ack — see §8. This route records a global signal on the leader and ships it to followers over the replicated WAL stream; it does not personalize (see the personalization note in §3). On a non-leader gateway it forwards to the leader transparently and still returns 204.

Hard negatives (user-scoped hides)

curl -X POST "$BASE/hardnegs" \
  -H 'Content-Type: application/json' \
  -d '{ "user_id": 42, "item_id": 7 }'
# → 204 No Content

Records a user-scoped hide on the leader (a non-leader gateway forwards). The item is filtered from that user's /feed?user_id=42. Hard negatives converge across regions via the LWW-resolved /cluster/reconcile CRDT path — NOT the signal WAL relay and NOT a broadcast.

Retrieve and search (region-pinned reads)

# Default read region is the LOCAL region (multi-process mode).
curl "$BASE/feed?user_id=42&profile=for_you&limit=20"
curl "$BASE/search?query=jazz%20piano&user_id=42&limit=5"

# Pin a read to a specific region. The gateway forwards to that region's process.
# Followers may lag the leader (and lag jumps during a partition) — use this for
# canary reads and lag verification.
curl "$BASE/feed?profile=trending&region=eu-west"

?region= accepts any declared region name; an unknown name returns 400. Omit it and the read serves from the LOCAL region (multi-process) / the current leader (single-process). limit is clamped to 1000 at the trust boundary (a larger value cannot amplify memory unboundedly). The for_you profile applies the built-in diversity defaults (max_per_creator=2, format_mix_max_fraction=0.4, exploration=0.1).

6. Cluster management API

Check cluster status

Two views. /cluster/status/local reports THIS node's own replication/leadership state (no peer calls); /cluster/status aggregates EVERY region (the gateway calls each peer's /cluster/status/local concurrently with a tight per-peer budget).

curl "$BASE/cluster/status/local" | jq

{
  "region": "ap-south",
  "is_leader": false,
  "leader": "us-east",
  "last_seq": 0,
  "applied_events": 124,
  "lag_events": 1,
  "partitioned": [],
  "reachable": true
}

curl "$BASE/cluster/status" | jq

{
  "leader": "us-east",
  "relay_log_len": 125,
  "regions": [
    { "name": "us-east",  "applied_events": 125, "lag_events": 0, "partitioned": false, "reachable": true },
    { "name": "eu-west",  "applied_events": 125, "lag_events": 0, "partitioned": false, "reachable": true },
    { "name": "ap-south", "applied_events": 124, "lag_events": 1, "partitioned": false, "reachable": true }
  ]
}

relay_log_len is the leader's high-water-mark (last_seq); each region's lag_events is relay_log_len − applied_events (saturating). Since m11p3 /cluster/status/local also reports the node's ack default and, on the leader, commit_index — the highest seqno a majority of the replica set durably holds (last_seq − commit_index is the quorum lag). A region the gateway cannot reach within the per-peer budget is reported honestly as reachable: false, partitioned: true, applied_events: 0, and worst-case lag (lag_events == relay_log_len). A non-zero, growing lag on a reachable region is the signal that it is partitioned (ship-skip) or its segment-receiver is wedged.

Promote a new leader

curl -X POST "$BASE/cluster/promote" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "eu-west" }'
# → { "ok": true, "leader": "eu-west", "acked": ["eu-west","ap-south"], "failed": [] }

The gateway first resolves the TARGET's stream baseline — the new leader's WAL flushed frontier at promotion, persisted in its data_dir/stream_baseline — then fans the promote out to every peer carrying it (internal marker, so each applies locally and does not re-fan). Peers jump their applied frontier for the new leader's shard to the baseline: everything at or below it is pre-stream history (replicated applies of the OLD stream), not data. The response carries baseline plus the acked/failed fan-out report; a peer in failed (e.g. a dead old leader) self-corrects — its first parked batch from the new stream triggers a catch-up pull whose chunks announce the baseline. After promotion /cluster/status reports the new leader; new writes route there; the new leader's ship queue activates and the demoted node's deactivates. An unknown region returns 400. Since m11p4 promote is a fenced transfer (it drains the target, then sanctions an election), not the availability mechanism — automatic failover handles a dead leader with no operator action (§9.1); this verb is for maintenance and deliberate successor choice (§9.2).

Simulate a partition & heal

There are two ways to partition a region; the runbook drills demonstrate both (§10):

1. Simulated ship-skip flag — /cluster/partition tells the leader to stop shipping to the named region (no sockets touched). Good for a controlled, reversible lag demo.
2. Real network partition — sever the actual TCP path between processes (firewall / proxy). The chaos suite uses a root-free in-harness TCP relay; operators can use iptables/pfctl (see §10).

# Simulated: isolate ap-south — leader ships skip this follower, so its lag climbs.
curl -X POST "$BASE/cluster/partition" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "ap-south" }'
# → { "ok": true, "partitioned": "ap-south" }

# Heal: resume shipping past everything the follower reports applied, and
# nudge it to PULL any history that rotated out of the leader's ship tail
# (the single recovery verb — log catch-up IS the full heal since m11p2).
curl -X POST "$BASE/cluster/heal" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "ap-south" }'
# → { "ok": true, "healed": "ap-south" }

/cluster/heal is the single recovery verb, and since m11p2 it is pure log catch-up: it clears the partition flag, resumes the ship queue past the follower's reported applied seqno (retries of data the follower already holds prune automatically — every ship ack piggybacks the follower's applied seqno), and nudges the follower to pull anything older than the leader's in-memory ship tail via its StreamSegments catch-up stream over the leader's durable WAL segments (POST /cluster/catchup, internal; the nudge forwards the healing operator's own bearer credential). Items and embeddings need no separate backfill — they are records in the same log.

Convergence is self-driving (m11p8 — the operator no longer loops heal). The per-peer ship senders retry parked batches every replication.retry_ms (default 100ms); a follower that detects a gap pulls the catch-up stream itself (also on boot, so a restarted node converges with no operator action at all — the tier-3 mp_items_ride_the_log_and_catchup_stream proves it). On top of that, a standing leader heal duty runs every ~3s: for any non-partitioned peer whose ship breaker is open AND that trails the leader, it re-arms the backlog re-ship from the peer's durable mark, so the moment the breaker half-opens (threshold 5, reset 30s — see §4) the leader pushes the WHOLE gap. This closes the old footgun: you no longer re-issue /cluster/heal until lag 0 — the server drives it. Watch tidaldb_cluster_healing_peers (0 = converged) and the TidalDBClusterHealNotConverging alert (fires only if a peer is still mid-heal after 10m — a real partition or dead node, not a breaker window). /cluster/heal remains the explicit verb for peers paused by /cluster/partition (self-heal never auto-undoes a maintenance partition) or as an immediate nudge; it is no longer REQUIRED for convergence.

Reconcile (cross-region CRDT convergence)

curl -X POST "$BASE/cluster/reconcile" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "ap-south" }'
# → { "ok": true, "region": "ap-south",
#     "local_elapsed_ms": 0, "remote_elapsed_ms": 1, "ops_applied": 3 }

/cluster/reconcile exchanges a CRDT state snapshot with the target region: this node ships its snapshot into the target's merge AND applies the target's pre-merge snapshot back, so both sides converge to identical state by deterministic LWW. This is how hard negatives (recorded via /hardnegs) converge across regions — they do not ride the WAL relay. local_elapsed_ms / remote_elapsed_ms are the merge+apply times each side measured (not the HTTP round-trip); both are typically 0–1ms. Reconcile is idempotent: a repeat reconcile of already-converged regions is an exact no-op on scores (no drift). /cluster/reconcile/snapshot is the INTERNAL snapshot-exchange leg this verb drives (marker required); operators never call it directly.

Membership verbs (m11p5 — online add / remove / inspect / reseed)

Membership is data on the replicated log (kind-4 records, latest wins, folded into a ClusterMembership cell). All four verbs route to / are answered by the leader and are quorum-commit-gated, one change at a time. The leader refuses every conf-change until all current voters report kind-4 capability (the mixed-version safety gate — complete the binary upgrade first).

# Add a node (idempotent by name): forwards to the leader, which assigns the
# next id, appends a Learner record, and answers AFTER quorum-commit.
curl -X POST "$BASE/cluster/join" \
  -H 'Content-Type: application/json' \
  -d '{ "name": "eu-2",
        "grpc_addr": "eu-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9600",
        "http_addr": "eu-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9504" }'
# → { "id": 4, "role": "learner", "term": 7, "leader": "us-east", "members": [ … ] }

The joiner normally seed-joins itself (§1b); this verb is the same conf-change for tooling. A re-join from a known name returns its existing id and current role and appends nothing. The leader's auto-promotion duty (a standing duty, re-armed on every activation and membership apply — it survives the joining-era leader's death) flips the learner to Voter once it is within replication.learner_promote_lag of the flushed frontier; promotion_pending (lag=N) in /cluster/status/local makes a stuck scale-up diagnosable.

# Inspect the applied roster (ids, names, addresses, roles, conf version).
curl "$BASE/cluster/members" | jq

# Remove a node: appends a Removed tombstone (quorum-commit-gated). The peer's
# ship cell is retired only AFTER the record is delivered-to/acked-by the removed
# peer (bounded give-up → tidaldb_cluster_remove_delivery_giveups_total); the
# removed node's readiness flips to 503 and it stops campaigning. Its id is BURNED
# (never renumbered, never reused).
curl -X POST "$BASE/cluster/members/remove" \
  -H 'Content-Type: application/json' \
  -d '{ "region": "eu-2" }'
# → { "removed": "eu-2", "membership_version": 9 }

# Force a reseed on demand: latches the durable reseed_required marker. The node
# keeps serving degraded (voting enabled) and reseeds via snapshot on its NEXT
# boot (or self-restarts if replication.reseed_self_restart is true and quorum
# can be sustained without it). See §9.1.
curl -X POST "$BASE/cluster/reseed"
# → { "reseed_required": true }

Scale-down order (decommission): call /cluster/members/remove first (so the cluster's quorum math shrinks before the node disappears), wait for the record to quorum-commit, then stop / delete the node — lowest ordinal last under a StatefulSet.

6a. Rebalancing (m11p6)

With a shards: topology (§3a) the operator moves load between nodes per group. Automatic rate-limited rebalancing is explicitly later; m11p6 ships the manual verbs (each reuses the proven m11p4/m11p5 machinery, scoped to one group).

Move a group's leadership (the rebalance-back-to-preferred path) — a fenced transfer (catch-up drain → TimeoutNow → term+1 election), identical to /cluster/promote?shard=<id>:

curl -X POST "$BASE/cluster/shards/0/transfer" \
  -H 'content-type: application/json' \
  -d '{"region":"eu-west"}'
# only group 0's leader moves; groups 1,2 are untouched

Add / remove a replica of a group — add runs the m11p5 join (the named node joins group {id} as a Learner, catches up via snapshot+stream, auto-promotes to Voter); remove runs the m11p5 fenced removal on that group's log:

curl -X POST "$BASE/cluster/shards/2/replicas" \
  -H 'content-type: application/json' \
  -d '{"action":"add","name":"region-3","grpc_addr":"10.0.4.10:9603","http_addr":"10.0.4.10:9504"}'

Multi-group caveat (tracked follow-up). A node's readiness is currently node-global across its co-hosted groups, so removing a node from ONE group of a node that hosts several would wrongly flip the whole node's readiness. Use the replicas verbs today for nodes that host a SINGLE group (or a brand-new node joining one group); per-group-aware readiness is the elasticity follow-up.

Inspect / target one group. Every admin verb takes ?shard=<id>; /cluster/status/local always lists a per-group shards[] array (leader, term, role, applied, lag, commit index per group), and /cluster/members?shard=<id> returns that group's roster.

7. Sharded scatter-gather API

The /sharded/* routes hash-partition entities across regions (hash(entity_id) % num_shards, using the engine's own ShardRouter so writes and reads never disagree). Writes route to the single owning region; reads fan out to all regions and K-way merge by score.

Write routes (each routes to the owning region; a non-owner gateway forwards):

curl -X POST "$BASE/sharded/items"      -d '{ "entity_id": 7, "metadata": { "title": "..." } }'  # 201
curl -X POST "$BASE/sharded/embeddings" -d '{ "entity_id": 7, "values": [0.1,0.2,0.3,0.4] }'      # 204
curl -X POST "$BASE/sharded/signals"    -d '{ "entity_id": 7, "signal": "view", "weight": 1.0 }'  # 204

Read routes (scatter-gather across all regions):

curl "$BASE/sharded/feed?profile=for_you&limit=20&deadline_ms=50"
curl "$BASE/sharded/search?query=jazz&limit=10&deadline_ms=100"

Each sharded read accepts an optional deadline_ms — the total scatter budget (default 50ms, server-clamped to 10s). The per-shard deadline is deadline_ms − 5ms network overhead. The response includes a scatter_gather block:

{
  "items": [ /* merged, deduped, diversity-enforced, re-ranked */ ],
  "total_candidates": 4210,
  "scatter_gather": {
    "degraded": false,
    "shards_queried": 3,
    "elapsed_ms": 12,
    "shard_deadline_ms": 45
  }
}

Degraded semantics: a shard that is partitioned, errors, or misses the deadline is reported in unavailable_shards (a name list) and flips degraded: true — it is never silently dropped, and the read still returns 200 with the live shards' results. The merge dedups replicated copies of an entity (keeping the best-scoring copy), reconciles total_candidates so replicated shards are not counted multiple times, and re-enforces max_per_creator across the merged set.

8. Write-durability contract: the ack knob and its cost (m11p3)

Every replicated write (/signals, /items, /embeddings) runs under one of two acknowledgment modes. Pick the deployment default with the topology's replication.ack; any caller overrides per request with the x-tidal-ack header (leader or quorum; anything else is a 400).

	ack=leader (default)	ack=quorum
Success means	Durable on the leader (storage + WAL group-commit fsync)	Durable on a majority of the replica set (leader + floor(n/2) followers, each storage-applied + own-WAL-fsynced)
Survives	Any follower failure; leader restart (WAL replay)	Any single node's permanent loss, including the leader's — promote the max-applied survivor and every acked write is there (the 167-kill-point ledger gate proves it)
Does NOT survive	Permanent leader loss before followers caught up (the un-shipped tail dies with it)	Simultaneous majority loss
Latency cost	One group-commit fsync (~ms-scale; the macOS F_FULLFSYNC tail is the local floor)	+ one ship RTT + the follower's group-commit fsync, pipelined: acks are batch-level, so concurrent writers share the round trip the same way they share fsyncs
Failure mode	5xx only for local faults	Additionally a retryable 503 when the quorum budget (replication.quorum_timeout_ms, default 2000) expires — the JSON body names the laggards, the commit_index, and needed/confirmed counts
Availability	Unaffected by follower outages	Blocks when a majority is unreachable. In a 2-region cluster quorum = leader + THE follower: one follower outage stops all quorum writes (by design — that is what the contract says). 3+ regions tolerate floor((n-1)/2) follower outages

Mechanics, in one paragraph: there is one replicated log (the leader's WAL — m11p2); peers only ever receive fsynced batches by construction. Since m11p3 every follower pushes its durably-applied frontier back to the leader once per apply round (ReportApplied — batch-level, decoupled from ship acks, flowing even when the follower converges by catch-up pull or the leader is quiet); ship acks additionally carry the same frontier as an instant floor hint. The leader folds both into per-peer durable marks; the commit index is the k-th largest mark (k = floor(n/2)), and ack=quorum responses gate on it passing the write's seqno — awaited asynchronously, so quorum waiters never hold threads or starve completions. The index is leadership-scoped: promote resets it to the new stream baseline, and a demoted leader fails its in-flight quorum waiters (it must never claim quorum for a stream it no longer owns).

Every cluster write's success response carries x-tidal-seq — the write's seqno in the replicated log (relayed through gateway forwards). Persist it if you need an exact durability cursor: commit_index >= seq on /cluster/status/local is "this write is majority-durable", regardless of which mode acked it. (The rare dedup-suppressed signal write — an identical event within the WAL's ~60s content-hash window is already durably logged, so no new record exists to track — carries x-tidal-deduplicated: 1 instead, relayed through forwards like the seq header.)

Retry semantics under ack=quorum — read this twice. A quorum-timeout 503 means not confirmed in budget, not not written: the write is in the leader's log and usually commits moments later. Retries are therefore at-least-once:

• /items and /embeddings retries are always safe — idempotent upserts keyed by entity_id.
• /signals retries can double-count the signal's weight when the original did commit (the server stamps each request's timestamp, so the WAL's content-hash dedup window cannot identify a client retry). The distortion is one extra decaying signal per retried timeout — bounded by your retry rate (tidaldb_cluster_quorum_timeouts_total is exactly that budget). Accounting that cannot tolerate it should route through session writes (which carry idempotency keys) or dedup client-side on its own key.
• The laggard names in the 503 are your runbook pointer: a persistent laggard is a down/partitioned region — heal it (§6) or accept leader-ack for the duration (x-tidal-ack: leader).

Rolling upgrades into m11p3 — upgrade the leader first. A pre-m11p3 leader neither serves the ReportApplied RPC nor recognizes x-tidal-ack: it silently applies leader-ack semantics to a quorum request — a durability downgrade the caller cannot see. Upgrade order:

1. Promote leadership off the leader node if needed, upgrade it, promote it back (or simply upgrade the standing leader per §9's restart procedure). From this moment ack=quorum is honored: the commit index rides the m11p2 ship-ack floor hints from not-yet-upgraded followers (correct, just laggier).
2. Upgrade followers one at a time. Each upgraded follower starts pushing ReportApplied and quorum freshness returns to batch-level. (An upgraded follower reporting to a still-old leader is harmless — the report is refused and logged, replication and heal are unaffected.)

Until step 1 completes, treat the cluster as ack=leader-only — do not point ack=quorum traffic at it expecting majority durability.

Other facts unchanged from m11p1/p2: the success code never waits on shipping for ack=leader (sender threads push the WAL flush feed's tail off the request path; replication.batch_max_events/window tune it); a failed fsync errors that write and nothing unfsynced can ship; long-outage data is follower-pulled via StreamSegments (or, when it has rotated past the leader's retained WAL, via the m11p5 FetchSnapshot snapshot transfer); a leader crash is an automatic failover since m11p4 (the survivors elect the up-to-date successor — see §9.1; the vote restriction only elects a node whose log covers every quorum-acked write, which is what makes the zero-acked-loss guarantee hold).

In short: ack=leader = leader durability. ack=quorum = failover-survivable durability, priced at one pipelined replication round trip and majority availability.

WAL segment format across upgrades (m11p4). Segment files carry an 8-byte version header (TSEG + version byte; headerless pre-m11p4 files stay readable — no migration). Three behaviors follow:

• Unreadable segments fail the boot, loudly. A node whose WAL dir holds segments written by an incompatible tidalDB version (or stray/foreign .seg files) refuses to start with WAL segment format unknown: <path> instead of booting with the data invisibly absent (segments=0 — the 2026-06-11 p3 rollout failure mode). Remedy: run a compatible binary, or reseed the node (delete its PVC and let it pull from the leader).
• Unservable catch-up resolves via snapshot transfer (m11p5). A leader that cannot serve a follower's requested range from its retained WAL answers the StreamSegments pull with FAILED_PRECONDITION — "segments not available from seq N; snapshot required", carrying the typed trailer x-tidal-catchup: snapshot-required. Since m11p5 the follower latches a durable reseed_required marker (only on that typed trailer — an ordinary election term-mismatch never latches it fleet-wide) and reseeds itself via the FetchSnapshot snapshot stream on its next boot (or self-restarts if replication.reseed_self_restart is set) — no operator verb, no wipe_data_dir. See §9.1. The marker is surfaced in /cluster/status/local (reseed_required: true) and the tidaldb_cluster_reseed_required gauge.
• Failed pulls self-heal on a timer. A catch-up pull that fails (e.g. the leader's gRPC server not yet ready during a rolling restart) retries every replication.catchup_retry_ms (default 30000) without waiting for a write to re-expose the gap — an idle cluster no longer strands lagged followers.

Downgrade hazard (m11p4): a pre-m11p4 binary reading a header-bearing segment treats the header as a torn tail and may truncate the final segment. Downgrading across the m11p4 boundary requires reseeding the node's WAL.

Membership records + capability gate (m11p5). A kind-4 membership record is a new WAL blob kind. A kind-4 record shipped to a pre-p5 follower is an unknown batch kind → WalError::Corruption → that follower's torn-state receiver halt, permanent across restarts (boot self-heal re-pulls the same record). Both followers halted = a quorum-write outage. To make this structurally impossible, HeartbeatResponse/ReportApplied carry a capabilities bit-field (proto3 zero-default = pre-p5 = incapable) and the leader refuses /cluster/join and every conf-change until all current voters report kind-4 capability — so the first conf-change cannot fire mid-upgrade. Downgrade rule (kind-3 precedent verbatim): once any kind-4 record is in a node's WAL, downgrading it below p5 requires a reseed. Complete the binary upgrade across all voters before adding, removing, or replacing a node.

9. Failover (multi-process)

9.1 Automatic failover (m11p4 — the default)

"A machine died" is a non-event. Every node runs a failure detector (leader heartbeats every election.heartbeat_interval_ms, default 300) and a Raft-style election (pre-vote + vote, randomized election.election_timeout_{min,max}_ms, default 1500–3000). Kill the leader and the survivors elect a successor — typically in under one second with the defaults, bounded well inside 10s — with zero operator verbs and zero acknowledged-write loss (the vote restriction only elects a node whose log covers every quorum-acked write; the tier-3 cluster_election.rs::mp_auto_failover_writes_resume_zero_acked_loss gate proves it across repeated random kill points under ack=quorum load).

What the operator sees:

• /cluster/status/local carries term (the election term, 0 = the pre-election "topology era"), role (leader/follower/pre-candidate/ candidate) and quarantined.
• During the brief leaderless window, writes return a retryable 503 naming the election (leader: "none (election in progress)" plus the responding node's term); clients retry and land on the new leader.
• A restarted ex-leader can never re-claim leadership from its topology file: its durable election state (data_dir/election_state) boots it as a follower, and every replication RPC is term-fenced — a deposed leader's ships, heartbeats and frontier reports are rejected until it rejoins the current term (the §1.4-1 split-brain incident is closed by construction; proven by mp_fenced_ex_leader_restart_cannot_write).
• A leader that loses contact with a majority steps down within election.leader_lease_ms (default 900) and stops accepting writes: ack=leader writes during a minority partition are bounded by the lease, and ack=quorum writes were never at risk.

Divergent suffix / quarantine → automated reseed (m11p5). A node that held leader-acked (never quorum-acked) writes when it died can rejoin into a cluster that elected past them. It detects this at term-join and quarantines: it serves status (quarantined: true, metric tidaldb_cluster_divergence_quarantined) and keeps voting, but refuses the data plane. Since m11p5 recovery is automatic and full-history — no wipe_data_dir: the quarantine latches the durable reseed_required marker, the node reseeds via the FetchSnapshot snapshot stream on its next boot (or self-restarts if replication.reseed_self_restart is set and quorum can be sustained without it), rejoins clean, and the divergence gauge clears. This is strictly leader-ack-only data, within the documented ack=leader crash contract (§8). The tier-3 cluster_reseed.rs proves the full quarantine → marker → restart → reseeded → gauges-cleared loop.

The three-way term-join rule (m11p5). The divergent-suffix check above is one of three outcomes a node reaches when it joins a new term, comparing its own stream position against the leader's election-time position (prev_log, carried on the heartbeat):

• own > prev_log → divergent suffix → quarantine (above);
• own < prev_log → the node is genuinely missing committed-era history that the new stream's baseline jump would silently skip (the p4 carried hazard) → it latches reseed_required and reseeds via snapshot (no silent gap);
• own == prev_log (or a within-term rejoin) → clean, catch-up via the stream.

A snapshot-installed node always joins clean by construction (its WAL is the leader's copy, so its tail term equals the leader's).

Failure mode: write-burst false-partition (the headline m12 fix — rc7).

• SYMPTOM. Under a sustained 1536-D ack=quorum ingest burst, ack=quorum writes start 503-storming and do not self-heal. The breaker gauge tidaldb_cluster_peer_breaker_state shows BOTH followers stuck at 1 (Open) while they are actually alive — still heartbeating, still applying. The commit index stalls because the leader believes it cannot ship to a majority.
• CAUSE. A follower's CPU-heavy HNSW apply (each apply an ef_construction=400 insert at 1536-D) momentarily starves its transport runtime, so a leader ship RPC misses the 10s request deadline. Pre-rc7 that tonic DeadlineExceeded was counted as a transport failure (record_failure) and opened the breaker even though the peer was alive and heartbeating — both followers' breakers latched Open, commit stalled, quorum writes 503-stormed with no self-heal.
• FIX. Shipped in m12-writeburst-rc7 (tidal-net record_timeout): a ship deadline opens the breaker only when there is no recent proof of life (last_contact stale ⇒ a genuine blackhole still opens it; DeadlineExceeded / Cancelled route through record_timeout, while a genuine Unavailable still opens immediately). Heuristic-only change — the commit / election / vote paths are untouched.
• RESPONSE if seen on an older image. Roll the StatefulSet to ≥ rc7 (@sha256:171505745b801dcf231b531de6167dbc309a7182957811cbc2228f0a302572b1). Confirm recovery by watching tidaldb_cluster_peer_breaker_state clear back to 0 once load eases and ack=quorum writes stop 503-ing.

To run the pre-m11p4 posture (operator-driven failover, no automatic elections, no check-quorum step-down), set in the topology:

election:
  auto_election: false

9.2 Manual promote: a FENCED TRANSFER (maintenance / override)

POST /cluster/promote { "region": "eu-west" } remains the maintenance verb — but it is now a fenced leadership transfer, not a view flip:

1. With a live leader: the leader waits for the target to hold the full flushed prefix (the catch-up wait IS the drain), then sanctions an immediate election (TimeoutNow); the target wins term+1 and the old leader steps down on first higher-term contact. Response: { ok, leader, term, transfer: "elected" }.
2. With a dead leader: the target campaigns among the survivors directly — the manual override of the automatic path (and the way to choose the successor).
3. The election can refuse a target that lags (its log loses the up-to-date comparison — e.g. the survivor you sampled fell behind between your status read and the vote). The verb 503s naming the cause; promote the other survivor. You can no longer accidentally promote a node that would discard acked data — the m11p3 "max-applied survivor" operator rule is now enforced by the protocol.
4. Promoting the node that already leads is a no-op 200.

The legacy term-0 fan-out promote survives only for clusters that have never elected (mixed-version rollouts mid-upgrade, and the deliberate isolated-node override used by the chaos drills); the first joined election permanently retires it on each node.

Rolling upgrade m11p3 → m11p4: upgrade ALL binaries before relying on auto-failover (pre-m11p4 peers answer vote RPCs with Unimplemented, so no election can reach quorum until a majority is upgraded — the cluster simply keeps its m11p3 behavior until then). The first ELECTED leader journals a kind-3 term-marker WAL record; pre-m11p4 binaries cannot decode it, so do not downgrade a node after the first election without reseeding.

10. Partition drill (multi-process)

The rewritten drill demonstrates both partition mechanisms, exactly as cluster_runbook.rs::runbook_s10_partition_drill scripts them.

1. Baseline. GET /cluster/status; all lag_events: 0, partitioned: false, reachable: true.
2. Inject. Use one of:
   • Real network partition — sever the TCP path peers use to reach the region. The chaos suite (cluster_chaos.rs) uses a root-free in-harness TCP relay proxy (the ROADMAP-sanctioned toxiproxy-style alternative). On a real host you can instead use iptables (Linux) or pfctl (macOS), e.g. iptables -A INPUT -p tcp --dport 9603 -j DROP to blackhole ap-south's gRPC port from a peer. A real cut shows reachable: false in the aggregate status.
   • Simulated ship-skip flag — POST /cluster/partition { "region": "ap-south" } (→ { ok, partitioned: "ap-south" }). The leader stops shipping to ap-south without touching sockets; the aggregate status shows partitioned: true.
3. Write through it. Send several POST /signals. Each must still 204 (the leader-durable contract holds). Watch ap-south's lag_events climb while its applied_events stalls — the leader's ships to it are dropped.
4. Read the stale follower. GET /feed?region=ap-south (or read ap-south's gateway directly) returns the pre-partition view — eventual, not strong, read consistency. The operator console survives a real partition because clients talk to each region's gateway directly.
5. Scatter-gather degradation. While partitioned, GET /sharded/feed returns 200 with degraded: true and ap-south in unavailable_shards — never an error, and the live shards' items are still returned.
6. Heal. Clear the cut (heal the proxy / drop the firewall rule, or just call heal for the simulated flag), then POST /cluster/heal { "region": "ap-south" }. Re-issue heal until /cluster/status shows ap-south at lag_events: 0 — the gRPC circuit breaker opened during the partition (threshold 5, reset 30s), so the first heal may ship into an open breaker and no-op. This is genuine production behavior, not a flag.
7. Verify convergence. GET /cluster/status shows ap-south lag_events: 0, partitioned: false, reachable: true; /sharded/feed is no longer degraded; feed scores on ap-south match the leader to within float tolerance (no loss, no duplication).

11. Shutdown

Send SIGTERM (or SIGINT / Ctrl+C, or stop the container). The server flips readiness to 503 (so a load balancer stops routing to it), drains in-flight requests, then drops the region's TidalDb shutdown path: checkpoint in-memory signal state → flush storage → write the WAL checkpoint marker + fsync → join the WAL, sweeper, checkpoint, text-syncer, and replication-receiver threads. The drop is idempotent; the process exits 0. You will see region cluster node shutdown: database closed (checkpoint + WAL fsync) (multi-process) / cluster shutdown: closing all nodes (checkpoint + WAL fsync) (single-process).

On restart with the same --data-dir, the region recovers its pre-shutdown state from the WAL (verified by cluster_runbook.rs::runbook_s11_shutdown_and_wal_recovery: SIGTERM → exit 0 → restart → the same items are served). In multi-process mode, the restarted node rejoins the cluster and pulls anything it missed while down via its boot-time StreamSegments catch-up request — no operator verb needed (m11p2; /cluster/heal remains the explicit lever for paused peers). This is the per-node step of a rolling upgrade (promote leadership off the node, SIGTERM, restart on the same data dir with the new binary, heal) — see cluster_lifecycle.rs::mp_rolling_upgrade_no_loss_no_stall, which proves zero acknowledged-write loss across a full rolling upgrade under load.

Node-replace drill (m11p5)

Replacing a dead or recycled node is kubectl delete pod — no topology edit, no operator verb:

• PVC retained (the data dir survives the pod): the replacement boots on the same data dir and converges via its boot-time StreamSegments catch-up — the ordinary restart path above. If it rotated past the leader's retained WAL while down, it reseeds via FetchSnapshot automatically (the reseed_required marker + snapshot path, §9.1).
• PVC deleted + pod deleted (fresh node): the replacement comes up empty and seed-joins (§1b) — it --seeds a survivor, FetchSnapshots the current state, and the leader auto-promotes it back to Voter. Under the k8s/cluster/ StatefulSet this is the default: the pod's args carry --seed against the headless Service, so a recreated pod rejoins with no human in the loop.

Add a node: scale the StatefulSet up (pod N≥3 seed-joins as a learner and auto-promotes). Remove a node: POST /cluster/members/remove first (§6), wait for the record to quorum-commit, then scale down (lowest ordinal last).

12. Security (m11p7): mTLS, rotation, identity, audit, rate limits

The cluster does not trust the network. Everything here is opt-in — absent the grpc_tls block and the cluster key, the cluster behaves exactly as pre-m11p7 (plaintext, hint-only marker, no audit/limit). A reference (k8s) deployment turns it all on.

12.1 mTLS (gRPC replication) — the default posture

• Configure the grpc_tls block per region (ca_cert, server_cert, server_key, client_cert, client_key). The gRPC server then REQUIRES a client cert chained to the cluster CA (mutual TLS): a foreign pod with no cert, a cert from another CA, or a plaintext probe fails the TLS handshake and never reaches an RPC.
• No grpc_tls ⇒ plaintext, with a loud startup WARN on both the server and the client. Acceptable only on a trusted single-host / loopback topology. To serve plaintext intentionally there is nothing else to set — the WARN is the signal that you are on the insecure path.
• The inter-node HTTP plane (forwards, broadcasts, scatter, status, seed-join) is served over TLS with the SAME cert and dials https:// with the cluster CA whenever grpc_tls is set, so enabling it gives zero plaintext inter-node links on both planes at once.

12.2 Cert + bearer rotation WITHOUT restart

• A background poller (TIDAL_ROTATION_POLL_MS, default 30000) content-hashes the cert files and the credential files; on a change it atomically swaps the served cert (in-flight TLS sessions keep their negotiated keys — zero dropped requests) and rebuilds the outbound peer channels.
• Procedure: issue a new cert under the same CA (cert-manager renewal, or re-run scripts/gen-cluster-certs.sh and re-apply the Secret) — the files change in place, the poller swaps within one interval, no pod restart. Verify with tidaldb_cluster_* logs (TLS material rotated…) or the tidal_audit /tracing stream.
• The bearer (TIDAL_API_KEY_FILE) and the cluster key (TIDAL_CLUSTER_KEY_FILE) rotate the same way. Use FILE mounts (not inline env) so a Secret rotation is picked up live. During a CA roll, keep both old and new CAs trusted for one cycle (CA-overlap) so in-flight connections complete.

12.3 Per-node identity + the marker

• Set a shared cluster key (TIDAL_CLUSTER_KEY / TIDAL_CLUSTER_KEY_FILE, any random string — BLAKE3-derived to the MAC key). Each node then mints a signed x-tidal-node-token on every forward/broadcast; the receiver verifies it. This gives inter-node calls a verifiable node identity and is the defense-in-depth layer beyond the shared bearer.
• With a cluster key configured, the x-tidal-internal marker is honored ONLY from a verified sibling: a request that sets the marker without a valid node token is rejected 403 (the marker is a routing hint, never an auth bypass). Never hand the cluster key to external clients.

12.4 Admin audit log

• promote / partition / heal / join / member-remove / reseed each emit one structured record: {principal, verb, target, term, outcome}. The principal is the verified node (node:<id>) for inter-node calls or external for an operator with the bearer.
• Sinks: a tidal_audit tracing target (always — capture it in your log pipeline), plus an append-only JSONL file when TIDAL_AUDIT_LOG=<path> is set. Recorded on the operator-originated leg only (no double-record on a forwarded re-apply).
• At-rest encryption of the JSONL file is delegated to the volume — mount TIDAL_AUDIT_LOG on an encrypted PV (or a gVisor/LUKS-backed volume); the server does not encrypt it in-engine.

12.5 Per-principal rate limits

• TIDAL_RATE_LIMIT_RPS (+ optional TIDAL_RATE_LIMIT_BURST, default 2×) caps per-principal request rate; a deny is 429 + Retry-After. Off by default.
• Verified sibling nodes are EXEMPT — replication/forward traffic is never throttled by the external-client budget. Today external callers share ONE bucket: TIDAL_RATE_LIMIT_RPS is an AGGREGATE cap across all external clients, not a per-client budget — set it to your total external ceiling, not a per-client one (one noisy client can consume it). A future multi-key registry adds per-key buckets.

12.6 Foreign-pod / negative behavior (what an attacker on the network sees)

Attempt	Result
Ship a gRPC segment without a cluster client cert	TLS handshake fails — no RPC dispatched
Call an internal HTTP route without trusting the cluster CA	TLS handshake fails — no route reached
Set x-tidal-internal without a valid node token (key configured)	403 — marker honored only from a verified sibling
Call a protected route without the bearer	401 (unchanged)

13. Coordinated backup / restore + point-in-time recovery (m11p8)

See also the dedicated DR runbook: docs/runbooks/disaster-recovery.md for full disaster-recovery procedures (region/cluster loss, restore drills, RPO/RTO).

The building blocks: the engine's crash-consistent create_backup, the WAL archive (wal.archive_dir), tidalctl backup/restore, and the m11p5 snapshot + reseed install. Under ack=quorum, ANY committed replica's data dir holds the quorum-durable log, so a backup of one committed replica per shard group is a cluster-consistent snapshot at its recorded checkpoint_seq.

13.1 Enable the WAL archive (point-in-time recovery)

Set wal.archive_dir in the topology (or TidalDb::builder().wal_archive_dir(path) for the embedded engine). Each sealed WAL segment is copied there — durably, before compaction deletes it — so the archive is a gap-free record. Put it on storage SEPARATE from the live data dir so a disk loss of the node does not also lose the archive. Segment filenames encode shard + first_seq, so co-located groups share one archive dir without collision.

wal:
  archive_dir: "/archive/tidaldb"   # durable, off-node storage

13.2 Coordinated backup drill

1. Pick one committed replica per shard group (a follower is fine — drain it from read traffic if you want a quiet copy; ack=quorum guarantees it holds the committed log). Stop it (or snapshot its volume).
2. tidalctl backup --path /data/<node> --out /backups/<cluster>-<ts>/shard-<id> — writes a recursive copy + BACKUP_MANIFEST.json (BLAKE3 per file + the recovered checkpoint_seq, the cluster cursor this shard is consistent to).
3. Repeat per shard group. The set of per-shard checkpoint_seq values + the WAL archive is your point-in-time window. Restart the replica; self-heal/catch-up reconverges it.

13.3 Restore drill (timed)

1. tidalctl restore --from /backups/<cluster>-<ts>/shard-<id> --path /data/<new-node> — verifies EVERY file's BLAKE3 against the manifest BEFORE writing, and refuses a non-empty target (it never overwrites a live data dir).
2. Point a stopped node at the restored dir and boot it. Under ack=quorum its group's followers catch up via the live stream; promote it if it is the group's chosen leader (the highest-applied survivor rule, §9).

Recovery granularity today is full-snapshot-to-frontier, NOT arbitrary point-in-time. A restore recovers a node to the backup's checkpoint_seq, then the live stream reconverges it to the cluster's CURRENT frontier. The WAL archive (wal.archive_dir) is the durable, gap-free primitive that backs a future point-in-time replay — but it is write-only today: no tool replays archived segments up to a chosen target seq. tidalctl recover is verify-only (--verify-only; an in-place replay mode is reserved for a future release), and tidalctl restore copies a full snapshot with no seq bound. So the inputs you retain for a future PITR are the per-shard checkpoint_seq set + the archive; treat them as the window, not a one-command restore-to-instant. Do not plan an incident around seq-bounded replay until a tidalctl replay --until <seq> verb ships.

Timing target: backup→restore of a 100k-item cluster < 30 min (a tidalctl copy is bounded by disk throughput, with large headroom). The Ref-A timed figure is a k3s line item (the standing M11 access caveat).

14. Rolling upgrade + version skew (m11p8)

Nodes carry a build version on the wire (HeartbeatRequest.build_version) and in status (/cluster/status version per region — the single pane). Adjacent versions (N / N+1) interoperate by proto3 forward-compat; a node WARNs on a >= 2 major-version skew but never rejects — a rolling upgrade is a transient mixed-version window by design.

Procedure (one node at a time):

1. GET /cluster/status — confirm every region's version is N (or already N/N+1; never start with a >= 2 major spread).
2. Graceful SIGTERM one follower → it drains (readiness 503 → checkpoint).
3. Restart it on N+1 (same ports, same data dir). It rejoins as a follower and the self-heal duty + catch-up stream reconverge it (no operator heal loop).
4. Repeat for each follower. Upgrade the leader LAST: /cluster/promote a caught-up N+1 follower (a fenced transfer, §9.2), then upgrade the old leader as a follower.
5. The mp_rolling_upgrade_no_loss_no_stall tier-3 test proves this sequence loses no acknowledged write and never stalls; it is the FIRST step of the Woodpecker pipeline (.woodpecker.yaml) — a failure blocks the image build.

Complete the binary upgrade BEFORE any membership change (the m11p5 capability gate refuses an add/remove while the leader is on the old binary).

Performance

Read this table by era. The /signals-throughput row below is the m11p1 signal-write benchmark (3-byte signal writes, small payloads, measured over real localhost processes) — it is NOT the m12 1536-D production shape and must not be cited as the live cluster's read/write ceiling. The m12 reality on the 3-node k3s fleet at the 1536-D production corpus (see docs/profiling/m12-cluster-deploy-findings.md):

• Reads: p99 7.97–11.47 ms at 100–500 rps; G1 (p99 ≤ 10 ms) MET at 100k with recall@10 0.9989 (clean index). Read ceiling ~1000 rps clean (~1500 rps saturated), CPU-bound — beyond it a node sheds/errors.
• Writes: the peach mix is write-heavy and the write knee is ~250 rps on this fleet (1536-D ack=quorum ingest; each apply is an HNSW insert). This is why the soak (§15) was re-scoped to 200 rps on 2026-06-19.

The legacy m11p1 signal-write benchmark (measured over real localhost processes):

Operation	SLA	Measured (p99 / typical) — m11p1 signal-write benchmark, pre-m12, NOT the 1536-D shape
Replicated /signals throughput (m11p1, 3 nodes, release build)	≥ 2,000/s	4,534 signal-writes/s within SLO on the ramp (knee ~5.5k/s); 2,739/s sustained 10 min (1.65M writes, 0.35% errors); was ~90/s pre-m11p1. Superseded for production sizing by the m12 1536-D figures above.
Replication lag under that load (m11p1)	< 2s	≤ 103 events (~40ms) across the 10-min sustain; ≤ 377 events on the 5k/s ramp (follower group-commit coalescing)
Cross-region replication (write → follower applied)	< 2s	~110–133ms p99 (m8p10)
Failover (/cluster/promote → first successful write)	< 10s	~31–34ms
CRDT reconcile (merge+apply, each side)	< 100ms	0–1ms

Write-latency note: signal p50 ≈ 17–25ms with a p99 tail of 160–220ms on macOS, where F_FULLFSYNC averages ~7.4ms with a 10–50ms tail (tidaldb_cluster_wal_fsync_us — 0% complete under 1ms). On Linux fdatasync volumes the same pipeline's fsync floor is far lower; validate the p99 gate on the reference environment, and tune wal.batch_timeout_ms against the measured fsync histogram.

15. Continuous correctness: chaos suites + soak (m11p9)

Correctness is a pipeline, not a one-time gate. The tier-3 suites run REAL OS processes; the new fault classes inject REAL faults.

Run the chaos suites locally (serial — they bind fixed ports and spawn processes, so suites must not overlap):

cargo test -p tidal-server --features "cluster-e2e fault-injection" \
  --test cluster_faults -- --test-threads 1 --nocapture

cluster_faults covers the fault classes the partition/crash/skew suites lacked:

Test	Fault	Asserts
mp_disk_full_follower_degrades_no_acked_loss	follower WAL ENOSPC	receiver halts (degraded, alive), healthy majority keeps acking quorum, zero acked loss, restart recovers to parity
mp_slow_fsync_follower_lags_but_quorum_holds	one follower's fsync slowed	fast follower supplies quorum, slow node lags then converges, no loss
mp_slow_fsync_both_followers_force_honest_quorum_timeout	both followers slowed below the budget	ack=quorum → retryable 503 naming laggards; ack=leader → 204; recover
mp_asymmetric_partition_no_split_brain_no_loss	inbound to one node severed (outbound up)	pre-vote + check-quorum hold the leader; single-leader-per-term; no loss

Fault knobs (behind the fault-injection feature — compiled OUT of the production image, inert until armed): TIDAL_FAULT_FSYNC_DELAY_MS=<ms> slows every durable WAL fsync; TIDAL_FAULT_DISK_FULL_AFTER_BYTES=<n> fails segment writes with ENOSPC after n cumulative bytes this process lifetime (arm it on a node at restart to fail after n bytes of post-restart writes). NEVER set these on a production node.

Soak with regression gates (tidal-stress):

tidal-stress --target https://<gateway> --ramp "200:3600" --mix peach \
  --json-summary soak.json --max-error-pct 1 --max-p99-ms 250 --fail-on-knee

Soak runs at 200 rps (re-scoped 2026-06-19). The peach mix is write-heavy and the write knee on this 3-node fleet is ~250 rps (1536-D ack=quorum ingest), so the soak ramp is a single measured-sustainable 200-rps stage (--ramp "200:3600") rather than the retired 3900:3600 constant (a pre-m12 signal-write figure that does not hold at the 1536-D production shape). Point --target at an https:// gateway — the :9500 plane serves TLS.

--fail-on-knee (built-in SLO), --max-p99-ms, and --max-error-pct make the run exit non-zero on a regression; --json-summary writes a machine-readable per-stage roll-up for trend lines. A bounded version runs nightly; the GA-bar 100k-DAU soak points --target at the live Ref-A cluster at 200 rps.

Nightly CI (.woodpecker.yaml, cron nightly — Woodpecker, never GitHub Actions): the chaos suites with elevated kill-points (TIDAL_QUORUM_KILLPOINTS, TIDAL_ELECTION_KILLPOINTS) then the gated soak. A nightly failure flags the day's correctness or performance regression. The guarantee→test map is docs/planning/milestone-11/guarantee-traceability.md.

Cross-references

• Kubernetes deployment — docs/runbooks/kubernetes.md (the hardened standalone single-replica set in k8s/, and the multi-region cluster reference in k8s/cluster/ — one StatefulSet + headless Service peer discovery + PDB, with --seed-based scale and kubectl delete pod node-replace, shipped in m11p5).
• Disaster recovery — docs/runbooks/disaster-recovery.md (backup/restore/PITR procedures, region/cluster loss, RPO/RTO; the operational companion to §13).
• Server deployment guide — docs/guides/server-deployment.md (standalone and cluster launch, config, env, health probes).
• Monitoring & alerts — docs/ops/monitoring.md (Prometheus scrape config, replication-lag and WAL metrics, recommended alerts; per-region replication lag is observable via /cluster/status lag_events; remember the /metrics endpoint is unauthenticated — bind it internally).
• Roadmap / cluster status & known gaps — docs/planning/ROADMAP.md and docs/roadmap-to-cluster.md for the M11 cluster status — quorum-ack writes (m11p3), automatic failover (m11p4), membership/discovery/elasticity (m11p5), security hardening (m11p7), observability + operations (m11p8), and continuous correctness (m11p9) all shipped; sharding × replication (p6) data plane in progress.
• API & schema reference — API.md, QUICKSTART.md, and the live /openapi.json document.
• Scope & vision — VISION.md.