a6f663f002
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
Fixes the two defects a malformed probe exposed on the live cluster, plus the
coverage gap that let a stale assertion survive the same day it was falsified.
TASK 17 — validate before the WAL append. A 128-dim vector against a 1536-dim
slot was appended to the WAL FIRST, then validated, then answered 500 — so an
already-durable, unapplicable record shipped to both followers, halted both
receivers, and put shard 1 into a quorum-write outage. Validation now runs before
the append and returns 400 via invalid_input; nothing enters the log.
`storage::vector::validate_dimensions` is now the single comparison, replacing an
inline duplicate of the same rule in lifecycle/ops.rs:57-62 — two copies of a
dimension check drift, and the apply-path copy is the one that halts replication
when it disagrees.
The receiver's halt-vs-skip decision is now explicit instead of "halt on
anything". A record whose failure is deterministic and node-independent (schema
width) is skipped, counted on blobs_apply_failed_total and ERROR-logged, so the
frontier advances; a record that could become applicable after a binary upgrade
(unknown batch kind, capability skew) still halts, because skipping那 would
silently drop replicated data. Both branches are proven reachable by tests.
TASK 18 — the reseed latch outlived its discharge. A node hosting 3 shard groups
latched a marker per group but discharged on a single seqno, so two latches meant
permanent 503 on a node whose every shard read lag 0 — it hit all three pods
during the roll and each needed a manual delete. Gaps are now tracked per group
in a ReseedGapSet and cleared on evidence about themselves; a REFUSED
reseed_self_restart re-evaluates every 15s instead of waiting for a latch that
never arrives. /health's cause ladder was also lying: it printed "joiner boot not
yet converged" for a node whose groups had all converged, because the fallback
asserted a state it never tested. It now names the outstanding gaps, gained the
decommissioned-by-signal arm that is_ready checked but the ladder did not, and
its terminal arm says "reason unavailable" rather than inventing one.
COVERAGE — 14 of 23 integration suites were run by NO pipeline. Not theoretical:
cluster_routes still asserted the wire fabrication removed hours earlier
(applied_events == 0 with a lag derived from it) and nothing caught it because
nothing ran it. cluster_sharding (dense-rank, /sharded/* opt-in), vector_search
(distance contract) and cluster_poison_embedding (task 17's own gate) were in the
same position, so those guards would have rotted identically. Every suite now has
a runner: 8 in-process ones in a new `fast-suites` push step (measured 71s, runs
FIRST so a cheap failure precedes the 6.5-min gate), 6 multiproc ones in the
nightly. All 23 scheduled; all 4 never-before-run heavy suites verified passing
before being scheduled.
Also fixes cluster_chaos.rs:329, which the nightly's FIRST EVER run caught 13
minutes in — it demanded an unreachable peer report worst-case lag, i.e. it
required the fabrication task 04a deleted.
Verified: fmt clean; clippy 72 vs 73 baseline (one FEWER, zero added, measured on
touched trees at 431340f); lib 2115 passed; all 8 fast suites green;
cluster_chaos 5, cluster_sharding 5, cluster_poison_embedding 1,
cluster_cross_shard_reads 2, cluster_graph_persistence 1, cluster_multiproc 5,
cluster_e2e 2; doc-guard OK.
1869 lines
107 KiB
Markdown
1869 lines
107 KiB
Markdown
# tidalDB Cluster Runbook
|
||
|
||
Operating the multi-region `tidal-server cluster` surface: launch (single-process
|
||
dev fabric **and** the multi-process region nodes), the operational API,
|
||
replication transport facts, failover and partition drills, and the honest
|
||
write-durability contract.
|
||
|
||
> ## STATUS: QUORUM-ACKED HA IS LIVE — RUNNING IN PRODUCTION ON k3s
|
||
>
|
||
> Quorum-ack writes (m11p3) and automatic election/failover (m11p4) are **LIVE and
|
||
> deployed**. The reference cluster runs in production on k3s as a **single
|
||
> StatefulSet, full-placement RF3** deployment: ns `tidaldb-cluster`, `replicas: 3`,
|
||
> every pod a region (`tidaldb-0/1/2`) hosting **all three shard groups**, image
|
||
> `m12-writeburst-rc7`. ack=quorum is the **cluster deployment default** (topology
|
||
> `replication.ack: quorum`) and leader election + failover are **automatic** — kill
|
||
> the leader and the survivors elect a successor with zero operator verbs and zero
|
||
> acknowledged-write loss. See the live production topology in
|
||
> [§1](#1-launch-the-cluster-locally) and the deployed shape in
|
||
> [§3a](#3a-sharding--replication-shards-m11p6).
|
||
>
|
||
> **Honest caveats that still hold:** cluster mode replicates **global** retrieval
|
||
> signals only (no per-user personalization on the `/signals` route — see the
|
||
> personalization note in [§3](#3-topology-yaml)); and because a standalone node is
|
||
> the right answer for most deployments, both launch modes **refuse to start**
|
||
> unless you explicitly opt in with `--experimental-cluster` or
|
||
> `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1` (the k8s manifests set the env var) — a guard
|
||
> against standing up a multi-node fabric by accident, not a readiness warning. The
|
||
> two launch shapes below remain:
|
||
>
|
||
> **1. Multi-process (`--region`) — real process isolation (the production shape).** Each
|
||
> `tidal-server cluster --region <name>` process owns **exactly one region**: one
|
||
> `TidalDb`, one [`GrpcTransport`](#4-grpc-replication-transport-tidal-net) whose
|
||
> server binds *this* region's `grpc_addr` and whose peers are every **sibling
|
||
> region's real `grpc_addr`**. Processes peer over real gRPC and forward over real
|
||
> HTTP, so a crash of one region's process takes down **only that region** — the
|
||
> survivors keep serving. This is genuine process (and, across hosts, host)
|
||
> isolation. It is verified end-to-end by the tier-3 suites
|
||
> (`cluster_multiproc`, `cluster_chaos`, `cluster_lifecycle`, `cluster_runbook`)
|
||
> over real OS processes with real network-partition injection.
|
||
>
|
||
> **2. Single-process (no `--region`) — the dev/demo default.** Every region runs
|
||
> inside **one** process. Replication still traverses the **real `tidal-net` gRPC
|
||
> transport** on loopback (faithful multi-region semantics over a real wire), but
|
||
> there is **no process isolation**: a crash, OOM, or host failure takes the whole
|
||
> "cluster" down at once. This is a development / staging / demo fabric and a
|
||
> correctness harness for the replication paths — **not** production HA. It remains
|
||
> the default because it needs no per-region topology addresses and no process
|
||
> orchestration.
|
||
>
|
||
> **Durability / leadership / membership facts (both modes):**
|
||
> * **Quorum durability is the cluster default, per-request overridable.**
|
||
> `replication.ack: quorum` is the topology default the k8s reference cluster
|
||
> ships, so a write succeeds only once a **majority of the replica set durably
|
||
> holds it** (m11p3), surviving permanent leader loss. A caller can downgrade a
|
||
> single write to leader-durable with `x-tidal-ack: leader` (storage + WAL fsync;
|
||
> follower ship off the request path). See
|
||
> [§8](#8-write-durability-contract-the-ack-knob-and-its-cost-m11p3).
|
||
> * **Leadership is automatic** (m11p4). Every node runs a failure detector and
|
||
> a Raft-style election (pre-vote + vote + check-quorum + fenced transfer):
|
||
> kill the leader and the survivors elect a successor — typically under a
|
||
> second with the defaults — with **zero operator verbs** and zero
|
||
> acknowledged-write loss (see [§9.1](#91-automatic-failover-m11p4--the-default)).
|
||
> `/cluster/promote` is now a **fenced transfer** for maintenance/override, not
|
||
> the availability mechanism. `election.auto_election: false` preserves the
|
||
> pre-m11p4 operator-driven posture.
|
||
> * **Membership is elastic and addresses are DNS names** (m11p5). `grpc_addr`
|
||
> is an advertised hostname or IP (DNS-resolved on every reconnect); nodes join
|
||
> online via `--seed` and catch up via `FetchSnapshot` + the `StreamSegments`
|
||
> stream; add/remove ride kind-4 membership records on the replicated log
|
||
> (see [§1b](#1b-multi-process-one-process-per-region),
|
||
> [§3](#3-topology-yaml), [§6](#6-cluster-management-api),
|
||
> [§9.1](#91-automatic-failover-m11p4--the-default)).
|
||
>
|
||
> **Production deployment today** is the multi-process cluster on k3s described
|
||
> above — the single-StatefulSet full-placement RF3 reference in [`k8s/cluster/`](../../k8s/cluster/)
|
||
> (see [§1](#1-launch-the-cluster-locally), [§3a](#3a-sharding--replication-shards-m11p6),
|
||
> and [kubernetes.md](kubernetes.md)). The single-process shape below remains the
|
||
> dev/demo fabric and replication-correctness harness, and a single
|
||
> `tidal-server standalone` node (see [server-deployment.md](../guides/server-deployment.md))
|
||
> stays valid for deployments that do not need multi-region / `ack=quorum`.
|
||
>
|
||
> **Both modes refuse to start** unless you explicitly opt in with either the
|
||
> `--experimental-cluster` flag or the `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1`
|
||
> environment variable. On start each emits a loud, **mode-specific** `WARN`
|
||
> restating exactly what it does and does not provide. Do not wire either into a
|
||
> production load balancer.
|
||
|
||
## Scope: what cluster mode does and does not own
|
||
|
||
tidalDB owns **retrieval and ranking only**. Cluster mode adds multi-region
|
||
replication of that, nothing more. It does **not** generate embeddings, store
|
||
video/blobs, run a CDN or transcoder, do auth/moderation/payments, or provide a
|
||
distributed consensus log. Bring your vectors; tidalDB retrieves and ranks over
|
||
them. See [VISION.md](../../VISION.md) for the scope boundary.
|
||
|
||
## Prerequisites
|
||
|
||
- Rust toolchain ≥ 1.91 if running directly (the Docker build pins
|
||
`rust:1.91-bookworm`).
|
||
- `protobuf-compiler` (`protoc`) and a C++ toolchain (`g++`) on the build host —
|
||
`tidal-net`'s build script compiles the WAL-shipping `.proto`, and USearch's
|
||
HNSW core is C++. The Docker image installs both.
|
||
- Docker 25+ if running via container.
|
||
- For **single-process** mode: one HTTP port (default `9500`); each follower
|
||
region also binds an **OS-assigned loopback port** for its gRPC replication
|
||
server unless you pin one in the topology (see [§3](#3-topology-yaml)).
|
||
- For **multi-process** mode: per-region `grpc_addr` **and** `http_addr` declared
|
||
in the topology, and the ports they name available on each host
|
||
(see [§3](#3-topology-yaml)).
|
||
|
||
## 1. Launch the cluster locally
|
||
|
||
Cluster mode is gated. Pass `--experimental-cluster` (or set
|
||
`TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1`) or the server exits with a mode-specific
|
||
error explaining why.
|
||
|
||
> **Live production topology (the deployed shape on k3s — 2026-06-19).** The
|
||
> reference cluster is **ONE StatefulSet `tidaldb`** in namespace
|
||
> `tidaldb-cluster`, `replicas: 3` = **3 pods = 3 regions = 3 voters**
|
||
> (`tidaldb-0/1/2`), **full-placement RF3**: every pod hosts **all three shard
|
||
> groups** (the 3-group `shards:` block in [§3a](#3a-sharding--replication-shards-m11p6)
|
||
> is the DEPLOYED shape, not optional). Each group's data lives under
|
||
> `/data/db/shard-0000N` on the pod's one PVC. Each pod binds one gRPC port per
|
||
> group — **shard 0 → 9601, shard 1 → 9602, shard 2 → 9603** (derived
|
||
> `node base port + shard id`). The HTTP plane is **`:9500` over HTTPS with
|
||
> inter-node mTLS** (m11p7 — every probe/curl uses `https://`); Prometheus
|
||
> `/metrics` is `:9091`. `replication.ack: quorum` is the deployment default and
|
||
> election/failover is automatic (`election.auto_election: true`). Live image
|
||
> `m12-writeburst-rc7`
|
||
> (`@sha256:171505745b801dcf231b531de6167dbc309a7182957811cbc2228f0a302572b1`).
|
||
> Networking is a **headless peer Service `tidaldb-peers`** (per-pod DNS
|
||
> `tidaldb-N.tidaldb-peers.tidaldb-cluster.svc.cluster.local`) plus a **ready-only
|
||
> client Service `tidaldb`** (the seed-join discovery target). See
|
||
> [`k8s/cluster/statefulset.yaml`](../../k8s/cluster/statefulset.yaml) and
|
||
> [`k8s/cluster/topology-configmap.yaml`](../../k8s/cluster/topology-configmap.yaml).
|
||
|
||
### 1a. Single-process (dev/demo default)
|
||
|
||
```bash
|
||
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
|
||
cargo run -p tidal-server -- \
|
||
cluster \
|
||
--listen 127.0.0.1:9500 \
|
||
--schema tidal-server/config/default-schema.yaml \
|
||
--topology tidal-server/config/default-cluster.yaml \
|
||
--experimental-cluster
|
||
```
|
||
|
||
The default topology spins up three regions (`us-east`, `eu-west`, `ap-south`)
|
||
with `us-east` as leader, all inside one process. On a clean start you will see a
|
||
`WARN` line stating this is experimental and single-process, an `info` line per
|
||
follower (`follower gRPC transport ready`), and finally
|
||
`listening on http://127.0.0.1:9500`.
|
||
|
||
### 1b. Multi-process (one process per region)
|
||
|
||
Pass `--region <name>` (or set `TIDAL_REGION`) to run **only that region** in this
|
||
process. The topology must declare a per-region `grpc_addr` **and** `http_addr` for
|
||
every region (see [§3](#3-topology-yaml)); siblings reach each other over those.
|
||
Launch one process per region — typically one per host, each with its own
|
||
`--data-dir`. **`--data-dir` is REQUIRED in multi-process mode** (m11p2): the
|
||
durable WAL is the replicated log itself — it is what ships to peers and what
|
||
serves their catch-up streams — so a node without one is rejected at startup:
|
||
|
||
```bash
|
||
# Region us-east (the initial leader) on host A
|
||
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
|
||
tidal-server cluster --experimental-cluster \
|
||
--region us-east \
|
||
--listen 0.0.0.0:9501 \
|
||
--schema /etc/tidal/schema.yaml \
|
||
--topology /etc/tidal/topology.yaml \
|
||
--data-dir /var/lib/tidal/us-east
|
||
|
||
# Region eu-west on host B
|
||
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
|
||
tidal-server cluster --experimental-cluster \
|
||
--region eu-west --listen 0.0.0.0:9502 \
|
||
--schema /etc/tidal/schema.yaml --topology /etc/tidal/topology.yaml \
|
||
--data-dir /var/lib/tidal/eu-west
|
||
|
||
# Region ap-south on host C
|
||
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
|
||
tidal-server cluster --experimental-cluster \
|
||
--region ap-south --listen 0.0.0.0:9503 \
|
||
--schema /etc/tidal/schema.yaml --topology /etc/tidal/topology.yaml \
|
||
--data-dir /var/lib/tidal/ap-south
|
||
```
|
||
|
||
Every process parses the **same** topology file; `RegionId`s are assigned by
|
||
declaration order, so all processes agree on the region→id mapping. Each process
|
||
binds its own region's `grpc_addr`/`http_addr` and dials its siblings' addresses.
|
||
The `--listen` HTTP address is the public gateway for that region.
|
||
|
||
#### Seed-join boot (m11p5 — adding a node without editing the topology)
|
||
|
||
A node can join an existing cluster **online**, without appearing in any
|
||
declared topology, by contacting a running peer as a **seed**:
|
||
|
||
```bash
|
||
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
|
||
tidal-server cluster --experimental-cluster \
|
||
--region eu-2 \
|
||
--listen 0.0.0.0:9504 \
|
||
--schema /etc/tidal/schema.yaml \
|
||
--topology /etc/tidal/topology.yaml \
|
||
--data-dir /var/lib/tidal/eu-2 \
|
||
--seed http://10.0.1.10:9501 \
|
||
--seed http://10.0.2.10:9502 \
|
||
--advertise-grpc eu-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9600 \
|
||
--advertise-http eu-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9504 \
|
||
--metrics 0.0.0.0:9091
|
||
```
|
||
|
||
| Flag | Meaning |
|
||
|------|---------|
|
||
| `--seed <url>` (repeatable) | One or more seed HTTP base URLs. The joiner polls each for the current leader, then `POST /cluster/join`s through it. Any reachable seed works; list a few for resilience. |
|
||
| `--advertise-grpc <host:port>` | This node's **advertised** gRPC address — what siblings dial. Required with `--seed` (the joiner has no topology entry of its own). DNS-capable. |
|
||
| `--advertise-http <host:port>` | This node's advertised HTTP gateway — what peers forward writes/status to. Required with `--seed`. |
|
||
| `--metrics <addr>` | Prometheus `/metrics` bind (the joiner has no topology `metrics_addr`). |
|
||
|
||
What happens: the joiner skips the topology's "every region declared" gate,
|
||
learns its **roster + assigned id + current term** from the seed's join
|
||
response, appends a **Learner** record to the replicated log (the leader answers
|
||
only after it is quorum-committed), persists the roster to a durable membership
|
||
cache and the term to `election_state` (persist-before-act), installs a
|
||
[snapshot](#8-write-durability-contract-the-ack-knob-and-its-cost-m11p3) when it
|
||
is behind the leader's retained WAL, then streams the live tail. The leader's
|
||
**auto-promotion** duty flips it Learner → Voter once it is within
|
||
`replication.learner_promote_lag` (default 1024) of the flushed frontier. A
|
||
**restart** boots from the cache without the seed.
|
||
|
||
> **A `--seed` boot still requires the local topology/config file** for the
|
||
> behavioral knob blocks (`replication:`, `wal:`, `election:`, `timeouts:`,
|
||
> `grpc_tls`) — its `regions:` list is ignored for the roster (the join response
|
||
> is the roster), but a bare `--seed` with neither `--topology` nor
|
||
> `TIDAL_CONFIG` **refuses to boot** naming the rule, so a joiner never silently
|
||
> inherits the compiled-in defaults' wrong ack mode, quorum timeout, or election
|
||
> timing. The k8s manifests mount the shared bootstrap ConfigMap on every pod
|
||
> including N≥3, so this costs nothing there.
|
||
|
||
> **Capability gate (m11p5 mixed-version safety).** The leader **refuses
|
||
> `/cluster/join` and every conf-change until all current voters report kind-4
|
||
> capability** — see [§8](#8-write-durability-contract-the-ack-knob-and-its-cost-m11p3)'s
|
||
> downgrade rule. Complete the binary upgrade before adding or removing nodes.
|
||
|
||
**Auth:** set `TIDAL_API_KEY=<secret>` to require `Authorization: Bearer <secret>`
|
||
on the data and `/cluster/*` mutation routes. If it is **unset the server runs
|
||
UNAUTHENTICATED and logs a WARN** — never expose an unauthenticated cluster
|
||
beyond loopback / a trusted VPC. Health probes and `/openapi.json` are always
|
||
unauthenticated. In multi-process mode set the **same** key on every process: a
|
||
forwarded/broadcast request passes the caller's `Authorization` through verbatim,
|
||
and the internal-propagation marker (`x-tidal-internal: 1`) is an inter-sibling
|
||
trust signal, **not** an auth bypass (the bearer middleware still runs first).
|
||
|
||
Useful environment variables:
|
||
|
||
| Var | Mode | Effect |
|
||
|-----|------|--------|
|
||
| `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1` | both | Opt in to cluster mode (alternative to `--experimental-cluster`). |
|
||
| `TIDAL_REGION` | multi-process | Selects the single region this process owns (alternative to `--region`). |
|
||
| `TIDAL_API_KEY` | both | Bearer token for protected routes. Unset ⇒ unauthenticated + WARN. Set the SAME key on every region in multi-process mode. |
|
||
| `TIDAL_HLC_SKEW_MS` | multi-process | Signed ms offset applied to THIS process's HLC. Affects reconcile-time LWW stamping ONLY (not signal-decay timestamps). A test/ops escape hatch for verifying causal convergence under clock skew — do not set it in normal operation. |
|
||
| `TIDAL_CONFIG` | both | Config dir holding `default-schema.yaml` / `default-cluster.yaml` (used when `--schema` / `--topology` omitted). |
|
||
| `TIDAL_SEED_STATUS_TIMEOUT_MS` | seed-join | Per-status-poll HTTP timeout during seed-join leader discovery (default `5000`). Raise on a TLS cluster under heavy CPU contention where a cold rustls handshake alone can blow a tighter budget — the joiner would then time out every poll and burn the whole 120s discovery window despite the peer being reachable. Lower only for fast loopback/test rigs. |
|
||
| `PORT` | both | Listen address. A bare port (`9500`) normalises to `0.0.0.0:9500`. |
|
||
| `TIDAL_SERVER_LOG` | both | `tracing` filter (default `info`). |
|
||
|
||
## 2. Launch via Docker
|
||
|
||
```bash
|
||
# Build the image once (build context is the repo root).
|
||
docker build -f docker/cluster/Dockerfile -t tidaldb:cluster .
|
||
|
||
# Run (press Ctrl+C to stop). The image already sets
|
||
# TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 so cluster mode starts; it still logs the
|
||
# loud experimental WARN. The default CMD is SINGLE-process cluster mode.
|
||
docker run --rm -p 9500:9500 tidaldb:cluster
|
||
```
|
||
|
||
The image bakes the default schema/topology under `/etc/tidal-server` and uses
|
||
`ENTRYPOINT ["tidal-server"]` + a `cluster …` `CMD`, so you can override the
|
||
subcommand (e.g. `docker run tidaldb:cluster standalone …`), pass `--region` for a
|
||
multi-process container, or supply your own config files:
|
||
|
||
```bash
|
||
docker run --rm -p 9500:9500 \
|
||
-e TIDAL_API_KEY=changeme \
|
||
-v "$PWD/configs/my-schema.yaml:/srv/schema.yaml:ro" \
|
||
-v "$PWD/configs/my-topology.yaml:/srv/topology.yaml:ro" \
|
||
tidaldb:cluster \
|
||
cluster \
|
||
--listen 0.0.0.0:9500 \
|
||
--schema /srv/schema.yaml \
|
||
--topology /srv/topology.yaml
|
||
```
|
||
|
||
The container ships a `HEALTHCHECK` that curls `/health`. For Kubernetes/Compose
|
||
deployment patterns see [Cross-references](#cross-references).
|
||
|
||
## 3. Topology YAML
|
||
|
||
The topology file declares regions and the leader.
|
||
|
||
**Single-process minimal default** (`tidal-server/config/default-cluster.yaml`):
|
||
|
||
```yaml
|
||
regions:
|
||
- name: us-east
|
||
- name: eu-west
|
||
- name: ap-south
|
||
leader: us-east
|
||
# Optional: OS worker threads serving cluster write/heal requests (gRPC ship).
|
||
# Defaults to available parallelism when omitted.
|
||
# write_workers: 4
|
||
```
|
||
|
||
**Multi-process** requires a per-region `grpc_addr` **and** `http_addr` (validated
|
||
at startup — a missing or syntactically invalid `host:port` is a hard error naming
|
||
the region; reachability is deliberately not probed, since siblings boot in any
|
||
order):
|
||
|
||
`grpc_addr` is an **advertised** address — a literal `host:port` OR a DNS
|
||
hostname (m11p5). A new optional per-region `grpc_bind` controls the local bind
|
||
independently of what siblings dial; the example below shows the DNS shape (one
|
||
shared file names every region by its per-pod DNS name while each pod binds
|
||
`0.0.0.0`):
|
||
|
||
```yaml
|
||
regions:
|
||
- name: us-east
|
||
grpc_addr: "tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9601" # ADVERTISED (siblings dial; DNS re-resolved on reconnect)
|
||
grpc_bind: "0.0.0.0:9601" # LOCAL bind (optional; see derivation rule below)
|
||
http_addr: "10.0.1.10:9501" # this region's public HTTP gateway (forwarding + status)
|
||
metrics_addr: "10.0.1.10:9091" # optional Prometheus /metrics listener (set it in production)
|
||
- name: eu-west
|
||
grpc_addr: "tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9602"
|
||
grpc_bind: "0.0.0.0:9602"
|
||
http_addr: "10.0.2.10:9502"
|
||
metrics_addr: "10.0.2.10:9091"
|
||
- name: ap-south
|
||
grpc_addr: "tidaldb-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9603"
|
||
grpc_bind: "0.0.0.0:9603"
|
||
http_addr: "10.0.3.10:9503"
|
||
metrics_addr: "10.0.3.10:9091"
|
||
leader: us-east
|
||
# Optional m11p1 tuning blocks (engine defaults shown):
|
||
# replication:
|
||
# batch_max_events: 256 # events coalesced per shipped batch (1-256)
|
||
# window: 4 # in-flight batches per peer (1-64)
|
||
# retry_ms: 100 # backoff before a transiently-failed batch retries
|
||
# catchup_retry_ms: 30000 # backoff before a FAILED catch-up pull re-pulls (m11p4)
|
||
# snapshot_artifact_ttl_ms: 600000 # staged-snapshot reuse window (m11p5; never unpins an active consumer)
|
||
# learner_promote_lag: 1024 # learner→voter promotion distance + readiness hysteresis (m11p5)
|
||
# reseed_self_restart: false # drain+exit(0) when reseed_required latches (m11p5; k8s sets true)
|
||
# wal:
|
||
# batch_size: 100 # events per group-commit fsync (1-256)
|
||
# batch_timeout_ms: 10 # max wait before a partial batch flushes
|
||
# election: # m11p4 failure-detector / election timers (defaults shown)
|
||
# heartbeat_interval_ms: 300
|
||
# election_timeout_min_ms: 1500 # validated: leader_lease_ms + heartbeat_interval_ms < this
|
||
# election_timeout_max_ms: 3000
|
||
# leader_lease_ms: 900
|
||
# auto_election: true # false → pre-m11p4 operator-driven failover
|
||
```
|
||
|
||
**`grpc_bind` derivation (m11p5 §1):** `grpc_bind` present → bind it. Absent +
|
||
`grpc_addr` parses as a literal SocketAddr → bind that (today's behavior,
|
||
byte-for-byte — every existing IP topology keeps working). Absent + `grpc_addr`
|
||
is a hostname → bind `0.0.0.0:<port from grpc_addr>`. Because the peer-dial path
|
||
no longer parses `grpc_addr` as a SocketAddr, hyper re-resolves the hostname on
|
||
every reconnect — a pod rescheduled onto a new IP is reached with no peer
|
||
restarts. **TLS note:** SNI follows the URI host, so DNS peer names require
|
||
DNS-SAN certs (see `grpc_tls` below); no code change.
|
||
|
||
### 3a. Sharding × replication (`shards:`, m11p6)
|
||
|
||
Absent `shards:`, the cluster is **1 shard × RF = all regions** — one replicated
|
||
log, one elected leader, byte-for-byte everything above. Add a `shards:` block to
|
||
split the entity space into **S groups, each a replication group at RF with its
|
||
own elected leader**, leaders balanced across nodes.
|
||
|
||
> **The k3s reference cluster runs this block enabled: S=3, full placement, RF3**
|
||
> ([`k8s/cluster/topology-configmap.yaml`](../../k8s/cluster/topology-configmap.yaml)).
|
||
> Every pod replicates all three groups; tidaldb-0 leads shard 0, tidaldb-1 shard 1,
|
||
> tidaldb-2 shard 2, binding gRPC 9601/9602/9603 respectively. Full placement is
|
||
> the production shape, not a sharding experiment — a single pod loss never loses a
|
||
> group's quorum (2 of 3 survive per group) and any pod can serve a corpus-wide
|
||
> read locally.
|
||
|
||
Writes
|
||
to `/items`//`/embeddings`//`/signals` hash-route (the engine's FNV-1a router) to
|
||
the owning group's leader and replicate at RF; any gateway accepts any write.
|
||
|
||
```yaml
|
||
regions: # the NODE list (identity + addresses)
|
||
- { name: us-east, grpc_addr: "10.0.1.10:9601", http_addr: "10.0.1.10:9501" }
|
||
- { name: eu-west, grpc_addr: "10.0.2.10:9601", http_addr: "10.0.2.10:9501" }
|
||
- { name: ap-south, grpc_addr: "10.0.3.10:9601", http_addr: "10.0.3.10:9501" }
|
||
shards: # NEW — S groups (dense ids 0..S)
|
||
- id: 0
|
||
leader: us-east # term-0 / preferred leader (balance: group i → node i)
|
||
replicas: # the RF nodes hosting this group
|
||
- { node: us-east } # grpc_addr optional → derived node base port + shard id
|
||
- { node: eu-west }
|
||
- { node: ap-south }
|
||
- { id: 1, leader: eu-west, replicas: [ {node: us-east}, {node: eu-west}, {node: ap-south} ] }
|
||
- { id: 2, leader: ap-south, replicas: [ {node: us-east}, {node: eu-west}, {node: ap-south} ] }
|
||
leader: us-east # legacy field, ignored once shards: is set
|
||
```
|
||
|
||
- **Ports & dirs.** A node hosting several groups binds one gRPC port PER group:
|
||
set `replicas[].grpc_addr` explicitly, or omit it to derive `node base port +
|
||
shard id`. Each group's data lives under `<data_dir>/shard-{id:05}/`; the
|
||
single-group legacy layout (`shards:` absent) keeps `<data_dir>` verbatim, so
|
||
existing clusters restart unchanged.
|
||
- **Placement.** `replicas` need not be every node (RF < N is valid). The exit
|
||
gate and the simplest production shape are **full placement** (every node
|
||
replicates every group); a strict-subset placement is supported for writes but
|
||
see the read caveat in [§7](#7-sharded-scatter-gather-api).
|
||
- **Validation.** Shard ids must be dense and unique; every `leader`/`replica.node`
|
||
names a declared region; a group needs ≥1 replica; co-hosted groups must not
|
||
collide on a derived gRPC port.
|
||
|
||
**Per-shard admin (`?shard=`).** With `shards:` set, every cluster admin verb
|
||
takes an optional `?shard=<id>` selecting which hosted group to act on; omitted =
|
||
the node's lowest-id hosted group (and the only group when S=1, so S=1 URLs are
|
||
unchanged). The selector is forwarded on intra-group hops, and `NotLeader` names
|
||
the group. See [§6a](#6a-rebalancing-m11p6) for the rebalancing verbs.
|
||
|
||
Fields:
|
||
|
||
| Field | Single-process | Multi-process | Meaning |
|
||
|-------|----------------|---------------|---------|
|
||
| `regions[].name` | required | required | Region name used everywhere in the HTTP API (`?region=`, `/cluster/promote`, etc.). Must be unique. |
|
||
| `regions[].grpc_addr` | optional | **required** | This region's **advertised** gRPC replication address — what siblings dial. Since m11p5 it may be a literal `host:port` OR a **DNS hostname** (re-resolved on every reconnect). In single-process mode **omit it** — the server allocates a free loopback port and self-heals a bind race by retrying on a fresh port. In multi-process mode it is required and is tried exactly once. |
|
||
| `regions[].grpc_bind` | unused | optional | This region's **local** gRPC bind `host:port` (m11p5), independent of the advertised `grpc_addr`. Omitted: a literal `grpc_addr` binds itself; a hostname `grpc_addr` binds `0.0.0.0:<its port>`. Set it to bind a specific interface while advertising a DNS name. |
|
||
| `regions[].http_addr` | unused | **required** | This region's public HTTP gateway address, used by siblings for write/read forwarding and status aggregation. Unused in single-process mode. |
|
||
| `regions[].grpc_tls` | unused | optional | TLS material for this region's gRPC transport: `ca_cert`, `server_cert`, `server_key` (PEM paths), plus optional `client_cert`/`client_key` for mTLS. Omitted ⇒ plaintext, the right posture for loopback/VPC topologies. **DNS `grpc_addr` requires DNS-SAN certs** (SNI follows the dialed hostname). |
|
||
| `regions[].metrics_addr` | unused | optional | Bind address for this region's Prometheus `/metrics` listener (m11p1; cluster mode previously had none). Omitted ⇒ no metrics endpoint. Set it in every production topology, and bind it internally — the endpoint is unauthenticated. |
|
||
| `leader` | required | required | Must name one of the declared regions. The initial write leader. |
|
||
| `write_workers` | optional | optional | Size of the runtime-free OS-thread pool that admission-controls cluster writes (`/signals` staging) and runs `/cluster/heal`'s blocking redelivery. Bounded queue ⇒ 429 backpressure. Must be ≥ 1 when given. |
|
||
| `timeouts.broadcast_peer_secs` | unused | optional | Per-peer budget (seconds) for the `/cluster/promote` fan-out (the only remaining peer fan-out — the m11p2 log replaced the item/embedding broadcast). Default 2s — right for loopback/VPC; raise it for WAN topologies where a distant region cannot answer in 2s. Must be ≥ 1 when given. |
|
||
| `replication.batch_max_events` | unused | optional | Max relay events coalesced into one shipped batch (1–256, the WAL wire-format ceiling). Default 256. |
|
||
| `replication.window` | unused | optional | In-flight batches per peer (1–64). 1 = strictly in-order shipping; higher pipelines across the peer RTT (out-of-order arrivals park gap-aware on the receiver). Default 4. |
|
||
| `replication.retry_ms` | unused | optional | Backoff (ms) before a transiently-failed batch ship retries. Default 100. |
|
||
| `replication.ack` | unused | optional | Deployment-default write acknowledgment: `leader` (default) or `quorum` (majority-durable — see [§8](#8-write-durability-contract-the-ack-knob-and-its-cost-m11p3)). Per-request override: the `x-tidal-ack` header. |
|
||
| `replication.quorum_timeout_ms` | unused | optional | Budget an `ack=quorum` write waits for the commit index before the retryable 503 naming the laggards. Default 2000. |
|
||
| `replication.catchup_retry_ms` | unused | optional | Backoff (ms) before a FAILED catch-up pull re-pulls on a timer (m11p4) — lets an idle cluster self-heal a follower whose pull failed during a rolling restart. Default 30000. Must be ≥ 1 when given. |
|
||
| `replication.snapshot_artifact_ttl_ms` | unused | optional | Staged-snapshot reuse window (m11p5), counted from the last fetch's completion. Governs artifact REUSE only — **never** unpins the WAL retention of an active consumer; a hard cap (4×) force-drops a never-releasing pin (`tidaldb_cluster_snapshot_pin_force_drops_total`). Default 600000 (10 min). Must be ≥ 1. |
|
||
| `replication.learner_promote_lag` | unused | optional | Learner→voter promotion distance AND the readiness-convergence hysteresis threshold (m11p5), in events. Default 1024. Must be ≥ 1. |
|
||
| `replication.reseed_self_restart` | unused | optional | Drain + clean-exit(0) once the durable `reseed_required` marker latches (m11p5; k8s sets it `true`). **Refused** when the remaining voters can't sustain quorum without this node. Default `false`. |
|
||
| `election.heartbeat_interval_ms` | unused | optional | Leader heartbeat interval (m11p4). Default 300. |
|
||
| `election.election_timeout_min_ms` / `..max_ms` | unused | optional | Randomized follower election timeout window (m11p4). Defaults 1500 / 3000. Validated: `leader_lease_ms + heartbeat_interval_ms < election_timeout_min_ms`. |
|
||
| `election.leader_lease_ms` | unused | optional | Leader freshness lease — a leader that loses majority contact steps down within it (m11p4). Default 900. |
|
||
| `election.auto_election` | unused | optional | `true` (default) = automatic failover; `false` = pre-m11p4 operator-driven posture (no auto elections, no check-quorum step-down). |
|
||
| `wal.batch_size` | optional | optional | Events per WAL group-commit fsync (1–256). Default 100. |
|
||
| `wal.batch_timeout_ms` | optional | optional | Max ms a partial group-commit batch waits before flushing. Default 10. Tune against the measured `tidaldb_cluster_wal_fsync_us` on the deployment's volume. |
|
||
|
||
The schema YAML (signals / text_fields / embedding_slots / profiles) is the same
|
||
format the standalone server loads — see [API.md](../../API.md) and
|
||
[QUICKSTART.md](../../QUICKSTART.md) for the full schema/profile grammar and the
|
||
built-in ranking profiles. Topology only adds the region layout above.
|
||
|
||
> **Personalization correctness note.** Preference-vector personalization (the
|
||
> taste vector that powers `for_you`) only updates for signals declared
|
||
> `positive_engagement: true` in the schema. The cluster `/signals` route writes
|
||
> **global** signals (no `user_id` / `creator_id` context), so it never folds
|
||
> item embeddings into a user's taste vector regardless of the
|
||
> `positive_engagement` flag. Personalized writes go through the embedded engine
|
||
> (`signal_with_context`), not this HTTP route. The exception is `/hardnegs`,
|
||
> which records a **user-scoped** hide that converges across regions via
|
||
> [`/cluster/reconcile`](#6-cluster-management-api).
|
||
|
||
## 4. gRPC replication transport (tidal-net)
|
||
|
||
Replication between regions is the real `tidal-net` `WalShipping` gRPC service,
|
||
not in-process channels. The facts you need to operate and firewall it:
|
||
|
||
| Property | Value | Notes |
|
||
|----------|-------|-------|
|
||
| Service | `WalShipping` (proto `tidal.replication.v1`) | RPCs: **`ShipSegment`** (unary; the live-tail push path), **`StreamSegments`** (server-streaming; the follower-pulled catch-up path since m11p2), **`FetchSnapshot`** (server-streaming; the m11p5 snapshot transfer for joiners + reseeds), **`JoinCluster`** (m11p5 conf-change), and **`Heartbeat`** (ControlPlane health). |
|
||
| Bind address | per-region `grpc_addr` (multi-process); an **auto-allocated** loopback port (single-process) | Default dev band `59520–59529` if you pin one. |
|
||
| Transport security | mTLS via `TlsConfig` (`ca_cert`, `server_cert`, `server_key`, optional `client_cert` / `client_key`) | `insecure = true` (plaintext) is the loopback/VPC default in this phase. A peer with no TLS config **must** set `insecure`. |
|
||
| Max payload | **64 MiB** | Both encoder and decoder codec limits are raised to this; pinned by a compile-time assert to the engine's `InProcessTransport` limit so both ends agree. WAL segments default to 16 MiB. |
|
||
| Circuit breaker | per-peer, **threshold 5**, **reset 30s** | After 5 consecutive ship failures the breaker opens; it stays open for 30s, then the next attempt probes (HalfOpen → Closed on success, Open again on failure). Backpressure (channel full) does **not** trip it. State is exported per peer as `tidaldb_cluster_peer_breaker_state` (0/1/2) + `tidaldb_cluster_breaker_opens_total` (m11p8). **Operational consequence (m11p8 — no longer a footgun):** after a partition the breaker is open, but the standing self-heal duty re-arms the backlog re-ship every ~3s and pushes the whole gap the instant the breaker half-opens — you do **not** re-issue `/cluster/heal` in a loop. Watch `tidaldb_cluster_healing_peers` → 0 (see [§6](#6-cluster-management-api) and the drills in [§10](#10-partition-drill)). |
|
||
| Timeouts (defaults) | connect 5s, request 10s, keep-alive PING every 10s / 5s ACK | A blackholed peer fails fast instead of stalling the single-threaded shipper. |
|
||
|
||
`ShipSegment` carries the WAL segment id, BLAKE3-validated payload bytes, event
|
||
count, and the leader's authoritative `leader_last_seq` (so a follower can
|
||
advance its replication-lag high-water-mark even for an all-local segment that
|
||
filters to empty). The follower's segment-receiver thread drains and applies it
|
||
on arrival. The engine's applied high-water-mark is a **contiguous frontier with a
|
||
bounded ahead-buffer**, so out-of-order eager ships can never swallow a sequence
|
||
gap (a silent-data-loss class fixed in m8p10).
|
||
|
||
> **Operational guard:** the gRPC replication ports and the Prometheus `/metrics`
|
||
> endpoint are **UNAUTHENTICATED**. Bind them to loopback or a cluster-internal
|
||
> network only — never the public interface.
|
||
|
||
## 5. Core HTTP API
|
||
|
||
All routes are JSON unless noted. Examples assume `BASE=http://localhost:9501`
|
||
(any region's gateway in multi-process mode; the single `--listen` address in
|
||
single-process mode) and, when `TIDAL_API_KEY` is set,
|
||
`AUTH='-H "Authorization: Bearer $TIDAL_API_KEY"'`. Drop the `-H` header when
|
||
running unauthenticated.
|
||
|
||
Middleware mirrors standalone: 30s request timeout (408), 100 max in-flight
|
||
(429), 2 MB body limit (413), and an `x-request-id` on every response. Health
|
||
probes sit **outside** the load-shedding stack so liveness/readiness are never
|
||
queued or timed out under saturation.
|
||
|
||
> **Multi-process routing (one coherent surface).** Any region's gateway accepts
|
||
> any operation and routes it to the node that owns it:
|
||
> * **Writes** (`/signals`, `/items`, `/embeddings`, `/hardnegs`) on a non-leader
|
||
> **forward to the leader** (the caller's `Authorization` passes through); the
|
||
> client sees the leader's status/body. A leader that is unreachable degrades to
|
||
> a `503` naming the leader, never a hang.
|
||
> * **Reads** (`/feed`, `/search`) default to the **LOCAL** region. This is the
|
||
> key difference from single-process mode, whose default read is the leader. A
|
||
> `?region=<other>` read **forwards** to that region's process.
|
||
> * `/cluster/promote`, `/cluster/partition`, `/cluster/heal` route to / fan out
|
||
> from the leader as documented in [§6](#6-cluster-management-api).
|
||
|
||
### Health
|
||
|
||
```bash
|
||
curl "$BASE/health" # 200 ok / 503 while draining; reports mode, region, leader
|
||
curl "$BASE/health/startup" # always 200
|
||
curl "$BASE/health/live" # always 200
|
||
curl "$BASE/openapi.json" # served OpenAPI 3.1, UNAUTHENTICATED — canonical HTTP reference
|
||
```
|
||
|
||
`/health` returns `{ "ok": true, "service": "tidaldb", "mode": "cluster",
|
||
"region": "us-east", "leader": "us-east", ... }`. The `/openapi.json` document is
|
||
the machine-readable source of truth for the **data**, **`/cluster/*`**,
|
||
**`/sharded/*`**, and **`/hardnegs`** request/response shapes (the health probes
|
||
are intentionally outside the documented API surface).
|
||
|
||
### Register items & embeddings (one replicated log — m11p2)
|
||
|
||
Items and embeddings ride the **same replicated WAL stream as signals**: the
|
||
leader journals each write as a kind-1 (item metadata) or kind-2 (embedding)
|
||
WAL record BEFORE touching storage, and followers apply it from the log —
|
||
live pushes for the hot tail, the `StreamSegments` catch-up stream for
|
||
history. There is **no HTTP broadcast** anymore (the m8p10 side channel and
|
||
its bug classes were deleted in m11p2), so the responses are plain statuses:
|
||
|
||
```bash
|
||
curl -X POST "$BASE/items" \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{ "entity_id": 1, "metadata": { "title": "Jazz Piano", "category": "music" } }'
|
||
# → 201 Created (item durably journaled into the replicated log + applied)
|
||
|
||
curl -X POST "$BASE/embeddings" \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{ "entity_id": 1, "values": [0.1, 0.2, 0.3, 0.4] }'
|
||
# → 204 No Content
|
||
```
|
||
|
||
The `201`/`204` asserts **leader durability** and carries **`x-tidal-seq`**
|
||
(the record's seqno in the replicated log); `x-tidal-ack: quorum` upgrades it
|
||
to a majority-durable ack (see
|
||
[§8](#8-write-durability-contract-the-ack-knob-and-its-cost-m11p3)). The
|
||
record is fsynced into the stream every follower receives (live push, or
|
||
pull-based catch-up after downtime — see
|
||
[§6 heal](#6-cluster-management-api)). A write to a non-leader forwards to
|
||
the leader transparently (the ack header and seq header travel through the
|
||
forward). A down/partitioned peer needs no backfill bookkeeping: it
|
||
converges from the log when it returns.
|
||
|
||
Embeddings: tidalDB does **not** generate vectors — the caller brings them. The
|
||
write L2-normalizes and inserts into the HNSW index. Dimensions are **strict**:
|
||
they must equal the slot's declared dimensions (min 2, max 4096) or the insert is
|
||
**rejected with a 400**, and **zero-norm / non-finite vectors are rejected (400)**
|
||
too. Both are checked **before the WAL append**, so a malformed vector never
|
||
becomes durable and never enters the replication stream — see
|
||
[§16.6](#166-a-halted-receiver-a-poison-record-in-the-log) for the outage that
|
||
ordering caused when it was the other way round. (These used to surface as `500`;
|
||
that was a real defect, not just an imprecise status code — the record had already
|
||
been journaled by the time the error was produced.) `RETRIEVE` / `SEARCH` route
|
||
through the **first declared embedding slot only**; multi-modal apps must fuse
|
||
offline or use separate entity kinds.
|
||
|
||
### Record signals (cluster `/signals` = global only)
|
||
|
||
```bash
|
||
curl -X POST "$BASE/signals" \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{ "entity_id": 1, "signal": "view", "weight": 1.0 }'
|
||
# → 204 No Content (see the durability contract in §8)
|
||
```
|
||
|
||
The signal name must be declared in the schema (an undeclared name returns **400**
|
||
naming it). The 204 carries **`x-tidal-seq`** (the write's replicated-log seqno),
|
||
and `x-tidal-ack: quorum` upgrades it to a majority-durable ack — see
|
||
[§8](#8-write-durability-contract-the-ack-knob-and-its-cost-m11p3). This route
|
||
records a **global** signal on the leader and ships it to
|
||
followers over the replicated WAL stream; it does not personalize (see the
|
||
personalization note in [§3](#3-topology-yaml)). On a non-leader gateway it
|
||
forwards to the leader transparently and still returns `204`.
|
||
|
||
### Hard negatives (user-scoped hides)
|
||
|
||
```bash
|
||
curl -X POST "$BASE/hardnegs" \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{ "user_id": 42, "item_id": 7 }'
|
||
# → 204 No Content
|
||
```
|
||
|
||
Records a user-scoped hide on the leader (a non-leader gateway forwards). The item
|
||
is filtered from that user's `/feed?user_id=42`. Hard negatives converge across
|
||
regions via the LWW-resolved [`/cluster/reconcile`](#6-cluster-management-api) CRDT
|
||
path — NOT the signal WAL relay and NOT a broadcast.
|
||
|
||
### Retrieve and search (region-pinned reads)
|
||
|
||
```bash
|
||
# Default read region is the LOCAL region (multi-process mode).
|
||
curl "$BASE/feed?user_id=42&profile=for_you&limit=20"
|
||
curl "$BASE/search?query=jazz%20piano&user_id=42&limit=5"
|
||
|
||
# Pin a read to a specific region. The gateway forwards to that region's process.
|
||
# Followers may lag the leader (and lag jumps during a partition) — use this for
|
||
# canary reads and lag verification.
|
||
curl "$BASE/feed?profile=trending®ion=eu-west"
|
||
```
|
||
|
||
`?region=` accepts any declared region name; an unknown name returns **400**. Omit
|
||
it and the read serves from the **LOCAL** region (multi-process) / the current
|
||
leader (single-process). `limit` is clamped to 1000 at the trust boundary (a
|
||
larger value cannot amplify memory unboundedly). The `for_you` profile applies the
|
||
built-in diversity defaults (`max_per_creator=2`, `format_mix_max_fraction=0.4`,
|
||
`exploration=0.1`).
|
||
|
||
## 6. Cluster management API
|
||
|
||
### Check cluster status
|
||
|
||
Two views. `/cluster/status/local` reports THIS node's own replication/leadership
|
||
state (no peer calls); `/cluster/status` aggregates EVERY region (the gateway calls
|
||
each peer's `/cluster/status/local` concurrently with a tight per-peer budget).
|
||
|
||
```bash
|
||
curl "$BASE/cluster/status/local" | jq
|
||
```
|
||
|
||
```json
|
||
{
|
||
"region": "ap-south",
|
||
"is_leader": false,
|
||
"leader": "us-east",
|
||
"last_seq": 0,
|
||
"applied_events": 124,
|
||
"lag_events": 1,
|
||
"partitioned": [],
|
||
"reachable": true
|
||
}
|
||
```
|
||
|
||
```bash
|
||
curl "$BASE/cluster/status" | jq
|
||
```
|
||
|
||
```json
|
||
{
|
||
"leader": "us-east",
|
||
"relay_log_len": 125,
|
||
"regions": [
|
||
{ "name": "us-east", "applied_events": 125, "lag_events": 0, "partitioned": false, "reachable": true },
|
||
{ "name": "eu-west", "applied_events": 125, "lag_events": 0, "partitioned": false, "reachable": true },
|
||
{ "name": "ap-south", "applied_events": 124, "lag_events": 1, "partitioned": false, "reachable": true }
|
||
]
|
||
}
|
||
```
|
||
|
||
`relay_log_len` is the leader's high-water-mark (`last_seq`); each region's
|
||
`lag_events` is `relay_log_len − applied_events` (saturating). Since m11p3
|
||
`/cluster/status/local` also reports the node's `ack` default and, on the
|
||
leader, `commit_index` — the highest seqno a majority of the replica set
|
||
durably holds (`last_seq − commit_index` is the quorum lag). A region the
|
||
gateway **cannot reach** within the per-peer budget is reported honestly as
|
||
`reachable: false`, `partitioned: true`, `applied_events: 0`, and worst-case lag
|
||
(`lag_events == relay_log_len`). A non-zero, *growing* lag on a reachable region is
|
||
the signal that it is partitioned (ship-skip) or its segment-receiver is wedged.
|
||
|
||
### Promote a new leader
|
||
|
||
```bash
|
||
curl -X POST "$BASE/cluster/promote" \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{ "region": "eu-west" }'
|
||
# → { "ok": true, "leader": "eu-west", "acked": ["eu-west","ap-south"], "failed": [] }
|
||
```
|
||
|
||
The gateway first resolves the TARGET's **stream baseline** — the new leader's
|
||
WAL flushed frontier at promotion, persisted in its `data_dir/stream_baseline`
|
||
— then fans the promote out to every peer carrying it (internal marker, so
|
||
each applies locally and does not re-fan). Peers jump their applied frontier
|
||
for the new leader's shard to the baseline: everything at or below it is
|
||
pre-stream history (replicated applies of the OLD stream), not data. The
|
||
response carries `baseline` plus the `acked`/`failed` fan-out report; a peer
|
||
in `failed` (e.g. a dead old leader) self-corrects — its first parked batch
|
||
from the new stream triggers a catch-up pull whose chunks announce the
|
||
baseline. After promotion `/cluster/status` reports the new leader; new
|
||
writes route there; the new leader's ship queue activates and the demoted
|
||
node's deactivates. An unknown region returns **400**. Since m11p4 `promote` is
|
||
a **fenced transfer** (it drains the target, then sanctions an election), not
|
||
the availability mechanism — automatic failover handles a dead leader with no
|
||
operator action ([§9.1](#91-automatic-failover-m11p4--the-default)); this verb
|
||
is for maintenance and deliberate successor choice
|
||
([§9.2](#92-manual-promote-a-fenced-transfer-maintenance--override)).
|
||
|
||
### Simulate a partition & heal
|
||
|
||
There are **two** ways to partition a region; the runbook drills demonstrate both
|
||
([§10](#10-partition-drill)):
|
||
|
||
1. **Simulated ship-skip flag** — `/cluster/partition` tells the leader to stop
|
||
shipping to the named region (no sockets touched). Good for a controlled,
|
||
reversible lag demo.
|
||
2. **Real network partition** — sever the actual TCP path between processes
|
||
(firewall / proxy). The chaos suite uses a root-free in-harness TCP relay;
|
||
operators can use `iptables`/`pfctl` (see [§10](#10-partition-drill)).
|
||
|
||
```bash
|
||
# Simulated: isolate ap-south — leader ships skip this follower, so its lag climbs.
|
||
curl -X POST "$BASE/cluster/partition" \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{ "region": "ap-south" }'
|
||
# → { "ok": true, "partitioned": "ap-south" }
|
||
|
||
# Heal: resume shipping past everything the follower reports applied, and
|
||
# nudge it to PULL any history that rotated out of the leader's ship tail
|
||
# (the single recovery verb — log catch-up IS the full heal since m11p2).
|
||
curl -X POST "$BASE/cluster/heal" \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{ "region": "ap-south" }'
|
||
# → { "ok": true, "healed": "ap-south" }
|
||
```
|
||
|
||
`/cluster/heal` is the **single recovery verb**, and since m11p2 it is pure
|
||
log catch-up: it clears the partition flag, **resumes** the ship queue past
|
||
the follower's reported applied seqno (retries of data the follower already
|
||
holds prune automatically — every ship ack piggybacks the follower's applied
|
||
seqno), and **nudges** the follower to pull anything older than the leader's
|
||
in-memory ship tail via its `StreamSegments` catch-up stream over the
|
||
leader's durable WAL segments (`POST /cluster/catchup`, internal; the nudge
|
||
forwards the healing operator's own bearer credential). Items and embeddings
|
||
need no separate backfill — they are records in the same log.
|
||
|
||
> **Convergence is self-driving (m11p8 — the operator no longer loops heal).**
|
||
> The per-peer ship senders retry parked batches every `replication.retry_ms`
|
||
> (default 100ms); a follower that detects a gap pulls the catch-up stream
|
||
> itself (also on boot, so a restarted node converges with no operator action
|
||
> at all — the tier-3 `mp_items_ride_the_log_and_catchup_stream` proves it).
|
||
> On top of that, a **standing leader heal duty** runs every ~3s: for any
|
||
> non-partitioned peer whose ship breaker is open AND that trails the leader,
|
||
> it re-arms the backlog re-ship from the peer's durable mark, so the moment the
|
||
> breaker half-opens (threshold 5, reset 30s — see
|
||
> [§4](#4-grpc-replication-transport-tidal-net)) the leader pushes the WHOLE
|
||
> gap. This closes the old footgun: you **no longer re-issue `/cluster/heal`
|
||
> until lag 0** — the server drives it. Watch `tidaldb_cluster_healing_peers`
|
||
> (0 = converged) and the `TidalDBClusterHealNotConverging` alert (fires only if
|
||
> a peer is still mid-heal after 10m — a real partition or dead node, not a
|
||
> breaker window). `/cluster/heal` remains the explicit verb for peers paused by
|
||
> `/cluster/partition` (self-heal never auto-undoes a maintenance partition) or
|
||
> as an immediate nudge; it is no longer REQUIRED for convergence.
|
||
|
||
### Reconcile (cross-region CRDT convergence)
|
||
|
||
```bash
|
||
curl -X POST "$BASE/cluster/reconcile" \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{ "region": "ap-south" }'
|
||
# → { "ok": true, "region": "ap-south",
|
||
# "local_elapsed_ms": 0, "remote_elapsed_ms": 1, "ops_applied": 3 }
|
||
```
|
||
|
||
`/cluster/reconcile` exchanges a CRDT state snapshot with the target region: this
|
||
node ships its snapshot into the target's merge AND applies the target's pre-merge
|
||
snapshot back, so **both sides converge to identical state** by deterministic LWW.
|
||
This is how **hard negatives** (recorded via `/hardnegs`) converge across regions —
|
||
they do not ride the WAL relay. `local_elapsed_ms` / `remote_elapsed_ms` are the
|
||
merge+apply times each side measured (not the HTTP round-trip); both are typically
|
||
0–1ms. Reconcile is **idempotent**: a repeat reconcile of already-converged regions
|
||
is an exact no-op on scores (no drift). `/cluster/reconcile/snapshot` is the
|
||
INTERNAL snapshot-exchange leg this verb drives (marker required); operators never
|
||
call it directly.
|
||
|
||
### Membership verbs (m11p5 — online add / remove / inspect / reseed)
|
||
|
||
Membership is **data on the replicated log** (kind-4 records, latest wins,
|
||
folded into a `ClusterMembership` cell). All four verbs route to / are answered
|
||
by the leader and are **quorum-commit-gated**, one change at a time. The leader
|
||
**refuses every conf-change until all current voters report kind-4 capability**
|
||
(the mixed-version safety gate — complete the binary upgrade first).
|
||
|
||
```bash
|
||
# Add a node (idempotent by name): forwards to the leader, which assigns the
|
||
# next id, appends a Learner record, and answers AFTER quorum-commit.
|
||
curl -X POST "$BASE/cluster/join" \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{ "name": "eu-2",
|
||
"grpc_addr": "eu-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9600",
|
||
"http_addr": "eu-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9504" }'
|
||
# → { "id": 4, "role": "learner", "term": 7, "leader": "us-east", "members": [ … ] }
|
||
```
|
||
|
||
The joiner normally seed-joins itself ([§1b](#1b-multi-process-one-process-per-region));
|
||
this verb is the same conf-change for tooling. A re-join from a known name
|
||
returns its existing id and current role and appends nothing. The leader's
|
||
**auto-promotion** duty (a standing duty, re-armed on every activation and
|
||
membership apply — it survives the joining-era leader's death) flips the learner
|
||
to Voter once it is within `replication.learner_promote_lag` of the flushed
|
||
frontier; `promotion_pending (lag=N)` in `/cluster/status/local` makes a stuck
|
||
scale-up diagnosable.
|
||
|
||
```bash
|
||
# Inspect the applied roster (ids, names, addresses, roles, conf version).
|
||
curl "$BASE/cluster/members" | jq
|
||
```
|
||
|
||
```bash
|
||
# Remove a node: appends a Removed tombstone (quorum-commit-gated). The peer's
|
||
# ship cell is retired only AFTER the record is delivered-to/acked-by the removed
|
||
# peer (bounded give-up → tidaldb_cluster_remove_delivery_giveups_total); the
|
||
# removed node's readiness flips to 503 and it stops campaigning. Its id is BURNED
|
||
# (never renumbered, never reused).
|
||
curl -X POST "$BASE/cluster/members/remove" \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{ "region": "eu-2" }'
|
||
# → { "removed": "eu-2", "membership_version": 9 }
|
||
```
|
||
|
||
```bash
|
||
# Force a reseed on demand: latches the durable reseed_required marker. The node
|
||
# keeps serving degraded (voting enabled) and reseeds via snapshot on its NEXT
|
||
# boot (or self-restarts if replication.reseed_self_restart is true and quorum
|
||
# can be sustained without it). See §9.1.
|
||
curl -X POST "$BASE/cluster/reseed"
|
||
# → { "reseed_required": true }
|
||
```
|
||
|
||
> **Reseed escapes a divergent suffix, not a poison record.** The snapshot is
|
||
> captured at the **leader's applied frontier**. If the group is stalled because
|
||
> a record in the log cannot be applied by anyone, that frontier is itself
|
||
> *behind* the bad record — the reseeded node installs the snapshot and replays
|
||
> straight back into it. Do not reach for this verb on a halted receiver; see
|
||
> [§16.6](#166-a-halted-receiver-a-poison-record-in-the-log).
|
||
|
||
**Scale-down order (decommission):** call `/cluster/members/remove` **first**
|
||
(so the cluster's quorum math shrinks before the node disappears), wait for the
|
||
record to quorum-commit, then stop / delete the node — lowest ordinal last under
|
||
a StatefulSet.
|
||
|
||
### 6a. Rebalancing (m11p6)
|
||
|
||
With a `shards:` topology ([§3a](#3a-sharding--replication-shards-m11p6)) the
|
||
operator moves load between nodes per group. Automatic rate-limited rebalancing
|
||
is explicitly later; m11p6 ships the manual verbs (each reuses the proven
|
||
m11p4/m11p5 machinery, scoped to one group).
|
||
|
||
**Move a group's leadership** (the rebalance-back-to-preferred path) — a fenced
|
||
transfer (catch-up drain → `TimeoutNow` → term+1 election), identical to
|
||
`/cluster/promote?shard=<id>`:
|
||
|
||
```bash
|
||
curl -X POST "$BASE/cluster/shards/0/transfer" \
|
||
-H 'content-type: application/json' \
|
||
-d '{"region":"eu-west"}'
|
||
# only group 0's leader moves; groups 1,2 are untouched
|
||
```
|
||
|
||
**Add / remove a replica of a group** — `add` runs the m11p5 join (the named node
|
||
joins group `{id}` as a Learner, catches up via snapshot+stream, auto-promotes to
|
||
Voter); `remove` runs the m11p5 fenced removal on that group's log:
|
||
|
||
```bash
|
||
curl -X POST "$BASE/cluster/shards/2/replicas" \
|
||
-H 'content-type: application/json' \
|
||
-d '{"action":"add","name":"region-3","grpc_addr":"10.0.4.10:9603","http_addr":"10.0.4.10:9504"}'
|
||
```
|
||
|
||
> **Multi-group caveat (tracked follow-up).** A node's readiness is currently
|
||
> node-global across its co-hosted groups, so removing a node from ONE group of a
|
||
> node that hosts several would wrongly flip the whole node's readiness. Use the
|
||
> `replicas` verbs today for nodes that host a SINGLE group (or a brand-new node
|
||
> joining one group); per-group-aware readiness is the elasticity follow-up.
|
||
|
||
**Inspect / target one group.** Every admin verb takes `?shard=<id>`;
|
||
`/cluster/status/local` always lists a per-group `shards[]` array (leader, term,
|
||
role, applied, lag, commit index per group), and `/cluster/members?shard=<id>`
|
||
returns that group's roster.
|
||
|
||
## 7. Sharded scatter-gather API
|
||
|
||
The `/sharded/*` routes hash-partition entities across regions
|
||
(`hash(entity_id) % num_shards`, using the engine's own `ShardRouter` so writes
|
||
and reads never disagree). Writes route to the single owning region; reads fan out
|
||
to **all** regions and K-way merge by score.
|
||
|
||
**Writes are SINGLE-COPY and require an explicit opt-in.** A `/sharded/*` write is
|
||
applied to the owning region's LOCAL store with **no WAL append**, so it does not
|
||
ride the leader relay and has **redundancy 1 regardless of the replication
|
||
factor** — RF3 with `ack: quorum` does *not* replicate it. That is the design
|
||
(parallel write throughput across shard owners: §11 measured 3,669 signals/s vs
|
||
~90/s replicated), but the endpoint used to answer `201`/`204` with nothing saying
|
||
so. It now requires `x-tidal-ack: local` and returns **400** without it, naming
|
||
the header and the replicating alternative in the body.
|
||
|
||
Write routes (each routes to the owning region; a non-owner gateway forwards):
|
||
|
||
```bash
|
||
ACK='x-tidal-ack: local' # the single-copy opt-in; without it every line below is 400
|
||
curl -X POST "$BASE/sharded/items" -H "$ACK" -d '{ "entity_id": 7, "metadata": { "title": "..." } }' # 201
|
||
curl -X POST "$BASE/sharded/embeddings" -H "$ACK" -d '{ "entity_id": 7, "values": [0.1,0.2,0.3,0.4] }' # 204
|
||
curl -X POST "$BASE/sharded/signals" -H "$ACK" -d '{ "entity_id": 7, "signal": "view", "weight": 1.0 }' # 204
|
||
```
|
||
|
||
For a **replicated** write use `POST /items` / `/embeddings` / `/signals` (leader
|
||
WAL relay, `x-tidal-ack: leader|quorum`). `local` is rejected on those routes.
|
||
|
||
Read routes (scatter-gather across all regions):
|
||
|
||
```bash
|
||
curl "$BASE/sharded/feed?profile=for_you&limit=20&deadline_ms=50"
|
||
curl "$BASE/sharded/search?query=jazz&limit=10&deadline_ms=100"
|
||
```
|
||
|
||
Each sharded read accepts an optional `deadline_ms` — the **total** scatter budget
|
||
(default 50ms, server-clamped to 10s). The per-shard deadline is `deadline_ms − 5ms`
|
||
network overhead. The response includes a `scatter_gather` block:
|
||
|
||
```json
|
||
{
|
||
"items": [ /* merged, deduped, diversity-enforced, re-ranked */ ],
|
||
"total_candidates": 4210,
|
||
"scatter_gather": {
|
||
"degraded": false,
|
||
"shards_queried": 3,
|
||
"elapsed_ms": 12,
|
||
"shard_deadline_ms": 45
|
||
}
|
||
}
|
||
```
|
||
|
||
Degraded semantics: a shard that is partitioned, errors, or misses the deadline is
|
||
reported in `unavailable_shards` (a name list) and flips `degraded: true` — it is
|
||
**never silently dropped**, and the read **still returns 200** with the live
|
||
shards' results. The merge dedups replicated copies of an entity (keeping the
|
||
best-scoring copy), reconciles `total_candidates` so replicated shards are not
|
||
counted multiple times, and re-enforces `max_per_creator` across the merged set.
|
||
|
||
## 8. Write-durability contract: the ack knob and its cost (m11p3)
|
||
|
||
Every replicated write (`/signals`, `/items`, `/embeddings`) runs under one of
|
||
two acknowledgment modes. Pick the deployment default with the topology's
|
||
`replication.ack`; any caller overrides per request with the **`x-tidal-ack`
|
||
header** (`leader` or `quorum`; anything else is a 400).
|
||
|
||
| | `ack=leader` (default) | `ack=quorum` |
|
||
|---|---|---|
|
||
| Success means | Durable on the **leader** (storage + WAL group-commit fsync) | Durable on a **majority of the replica set** (leader + `floor(n/2)` followers, each storage-applied + own-WAL-fsynced) |
|
||
| Survives | Any follower failure; leader restart (WAL replay) | **Any single node's permanent loss, including the leader's** — promote the max-applied survivor and every acked write is there (the 167-kill-point ledger gate proves it) |
|
||
| Does NOT survive | Permanent leader loss before followers caught up (the un-shipped tail dies with it) | Simultaneous majority loss |
|
||
| Latency cost | One group-commit fsync (~ms-scale; the macOS F_FULLFSYNC tail is the local floor) | + one ship RTT + the follower's group-commit fsync, **pipelined**: acks are batch-level, so concurrent writers share the round trip the same way they share fsyncs |
|
||
| Failure mode | 5xx only for local faults | Additionally a **retryable 503** when the quorum budget (`replication.quorum_timeout_ms`, default 2000) expires — the JSON body names the `laggards`, the `commit_index`, and `needed`/`confirmed` counts |
|
||
| Availability | Unaffected by follower outages | **Blocks when a majority is unreachable.** In a 2-region cluster quorum = leader + THE follower: one follower outage stops all quorum writes (by design — that is what the contract says). 3+ regions tolerate `floor((n-1)/2)` follower outages |
|
||
|
||
Mechanics, in one paragraph: there is **one replicated log** (the leader's
|
||
WAL — m11p2); peers only ever receive **fsynced batches** by construction.
|
||
Since m11p3 every follower **pushes its durably-applied frontier** back to
|
||
the leader once per apply round (`ReportApplied` — batch-level, decoupled
|
||
from ship acks, flowing even when the follower converges by catch-up pull or
|
||
the leader is quiet); ship acks additionally carry the same frontier as an
|
||
instant floor hint. The leader folds both into per-peer durable marks; the
|
||
**commit index** is the k-th largest mark (k = `floor(n/2)`), and
|
||
`ack=quorum` responses gate on it passing the write's seqno — awaited
|
||
asynchronously, so quorum waiters never hold threads or starve completions.
|
||
The index is leadership-scoped: promote resets it to the new stream
|
||
baseline, and a demoted leader fails its in-flight quorum waiters (it must
|
||
never claim quorum for a stream it no longer owns).
|
||
|
||
Every cluster write's success response carries **`x-tidal-seq`** — the
|
||
write's seqno in the replicated log (relayed through gateway forwards).
|
||
Persist it if you need an exact durability cursor: `commit_index >= seq` on
|
||
`/cluster/status/local` is "this write is majority-durable", regardless of
|
||
which mode acked it. (The rare dedup-suppressed signal write — an identical
|
||
event within the WAL's ~60s content-hash window is already durably logged,
|
||
so no new record exists to track — carries **`x-tidal-deduplicated: 1`**
|
||
instead, relayed through forwards like the seq header.)
|
||
|
||
**Retry semantics under `ack=quorum` — read this twice.** A quorum-timeout
|
||
503 means *not confirmed in budget*, not *not written*: the write is in the
|
||
leader's log and usually commits moments later. Retries are therefore
|
||
**at-least-once**:
|
||
|
||
- `/items` and `/embeddings` retries are **always safe** — idempotent
|
||
upserts keyed by `entity_id`.
|
||
- `/signals` retries can **double-count** the signal's weight when the
|
||
original did commit (the server stamps each request's timestamp, so the
|
||
WAL's content-hash dedup window cannot identify a client retry). The
|
||
distortion is one extra decaying signal per retried timeout — bounded by
|
||
your retry rate (`tidaldb_cluster_quorum_timeouts_total` is exactly that
|
||
budget). Accounting that cannot tolerate it should route through session
|
||
writes (which carry idempotency keys) or dedup client-side on its own key.
|
||
- The laggard names in the 503 are your runbook pointer: a persistent
|
||
laggard is a down/partitioned region — heal it ([§6](#6-cluster-management-api))
|
||
or accept leader-ack for the duration (`x-tidal-ack: leader`).
|
||
|
||
**Rolling upgrades into m11p3 — upgrade the leader first.** A pre-m11p3
|
||
leader neither serves the `ReportApplied` RPC nor recognizes `x-tidal-ack`:
|
||
it silently applies **leader-ack semantics to a `quorum` request** — a
|
||
durability downgrade the caller cannot see. Upgrade order:
|
||
|
||
1. Promote leadership off the leader node if needed, upgrade it, promote it
|
||
back (or simply upgrade the standing leader per [§9](#9-failover-multi-process)'s
|
||
restart procedure). From this moment `ack=quorum` is honored: the commit
|
||
index rides the m11p2 ship-ack floor hints from not-yet-upgraded
|
||
followers (correct, just laggier).
|
||
2. Upgrade followers one at a time. Each upgraded follower starts pushing
|
||
`ReportApplied` and quorum freshness returns to batch-level. (An upgraded
|
||
follower reporting to a still-old leader is harmless — the report is
|
||
refused and logged, replication and heal are unaffected.)
|
||
|
||
Until step 1 completes, treat the cluster as `ack=leader`-only — do not
|
||
point `ack=quorum` traffic at it expecting majority durability.
|
||
|
||
Other facts unchanged from m11p1/p2: the success code never waits on
|
||
shipping for `ack=leader` (sender threads push the WAL flush feed's tail off
|
||
the request path; `replication.batch_max_events`/`window` tune it); a failed
|
||
fsync errors that write and nothing unfsynced can ship; long-outage data is
|
||
follower-pulled via `StreamSegments` (or, when it has rotated past the leader's
|
||
retained WAL, via the m11p5 `FetchSnapshot` snapshot transfer); a leader crash
|
||
is an **automatic failover** since m11p4 (the survivors elect the
|
||
up-to-date successor — see [§9.1](#91-automatic-failover-m11p4--the-default);
|
||
the vote restriction only elects a node whose log covers every quorum-acked
|
||
write, which is what makes the zero-acked-loss guarantee hold).
|
||
|
||
In short: **`ack=leader` = leader durability. `ack=quorum` = failover-survivable
|
||
durability, priced at one pipelined replication round trip and majority
|
||
availability.**
|
||
|
||
**WAL segment format across upgrades (m11p4).** Segment files carry an
|
||
8-byte version header (`TSEG` + version byte; headerless pre-m11p4 files
|
||
stay readable — no migration). Three behaviors follow:
|
||
|
||
- **Unreadable segments fail the boot, loudly.** A node whose WAL dir holds
|
||
segments written by an incompatible tidalDB version (or stray/foreign
|
||
`.seg` files) refuses to start with `WAL segment format unknown: <path>`
|
||
instead of booting with the data invisibly absent (`segments=0` — the
|
||
2026-06-11 p3 rollout failure mode). Remedy: run a compatible binary, or
|
||
reseed the node (delete its PVC and let it pull from the leader).
|
||
- **Unservable catch-up resolves via snapshot transfer (m11p5).** A leader
|
||
that cannot serve a follower's requested range from its retained WAL answers
|
||
the `StreamSegments` pull with `FAILED_PRECONDITION` — `"segments not
|
||
available from seq N; snapshot required"`, carrying the typed trailer
|
||
`x-tidal-catchup: snapshot-required`. Since m11p5 the follower **latches a
|
||
durable `reseed_required` marker** (only on that typed trailer — an ordinary
|
||
election term-mismatch never latches it fleet-wide) and reseeds itself via the
|
||
`FetchSnapshot` snapshot stream on its next boot (or self-restarts if
|
||
`replication.reseed_self_restart` is set) — no operator verb, no
|
||
`wipe_data_dir`. See [§9.1](#91-automatic-failover-m11p4--the-default). The
|
||
marker is surfaced in `/cluster/status/local` (`reseed_required: true`) and the
|
||
`tidaldb_cluster_reseed_required` gauge.
|
||
- **Failed pulls self-heal on a timer.** A catch-up pull that fails (e.g.
|
||
the leader's gRPC server not yet ready during a rolling restart) retries
|
||
every `replication.catchup_retry_ms` (default 30000) without waiting for
|
||
a write to re-expose the gap — an idle cluster no longer strands lagged
|
||
followers.
|
||
|
||
**Downgrade hazard (m11p4):** a pre-m11p4 binary reading a header-bearing
|
||
segment treats the header as a torn tail and may truncate the final segment.
|
||
Downgrading across the m11p4 boundary requires reseeding the node's WAL.
|
||
|
||
**Membership records + capability gate (m11p5).** A **kind-4 membership
|
||
record** is a new WAL blob kind. A kind-4 record shipped to a **pre-p5
|
||
follower** is an unknown batch kind → `WalError::Corruption` → that follower's
|
||
torn-state receiver halt, **permanent across restarts** (boot self-heal
|
||
re-pulls the same record). Both followers halted = a quorum-write outage. To
|
||
make this structurally impossible, `HeartbeatResponse`/`ReportApplied` carry a
|
||
`capabilities` bit-field (proto3 zero-default = pre-p5 = incapable) and the
|
||
leader **refuses `/cluster/join` and every conf-change until all current voters
|
||
report kind-4 capability** — so the first conf-change cannot fire mid-upgrade.
|
||
**Downgrade rule (kind-3 precedent verbatim): once any kind-4 record is in a
|
||
node's WAL, downgrading it below p5 requires a reseed.** Complete the binary
|
||
upgrade across all voters before adding, removing, or replacing a node.
|
||
|
||
**The same halt was reachable from ordinary malformed input, and that path had
|
||
no gate.** A dimension-mismatched embedding was appended to the WAL *before* it
|
||
was validated, so it shipped and halted both followers — one client request, one
|
||
quorum-write outage (2026-08-31, shard 1). Closed on both sides now: the write
|
||
path validates before the append, and the receiver skips a record that is
|
||
unapplicable on every replica instead of halting on it. See
|
||
[§16.6](#166-a-halted-receiver-a-poison-record-in-the-log).
|
||
|
||
## 9. Failover (multi-process)
|
||
|
||
### 9.1 Automatic failover (m11p4 — the default)
|
||
|
||
**"A machine died" is a non-event.** Every node runs a failure detector
|
||
(leader heartbeats every `election.heartbeat_interval_ms`, default 300) and a
|
||
Raft-style election (pre-vote + vote, randomized
|
||
`election.election_timeout_{min,max}_ms`, default 1500–3000). Kill the
|
||
leader and the survivors elect a successor — typically in **under one
|
||
second** with the defaults, bounded well inside 10s — with **zero operator
|
||
verbs** and zero acknowledged-write loss (the vote restriction only elects a
|
||
node whose log covers every quorum-acked write; the tier-3
|
||
`cluster_election.rs::mp_auto_failover_writes_resume_zero_acked_loss` gate
|
||
proves it across repeated random kill points under `ack=quorum` load).
|
||
|
||
What the operator sees:
|
||
|
||
- `/cluster/status/local` carries `term` (the election term, 0 = the
|
||
pre-election "topology era"), `role` (`leader`/`follower`/`pre-candidate`/
|
||
`candidate`) and `quarantined`.
|
||
- During the brief leaderless window, writes return a retryable 503 naming
|
||
the election (`leader: "none (election in progress)"` plus the responding
|
||
node's `term`); clients retry and land on the new leader.
|
||
- A **restarted ex-leader can never re-claim leadership from its topology
|
||
file**: its durable election state (`data_dir/election_state`) boots it as
|
||
a follower, and every replication RPC is term-fenced — a deposed leader's
|
||
ships, heartbeats and frontier reports are rejected until it rejoins the
|
||
current term (the §1.4-1 split-brain incident is closed by construction;
|
||
proven by `mp_fenced_ex_leader_restart_cannot_write`).
|
||
- A leader that loses contact with a majority **steps down within
|
||
`election.leader_lease_ms`** (default 900) and stops accepting writes:
|
||
`ack=leader` writes during a minority partition are bounded by the lease,
|
||
and `ack=quorum` writes were never at risk.
|
||
|
||
**Divergent suffix / quarantine → automated reseed (m11p5).** A node that
|
||
held leader-acked (never quorum-acked) writes when it died can rejoin into a
|
||
cluster that elected past them. It detects this at term-join and
|
||
**quarantines**: it serves status (`quarantined: true`, metric
|
||
`tidaldb_cluster_divergence_quarantined`) and keeps voting, but refuses the
|
||
data plane. Since m11p5 recovery is **automatic and full-history** — no
|
||
`wipe_data_dir`: the quarantine latches the durable `reseed_required` marker,
|
||
the node reseeds via the `FetchSnapshot` snapshot stream on its next boot (or
|
||
self-restarts if `replication.reseed_self_restart` is set and quorum can be
|
||
sustained without it), rejoins clean, and the divergence gauge clears. This is
|
||
strictly leader-ack-only data, within the documented `ack=leader` crash
|
||
contract (§8). The tier-3 `cluster_reseed.rs` proves the full quarantine →
|
||
marker → restart → reseeded → gauges-cleared loop.
|
||
|
||
**Scope of "automatic recovery".** The reseed path above repairs a **divergent
|
||
suffix** — records this replica holds that the elected majority does not. It does
|
||
**not** repair a record the whole group holds and none of them can apply: the
|
||
snapshot is taken at the leader's applied frontier, which in that case is behind
|
||
the offending record, so the reseeded node replays back into it. That failure is
|
||
[§16.6](#166-a-halted-receiver-a-poison-record-in-the-log), and its remedy is a
|
||
leadership change, not a reseed.
|
||
|
||
**The three-way term-join rule (m11p5).** The divergent-suffix check above is
|
||
one of three outcomes a node reaches when it joins a new term, comparing its own
|
||
stream position against the leader's election-time position (`prev_log`, carried
|
||
on the heartbeat):
|
||
|
||
- `own > prev_log` → **divergent suffix** → quarantine (above);
|
||
- `own < prev_log` → the node is genuinely **missing committed-era history**
|
||
that the new stream's baseline jump would silently skip (the p4 carried
|
||
hazard) → it latches `reseed_required` and reseeds via snapshot (no silent
|
||
gap);
|
||
- `own == prev_log` (or a within-term rejoin) → **clean**, catch-up via the
|
||
stream.
|
||
|
||
A snapshot-installed node always joins clean by construction (its WAL is the
|
||
leader's copy, so its tail term equals the leader's).
|
||
|
||
**Failure mode: write-burst false-partition (the headline m12 fix — rc7).**
|
||
|
||
- **SYMPTOM.** Under a sustained 1536-D `ack=quorum` ingest burst, `ack=quorum`
|
||
writes start **503-storming and do not self-heal**. The breaker gauge
|
||
`tidaldb_cluster_peer_breaker_state` shows **BOTH followers stuck at `1` (Open)**
|
||
while they are actually alive — still heartbeating, still applying. The commit
|
||
index stalls because the leader believes it cannot ship to a majority.
|
||
- **CAUSE.** A follower's CPU-heavy HNSW apply (each apply an `ef_construction=400`
|
||
insert at 1536-D) momentarily **starves its transport runtime**, so a leader ship
|
||
RPC misses the 10s request deadline. Pre-rc7 that tonic `DeadlineExceeded` was
|
||
counted as a transport failure (`record_failure`) and **opened the breaker** even
|
||
though the peer was alive and heartbeating — both followers' breakers latched
|
||
Open, commit stalled, quorum writes 503-stormed with no self-heal.
|
||
- **FIX.** Shipped in **`m12-writeburst-rc7`** (tidal-net `record_timeout`): a ship
|
||
deadline opens the breaker **only when there is no recent proof of life**
|
||
(`last_contact` stale ⇒ a genuine blackhole still opens it; `DeadlineExceeded` /
|
||
`Cancelled` route through `record_timeout`, while a genuine `Unavailable` still
|
||
opens immediately). Heuristic-only change — the commit / election / vote paths are
|
||
untouched.
|
||
- **RESPONSE if seen on an older image.** Roll the StatefulSet to **≥ rc7**
|
||
(`@sha256:171505745b801dcf231b531de6167dbc309a7182957811cbc2228f0a302572b1`).
|
||
Confirm recovery by watching `tidaldb_cluster_peer_breaker_state` clear back to
|
||
`0` once load eases and `ack=quorum` writes stop 503-ing.
|
||
|
||
To run the pre-m11p4 posture (operator-driven failover, no automatic
|
||
elections, no check-quorum step-down), set in the topology:
|
||
|
||
```yaml
|
||
election:
|
||
auto_election: false
|
||
```
|
||
|
||
### 9.2 Manual promote: a FENCED TRANSFER (maintenance / override)
|
||
|
||
`POST /cluster/promote { "region": "eu-west" }` remains the maintenance verb
|
||
— but it is now a **fenced leadership transfer**, not a view flip:
|
||
|
||
1. With a live leader: the leader waits for the target to hold the full
|
||
flushed prefix (the catch-up wait IS the drain), then sanctions an
|
||
immediate election (`TimeoutNow`); the target wins term+1 and the old
|
||
leader steps down on first higher-term contact. Response:
|
||
`{ ok, leader, term, transfer: "elected" }`.
|
||
2. With a dead leader: the target campaigns among the survivors directly —
|
||
the manual override of the automatic path (and the way to *choose* the
|
||
successor).
|
||
3. The election can **refuse a target that lags** (its log loses the
|
||
up-to-date comparison — e.g. the survivor you sampled fell behind between
|
||
your status read and the vote). The verb 503s naming the cause; promote
|
||
the other survivor. **You can no longer accidentally promote a node that
|
||
would discard acked data** — the m11p3 "max-applied survivor" operator
|
||
rule is now enforced by the protocol.
|
||
4. Promoting the node that already leads is a no-op 200.
|
||
|
||
The legacy term-0 fan-out promote survives only for clusters that have never
|
||
elected (mixed-version rollouts mid-upgrade, and the deliberate
|
||
isolated-node override used by the chaos drills); the first joined election
|
||
permanently retires it on each node.
|
||
|
||
> Rolling upgrade m11p3 → m11p4: upgrade ALL binaries before relying on
|
||
> auto-failover (pre-m11p4 peers answer vote RPCs with `Unimplemented`, so
|
||
> no election can reach quorum until a majority is upgraded — the cluster
|
||
> simply keeps its m11p3 behavior until then). The first ELECTED leader
|
||
> journals a kind-3 term-marker WAL record; pre-m11p4 binaries cannot decode
|
||
> it, so do not downgrade a node after the first election without reseeding.
|
||
|
||
## 10. Partition drill (multi-process)
|
||
|
||
The rewritten drill demonstrates **both** partition mechanisms, exactly as
|
||
`cluster_runbook.rs::runbook_s10_partition_drill` scripts them.
|
||
|
||
1. **Baseline.** `GET /cluster/status`; all `lag_events: 0`, `partitioned: false`,
|
||
`reachable: true`.
|
||
2. **Inject.** Use **one** of:
|
||
- **Real network partition** — sever the TCP path peers use to reach the region.
|
||
The chaos suite (`cluster_chaos.rs`) uses a root-free in-harness TCP relay
|
||
proxy (the ROADMAP-sanctioned toxiproxy-style alternative). On a real host you
|
||
can instead use `iptables` (Linux) or `pfctl` (macOS), e.g.
|
||
`iptables -A INPUT -p tcp --dport 9603 -j DROP` to blackhole ap-south's gRPC
|
||
port from a peer. A real cut shows `reachable: false` in the aggregate status.
|
||
- **Simulated ship-skip flag** — `POST /cluster/partition { "region": "ap-south" }`
|
||
(`→ { ok, partitioned: "ap-south" }`). The leader stops shipping to ap-south
|
||
without touching sockets; the aggregate status shows `partitioned: true`.
|
||
3. **Write through it.** Send several `POST /signals`. Each must still `204` (the
|
||
leader-durable contract holds). Watch ap-south's `lag_events` climb while its
|
||
`applied_events` stalls — the leader's ships to it are dropped.
|
||
4. **Read the stale follower.** `GET /feed?region=ap-south` (or read ap-south's
|
||
gateway directly) returns the pre-partition view — eventual, not strong, read
|
||
consistency. The operator console survives a real partition because clients talk
|
||
to each region's gateway directly.
|
||
5. **Scatter-gather degradation.** While partitioned, `GET /sharded/feed` returns
|
||
**200** with `degraded: true` and `ap-south` in `unavailable_shards` — never an
|
||
error, and the live shards' items are still returned.
|
||
6. **Heal.** Clear the cut (heal the proxy / drop the firewall rule, or just call
|
||
heal for the simulated flag), then `POST /cluster/heal { "region": "ap-south" }`.
|
||
**Re-issue heal until `/cluster/status` shows ap-south at `lag_events: 0`** — the
|
||
gRPC circuit breaker opened during the partition (threshold 5, reset 30s), so the
|
||
first heal may ship into an open breaker and no-op. This is genuine production
|
||
behavior, not a flag.
|
||
7. **Verify convergence.** `GET /cluster/status` shows ap-south `lag_events: 0`,
|
||
`partitioned: false`, `reachable: true`; `/sharded/feed` is no longer degraded;
|
||
feed scores on ap-south match the leader to within float tolerance (no loss, no
|
||
duplication).
|
||
|
||
## 11. Shutdown
|
||
|
||
Send `SIGTERM` (or `SIGINT` / Ctrl+C, or stop the container). The server flips
|
||
readiness to **503** (so a load balancer stops routing to it), **drains** in-flight
|
||
requests, then drops the region's `TidalDb` shutdown path: checkpoint in-memory
|
||
signal state → flush storage → write the WAL checkpoint marker + **fsync** → join
|
||
the WAL, sweeper, checkpoint, text-syncer, and replication-receiver threads. The
|
||
drop is idempotent; the process exits 0. You will see
|
||
`region cluster node shutdown: database closed (checkpoint + WAL fsync)`
|
||
(multi-process) / `cluster shutdown: closing all nodes (checkpoint + WAL fsync)`
|
||
(single-process).
|
||
|
||
On **restart with the same `--data-dir`**, the region recovers its pre-shutdown
|
||
state from the WAL (verified by `cluster_runbook.rs::runbook_s11_shutdown_and_wal_recovery`:
|
||
SIGTERM → exit 0 → restart → the same items are served). In multi-process mode, the
|
||
restarted node rejoins the cluster and pulls anything it missed while down via
|
||
its boot-time `StreamSegments` catch-up request — no operator verb needed
|
||
(m11p2; `/cluster/heal` remains the explicit lever for paused peers). This is the per-node step of a
|
||
**rolling upgrade** (promote leadership off the node, SIGTERM, restart on the same
|
||
data dir with the new binary, heal) — see `cluster_lifecycle.rs::mp_rolling_upgrade_no_loss_no_stall`,
|
||
which proves zero acknowledged-write loss across a full rolling upgrade under load.
|
||
|
||
### Node-replace drill (m11p5)
|
||
|
||
Replacing a dead or recycled node is `kubectl delete pod` — no topology edit, no
|
||
operator verb:
|
||
|
||
- **PVC retained** (the data dir survives the pod): the replacement boots on the
|
||
same data dir and converges via its boot-time `StreamSegments` catch-up — the
|
||
ordinary restart path above. If it rotated past the leader's retained WAL while
|
||
down, it reseeds via `FetchSnapshot` automatically (the `reseed_required`
|
||
marker + snapshot path, [§9.1](#91-automatic-failover-m11p4--the-default)).
|
||
- **PVC deleted + pod deleted** (fresh node): the replacement comes up empty and
|
||
**seed-joins** ([§1b](#1b-multi-process-one-process-per-region)) — it `--seed`s
|
||
a survivor, `FetchSnapshot`s the current state, and the leader auto-promotes it
|
||
back to Voter. Under the `k8s/cluster/` StatefulSet this is the default: the
|
||
pod's args carry `--seed` against the headless Service, so a recreated pod
|
||
rejoins with no human in the loop.
|
||
|
||
**Add a node:** scale the StatefulSet up (pod N≥3 seed-joins as a learner and
|
||
auto-promotes). **Remove a node:** `POST /cluster/members/remove` **first**
|
||
([§6](#6-cluster-management-api)), wait for the record to quorum-commit, then
|
||
scale down (lowest ordinal last).
|
||
|
||
## 12. Security (m11p7): mTLS, rotation, identity, audit, rate limits
|
||
|
||
The cluster does not trust the network. Everything here is **opt-in** — absent
|
||
the `grpc_tls` block and the cluster key, the cluster behaves exactly as pre-m11p7
|
||
(plaintext, hint-only marker, no audit/limit). A reference (k8s) deployment turns
|
||
it all on.
|
||
|
||
### 12.1 mTLS (gRPC replication) — the default posture
|
||
|
||
- Configure the `grpc_tls` block per region (`ca_cert`, `server_cert`,
|
||
`server_key`, `client_cert`, `client_key`). The gRPC server then REQUIRES a
|
||
client cert chained to the cluster CA (mutual TLS): a foreign pod with no cert,
|
||
a cert from another CA, or a plaintext probe fails the TLS handshake and never
|
||
reaches an RPC.
|
||
- **No `grpc_tls` ⇒ plaintext, with a loud startup WARN** on both the server and
|
||
the client. Acceptable only on a trusted single-host / loopback topology. To
|
||
serve plaintext intentionally there is nothing else to set — the WARN is the
|
||
signal that you are on the insecure path.
|
||
- The inter-node HTTP plane (forwards, broadcasts, scatter, status, seed-join)
|
||
is served over TLS with the SAME cert and dials `https://` with the cluster CA
|
||
whenever `grpc_tls` is set, so enabling it gives **zero plaintext inter-node
|
||
links** on both planes at once.
|
||
|
||
### 12.2 Cert + bearer rotation WITHOUT restart
|
||
|
||
- A background poller (`TIDAL_ROTATION_POLL_MS`, default 30000) content-hashes the
|
||
cert files and the credential files; on a change it **atomically swaps the
|
||
served cert** (in-flight TLS sessions keep their negotiated keys — zero dropped
|
||
requests) and rebuilds the outbound peer channels.
|
||
- **Procedure:** issue a new cert under the same CA (cert-manager renewal, or
|
||
re-run `scripts/gen-cluster-certs.sh` and re-apply the Secret) — the files
|
||
change in place, the poller swaps within one interval, no pod restart. Verify
|
||
with `tidaldb_cluster_*` logs (`TLS material rotated…`) or the `tidal_audit`
|
||
/tracing stream.
|
||
- The bearer (`TIDAL_API_KEY_FILE`) and the cluster key (`TIDAL_CLUSTER_KEY_FILE`)
|
||
rotate the same way. Use FILE mounts (not inline env) so a Secret rotation is
|
||
picked up live. During a CA roll, keep both old and new CAs trusted for one
|
||
cycle (CA-overlap) so in-flight connections complete.
|
||
|
||
### 12.3 Per-node identity + the marker
|
||
|
||
- Set a shared **cluster key** (`TIDAL_CLUSTER_KEY` / `TIDAL_CLUSTER_KEY_FILE`,
|
||
any random string — BLAKE3-derived to the MAC key). Each node then mints a
|
||
signed `x-tidal-node-token` on every forward/broadcast; the receiver verifies
|
||
it. This gives inter-node calls a verifiable node identity and is the
|
||
defense-in-depth layer beyond the shared bearer.
|
||
- With a cluster key configured, the `x-tidal-internal` marker is honored ONLY
|
||
from a verified sibling: a request that sets the marker without a valid node
|
||
token is rejected **403** (the marker is a routing hint, never an auth bypass).
|
||
Never hand the cluster key to external clients.
|
||
|
||
### 12.4 Admin audit log
|
||
|
||
- promote / partition / heal / join / member-remove / reseed each emit one
|
||
structured record: `{principal, verb, target, term, outcome}`. The principal is
|
||
the verified node (`node:<id>`) for inter-node calls or `external` for an
|
||
operator with the bearer.
|
||
- Sinks: a `tidal_audit` **tracing target** (always — capture it in your log
|
||
pipeline), plus an **append-only JSONL file** when `TIDAL_AUDIT_LOG=<path>` is
|
||
set. Recorded on the operator-originated leg only (no double-record on a
|
||
forwarded re-apply).
|
||
- **At-rest encryption** of the JSONL file is delegated to the volume — mount
|
||
`TIDAL_AUDIT_LOG` on an encrypted PV (or a `gVisor`/LUKS-backed volume); the
|
||
server does not encrypt it in-engine.
|
||
|
||
### 12.5 Per-principal rate limits
|
||
|
||
- `TIDAL_RATE_LIMIT_RPS` (+ optional `TIDAL_RATE_LIMIT_BURST`, default 2×) caps
|
||
per-principal request rate; a deny is **429 + `Retry-After`**. Off by default.
|
||
- Verified sibling nodes are EXEMPT — replication/forward traffic is never
|
||
throttled by the external-client budget. Today external callers share ONE
|
||
bucket: `TIDAL_RATE_LIMIT_RPS` is an AGGREGATE cap across all external clients,
|
||
not a per-client budget — set it to your total external ceiling, not a
|
||
per-client one (one noisy client can consume it). A future multi-key registry
|
||
adds per-key buckets.
|
||
|
||
### 12.6 Foreign-pod / negative behavior (what an attacker on the network sees)
|
||
|
||
| Attempt | Result |
|
||
|---------|--------|
|
||
| Ship a gRPC segment without a cluster client cert | TLS handshake fails — no RPC dispatched |
|
||
| Call an internal HTTP route without trusting the cluster CA | TLS handshake fails — no route reached |
|
||
| Set `x-tidal-internal` without a valid node token (key configured) | 403 — marker honored only from a verified sibling |
|
||
| Call a protected route without the bearer | 401 (unchanged) |
|
||
|
||
## 13. Coordinated backup / restore + point-in-time recovery (m11p8)
|
||
|
||
> **See also the dedicated DR runbook:**
|
||
> [docs/runbooks/disaster-recovery.md](disaster-recovery.md) for full
|
||
> disaster-recovery procedures (region/cluster loss, restore drills, RPO/RTO).
|
||
|
||
The building blocks: the engine's crash-consistent `create_backup`, the WAL
|
||
**archive** (`wal.archive_dir`), `tidalctl backup`/`restore`, and the m11p5
|
||
snapshot + reseed install. Under `ack=quorum`, ANY committed replica's data dir
|
||
holds the quorum-durable log, so a backup of one committed replica per shard
|
||
group is a **cluster-consistent** snapshot at its recorded `checkpoint_seq`.
|
||
|
||
### 13.1 Enable the WAL archive (point-in-time recovery)
|
||
|
||
Set `wal.archive_dir` in the topology (or `TidalDb::builder().wal_archive_dir(path)`
|
||
for the embedded engine). Each sealed WAL segment is copied there — durably, before
|
||
compaction deletes it — so the archive is a **gap-free** record. Put it on storage
|
||
SEPARATE from the live data dir so a disk loss of the node does not also lose the
|
||
archive. Segment filenames encode `shard + first_seq`, so co-located groups share
|
||
one archive dir without collision.
|
||
|
||
```yaml
|
||
wal:
|
||
archive_dir: "/archive/tidaldb" # durable, off-node storage
|
||
```
|
||
|
||
### 13.2 Coordinated backup drill
|
||
|
||
1. Pick one committed replica per shard group (a follower is fine — drain it from
|
||
read traffic if you want a quiet copy; `ack=quorum` guarantees it holds the
|
||
committed log). Stop it (or snapshot its volume).
|
||
2. `tidalctl backup --path /data/<node> --out /backups/<cluster>-<ts>/shard-<id>`
|
||
— writes a recursive copy + `BACKUP_MANIFEST.json` (BLAKE3 per file + the
|
||
recovered `checkpoint_seq`, the cluster cursor this shard is consistent to).
|
||
3. Repeat per shard group. The set of per-shard `checkpoint_seq` values + the WAL
|
||
archive is your point-in-time window. Restart the replica; self-heal/catch-up
|
||
reconverges it.
|
||
|
||
### 13.3 Restore drill (timed)
|
||
|
||
1. `tidalctl restore --from /backups/<cluster>-<ts>/shard-<id> --path /data/<new-node>`
|
||
— verifies EVERY file's BLAKE3 against the manifest BEFORE writing, and refuses
|
||
a non-empty target (it never overwrites a live data dir).
|
||
2. Point a stopped node at the restored dir and boot it. Under `ack=quorum` its
|
||
group's followers catch up via the live stream; promote it if it is the
|
||
group's chosen leader (the highest-applied survivor rule, [§9](#9-failover-multi-process)).
|
||
|
||
**Recovery granularity today is full-snapshot-to-frontier, NOT arbitrary
|
||
point-in-time.** A restore recovers a node to the backup's `checkpoint_seq`, then
|
||
the live stream reconverges it to the cluster's CURRENT frontier. The WAL archive
|
||
(`wal.archive_dir`) is the durable, gap-free **primitive** that *backs* a future
|
||
point-in-time replay — but it is write-only today: no tool replays archived
|
||
segments up to a chosen target seq. `tidalctl recover` is verify-only
|
||
(`--verify-only`; an in-place replay mode is reserved for a future release), and
|
||
`tidalctl restore` copies a full snapshot with no seq bound. So the inputs you
|
||
retain for a future PITR are the per-shard `checkpoint_seq` set + the archive;
|
||
treat them as the **window**, not a one-command restore-to-instant. Do not plan an
|
||
incident around seq-bounded replay until a `tidalctl replay --until <seq>` verb
|
||
ships.
|
||
|
||
Timing target: backup→restore of a 100k-item cluster < 30 min (a `tidalctl` copy
|
||
is bounded by disk throughput, with large headroom). The Ref-A timed figure is a
|
||
k3s line item (the standing M11 access caveat).
|
||
|
||
## 14. Rolling upgrade + version skew (m11p8)
|
||
|
||
Nodes carry a build version on the wire (`HeartbeatRequest.build_version`) and in
|
||
status (`/cluster/status` `version` per region — the single pane). Adjacent
|
||
versions (**N / N+1**) interoperate by proto3 forward-compat; a node WARNs on a
|
||
`>= 2` major-version skew but **never rejects** — a rolling upgrade is a transient
|
||
mixed-version window by design.
|
||
|
||
**Procedure (one node at a time):**
|
||
|
||
1. `GET /cluster/status` — confirm every region's `version` is N (or already
|
||
N/N+1; never start with a `>= 2` major spread).
|
||
2. Graceful SIGTERM one follower → it drains (readiness 503 → checkpoint).
|
||
3. Restart it on N+1 (same ports, same data dir). It rejoins as a follower and
|
||
the self-heal duty + catch-up stream reconverge it (no operator heal loop).
|
||
4. Repeat for each follower. Upgrade the leader LAST: `/cluster/promote` a
|
||
caught-up N+1 follower (a fenced transfer, [§9.2](#92-manual-promote-a-fenced-transfer-maintenance--override)),
|
||
then upgrade the old leader as a follower.
|
||
5. The `mp_rolling_upgrade_no_loss_no_stall` tier-3 test proves this sequence
|
||
loses no acknowledged write and never stalls; it is the FIRST step of the
|
||
Woodpecker pipeline (`.woodpecker.yaml`) — a failure blocks the image build.
|
||
|
||
> Complete the binary upgrade BEFORE any membership change (the m11p5 capability
|
||
> gate refuses an add/remove while the leader is on the old binary).
|
||
|
||
## Performance
|
||
|
||
> **Read this table by era.** The `/signals`-throughput row below is the **m11p1
|
||
> signal-write benchmark** (3-byte signal writes, small payloads, measured over
|
||
> real localhost processes) — it is **NOT** the m12 1536-D production shape and
|
||
> must not be cited as the live cluster's read/write ceiling. The m12 reality on
|
||
> the 3-node k3s fleet at the 1536-D production corpus (see
|
||
> [`docs/profiling/m12-cluster-deploy-findings.md`](../profiling/m12-cluster-deploy-findings.md)):
|
||
> * **Reads:** p99 **7.97–11.47 ms** at 100–500 rps; G1 (p99 ≤ 10 ms) **MET at
|
||
> 100k** with **recall@10 0.9989** (clean index). Read **ceiling ~1000 rps
|
||
> clean** (~1500 rps saturated), **CPU-bound** — beyond it a node sheds/errors.
|
||
> * **Writes:** the peach mix is write-heavy and the **write knee is ~250 rps** on
|
||
> this fleet (1536-D `ack=quorum` ingest; each apply is an HNSW insert). This is
|
||
> why the soak ([§15](#15-continuous-correctness-chaos-suites--soak-m11p9)) was
|
||
> re-scoped to **200 rps** on 2026-06-19.
|
||
|
||
The legacy m11p1 signal-write benchmark (measured over real localhost processes):
|
||
|
||
| Operation | SLA | Measured (p99 / typical) — **m11p1 signal-write benchmark, pre-m12, NOT the 1536-D shape** |
|
||
|-----------|-----|--------------------------|
|
||
| Replicated `/signals` throughput (m11p1, 3 nodes, release build) | ≥ 2,000/s | **4,534 signal-writes/s** within SLO on the ramp (knee ~5.5k/s); **2,739/s sustained 10 min** (1.65M writes, 0.35% errors); was ~90/s pre-m11p1. **Superseded for production sizing by the m12 1536-D figures above.** |
|
||
| Replication lag under that load (m11p1) | < 2s | ≤ 103 events (~40ms) across the 10-min sustain; ≤ 377 events on the 5k/s ramp (follower group-commit coalescing) |
|
||
| Cross-region replication (write → follower applied) | < 2s | ~110–133ms p99 (m8p10) |
|
||
| Failover (`/cluster/promote` → first successful write) | < 10s | ~31–34ms |
|
||
| CRDT reconcile (merge+apply, each side) | < 100ms | 0–1ms |
|
||
|
||
Write-latency note: signal p50 ≈ 17–25ms with a p99 tail of 160–220ms on
|
||
macOS, where `F_FULLFSYNC` averages ~7.4ms with a 10–50ms tail
|
||
(`tidaldb_cluster_wal_fsync_us` — 0% complete under 1ms). On Linux
|
||
`fdatasync` volumes the same pipeline's fsync floor is far lower; validate
|
||
the p99 gate on the reference environment, and tune `wal.batch_timeout_ms`
|
||
against the measured fsync histogram.
|
||
|
||
## 15. Continuous correctness: chaos suites + soak (m11p9)
|
||
|
||
Correctness is a pipeline, not a one-time gate. The tier-3 suites run REAL OS
|
||
processes; the new fault classes inject REAL faults.
|
||
|
||
**Run the chaos suites locally** (serial — they bind fixed ports and spawn
|
||
processes, so suites must not overlap):
|
||
|
||
```
|
||
cargo test -p tidal-server --features "cluster-e2e fault-injection" \
|
||
--test cluster_faults -- --test-threads 1 --nocapture
|
||
```
|
||
|
||
`cluster_faults` covers the fault classes the partition/crash/skew suites lacked:
|
||
|
||
| Test | Fault | Asserts |
|
||
|------|-------|---------|
|
||
| `mp_disk_full_follower_degrades_no_acked_loss` | follower WAL `ENOSPC` | receiver halts (degraded, alive), healthy majority keeps acking quorum, zero acked loss, restart recovers to parity |
|
||
| `mp_slow_fsync_follower_lags_but_quorum_holds` | one follower's fsync slowed | fast follower supplies quorum, slow node lags then converges, no loss |
|
||
| `mp_slow_fsync_both_followers_force_honest_quorum_timeout` | both followers slowed below the budget | `ack=quorum` → retryable 503 naming laggards; `ack=leader` → 204; recover |
|
||
| `mp_asymmetric_partition_no_split_brain_no_loss` | inbound to one node severed (outbound up) | pre-vote + check-quorum hold the leader; single-leader-per-term; no loss |
|
||
|
||
**Fault knobs** (behind the `fault-injection` feature — compiled OUT of the
|
||
production image, inert until armed): `TIDAL_FAULT_FSYNC_DELAY_MS=<ms>` slows
|
||
every durable WAL fsync; `TIDAL_FAULT_DISK_FULL_AFTER_BYTES=<n>` fails segment
|
||
writes with `ENOSPC` after `n` cumulative bytes this process lifetime (arm it on
|
||
a node at restart to fail after `n` bytes of post-restart writes). NEVER set
|
||
these on a production node.
|
||
|
||
**Fleet soak with regression gates** (`tidal-stress`):
|
||
|
||
```
|
||
tidal-stress --target https://<cluster-node>:9500 \
|
||
--ca-cert /etc/tidaldb/tls/ca.crt --skip-seed \
|
||
--corpus 100000 --embedding-dim 1536 \
|
||
--ramp "200:3600" --mix peach --json-summary soak.json \
|
||
--max-error-pct 1 --max-p99-ms 150 --fail-on-knee
|
||
```
|
||
|
||
> **The Ref-A gate is 200 rps (re-scoped 2026-06-19).** The peach mix is
|
||
> write-heavy and the measured write knee on this full-placement three-node
|
||
> fleet is about 250 rps. The one-hour gate therefore uses the measured 200-rps
|
||
> stage, not the retired pre-m12 `3900:3600` figure. The `:9500` plane is TLS.
|
||
|
||
`--fail-on-knee`, `--max-p99-ms`, and `--max-error-pct` make the load process
|
||
exit non-zero on regression. The in-cluster `tidal-soak-nightly` CronJob adds
|
||
the recovery half of the gate: one ledger row per UTC date, one-minute pod UID
|
||
and restart-count evidence immediately before and after the exact load window,
|
||
and an atomic `streak.tsv`. A date advances the streak only when the load passes,
|
||
all three pods have fresh boundary samples, and no UID or counter changes.
|
||
Duplicate dates, gaps, stale evidence, pod replacement, and load failure cannot
|
||
inflate the 30-night streak.
|
||
|
||
**Nightly CI** (`.woodpecker.yaml`, cron `nightly` — Woodpecker, never GitHub
|
||
Actions) remains the bounded code-regression signal: tier-3 chaos with elevated
|
||
kill-points, then a local gated soak. It is not the Ref-A calendar gate unless
|
||
`TIDAL_SOAK_TARGET` and its duration/rate are explicitly pointed at that fleet.
|
||
The guarantee→test map is
|
||
[docs/planning/milestone-11/guarantee-traceability.md](../planning/milestone-11/guarantee-traceability.md).
|
||
|
||
## 16. Vector index divergence
|
||
|
||
Target of the `runbook_url` on `TidalDBClusterVectorIndexDiverged`,
|
||
`TidalDBClusterBlobApplyFailing`, and `TidalDBClusterPeerShipFailing`.
|
||
|
||
**Symptom.** Replicas of one shard group answer the same vector query differently: an
|
||
entity is the top hit on one replica and absent on the others. Health surfaces stay
|
||
green throughout — `lag_events` counts WAL apply, not blob apply, so a blob that never
|
||
lands moves no lag number.
|
||
|
||
### 16.0 First: rule out the two benign causes
|
||
|
||
Both look exactly like a replication defect. On 2026-08-30 the first one consumed several
|
||
rounds of investigation and produced a retracted durability incident report.
|
||
|
||
1. **The write used `/sharded/*`.** That surface hash-partitions and applies to the
|
||
owning region's **local store with no WAL append**, so it **does not replicate — by
|
||
design** (`tidal-server/src/cluster/node.rs:8828-8829`;
|
||
`sharded_write_embedding` → `ShardReplica::apply_embedding_local`). An entity written
|
||
that way living on exactly one node is the contract, not a fault. Only the
|
||
**non-sharded** surface (`/items`, `/embeddings`, `/signals`) rides the leader WAL
|
||
relay and replicates.
|
||
*Check:* re-probe with `/embeddings` and see whether parity holds. It settles in
|
||
seconds what measuring the sharded path more precisely never will.
|
||
2. **A rolling deploy is in flight.** A restarted pod rebuilds its index from store and
|
||
legitimately reads a different count until it finishes. This is why the alert carries
|
||
`for: 15m`.
|
||
|
||
Both are covered by tests: `mp_embedding_is_searchable_on_every_replica_without_restart`
|
||
and `mp_sharded_surface_writes_are_local_to_the_owner`
|
||
(`tidal-server/tests/cluster_sharding.rs`).
|
||
|
||
### DO NOT restart a pod first
|
||
|
||
`rebuild_from_store` runs at open and re-derives the whole index from the durable
|
||
store, which **converges the counts and destroys the evidence**. Capture every number
|
||
below *before* touching a pod. The restart is a diagnostic step with a specific
|
||
meaning (§16.3), not a remedy.
|
||
|
||
### 16.1 Read the counts per shard group
|
||
|
||
`tidaldb_usearch_vector_count` is per shard **group**, not per node: the node's
|
||
metrics-owner group renders unlabeled and each co-located group renders `shard="N"`.
|
||
The owner is deterministically the same group on every node, so compare within a
|
||
label set, never across:
|
||
|
||
```bash
|
||
kubectl -n observability port-forward svc/vmsingle 18428:8428 &
|
||
curl -s --get --data-urlencode \
|
||
'query=tidaldb_usearch_vector_count{namespace="tidaldb-cluster"}' \
|
||
localhost:18428/prometheus/api/v1/query | jq -r \
|
||
'.data.result[] | "\(.metric.pod) shard=\(.metric.shard // "0(owner)") \(.value[1])"'
|
||
```
|
||
|
||
Replicas of one group MUST agree. A spread that persists past a rolling deploy is real.
|
||
|
||
### 16.2 Localise it with the blob ledger
|
||
|
||
```bash
|
||
# on the node that ACCEPTED the write:
|
||
# tidaldb_cluster_blobs_originated_total{kind="embedding"}
|
||
# on every peer:
|
||
# tidaldb_cluster_blobs_applied_total{kind="embedding"}
|
||
# tidaldb_cluster_blobs_apply_failed_total{kind="embedding"}
|
||
```
|
||
|
||
With RF3 a healthy cluster shows `applied ≈ (RF-1) × originated` in aggregate — one
|
||
apply per follower. Read it per (pod, kind), not as a fleet-wide subtraction.
|
||
|
||
| reading | meaning | next step |
|
||
| --- | --- | --- |
|
||
| `apply_failed` > 0 **and** that pod's `applied` is flat / `lag` growing | the receiver **halted** that stream (it stopped applying entirely) | §16.6 — read that pod's logs for the apply error; the stream is stalled, not silently lossy |
|
||
| `apply_failed` > 0 but `applied` keeps advancing and `lag` returns to 0 | the receiver **skipped** a record no replica could ever apply and continued (logged at ERROR with the seqno range). Expected for a malformed record already in the log; the group stays available | find the ERROR line, fix the producer; nothing to recover on the replica |
|
||
| peer's `applied` never advances while the writer's `originated` does | the record is not arriving — enqueue or ship | §16.4 |
|
||
| `applied` advances but that group's `usearch_vector_count` lags | it arrived and was not indexed | index-gap; the store is intact, a restart recovers it |
|
||
|
||
### 16.3 The decisive test (only after 16.1 and 16.2 are captured)
|
||
|
||
Confirm `kubectl -n tidaldb-cluster get pdb tidaldb -o jsonpath='{.status.disruptionsAllowed}'`
|
||
is `1`, run a quorum-write probe (`POST /items` with `x-tidal-ack: quorum` → expect
|
||
201), then restart **one follower** that lacks the entity — never the leader.
|
||
|
||
> **The probe MUST go through the replicating surface, and `/sharded/*` can never
|
||
> serve as one.** This step used to say `POST /sharded/items` → expect 201. A
|
||
> `/sharded/*` write is applied to the owning region's local store **with no WAL
|
||
> append**, so it never enters the replication stream and no quorum is ever
|
||
> consulted — its `201` means "one region accepted a single-copy write", which is
|
||
> true whether quorum is intact, degraded, or gone. A green probe therefore
|
||
> carried **no** information about the property it was run to check.
|
||
>
|
||
> Do not reintroduce it on the reasoning that it answers faster: speed is exactly
|
||
> what it buys by skipping the WAL append, and skipping the WAL append is precisely
|
||
> what makes it blind. Only a write that appends to the leader WAL and waits for a
|
||
> quorum ack can verify quorum, which is `POST /items|/embeddings|/signals` with
|
||
> `x-tidal-ack: quorum`. (Since the opt-in landed, `/sharded/*` also returns 400
|
||
> without `x-tidal-ack: local` — see §7.)
|
||
>
|
||
> This probe was used as the between-step safety check of a staged rolling deploy
|
||
> on 2026-08-30; that run's quorum verification must be treated as never having
|
||
> happened. See `k3s-fleet/cluster-state.yaml`.
|
||
|
||
- entity becomes retrievable after the rebuild ⇒ its **durable store had it**, only the
|
||
live index was missing it. Data was safe.
|
||
- entity still absent ⇒ its **store never had it**. Before calling this a durability
|
||
event, re-check §16.0: for a `/sharded/*` write this is the intended outcome and the
|
||
store was never supposed to hold it. If the write went through the **non-sharded**
|
||
surface and the entity is still absent after a rebuild, that *is* a durability event —
|
||
the entity exists on fewer replicas than the replication factor claims. Record the
|
||
affected count and escalate.
|
||
|
||
Re-probe the quorum write afterwards and confirm 201 again.
|
||
|
||
### 16.4 Check the ship path
|
||
|
||
```bash
|
||
kubectl -n tidaldb-cluster logs tidaldb-<leader> -c tidaldb | grep 'ship sender'
|
||
```
|
||
|
||
`batch ship failing; retrying every 100ms … consecutive_failures=N` means the leader
|
||
cannot reach that peer. **`tidaldb_cluster_peer_breaker_state` is not a reliable
|
||
signal here** — on 2026-08-30 it read `0` (closed/healthy) through 2697 consecutive
|
||
failures. Trust `tidaldb_cluster_peer_ship_failures_total`'s rate, which
|
||
`TidalDBClusterPeerShipFailing` now alerts on.
|
||
|
||
Also note `GET /cluster/status`'s `regions[]` block can report peers
|
||
`partitioned: true, reachable: false` with full `lag_events` while the same response's
|
||
`shards[]` and `peer_acked_seqno` show complete convergence. **The `shards[]` block is
|
||
authoritative**; `regions[]` is a stale legacy view.
|
||
|
||
### 16.5 Expected convergence window
|
||
|
||
A write is not instantly visible on every replica, and that is not this bug. Ship and
|
||
apply normally complete in well under a second on an in-cluster (low-RTT) deployment.
|
||
"Still converging" vs "diverged" is decided by whether `applied` on the lagging peer is
|
||
**advancing**: if it is, wait; if it is flat while the writer's `originated` climbs, it
|
||
is the failure in §16.2. The alert's `for: 15m` exists so a rolling deploy's legitimate
|
||
rebuild window never pages.
|
||
|
||
### 16.6 A halted receiver: a poison record in the log
|
||
|
||
**Symptom.** One shard group's followers stop advancing while its leader keeps
|
||
writing: leader at seqno `N`, followers pinned at `N-4` / `N-5`, `lag_events`
|
||
growing, `tidaldb_cluster_blobs_apply_failed_total{kind="..."}` non-zero, and
|
||
**writes to that group return 503** — the followers cannot ack, so quorum is
|
||
gone. The pod log names the record:
|
||
|
||
```text
|
||
replicated blob batch apply failed (1 records):
|
||
[op=write_item_embedding] dimension mismatch: expected 1536, got 128
|
||
replication apply failed; receiver halting (health degraded)
|
||
```
|
||
|
||
This is a **quorum-write outage caused by a single record in the WAL**, not by
|
||
any node being unhealthy. It happened on 2026-08-31 on shard 1: one probe posted
|
||
a 128-dimension vector to a slot the live ConfigMap declares at 1536.
|
||
|
||
#### Restarting does NOT fix it
|
||
|
||
Boot self-heal re-pulls the same record from the leader and the receiver halts
|
||
again at the same seqno. The halt is **permanent across restarts** — the same
|
||
shape [§8](#8-write-durability-contract-the-ack-knob-and-its-cost-m11p3)'s
|
||
kind-4/pre-p5 capability note describes, reached there from a version skew and
|
||
here from ordinary malformed input.
|
||
|
||
#### `POST /cluster/reseed` does NOT fix it either — and can make it worse
|
||
|
||
This is the opposite of what the reseed narrative in
|
||
[§9.1](#91-automatic-failover-m11p4--the-default) and the
|
||
[`/cluster/reseed` verb](#membership-verbs-m11p5--online-add--remove--inspect--reseed)
|
||
imply, and it is worth understanding before reaching for it:
|
||
|
||
> **A snapshot is captured at the leader's APPLIED frontier, and on a halted
|
||
> group that frontier is itself BEHIND the poison record.** So the reseeded
|
||
> follower installs a snapshot from *before* the bad record, then replays forward
|
||
> **into it** and halts again. On 2026-08-31 the reseed left the node reporting 3
|
||
> apply failures instead of 2 — marginally worse than doing nothing.
|
||
|
||
Reseed escapes a **divergent suffix** (this replica holds records the elected
|
||
majority does not). It cannot escape a record the whole group holds and none of
|
||
them can apply. Different failure, different remedy.
|
||
|
||
#### What DOES fix it: force a leadership change on the affected group
|
||
|
||
Send the promote **to the node you want to lead, naming itself**, not to the
|
||
current leader:
|
||
|
||
```bash
|
||
# $NODE = a surviving FOLLOWER of the stalled group, addressed directly. It is by
|
||
# definition not caught up — that is what makes the fenced path below unusable.
|
||
curl -sk -X POST "$NODE/cluster/promote?shard=<id>" \
|
||
-H 'content-type: application/json' -H "authorization: Bearer $ADMIN_KEY" \
|
||
-d '{"region":"<that same node>"}'
|
||
```
|
||
|
||
On the target-side path the node first asks the current leader for a sanctioned
|
||
transfer; that request **fails** here (see below), so the node campaigns directly
|
||
and the group elects at term+1. This is the same path the dead-leader failover
|
||
drill uses, and a stalled group is the same situation from the survivors' side.
|
||
|
||
> **Do NOT reach for `POST /cluster/shards/<id>/transfer`, and do not send the
|
||
> promote to the LEADER.** Both run the *fenced* transfer, whose drain waits (5s)
|
||
> for the target to hold the leader's full flushed prefix before `TimeoutNow`. On
|
||
> a stalled group the target can never get there — that is the entire problem —
|
||
> so it returns:
|
||
>
|
||
> ```text
|
||
> transfer target '<node>' lags the flushed frontier (13540694 < 13540698);
|
||
> heal it first, then retry the promote
|
||
> ```
|
||
>
|
||
> The message is correct and its advice is a dead end here: `/cluster/heal`
|
||
> re-drives the same stream into the same halt. Do not chase it.
|
||
|
||
If no verb can be reached, deleting the group's LEADER pod forces the election —
|
||
this is what recovered the live cluster on 2026-08-31. Check
|
||
`disruptionsAllowed` first (§16.3). This is the one case where restarting a pod
|
||
is the remedy rather than evidence destruction (§16's "DO NOT restart a pod
|
||
first"), because the group is already stalled and its counters are already
|
||
captured:
|
||
|
||
```bash
|
||
kubectl -n tidaldb-cluster delete pod tidaldb-<leader-of-that-group>
|
||
```
|
||
|
||
Either way the receivers restart under the new term and re-derive their position,
|
||
and the group converges. On 2026-08-31 `tidaldb-1` took shard 1 at term 113, all
|
||
three nodes converged to `13540698` with `lag=0`, and writes returned to `201`.
|
||
|
||
Verify, per node:
|
||
|
||
```bash
|
||
curl -sk "$NODE/cluster/status/local" | jq '{applied_events, lag_events, last_seq}'
|
||
kubectl -n tidaldb-cluster logs <pod> --since=5m | grep -c 'receiver halting' # expect 0
|
||
```
|
||
|
||
Leave the poison records where they are. They sit below the converged frontier
|
||
and are harmless once every replica has moved past them.
|
||
|
||
#### Why a new malformed write can no longer do this
|
||
|
||
The origin now validates an embedding **before** `wal_blob_first` appends
|
||
(`tidal/src/db/items.rs`), so a dimension-mismatched vector is rejected **400**
|
||
and never becomes durable — it cannot ship, and no follower can be poisoned by
|
||
it. And a record that *is* in the log and is unapplicable on **every** replica
|
||
(its verdict reads only the record's bytes and the shared schema) is now
|
||
**skipped** by the receiver rather than halted on: counted on
|
||
`tidaldb_cluster_blobs_apply_failed_total`, logged at ERROR with the seqno range,
|
||
stream continues. Failures that might apply later — a node-local disk/lock fault,
|
||
an unknown batch kind a newer binary understands, a malformed term/membership
|
||
record whose loss would diverge the roster — still halt, deliberately: skipping
|
||
those would trade an availability bug for silent data loss.
|
||
|
||
Regression gate: `tidal-server/tests/cluster_poison_embedding.rs`
|
||
(tier-3, 3 real processes) and the `db::items` unit tests
|
||
`a_dimension_mismatched_embedding_is_rejected_before_the_wal_append` and
|
||
`a_schema_unapplicable_replicated_embedding_is_skipped_not_halted`.
|
||
|
||
## Cross-references
|
||
|
||
- **Kubernetes deployment** — [docs/runbooks/kubernetes.md](kubernetes.md)
|
||
(the hardened standalone single-replica set in [`k8s/`](../../k8s/), and the
|
||
multi-region cluster reference in [`k8s/cluster/`](../../k8s/cluster/) — one
|
||
StatefulSet + headless Service peer discovery + PDB, with `--seed`-based scale
|
||
and `kubectl delete pod` node-replace, shipped in m11p5).
|
||
- **Disaster recovery** — [docs/runbooks/disaster-recovery.md](disaster-recovery.md)
|
||
(backup/restore/PITR procedures, region/cluster loss, RPO/RTO; the operational
|
||
companion to [§13](#13-coordinated-backup--restore--point-in-time-recovery-m11p8)).
|
||
- **Server deployment guide** — [docs/guides/server-deployment.md](../guides/server-deployment.md)
|
||
(standalone and cluster launch, config, env, health probes).
|
||
- **Monitoring & alerts** — [docs/ops/monitoring.md](../ops/monitoring.md)
|
||
(Prometheus scrape config, replication-lag and WAL metrics, recommended alerts;
|
||
per-region replication lag is observable via `/cluster/status` `lag_events`;
|
||
remember the `/metrics` endpoint is unauthenticated — bind it internally).
|
||
- **Roadmap / cluster status & known gaps** —
|
||
[docs/planning/ROADMAP.md](../planning/ROADMAP.md) and
|
||
[docs/roadmap-to-cluster.md](../roadmap-to-cluster.md) for the M11 cluster
|
||
status — quorum-ack writes (m11p3), automatic failover (m11p4),
|
||
membership/discovery/elasticity (m11p5), security hardening (m11p7),
|
||
observability + operations (m11p8), and continuous correctness (m11p9) all
|
||
shipped; sharding × replication (p6) data plane in progress.
|
||
- **API & schema reference** — [API.md](../../API.md),
|
||
[QUICKSTART.md](../../QUICKSTART.md), and the live `/openapi.json` document.
|
||
- **Scope & vision** — [VISION.md](../../VISION.md).
|