1265140e28
- fault-injection cargo feature (compiled OUT of prod): slow-fsync + disk-full WAL hooks in tidal/src/fault.rs, inert until armed, tier-3 builds with feature - first-class invariant checkers (tests/support/invariants.rs): AckLedger no-acked-loss (now consumed by m11p3 gate), feed parity, single-leader-per-term, monotonic frontiers - cluster_faults.rs tier-3 suite 4/4: disk-full degrade+recover, slow-fsync lag+converge, both-slow quorum 503, asymmetric partition no-split-brain - tidal-stress soak gates: --json-summary + --max-p99-ms/--max-error-pct/ --fail-on-knee → non-zero exit on regression - Woodpecker cron nightly flow (chaos + gated soak), event-routed, not GH Actions - guarantee-traceability.md: roadmap §2 guarantees → named tests (closes G-C apparatus; 30-day-green is a calendar criterion)
1376 lines
79 KiB
Markdown
1376 lines
79 KiB
Markdown
# tidalDB Cluster Runbook
|
||
|
||
Operating the multi-region `tidal-server cluster` surface: launch (single-process
|
||
dev fabric **and** the multi-process region nodes), the operational API,
|
||
replication transport facts, failover and partition drills, and the honest
|
||
write-durability contract.
|
||
|
||
> ## STATUS: EXPERIMENTAL — TWO MODES, NEITHER IS QUORUM-ACKED HA YET
|
||
>
|
||
> Cluster mode has **two shapes**, both behind the same experimental opt-in:
|
||
>
|
||
> **1. Multi-process (`--region`) — real process isolation.** Each
|
||
> `tidal-server cluster --region <name>` process owns **exactly one region**: one
|
||
> `TidalDb`, one [`GrpcTransport`](#4-grpc-replication-transport-tidal-net) whose
|
||
> server binds *this* region's `grpc_addr` and whose peers are every **sibling
|
||
> region's real `grpc_addr`**. Processes peer over real gRPC and forward over real
|
||
> HTTP, so a crash of one region's process takes down **only that region** — the
|
||
> survivors keep serving. This is genuine process (and, across hosts, host)
|
||
> isolation. It is verified end-to-end by the tier-3 suites
|
||
> (`cluster_multiproc`, `cluster_chaos`, `cluster_lifecycle`, `cluster_runbook`)
|
||
> over real OS processes with real network-partition injection.
|
||
>
|
||
> **2. Single-process (no `--region`) — the dev/demo default.** Every region runs
|
||
> inside **one** process. Replication still traverses the **real `tidal-net` gRPC
|
||
> transport** on loopback (faithful multi-region semantics over a real wire), but
|
||
> there is **no process isolation**: a crash, OOM, or host failure takes the whole
|
||
> "cluster" down at once. This is a development / staging / demo fabric and a
|
||
> correctness harness for the replication paths — **not** production HA. It remains
|
||
> the default because it needs no per-region topology addresses and no process
|
||
> orchestration.
|
||
>
|
||
> **Honest remaining limits (both modes):**
|
||
> * **Quorum durability is opt-in.** The default `204` is leader-durable
|
||
> (storage + WAL fsync; follower ship off the request path). Since m11p3,
|
||
> `ack=quorum` — topology default or per-request `x-tidal-ack` header —
|
||
> gates success on a **majority of the replica set durably holding the
|
||
> write**, surviving permanent leader loss (see
|
||
> [§8](#8-write-durability-contract-the-ack-knob-and-its-cost-m11p3)).
|
||
> * **Leadership is automatic** (m11p4). Every node runs a failure detector and
|
||
> a Raft-style election (pre-vote + vote + check-quorum + fenced transfer):
|
||
> kill the leader and the survivors elect a successor — typically under a
|
||
> second with the defaults — with **zero operator verbs** and zero
|
||
> acknowledged-write loss (see [§9.1](#91-automatic-failover-m11p4--the-default)).
|
||
> `/cluster/promote` is now a **fenced transfer** for maintenance/override, not
|
||
> the availability mechanism. `election.auto_election: false` preserves the
|
||
> pre-m11p4 operator-driven posture.
|
||
> * **Membership is elastic and addresses are DNS names** (m11p5). `grpc_addr`
|
||
> is an advertised hostname or IP (DNS-resolved on every reconnect); nodes join
|
||
> online via `--seed` and catch up via `FetchSnapshot` + the `StreamSegments`
|
||
> stream; add/remove ride kind-4 membership records on the replicated log
|
||
> (see [§1b](#1b-multi-process-one-process-per-region),
|
||
> [§3](#3-topology-yaml), [§6](#6-cluster-management-api),
|
||
> [§9.1](#91-automatic-failover-m11p4--the-default)).
|
||
>
|
||
> **For a production deployment today**, run a **single `tidal-server standalone`**
|
||
> node backed by host-level redundancy and disk durability (see
|
||
> [kubernetes.md](kubernetes.md) and [server-deployment.md](../guides/server-deployment.md)),
|
||
> and reach for multi-process cluster mode for read-scale / multi-region
|
||
> deployments whose writes need `ack=quorum`'s failover-survivable contract.
|
||
>
|
||
> **Both modes refuse to start** unless you explicitly opt in with either the
|
||
> `--experimental-cluster` flag or the `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1`
|
||
> environment variable. On start each emits a loud, **mode-specific** `WARN`
|
||
> restating exactly what it does and does not provide. Do not wire either into a
|
||
> production load balancer.
|
||
|
||
## Scope: what cluster mode does and does not own
|
||
|
||
tidalDB owns **retrieval and ranking only**. Cluster mode adds multi-region
|
||
replication of that, nothing more. It does **not** generate embeddings, store
|
||
video/blobs, run a CDN or transcoder, do auth/moderation/payments, or provide a
|
||
distributed consensus log. Bring your vectors; tidalDB retrieves and ranks over
|
||
them. See [VISION.md](../../VISION.md) for the scope boundary.
|
||
|
||
## Prerequisites
|
||
|
||
- Rust toolchain ≥ 1.91 if running directly (the Docker build pins
|
||
`rust:1.91-bookworm`).
|
||
- `protobuf-compiler` (`protoc`) and a C++ toolchain (`g++`) on the build host —
|
||
`tidal-net`'s build script compiles the WAL-shipping `.proto`, and USearch's
|
||
HNSW core is C++. The Docker image installs both.
|
||
- Docker 25+ if running via container.
|
||
- For **single-process** mode: one HTTP port (default `9500`); each follower
|
||
region also binds an **OS-assigned loopback port** for its gRPC replication
|
||
server unless you pin one in the topology (see [§3](#3-topology-yaml)).
|
||
- For **multi-process** mode: per-region `grpc_addr` **and** `http_addr` declared
|
||
in the topology, and the ports they name available on each host
|
||
(see [§3](#3-topology-yaml)).
|
||
|
||
## 1. Launch the cluster locally
|
||
|
||
Cluster mode is gated. Pass `--experimental-cluster` (or set
|
||
`TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1`) or the server exits with a mode-specific
|
||
error explaining why.
|
||
|
||
### 1a. Single-process (dev/demo default)
|
||
|
||
```bash
|
||
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
|
||
cargo run -p tidal-server -- \
|
||
cluster \
|
||
--listen 127.0.0.1:9500 \
|
||
--schema tidal-server/config/default-schema.yaml \
|
||
--topology tidal-server/config/default-cluster.yaml \
|
||
--experimental-cluster
|
||
```
|
||
|
||
The default topology spins up three regions (`us-east`, `eu-west`, `ap-south`)
|
||
with `us-east` as leader, all inside one process. On a clean start you will see a
|
||
`WARN` line stating this is experimental and single-process, an `info` line per
|
||
follower (`follower gRPC transport ready`), and finally
|
||
`listening on http://127.0.0.1:9500`.
|
||
|
||
### 1b. Multi-process (one process per region)
|
||
|
||
Pass `--region <name>` (or set `TIDAL_REGION`) to run **only that region** in this
|
||
process. The topology must declare a per-region `grpc_addr` **and** `http_addr` for
|
||
every region (see [§3](#3-topology-yaml)); siblings reach each other over those.
|
||
Launch one process per region — typically one per host, each with its own
|
||
`--data-dir`. **`--data-dir` is REQUIRED in multi-process mode** (m11p2): the
|
||
durable WAL is the replicated log itself — it is what ships to peers and what
|
||
serves their catch-up streams — so a node without one is rejected at startup:
|
||
|
||
```bash
|
||
# Region us-east (the initial leader) on host A
|
||
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
|
||
tidal-server cluster --experimental-cluster \
|
||
--region us-east \
|
||
--listen 0.0.0.0:9501 \
|
||
--schema /etc/tidal/schema.yaml \
|
||
--topology /etc/tidal/topology.yaml \
|
||
--data-dir /var/lib/tidal/us-east
|
||
|
||
# Region eu-west on host B
|
||
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
|
||
tidal-server cluster --experimental-cluster \
|
||
--region eu-west --listen 0.0.0.0:9502 \
|
||
--schema /etc/tidal/schema.yaml --topology /etc/tidal/topology.yaml \
|
||
--data-dir /var/lib/tidal/eu-west
|
||
|
||
# Region ap-south on host C
|
||
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
|
||
tidal-server cluster --experimental-cluster \
|
||
--region ap-south --listen 0.0.0.0:9503 \
|
||
--schema /etc/tidal/schema.yaml --topology /etc/tidal/topology.yaml \
|
||
--data-dir /var/lib/tidal/ap-south
|
||
```
|
||
|
||
Every process parses the **same** topology file; `RegionId`s are assigned by
|
||
declaration order, so all processes agree on the region→id mapping. Each process
|
||
binds its own region's `grpc_addr`/`http_addr` and dials its siblings' addresses.
|
||
The `--listen` HTTP address is the public gateway for that region.
|
||
|
||
#### Seed-join boot (m11p5 — adding a node without editing the topology)
|
||
|
||
A node can join an existing cluster **online**, without appearing in any
|
||
declared topology, by contacting a running peer as a **seed**:
|
||
|
||
```bash
|
||
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 \
|
||
tidal-server cluster --experimental-cluster \
|
||
--region eu-2 \
|
||
--listen 0.0.0.0:9504 \
|
||
--schema /etc/tidal/schema.yaml \
|
||
--topology /etc/tidal/topology.yaml \
|
||
--data-dir /var/lib/tidal/eu-2 \
|
||
--seed http://10.0.1.10:9501 \
|
||
--seed http://10.0.2.10:9502 \
|
||
--advertise-grpc eu-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9600 \
|
||
--advertise-http eu-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9504 \
|
||
--metrics 0.0.0.0:9091
|
||
```
|
||
|
||
| Flag | Meaning |
|
||
|------|---------|
|
||
| `--seed <url>` (repeatable) | One or more seed HTTP base URLs. The joiner polls each for the current leader, then `POST /cluster/join`s through it. Any reachable seed works; list a few for resilience. |
|
||
| `--advertise-grpc <host:port>` | This node's **advertised** gRPC address — what siblings dial. Required with `--seed` (the joiner has no topology entry of its own). DNS-capable. |
|
||
| `--advertise-http <host:port>` | This node's advertised HTTP gateway — what peers forward writes/status to. Required with `--seed`. |
|
||
| `--metrics <addr>` | Prometheus `/metrics` bind (the joiner has no topology `metrics_addr`). |
|
||
|
||
What happens: the joiner skips the topology's "every region declared" gate,
|
||
learns its **roster + assigned id + current term** from the seed's join
|
||
response, appends a **Learner** record to the replicated log (the leader answers
|
||
only after it is quorum-committed), persists the roster to a durable membership
|
||
cache and the term to `election_state` (persist-before-act), installs a
|
||
[snapshot](#8-write-durability-contract-the-ack-knob-and-its-cost-m11p3) when it
|
||
is behind the leader's retained WAL, then streams the live tail. The leader's
|
||
**auto-promotion** duty flips it Learner → Voter once it is within
|
||
`replication.learner_promote_lag` (default 1024) of the flushed frontier. A
|
||
**restart** boots from the cache without the seed.
|
||
|
||
> **A `--seed` boot still requires the local topology/config file** for the
|
||
> behavioral knob blocks (`replication:`, `wal:`, `election:`, `timeouts:`,
|
||
> `grpc_tls`) — its `regions:` list is ignored for the roster (the join response
|
||
> is the roster), but a bare `--seed` with neither `--topology` nor
|
||
> `TIDAL_CONFIG` **refuses to boot** naming the rule, so a joiner never silently
|
||
> inherits the compiled-in defaults' wrong ack mode, quorum timeout, or election
|
||
> timing. The k8s manifests mount the shared bootstrap ConfigMap on every pod
|
||
> including N≥3, so this costs nothing there.
|
||
|
||
> **Capability gate (m11p5 mixed-version safety).** The leader **refuses
|
||
> `/cluster/join` and every conf-change until all current voters report kind-4
|
||
> capability** — see [§8](#8-write-durability-contract-the-ack-knob-and-its-cost-m11p3)'s
|
||
> downgrade rule. Complete the binary upgrade before adding or removing nodes.
|
||
|
||
**Auth:** set `TIDAL_API_KEY=<secret>` to require `Authorization: Bearer <secret>`
|
||
on the data and `/cluster/*` mutation routes. If it is **unset the server runs
|
||
UNAUTHENTICATED and logs a WARN** — never expose an unauthenticated cluster
|
||
beyond loopback / a trusted VPC. Health probes and `/openapi.json` are always
|
||
unauthenticated. In multi-process mode set the **same** key on every process: a
|
||
forwarded/broadcast request passes the caller's `Authorization` through verbatim,
|
||
and the internal-propagation marker (`x-tidal-internal: 1`) is an inter-sibling
|
||
trust signal, **not** an auth bypass (the bearer middleware still runs first).
|
||
|
||
Useful environment variables:
|
||
|
||
| Var | Mode | Effect |
|
||
|-----|------|--------|
|
||
| `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1` | both | Opt in to cluster mode (alternative to `--experimental-cluster`). |
|
||
| `TIDAL_REGION` | multi-process | Selects the single region this process owns (alternative to `--region`). |
|
||
| `TIDAL_API_KEY` | both | Bearer token for protected routes. Unset ⇒ unauthenticated + WARN. Set the SAME key on every region in multi-process mode. |
|
||
| `TIDAL_HLC_SKEW_MS` | multi-process | Signed ms offset applied to THIS process's HLC. Affects reconcile-time LWW stamping ONLY (not signal-decay timestamps). A test/ops escape hatch for verifying causal convergence under clock skew — do not set it in normal operation. |
|
||
| `TIDAL_CONFIG` | both | Config dir holding `default-schema.yaml` / `default-cluster.yaml` (used when `--schema` / `--topology` omitted). |
|
||
| `PORT` | both | Listen address. A bare port (`9500`) normalises to `0.0.0.0:9500`. |
|
||
| `TIDAL_SERVER_LOG` | both | `tracing` filter (default `info`). |
|
||
|
||
## 2. Launch via Docker
|
||
|
||
```bash
|
||
# Build the image once (build context is the repo root).
|
||
docker build -f docker/cluster/Dockerfile -t tidaldb:cluster .
|
||
|
||
# Run (press Ctrl+C to stop). The image already sets
|
||
# TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1 so cluster mode starts; it still logs the
|
||
# loud experimental WARN. The default CMD is SINGLE-process cluster mode.
|
||
docker run --rm -p 9500:9500 tidaldb:cluster
|
||
```
|
||
|
||
The image bakes the default schema/topology under `/etc/tidal-server` and uses
|
||
`ENTRYPOINT ["tidal-server"]` + a `cluster …` `CMD`, so you can override the
|
||
subcommand (e.g. `docker run tidaldb:cluster standalone …`), pass `--region` for a
|
||
multi-process container, or supply your own config files:
|
||
|
||
```bash
|
||
docker run --rm -p 9500:9500 \
|
||
-e TIDAL_API_KEY=changeme \
|
||
-v "$PWD/configs/my-schema.yaml:/srv/schema.yaml:ro" \
|
||
-v "$PWD/configs/my-topology.yaml:/srv/topology.yaml:ro" \
|
||
tidaldb:cluster \
|
||
cluster \
|
||
--listen 0.0.0.0:9500 \
|
||
--schema /srv/schema.yaml \
|
||
--topology /srv/topology.yaml
|
||
```
|
||
|
||
The container ships a `HEALTHCHECK` that curls `/health`. For Kubernetes/Compose
|
||
deployment patterns see [Cross-references](#cross-references).
|
||
|
||
## 3. Topology YAML
|
||
|
||
The topology file declares regions and the leader.
|
||
|
||
**Single-process minimal default** (`tidal-server/config/default-cluster.yaml`):
|
||
|
||
```yaml
|
||
regions:
|
||
- name: us-east
|
||
- name: eu-west
|
||
- name: ap-south
|
||
leader: us-east
|
||
# Optional: OS worker threads serving cluster write/heal requests (gRPC ship).
|
||
# Defaults to available parallelism when omitted.
|
||
# write_workers: 4
|
||
```
|
||
|
||
**Multi-process** requires a per-region `grpc_addr` **and** `http_addr` (validated
|
||
at startup — a missing or syntactically invalid `host:port` is a hard error naming
|
||
the region; reachability is deliberately not probed, since siblings boot in any
|
||
order):
|
||
|
||
`grpc_addr` is an **advertised** address — a literal `host:port` OR a DNS
|
||
hostname (m11p5). A new optional per-region `grpc_bind` controls the local bind
|
||
independently of what siblings dial; the example below shows the DNS shape (one
|
||
shared file names every region by its per-pod DNS name while each pod binds
|
||
`0.0.0.0`):
|
||
|
||
```yaml
|
||
regions:
|
||
- name: us-east
|
||
grpc_addr: "tidaldb-0.tidaldb-peers.svc.cluster.local:9601" # ADVERTISED (siblings dial; DNS re-resolved on reconnect)
|
||
grpc_bind: "0.0.0.0:9601" # LOCAL bind (optional; see derivation rule below)
|
||
http_addr: "10.0.1.10:9501" # this region's public HTTP gateway (forwarding + status)
|
||
metrics_addr: "10.0.1.10:9091" # optional Prometheus /metrics listener (set it in production)
|
||
- name: eu-west
|
||
grpc_addr: "tidaldb-1.tidaldb-peers.svc.cluster.local:9602"
|
||
grpc_bind: "0.0.0.0:9602"
|
||
http_addr: "10.0.2.10:9502"
|
||
metrics_addr: "10.0.2.10:9091"
|
||
- name: ap-south
|
||
grpc_addr: "tidaldb-2.tidaldb-peers.svc.cluster.local:9603"
|
||
grpc_bind: "0.0.0.0:9603"
|
||
http_addr: "10.0.3.10:9503"
|
||
metrics_addr: "10.0.3.10:9091"
|
||
leader: us-east
|
||
# Optional m11p1 tuning blocks (engine defaults shown):
|
||
# replication:
|
||
# batch_max_events: 256 # events coalesced per shipped batch (1-256)
|
||
# window: 4 # in-flight batches per peer (1-64)
|
||
# retry_ms: 100 # backoff before a transiently-failed batch retries
|
||
# catchup_retry_ms: 30000 # backoff before a FAILED catch-up pull re-pulls (m11p4)
|
||
# snapshot_artifact_ttl_ms: 600000 # staged-snapshot reuse window (m11p5; never unpins an active consumer)
|
||
# learner_promote_lag: 1024 # learner→voter promotion distance + readiness hysteresis (m11p5)
|
||
# reseed_self_restart: false # drain+exit(0) when reseed_required latches (m11p5; k8s sets true)
|
||
# wal:
|
||
# batch_size: 100 # events per group-commit fsync (1-256)
|
||
# batch_timeout_ms: 10 # max wait before a partial batch flushes
|
||
# election: # m11p4 failure-detector / election timers (defaults shown)
|
||
# heartbeat_interval_ms: 300
|
||
# election_timeout_min_ms: 1500 # validated: leader_lease_ms + heartbeat_interval_ms < this
|
||
# election_timeout_max_ms: 3000
|
||
# leader_lease_ms: 900
|
||
# auto_election: true # false → pre-m11p4 operator-driven failover
|
||
```
|
||
|
||
**`grpc_bind` derivation (m11p5 §1):** `grpc_bind` present → bind it. Absent +
|
||
`grpc_addr` parses as a literal SocketAddr → bind that (today's behavior,
|
||
byte-for-byte — every existing IP topology keeps working). Absent + `grpc_addr`
|
||
is a hostname → bind `0.0.0.0:<port from grpc_addr>`. Because the peer-dial path
|
||
no longer parses `grpc_addr` as a SocketAddr, hyper re-resolves the hostname on
|
||
every reconnect — a pod rescheduled onto a new IP is reached with no peer
|
||
restarts. **TLS note:** SNI follows the URI host, so DNS peer names require
|
||
DNS-SAN certs (see `grpc_tls` below); no code change.
|
||
|
||
Fields:
|
||
|
||
| Field | Single-process | Multi-process | Meaning |
|
||
|-------|----------------|---------------|---------|
|
||
| `regions[].name` | required | required | Region name used everywhere in the HTTP API (`?region=`, `/cluster/promote`, etc.). Must be unique. |
|
||
| `regions[].grpc_addr` | optional | **required** | This region's **advertised** gRPC replication address — what siblings dial. Since m11p5 it may be a literal `host:port` OR a **DNS hostname** (re-resolved on every reconnect). In single-process mode **omit it** — the server allocates a free loopback port and self-heals a bind race by retrying on a fresh port. In multi-process mode it is required and is tried exactly once. |
|
||
| `regions[].grpc_bind` | unused | optional | This region's **local** gRPC bind `host:port` (m11p5), independent of the advertised `grpc_addr`. Omitted: a literal `grpc_addr` binds itself; a hostname `grpc_addr` binds `0.0.0.0:<its port>`. Set it to bind a specific interface while advertising a DNS name. |
|
||
| `regions[].http_addr` | unused | **required** | This region's public HTTP gateway address, used by siblings for write/read forwarding and status aggregation. Unused in single-process mode. |
|
||
| `regions[].grpc_tls` | unused | optional | TLS material for this region's gRPC transport: `ca_cert`, `server_cert`, `server_key` (PEM paths), plus optional `client_cert`/`client_key` for mTLS. Omitted ⇒ plaintext, the right posture for loopback/VPC topologies. **DNS `grpc_addr` requires DNS-SAN certs** (SNI follows the dialed hostname). |
|
||
| `regions[].metrics_addr` | unused | optional | Bind address for this region's Prometheus `/metrics` listener (m11p1; cluster mode previously had none). Omitted ⇒ no metrics endpoint. Set it in every production topology, and bind it internally — the endpoint is unauthenticated. |
|
||
| `leader` | required | required | Must name one of the declared regions. The initial write leader. |
|
||
| `write_workers` | optional | optional | Size of the runtime-free OS-thread pool that admission-controls cluster writes (`/signals` staging) and runs `/cluster/heal`'s blocking redelivery. Bounded queue ⇒ 429 backpressure. Must be ≥ 1 when given. |
|
||
| `timeouts.broadcast_peer_secs` | unused | optional | Per-peer budget (seconds) for the `/cluster/promote` fan-out (the only remaining peer fan-out — the m11p2 log replaced the item/embedding broadcast). Default 2s — right for loopback/VPC; raise it for WAN topologies where a distant region cannot answer in 2s. Must be ≥ 1 when given. |
|
||
| `replication.batch_max_events` | unused | optional | Max relay events coalesced into one shipped batch (1–256, the WAL wire-format ceiling). Default 256. |
|
||
| `replication.window` | unused | optional | In-flight batches per peer (1–64). 1 = strictly in-order shipping; higher pipelines across the peer RTT (out-of-order arrivals park gap-aware on the receiver). Default 4. |
|
||
| `replication.retry_ms` | unused | optional | Backoff (ms) before a transiently-failed batch ship retries. Default 100. |
|
||
| `replication.ack` | unused | optional | Deployment-default write acknowledgment: `leader` (default) or `quorum` (majority-durable — see [§8](#8-write-durability-contract-the-ack-knob-and-its-cost-m11p3)). Per-request override: the `x-tidal-ack` header. |
|
||
| `replication.quorum_timeout_ms` | unused | optional | Budget an `ack=quorum` write waits for the commit index before the retryable 503 naming the laggards. Default 2000. |
|
||
| `replication.catchup_retry_ms` | unused | optional | Backoff (ms) before a FAILED catch-up pull re-pulls on a timer (m11p4) — lets an idle cluster self-heal a follower whose pull failed during a rolling restart. Default 30000. Must be ≥ 1 when given. |
|
||
| `replication.snapshot_artifact_ttl_ms` | unused | optional | Staged-snapshot reuse window (m11p5), counted from the last fetch's completion. Governs artifact REUSE only — **never** unpins the WAL retention of an active consumer; a hard cap (4×) force-drops a never-releasing pin (`tidaldb_cluster_snapshot_pin_force_drops_total`). Default 600000 (10 min). Must be ≥ 1. |
|
||
| `replication.learner_promote_lag` | unused | optional | Learner→voter promotion distance AND the readiness-convergence hysteresis threshold (m11p5), in events. Default 1024. Must be ≥ 1. |
|
||
| `replication.reseed_self_restart` | unused | optional | Drain + clean-exit(0) once the durable `reseed_required` marker latches (m11p5; k8s sets it `true`). **Refused** when the remaining voters can't sustain quorum without this node. Default `false`. |
|
||
| `election.heartbeat_interval_ms` | unused | optional | Leader heartbeat interval (m11p4). Default 300. |
|
||
| `election.election_timeout_min_ms` / `..max_ms` | unused | optional | Randomized follower election timeout window (m11p4). Defaults 1500 / 3000. Validated: `leader_lease_ms + heartbeat_interval_ms < election_timeout_min_ms`. |
|
||
| `election.leader_lease_ms` | unused | optional | Leader freshness lease — a leader that loses majority contact steps down within it (m11p4). Default 900. |
|
||
| `election.auto_election` | unused | optional | `true` (default) = automatic failover; `false` = pre-m11p4 operator-driven posture (no auto elections, no check-quorum step-down). |
|
||
| `wal.batch_size` | optional | optional | Events per WAL group-commit fsync (1–256). Default 100. |
|
||
| `wal.batch_timeout_ms` | optional | optional | Max ms a partial group-commit batch waits before flushing. Default 10. Tune against the measured `tidaldb_cluster_wal_fsync_us` on the deployment's volume. |
|
||
|
||
The schema YAML (signals / text_fields / embedding_slots / profiles) is the same
|
||
format the standalone server loads — see [API.md](../../API.md) and
|
||
[QUICKSTART.md](../../QUICKSTART.md) for the full schema/profile grammar and the
|
||
built-in ranking profiles. Topology only adds the region layout above.
|
||
|
||
> **Personalization correctness note.** Preference-vector personalization (the
|
||
> taste vector that powers `for_you`) only updates for signals declared
|
||
> `positive_engagement: true` in the schema. The cluster `/signals` route writes
|
||
> **global** signals (no `user_id` / `creator_id` context), so it never folds
|
||
> item embeddings into a user's taste vector regardless of the
|
||
> `positive_engagement` flag. Personalized writes go through the embedded engine
|
||
> (`signal_with_context`), not this HTTP route. The exception is `/hardnegs`,
|
||
> which records a **user-scoped** hide that converges across regions via
|
||
> [`/cluster/reconcile`](#6-cluster-management-api).
|
||
|
||
## 4. gRPC replication transport (tidal-net)
|
||
|
||
Replication between regions is the real `tidal-net` `WalShipping` gRPC service,
|
||
not in-process channels. The facts you need to operate and firewall it:
|
||
|
||
| Property | Value | Notes |
|
||
|----------|-------|-------|
|
||
| Service | `WalShipping` (proto `tidal.replication.v1`) | RPCs: **`ShipSegment`** (unary; the live-tail push path), **`StreamSegments`** (server-streaming; the follower-pulled catch-up path since m11p2), **`FetchSnapshot`** (server-streaming; the m11p5 snapshot transfer for joiners + reseeds), **`JoinCluster`** (m11p5 conf-change), and **`Heartbeat`** (ControlPlane health). |
|
||
| Bind address | per-region `grpc_addr` (multi-process); an **auto-allocated** loopback port (single-process) | Default dev band `59520–59529` if you pin one. |
|
||
| Transport security | mTLS via `TlsConfig` (`ca_cert`, `server_cert`, `server_key`, optional `client_cert` / `client_key`) | `insecure = true` (plaintext) is the loopback/VPC default in this phase. A peer with no TLS config **must** set `insecure`. |
|
||
| Max payload | **64 MiB** | Both encoder and decoder codec limits are raised to this; pinned by a compile-time assert to the engine's `InProcessTransport` limit so both ends agree. WAL segments default to 16 MiB. |
|
||
| Circuit breaker | per-peer, **threshold 5**, **reset 30s** | After 5 consecutive ship failures the breaker opens; it stays open for 30s, then the next attempt probes (HalfOpen → Closed on success, Open again on failure). Backpressure (channel full) does **not** trip it. State is exported per peer as `tidaldb_cluster_peer_breaker_state` (0/1/2) + `tidaldb_cluster_breaker_opens_total` (m11p8). **Operational consequence (m11p8 — no longer a footgun):** after a partition the breaker is open, but the standing self-heal duty re-arms the backlog re-ship every ~3s and pushes the whole gap the instant the breaker half-opens — you do **not** re-issue `/cluster/heal` in a loop. Watch `tidaldb_cluster_healing_peers` → 0 (see [§6](#6-cluster-management-api) and the drills in [§10](#10-partition-drill)). |
|
||
| Timeouts (defaults) | connect 5s, request 10s, keep-alive PING every 10s / 5s ACK | A blackholed peer fails fast instead of stalling the single-threaded shipper. |
|
||
|
||
`ShipSegment` carries the WAL segment id, BLAKE3-validated payload bytes, event
|
||
count, and the leader's authoritative `leader_last_seq` (so a follower can
|
||
advance its replication-lag high-water-mark even for an all-local segment that
|
||
filters to empty). The follower's segment-receiver thread drains and applies it
|
||
on arrival. The engine's applied high-water-mark is a **contiguous frontier with a
|
||
bounded ahead-buffer**, so out-of-order eager ships can never swallow a sequence
|
||
gap (a silent-data-loss class fixed in m8p10).
|
||
|
||
> **Operational guard:** the gRPC replication ports and the Prometheus `/metrics`
|
||
> endpoint are **UNAUTHENTICATED**. Bind them to loopback or a cluster-internal
|
||
> network only — never the public interface.
|
||
|
||
## 5. Core HTTP API
|
||
|
||
All routes are JSON unless noted. Examples assume `BASE=http://localhost:9501`
|
||
(any region's gateway in multi-process mode; the single `--listen` address in
|
||
single-process mode) and, when `TIDAL_API_KEY` is set,
|
||
`AUTH='-H "Authorization: Bearer $TIDAL_API_KEY"'`. Drop the `-H` header when
|
||
running unauthenticated.
|
||
|
||
Middleware mirrors standalone: 30s request timeout (408), 100 max in-flight
|
||
(429), 2 MB body limit (413), and an `x-request-id` on every response. Health
|
||
probes sit **outside** the load-shedding stack so liveness/readiness are never
|
||
queued or timed out under saturation.
|
||
|
||
> **Multi-process routing (one coherent surface).** Any region's gateway accepts
|
||
> any operation and routes it to the node that owns it:
|
||
> * **Writes** (`/signals`, `/items`, `/embeddings`, `/hardnegs`) on a non-leader
|
||
> **forward to the leader** (the caller's `Authorization` passes through); the
|
||
> client sees the leader's status/body. A leader that is unreachable degrades to
|
||
> a `503` naming the leader, never a hang.
|
||
> * **Reads** (`/feed`, `/search`) default to the **LOCAL** region. This is the
|
||
> key difference from single-process mode, whose default read is the leader. A
|
||
> `?region=<other>` read **forwards** to that region's process.
|
||
> * `/cluster/promote`, `/cluster/partition`, `/cluster/heal` route to / fan out
|
||
> from the leader as documented in [§6](#6-cluster-management-api).
|
||
|
||
### Health
|
||
|
||
```bash
|
||
curl "$BASE/health" # 200 ok / 503 while draining; reports mode, region, leader
|
||
curl "$BASE/health/startup" # always 200
|
||
curl "$BASE/health/live" # always 200
|
||
curl "$BASE/openapi.json" # served OpenAPI 3.1, UNAUTHENTICATED — canonical HTTP reference
|
||
```
|
||
|
||
`/health` returns `{ "ok": true, "service": "tidaldb", "mode": "cluster",
|
||
"region": "us-east", "leader": "us-east", ... }`. The `/openapi.json` document is
|
||
the machine-readable source of truth for the **data**, **`/cluster/*`**,
|
||
**`/sharded/*`**, and **`/hardnegs`** request/response shapes (the health probes
|
||
are intentionally outside the documented API surface).
|
||
|
||
### Register items & embeddings (one replicated log — m11p2)
|
||
|
||
Items and embeddings ride the **same replicated WAL stream as signals**: the
|
||
leader journals each write as a kind-1 (item metadata) or kind-2 (embedding)
|
||
WAL record BEFORE touching storage, and followers apply it from the log —
|
||
live pushes for the hot tail, the `StreamSegments` catch-up stream for
|
||
history. There is **no HTTP broadcast** anymore (the m8p10 side channel and
|
||
its bug classes were deleted in m11p2), so the responses are plain statuses:
|
||
|
||
```bash
|
||
curl -X POST "$BASE/items" \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{ "entity_id": 1, "metadata": { "title": "Jazz Piano", "category": "music" } }'
|
||
# → 201 Created (item durably journaled into the replicated log + applied)
|
||
|
||
curl -X POST "$BASE/embeddings" \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{ "entity_id": 1, "values": [0.1, 0.2, 0.3, 0.4] }'
|
||
# → 204 No Content
|
||
```
|
||
|
||
The `201`/`204` asserts **leader durability** and carries **`x-tidal-seq`**
|
||
(the record's seqno in the replicated log); `x-tidal-ack: quorum` upgrades it
|
||
to a majority-durable ack (see
|
||
[§8](#8-write-durability-contract-the-ack-knob-and-its-cost-m11p3)). The
|
||
record is fsynced into the stream every follower receives (live push, or
|
||
pull-based catch-up after downtime — see
|
||
[§6 heal](#6-cluster-management-api)). A write to a non-leader forwards to
|
||
the leader transparently (the ack header and seq header travel through the
|
||
forward). A down/partitioned peer needs no backfill bookkeeping: it
|
||
converges from the log when it returns.
|
||
|
||
Embeddings: tidalDB does **not** generate vectors — the caller brings them. The
|
||
write L2-normalizes and inserts into the HNSW index. Dimensions are **strict**:
|
||
they must equal the slot's declared dimensions (min 2, max 4096) or the insert is
|
||
**rejected with a 500** (the engine surfaces a dimension mismatch as an internal
|
||
error, not a 400 — a known wart, tracked post-M8); **zero-norm vectors are also
|
||
rejected (500)**. `RETRIEVE` / `SEARCH` route through the **first declared
|
||
embedding slot only**; multi-modal apps must fuse offline or use separate entity
|
||
kinds.
|
||
|
||
### Record signals (cluster `/signals` = global only)
|
||
|
||
```bash
|
||
curl -X POST "$BASE/signals" \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{ "entity_id": 1, "signal": "view", "weight": 1.0 }'
|
||
# → 204 No Content (see the durability contract in §8)
|
||
```
|
||
|
||
The signal name must be declared in the schema (an undeclared name returns **400**
|
||
naming it). The 204 carries **`x-tidal-seq`** (the write's replicated-log seqno),
|
||
and `x-tidal-ack: quorum` upgrades it to a majority-durable ack — see
|
||
[§8](#8-write-durability-contract-the-ack-knob-and-its-cost-m11p3). This route
|
||
records a **global** signal on the leader and ships it to
|
||
followers over the replicated WAL stream; it does not personalize (see the
|
||
personalization note in [§3](#3-topology-yaml)). On a non-leader gateway it
|
||
forwards to the leader transparently and still returns `204`.
|
||
|
||
### Hard negatives (user-scoped hides)
|
||
|
||
```bash
|
||
curl -X POST "$BASE/hardnegs" \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{ "user_id": 42, "item_id": 7 }'
|
||
# → 204 No Content
|
||
```
|
||
|
||
Records a user-scoped hide on the leader (a non-leader gateway forwards). The item
|
||
is filtered from that user's `/feed?user_id=42`. Hard negatives converge across
|
||
regions via the LWW-resolved [`/cluster/reconcile`](#6-cluster-management-api) CRDT
|
||
path — NOT the signal WAL relay and NOT a broadcast.
|
||
|
||
### Retrieve and search (region-pinned reads)
|
||
|
||
```bash
|
||
# Default read region is the LOCAL region (multi-process mode).
|
||
curl "$BASE/feed?user_id=42&profile=for_you&limit=20"
|
||
curl "$BASE/search?query=jazz%20piano&user_id=42&limit=5"
|
||
|
||
# Pin a read to a specific region. The gateway forwards to that region's process.
|
||
# Followers may lag the leader (and lag jumps during a partition) — use this for
|
||
# canary reads and lag verification.
|
||
curl "$BASE/feed?profile=trending®ion=eu-west"
|
||
```
|
||
|
||
`?region=` accepts any declared region name; an unknown name returns **400**. Omit
|
||
it and the read serves from the **LOCAL** region (multi-process) / the current
|
||
leader (single-process). `limit` is clamped to 1000 at the trust boundary (a
|
||
larger value cannot amplify memory unboundedly). The `for_you` profile applies the
|
||
built-in diversity defaults (`max_per_creator=2`, `format_mix_max_fraction=0.4`,
|
||
`exploration=0.1`).
|
||
|
||
## 6. Cluster management API
|
||
|
||
### Check cluster status
|
||
|
||
Two views. `/cluster/status/local` reports THIS node's own replication/leadership
|
||
state (no peer calls); `/cluster/status` aggregates EVERY region (the gateway calls
|
||
each peer's `/cluster/status/local` concurrently with a tight per-peer budget).
|
||
|
||
```bash
|
||
curl "$BASE/cluster/status/local" | jq
|
||
```
|
||
|
||
```json
|
||
{
|
||
"region": "ap-south",
|
||
"is_leader": false,
|
||
"leader": "us-east",
|
||
"last_seq": 0,
|
||
"applied_events": 124,
|
||
"lag_events": 1,
|
||
"partitioned": [],
|
||
"reachable": true
|
||
}
|
||
```
|
||
|
||
```bash
|
||
curl "$BASE/cluster/status" | jq
|
||
```
|
||
|
||
```json
|
||
{
|
||
"leader": "us-east",
|
||
"relay_log_len": 125,
|
||
"regions": [
|
||
{ "name": "us-east", "applied_events": 125, "lag_events": 0, "partitioned": false, "reachable": true },
|
||
{ "name": "eu-west", "applied_events": 125, "lag_events": 0, "partitioned": false, "reachable": true },
|
||
{ "name": "ap-south", "applied_events": 124, "lag_events": 1, "partitioned": false, "reachable": true }
|
||
]
|
||
}
|
||
```
|
||
|
||
`relay_log_len` is the leader's high-water-mark (`last_seq`); each region's
|
||
`lag_events` is `relay_log_len − applied_events` (saturating). Since m11p3
|
||
`/cluster/status/local` also reports the node's `ack` default and, on the
|
||
leader, `commit_index` — the highest seqno a majority of the replica set
|
||
durably holds (`last_seq − commit_index` is the quorum lag). A region the
|
||
gateway **cannot reach** within the per-peer budget is reported honestly as
|
||
`reachable: false`, `partitioned: true`, `applied_events: 0`, and worst-case lag
|
||
(`lag_events == relay_log_len`). A non-zero, *growing* lag on a reachable region is
|
||
the signal that it is partitioned (ship-skip) or its segment-receiver is wedged.
|
||
|
||
### Promote a new leader
|
||
|
||
```bash
|
||
curl -X POST "$BASE/cluster/promote" \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{ "region": "eu-west" }'
|
||
# → { "ok": true, "leader": "eu-west", "acked": ["eu-west","ap-south"], "failed": [] }
|
||
```
|
||
|
||
The gateway first resolves the TARGET's **stream baseline** — the new leader's
|
||
WAL flushed frontier at promotion, persisted in its `data_dir/stream_baseline`
|
||
— then fans the promote out to every peer carrying it (internal marker, so
|
||
each applies locally and does not re-fan). Peers jump their applied frontier
|
||
for the new leader's shard to the baseline: everything at or below it is
|
||
pre-stream history (replicated applies of the OLD stream), not data. The
|
||
response carries `baseline` plus the `acked`/`failed` fan-out report; a peer
|
||
in `failed` (e.g. a dead old leader) self-corrects — its first parked batch
|
||
from the new stream triggers a catch-up pull whose chunks announce the
|
||
baseline. After promotion `/cluster/status` reports the new leader; new
|
||
writes route there; the new leader's ship queue activates and the demoted
|
||
node's deactivates. An unknown region returns **400**. Since m11p4 `promote` is
|
||
a **fenced transfer** (it drains the target, then sanctions an election), not
|
||
the availability mechanism — automatic failover handles a dead leader with no
|
||
operator action ([§9.1](#91-automatic-failover-m11p4--the-default)); this verb
|
||
is for maintenance and deliberate successor choice
|
||
([§9.2](#92-manual-promote-a-fenced-transfer-maintenance--override)).
|
||
|
||
### Simulate a partition & heal
|
||
|
||
There are **two** ways to partition a region; the runbook drills demonstrate both
|
||
([§10](#10-partition-drill)):
|
||
|
||
1. **Simulated ship-skip flag** — `/cluster/partition` tells the leader to stop
|
||
shipping to the named region (no sockets touched). Good for a controlled,
|
||
reversible lag demo.
|
||
2. **Real network partition** — sever the actual TCP path between processes
|
||
(firewall / proxy). The chaos suite uses a root-free in-harness TCP relay;
|
||
operators can use `iptables`/`pfctl` (see [§10](#10-partition-drill)).
|
||
|
||
```bash
|
||
# Simulated: isolate ap-south — leader ships skip this follower, so its lag climbs.
|
||
curl -X POST "$BASE/cluster/partition" \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{ "region": "ap-south" }'
|
||
# → { "ok": true, "partitioned": "ap-south" }
|
||
|
||
# Heal: resume shipping past everything the follower reports applied, and
|
||
# nudge it to PULL any history that rotated out of the leader's ship tail
|
||
# (the single recovery verb — log catch-up IS the full heal since m11p2).
|
||
curl -X POST "$BASE/cluster/heal" \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{ "region": "ap-south" }'
|
||
# → { "ok": true, "healed": "ap-south" }
|
||
```
|
||
|
||
`/cluster/heal` is the **single recovery verb**, and since m11p2 it is pure
|
||
log catch-up: it clears the partition flag, **resumes** the ship queue past
|
||
the follower's reported applied seqno (retries of data the follower already
|
||
holds prune automatically — every ship ack piggybacks the follower's applied
|
||
seqno), and **nudges** the follower to pull anything older than the leader's
|
||
in-memory ship tail via its `StreamSegments` catch-up stream over the
|
||
leader's durable WAL segments (`POST /cluster/catchup`, internal; the nudge
|
||
forwards the healing operator's own bearer credential). Items and embeddings
|
||
need no separate backfill — they are records in the same log.
|
||
|
||
> **Convergence is self-driving (m11p8 — the operator no longer loops heal).**
|
||
> The per-peer ship senders retry parked batches every `replication.retry_ms`
|
||
> (default 100ms); a follower that detects a gap pulls the catch-up stream
|
||
> itself (also on boot, so a restarted node converges with no operator action
|
||
> at all — the tier-3 `mp_items_ride_the_log_and_catchup_stream` proves it).
|
||
> On top of that, a **standing leader heal duty** runs every ~3s: for any
|
||
> non-partitioned peer whose ship breaker is open AND that trails the leader,
|
||
> it re-arms the backlog re-ship from the peer's durable mark, so the moment the
|
||
> breaker half-opens (threshold 5, reset 30s — see
|
||
> [§4](#4-grpc-replication-transport-tidal-net)) the leader pushes the WHOLE
|
||
> gap. This closes the old footgun: you **no longer re-issue `/cluster/heal`
|
||
> until lag 0** — the server drives it. Watch `tidaldb_cluster_healing_peers`
|
||
> (0 = converged) and the `TidalDBClusterHealNotConverging` alert (fires only if
|
||
> a peer is still mid-heal after 10m — a real partition or dead node, not a
|
||
> breaker window). `/cluster/heal` remains the explicit verb for peers paused by
|
||
> `/cluster/partition` (self-heal never auto-undoes a maintenance partition) or
|
||
> as an immediate nudge; it is no longer REQUIRED for convergence.
|
||
|
||
### Reconcile (cross-region CRDT convergence)
|
||
|
||
```bash
|
||
curl -X POST "$BASE/cluster/reconcile" \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{ "region": "ap-south" }'
|
||
# → { "ok": true, "region": "ap-south",
|
||
# "local_elapsed_ms": 0, "remote_elapsed_ms": 1, "ops_applied": 3 }
|
||
```
|
||
|
||
`/cluster/reconcile` exchanges a CRDT state snapshot with the target region: this
|
||
node ships its snapshot into the target's merge AND applies the target's pre-merge
|
||
snapshot back, so **both sides converge to identical state** by deterministic LWW.
|
||
This is how **hard negatives** (recorded via `/hardnegs`) converge across regions —
|
||
they do not ride the WAL relay. `local_elapsed_ms` / `remote_elapsed_ms` are the
|
||
merge+apply times each side measured (not the HTTP round-trip); both are typically
|
||
0–1ms. Reconcile is **idempotent**: a repeat reconcile of already-converged regions
|
||
is an exact no-op on scores (no drift). `/cluster/reconcile/snapshot` is the
|
||
INTERNAL snapshot-exchange leg this verb drives (marker required); operators never
|
||
call it directly.
|
||
|
||
### Membership verbs (m11p5 — online add / remove / inspect / reseed)
|
||
|
||
Membership is **data on the replicated log** (kind-4 records, latest wins,
|
||
folded into a `ClusterMembership` cell). All four verbs route to / are answered
|
||
by the leader and are **quorum-commit-gated**, one change at a time. The leader
|
||
**refuses every conf-change until all current voters report kind-4 capability**
|
||
(the mixed-version safety gate — complete the binary upgrade first).
|
||
|
||
```bash
|
||
# Add a node (idempotent by name): forwards to the leader, which assigns the
|
||
# next id, appends a Learner record, and answers AFTER quorum-commit.
|
||
curl -X POST "$BASE/cluster/join" \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{ "name": "eu-2",
|
||
"grpc_addr": "eu-2.tidaldb-peers.svc.cluster.local:9600",
|
||
"http_addr": "eu-2.tidaldb-peers.svc.cluster.local:9504" }'
|
||
# → { "id": 4, "role": "learner", "term": 7, "leader": "us-east", "members": [ … ] }
|
||
```
|
||
|
||
The joiner normally seed-joins itself ([§1b](#1b-multi-process-one-process-per-region));
|
||
this verb is the same conf-change for tooling. A re-join from a known name
|
||
returns its existing id and current role and appends nothing. The leader's
|
||
**auto-promotion** duty (a standing duty, re-armed on every activation and
|
||
membership apply — it survives the joining-era leader's death) flips the learner
|
||
to Voter once it is within `replication.learner_promote_lag` of the flushed
|
||
frontier; `promotion_pending (lag=N)` in `/cluster/status/local` makes a stuck
|
||
scale-up diagnosable.
|
||
|
||
```bash
|
||
# Inspect the applied roster (ids, names, addresses, roles, conf version).
|
||
curl "$BASE/cluster/members" | jq
|
||
```
|
||
|
||
```bash
|
||
# Remove a node: appends a Removed tombstone (quorum-commit-gated). The peer's
|
||
# ship cell is retired only AFTER the record is delivered-to/acked-by the removed
|
||
# peer (bounded give-up → tidaldb_cluster_remove_delivery_giveups_total); the
|
||
# removed node's readiness flips to 503 and it stops campaigning. Its id is BURNED
|
||
# (never renumbered, never reused).
|
||
curl -X POST "$BASE/cluster/members/remove" \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{ "region": "eu-2" }'
|
||
# → { "removed": "eu-2", "membership_version": 9 }
|
||
```
|
||
|
||
```bash
|
||
# Force a reseed on demand: latches the durable reseed_required marker. The node
|
||
# keeps serving degraded (voting enabled) and reseeds via snapshot on its NEXT
|
||
# boot (or self-restarts if replication.reseed_self_restart is true and quorum
|
||
# can be sustained without it). See §9.1.
|
||
curl -X POST "$BASE/cluster/reseed"
|
||
# → { "reseed_required": true }
|
||
```
|
||
|
||
**Scale-down order (decommission):** call `/cluster/members/remove` **first**
|
||
(so the cluster's quorum math shrinks before the node disappears), wait for the
|
||
record to quorum-commit, then stop / delete the node — lowest ordinal last under
|
||
a StatefulSet.
|
||
|
||
## 7. Sharded scatter-gather API
|
||
|
||
The `/sharded/*` routes hash-partition entities across regions
|
||
(`hash(entity_id) % num_shards`, using the engine's own `ShardRouter` so writes
|
||
and reads never disagree). Writes route to the single owning region; reads fan out
|
||
to **all** regions and K-way merge by score.
|
||
|
||
Write routes (each routes to the owning region; a non-owner gateway forwards):
|
||
|
||
```bash
|
||
curl -X POST "$BASE/sharded/items" -d '{ "entity_id": 7, "metadata": { "title": "..." } }' # 201
|
||
curl -X POST "$BASE/sharded/embeddings" -d '{ "entity_id": 7, "values": [0.1,0.2,0.3,0.4] }' # 204
|
||
curl -X POST "$BASE/sharded/signals" -d '{ "entity_id": 7, "signal": "view", "weight": 1.0 }' # 204
|
||
```
|
||
|
||
Read routes (scatter-gather across all regions):
|
||
|
||
```bash
|
||
curl "$BASE/sharded/feed?profile=for_you&limit=20&deadline_ms=50"
|
||
curl "$BASE/sharded/search?query=jazz&limit=10&deadline_ms=100"
|
||
```
|
||
|
||
Each sharded read accepts an optional `deadline_ms` — the **total** scatter budget
|
||
(default 50ms, server-clamped to 10s). The per-shard deadline is `deadline_ms − 5ms`
|
||
network overhead. The response includes a `scatter_gather` block:
|
||
|
||
```json
|
||
{
|
||
"items": [ /* merged, deduped, diversity-enforced, re-ranked */ ],
|
||
"total_candidates": 4210,
|
||
"scatter_gather": {
|
||
"degraded": false,
|
||
"shards_queried": 3,
|
||
"elapsed_ms": 12,
|
||
"shard_deadline_ms": 45
|
||
}
|
||
}
|
||
```
|
||
|
||
Degraded semantics: a shard that is partitioned, errors, or misses the deadline is
|
||
reported in `unavailable_shards` (a name list) and flips `degraded: true` — it is
|
||
**never silently dropped**, and the read **still returns 200** with the live
|
||
shards' results. The merge dedups replicated copies of an entity (keeping the
|
||
best-scoring copy), reconciles `total_candidates` so replicated shards are not
|
||
counted multiple times, and re-enforces `max_per_creator` across the merged set.
|
||
|
||
## 8. Write-durability contract: the ack knob and its cost (m11p3)
|
||
|
||
Every replicated write (`/signals`, `/items`, `/embeddings`) runs under one of
|
||
two acknowledgment modes. Pick the deployment default with the topology's
|
||
`replication.ack`; any caller overrides per request with the **`x-tidal-ack`
|
||
header** (`leader` or `quorum`; anything else is a 400).
|
||
|
||
| | `ack=leader` (default) | `ack=quorum` |
|
||
|---|---|---|
|
||
| Success means | Durable on the **leader** (storage + WAL group-commit fsync) | Durable on a **majority of the replica set** (leader + `floor(n/2)` followers, each storage-applied + own-WAL-fsynced) |
|
||
| Survives | Any follower failure; leader restart (WAL replay) | **Any single node's permanent loss, including the leader's** — promote the max-applied survivor and every acked write is there (the 167-kill-point ledger gate proves it) |
|
||
| Does NOT survive | Permanent leader loss before followers caught up (the un-shipped tail dies with it) | Simultaneous majority loss |
|
||
| Latency cost | One group-commit fsync (~ms-scale; the macOS F_FULLFSYNC tail is the local floor) | + one ship RTT + the follower's group-commit fsync, **pipelined**: acks are batch-level, so concurrent writers share the round trip the same way they share fsyncs |
|
||
| Failure mode | 5xx only for local faults | Additionally a **retryable 503** when the quorum budget (`replication.quorum_timeout_ms`, default 2000) expires — the JSON body names the `laggards`, the `commit_index`, and `needed`/`confirmed` counts |
|
||
| Availability | Unaffected by follower outages | **Blocks when a majority is unreachable.** In a 2-region cluster quorum = leader + THE follower: one follower outage stops all quorum writes (by design — that is what the contract says). 3+ regions tolerate `floor((n-1)/2)` follower outages |
|
||
|
||
Mechanics, in one paragraph: there is **one replicated log** (the leader's
|
||
WAL — m11p2); peers only ever receive **fsynced batches** by construction.
|
||
Since m11p3 every follower **pushes its durably-applied frontier** back to
|
||
the leader once per apply round (`ReportApplied` — batch-level, decoupled
|
||
from ship acks, flowing even when the follower converges by catch-up pull or
|
||
the leader is quiet); ship acks additionally carry the same frontier as an
|
||
instant floor hint. The leader folds both into per-peer durable marks; the
|
||
**commit index** is the k-th largest mark (k = `floor(n/2)`), and
|
||
`ack=quorum` responses gate on it passing the write's seqno — awaited
|
||
asynchronously, so quorum waiters never hold threads or starve completions.
|
||
The index is leadership-scoped: promote resets it to the new stream
|
||
baseline, and a demoted leader fails its in-flight quorum waiters (it must
|
||
never claim quorum for a stream it no longer owns).
|
||
|
||
Every cluster write's success response carries **`x-tidal-seq`** — the
|
||
write's seqno in the replicated log (relayed through gateway forwards).
|
||
Persist it if you need an exact durability cursor: `commit_index >= seq` on
|
||
`/cluster/status/local` is "this write is majority-durable", regardless of
|
||
which mode acked it. (The rare dedup-suppressed signal write — an identical
|
||
event within the WAL's ~60s content-hash window is already durably logged,
|
||
so no new record exists to track — carries **`x-tidal-deduplicated: 1`**
|
||
instead, relayed through forwards like the seq header.)
|
||
|
||
**Retry semantics under `ack=quorum` — read this twice.** A quorum-timeout
|
||
503 means *not confirmed in budget*, not *not written*: the write is in the
|
||
leader's log and usually commits moments later. Retries are therefore
|
||
**at-least-once**:
|
||
|
||
- `/items` and `/embeddings` retries are **always safe** — idempotent
|
||
upserts keyed by `entity_id`.
|
||
- `/signals` retries can **double-count** the signal's weight when the
|
||
original did commit (the server stamps each request's timestamp, so the
|
||
WAL's content-hash dedup window cannot identify a client retry). The
|
||
distortion is one extra decaying signal per retried timeout — bounded by
|
||
your retry rate (`tidaldb_cluster_quorum_timeouts_total` is exactly that
|
||
budget). Accounting that cannot tolerate it should route through session
|
||
writes (which carry idempotency keys) or dedup client-side on its own key.
|
||
- The laggard names in the 503 are your runbook pointer: a persistent
|
||
laggard is a down/partitioned region — heal it ([§6](#6-cluster-management-api))
|
||
or accept leader-ack for the duration (`x-tidal-ack: leader`).
|
||
|
||
**Rolling upgrades into m11p3 — upgrade the leader first.** A pre-m11p3
|
||
leader neither serves the `ReportApplied` RPC nor recognizes `x-tidal-ack`:
|
||
it silently applies **leader-ack semantics to a `quorum` request** — a
|
||
durability downgrade the caller cannot see. Upgrade order:
|
||
|
||
1. Promote leadership off the leader node if needed, upgrade it, promote it
|
||
back (or simply upgrade the standing leader per [§9](#9-failover-multi-process)'s
|
||
restart procedure). From this moment `ack=quorum` is honored: the commit
|
||
index rides the m11p2 ship-ack floor hints from not-yet-upgraded
|
||
followers (correct, just laggier).
|
||
2. Upgrade followers one at a time. Each upgraded follower starts pushing
|
||
`ReportApplied` and quorum freshness returns to batch-level. (An upgraded
|
||
follower reporting to a still-old leader is harmless — the report is
|
||
refused and logged, replication and heal are unaffected.)
|
||
|
||
Until step 1 completes, treat the cluster as `ack=leader`-only — do not
|
||
point `ack=quorum` traffic at it expecting majority durability.
|
||
|
||
Other facts unchanged from m11p1/p2: the success code never waits on
|
||
shipping for `ack=leader` (sender threads push the WAL flush feed's tail off
|
||
the request path; `replication.batch_max_events`/`window` tune it); a failed
|
||
fsync errors that write and nothing unfsynced can ship; long-outage data is
|
||
follower-pulled via `StreamSegments` (or, when it has rotated past the leader's
|
||
retained WAL, via the m11p5 `FetchSnapshot` snapshot transfer); a leader crash
|
||
is an **automatic failover** since m11p4 (the survivors elect the
|
||
up-to-date successor — see [§9.1](#91-automatic-failover-m11p4--the-default);
|
||
the vote restriction only elects a node whose log covers every quorum-acked
|
||
write, which is what makes the zero-acked-loss guarantee hold).
|
||
|
||
In short: **`ack=leader` = leader durability. `ack=quorum` = failover-survivable
|
||
durability, priced at one pipelined replication round trip and majority
|
||
availability.**
|
||
|
||
**WAL segment format across upgrades (m11p4).** Segment files carry an
|
||
8-byte version header (`TSEG` + version byte; headerless pre-m11p4 files
|
||
stay readable — no migration). Three behaviors follow:
|
||
|
||
- **Unreadable segments fail the boot, loudly.** A node whose WAL dir holds
|
||
segments written by an incompatible tidalDB version (or stray/foreign
|
||
`.seg` files) refuses to start with `WAL segment format unknown: <path>`
|
||
instead of booting with the data invisibly absent (`segments=0` — the
|
||
2026-06-11 p3 rollout failure mode). Remedy: run a compatible binary, or
|
||
reseed the node (delete its PVC and let it pull from the leader).
|
||
- **Unservable catch-up resolves via snapshot transfer (m11p5).** A leader
|
||
that cannot serve a follower's requested range from its retained WAL answers
|
||
the `StreamSegments` pull with `FAILED_PRECONDITION` — `"segments not
|
||
available from seq N; snapshot required"`, carrying the typed trailer
|
||
`x-tidal-catchup: snapshot-required`. Since m11p5 the follower **latches a
|
||
durable `reseed_required` marker** (only on that typed trailer — an ordinary
|
||
election term-mismatch never latches it fleet-wide) and reseeds itself via the
|
||
`FetchSnapshot` snapshot stream on its next boot (or self-restarts if
|
||
`replication.reseed_self_restart` is set) — no operator verb, no
|
||
`wipe_data_dir`. See [§9.1](#91-automatic-failover-m11p4--the-default). The
|
||
marker is surfaced in `/cluster/status/local` (`reseed_required: true`) and the
|
||
`tidaldb_cluster_reseed_required` gauge.
|
||
- **Failed pulls self-heal on a timer.** A catch-up pull that fails (e.g.
|
||
the leader's gRPC server not yet ready during a rolling restart) retries
|
||
every `replication.catchup_retry_ms` (default 30000) without waiting for
|
||
a write to re-expose the gap — an idle cluster no longer strands lagged
|
||
followers.
|
||
|
||
**Downgrade hazard (m11p4):** a pre-m11p4 binary reading a header-bearing
|
||
segment treats the header as a torn tail and may truncate the final segment.
|
||
Downgrading across the m11p4 boundary requires reseeding the node's WAL.
|
||
|
||
**Membership records + capability gate (m11p5).** A **kind-4 membership
|
||
record** is a new WAL blob kind. A kind-4 record shipped to a **pre-p5
|
||
follower** is an unknown batch kind → `WalError::Corruption` → that follower's
|
||
torn-state receiver halt, **permanent across restarts** (boot self-heal
|
||
re-pulls the same record). Both followers halted = a quorum-write outage. To
|
||
make this structurally impossible, `HeartbeatResponse`/`ReportApplied` carry a
|
||
`capabilities` bit-field (proto3 zero-default = pre-p5 = incapable) and the
|
||
leader **refuses `/cluster/join` and every conf-change until all current voters
|
||
report kind-4 capability** — so the first conf-change cannot fire mid-upgrade.
|
||
**Downgrade rule (kind-3 precedent verbatim): once any kind-4 record is in a
|
||
node's WAL, downgrading it below p5 requires a reseed.** Complete the binary
|
||
upgrade across all voters before adding, removing, or replacing a node.
|
||
|
||
## 9. Failover (multi-process)
|
||
|
||
### 9.1 Automatic failover (m11p4 — the default)
|
||
|
||
**"A machine died" is a non-event.** Every node runs a failure detector
|
||
(leader heartbeats every `election.heartbeat_interval_ms`, default 300) and a
|
||
Raft-style election (pre-vote + vote, randomized
|
||
`election.election_timeout_{min,max}_ms`, default 1500–3000). Kill the
|
||
leader and the survivors elect a successor — typically in **under one
|
||
second** with the defaults, bounded well inside 10s — with **zero operator
|
||
verbs** and zero acknowledged-write loss (the vote restriction only elects a
|
||
node whose log covers every quorum-acked write; the tier-3
|
||
`cluster_election.rs::mp_auto_failover_writes_resume_zero_acked_loss` gate
|
||
proves it across repeated random kill points under `ack=quorum` load).
|
||
|
||
What the operator sees:
|
||
|
||
- `/cluster/status/local` carries `term` (the election term, 0 = the
|
||
pre-election "topology era"), `role` (`leader`/`follower`/`pre-candidate`/
|
||
`candidate`) and `quarantined`.
|
||
- During the brief leaderless window, writes return a retryable 503 naming
|
||
the election (`leader: "none (election in progress)"` plus the responding
|
||
node's `term`); clients retry and land on the new leader.
|
||
- A **restarted ex-leader can never re-claim leadership from its topology
|
||
file**: its durable election state (`data_dir/election_state`) boots it as
|
||
a follower, and every replication RPC is term-fenced — a deposed leader's
|
||
ships, heartbeats and frontier reports are rejected until it rejoins the
|
||
current term (the §1.4-1 split-brain incident is closed by construction;
|
||
proven by `mp_fenced_ex_leader_restart_cannot_write`).
|
||
- A leader that loses contact with a majority **steps down within
|
||
`election.leader_lease_ms`** (default 900) and stops accepting writes:
|
||
`ack=leader` writes during a minority partition are bounded by the lease,
|
||
and `ack=quorum` writes were never at risk.
|
||
|
||
**Divergent suffix / quarantine → automated reseed (m11p5).** A node that
|
||
held leader-acked (never quorum-acked) writes when it died can rejoin into a
|
||
cluster that elected past them. It detects this at term-join and
|
||
**quarantines**: it serves status (`quarantined: true`, metric
|
||
`tidaldb_cluster_divergence_quarantined`) and keeps voting, but refuses the
|
||
data plane. Since m11p5 recovery is **automatic and full-history** — no
|
||
`wipe_data_dir`: the quarantine latches the durable `reseed_required` marker,
|
||
the node reseeds via the `FetchSnapshot` snapshot stream on its next boot (or
|
||
self-restarts if `replication.reseed_self_restart` is set and quorum can be
|
||
sustained without it), rejoins clean, and the divergence gauge clears. This is
|
||
strictly leader-ack-only data, within the documented `ack=leader` crash
|
||
contract (§8). The tier-3 `cluster_reseed.rs` proves the full quarantine →
|
||
marker → restart → reseeded → gauges-cleared loop.
|
||
|
||
**The three-way term-join rule (m11p5).** The divergent-suffix check above is
|
||
one of three outcomes a node reaches when it joins a new term, comparing its own
|
||
stream position against the leader's election-time position (`prev_log`, carried
|
||
on the heartbeat):
|
||
|
||
- `own > prev_log` → **divergent suffix** → quarantine (above);
|
||
- `own < prev_log` → the node is genuinely **missing committed-era history**
|
||
that the new stream's baseline jump would silently skip (the p4 carried
|
||
hazard) → it latches `reseed_required` and reseeds via snapshot (no silent
|
||
gap);
|
||
- `own == prev_log` (or a within-term rejoin) → **clean**, catch-up via the
|
||
stream.
|
||
|
||
A snapshot-installed node always joins clean by construction (its WAL is the
|
||
leader's copy, so its tail term equals the leader's).
|
||
|
||
To run the pre-m11p4 posture (operator-driven failover, no automatic
|
||
elections, no check-quorum step-down), set in the topology:
|
||
|
||
```yaml
|
||
election:
|
||
auto_election: false
|
||
```
|
||
|
||
### 9.2 Manual promote: a FENCED TRANSFER (maintenance / override)
|
||
|
||
`POST /cluster/promote { "region": "eu-west" }` remains the maintenance verb
|
||
— but it is now a **fenced leadership transfer**, not a view flip:
|
||
|
||
1. With a live leader: the leader waits for the target to hold the full
|
||
flushed prefix (the catch-up wait IS the drain), then sanctions an
|
||
immediate election (`TimeoutNow`); the target wins term+1 and the old
|
||
leader steps down on first higher-term contact. Response:
|
||
`{ ok, leader, term, transfer: "elected" }`.
|
||
2. With a dead leader: the target campaigns among the survivors directly —
|
||
the manual override of the automatic path (and the way to *choose* the
|
||
successor).
|
||
3. The election can **refuse a target that lags** (its log loses the
|
||
up-to-date comparison — e.g. the survivor you sampled fell behind between
|
||
your status read and the vote). The verb 503s naming the cause; promote
|
||
the other survivor. **You can no longer accidentally promote a node that
|
||
would discard acked data** — the m11p3 "max-applied survivor" operator
|
||
rule is now enforced by the protocol.
|
||
4. Promoting the node that already leads is a no-op 200.
|
||
|
||
The legacy term-0 fan-out promote survives only for clusters that have never
|
||
elected (mixed-version rollouts mid-upgrade, and the deliberate
|
||
isolated-node override used by the chaos drills); the first joined election
|
||
permanently retires it on each node.
|
||
|
||
> Rolling upgrade m11p3 → m11p4: upgrade ALL binaries before relying on
|
||
> auto-failover (pre-m11p4 peers answer vote RPCs with `Unimplemented`, so
|
||
> no election can reach quorum until a majority is upgraded — the cluster
|
||
> simply keeps its m11p3 behavior until then). The first ELECTED leader
|
||
> journals a kind-3 term-marker WAL record; pre-m11p4 binaries cannot decode
|
||
> it, so do not downgrade a node after the first election without reseeding.
|
||
|
||
## 10. Partition drill (multi-process)
|
||
|
||
The rewritten drill demonstrates **both** partition mechanisms, exactly as
|
||
`cluster_runbook.rs::runbook_s10_partition_drill` scripts them.
|
||
|
||
1. **Baseline.** `GET /cluster/status`; all `lag_events: 0`, `partitioned: false`,
|
||
`reachable: true`.
|
||
2. **Inject.** Use **one** of:
|
||
- **Real network partition** — sever the TCP path peers use to reach the region.
|
||
The chaos suite (`cluster_chaos.rs`) uses a root-free in-harness TCP relay
|
||
proxy (the ROADMAP-sanctioned toxiproxy-style alternative). On a real host you
|
||
can instead use `iptables` (Linux) or `pfctl` (macOS), e.g.
|
||
`iptables -A INPUT -p tcp --dport 9603 -j DROP` to blackhole ap-south's gRPC
|
||
port from a peer. A real cut shows `reachable: false` in the aggregate status.
|
||
- **Simulated ship-skip flag** — `POST /cluster/partition { "region": "ap-south" }`
|
||
(`→ { ok, partitioned: "ap-south" }`). The leader stops shipping to ap-south
|
||
without touching sockets; the aggregate status shows `partitioned: true`.
|
||
3. **Write through it.** Send several `POST /signals`. Each must still `204` (the
|
||
leader-durable contract holds). Watch ap-south's `lag_events` climb while its
|
||
`applied_events` stalls — the leader's ships to it are dropped.
|
||
4. **Read the stale follower.** `GET /feed?region=ap-south` (or read ap-south's
|
||
gateway directly) returns the pre-partition view — eventual, not strong, read
|
||
consistency. The operator console survives a real partition because clients talk
|
||
to each region's gateway directly.
|
||
5. **Scatter-gather degradation.** While partitioned, `GET /sharded/feed` returns
|
||
**200** with `degraded: true` and `ap-south` in `unavailable_shards` — never an
|
||
error, and the live shards' items are still returned.
|
||
6. **Heal.** Clear the cut (heal the proxy / drop the firewall rule, or just call
|
||
heal for the simulated flag), then `POST /cluster/heal { "region": "ap-south" }`.
|
||
**Re-issue heal until `/cluster/status` shows ap-south at `lag_events: 0`** — the
|
||
gRPC circuit breaker opened during the partition (threshold 5, reset 30s), so the
|
||
first heal may ship into an open breaker and no-op. This is genuine production
|
||
behavior, not a flag.
|
||
7. **Verify convergence.** `GET /cluster/status` shows ap-south `lag_events: 0`,
|
||
`partitioned: false`, `reachable: true`; `/sharded/feed` is no longer degraded;
|
||
feed scores on ap-south match the leader to within float tolerance (no loss, no
|
||
duplication).
|
||
|
||
## 11. Shutdown
|
||
|
||
Send `SIGTERM` (or `SIGINT` / Ctrl+C, or stop the container). The server flips
|
||
readiness to **503** (so a load balancer stops routing to it), **drains** in-flight
|
||
requests, then drops the region's `TidalDb` shutdown path: checkpoint in-memory
|
||
signal state → flush storage → write the WAL checkpoint marker + **fsync** → join
|
||
the WAL, sweeper, checkpoint, text-syncer, and replication-receiver threads. The
|
||
drop is idempotent; the process exits 0. You will see
|
||
`region cluster node shutdown: database closed (checkpoint + WAL fsync)`
|
||
(multi-process) / `cluster shutdown: closing all nodes (checkpoint + WAL fsync)`
|
||
(single-process).
|
||
|
||
On **restart with the same `--data-dir`**, the region recovers its pre-shutdown
|
||
state from the WAL (verified by `cluster_runbook.rs::runbook_s11_shutdown_and_wal_recovery`:
|
||
SIGTERM → exit 0 → restart → the same items are served). In multi-process mode, the
|
||
restarted node rejoins the cluster and pulls anything it missed while down via
|
||
its boot-time `StreamSegments` catch-up request — no operator verb needed
|
||
(m11p2; `/cluster/heal` remains the explicit lever for paused peers). This is the per-node step of a
|
||
**rolling upgrade** (promote leadership off the node, SIGTERM, restart on the same
|
||
data dir with the new binary, heal) — see `cluster_lifecycle.rs::mp_rolling_upgrade_no_loss_no_stall`,
|
||
which proves zero acknowledged-write loss across a full rolling upgrade under load.
|
||
|
||
### Node-replace drill (m11p5)
|
||
|
||
Replacing a dead or recycled node is `kubectl delete pod` — no topology edit, no
|
||
operator verb:
|
||
|
||
- **PVC retained** (the data dir survives the pod): the replacement boots on the
|
||
same data dir and converges via its boot-time `StreamSegments` catch-up — the
|
||
ordinary restart path above. If it rotated past the leader's retained WAL while
|
||
down, it reseeds via `FetchSnapshot` automatically (the `reseed_required`
|
||
marker + snapshot path, [§9.1](#91-automatic-failover-m11p4--the-default)).
|
||
- **PVC deleted + pod deleted** (fresh node): the replacement comes up empty and
|
||
**seed-joins** ([§1b](#1b-multi-process-one-process-per-region)) — it `--seed`s
|
||
a survivor, `FetchSnapshot`s the current state, and the leader auto-promotes it
|
||
back to Voter. Under the `k8s/cluster/` StatefulSet this is the default: the
|
||
pod's args carry `--seed` against the headless Service, so a recreated pod
|
||
rejoins with no human in the loop.
|
||
|
||
**Add a node:** scale the StatefulSet up (pod N≥3 seed-joins as a learner and
|
||
auto-promotes). **Remove a node:** `POST /cluster/members/remove` **first**
|
||
([§6](#6-cluster-management-api)), wait for the record to quorum-commit, then
|
||
scale down (lowest ordinal last).
|
||
|
||
## 12. Security (m11p7): mTLS, rotation, identity, audit, rate limits
|
||
|
||
The cluster does not trust the network. Everything here is **opt-in** — absent
|
||
the `grpc_tls` block and the cluster key, the cluster behaves exactly as pre-m11p7
|
||
(plaintext, hint-only marker, no audit/limit). A reference (k8s) deployment turns
|
||
it all on.
|
||
|
||
### 12.1 mTLS (gRPC replication) — the default posture
|
||
|
||
- Configure the `grpc_tls` block per region (`ca_cert`, `server_cert`,
|
||
`server_key`, `client_cert`, `client_key`). The gRPC server then REQUIRES a
|
||
client cert chained to the cluster CA (mutual TLS): a foreign pod with no cert,
|
||
a cert from another CA, or a plaintext probe fails the TLS handshake and never
|
||
reaches an RPC.
|
||
- **No `grpc_tls` ⇒ plaintext, with a loud startup WARN** on both the server and
|
||
the client. Acceptable only on a trusted single-host / loopback topology. To
|
||
serve plaintext intentionally there is nothing else to set — the WARN is the
|
||
signal that you are on the insecure path.
|
||
- The inter-node HTTP plane (forwards, broadcasts, scatter, status, seed-join)
|
||
is served over TLS with the SAME cert and dials `https://` with the cluster CA
|
||
whenever `grpc_tls` is set, so enabling it gives **zero plaintext inter-node
|
||
links** on both planes at once.
|
||
|
||
### 12.2 Cert + bearer rotation WITHOUT restart
|
||
|
||
- A background poller (`TIDAL_ROTATION_POLL_MS`, default 30000) content-hashes the
|
||
cert files and the credential files; on a change it **atomically swaps the
|
||
served cert** (in-flight TLS sessions keep their negotiated keys — zero dropped
|
||
requests) and rebuilds the outbound peer channels.
|
||
- **Procedure:** issue a new cert under the same CA (cert-manager renewal, or
|
||
re-run `scripts/gen-cluster-certs.sh` and re-apply the Secret) — the files
|
||
change in place, the poller swaps within one interval, no pod restart. Verify
|
||
with `tidaldb_cluster_*` logs (`TLS material rotated…`) or the `tidal_audit`
|
||
/tracing stream.
|
||
- The bearer (`TIDAL_API_KEY_FILE`) and the cluster key (`TIDAL_CLUSTER_KEY_FILE`)
|
||
rotate the same way. Use FILE mounts (not inline env) so a Secret rotation is
|
||
picked up live. During a CA roll, keep both old and new CAs trusted for one
|
||
cycle (CA-overlap) so in-flight connections complete.
|
||
|
||
### 12.3 Per-node identity + the marker
|
||
|
||
- Set a shared **cluster key** (`TIDAL_CLUSTER_KEY` / `TIDAL_CLUSTER_KEY_FILE`,
|
||
any random string — BLAKE3-derived to the MAC key). Each node then mints a
|
||
signed `x-tidal-node-token` on every forward/broadcast; the receiver verifies
|
||
it. This gives inter-node calls a verifiable node identity and is the
|
||
defense-in-depth layer beyond the shared bearer.
|
||
- With a cluster key configured, the `x-tidal-internal` marker is honored ONLY
|
||
from a verified sibling: a request that sets the marker without a valid node
|
||
token is rejected **403** (the marker is a routing hint, never an auth bypass).
|
||
Never hand the cluster key to external clients.
|
||
|
||
### 12.4 Admin audit log
|
||
|
||
- promote / partition / heal / join / member-remove / reseed each emit one
|
||
structured record: `{principal, verb, target, term, outcome}`. The principal is
|
||
the verified node (`node:<id>`) for inter-node calls or `external` for an
|
||
operator with the bearer.
|
||
- Sinks: a `tidal_audit` **tracing target** (always — capture it in your log
|
||
pipeline), plus an **append-only JSONL file** when `TIDAL_AUDIT_LOG=<path>` is
|
||
set. Recorded on the operator-originated leg only (no double-record on a
|
||
forwarded re-apply).
|
||
- **At-rest encryption** of the JSONL file is delegated to the volume — mount
|
||
`TIDAL_AUDIT_LOG` on an encrypted PV (or a `gVisor`/LUKS-backed volume); the
|
||
server does not encrypt it in-engine.
|
||
|
||
### 12.5 Per-principal rate limits
|
||
|
||
- `TIDAL_RATE_LIMIT_RPS` (+ optional `TIDAL_RATE_LIMIT_BURST`, default 2×) caps
|
||
per-principal request rate; a deny is **429 + `Retry-After`**. Off by default.
|
||
- Verified sibling nodes are EXEMPT — replication/forward traffic is never
|
||
throttled by the external-client budget. Today external callers share ONE
|
||
bucket: `TIDAL_RATE_LIMIT_RPS` is an AGGREGATE cap across all external clients,
|
||
not a per-client budget — set it to your total external ceiling, not a
|
||
per-client one (one noisy client can consume it). A future multi-key registry
|
||
adds per-key buckets.
|
||
|
||
### 12.6 Foreign-pod / negative behavior (what an attacker on the network sees)
|
||
|
||
| Attempt | Result |
|
||
|---------|--------|
|
||
| Ship a gRPC segment without a cluster client cert | TLS handshake fails — no RPC dispatched |
|
||
| Call an internal HTTP route without trusting the cluster CA | TLS handshake fails — no route reached |
|
||
| Set `x-tidal-internal` without a valid node token (key configured) | 403 — marker honored only from a verified sibling |
|
||
| Call a protected route without the bearer | 401 (unchanged) |
|
||
|
||
## 13. Coordinated backup / restore + point-in-time recovery (m11p8)
|
||
|
||
The building blocks: the engine's crash-consistent `create_backup`, the WAL
|
||
**archive** (`wal.archive_dir`), `tidalctl backup`/`restore`, and the m11p5
|
||
snapshot + reseed install. Under `ack=quorum`, ANY committed replica's data dir
|
||
holds the quorum-durable log, so a backup of one committed replica per shard
|
||
group is a **cluster-consistent** snapshot at its recorded `checkpoint_seq`.
|
||
|
||
### 13.1 Enable the WAL archive (point-in-time recovery)
|
||
|
||
Set `wal.archive_dir` in the topology (or `TidalDb::builder().wal_archive_dir(path)`
|
||
for the embedded engine). Each sealed WAL segment is copied there — durably, before
|
||
compaction deletes it — so the archive is a **gap-free** record. Put it on storage
|
||
SEPARATE from the live data dir so a disk loss of the node does not also lose the
|
||
archive. Segment filenames encode `shard + first_seq`, so co-located groups share
|
||
one archive dir without collision.
|
||
|
||
```yaml
|
||
wal:
|
||
archive_dir: "/archive/tidaldb" # durable, off-node storage
|
||
```
|
||
|
||
### 13.2 Coordinated backup drill
|
||
|
||
1. Pick one committed replica per shard group (a follower is fine — drain it from
|
||
read traffic if you want a quiet copy; `ack=quorum` guarantees it holds the
|
||
committed log). Stop it (or snapshot its volume).
|
||
2. `tidalctl backup --path /data/<node> --out /backups/<cluster>-<ts>/shard-<id>`
|
||
— writes a recursive copy + `BACKUP_MANIFEST.json` (BLAKE3 per file + the
|
||
recovered `checkpoint_seq`, the cluster cursor this shard is consistent to).
|
||
3. Repeat per shard group. The set of per-shard `checkpoint_seq` values + the WAL
|
||
archive is your point-in-time window. Restart the replica; self-heal/catch-up
|
||
reconverges it.
|
||
|
||
### 13.3 Restore drill (timed)
|
||
|
||
1. `tidalctl restore --from /backups/<cluster>-<ts>/shard-<id> --path /data/<new-node>`
|
||
— verifies EVERY file's BLAKE3 against the manifest BEFORE writing, and refuses
|
||
a non-empty target (it never overwrites a live data dir).
|
||
2. Point a stopped node at the restored dir and boot it. Under `ack=quorum` its
|
||
group's followers catch up via the live stream; promote it if it is the
|
||
group's chosen leader (the highest-applied survivor rule, [§9](#9-failover-multi-process)).
|
||
|
||
**Recovery granularity today is full-snapshot-to-frontier, NOT arbitrary
|
||
point-in-time.** A restore recovers a node to the backup's `checkpoint_seq`, then
|
||
the live stream reconverges it to the cluster's CURRENT frontier. The WAL archive
|
||
(`wal.archive_dir`) is the durable, gap-free **primitive** that *backs* a future
|
||
point-in-time replay — but it is write-only today: no tool replays archived
|
||
segments up to a chosen target seq. `tidalctl recover` is verify-only
|
||
(`--verify-only`; an in-place replay mode is reserved for a future release), and
|
||
`tidalctl restore` copies a full snapshot with no seq bound. So the inputs you
|
||
retain for a future PITR are the per-shard `checkpoint_seq` set + the archive;
|
||
treat them as the **window**, not a one-command restore-to-instant. Do not plan an
|
||
incident around seq-bounded replay until a `tidalctl replay --until <seq>` verb
|
||
ships.
|
||
|
||
Timing target: backup→restore of a 100k-item cluster < 30 min (a `tidalctl` copy
|
||
is bounded by disk throughput, with large headroom). The Ref-A timed figure is a
|
||
k3s line item (the standing M11 access caveat).
|
||
|
||
## 14. Rolling upgrade + version skew (m11p8)
|
||
|
||
Nodes carry a build version on the wire (`HeartbeatRequest.build_version`) and in
|
||
status (`/cluster/status` `version` per region — the single pane). Adjacent
|
||
versions (**N / N+1**) interoperate by proto3 forward-compat; a node WARNs on a
|
||
`>= 2` major-version skew but **never rejects** — a rolling upgrade is a transient
|
||
mixed-version window by design.
|
||
|
||
**Procedure (one node at a time):**
|
||
|
||
1. `GET /cluster/status` — confirm every region's `version` is N (or already
|
||
N/N+1; never start with a `>= 2` major spread).
|
||
2. Graceful SIGTERM one follower → it drains (readiness 503 → checkpoint).
|
||
3. Restart it on N+1 (same ports, same data dir). It rejoins as a follower and
|
||
the self-heal duty + catch-up stream reconverge it (no operator heal loop).
|
||
4. Repeat for each follower. Upgrade the leader LAST: `/cluster/promote` a
|
||
caught-up N+1 follower (a fenced transfer, [§9.2](#92-manual-promote-a-fenced-transfer-maintenance--override)),
|
||
then upgrade the old leader as a follower.
|
||
5. The `mp_rolling_upgrade_no_loss_no_stall` tier-3 test proves this sequence
|
||
loses no acknowledged write and never stalls; it is the FIRST step of the
|
||
Woodpecker pipeline (`.woodpecker.yaml`) — a failure blocks the image build.
|
||
|
||
> Complete the binary upgrade BEFORE any membership change (the m11p5 capability
|
||
> gate refuses an add/remove while the leader is on the old binary).
|
||
|
||
## Performance (measured over real localhost processes)
|
||
|
||
| Operation | SLA | Measured (p99 / typical) |
|
||
|-----------|-----|--------------------------|
|
||
| Replicated `/signals` throughput (m11p1, 3 nodes, release build) | ≥ 2,000/s | **4,534 signal-writes/s** within SLO on the ramp (knee ~5.5k/s); **2,739/s sustained 10 min** (1.65M writes, 0.35% errors); was ~90/s pre-m11p1 |
|
||
| Replication lag under that load (m11p1) | < 2s | ≤ 103 events (~40ms) across the 10-min sustain; ≤ 377 events on the 5k/s ramp (follower group-commit coalescing) |
|
||
| Cross-region replication (write → follower applied) | < 2s | ~110–133ms p99 (m8p10) |
|
||
| Failover (`/cluster/promote` → first successful write) | < 10s | ~31–34ms |
|
||
| CRDT reconcile (merge+apply, each side) | < 100ms | 0–1ms |
|
||
|
||
Write-latency note: signal p50 ≈ 17–25ms with a p99 tail of 160–220ms on
|
||
macOS, where `F_FULLFSYNC` averages ~7.4ms with a 10–50ms tail
|
||
(`tidaldb_cluster_wal_fsync_us` — 0% complete under 1ms). On Linux
|
||
`fdatasync` volumes the same pipeline's fsync floor is far lower; validate
|
||
the p99 gate on the reference environment, and tune `wal.batch_timeout_ms`
|
||
against the measured fsync histogram.
|
||
|
||
## 15. Continuous correctness: chaos suites + soak (m11p9)
|
||
|
||
Correctness is a pipeline, not a one-time gate. The tier-3 suites run REAL OS
|
||
processes; the new fault classes inject REAL faults.
|
||
|
||
**Run the chaos suites locally** (serial — they bind fixed ports and spawn
|
||
processes, so suites must not overlap):
|
||
|
||
```
|
||
cargo test -p tidal-server --features "cluster-e2e fault-injection" \
|
||
--test cluster_faults -- --test-threads 1 --nocapture
|
||
```
|
||
|
||
`cluster_faults` covers the fault classes the partition/crash/skew suites lacked:
|
||
|
||
| Test | Fault | Asserts |
|
||
|------|-------|---------|
|
||
| `mp_disk_full_follower_degrades_no_acked_loss` | follower WAL `ENOSPC` | receiver halts (degraded, alive), healthy majority keeps acking quorum, zero acked loss, restart recovers to parity |
|
||
| `mp_slow_fsync_follower_lags_but_quorum_holds` | one follower's fsync slowed | fast follower supplies quorum, slow node lags then converges, no loss |
|
||
| `mp_slow_fsync_both_followers_force_honest_quorum_timeout` | both followers slowed below the budget | `ack=quorum` → retryable 503 naming laggards; `ack=leader` → 204; recover |
|
||
| `mp_asymmetric_partition_no_split_brain_no_loss` | inbound to one node severed (outbound up) | pre-vote + check-quorum hold the leader; single-leader-per-term; no loss |
|
||
|
||
**Fault knobs** (behind the `fault-injection` feature — compiled OUT of the
|
||
production image, inert until armed): `TIDAL_FAULT_FSYNC_DELAY_MS=<ms>` slows
|
||
every durable WAL fsync; `TIDAL_FAULT_DISK_FULL_AFTER_BYTES=<n>` fails segment
|
||
writes with `ENOSPC` after `n` cumulative bytes this process lifetime (arm it on
|
||
a node at restart to fail after `n` bytes of post-restart writes). NEVER set
|
||
these on a production node.
|
||
|
||
**Soak with regression gates** (`tidal-stress`):
|
||
|
||
```
|
||
tidal-stress --target http://<gateway> --ramp "3900:3600" --mix peach \
|
||
--json-summary soak.json --max-error-pct 1 --max-p99-ms 250 --fail-on-knee
|
||
```
|
||
|
||
`--fail-on-knee` (built-in SLO), `--max-p99-ms`, and `--max-error-pct` make the
|
||
run exit non-zero on a regression; `--json-summary` writes a machine-readable
|
||
per-stage roll-up for trend lines. A bounded version runs nightly; the GA-bar
|
||
1-hour 100k-DAU soak points `--target` at the live Ref-A cluster (`--ramp 3900:3600`).
|
||
|
||
**Nightly CI** (`.woodpecker.yaml`, cron `nightly` — Woodpecker, never GitHub
|
||
Actions): the chaos suites with elevated kill-points (`TIDAL_QUORUM_KILLPOINTS`,
|
||
`TIDAL_ELECTION_KILLPOINTS`) then the gated soak. A nightly failure flags the
|
||
day's correctness or performance regression. The guarantee→test map is
|
||
[docs/planning/milestone-11/guarantee-traceability.md](../planning/milestone-11/guarantee-traceability.md).
|
||
|
||
## Cross-references
|
||
|
||
- **Kubernetes deployment** — [docs/runbooks/kubernetes.md](kubernetes.md)
|
||
(the hardened standalone single-replica set in [`k8s/`](../../k8s/), and the
|
||
multi-region cluster reference in [`k8s/cluster/`](../../k8s/cluster/) — one
|
||
StatefulSet + headless Service peer discovery + PDB, with `--seed`-based scale
|
||
and `kubectl delete pod` node-replace, shipped in m11p5).
|
||
- **Server deployment guide** — [docs/guides/server-deployment.md](../guides/server-deployment.md)
|
||
(standalone and cluster launch, config, env, health probes).
|
||
- **Monitoring & alerts** — [docs/ops/monitoring.md](../ops/monitoring.md)
|
||
(Prometheus scrape config, replication-lag and WAL metrics, recommended alerts;
|
||
per-region replication lag is observable via `/cluster/status` `lag_events`;
|
||
remember the `/metrics` endpoint is unauthenticated — bind it internally).
|
||
- **Roadmap / cluster status & known gaps** —
|
||
[docs/planning/ROADMAP.md](../planning/ROADMAP.md) and
|
||
[docs/roadmap-to-cluster.md](../roadmap-to-cluster.md) for the M11 cluster
|
||
status — quorum-ack writes (m11p3), automatic failover (m11p4),
|
||
membership/discovery/elasticity (m11p5), security hardening (m11p7),
|
||
observability + operations (m11p8), and continuous correctness (m11p9) all
|
||
shipped; sharding × replication (p6) data plane in progress.
|
||
- **API & schema reference** — [API.md](../../API.md),
|
||
[QUICKSTART.md](../../QUICKSTART.md), and the live `/openapi.json` document.
|
||
- **Scope & vision** — [VISION.md](../../VISION.md).
|