# tidalDB on Kubernetes How to run tidalDB on Kubernetes. Two manifest sets ship in this repo, mutually exclusive per namespace: - **[`k8s/`](../../k8s/)** — the hardened single-node **standalone** deployment (namespace `tidaldb`, `replicas: 1`). Apply with `kubectl apply -k k8s/`. This is the recommended default: a `StatefulSet` with a durable volume, the three health probes, metrics, secret-backed auth, and graceful rolling updates. - **[`k8s/cluster/`](../../k8s/cluster/)** — the multi-region **cluster** reference (namespace `tidaldb-cluster`, `replicas: 3`, automatic election, quorum-ack writes, elastic membership). Apply with `kubectl apply -k k8s/cluster/`. Still `--experimental-cluster`-gated. See [Cluster mode on Kubernetes](#cluster-mode-on-kubernetes) below. > ## Deploy the STANDALONE server, one replica (recommended default) > > tidalDB is single-node-first: the server wraps one embedded engine whose state > (WAL + checkpoints + indexes) lives on a data dir. It scales **vertically** > (a bigger pod), not by adding replicas — there is no shared-storage multi-writer > mode, so `replicas: 1` in the standalone StatefulSet is load-bearing. Run one > standalone pod, back it with a durable `PersistentVolume`, and recover from the > WAL on restart (see [recovery](../ops/recovery.md)). > > **The standalone pod remains the recommended deployment.** The multi-region > `cluster` mode is now genuinely HA — quorum-acked writes (m11p3), automatic > election/failover (m11p4), and elastic membership (m11p5) all exist — but it is > still `--experimental-cluster`-gated, so choose it deliberately when you need > multi-node availability, and keep a single standalone pod when you do not. The > cluster reference is [`k8s/cluster/`](../../k8s/cluster/); see > [Cluster mode on Kubernetes](#cluster-mode-on-kubernetes). ## What's in `k8s/` | File | Purpose | |------|---------| | `namespace.yaml` | The `tidaldb` namespace | | `schema-configmap.yaml` | The schema YAML the server loads (`--schema`); edit for your signals | | `statefulset.yaml` | The server: durable PVC, probes, security context, resources | | `service.yaml` | Headless `Service` for stable DNS + in-cluster clients | | `poddisruptionbudget.yaml` | `maxUnavailable: 0` — a drain can't silently kill the single node | | `secret.example.yaml` | Template for the API-key secret (create the real one out-of-band) | | `servicemonitor.yaml` | Optional Prometheus-Operator scrape config (apply separately) | | `kustomization.yaml` | Ties the core resources together for `kubectl apply -k` | ## Prerequisites - A Kubernetes cluster (1.25+) and `kubectl` pointed at it. For local testing, [`kind`](https://kind.sigs.k8s.io/) is used in the walkthrough below. - A container registry the cluster can pull from (for real clusters), or a local image loaded into the node (for `kind`). The image is built from [`docker/deploy/Dockerfile`](../../docker/deploy/Dockerfile). - A default `StorageClass` (for dynamic `PersistentVolumeClaim` provisioning). `kind`, GKE, EKS, and AKS all ship one. ## Deploy ### 1. Build and publish the image ```bash # From the repo root — the build context must be the workspace root. docker build -f docker/deploy/Dockerfile -t /tidaldb: . docker push /tidaldb: ``` Set that reference in `k8s/statefulset.yaml` (`image:`), pinned by digest in production (`@sha256:...`). ### 2. Create the namespace and the API-key secret The secret is deliberately **not** in the kustomization so no key lands in git. Create it directly: ```bash kubectl create namespace tidaldb kubectl -n tidaldb create secret generic tidaldb-api-key \ --from-literal=api-key="$(openssl rand -hex 32)" ``` In production, manage it with External Secrets Operator, Sealed Secrets, or Vault Agent instead. The StatefulSet injects it as `TIDAL_API_KEY` — clients then send `Authorization: Bearer ` on every data route. **If the secret is empty the server runs unauthenticated and logs a WARN — never do that on a shared network.** ### 3. Edit the schema (optional) `k8s/schema-configmap.yaml` carries the schema the server loads. Edit it to model your signals, text fields, embedding slots, and (optionally) ranking profiles — the format is documented in [server-deployment.md](../guides/server-deployment.md). The schema is read once at boot; roll the StatefulSet to apply changes. ### 4. Apply ```bash kubectl apply -k k8s/ kubectl -n tidaldb rollout status statefulset/tidaldb --timeout=180s ``` ### 5. Verify ```bash kubectl -n tidaldb port-forward statefulset/tidaldb 9400:9400 & KEY=$(kubectl -n tidaldb get secret tidaldb-api-key -o jsonpath='{.data.api-key}' | base64 -d) curl -s localhost:9400/health # {"ok":true,...} curl -s localhost:9400/openapi.json | jq .info # served API contract curl -s -X POST localhost:9400/items \ -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \ -d '{"entity_id":1,"metadata":{"title":"hello","category":"demo","created_at":"1700000000"}}' curl -s -H "Authorization: Bearer $KEY" "localhost:9400/feed?profile=trending&limit=5" ``` ## How the health probes map The server exposes three unauthenticated endpoints, wired to the three probe types in `statefulset.yaml`: | Probe | Endpoint | Behavior | |-------|----------|----------| | `startupProbe` | `GET /health/startup` | 200 once the HTTP listener is up; high `failureThreshold` covers slow WAL replay / index load on large data dirs (see [capacity-planning](../ops/capacity-planning.md)) | | `livenessProbe` | `GET /health/live` | 200 while the process is alive; restart if it stops answering | | `readinessProbe` | `GET /health` | 200 ready / **503 while draining** — on SIGTERM the pod leaves the Service endpoints before it stops accepting | ## Rolling updates and graceful shutdown On `kubectl rollout restart` (or any pod delete), the kubelet sends SIGTERM. The server flips readiness to 503 (so it leaves the Service), drains in-flight requests, then checkpoints and fsyncs the WAL before exit. `terminationGracePeriodSeconds: 60` gives that room — raise it if your data dir is large. Because there is one replica, a restart is a brief planned outage while the new pod replays the WAL; the `PodDisruptionBudget` (`maxUnavailable: 0`) prevents an *involuntary* drain from taking the node down without operator intent. ## Persistence and backup The `volumeClaimTemplate` provisions a `PersistentVolumeClaim` (`/data`, 10Gi by default — size it from [capacity-planning](../ops/capacity-planning.md)). The WAL + checkpoints there are the source of truth and survive pod restarts. For backup and disaster recovery (snapshotting the PVC, restoring a corrupt data dir), see [recovery](../ops/recovery.md). ## Metrics The pod exposes Prometheus metrics on `:9091/metrics` (**unauthenticated** — it is not exposed by the headless Service externally; keep it cluster-internal). Scrape it one of two ways: - **Prometheus Operator:** `kubectl apply -f k8s/servicemonitor.yaml` (requires the `monitoring.coreos.com` CRDs). - **Plain Prometheus:** the pod carries `prometheus.io/scrape`, `prometheus.io/port`, and `prometheus.io/path` annotations. Alert rules and a dashboard ship in [`docs/ops/prometheus-alerts.yaml`](../ops/prometheus-alerts.yaml) and [`docs/ops/grafana-dashboard.json`](../ops/grafana-dashboard.json); see [monitoring](../ops/monitoring.md). ## Local walkthrough with `kind` This is the exact flow used to verify the manifests end-to-end: ```bash # 1. Create a local cluster. kind create cluster --name tidaldb # 2. Build the image and load it into the kind node (no registry needed). docker build -f docker/deploy/Dockerfile -t tidaldb:deploy . kind load docker-image tidaldb:deploy --name tidaldb # 3. Namespace + API-key secret. kubectl create namespace tidaldb kubectl -n tidaldb create secret generic tidaldb-api-key \ --from-literal=api-key="$(openssl rand -hex 32)" # 4. Apply and wait for ready. kubectl apply -k k8s/ kubectl -n tidaldb rollout status statefulset/tidaldb --timeout=240s # 5. Verify, then tear down. kubectl -n tidaldb port-forward statefulset/tidaldb 9400:9400 & curl -s localhost:9400/health kind delete cluster --name tidaldb ``` The manifests set `image: tidaldb:deploy` with `imagePullPolicy: IfNotPresent`, which is exactly what `kind load` + a local tag need. For a real cluster, swap in your registry image. > **`kind create cluster` fails with "could not find a log line that matches … > Multi-User System"?** On Docker Desktop the node's `systemd` can die at boot > with `Failed to create control group inotify object: Too many open files` > (`docker logs -control-plane` shows it). The Docker VM's inotify > limits are too low; raise them in the VM kernel, then recreate: > > ```bash > docker run --rm --privileged alpine sysctl -w fs.inotify.max_user_instances=8192 > kind delete cluster --name tidaldb && kind create cluster --name tidaldb > ``` > > This is a kind-on-Docker-Desktop prerequisite, unrelated to tidalDB. ## Troubleshooting | Symptom | Likely cause | |---------|--------------| | Pod `Pending` | No default `StorageClass`, or the PVC can't bind — `kubectl -n tidaldb describe pvc data-tidaldb-0` | | Pod `CrashLoopBackOff` at boot | Bad schema YAML in the ConfigMap, or a data dir from an incompatible schema — check logs; see [recovery § schema mismatch](../ops/recovery.md) | | Pod never `Ready`, but `Running` | Readiness probe failing — `kubectl -n tidaldb logs statefulset/tidaldb`; a large data dir may need a longer `startupProbe` | | `401 Unauthorized` on data routes | Wrong/empty `tidaldb-api-key` secret; clients must send `Authorization: Bearer ` | | Writes lost after restart | Data dir not on the PVC — confirm `--data-dir /data` and the `data` volume mount | ## Cluster mode on Kubernetes The cluster reference is [`k8s/cluster/`](../../k8s/cluster/): **ONE `StatefulSet` named `tidaldb`, `replicas: 3`**, in its own namespace `tidaldb-cluster`. Each pod is a region; the three pods (`tidaldb-0/1/2`) form the initial voter set. This is real HA — automatic election/failover, quorum-acked writes, and membership changes that ride the replicated log — but the mode is still `--experimental-cluster`-gated (set via `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1` in the manifest). It is **mutually exclusive** with the standalone set per namespace: they share the StatefulSet name `tidaldb`, and the standalone set's `replicas: 1` is load-bearing. Deploy one or the other. ### Why one StatefulSet (not one per region) Before m11p5, a pod bound its own `grpc_addr` literally, and a pod cannot bind a Service ClusterIP — so each region needed a per-pod topology variant, defeating the "every process parses the same file" contract. The m11p5 **bind/advertise split** removes that: `grpc_addr` is the address peers *dial* (a per-pod headless DNS name, re-resolved by tonic on every reconnect, so a rescheduled pod on a new IP is reachable with no peer restart), while `grpc_bind` is the local socket (`0.0.0.0:9601`). One topology ConfigMap names all three regions by their stable pod DNS, and every pod mounts it unmodified. ### Shard layout: full placement (every pod hosts all three groups) The deployed shape is **3 shard groups, RF3, full placement** — the `shards:` block in the topology ConfigMap is enabled (an absent block would mean one group, RF = all pods). Every pod replicates **all three** groups; leadership balances one-per-pod (tidaldb-0 leads shard 0, tidaldb-1 shard 1, tidaldb-2 shard 2) and the entity space is hash-partitioned across them (~even, ≈⅓ each). - **Per-shard data dir:** each group's WAL + checkpoints + per-shard reseed marker live under `/data/db/shard-00000`, `/data/db/shard-00001`, `/data/db/shard-00002` in the one PVC. (Boot-install and divergent-suffix detection are per-shard — a divergent group heals from *its own* leader.) - **One gRPC port per group:** `replicas[].grpc_addr` is omitted in the `shards:` block, so each bind is **derived as `node base port + shard id`** — shard 0 → 9601, shard 1 → 9602, shard 2 → 9603. The StatefulSet declares all three containerPorts (`grpc`/`grpc-1`/`grpc-2`) for clarity and NetworkPolicy; the bind itself is driven by the topology. Collapse back to a single `grpc` port only if the `shards:` block is removed (legacy single group). > The m12 findings flagged the **m12p4 3-shard catch-up + cross-shard read** layer > as not yet production-ready on real k3s (rc7); those five bugs were root-caused > and **fixed in rc8/rc9**, and the LIVE rc7 image below carries the reseed-loop, > seed-join-promotion, election-divergence, read-SLA, and write-burst fixes. ### Resources at 1536-D The 1536-dim production shape sets the pod resources, deliberately: | Field | Value | Why | |-------|-------|-----| | `limits.cpu` | `"3"` (raised from 2) | The cgroup CPU quota is what the engine reads for `available_parallelism()` (SEARCH_GATE / worker-thread sizing). At `2` a cross-shard search burst **starved the async reactor + the election/heartbeat/apply control plane** — reads hung to the 30 s route timeout and the starved control plane churned elections into reseed self-exit. `3` leaves ~1 core for kubelet/system on the 4-core nodes. | | `requests.cpu` | `500m` | Kept low so the pod still schedules (server nodes alloc ≈ 3). | | `limits.memory` | `4Gi` | A 100k×1536-D HNSW load peaks **~1.9Gi**. 1M needs more headroom — plan **>16Gi nodes** for the 1M gate. | | `startupProbe.failureThreshold` | `240` (~20 min) | HNSW rebuild/load at 1536-D is CPU-bound: **~5 min single-core at 100k**, so the startup budget must cover a cold rebuild (plus headroom for the 1M gate). | | probe `scheme` | `HTTPS` (all three) | The `:9500` HTTP plane serves TLS (inter-node mTLS, m11p7). kubelet does not verify the server cert for httpGet probes, so the cert's DNS-only SANs are fine. | ### What's in `k8s/cluster/` | File | Purpose | |------|---------| | `namespace.yaml` | The `tidaldb-cluster` namespace (mutually exclusive with `tidaldb`) | | `topology-configmap.yaml` | The ONE bootstrap topology shared by all pods: 3 regions by per-pod DNS (`grpc_addr` advertised, `grpc_bind` `0.0.0.0`), `replication.ack: quorum`, `replication.reseed_self_restart: true`, the election block | | `schema-configmap.yaml` | The schema YAML every region loads (`--schema`) | | `statefulset.yaml` | `replicas: 3`, `podManagementPolicy: Parallel`, `TIDAL_REGION` from `POD_NAME`, durable PVC at `/data` with `--data-dir /data/db`, the three probes (readiness now cluster-aware, all `scheme: HTTPS`), uid 10001, `terminationGracePeriodSeconds: 600` with `TIDAL_SHUTDOWN_DRAIN_MS=3000`, the full-placement 3-shard layout (one gRPC port per group), 1536-D resources (`cpu: "3"`, `memory: 4Gi`), topology spread, and the ordinal-branching scale-up wrapper | | `service-peers.yaml` | Headless peer Service (`publishNotReadyAddresses: true`) — stable per-pod DNS, keeps not-ready joiners resolvable for peers | | `service-client.yaml` | Client Service — readiness-gated, drops not-ready/joining/quarantined pods from load balancing | | `poddisruptionbudget.yaml` | `maxUnavailable: 1` — a 3-voter cluster keeps quorum across one disruption | | `secret.example.yaml` | Template for the `tidaldb-credentials` / `TIDAL_API_KEY` secret (create the real one out-of-band) | | `kustomization.yaml` | Ties it together for `kubectl apply -k k8s/cluster/` (secret excluded) | ### The two Services - **`tidaldb-peers`** (headless, `clusterIP: None`, `publishNotReadyAddresses: true`): gives each pod the stable DNS name `tidaldb-N.tidaldb-peers.tidaldb-cluster.svc.cluster.local`, which the topology advertises. `publishNotReadyAddresses: true` is load-bearing — a joiner is *not ready* until it first converges, but peers must still resolve it to feed it a snapshot + catch-up stream; without this the joiner could never reach a seed to become ready (a deadlock). - **`tidaldb`** (client-facing, VIP, readiness-gated): the address in-cluster clients hit. Default readiness gating drops not-ready/joining/quarantined/ draining pods, so a client is never routed to a node that is still catching up. ### The readiness predicate (cluster-aware `/health`) The `readinessProbe` stays `GET /health`, but in cluster mode it is now predicate-driven (m11p5 §4). `/health` returns **503** when the node is: - shutting down (SIGTERM drain — leaves both Services before it stops accepting); - **quarantined** (an m11p4 divergent leader-acked suffix — serves status and votes, refuses the data plane until reseeded); - **removed / decommissioned** (a `Removed` membership record reached it, or a voter's typed `removed` signal told it so); - a **joiner** (seed-join learner) or **install boot** (snapshot-reseeded) that has **not yet first-converged** — convergence means the boot catch-up pull completed at least once AND lag fell to `≤ learner_promote_lag` (hysteresis; **never** `lag == 0`, which an open-loop write load keeps perpetually false). Sticky-ready after the first convergence. A **restarted, PVC-retained voter is Ready on today's terms** — no regression for ordinary pod restarts. `GET /cluster/status/local` surfaces the inputs (`lag_events`, `quarantined`, `role`, `term`, `reseed_required`) for diagnosis, plus the `tidaldb_cluster_reseed_required` and `tidaldb_cluster_divergence` gauges. ### Deploy the cluster ```bash # 1. Build/publish the release image (runtime user uid 10001 to match the # securityContext so the PVC is writable; one image serves every subcommand). # Use the release script — NOT a bare `docker build`: ./scripts/build-release.sh server # It HOST cross-compiles macOS-arm64 -> x86_64-unknown-linux-gnu # (GCC 15.2 / glibc 2.41), then packages a `debian:trixie-slim` runtime # (the binary needs `libmvec.so.1` — ABSENT on bookworm — plus libstdc++6 / # libgcc-s1) via the `amd64builder` buildx builder (QEMU), and pushes to # registry.threesix.ai. # # CRITICAL — pin the linux/amd64 PLATFORM manifest digest, NOT the OCI index # digest and NOT the `unknown/unknown` attestation manifest. Verify before pin: docker buildx imagetools inspect registry.threesix.ai/tidal/server: --raw # Expect mediaType application/vnd.oci.image.manifest.v1+json, ~6 layers, # architecture amd64. (Pinning the index or attestation digest yields an # ImagePullBackOff or a no-arch pull.) # 2. Namespace + the credentials secret (stress/Ref-A shape: name # tidaldb-credentials, key TIDAL_API_KEY). Same key on EVERY pod and client. kubectl create namespace tidaldb-cluster kubectl -n tidaldb-cluster create secret generic tidaldb-credentials \ --from-literal=TIDAL_API_KEY="$(openssl rand -hex 32)" # 3. Apply and wait for the 3-pod voter set. kubectl apply -k k8s/cluster/ kubectl -n tidaldb-cluster rollout status statefulset/tidaldb --timeout=300s # 4. Confirm three reachable regions with low lag, and check each node's role. kubectl -n tidaldb-cluster exec tidaldb-0 -- \ curl -s localhost:9500/cluster/status | jq '.regions[] | {name, lag_events, reachable}' for p in tidaldb-0 tidaldb-1 tidaldb-2; do kubectl -n tidaldb-cluster exec "$p" -- \ curl -s localhost:9500/cluster/status/local | jq '{role, term, membership_role, lag_events}' done ``` #### Upgrade by digest (RollingUpdate) The LIVE image is `registry.threesix.ai/tidal/server:m12-writeburst-rc7` (`@sha256:171505745b801dcf231b531de6167dbc309a7182957811cbc2228f0a302572b1`). It carries the **reseed-loop, seed-join-promotion, election-divergence, read-SLA, and write-burst** fixes (all shipped). To roll a new build: ```bash # Set the image by its amd64 platform-manifest DIGEST (verified above), then watch # the RollingUpdate. The StatefulSet rolls highest-ordinal-first, one pod at a time. kubectl set image statefulset/tidaldb \ tidaldb=registry.threesix.ai/tidal/server@sha256: \ -n tidaldb-cluster kubectl -n tidaldb-cluster rollout status statefulset/tidaldb --timeout=600s ``` Then verify **3/3 Ready** and check each pod's **boot-reseed outcome**: ```bash kubectl -n tidaldb-cluster get pods -l app.kubernetes.io/name=tidaldb for p in tidaldb-0 tidaldb-1 tidaldb-2; do kubectl -n tidaldb-cluster exec "$p" -- \ curl -s localhost:9500/cluster/status/local | jq '{role, term, lag_events, reseed_required}' done ``` The ideal boot-reseed outcome is `[(0,NotNeeded),(1,NotNeeded),(2,NotNeeded)]` — every pod's PVC-retained WAL caught up via stream, no reseed. A **snapshot-install fallback** (a pod rejoined behind WAL retention and reseeded via snapshot) is **SAFE, not a failure**: it converges to `lag=0` on its own; readiness simply stays 503 until the install first-converges. Only a pod that *loops* (quarantine → self-restart → re-detect) is a real problem — see [disaster-recovery.md](disaster-recovery.md). ### Scale up (3 → N): seed-join as a learner, auto-promote `kubectl scale` is the whole story — **no topology edits**. Pods with ordinal `≥ 3` boot with `--seed` (the StatefulSet's ordinal-branching wrapper adds it automatically) and learn their roster/id/term from a seed, joining as a **learner**. The leader auto-promotes a learner to a voter once its durable mark is within `learner_promote_lag` of the leader's frontier (or stops falling behind for K rounds under sustained load). ```bash # Grow to 5 voters. Pods tidaldb-3 and tidaldb-4 seed-join + auto-promote. kubectl -n tidaldb-cluster scale statefulset/tidaldb --replicas=5 kubectl -n tidaldb-cluster rollout status statefulset/tidaldb --timeout=600s # Watch the new members converge and promote (promotion_pending shows the lag): kubectl -n tidaldb-cluster exec tidaldb-0 -- \ curl -s localhost:9500/cluster/members | jq '.members[] | {id, name, role}' ``` A scaled pod still mounts the shared topology ConfigMap — a `--seed` boot **requires** the local config for the behavioral knob blocks (`replication`, `wal`, `election`, `timeouts`, `grpc_tls`); its `regions:` list is ignored for the seed joiner's roster (the join response is authoritative). ### Scale down (N → fewer): remove verb FIRST, then scale Decommission a member **before** removing its pod, so the cluster stops counting it toward quorum cleanly. Remove the highest-ordinal members (StatefulSet deletes lowest-ordinal-last on scale-down). ```bash # Going 5 -> 3: decommission tidaldb-4, then tidaldb-3, then scale. # Remove verb: POST /cluster/members/remove {"region": ""} (any node # forwards to the leader; one-at-a-time, quorum-commit-gated). kubectl -n tidaldb-cluster exec tidaldb-0 -- curl -s -X POST \ -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \ -d '{"region":"tidaldb-4"}' localhost:9500/cluster/members/remove kubectl -n tidaldb-cluster exec tidaldb-0 -- curl -s -X POST \ -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \ -d '{"region":"tidaldb-3"}' localhost:9500/cluster/members/remove # Each Remove is delivered to the removed peer (which flips to 503 and stops # campaigning) before its ship cell retires. Only then is it safe to scale. kubectl -n tidaldb-cluster scale statefulset/tidaldb --replicas=3 ``` Removing pods without the remove verb leaves zombie members the cluster still expects — never scale down before decommissioning. ### Node / pod replacement - **`kubectl delete pod tidaldb-N` (PVC retained):** the StatefulSet recreates the pod onto its existing PVC. It boots a follower, replays its WAL, and catches up via the `StreamSegments` stream (boot-time + the 30 s catch-up timer). No operator verb. This is the routine replace — already proven by the p4 leader-kill regression (lag → 0 within ~10 s). - **PVC + pod delete (fresh reseed):** delete the PVC *and* the pod. The new pod comes up with an empty data dir, latches an install boot, and **reseeds via snapshot** (`FetchSnapshot` + stream) from the leader, then rejoins clean. Use this when the data dir is corrupt or the node fell behind a compacted leader. Readiness stays 503 until the snapshot install first converges. ### Self-healing reseed (quarantine / behind-compaction) `replication.reseed_self_restart: true` (set in the topology ConfigMap) makes a node that durably latches `reseed_required` (an m11p4 divergence quarantine, or a typed `snapshot-required` refusal) drain and exit(0); the StatefulSet restarts it and the **boot-time install** reseeds it via snapshot, clearing the quarantine and the divergence gauge with no operator verb and no PVC wipe. The self-restart is **refused** (loudly, in `/cluster/status/local` + the gauge) when the remaining voters cannot sustain quorum without this node — exiting during a 2-voter window would be a total write outage. `POST /cluster/reseed` latches the marker on demand. ### PodDisruptionBudget and graceful shutdown `maxUnavailable: 1` lets a voluntary disruption (node drain, autoscaler, rolling upgrade) take at most one pod at a time, preserving quorum (2 of 3). On SIGTERM a pod flips readiness to 503 (leaving both Services), drains, lets the leader lease/heartbeat windows lapse so a successor is elected cleanly, then checkpoints + fsyncs the WAL **and saves every hosted shard's HNSW graph** before exit. `terminationGracePeriodSeconds: 600` (raised from 60 in m12p6) covers that sequence. At 1536-D the long pole is the graph save: each pod hosts all three shard groups, and a single group's USearch serialize+fsync (~32k vectors/slot) is slow, so the three concurrent saves need ample budget or k8s SIGKILLs mid-save and the next boot rebuilds from raw vectors (a ~5 min single-core stall at 100k). `TIDAL_SHUTDOWN_DRAIN_MS=3000` shortens the post-SIGTERM in-flight drain (from the 15 s default) so the save *starts* promptly inside the grace window; a clean save typically finishes in well under a minute, so 600 s is a ceiling, not the norm — but it makes rolling restarts on a loaded cluster slow, since each pod saves before it exits. ### The exit-gate harness The in-cluster load Jobs in [`tidal-stress/k8s/`](../../tidal-stress/k8s/) target the cluster by per-pod DNS (`tidaldb-N.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500`), in namespace `tidaldb-cluster`, reading the `tidaldb-credentials` secret. They run the capacity ramp, the quorum-throughput gate (`stress-job-t2a.yaml`, `--ack quorum`), and the leader-kill chaos drill (`stress-job-t2b.yaml`). ## See also - [Server deployment guide](../guides/server-deployment.md) — config, auth, the served OpenAPI spec - [Build a feed app](../guides/build-a-feed-app.md) — what to run against this server - [Cluster runbook](cluster.md) — the multi-region mode's operational API (launch, promote, heal, reseed) - [Disaster recovery](disaster-recovery.md) — backup, restore, and DR for the cluster (snapshot/restore, behind-compaction reseed, divergence recovery) - [Monitoring](../ops/monitoring.md) · [Capacity planning](../ops/capacity-planning.md) · [Recovery](../ops/recovery.md)