# tidalDB Runbooks — index

Operational documentation for the **cluster** deployment (ns `tidaldb-cluster`,
single StatefulSet, full-placement RF3, quorum-ack + automatic election). Live state
and history live in the `orchard9-k3sf` repo (`cluster-state.yaml`,
`deployments/history/tidaldb.md`).

| When you need to… | Open |
|---|---|
| **Respond to an incident / page** — symptom → signal → fix | [`on-call.md`](on-call.md) |
| Operate the cluster — API, failover, partition, membership, scale, rolling upgrade | [`cluster.md`](cluster.md) |
| Run it on Kubernetes — deploy, digest-pin, probes, scale up/down, troubleshoot pods | [`kubernetes.md`](kubernetes.md) |
| Back up / restore / DR — object-store export, restore, byte-verify, query-proof, R2, rebuild | [`disaster-recovery.md`](disaster-recovery.md) |
| Recover a single node / standalone engine — corrupt keyspace, WAL, lock, schema | [`../ops/recovery.md`](../ops/recovery.md) |
| Size a deployment — single-node tables + the measured Ref-A cluster envelope | [`../ops/capacity-planning.md`](../ops/capacity-planning.md) |
| Read the metrics / wire dashboards & alerts | [`../ops/monitoring.md`](../ops/monitoring.md), [`../ops/grafana-dashboard.json`](../ops/grafana-dashboard.json), [`../ops/prometheus-alerts.yaml`](../ops/prometheus-alerts.yaml) |
| Understand the live perf/topology findings (dev handoff) | [`../profiling/m12-cluster-deploy-findings.md`](../profiling/m12-cluster-deploy-findings.md) |

## Current production facts (2026-06-19)

- **Image:** `registry.threesix.ai/tidal/server@sha256:171505745b…` (`m12-writeburst-rc7`).
- **Shape:** 1 StatefulSet `tidaldb`, 3 pods = 3 regions = 3 voters, **full-placement RF3** (every pod hosts all 3 shard groups; gRPC 9601/9602/9603), HTTPS+mTLS on `:9500`, metrics `:9091`.
- **Guarantees live:** quorum-ack writes, automatic election/failover, elastic seed-join (learner→voter), inter-node mTLS, per-node Prometheus.
- **Shipped fixes in this image:** reseed-loop resolved, seed-join promotion, election-divergence/quarantine, read-SLA (CPU oversubscription), and the write-burst false-partition fix.
- **Known limits:** read p99 ≤10 ms to ~1000 rps (ceiling ~1–1.5k/s, CPU-bound); write knee ~250 rps; soak at 200 rps; 1M corpus needs >16 GiB nodes; 2.5× write-scaling needs ≥5 nodes + partitioned placement.
- **Must-watch signal:** `tidaldb_cluster_peer_breaker_state` (0/1/2) — a live peer stuck at 1 is the write-burst/partition pattern.

> Alert rules in `../ops/prometheus-alerts.yaml` are accurate but **design-reference
> — not yet loaded by the live Prometheus.** Promoting them to a `PrometheusRule` is
> the one open observability step (see [`on-call.md`](on-call.md) §3).