25296bcc5b
The runbooks had drifted to the retired m8/m11p5 design while all m12 production reality (topology, perf, fixes, DR) sat only in a profiling doc no operator opens. This promotes that reality into the runbooks and fixes the contradictions. Contradictions fixed: - runbooks/cluster.md: the "NEITHER IS QUORUM-ACKED HA YET" status banner was FALSE (quorum-ack + automatic election have been live since m11p3/p4). Rewritten to state the deployed reality (single-StatefulSet full-placement RF3, rc7). - README.md: the cluster section called the HA cluster a "built-in simulated cluster / multi-region fabric" demo and showed promote-by-region as failover. Rewritten — real quorum HA, automatic failover, /cluster/promote is a maintenance verb. Kept the honest caveats (experimental gate, global-signals-only). Reality promoted into the runbooks: - Live topology (single STS, ns tidaldb-cluster, 3 voters, full-placement RF3, gRPC 9601/9602/9603, HTTPS+mTLS :9500), the five shipped fixes, and the real build+digest-pin procedure (cross-compile -> trixie -> amd64 PLATFORM manifest, not the index/attestation digest) in cluster.md + kubernetes.md. - Stale constants: soak ramp 3900 -> 200 rps; cluster grace 60 -> 600s; the pre-m12 4.5k/s signal-write perf table annotated + the 1536-D read reality added. - ops/capacity-planning.md: new "Ref-A 3-node fleet — measured capacity" section (read p99/ceiling, ~250 rps write knee, 1M needs >16GiB nodes, pod resources). - ops/recovery.md: new cluster-recovery routing section + scoped the quiesce-and- copy note to standalone (the cluster uses tidalctl + the DR runbook). New docs: - runbooks/disaster-recovery.md: the proven S3/R2 backup -> restore -> byte-verify -> query-proof procedure, full-cluster rebuild, PITR posture (previously undocumented despite being proven against real S3). - runbooks/on-call.md: incident response — symptom -> golden signal -> runbook, severity, escalation, and the open alert-wiring step. - runbooks/README.md: the runbook index + current production facts. Open follow-up (infra, not docs): ops/prometheus-alerts.yaml is accurate but design-reference; promoting it to a live PrometheusRule is the one unwired step.
31 lines
2.7 KiB
Markdown
31 lines
2.7 KiB
Markdown
# tidalDB Runbooks — index
|
||
|
||
Operational documentation for the **cluster** deployment (ns `tidaldb-cluster`,
|
||
single StatefulSet, full-placement RF3, quorum-ack + automatic election). Live state
|
||
and history live in the `orchard9-k3sf` repo (`cluster-state.yaml`,
|
||
`deployments/history/tidaldb.md`).
|
||
|
||
| When you need to… | Open |
|
||
|---|---|
|
||
| **Respond to an incident / page** — symptom → signal → fix | [`on-call.md`](on-call.md) |
|
||
| Operate the cluster — API, failover, partition, membership, scale, rolling upgrade | [`cluster.md`](cluster.md) |
|
||
| Run it on Kubernetes — deploy, digest-pin, probes, scale up/down, troubleshoot pods | [`kubernetes.md`](kubernetes.md) |
|
||
| Back up / restore / DR — object-store export, restore, byte-verify, query-proof, R2, rebuild | [`disaster-recovery.md`](disaster-recovery.md) |
|
||
| Recover a single node / standalone engine — corrupt keyspace, WAL, lock, schema | [`../ops/recovery.md`](../ops/recovery.md) |
|
||
| Size a deployment — single-node tables + the measured Ref-A cluster envelope | [`../ops/capacity-planning.md`](../ops/capacity-planning.md) |
|
||
| Read the metrics / wire dashboards & alerts | [`../ops/monitoring.md`](../ops/monitoring.md), [`../ops/grafana-dashboard.json`](../ops/grafana-dashboard.json), [`../ops/prometheus-alerts.yaml`](../ops/prometheus-alerts.yaml) |
|
||
| Understand the live perf/topology findings (dev handoff) | [`../profiling/m12-cluster-deploy-findings.md`](../profiling/m12-cluster-deploy-findings.md) |
|
||
|
||
## Current production facts (2026-06-19)
|
||
|
||
- **Image:** `registry.threesix.ai/tidal/server@sha256:171505745b…` (`m12-writeburst-rc7`).
|
||
- **Shape:** 1 StatefulSet `tidaldb`, 3 pods = 3 regions = 3 voters, **full-placement RF3** (every pod hosts all 3 shard groups; gRPC 9601/9602/9603), HTTPS+mTLS on `:9500`, metrics `:9091`.
|
||
- **Guarantees live:** quorum-ack writes, automatic election/failover, elastic seed-join (learner→voter), inter-node mTLS, per-node Prometheus.
|
||
- **Shipped fixes in this image:** reseed-loop resolved, seed-join promotion, election-divergence/quarantine, read-SLA (CPU oversubscription), and the write-burst false-partition fix.
|
||
- **Known limits:** read p99 ≤10 ms to ~1000 rps (ceiling ~1–1.5k/s, CPU-bound); write knee ~250 rps; soak at 200 rps; 1M corpus needs >16 GiB nodes; 2.5× write-scaling needs ≥5 nodes + partitioned placement.
|
||
- **Must-watch signal:** `tidaldb_cluster_peer_breaker_state` (0/1/2) — a live peer stuck at 1 is the write-burst/partition pattern.
|
||
|
||
> Alert rules in `../ops/prometheus-alerts.yaml` are accurate but **design-reference
|
||
> — not yet loaded by the live Prometheus.** Promoting them to a `PrometheusRule` is
|
||
> the one open observability step (see [`on-call.md`](on-call.md) §3).
|