jordan/tidaldb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25296bcc5b
tidaldb/docs/runbooks
History
jx12n 25296bcc5b docs: refresh ops runbooks to the live rc7 / full-placement reality 
The runbooks had drifted to the retired m8/m11p5 design while all m12 production
reality (topology, perf, fixes, DR) sat only in a profiling doc no operator opens.
This promotes that reality into the runbooks and fixes the contradictions.

Contradictions fixed:
- runbooks/cluster.md: the "NEITHER IS QUORUM-ACKED HA YET" status banner was FALSE
  (quorum-ack + automatic election have been live since m11p3/p4). Rewritten to
  state the deployed reality (single-StatefulSet full-placement RF3, rc7).
- README.md: the cluster section called the HA cluster a "built-in simulated
  cluster / multi-region fabric" demo and showed promote-by-region as failover.
  Rewritten — real quorum HA, automatic failover, /cluster/promote is a maintenance
  verb. Kept the honest caveats (experimental gate, global-signals-only).

Reality promoted into the runbooks:
- Live topology (single STS, ns tidaldb-cluster, 3 voters, full-placement RF3,
  gRPC 9601/9602/9603, HTTPS+mTLS :9500), the five shipped fixes, and the real
  build+digest-pin procedure (cross-compile -> trixie -> amd64 PLATFORM manifest,
  not the index/attestation digest) in cluster.md + kubernetes.md.
- Stale constants: soak ramp 3900 -> 200 rps; cluster grace 60 -> 600s; the
  pre-m12 4.5k/s signal-write perf table annotated + the 1536-D read reality added.
- ops/capacity-planning.md: new "Ref-A 3-node fleet — measured capacity" section
  (read p99/ceiling, ~250 rps write knee, 1M needs >16GiB nodes, pod resources).
- ops/recovery.md: new cluster-recovery routing section + scoped the quiesce-and-
  copy note to standalone (the cluster uses tidalctl + the DR runbook).

New docs:
- runbooks/disaster-recovery.md: the proven S3/R2 backup -> restore -> byte-verify
  -> query-proof procedure, full-cluster rebuild, PITR posture (previously
  undocumented despite being proven against real S3).
- runbooks/on-call.md: incident response — symptom -> golden signal -> runbook,
  severity, escalation, and the open alert-wiring step.
- runbooks/README.md: the runbook index + current production facts.

Open follow-up (infra, not docs): ops/prometheus-alerts.yaml is accurate but
design-reference; promoting it to a live PrometheusRule is the one unwired step.
2026-06-19 19:53:29 -06:00
..
cluster.md docs: refresh ops runbooks to the live rc7 / full-placement reality 2026-06-19 19:53:29 -06:00
disaster-recovery.md docs: refresh ops runbooks to the live rc7 / full-placement reality 2026-06-19 19:53:29 -06:00
kubernetes.md docs: refresh ops runbooks to the live rc7 / full-placement reality 2026-06-19 19:53:29 -06:00
on-call.md docs: refresh ops runbooks to the live rc7 / full-placement reality 2026-06-19 19:53:29 -06:00
README.md docs: refresh ops runbooks to the live rc7 / full-placement reality 2026-06-19 19:53:29 -06:00

README.md

tidalDB Runbooks — index

Operational documentation for the cluster deployment (ns tidaldb-cluster, single StatefulSet, full-placement RF3, quorum-ack + automatic election). Live state and history live in the orchard9-k3sf repo (cluster-state.yaml, deployments/history/tidaldb.md).

When you need to… Open
Respond to an incident / page — symptom → signal → fix on-call.md
Operate the cluster — API, failover, partition, membership, scale, rolling upgrade cluster.md
Run it on Kubernetes — deploy, digest-pin, probes, scale up/down, troubleshoot pods kubernetes.md
Back up / restore / DR — object-store export, restore, byte-verify, query-proof, R2, rebuild disaster-recovery.md
Recover a single node / standalone engine — corrupt keyspace, WAL, lock, schema ../ops/recovery.md
Size a deployment — single-node tables + the measured Ref-A cluster envelope ../ops/capacity-planning.md
Read the metrics / wire dashboards & alerts ../ops/monitoring.md, ../ops/grafana-dashboard.json, ../ops/prometheus-alerts.yaml
Understand the live perf/topology findings (dev handoff) ../profiling/m12-cluster-deploy-findings.md

Current production facts (2026-06-19)

  • Image: registry.threesix.ai/tidal/server@sha256:171505745b… (m12-writeburst-rc7).
  • Shape: 1 StatefulSet tidaldb, 3 pods = 3 regions = 3 voters, full-placement RF3 (every pod hosts all 3 shard groups; gRPC 9601/9602/9603), HTTPS+mTLS on :9500, metrics :9091.
  • Guarantees live: quorum-ack writes, automatic election/failover, elastic seed-join (learner→voter), inter-node mTLS, per-node Prometheus.
  • Shipped fixes in this image: reseed-loop resolved, seed-join promotion, election-divergence/quarantine, read-SLA (CPU oversubscription), and the write-burst false-partition fix.
  • Known limits: read p99 ≤10 ms to ~1000 rps (ceiling ~1–1.5k/s, CPU-bound); write knee ~250 rps; soak at 200 rps; 1M corpus needs >16 GiB nodes; 2.5× write-scaling needs ≥5 nodes + partitioned placement.
  • Must-watch signal: tidaldb_cluster_peer_breaker_state (0/1/2) — a live peer stuck at 1 is the write-burst/partition pattern.

Alert rules in ../ops/prometheus-alerts.yaml are accurate but design-reference — not yet loaded by the live Prometheus. Promoting them to a PrometheusRule is the one open observability step (see on-call.md §3).