jordan/tidaldb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
c22a3b65a6
tidaldb/docs/runbooks
History
jx12n c22a3b65a6 docs: withdraw the pre-release "not ready for production" disclaimer 
M0-M12 are shipped and the HA cluster runs in production on k3s, so the
pre-release disclaimer no longer describes the project. Removes it from the
canonical doc set and corrects the readiness text that had gone stale.

- README.md: replace the "Pre-release / not yet recommended for production"
  banner with a production-ready statement; drop "(experimental)" from the
  cluster status bullet; state the post-1.0 versioning posture (additive in
  minor releases, breaking changes get a documented migration path).
- CLAUDE.md / QUICKSTART.md / docs/guides/server-deployment.md /
  docs/runbooks/cluster.md: same withdrawal; reframe the cluster opt-in as a
  guard against standing up a multi-node fabric by accident rather than a
  readiness warning.
- CHANGELOG.md: record the stability posture under [Unreleased], superseding
  the historical 0.1.0 "no stability guarantees" note (left intact as history).
- k8s/statefulset.yaml: the "NOT production HA, tracked as m8p10" comment was
  stale (m8p10 shipped); point at k8s/cluster/ for the HA deployment instead.

Also corrects text that was factually wrong since m11p3/m11p4: the
multi-process cluster gate, its CLI help, and the served OpenAPI description
all still claimed quorum-ack writes and automatic failure detection did not
exist. They do.

Historical records (docs/reviews/, docs/profiling/, past CHANGELOG entries,
the kubernetes.md rc7 fix note) are left unchanged.

Verified against a running binary, not just the build: the opt-in gate's
refusal message, the startup WARN, /health 200, and the served
/openapi.json description all carry the new text. cargo fmt clean; clippy
-D warnings clean on tidaldb and the tidal-server lib; 1943 engine + 155
server lib tests pass; scripts/check-docs.sh OK.

Claude-Session: https://claude.ai/code/session_01QdqSDw1tUhK1JT9Pb1vryP
2026-07-30 19:03:34 -06:00
..
cluster.md docs: withdraw the pre-release "not ready for production" disclaimer 2026-07-30 19:03:34 -06:00
disaster-recovery.md docs: refresh ops runbooks to the live rc7 / full-placement reality 2026-06-19 19:53:29 -06:00
kubernetes.md docs: refresh ops runbooks to the live rc7 / full-placement reality 2026-06-19 19:53:29 -06:00
on-call.md docs(m12): refresh API, specs, ops, and roadmap to the shipped M12 reality 2026-06-23 21:39:55 -06:00
README.md docs: refresh ops runbooks to the live rc7 / full-placement reality 2026-06-19 19:53:29 -06:00

README.md

tidalDB Runbooks — index

Operational documentation for the cluster deployment (ns tidaldb-cluster, single StatefulSet, full-placement RF3, quorum-ack + automatic election). Live state and history live in the orchard9-k3sf repo (cluster-state.yaml, deployments/history/tidaldb.md).

When you need to… Open
Respond to an incident / page — symptom → signal → fix on-call.md
Operate the cluster — API, failover, partition, membership, scale, rolling upgrade cluster.md
Run it on Kubernetes — deploy, digest-pin, probes, scale up/down, troubleshoot pods kubernetes.md
Back up / restore / DR — object-store export, restore, byte-verify, query-proof, R2, rebuild disaster-recovery.md
Recover a single node / standalone engine — corrupt keyspace, WAL, lock, schema ../ops/recovery.md
Size a deployment — single-node tables + the measured Ref-A cluster envelope ../ops/capacity-planning.md
Read the metrics / wire dashboards & alerts ../ops/monitoring.md, ../ops/grafana-dashboard.json, ../ops/prometheus-alerts.yaml
Understand the live perf/topology findings (dev handoff) ../profiling/m12-cluster-deploy-findings.md

Current production facts (2026-06-19)

  • Image: registry.threesix.ai/tidal/server@sha256:171505745b… (m12-writeburst-rc7).
  • Shape: 1 StatefulSet tidaldb, 3 pods = 3 regions = 3 voters, full-placement RF3 (every pod hosts all 3 shard groups; gRPC 9601/9602/9603), HTTPS+mTLS on :9500, metrics :9091.
  • Guarantees live: quorum-ack writes, automatic election/failover, elastic seed-join (learner→voter), inter-node mTLS, per-node Prometheus.
  • Shipped fixes in this image: reseed-loop resolved, seed-join promotion, election-divergence/quarantine, read-SLA (CPU oversubscription), and the write-burst false-partition fix.
  • Known limits: read p99 ≤10 ms to ~1000 rps (ceiling ~1–1.5k/s, CPU-bound); write knee ~250 rps; soak at 200 rps; 1M corpus needs >16 GiB nodes; 2.5× write-scaling needs ≥5 nodes + partitioned placement.
  • Must-watch signal: tidaldb_cluster_peer_breaker_state (0/1/2) — a live peer stuck at 1 is the write-burst/partition pattern.

Alert rules in ../ops/prometheus-alerts.yaml are accurate but design-reference — not yet loaded by the live Prometheus. Promoting them to a PrometheusRule is the one open observability step (see on-call.md §3).