c22a3b65a6
M0-M12 are shipped and the HA cluster runs in production on k3s, so the pre-release disclaimer no longer describes the project. Removes it from the canonical doc set and corrects the readiness text that had gone stale. - README.md: replace the "Pre-release / not yet recommended for production" banner with a production-ready statement; drop "(experimental)" from the cluster status bullet; state the post-1.0 versioning posture (additive in minor releases, breaking changes get a documented migration path). - CLAUDE.md / QUICKSTART.md / docs/guides/server-deployment.md / docs/runbooks/cluster.md: same withdrawal; reframe the cluster opt-in as a guard against standing up a multi-node fabric by accident rather than a readiness warning. - CHANGELOG.md: record the stability posture under [Unreleased], superseding the historical 0.1.0 "no stability guarantees" note (left intact as history). - k8s/statefulset.yaml: the "NOT production HA, tracked as m8p10" comment was stale (m8p10 shipped); point at k8s/cluster/ for the HA deployment instead. Also corrects text that was factually wrong since m11p3/m11p4: the multi-process cluster gate, its CLI help, and the served OpenAPI description all still claimed quorum-ack writes and automatic failure detection did not exist. They do. Historical records (docs/reviews/, docs/profiling/, past CHANGELOG entries, the kubernetes.md rc7 fix note) are left unchanged. Verified against a running binary, not just the build: the opt-in gate's refusal message, the startup WARN, /health 200, and the served /openapi.json description all carry the new text. cargo fmt clean; clippy -D warnings clean on tidaldb and the tidal-server lib; 1943 engine + 155 server lib tests pass; scripts/check-docs.sh OK. Claude-Session: https://claude.ai/code/session_01QdqSDw1tUhK1JT9Pb1vryP |
||
|---|---|---|
| .. | ||
| cluster.md | ||
| disaster-recovery.md | ||
| kubernetes.md | ||
| on-call.md | ||
| README.md | ||
tidalDB Runbooks — index
Operational documentation for the cluster deployment (ns tidaldb-cluster,
single StatefulSet, full-placement RF3, quorum-ack + automatic election). Live state
and history live in the orchard9-k3sf repo (cluster-state.yaml,
deployments/history/tidaldb.md).
| When you need to… | Open |
|---|---|
| Respond to an incident / page — symptom → signal → fix | on-call.md |
| Operate the cluster — API, failover, partition, membership, scale, rolling upgrade | cluster.md |
| Run it on Kubernetes — deploy, digest-pin, probes, scale up/down, troubleshoot pods | kubernetes.md |
| Back up / restore / DR — object-store export, restore, byte-verify, query-proof, R2, rebuild | disaster-recovery.md |
| Recover a single node / standalone engine — corrupt keyspace, WAL, lock, schema | ../ops/recovery.md |
| Size a deployment — single-node tables + the measured Ref-A cluster envelope | ../ops/capacity-planning.md |
| Read the metrics / wire dashboards & alerts | ../ops/monitoring.md, ../ops/grafana-dashboard.json, ../ops/prometheus-alerts.yaml |
| Understand the live perf/topology findings (dev handoff) | ../profiling/m12-cluster-deploy-findings.md |
Current production facts (2026-06-19)
- Image:
registry.threesix.ai/tidal/server@sha256:171505745b…(m12-writeburst-rc7). - Shape: 1 StatefulSet
tidaldb, 3 pods = 3 regions = 3 voters, full-placement RF3 (every pod hosts all 3 shard groups; gRPC 9601/9602/9603), HTTPS+mTLS on:9500, metrics:9091. - Guarantees live: quorum-ack writes, automatic election/failover, elastic seed-join (learner→voter), inter-node mTLS, per-node Prometheus.
- Shipped fixes in this image: reseed-loop resolved, seed-join promotion, election-divergence/quarantine, read-SLA (CPU oversubscription), and the write-burst false-partition fix.
- Known limits: read p99 ≤10 ms to ~1000 rps (ceiling ~1–1.5k/s, CPU-bound); write knee ~250 rps; soak at 200 rps; 1M corpus needs >16 GiB nodes; 2.5× write-scaling needs ≥5 nodes + partitioned placement.
- Must-watch signal:
tidaldb_cluster_peer_breaker_state(0/1/2) — a live peer stuck at 1 is the write-burst/partition pattern.
Alert rules in
../ops/prometheus-alerts.yamlare accurate but design-reference — not yet loaded by the live Prometheus. Promoting them to aPrometheusRuleis the one open observability step (seeon-call.md§3).