jordan/tidaldb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
fab5467b8f
tidaldb/docs/runbooks
History
jordan c58b18b994 docs(ops): un-blind the reseed alert and record the active VIP intervention 
The 2026-08-20 livelock ran 21h with no page. TidalDBClusterReseedPending is
`tidaldb_cluster_reseed_required == 1 for 10m`, written for exactly this, but the
defect cleared the marker ~200ms after each latch, so the gauge flapped 1->0
every ~30s and never held 1 for 10m. Add TidalDBClusterReseedFlapping, which
keys off `changes(...[15m]) > 2` instead of a hold duration, so a latch/clear
loop pages even when the gauge reads 0 at both ends of the window and even if a
future clear path reintroduces the spurious clear.

Also add an on-call banner for the operator intervention now in force: the
client Service selector is narrowed to keep tidaldb-0 out of the read path,
because its shard-1 frontier was cross-seeded from shard 2's snapshot artifact
and its reads are untrustworthy even at lag_events: 0. The banner carries the
footgun (the label is on the pods, not the template, so it does not survive pod
recreation) plus repair and revert commands.
2026-08-20 22:32:50 -06:00
..
cluster.md fleet remediation: make the workspace gate runnable, then fix what it caught 2026-08-16 12:38:14 -06:00
disaster-recovery.md docs: refresh ops runbooks to the live rc7 / full-placement reality 2026-06-19 19:53:29 -06:00
kubernetes.md docs: refresh ops runbooks to the live rc7 / full-placement reality 2026-06-19 19:53:29 -06:00
on-call.md docs(ops): un-blind the reseed alert and record the active VIP intervention 2026-08-20 22:32:50 -06:00
README.md docs: refresh ops runbooks to the live rc7 / full-placement reality 2026-06-19 19:53:29 -06:00

README.md

tidalDB Runbooks — index

Operational documentation for the cluster deployment (ns tidaldb-cluster, single StatefulSet, full-placement RF3, quorum-ack + automatic election). Live state and history live in the orchard9-k3sf repo (cluster-state.yaml, deployments/history/tidaldb.md).

When you need to… Open
Respond to an incident / page — symptom → signal → fix on-call.md
Operate the cluster — API, failover, partition, membership, scale, rolling upgrade cluster.md
Run it on Kubernetes — deploy, digest-pin, probes, scale up/down, troubleshoot pods kubernetes.md
Back up / restore / DR — object-store export, restore, byte-verify, query-proof, R2, rebuild disaster-recovery.md
Recover a single node / standalone engine — corrupt keyspace, WAL, lock, schema ../ops/recovery.md
Size a deployment — single-node tables + the measured Ref-A cluster envelope ../ops/capacity-planning.md
Read the metrics / wire dashboards & alerts ../ops/monitoring.md, ../ops/grafana-dashboard.json, ../ops/prometheus-alerts.yaml
Understand the live perf/topology findings (dev handoff) ../profiling/m12-cluster-deploy-findings.md

Current production facts (2026-06-19)

  • Image: registry.threesix.ai/tidal/server@sha256:171505745b… (m12-writeburst-rc7).
  • Shape: 1 StatefulSet tidaldb, 3 pods = 3 regions = 3 voters, full-placement RF3 (every pod hosts all 3 shard groups; gRPC 9601/9602/9603), HTTPS+mTLS on :9500, metrics :9091.
  • Guarantees live: quorum-ack writes, automatic election/failover, elastic seed-join (learner→voter), inter-node mTLS, per-node Prometheus.
  • Shipped fixes in this image: reseed-loop resolved, seed-join promotion, election-divergence/quarantine, read-SLA (CPU oversubscription), and the write-burst false-partition fix.
  • Known limits: read p99 ≤10 ms to ~1000 rps (ceiling ~1–1.5k/s, CPU-bound); write knee ~250 rps; soak at 200 rps; 1M corpus needs >16 GiB nodes; 2.5× write-scaling needs ≥5 nodes + partitioned placement.
  • Must-watch signal: tidaldb_cluster_peer_breaker_state (0/1/2) — a live peer stuck at 1 is the write-burst/partition pattern.

Alert rules in ../ops/prometheus-alerts.yaml are accurate but design-reference — not yet loaded by the live Prometheus. Promoting them to a PrometheusRule is the one open observability step (see on-call.md §3).