jordan/tidaldb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
4051077cff
tidaldb/docs/runbooks
History
jx12n 4051077cff docs(m12): refresh API, specs, ops, and roadmap to the shipped M12 reality 
- API.md: document `similar_to`/`region`/`unavailable_shards` on /feed and
  /search, the new POST /vector_search k-NN probe, and the cluster-node-only
  routes (/cluster/*, /sharded/*, /hardnegs)
- CHANGELOG.md: M12 entries — multi-vector preference + ANN candidate-gen,
  idle-readiness + TLS scale-up (m12p5/p6), sharded ingestion (m12p4)
- ROADMAP.md: mark M11 + M12 COMPLETE; restate the v1.0 bar (30-day-green
  nightly calendar + Ref-A/k3s throughput re-runs)
- prometheus-alerts.yaml: add ship-stall, quorum-lag, divergence-quarantine,
  reseed-pending, and snapshot-pin-force-drop cluster alerts
- check-docs.sh: self-updating milestone-status freshness guard derived from
  ROADMAP's latest COMPLETE milestone
- refresh specs (00-14), ai-lookup, guides, and runbooks to M0-M12
2026-06-23 21:39:55 -06:00
..
cluster.md docs: refresh ops runbooks to the live rc7 / full-placement reality 2026-06-19 19:53:29 -06:00
disaster-recovery.md docs: refresh ops runbooks to the live rc7 / full-placement reality 2026-06-19 19:53:29 -06:00
kubernetes.md docs: refresh ops runbooks to the live rc7 / full-placement reality 2026-06-19 19:53:29 -06:00
on-call.md docs(m12): refresh API, specs, ops, and roadmap to the shipped M12 reality 2026-06-23 21:39:55 -06:00
README.md docs: refresh ops runbooks to the live rc7 / full-placement reality 2026-06-19 19:53:29 -06:00

README.md

tidalDB Runbooks — index

Operational documentation for the cluster deployment (ns tidaldb-cluster, single StatefulSet, full-placement RF3, quorum-ack + automatic election). Live state and history live in the orchard9-k3sf repo (cluster-state.yaml, deployments/history/tidaldb.md).

When you need to… Open
Respond to an incident / page — symptom → signal → fix on-call.md
Operate the cluster — API, failover, partition, membership, scale, rolling upgrade cluster.md
Run it on Kubernetes — deploy, digest-pin, probes, scale up/down, troubleshoot pods kubernetes.md
Back up / restore / DR — object-store export, restore, byte-verify, query-proof, R2, rebuild disaster-recovery.md
Recover a single node / standalone engine — corrupt keyspace, WAL, lock, schema ../ops/recovery.md
Size a deployment — single-node tables + the measured Ref-A cluster envelope ../ops/capacity-planning.md
Read the metrics / wire dashboards & alerts ../ops/monitoring.md, ../ops/grafana-dashboard.json, ../ops/prometheus-alerts.yaml
Understand the live perf/topology findings (dev handoff) ../profiling/m12-cluster-deploy-findings.md

Current production facts (2026-06-19)

  • Image: registry.threesix.ai/tidal/server@sha256:171505745b… (m12-writeburst-rc7).
  • Shape: 1 StatefulSet tidaldb, 3 pods = 3 regions = 3 voters, full-placement RF3 (every pod hosts all 3 shard groups; gRPC 9601/9602/9603), HTTPS+mTLS on :9500, metrics :9091.
  • Guarantees live: quorum-ack writes, automatic election/failover, elastic seed-join (learner→voter), inter-node mTLS, per-node Prometheus.
  • Shipped fixes in this image: reseed-loop resolved, seed-join promotion, election-divergence/quarantine, read-SLA (CPU oversubscription), and the write-burst false-partition fix.
  • Known limits: read p99 ≤10 ms to ~1000 rps (ceiling ~1–1.5k/s, CPU-bound); write knee ~250 rps; soak at 200 rps; 1M corpus needs >16 GiB nodes; 2.5× write-scaling needs ≥5 nodes + partitioned placement.
  • Must-watch signal: tidaldb_cluster_peer_breaker_state (0/1/2) — a live peer stuck at 1 is the write-burst/partition pattern.

Alert rules in ../ops/prometheus-alerts.yaml are accurate but design-reference — not yet loaded by the live Prometheus. Promoting them to a PrometheusRule is the one open observability step (see on-call.md §3).