25296bcc5b
The runbooks had drifted to the retired m8/m11p5 design while all m12 production reality (topology, perf, fixes, DR) sat only in a profiling doc no operator opens. This promotes that reality into the runbooks and fixes the contradictions. Contradictions fixed: - runbooks/cluster.md: the "NEITHER IS QUORUM-ACKED HA YET" status banner was FALSE (quorum-ack + automatic election have been live since m11p3/p4). Rewritten to state the deployed reality (single-StatefulSet full-placement RF3, rc7). - README.md: the cluster section called the HA cluster a "built-in simulated cluster / multi-region fabric" demo and showed promote-by-region as failover. Rewritten — real quorum HA, automatic failover, /cluster/promote is a maintenance verb. Kept the honest caveats (experimental gate, global-signals-only). Reality promoted into the runbooks: - Live topology (single STS, ns tidaldb-cluster, 3 voters, full-placement RF3, gRPC 9601/9602/9603, HTTPS+mTLS :9500), the five shipped fixes, and the real build+digest-pin procedure (cross-compile -> trixie -> amd64 PLATFORM manifest, not the index/attestation digest) in cluster.md + kubernetes.md. - Stale constants: soak ramp 3900 -> 200 rps; cluster grace 60 -> 600s; the pre-m12 4.5k/s signal-write perf table annotated + the 1536-D read reality added. - ops/capacity-planning.md: new "Ref-A 3-node fleet — measured capacity" section (read p99/ceiling, ~250 rps write knee, 1M needs >16GiB nodes, pod resources). - ops/recovery.md: new cluster-recovery routing section + scoped the quiesce-and- copy note to standalone (the cluster uses tidalctl + the DR runbook). New docs: - runbooks/disaster-recovery.md: the proven S3/R2 backup -> restore -> byte-verify -> query-proof procedure, full-cluster rebuild, PITR posture (previously undocumented despite being proven against real S3). - runbooks/on-call.md: incident response — symptom -> golden signal -> runbook, severity, escalation, and the open alert-wiring step. - runbooks/README.md: the runbook index + current production facts. Open follow-up (infra, not docs): ops/prometheus-alerts.yaml is accurate but design-reference; promoting it to a live PrometheusRule is the one unwired step.
93 lines
6.2 KiB
Markdown
93 lines
6.2 KiB
Markdown
# tidalDB On-Call & Incident Response
|
||
|
||
First page to open during a tidalDB incident. Routes a symptom → the golden signal
|
||
that confirms it → the runbook that fixes it. For the operational API see
|
||
[`cluster.md`](cluster.md); for k8s mechanics [`kubernetes.md`](kubernetes.md); for
|
||
backup/restore [`disaster-recovery.md`](disaster-recovery.md).
|
||
|
||
**Ownership:** the tidalDB maintainers (source: `git.threesix.ai/jordan/tidaldb`).
|
||
The live cluster runs in `orchard9-k3sf` (ns `tidaldb-cluster`); deployment history
|
||
is `orchard9-k3sf/deployments/history/tidaldb.md`; live state is
|
||
`orchard9-k3sf/cluster-state.yaml`.
|
||
|
||
---
|
||
|
||
## 0. First 60 seconds — orient
|
||
|
||
```bash
|
||
export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
|
||
kubectl get pods -n tidaldb-cluster -l app.kubernetes.io/name=tidaldb -o wide # 3/3 Ready? restarts?
|
||
# health + leader (HTTPS + mTLS plane), via port-forward:
|
||
kubectl port-forward -n tidaldb-cluster tidaldb-0 19500:9500 >/dev/null 2>&1 &
|
||
curl -fsSk https://127.0.0.1:19500/health # ok:true + agreed leader across pods
|
||
curl -fsSk https://127.0.0.1:19500/cluster/status # per-shard leaders, commit index, quarantine, reseed
|
||
```
|
||
|
||
Golden signals (Grafana "tidalDB Overview" → Cluster Replication row, or `/metrics`):
|
||
|
||
| Signal | Metric | Healthy |
|
||
|---|---|---|
|
||
| Per-peer circuit breaker | `tidaldb_cluster_peer_breaker_state` | 0 (closed). **1 (open) on a live peer = the write-burst/partition pattern.** |
|
||
| Commit progress / quorum lag | `relay_durable_seq` vs `relay_last_seq` | lag bounded, advancing under load |
|
||
| Reseed pending | `reseed_required` | 0 |
|
||
| Election churn | `leader_changes_total`, `election_term` | flat |
|
||
| Quorum timeouts | `quorum_timeouts_total` | flat (a rising rate = writes can't reach quorum) |
|
||
| Divergence | `divergence_quarantined` | 0 |
|
||
|
||
---
|
||
|
||
## 1. Severity
|
||
|
||
| SEV | Definition | Examples |
|
||
|---|---|---|
|
||
| **SEV1** | Writes unavailable or acked data at risk | quorum lost (≥2 voters down or both breakers stuck open), divergent prefix served, total cluster loss |
|
||
| **SEV2** | Degraded but serving | one voter down (still 2/3 quorum), read p99 > 10 ms sustained, a pod CrashLooping, write-burst 503s on an old image |
|
||
| **SEV3** | No user impact yet | single transient reseed, elevated election term, one breaker briefly open then clears, soak night failed |
|
||
|
||
---
|
||
|
||
## 2. Symptom → signal → runbook
|
||
|
||
| Symptom | Confirm with | Action |
|
||
|---|---|---|
|
||
| **Pod CrashLooping** ("reseed marker latched") | `kubectl logs`, `reseed_required` | Reseed-loop is resolved on the live image (rc5+); a genuine reseed snapshot-installs and converges lag=0. If looping on an OLD image → roll to the live digest. [`cluster.md` reseed] |
|
||
| **Writes 503 / quorum lost** | `/cluster/status` (voters down?), `quorum_timeouts_total` rising | One voter down is survivable (2/3). Both followers down → honest retryable 503 until one returns. Restart/replace the down voter; check node health. [`cluster.md` §9] |
|
||
| **Write-burst 503-storm, no self-heal; both breakers stuck at 1 while peers alive** | `tidaldb_cluster_peer_breaker_state == 1` on live (heartbeating) followers | Fixed in **rc7** (write-burst false-partition). If on an older image, **roll to ≥ rc7**. Confirm the gauge clears as load eases. [`cluster.md` §9.1] |
|
||
| **Stuck leader / dual-leader view** (leader sees peers partitioned, reads 408) | `/cluster/status` disagrees across pods | Restart the leader pod to force re-election; auto-election converges. [`cluster.md` §9] |
|
||
| **Divergence / node serving a stale prefix** | `divergence_quarantined > 0` | The node auto-quarantines (503 readiness) and reseeds without wiping — it never serves a divergent prefix. Let it reseed; verify it rejoins lag=0. [`cluster.md` §9.1] |
|
||
| **Read p99 > 10 ms, reads hang** | Grafana read-latency; check load spread | Read-SLA collapse was CPU oversubscription, fixed in rc12 (SEARCH_GATE + parallel scatter + cpu limit 3). Ensure reads are spread round-robin across all 3 pods; check the ~1000-rps read ceiling isn't exceeded. [`capacity-planning.md`] |
|
||
| **OOMKilled** | pod `lastState.terminated.reason=OOMKilled` | 1536-D full placement peaks ~1.9 GiB at 100k; **1M needs > 16 GiB nodes** (≈7–8 GB/pod). Do not raise the limit past node headroom — it's a node-size limit. [`capacity-planning.md`] |
|
||
| **One pod's PVC lost/corrupt** | pod won't open its data dir | Delete that pod's PVC + pod → it reseeds fresh from the live quorum (snapshot install), converges lag=0. Do NOT object-store-restore for a single-pod loss. [`ops/recovery.md`] |
|
||
| **Total cluster loss** | — | Rebuild fresh + `tidalctl restore` each shard before boot. [`disaster-recovery.md`] |
|
||
| **Soak night failed** | `kubectl get jobs -n tidaldb-cluster -l app.kubernetes.io/name=tidal-soak` | A FAIL resets the 30-night streak. Read the night's log for the SLO breach (p99 > 150 ms or error > 1%) or an under-load restart. [`cluster.md` §15] |
|
||
|
||
---
|
||
|
||
## 3. Alerting status (action required)
|
||
|
||
The alert rules in [`../ops/prometheus-alerts.yaml`](../ops/prometheus-alerts.yaml)
|
||
are **accurate but DESIGN-REFERENCE — they are not yet loaded by the live
|
||
Prometheus.** The rules already cover the must-watch signals
|
||
(`TidalDBClusterBreakerOpen` on `peer_breaker_state == 1`,
|
||
`TidalDBClusterCommitIndexStall`, `TidalDBClusterElectionChurn`,
|
||
`TidalDBClusterQuorumTimeouts`, `DivergenceQuarantine`, `ReseedPending`).
|
||
|
||
**Open infra step:** promote them into the observability stack as a `PrometheusRule`
|
||
(or vmalert rule file) so they actually page. Until then, on-call watches the
|
||
Grafana Cluster Replication row manually. This is the one item that is an
|
||
observability *deployment*, not a doc fix — track it in `orchard9-k3sf`.
|
||
|
||
---
|
||
|
||
## 4. Escalation
|
||
|
||
1. Page the on-call operator → run §0, classify per §1.
|
||
2. SEV1 (acked data at risk / writes down): stop further rollouts, capture
|
||
`/cluster/status` + breaker gauge + `kubectl logs` from all 3 pods, then engage
|
||
the tidalDB maintainers. Do **not** delete PVCs or force-promote under a
|
||
divergence/quorum-loss condition without confirming the data-safety path in
|
||
[`cluster.md`](cluster.md) — the system is designed to quarantine rather than
|
||
serve bad data; let it.
|
||
3. Record the incident + resolution in
|
||
`orchard9-k3sf/deployments/history/tidaldb.md` (newest first).
|