4051077cff
- API.md: document `similar_to`/`region`/`unavailable_shards` on /feed and /search, the new POST /vector_search k-NN probe, and the cluster-node-only routes (/cluster/*, /sharded/*, /hardnegs) - CHANGELOG.md: M12 entries — multi-vector preference + ANN candidate-gen, idle-readiness + TLS scale-up (m12p5/p6), sharded ingestion (m12p4) - ROADMAP.md: mark M11 + M12 COMPLETE; restate the v1.0 bar (30-day-green nightly calendar + Ref-A/k3s throughput re-runs) - prometheus-alerts.yaml: add ship-stall, quorum-lag, divergence-quarantine, reseed-pending, and snapshot-pin-force-drop cluster alerts - check-docs.sh: self-updating milestone-status freshness guard derived from ROADMAP's latest COMPLETE milestone - refresh specs (00-14), ai-lookup, guides, and runbooks to M0-M12
94 lines
6.2 KiB
Markdown
94 lines
6.2 KiB
Markdown
# tidalDB On-Call & Incident Response
|
||
|
||
First page to open during a tidalDB incident. Routes a symptom → the golden signal
|
||
that confirms it → the runbook that fixes it. For the operational API see
|
||
[`cluster.md`](cluster.md); for k8s mechanics [`kubernetes.md`](kubernetes.md); for
|
||
backup/restore [`disaster-recovery.md`](disaster-recovery.md).
|
||
|
||
**Ownership:** the tidalDB maintainers (source: `git.threesix.ai/jordan/tidaldb`).
|
||
The live cluster runs in `orchard9-k3sf` (ns `tidaldb-cluster`); deployment history
|
||
is `orchard9-k3sf/deployments/history/tidaldb.md`; live state is
|
||
`orchard9-k3sf/cluster-state.yaml`.
|
||
|
||
---
|
||
|
||
## 0. First 60 seconds — orient
|
||
|
||
```bash
|
||
export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
|
||
kubectl get pods -n tidaldb-cluster -l app.kubernetes.io/name=tidaldb -o wide # 3/3 Ready? restarts?
|
||
# health + leader (HTTPS + mTLS plane), via port-forward:
|
||
kubectl port-forward -n tidaldb-cluster tidaldb-0 19500:9500 >/dev/null 2>&1 &
|
||
curl -fsSk https://127.0.0.1:19500/health # ok:true + agreed leader across pods
|
||
curl -fsSk https://127.0.0.1:19500/cluster/status # per-shard leaders, commit index, quarantine, reseed
|
||
```
|
||
|
||
Golden signals (Grafana "tidalDB Overview" → Cluster Replication row, or `/metrics`):
|
||
|
||
| Signal | Metric | Healthy |
|
||
|---|---|---|
|
||
| Per-peer circuit breaker | `tidaldb_cluster_peer_breaker_state` | 0 (closed). **1 (open) on a live peer = the write-burst/partition pattern.** |
|
||
| Commit progress / quorum lag | `relay_durable_seq` vs `relay_last_seq` | lag bounded, advancing under load |
|
||
| Reseed pending | `reseed_required` | 0 |
|
||
| Election churn | `leader_changes_total`, `election_term` | flat |
|
||
| Quorum timeouts | `quorum_timeouts_total` | flat (a rising rate = writes can't reach quorum) |
|
||
| Divergence | `divergence_quarantined` | 0 |
|
||
|
||
---
|
||
|
||
## 1. Severity
|
||
|
||
| SEV | Definition | Examples |
|
||
|---|---|---|
|
||
| **SEV1** | Writes unavailable or acked data at risk | quorum lost (≥2 voters down or both breakers stuck open), divergent prefix served, total cluster loss |
|
||
| **SEV2** | Degraded but serving | one voter down (still 2/3 quorum), read p99 > 10 ms sustained, a pod CrashLooping, write-burst 503s on an old image |
|
||
| **SEV3** | No user impact yet | single transient reseed, elevated election term, one breaker briefly open then clears, soak night failed |
|
||
|
||
---
|
||
|
||
## 2. Symptom → signal → runbook
|
||
|
||
| Symptom | Confirm with | Action |
|
||
|---|---|---|
|
||
| **Pod CrashLooping** ("reseed marker latched") | `kubectl logs`, `reseed_required` | Reseed-loop is resolved on the live image (rc5+); a genuine reseed snapshot-installs and converges lag=0. If looping on an OLD image → roll to the live digest. [`cluster.md` reseed] |
|
||
| **Writes 503 / quorum lost** | `/cluster/status` (voters down?), `quorum_timeouts_total` rising | One voter down is survivable (2/3). Both followers down → honest retryable 503 until one returns. Restart/replace the down voter; check node health. [`cluster.md` §9] |
|
||
| **Write-burst 503-storm, no self-heal; both breakers stuck at 1 while peers alive** | `tidaldb_cluster_peer_breaker_state == 1` on live (heartbeating) followers | Fixed in **rc7** (write-burst false-partition). If on an older image, **roll to ≥ rc7**. Confirm the gauge clears as load eases. [`cluster.md` §9.1] |
|
||
| **Stuck leader / dual-leader view** (leader sees peers partitioned, reads 408) | `/cluster/status` disagrees across pods | Restart the leader pod to force re-election; auto-election converges. [`cluster.md` §9] |
|
||
| **Divergence / node serving a stale prefix** | `divergence_quarantined > 0` | The node auto-quarantines (503 readiness) and reseeds without wiping — it never serves a divergent prefix. Let it reseed; verify it rejoins lag=0. [`cluster.md` §9.1] |
|
||
| **Read p99 > 10 ms, reads hang** | Grafana read-latency; check load spread | Read-SLA collapse was CPU oversubscription, fixed in rc12 (SEARCH_GATE + parallel scatter + cpu limit 3). Ensure reads are spread round-robin across all 3 pods; check the ~1000-rps read ceiling isn't exceeded. [`capacity-planning.md`] |
|
||
| **OOMKilled** | pod `lastState.terminated.reason=OOMKilled` | 1536-D full placement peaks ~1.9 GiB at 100k; **1M needs > 16 GiB nodes** (≈7–8 GB/pod). Do not raise the limit past node headroom — it's a node-size limit. [`capacity-planning.md`] |
|
||
| **One pod's PVC lost/corrupt** | pod won't open its data dir | Delete that pod's PVC + pod → it reseeds fresh from the live quorum (snapshot install), converges lag=0. Do NOT object-store-restore for a single-pod loss. [`ops/recovery.md`] |
|
||
| **Total cluster loss** | — | Rebuild fresh + `tidalctl restore` each shard before boot. [`disaster-recovery.md`] |
|
||
| **Soak night failed** | `kubectl get jobs -n tidaldb-cluster -l app.kubernetes.io/name=tidal-soak` | A FAIL resets the 30-night streak. Read the night's log for the SLO breach (p99 > 150 ms or error > 1%) or an under-load restart. [`cluster.md` §15] |
|
||
|
||
---
|
||
|
||
## 3. Alerting status (action required)
|
||
|
||
The alert rules in [`../ops/prometheus-alerts.yaml`](../ops/prometheus-alerts.yaml)
|
||
are **accurate but DESIGN-REFERENCE — they are not yet loaded by the live
|
||
Prometheus.** The rules already cover the must-watch signals
|
||
(`TidalDBClusterBreakerOpen` on `peer_breaker_state == 1`,
|
||
`TidalDBClusterCommitIndexStall`, `TidalDBClusterElectionChurn`,
|
||
`TidalDBClusterQuorumTimeouts`, `TidalDBClusterDivergenceQuarantine`,
|
||
`TidalDBClusterReseedPending`).
|
||
|
||
**Open infra step:** promote them into the observability stack as a `PrometheusRule`
|
||
(or vmalert rule file) so they actually page. Until then, on-call watches the
|
||
Grafana Cluster Replication row manually. This is the one item that is an
|
||
observability *deployment*, not a doc fix — track it in `orchard9-k3sf`.
|
||
|
||
---
|
||
|
||
## 4. Escalation
|
||
|
||
1. Page the on-call operator → run §0, classify per §1.
|
||
2. SEV1 (acked data at risk / writes down): stop further rollouts, capture
|
||
`/cluster/status` + breaker gauge + `kubectl logs` from all 3 pods, then engage
|
||
the tidalDB maintainers. Do **not** delete PVCs or force-promote under a
|
||
divergence/quorum-loss condition without confirming the data-safety path in
|
||
[`cluster.md`](cluster.md) — the system is designed to quarantine rather than
|
||
serve bad data; let it.
|
||
3. Record the incident + resolution in
|
||
`orchard9-k3sf/deployments/history/tidaldb.md` (newest first).
|