# tidalDB On-Call & Incident Response

First page to open during a tidalDB incident. Routes a symptom → the golden signal
that confirms it → the runbook that fixes it. For the operational API see
[`cluster.md`](cluster.md); for k8s mechanics [`kubernetes.md`](kubernetes.md); for
backup/restore [`disaster-recovery.md`](disaster-recovery.md).

**Ownership:** the tidalDB maintainers (source: `git.threesix.ai/jordan/tidaldb`).
The live cluster runs in `orchard9-k3sf` (ns `tidaldb-cluster`); deployment history
is `orchard9-k3sf/deployments/history/tidaldb.md`; live state is
`orchard9-k3sf/cluster-state.yaml`.

---

> ## ⚠ ACTIVE OPERATOR INTERVENTION — read before touching the client Service
>
> **As of 2026-08-21 the client Service `tidaldb` carries a NARROWED selector.**
> `tidaldb-0` serves a shard-1 frontier that was cross-seeded from shard 2's
> snapshot artifact (see `incident_2026_08_20_reseed_livelock` in
> `orchard9-k3sf/cluster-state.yaml`), so its reads are not trustworthy even when
> it reports `lag_events: 0`. It is held out of the read path by an extra
> selector label rather than by readiness, because the marker-clear bug lets it
> report Ready while degraded.
>
> ```bash
> # what is in force
> kubectl -n tidaldb-cluster get svc tidaldb -o jsonpath='{.spec.selector}'
> #   => includes tidaldb.orchard9.ai/serving: "true"
> # only tidaldb-1 and tidaldb-2 carry that label (applied to the PODS, not the template)
> ```
>
> **FOOTGUN — this label does NOT survive pod recreation.** It is on the pods, not
> the StatefulSet template (adding it to the template would trigger a rolling
> update, and with only two healthy voters that loses quorum). If `tidaldb-1` or
> `tidaldb-2` is recreated it silently drops out of the client Service. If BOTH
> are recreated the Service has ZERO endpoints and all reads fail.
>
> ```bash
> # SYMPTOM: reads fail / Service has no endpoints
> kubectl -n tidaldb-cluster get endpoints tidaldb
> # REPAIR: re-label whichever healthy pods lost it
> kubectl -n tidaldb-cluster label pod tidaldb-1 tidaldb-2 tidaldb.orchard9.ai/serving=true --overwrite
> ```
>
> **REVERT once tidaldb-0 genuinely holds every shard** (per-shard `applied_events`
> matching each shard's leader AND a real query agreeing with a healthy peer — a
> frontier number alone is what misled us here):
>
> ```bash
> kubectl -n tidaldb-cluster patch svc tidaldb --type=merge \
>   -p '{"spec":{"selector":{"app.kubernetes.io/name":"tidaldb","app.kubernetes.io/component":"cluster-node"}}}'
> kubectl -n tidaldb-cluster label pod tidaldb-1 tidaldb-2 tidaldb.orchard9.ai/serving-
> ```

---

## 0. First 60 seconds — orient

```bash
export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
kubectl get pods -n tidaldb-cluster -l app.kubernetes.io/name=tidaldb -o wide   # 3/3 Ready? restarts?
# health + leader (HTTPS + mTLS plane), via port-forward:
kubectl port-forward -n tidaldb-cluster tidaldb-0 19500:9500 >/dev/null 2>&1 &
curl -fsSk https://127.0.0.1:19500/health          # ok:true + agreed leader across pods
curl -fsSk https://127.0.0.1:19500/cluster/status  # per-shard leaders, commit index, quarantine, reseed
```

Golden signals (Grafana "tidalDB Overview" → Cluster Replication row, or `/metrics`):

| Signal | Metric | Healthy |
|---|---|---|
| Per-peer circuit breaker | `tidaldb_cluster_peer_breaker_state` | 0 (closed). **1 (open) on a live peer = the write-burst/partition pattern.** |
| Commit progress / quorum lag | `relay_durable_seq` vs `relay_last_seq` | lag bounded, advancing under load |
| Reseed pending | `reseed_required` | 0 |
| Election churn | `leader_changes_total`, `election_term` | flat |
| Quorum timeouts | `quorum_timeouts_total` | flat (a rising rate = writes can't reach quorum) |
| Divergence | `divergence_quarantined` | 0 |

---

## 1. Severity

| SEV | Definition | Examples |
|---|---|---|
| **SEV1** | Writes unavailable or acked data at risk | quorum lost (≥2 voters down or both breakers stuck open), divergent prefix served, total cluster loss |
| **SEV2** | Degraded but serving | one voter down (still 2/3 quorum), read p99 > 10 ms sustained, a pod CrashLooping, write-burst 503s on an old image |
| **SEV3** | No user impact yet | single transient reseed, elevated election term, one breaker briefly open then clears, soak night failed |

---

## 2. Symptom → signal → runbook

| Symptom | Confirm with | Action |
|---|---|---|
| **Pod CrashLooping** ("reseed marker latched") | `kubectl logs`, `reseed_required` | Reseed-loop is resolved on the live image (rc5+); a genuine reseed snapshot-installs and converges lag=0. If looping on an OLD image → roll to the live digest. [`cluster.md` reseed] |
| **Writes 503 / quorum lost** | `/cluster/status` (voters down?), `quorum_timeouts_total` rising | One voter down is survivable (2/3). Both followers down → honest retryable 503 until one returns. Restart/replace the down voter; check node health. [`cluster.md` §9] |
| **Write-burst 503-storm, no self-heal; both breakers stuck at 1 while peers alive** | `tidaldb_cluster_peer_breaker_state == 1` on live (heartbeating) followers | Fixed in **rc7** (write-burst false-partition). If on an older image, **roll to ≥ rc7**. Confirm the gauge clears as load eases. [`cluster.md` §9.1] |
| **Stuck leader / dual-leader view** (leader sees peers partitioned, reads 408) | `/cluster/status` disagrees across pods | Restart the leader pod to force re-election; auto-election converges. [`cluster.md` §9] |
| **Divergence / node serving a stale prefix** | `divergence_quarantined > 0` | The node auto-quarantines (503 readiness) and reseeds without wiping — it never serves a divergent prefix. Let it reseed; verify it rejoins lag=0. [`cluster.md` §9.1] |
| **Read p99 > 10 ms, reads hang** | Grafana read-latency; check load spread | Read-SLA collapse was CPU oversubscription, fixed in rc12 (SEARCH_GATE + parallel scatter + cpu limit 3). Ensure reads are spread round-robin across all 3 pods; check the ~1000-rps read ceiling isn't exceeded. [`capacity-planning.md`] |
| **OOMKilled** | pod `lastState.terminated.reason=OOMKilled` | 1536-D full placement peaks ~1.9 GiB at 100k; **1M needs > 16 GiB nodes** (≈7–8 GB/pod). Do not raise the limit past node headroom — it's a node-size limit. [`capacity-planning.md`] |
| **One pod's PVC lost/corrupt** | pod won't open its data dir | Delete that pod's PVC + pod → it reseeds fresh from the live quorum (snapshot install), converges lag=0. Do NOT object-store-restore for a single-pod loss. [`ops/recovery.md`] |
| **Total cluster loss** | — | Rebuild fresh + `tidalctl restore` each shard before boot. [`disaster-recovery.md`] |
| **Soak night failed** | `kubectl get jobs -n tidaldb-cluster -l app.kubernetes.io/name=tidal-soak` | A FAIL resets the 30-night streak. Read the night's log for the SLO breach (p99 > 150 ms or error > 1%) or an under-load restart. [`cluster.md` §15] |

---

## 3. Alerting status (action required)

The alert rules in [`../ops/prometheus-alerts.yaml`](../ops/prometheus-alerts.yaml)
are **accurate but DESIGN-REFERENCE — they are not yet loaded by the live
Prometheus.** The rules already cover the must-watch signals
(`TidalDBClusterBreakerOpen` on `peer_breaker_state == 1`,
`TidalDBClusterCommitIndexStall`, `TidalDBClusterElectionChurn`,
`TidalDBClusterQuorumTimeouts`, `TidalDBClusterDivergenceQuarantine`,
`TidalDBClusterReseedPending`).

**Open infra step:** promote them into the observability stack as a `PrometheusRule`
(or vmalert rule file) so they actually page. Until then, on-call watches the
Grafana Cluster Replication row manually. This is the one item that is an
observability *deployment*, not a doc fix — track it in `orchard9-k3sf`.

---

## 4. Escalation

1. Page the on-call operator → run §0, classify per §1.
2. SEV1 (acked data at risk / writes down): stop further rollouts, capture
   `/cluster/status` + breaker gauge + `kubectl logs` from all 3 pods, then engage
   the tidalDB maintainers. Do **not** delete PVCs or force-promote under a
   divergence/quorum-loss condition without confirming the data-safety path in
   [`cluster.md`](cluster.md) — the system is designed to quarantine rather than
   serve bad data; let it.
3. Record the incident + resolution in
   `orchard9-k3sf/deployments/history/tidaldb.md` (newest first).