c58b18b994
The 2026-08-20 livelock ran 21h with no page. TidalDBClusterReseedPending is `tidaldb_cluster_reseed_required == 1 for 10m`, written for exactly this, but the defect cleared the marker ~200ms after each latch, so the gauge flapped 1->0 every ~30s and never held 1 for 10m. Add TidalDBClusterReseedFlapping, which keys off `changes(...[15m]) > 2` instead of a hold duration, so a latch/clear loop pages even when the gauge reads 0 at both ends of the window and even if a future clear path reintroduces the spurious clear. Also add an on-call banner for the operator intervention now in force: the client Service selector is narrowed to keep tidaldb-0 out of the read path, because its shard-1 frontier was cross-seeded from shard 2's snapshot artifact and its reads are untrustworthy even at lag_events: 0. The banner carries the footgun (the label is on the pods, not the template, so it does not survive pod recreation) plus repair and revert commands.
136 lines
8.2 KiB
Markdown
136 lines
8.2 KiB
Markdown
# tidalDB On-Call & Incident Response
|
||
|
||
First page to open during a tidalDB incident. Routes a symptom → the golden signal
|
||
that confirms it → the runbook that fixes it. For the operational API see
|
||
[`cluster.md`](cluster.md); for k8s mechanics [`kubernetes.md`](kubernetes.md); for
|
||
backup/restore [`disaster-recovery.md`](disaster-recovery.md).
|
||
|
||
**Ownership:** the tidalDB maintainers (source: `git.threesix.ai/jordan/tidaldb`).
|
||
The live cluster runs in `orchard9-k3sf` (ns `tidaldb-cluster`); deployment history
|
||
is `orchard9-k3sf/deployments/history/tidaldb.md`; live state is
|
||
`orchard9-k3sf/cluster-state.yaml`.
|
||
|
||
---
|
||
|
||
> ## ⚠ ACTIVE OPERATOR INTERVENTION — read before touching the client Service
|
||
>
|
||
> **As of 2026-08-21 the client Service `tidaldb` carries a NARROWED selector.**
|
||
> `tidaldb-0` serves a shard-1 frontier that was cross-seeded from shard 2's
|
||
> snapshot artifact (see `incident_2026_08_20_reseed_livelock` in
|
||
> `orchard9-k3sf/cluster-state.yaml`), so its reads are not trustworthy even when
|
||
> it reports `lag_events: 0`. It is held out of the read path by an extra
|
||
> selector label rather than by readiness, because the marker-clear bug lets it
|
||
> report Ready while degraded.
|
||
>
|
||
> ```bash
|
||
> # what is in force
|
||
> kubectl -n tidaldb-cluster get svc tidaldb -o jsonpath='{.spec.selector}'
|
||
> # => includes tidaldb.orchard9.ai/serving: "true"
|
||
> # only tidaldb-1 and tidaldb-2 carry that label (applied to the PODS, not the template)
|
||
> ```
|
||
>
|
||
> **FOOTGUN — this label does NOT survive pod recreation.** It is on the pods, not
|
||
> the StatefulSet template (adding it to the template would trigger a rolling
|
||
> update, and with only two healthy voters that loses quorum). If `tidaldb-1` or
|
||
> `tidaldb-2` is recreated it silently drops out of the client Service. If BOTH
|
||
> are recreated the Service has ZERO endpoints and all reads fail.
|
||
>
|
||
> ```bash
|
||
> # SYMPTOM: reads fail / Service has no endpoints
|
||
> kubectl -n tidaldb-cluster get endpoints tidaldb
|
||
> # REPAIR: re-label whichever healthy pods lost it
|
||
> kubectl -n tidaldb-cluster label pod tidaldb-1 tidaldb-2 tidaldb.orchard9.ai/serving=true --overwrite
|
||
> ```
|
||
>
|
||
> **REVERT once tidaldb-0 genuinely holds every shard** (per-shard `applied_events`
|
||
> matching each shard's leader AND a real query agreeing with a healthy peer — a
|
||
> frontier number alone is what misled us here):
|
||
>
|
||
> ```bash
|
||
> kubectl -n tidaldb-cluster patch svc tidaldb --type=merge \
|
||
> -p '{"spec":{"selector":{"app.kubernetes.io/name":"tidaldb","app.kubernetes.io/component":"cluster-node"}}}'
|
||
> kubectl -n tidaldb-cluster label pod tidaldb-1 tidaldb-2 tidaldb.orchard9.ai/serving-
|
||
> ```
|
||
|
||
---
|
||
|
||
## 0. First 60 seconds — orient
|
||
|
||
```bash
|
||
export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
|
||
kubectl get pods -n tidaldb-cluster -l app.kubernetes.io/name=tidaldb -o wide # 3/3 Ready? restarts?
|
||
# health + leader (HTTPS + mTLS plane), via port-forward:
|
||
kubectl port-forward -n tidaldb-cluster tidaldb-0 19500:9500 >/dev/null 2>&1 &
|
||
curl -fsSk https://127.0.0.1:19500/health # ok:true + agreed leader across pods
|
||
curl -fsSk https://127.0.0.1:19500/cluster/status # per-shard leaders, commit index, quarantine, reseed
|
||
```
|
||
|
||
Golden signals (Grafana "tidalDB Overview" → Cluster Replication row, or `/metrics`):
|
||
|
||
| Signal | Metric | Healthy |
|
||
|---|---|---|
|
||
| Per-peer circuit breaker | `tidaldb_cluster_peer_breaker_state` | 0 (closed). **1 (open) on a live peer = the write-burst/partition pattern.** |
|
||
| Commit progress / quorum lag | `relay_durable_seq` vs `relay_last_seq` | lag bounded, advancing under load |
|
||
| Reseed pending | `reseed_required` | 0 |
|
||
| Election churn | `leader_changes_total`, `election_term` | flat |
|
||
| Quorum timeouts | `quorum_timeouts_total` | flat (a rising rate = writes can't reach quorum) |
|
||
| Divergence | `divergence_quarantined` | 0 |
|
||
|
||
---
|
||
|
||
## 1. Severity
|
||
|
||
| SEV | Definition | Examples |
|
||
|---|---|---|
|
||
| **SEV1** | Writes unavailable or acked data at risk | quorum lost (≥2 voters down or both breakers stuck open), divergent prefix served, total cluster loss |
|
||
| **SEV2** | Degraded but serving | one voter down (still 2/3 quorum), read p99 > 10 ms sustained, a pod CrashLooping, write-burst 503s on an old image |
|
||
| **SEV3** | No user impact yet | single transient reseed, elevated election term, one breaker briefly open then clears, soak night failed |
|
||
|
||
---
|
||
|
||
## 2. Symptom → signal → runbook
|
||
|
||
| Symptom | Confirm with | Action |
|
||
|---|---|---|
|
||
| **Pod CrashLooping** ("reseed marker latched") | `kubectl logs`, `reseed_required` | Reseed-loop is resolved on the live image (rc5+); a genuine reseed snapshot-installs and converges lag=0. If looping on an OLD image → roll to the live digest. [`cluster.md` reseed] |
|
||
| **Writes 503 / quorum lost** | `/cluster/status` (voters down?), `quorum_timeouts_total` rising | One voter down is survivable (2/3). Both followers down → honest retryable 503 until one returns. Restart/replace the down voter; check node health. [`cluster.md` §9] |
|
||
| **Write-burst 503-storm, no self-heal; both breakers stuck at 1 while peers alive** | `tidaldb_cluster_peer_breaker_state == 1` on live (heartbeating) followers | Fixed in **rc7** (write-burst false-partition). If on an older image, **roll to ≥ rc7**. Confirm the gauge clears as load eases. [`cluster.md` §9.1] |
|
||
| **Stuck leader / dual-leader view** (leader sees peers partitioned, reads 408) | `/cluster/status` disagrees across pods | Restart the leader pod to force re-election; auto-election converges. [`cluster.md` §9] |
|
||
| **Divergence / node serving a stale prefix** | `divergence_quarantined > 0` | The node auto-quarantines (503 readiness) and reseeds without wiping — it never serves a divergent prefix. Let it reseed; verify it rejoins lag=0. [`cluster.md` §9.1] |
|
||
| **Read p99 > 10 ms, reads hang** | Grafana read-latency; check load spread | Read-SLA collapse was CPU oversubscription, fixed in rc12 (SEARCH_GATE + parallel scatter + cpu limit 3). Ensure reads are spread round-robin across all 3 pods; check the ~1000-rps read ceiling isn't exceeded. [`capacity-planning.md`] |
|
||
| **OOMKilled** | pod `lastState.terminated.reason=OOMKilled` | 1536-D full placement peaks ~1.9 GiB at 100k; **1M needs > 16 GiB nodes** (≈7–8 GB/pod). Do not raise the limit past node headroom — it's a node-size limit. [`capacity-planning.md`] |
|
||
| **One pod's PVC lost/corrupt** | pod won't open its data dir | Delete that pod's PVC + pod → it reseeds fresh from the live quorum (snapshot install), converges lag=0. Do NOT object-store-restore for a single-pod loss. [`ops/recovery.md`] |
|
||
| **Total cluster loss** | — | Rebuild fresh + `tidalctl restore` each shard before boot. [`disaster-recovery.md`] |
|
||
| **Soak night failed** | `kubectl get jobs -n tidaldb-cluster -l app.kubernetes.io/name=tidal-soak` | A FAIL resets the 30-night streak. Read the night's log for the SLO breach (p99 > 150 ms or error > 1%) or an under-load restart. [`cluster.md` §15] |
|
||
|
||
---
|
||
|
||
## 3. Alerting status (action required)
|
||
|
||
The alert rules in [`../ops/prometheus-alerts.yaml`](../ops/prometheus-alerts.yaml)
|
||
are **accurate but DESIGN-REFERENCE — they are not yet loaded by the live
|
||
Prometheus.** The rules already cover the must-watch signals
|
||
(`TidalDBClusterBreakerOpen` on `peer_breaker_state == 1`,
|
||
`TidalDBClusterCommitIndexStall`, `TidalDBClusterElectionChurn`,
|
||
`TidalDBClusterQuorumTimeouts`, `TidalDBClusterDivergenceQuarantine`,
|
||
`TidalDBClusterReseedPending`).
|
||
|
||
**Open infra step:** promote them into the observability stack as a `PrometheusRule`
|
||
(or vmalert rule file) so they actually page. Until then, on-call watches the
|
||
Grafana Cluster Replication row manually. This is the one item that is an
|
||
observability *deployment*, not a doc fix — track it in `orchard9-k3sf`.
|
||
|
||
---
|
||
|
||
## 4. Escalation
|
||
|
||
1. Page the on-call operator → run §0, classify per §1.
|
||
2. SEV1 (acked data at risk / writes down): stop further rollouts, capture
|
||
`/cluster/status` + breaker gauge + `kubectl logs` from all 3 pods, then engage
|
||
the tidalDB maintainers. Do **not** delete PVCs or force-promote under a
|
||
divergence/quorum-loss condition without confirming the data-safety path in
|
||
[`cluster.md`](cluster.md) — the system is designed to quarantine rather than
|
||
serve bad data; let it.
|
||
3. Record the incident + resolution in
|
||
`orchard9-k3sf/deployments/history/tidaldb.md` (newest first).
|