tidaldb/docs/runbooks/on-call.md

tidalDB On-Call & Incident Response

First page to open during a tidalDB incident. Routes a symptom → the golden signal that confirms it → the runbook that fixes it. For the operational API see cluster.md; for k8s mechanics kubernetes.md; for backup/restore disaster-recovery.md.

Ownership: the tidalDB maintainers (source: git.threesix.ai/jordan/tidaldb). The live cluster runs in orchard9-k3sf (ns tidaldb-cluster); deployment history is orchard9-k3sf/deployments/history/tidaldb.md; live state is orchard9-k3sf/cluster-state.yaml.

---

⚠ ACTIVE OPERATOR INTERVENTION — read before touching the client Service

As of 2026-08-21 the client Service tidaldb carries a NARROWED selector. tidaldb-0 serves a shard-1 frontier that was cross-seeded from shard 2's snapshot artifact (see incident_2026_08_20_reseed_livelock in orchard9-k3sf/cluster-state.yaml), so its reads are not trustworthy even when it reports lag_events: 0. It is held out of the read path by an extra selector label rather than by readiness, because the marker-clear bug lets it report Ready while degraded.

# what is in force
kubectl -n tidaldb-cluster get svc tidaldb -o jsonpath='{.spec.selector}'
#   => includes tidaldb.orchard9.ai/serving: "true"
# only tidaldb-1 and tidaldb-2 carry that label (applied to the PODS, not the template)

FOOTGUN — this label does NOT survive pod recreation. It is on the pods, not the StatefulSet template (adding it to the template would trigger a rolling update, and with only two healthy voters that loses quorum). If tidaldb-1 or tidaldb-2 is recreated it silently drops out of the client Service. If BOTH are recreated the Service has ZERO endpoints and all reads fail.

# SYMPTOM: reads fail / Service has no endpoints
kubectl -n tidaldb-cluster get endpoints tidaldb
# REPAIR: re-label whichever healthy pods lost it
kubectl -n tidaldb-cluster label pod tidaldb-1 tidaldb-2 tidaldb.orchard9.ai/serving=true --overwrite

REVERT once tidaldb-0 genuinely holds every shard (per-shard applied_events matching each shard's leader AND a real query agreeing with a healthy peer — a frontier number alone is what misled us here):

kubectl -n tidaldb-cluster patch svc tidaldb --type=merge \
  -p '{"spec":{"selector":{"app.kubernetes.io/name":"tidaldb","app.kubernetes.io/component":"cluster-node"}}}'
kubectl -n tidaldb-cluster label pod tidaldb-1 tidaldb-2 tidaldb.orchard9.ai/serving-

---

0. First 60 seconds — orient

export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
kubectl get pods -n tidaldb-cluster -l app.kubernetes.io/name=tidaldb -o wide   # 3/3 Ready? restarts?
# health + leader (HTTPS + mTLS plane), via port-forward:
kubectl port-forward -n tidaldb-cluster tidaldb-0 19500:9500 >/dev/null 2>&1 &
curl -fsSk https://127.0.0.1:19500/health          # ok:true + agreed leader across pods
curl -fsSk https://127.0.0.1:19500/cluster/status  # per-shard leaders, commit index, quarantine, reseed

Golden signals (Grafana "tidalDB Overview" → Cluster Replication row, or /metrics):

Signal	Metric	Healthy
Per-peer circuit breaker	tidaldb_cluster_peer_breaker_state	0 (closed). 1 (open) on a live peer = the write-burst/partition pattern.
Commit progress / quorum lag	relay_durable_seq vs relay_last_seq	lag bounded, advancing under load
Reseed pending	reseed_required	0
Election churn	leader_changes_total, election_term	flat
Quorum timeouts	quorum_timeouts_total	flat (a rising rate = writes can't reach quorum)
Divergence	divergence_quarantined	0

---

1. Severity

SEV	Definition	Examples
SEV1	Writes unavailable or acked data at risk	quorum lost (≥2 voters down or both breakers stuck open), divergent prefix served, total cluster loss
SEV2	Degraded but serving	one voter down (still 2/3 quorum), read p99 > 10 ms sustained, a pod CrashLooping, write-burst 503s on an old image
SEV3	No user impact yet	single transient reseed, elevated election term, one breaker briefly open then clears, soak night failed

---

2. Symptom → signal → runbook

Symptom	Confirm with	Action
Pod CrashLooping ("reseed marker latched")	kubectl logs, reseed_required	Reseed-loop is resolved on the live image (rc5+); a genuine reseed snapshot-installs and converges lag=0. If looping on an OLD image → roll to the live digest. [cluster.md reseed]
Writes 503 / quorum lost	/cluster/status (voters down?), quorum_timeouts_total rising	One voter down is survivable (2/3). Both followers down → honest retryable 503 until one returns. Restart/replace the down voter; check node health. [cluster.md §9]
Write-burst 503-storm, no self-heal; both breakers stuck at 1 while peers alive	tidaldb_cluster_peer_breaker_state == 1 on live (heartbeating) followers	Fixed in rc7 (write-burst false-partition). If on an older image, roll to ≥ rc7. Confirm the gauge clears as load eases. [cluster.md §9.1]
Stuck leader / dual-leader view (leader sees peers partitioned, reads 408)	/cluster/status disagrees across pods	Restart the leader pod to force re-election; auto-election converges. [cluster.md §9]
Divergence / node serving a stale prefix	divergence_quarantined > 0	The node auto-quarantines (503 readiness) and reseeds without wiping — it never serves a divergent prefix. Let it reseed; verify it rejoins lag=0. [cluster.md §9.1]
Read p99 > 10 ms, reads hang	Grafana read-latency; check load spread	Read-SLA collapse was CPU oversubscription, fixed in rc12 (SEARCH_GATE + parallel scatter + cpu limit 3). Ensure reads are spread round-robin across all 3 pods; check the ~1000-rps read ceiling isn't exceeded. [capacity-planning.md]
OOMKilled	pod lastState.terminated.reason=OOMKilled	1536-D full placement peaks ~1.9 GiB at 100k; 1M needs > 16 GiB nodes (≈7–8 GB/pod). Do not raise the limit past node headroom — it's a node-size limit. [capacity-planning.md]
One pod's PVC lost/corrupt	pod won't open its data dir	Delete that pod's PVC + pod → it reseeds fresh from the live quorum (snapshot install), converges lag=0. Do NOT object-store-restore for a single-pod loss. [ops/recovery.md]
Total cluster loss	—	Rebuild fresh + tidalctl restore each shard before boot. [disaster-recovery.md]
Soak night failed	kubectl get jobs -n tidaldb-cluster -l app.kubernetes.io/name=tidal-soak	A FAIL resets the 30-night streak. Read the night's log for the SLO breach (p99 > 150 ms or error > 1%) or an under-load restart. [cluster.md §15]

---

3. Alerting status (action required)

The alert rules in ../ops/prometheus-alerts.yaml are accurate but DESIGN-REFERENCE — they are not yet loaded by the live Prometheus. The rules already cover the must-watch signals (TidalDBClusterBreakerOpen on peer_breaker_state == 1, TidalDBClusterCommitIndexStall, TidalDBClusterElectionChurn, TidalDBClusterQuorumTimeouts, TidalDBClusterDivergenceQuarantine, TidalDBClusterReseedPending).

Open infra step: promote them into the observability stack as a PrometheusRule (or vmalert rule file) so they actually page. Until then, on-call watches the Grafana Cluster Replication row manually. This is the one item that is an observability deployment, not a doc fix — track it in orchard9-k3sf.

---

4. Escalation

1. Page the on-call operator → run §0, classify per §1.
2. SEV1 (acked data at risk / writes down): stop further rollouts, capture /cluster/status + breaker gauge + kubectl logs from all 3 pods, then engage the tidalDB maintainers. Do not delete PVCs or force-promote under a divergence/quorum-loss condition without confirming the data-safety path in cluster.md — the system is designed to quarantine rather than serve bad data; let it.
3. Record the incident + resolution in orchard9-k3sf/deployments/history/tidaldb.md (newest first).