tidaldb/docs/runbooks/on-call.md

# tidalDB On-Call & Incident Response

First page to open during a tidalDB incident. Routes a symptom → the golden signal
that confirms it → the runbook that fixes it. For the operational API see
[`cluster.md`](cluster.md); for k8s mechanics [`kubernetes.md`](kubernetes.md); for
backup/restore [`disaster-recovery.md`](disaster-recovery.md).

**Ownership:** the tidalDB maintainers (source: `git.threesix.ai/jordan/tidaldb`).
The live cluster runs in `orchard9-k3sf` (ns `tidaldb-cluster`); deployment history
is `orchard9-k3sf/deployments/history/tidaldb.md`; live state is
`orchard9-k3sf/cluster-state.yaml`.

---

> ## ✅ RESOLVED — the narrowed client-Service selector is NO LONGER in force
>
> **Between 2026-08-21 and 2026-08-21T20:10Z** the client Service `tidaldb`
> carried an extra `tidaldb.orchard9.ai/serving: "true"` selector label that held
> `tidaldb-0` out of the read path, because it served a shard-1 frontier
> cross-seeded from shard 2's snapshot (`incident_2026_08_20_reseed_livelock` in
> `orchard9-k3sf/cluster-state.yaml`). **That intervention was reverted and this
> section is history, not instruction.** It is kept because the footgun below is
> worth knowing if anyone ever re-narrows the selector.
>
> Verified 2026-08-31, three ways:
>
> ```bash
> kubectl -n tidaldb-cluster get svc tidaldb -o jsonpath='{.spec.selector}'
> #   => {"app.kubernetes.io/component":"cluster-node","app.kubernetes.io/name":"tidaldb"}
> #      the plain manifest selector; no serving label
> kubectl -n tidaldb-cluster get endpoints tidaldb \
>   -o jsonpath='{range .subsets[*].addresses[*]}{.targetRef.name}{"\n"}{end}'
> #   => tidaldb-0, tidaldb-1, tidaldb-2 — all three serving
> kubectl -n tidaldb-cluster get pods -l app.kubernetes.io/name=tidaldb \
>   -o custom-columns=NAME:.metadata.name,SERVING:'.metadata.labels.tidaldb\.orchard9\.ai/serving'
> #   => <none> on all three
> ```
>
> `cluster-state.yaml` records the same: *"RESOLVED 2026-08-21T20:10Z (verified,
> not inferred) … Client Service selector reverted to the manifest's original."*
>
> **If you ever re-apply that narrowing, the footgun is that the label goes on the
> PODS, not the StatefulSet template** (templating it triggers a rolling update,
> which with two healthy voters loses quorum). Pod labels do not survive pod
> recreation, so a recreated pod silently drops out of the client Service, and if
> every labelled pod is recreated the Service has ZERO endpoints and all reads
> fail. Symptom and repair:
>
> ```bash
> kubectl -n tidaldb-cluster get endpoints tidaldb          # empty subsets?
> kubectl -n tidaldb-cluster label pod tidaldb-1 tidaldb-2 \
>   tidaldb.orchard9.ai/serving=true --overwrite
> ```
>
> Do not re-narrow on a frontier number alone — a matching `lag_events: 0` is
> exactly what misled the original diagnosis. Require per-shard `applied_events`
> matching each shard's leader AND a real query agreeing with a healthy peer.

---

## 0. First 60 seconds — orient

```bash
export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
kubectl get pods -n tidaldb-cluster -l app.kubernetes.io/name=tidaldb -o wide   # 3/3 Ready? restarts?
# health + leader (HTTPS + mTLS plane), via port-forward:
kubectl port-forward -n tidaldb-cluster tidaldb-0 19500:9500 >/dev/null 2>&1 &
curl -fsSk https://127.0.0.1:19500/health          # ok:true + agreed leader across pods
curl -fsSk https://127.0.0.1:19500/cluster/status  # per-shard leaders, commit index, quarantine, reseed
```

Golden signals (Grafana "tidalDB Overview" → Cluster Replication row, or `/metrics`):

| Signal | Metric | Healthy |
|---|---|---|
| Per-peer circuit breaker | `tidaldb_cluster_peer_breaker_state` | 0 (closed). **1 (open) on a live peer = the write-burst/partition pattern.** |
| Commit progress / quorum lag | `relay_durable_seq` vs `relay_last_seq` | lag bounded, advancing under load |
| Reseed pending | `reseed_required` | 0 |
| Election churn | `leader_changes_total`, `election_term` | flat |
| Quorum timeouts | `quorum_timeouts_total` | flat (a rising rate = writes can't reach quorum) |
| Divergence | `divergence_quarantined` | 0 |

---

## 1. Severity

| SEV | Definition | Examples |
|---|---|---|
| **SEV1** | Writes unavailable or acked data at risk | quorum lost (≥2 voters down or both breakers stuck open), divergent prefix served, total cluster loss |
| **SEV2** | Degraded but serving | one voter down (still 2/3 quorum), read p99 > 10 ms sustained, a pod CrashLooping, write-burst 503s on an old image |
| **SEV3** | No user impact yet | single transient reseed, elevated election term, one breaker briefly open then clears, soak night failed |

---

## 2. Symptom → signal → runbook

| Symptom | Confirm with | Action |
|---|---|---|
| **Pod CrashLooping** ("reseed marker latched") | `kubectl logs`, `reseed_required` | Reseed-loop is resolved on the live image (rc5+); a genuine reseed snapshot-installs and converges lag=0. If looping on an OLD image → roll to the live digest. [`cluster.md` reseed] |
| **Writes 503 / quorum lost** | `/cluster/status` (voters down?), `quorum_timeouts_total` rising | One voter down is survivable (2/3). Both followers down → honest retryable 503 until one returns. Restart/replace the down voter; check node health. [`cluster.md` §9] |
| **Write-burst 503-storm, no self-heal; both breakers stuck at 1 while peers alive** | `tidaldb_cluster_peer_breaker_state == 1` on live (heartbeating) followers | Fixed in **rc7** (write-burst false-partition). If on an older image, **roll to ≥ rc7**. Confirm the gauge clears as load eases. [`cluster.md` §9.1] |
| **Stuck leader / dual-leader view** (leader sees peers partitioned, reads 408) | `/cluster/status` disagrees across pods | Restart the leader pod to force re-election; auto-election converges. [`cluster.md` §9] |
| **Divergence / node serving a stale prefix** | `divergence_quarantined > 0` | The node auto-quarantines (503 readiness) and reseeds without wiping — it never serves a divergent prefix. Let it reseed; verify it rejoins lag=0. [`cluster.md` §9.1] |
| **Read p99 > 10 ms, reads hang** | Grafana read-latency; check load spread | Read-SLA collapse was CPU oversubscription, fixed in rc12 (SEARCH_GATE + parallel scatter + cpu limit 3). Ensure reads are spread round-robin across all 3 pods; check the ~1000-rps read ceiling isn't exceeded. [`capacity-planning.md`] |
| **OOMKilled** | pod `lastState.terminated.reason=OOMKilled` | 1536-D full placement peaks ~1.9 GiB at 100k; **1M needs > 16 GiB nodes** (≈7–8 GB/pod). Do not raise the limit past node headroom — it's a node-size limit. [`capacity-planning.md`] |
| **One pod's PVC lost/corrupt** | pod won't open its data dir | Delete that pod's PVC + pod → it reseeds fresh from the live quorum (snapshot install), converges lag=0. Do NOT object-store-restore for a single-pod loss. [`ops/recovery.md`] |
| **Total cluster loss** | — | Rebuild fresh + `tidalctl restore` each shard before boot. [`disaster-recovery.md`] |
| **Soak night failed** | `kubectl get jobs -n tidaldb-cluster -l app.kubernetes.io/name=tidal-soak` | A FAIL resets the 30-night streak. Read the night's log for the SLO breach (p99 > 150 ms or error > 1%) or an under-load restart. [`cluster.md` §15] |
| **401 rate spike / "is someone attacking us?"** | `tidaldb_http_requests_total{status="401"}` for the RATE; the `rejected request` WARN for the REASON | Most 401s here are unauthenticated scanning of a public ingress and are working as designed. Do not page on volume — page on `reason=invalid_token`. Full triage: §2.1 below. |

### 2.1 Auth rejections (401) — triage

**Do not page on 401 volume.** As measured 2026-08-31T02:28Z, **87.5%** of all
traffic reaching the public ingress is a 401 (281,093 of 321,342 requests over
~14 days), and it has been that way continuously. Volume is the background rate
of an internet-facing endpoint being scanned. What is worth waking someone for is
a *change in kind*, and only the server-side WARN can tell you which kind.

#### Two counters, two different questions — never compare them

| Counter | What it measures | Use it for |
|---|---|---|
| `traefik_service_requests_total{service="tidaldb-cluster-tidaldb-9500@kubernetes",code="401"}` | Cumulative since **each Traefik pod** started (both started 2026-08-16T22:26Z) | **History.** Is this normal for this endpoint? |
| `tidaldb_http_requests_total{status="401"}` | Cumulative since **each tidalDB pod** started (current pods: 2026-08-30T20:00–20:03Z) | **Live rate.** Is it changing right now? |

**The trap:** these two have different epochs and different denominators. A
change that appears to "cut 401s from 281,000 to 3,600" cut nothing — it
restarted the pods. Any claim that a restriction worked must compare a *rate*
before and after, from the same counter, across a pod lifetime that did not
reset in between.

**Two more traps in the history number itself.** There are **two** Traefik pods
and each keeps its own counters, so a single-pod scrape undercounts — reading
only `cxz7f` gives 101,901 where the fleet total is 281,093, a 64% undercount.
And the metric is `traefik_service_requests_total`, **not**
`traefik_router_requests_total`: router labels are disabled on this build and
that metric does not exist here.

```bash
export KUBECONFIG=~/.kube/orchard9-k3sf.yaml

# HISTORY — sum across BOTH traefik pods
for p in $(kubectl -n kube-system get pods -l app.kubernetes.io/name=traefik -o name); do
  IP=$(kubectl -n kube-system get "$p" -o jsonpath='{.status.podIP}')
  echo "== $p"
  kubectl -n observability exec deploy/vmagent -- sh -c \
    "wget -qO- --timeout=20 http://$IP:9100/metrics | grep '^traefik_service_requests_total' | grep tidaldb"
done

# LIVE RATE — sample the same counter twice and difference it
sample() {
  for p in tidaldb-0 tidaldb-1 tidaldb-2; do
    IP=$(kubectl -n tidaldb-cluster get pod "$p" -o jsonpath='{.status.podIP}')
    N=$(kubectl -n observability exec deploy/vmagent -- sh -c \
      "wget -qO- --timeout=20 http://$IP:9091/metrics | awk '/^tidaldb_http_requests_total.*status=\"401\"/{s+=\$2} END{print s+0}'")
    echo -n "$p=$N "
  done; echo
}
sample; sleep 90; sample
```

Reference reading, 2026-08-31T02:31–02:32Z (94s): `3640→3657`, `3664→3675`,
`3144→3161` — **+45 fleet-wide, 28.7/min, ~9.6/min per pod.** That is the
baseline. Triage a departure from it, not the number itself.

#### The reason field is the whole diagnosis

`unauthorized_response()` in `tidal-server/src/router.rs` is the single funnel for
every 401 on every surface, and it emits one WARN per rejection carrying `reason`
and `client` (the token itself is never logged, in either form). Correlate to the
request via `request_id` on the surrounding span.

| `reason` | `client` | What it means | Page? |
|---|---|---|---|
| `invalid_token` | one consistent value, steady rate | **A credential that STOPPED working.** A real client just lost access and is failing right now — almost always a key rotation that missed a consumer. | **YES.** Check the last rotation of `tidaldb-credentials` and find the consumer still holding the old key. |
| `invalid_token` | many/varied values, bursty | A scanner spraying guessed bearers. Read the caveat below before paging. | No — but confirm it is varied, not one client. |
| `missing_token` | an external address | Unauthenticated scanning of a public ingress. Expected, correctly rejected, nothing leaked. This is the 87.5% baseline. | **NO.** Never page on this. |
| `missing_token` | `-` (absent) | **Most likely one of our workloads deployed without its key** — an absent XFF should mean the request never traversed Traefik, i.e. an in-cluster caller. Confirm that reading with the calibration below before relying on it. Not urgent, but the workload is broken until fixed. | No — file it, find the workload, give it `TIDAL_API_KEY`. |

**Read `invalid_token` precisely.** The implementation
(`unauthorized_response()`, `tidal-server/src/router.rs`) sets it whenever an
`Authorization` header is **present and did not validate** — it does not, and
cannot, know whether the token was ever valid. So a scanner sending
`Authorization: Bearer admin` lands in the page-worthy bucket alongside a genuine
rotation miss. The discriminator is `client` and the shape of the rate: a
rotation miss is a *steady* rate from *one* client that starts at a rotation; a
scanner is bursty and varied. Check both before waking anyone.

**`client="-"` should be a signal, not a gap — confirm it in the same calibration
run.** The field is `x-forwarded-for` with a `-` default. Traefik sets XFF on
what it proxies, so an absent value should mean the request **did not come
through the ingress** — it reached the pod or the ClusterIP Service directly,
which on this cluster means an in-cluster caller. That is the cleanest
internal/external split available and, unlike the address itself, it does not
depend on SNAT fidelity. The three-request test below proves it in passing: if
those known-external requests come back with a non-`-` `client`, XFF-on-proxied
holds and `-` can be read as "in-cluster". This has not been observed yet — the
WARN is not deployed — so do not lean on it until it has.

A sudden move from a steady `missing_token` baseline to `invalid_token` is the
shape that matters. Volume alone is noise; `reason` is signal.

```bash
# now: plain text on stdout
kubectl -n tidaldb-cluster logs tidaldb-0 -c tidaldb --since=15m | grep 'rejected request'

# after the JSON_LOGS roll (deploy-verification.md §9.3), ask the store instead:
kubectl -n observability exec deploy/vmagent -- sh -c \
  'wget -qO- --timeout=25 \
   --post-data="query=_time:15m AND unit:tidaldb-cluster AND _msg:rejected AND _msg:request | stats by (reason, client) count() n" \
   http://victoria-logs:9428/select/logsql/query'
```

> **Two things that will bite you here, both already paid for.**
>
> The message field in LogsQL is **`_msg`**, not `msg` — Vector's sink declares
> `_msg_field: msg`, so VictoriaLogs renames it on ingest. `msg:rejected` returns
> nothing forever and looks exactly like "no rejections". Verified 2026-08-31: the
> identical query shape with a phrase that *does* exist today
> (`_msg:compaction AND _msg:complete`) returns `{"n":"226"}`, while `msg:` in the
> same position returns empty.
>
> Word filters, not a quoted phrase. `--post-data="query=…"` is already inside a
> double-quoted shell string; a `"rejected request"` inside it terminates the
> quoting and `wget` fails with `bad address 'request | stats…'`. Two `_msg:`
> word filters ANDed need no quotes and cannot break.

> **Not yet runnable.** The WARN ships with the next image, so the two commands
> above are the only ones in §2.1 not yet executed against real data. The LogsQL
> one runs clean today and returns **nothing** — `stats by (...)` emits no rows
> when nothing matches, unlike `stats count() n` which returns `{"n":"0"}`. Its
> *form* is verified; only the content is pending. Everything else in §2.1 was
> run against the live cluster.

#### Before trusting `client`, calibrate it

`client` carries `x-forwarded-for`, because behind Traefik the socket peer is the
ingress, not the caller. Its *presence* is the part expected to be reliable (see
above); its **value**
is not yet verified on this cluster, and the "an external address" row above
depends on it. Two measured facts say to check rather than assume: the Traefik
LoadBalancer Service is `externalTrafficPolicy: Cluster` (so the external source
IP is SNATed before Traefik ever sees it), and Traefik access logging is off
(no `--accesslog` argument), so there is no second opinion to compare against.

Calibrate with a request whose origin you know — this exact procedure was run on
2026-08-31 and the counter deltas below are its real output:

```bash
# 3 unauthenticated GETs from outside the cluster
for i in 1 2 3; do
  curl -s -o /dev/null -w "http=%{http_code} remote=%{remote_ip}\n" \
    --max-time 15 "https://tidaldb.threesix.ai/search?query=probe"
done
#   -> http=401 remote=208.122.204.173
#      http=401 remote=208.122.204.174
#      http=401 remote=208.122.204.173
# /search 401 counters moved 532/526/451 -> 533/527/453: +1 +1 +2 == the 3 requests
```

Then read the `client` value those three requests produced. If it is the address
you sent from, the table above works as written. If it is a node IP
(`208.122.204.172/173/174`) or an svclb pod IP (`10.42.0.13`, `10.42.1.182`,
`10.42.2.58`), then XFF is carrying the SNAT hop: external callers are no longer
distinguishable *from each other*, and a scanner will look like it came from your
own infrastructure. The `client="-"` split still holds in that case — an absent
XFF still means "never traversed Traefik" — but you lose the ability to
attribute an external caller. Fix it (Traefik trusted-IP config, or
`externalTrafficPolicy: Local`) before using the address itself to route
anything.

#### Corrected: the "all 401s land on tidaldb-0" claim

Earlier notes recorded 3,535 401s "all on `tidaldb-0`" and flagged it as an
unexplained asymmetry possibly caused by Traefik pinning an endpoint. **Measured
2026-08-31, that asymmetry does not exist** — the earlier figure was one pod's
counter read in isolation and generalized. All three pods take the load, at
within-noise-equal rates:

- Steady state: `tidaldb-0=3640 tidaldb-1=3664 tidaldb-2=3144`, all three climbing
  (+17 / +11 / +17 over 94s).
- Controlled test: 3 external requests produced deltas of **+1 / +1 / +2** across
  the three pods.
- The Service has all three as endpoints and no pod carries a restricting label
  (see the RESOLVED banner at the top of this file).

There is nothing to investigate here. Do not spend an incident chasing it.

---

## 3. Alerting status (action required)

The alert rules in [`../ops/prometheus-alerts.yaml`](../ops/prometheus-alerts.yaml)
are **accurate but DESIGN-REFERENCE — they are not yet loaded by the live
Prometheus.** The rules already cover the must-watch signals
(`TidalDBClusterBreakerOpen` on `peer_breaker_state == 1`,
`TidalDBClusterCommitIndexStall`, `TidalDBClusterElectionChurn`,
`TidalDBClusterQuorumTimeouts`, `TidalDBClusterDivergenceQuarantine`,
`TidalDBClusterReseedPending`).

**Open infra step:** promote them into the observability stack as a `PrometheusRule`
(or vmalert rule file) so they actually page. Until then, on-call watches the
Grafana Cluster Replication row manually. This is the one item that is an
observability *deployment*, not a doc fix — track it in `orchard9-k3sf`.

---

## 4. Escalation

1. Page the on-call operator → run §0, classify per §1.
2. SEV1 (acked data at risk / writes down): stop further rollouts, capture
   `/cluster/status` + breaker gauge + `kubectl logs` from all 3 pods, then engage
   the tidalDB maintainers. Do **not** delete PVCs or force-promote under a
   divergence/quorum-loss condition without confirming the data-safety path in
   [`cluster.md`](cluster.md) — the system is designed to quarantine rather than
   serve bad data; let it.
3. Record the incident + resolution in
   `orchard9-k3sf/deployments/history/tidaldb.md` (newest first).