diff --git a/docs/ops/prometheus-alerts.yaml b/docs/ops/prometheus-alerts.yaml index faacee1..6e3b948 100644 --- a/docs/ops/prometheus-alerts.yaml +++ b/docs/ops/prometheus-alerts.yaml @@ -225,6 +225,17 @@ groups: # Durable reseed-marker latch. Expected briefly after a quarantine or a # behind-a-compacted-leader restart; a persistent latch means the snapshot # fetch is failing. + # + # BLIND SPOT (2026-08-20 incident, incident_2026_08_20_reseed_livelock in + # orchard9-k3sf/cluster-state.yaml): `== 1 for 10m` cannot catch a node + # that LATCHES AND CLEARS the marker in a loop. tidaldb-0 flapped this + # gauge 1->0 every ~30s for 21h across 196 restarts and never once held 1 + # for 10m, so this alert stayed silent through a hard-down voter. The + # marker-clear fix (ReseedMarker::discharged_by) stops the spurious clear, + # which is what makes the `for: 10m` window reachable again — but a + # duration alert on a self-clearing gauge is structurally fragile, so + # TidalDBClusterReseedFlapping below catches the loop directly rather than + # trusting that no future clear path reintroduces it. expr: tidaldb_cluster_reseed_required == 1 for: 10m labels: { severity: warning } @@ -232,6 +243,20 @@ groups: summary: "A node has a reseed pending but has not completed it" description: "A node has latched the reseed marker but has not completed a snapshot reseed in 10m. Expected briefly after a quarantine or a behind-a-compacted-leader restart; a persistent latch means the snapshot fetch is failing (no reachable leader, capability gate, or staging fault)." + - alert: TidalDBClusterReseedFlapping + # The companion to the blind spot above: catch a marker that is being + # latched repeatedly, whether or not it ever STAYS latched. `changes()` + # counts every transition, so a latch/clear loop registers even when the + # gauge reads 0 at both ends of the window. Two transitions in 15m is + # already abnormal (a healthy reseed latches once and clears once, after + # the install); the 2026-08-20 loop ran ~60 transitions per 15m. + expr: changes(tidaldb_cluster_reseed_required[15m]) > 2 + for: 5m + labels: { severity: critical } + annotations: + summary: "A node is latching and clearing its reseed marker in a loop" + description: "tidaldb_cluster_reseed_required has changed more than twice in 15m: the node keeps deciding it needs a reseed and then deciding it does not. It is not converging, and a duration-based reseed alert will NOT fire for it. Check whether the catch-up puller's from_seqno is advancing and whether a snapshot install is completing for EVERY hosted shard (a multi-shard node can install one group's artifact under another group's frontier)." + - alert: TidalDBClusterSnapshotPinForceDrop # A staged-snapshot retention pin was force-dropped past the hard cap — a # joiner started a reseed and never released (died mid-fetch). The dropped diff --git a/docs/runbooks/on-call.md b/docs/runbooks/on-call.md index db380db..192691b 100644 --- a/docs/runbooks/on-call.md +++ b/docs/runbooks/on-call.md @@ -12,6 +12,48 @@ is `orchard9-k3sf/deployments/history/tidaldb.md`; live state is --- +> ## ⚠ ACTIVE OPERATOR INTERVENTION — read before touching the client Service +> +> **As of 2026-08-21 the client Service `tidaldb` carries a NARROWED selector.** +> `tidaldb-0` serves a shard-1 frontier that was cross-seeded from shard 2's +> snapshot artifact (see `incident_2026_08_20_reseed_livelock` in +> `orchard9-k3sf/cluster-state.yaml`), so its reads are not trustworthy even when +> it reports `lag_events: 0`. It is held out of the read path by an extra +> selector label rather than by readiness, because the marker-clear bug lets it +> report Ready while degraded. +> +> ```bash +> # what is in force +> kubectl -n tidaldb-cluster get svc tidaldb -o jsonpath='{.spec.selector}' +> # => includes tidaldb.orchard9.ai/serving: "true" +> # only tidaldb-1 and tidaldb-2 carry that label (applied to the PODS, not the template) +> ``` +> +> **FOOTGUN — this label does NOT survive pod recreation.** It is on the pods, not +> the StatefulSet template (adding it to the template would trigger a rolling +> update, and with only two healthy voters that loses quorum). If `tidaldb-1` or +> `tidaldb-2` is recreated it silently drops out of the client Service. If BOTH +> are recreated the Service has ZERO endpoints and all reads fail. +> +> ```bash +> # SYMPTOM: reads fail / Service has no endpoints +> kubectl -n tidaldb-cluster get endpoints tidaldb +> # REPAIR: re-label whichever healthy pods lost it +> kubectl -n tidaldb-cluster label pod tidaldb-1 tidaldb-2 tidaldb.orchard9.ai/serving=true --overwrite +> ``` +> +> **REVERT once tidaldb-0 genuinely holds every shard** (per-shard `applied_events` +> matching each shard's leader AND a real query agreeing with a healthy peer — a +> frontier number alone is what misled us here): +> +> ```bash +> kubectl -n tidaldb-cluster patch svc tidaldb --type=merge \ +> -p '{"spec":{"selector":{"app.kubernetes.io/name":"tidaldb","app.kubernetes.io/component":"cluster-node"}}}' +> kubectl -n tidaldb-cluster label pod tidaldb-1 tidaldb-2 tidaldb.orchard9.ai/serving- +> ``` + +--- + ## 0. First 60 seconds — orient ```bash