From c58b18b994ad032e3a9eb1481f94107121aa2735 Mon Sep 17 00:00:00 2001 From: jordan Date: Thu, 20 Aug 2026 22:32:50 -0600 Subject: [PATCH] docs(ops): un-blind the reseed alert and record the active VIP intervention The 2026-08-20 livelock ran 21h with no page. TidalDBClusterReseedPending is `tidaldb_cluster_reseed_required == 1 for 10m`, written for exactly this, but the defect cleared the marker ~200ms after each latch, so the gauge flapped 1->0 every ~30s and never held 1 for 10m. Add TidalDBClusterReseedFlapping, which keys off `changes(...[15m]) > 2` instead of a hold duration, so a latch/clear loop pages even when the gauge reads 0 at both ends of the window and even if a future clear path reintroduces the spurious clear. Also add an on-call banner for the operator intervention now in force: the client Service selector is narrowed to keep tidaldb-0 out of the read path, because its shard-1 frontier was cross-seeded from shard 2's snapshot artifact and its reads are untrustworthy even at lag_events: 0. The banner carries the footgun (the label is on the pods, not the template, so it does not survive pod recreation) plus repair and revert commands. --- docs/ops/prometheus-alerts.yaml | 25 ++++++++++++++++++++ docs/runbooks/on-call.md | 42 +++++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+) diff --git a/docs/ops/prometheus-alerts.yaml b/docs/ops/prometheus-alerts.yaml index faacee1..6e3b948 100644 --- a/docs/ops/prometheus-alerts.yaml +++ b/docs/ops/prometheus-alerts.yaml @@ -225,6 +225,17 @@ groups: # Durable reseed-marker latch. Expected briefly after a quarantine or a # behind-a-compacted-leader restart; a persistent latch means the snapshot # fetch is failing. + # + # BLIND SPOT (2026-08-20 incident, incident_2026_08_20_reseed_livelock in + # orchard9-k3sf/cluster-state.yaml): `== 1 for 10m` cannot catch a node + # that LATCHES AND CLEARS the marker in a loop. tidaldb-0 flapped this + # gauge 1->0 every ~30s for 21h across 196 restarts and never once held 1 + # for 10m, so this alert stayed silent through a hard-down voter. The + # marker-clear fix (ReseedMarker::discharged_by) stops the spurious clear, + # which is what makes the `for: 10m` window reachable again — but a + # duration alert on a self-clearing gauge is structurally fragile, so + # TidalDBClusterReseedFlapping below catches the loop directly rather than + # trusting that no future clear path reintroduces it. expr: tidaldb_cluster_reseed_required == 1 for: 10m labels: { severity: warning } @@ -232,6 +243,20 @@ groups: summary: "A node has a reseed pending but has not completed it" description: "A node has latched the reseed marker but has not completed a snapshot reseed in 10m. Expected briefly after a quarantine or a behind-a-compacted-leader restart; a persistent latch means the snapshot fetch is failing (no reachable leader, capability gate, or staging fault)." + - alert: TidalDBClusterReseedFlapping + # The companion to the blind spot above: catch a marker that is being + # latched repeatedly, whether or not it ever STAYS latched. `changes()` + # counts every transition, so a latch/clear loop registers even when the + # gauge reads 0 at both ends of the window. Two transitions in 15m is + # already abnormal (a healthy reseed latches once and clears once, after + # the install); the 2026-08-20 loop ran ~60 transitions per 15m. + expr: changes(tidaldb_cluster_reseed_required[15m]) > 2 + for: 5m + labels: { severity: critical } + annotations: + summary: "A node is latching and clearing its reseed marker in a loop" + description: "tidaldb_cluster_reseed_required has changed more than twice in 15m: the node keeps deciding it needs a reseed and then deciding it does not. It is not converging, and a duration-based reseed alert will NOT fire for it. Check whether the catch-up puller's from_seqno is advancing and whether a snapshot install is completing for EVERY hosted shard (a multi-shard node can install one group's artifact under another group's frontier)." + - alert: TidalDBClusterSnapshotPinForceDrop # A staged-snapshot retention pin was force-dropped past the hard cap — a # joiner started a reseed and never released (died mid-fetch). The dropped diff --git a/docs/runbooks/on-call.md b/docs/runbooks/on-call.md index db380db..192691b 100644 --- a/docs/runbooks/on-call.md +++ b/docs/runbooks/on-call.md @@ -12,6 +12,48 @@ is `orchard9-k3sf/deployments/history/tidaldb.md`; live state is --- +> ## ⚠ ACTIVE OPERATOR INTERVENTION — read before touching the client Service +> +> **As of 2026-08-21 the client Service `tidaldb` carries a NARROWED selector.** +> `tidaldb-0` serves a shard-1 frontier that was cross-seeded from shard 2's +> snapshot artifact (see `incident_2026_08_20_reseed_livelock` in +> `orchard9-k3sf/cluster-state.yaml`), so its reads are not trustworthy even when +> it reports `lag_events: 0`. It is held out of the read path by an extra +> selector label rather than by readiness, because the marker-clear bug lets it +> report Ready while degraded. +> +> ```bash +> # what is in force +> kubectl -n tidaldb-cluster get svc tidaldb -o jsonpath='{.spec.selector}' +> # => includes tidaldb.orchard9.ai/serving: "true" +> # only tidaldb-1 and tidaldb-2 carry that label (applied to the PODS, not the template) +> ``` +> +> **FOOTGUN — this label does NOT survive pod recreation.** It is on the pods, not +> the StatefulSet template (adding it to the template would trigger a rolling +> update, and with only two healthy voters that loses quorum). If `tidaldb-1` or +> `tidaldb-2` is recreated it silently drops out of the client Service. If BOTH +> are recreated the Service has ZERO endpoints and all reads fail. +> +> ```bash +> # SYMPTOM: reads fail / Service has no endpoints +> kubectl -n tidaldb-cluster get endpoints tidaldb +> # REPAIR: re-label whichever healthy pods lost it +> kubectl -n tidaldb-cluster label pod tidaldb-1 tidaldb-2 tidaldb.orchard9.ai/serving=true --overwrite +> ``` +> +> **REVERT once tidaldb-0 genuinely holds every shard** (per-shard `applied_events` +> matching each shard's leader AND a real query agreeing with a healthy peer — a +> frontier number alone is what misled us here): +> +> ```bash +> kubectl -n tidaldb-cluster patch svc tidaldb --type=merge \ +> -p '{"spec":{"selector":{"app.kubernetes.io/name":"tidaldb","app.kubernetes.io/component":"cluster-node"}}}' +> kubectl -n tidaldb-cluster label pod tidaldb-1 tidaldb-2 tidaldb.orchard9.ai/serving- +> ``` + +--- + ## 0. First 60 seconds — orient ```bash