docs(ops): un-blind the reseed alert and record the active VIP intervention

The 2026-08-20 livelock ran 21h with no page. TidalDBClusterReseedPending is
`tidaldb_cluster_reseed_required == 1 for 10m`, written for exactly this, but the
defect cleared the marker ~200ms after each latch, so the gauge flapped 1->0
every ~30s and never held 1 for 10m. Add TidalDBClusterReseedFlapping, which
keys off `changes(...[15m]) > 2` instead of a hold duration, so a latch/clear
loop pages even when the gauge reads 0 at both ends of the window and even if a
future clear path reintroduces the spurious clear.

Also add an on-call banner for the operator intervention now in force: the
client Service selector is narrowed to keep tidaldb-0 out of the read path,
because its shard-1 frontier was cross-seeded from shard 2's snapshot artifact
and its reads are untrustworthy even at lag_events: 0. The banner carries the
footgun (the label is on the pods, not the template, so it does not survive pod
recreation) plus repair and revert commands.

docs/ops/prometheus-alerts.yaml

@@ -225,6 +225,17 @@ groups:
         # Durable reseed-marker latch. Expected briefly after a quarantine or a
         # behind-a-compacted-leader restart; a persistent latch means the snapshot
         # fetch is failing.
+        #
+        # BLIND SPOT (2026-08-20 incident, incident_2026_08_20_reseed_livelock in
+        # orchard9-k3sf/cluster-state.yaml): `== 1 for 10m` cannot catch a node
+        # that LATCHES AND CLEARS the marker in a loop. tidaldb-0 flapped this
+        # gauge 1->0 every ~30s for 21h across 196 restarts and never once held 1
+        # for 10m, so this alert stayed silent through a hard-down voter. The
+        # marker-clear fix (ReseedMarker::discharged_by) stops the spurious clear,
+        # which is what makes the `for: 10m` window reachable again — but a
+        # duration alert on a self-clearing gauge is structurally fragile, so
+        # TidalDBClusterReseedFlapping below catches the loop directly rather than
+        # trusting that no future clear path reintroduces it.
         expr: tidaldb_cluster_reseed_required == 1
         for: 10m
         labels: { severity: warning }
@@ -232,6 +243,20 @@ groups:
           summary: "A node has a reseed pending but has not completed it"
           description: "A node has latched the reseed marker but has not completed a snapshot reseed in 10m. Expected briefly after a quarantine or a behind-a-compacted-leader restart; a persistent latch means the snapshot fetch is failing (no reachable leader, capability gate, or staging fault)."
 
+      - alert: TidalDBClusterReseedFlapping
+        # The companion to the blind spot above: catch a marker that is being
+        # latched repeatedly, whether or not it ever STAYS latched. `changes()`
+        # counts every transition, so a latch/clear loop registers even when the
+        # gauge reads 0 at both ends of the window. Two transitions in 15m is
+        # already abnormal (a healthy reseed latches once and clears once, after
+        # the install); the 2026-08-20 loop ran ~60 transitions per 15m.
+        expr: changes(tidaldb_cluster_reseed_required[15m]) > 2
+        for: 5m
+        labels: { severity: critical }
+        annotations:
+          summary: "A node is latching and clearing its reseed marker in a loop"
+          description: "tidaldb_cluster_reseed_required has changed more than twice in 15m: the node keeps deciding it needs a reseed and then deciding it does not. It is not converging, and a duration-based reseed alert will NOT fire for it. Check whether the catch-up puller's from_seqno is advancing and whether a snapshot install is completing for EVERY hosted shard (a multi-shard node can install one group's artifact under another group's frontier)."
+
       - alert: TidalDBClusterSnapshotPinForceDrop
         # A staged-snapshot retention pin was force-dropped past the hard cap — a
         # joiner started a reseed and never released (died mid-fetch). The dropped

docs/runbooks/on-call.md

@@ -12,6 +12,48 @@ is `orchard9-k3sf/deployments/history/tidaldb.md`; live state is
 
 ---
 
+> ## ⚠ ACTIVE OPERATOR INTERVENTION — read before touching the client Service
+>
+> **As of 2026-08-21 the client Service `tidaldb` carries a NARROWED selector.**
+> `tidaldb-0` serves a shard-1 frontier that was cross-seeded from shard 2's
+> snapshot artifact (see `incident_2026_08_20_reseed_livelock` in
+> `orchard9-k3sf/cluster-state.yaml`), so its reads are not trustworthy even when
+> it reports `lag_events: 0`. It is held out of the read path by an extra
+> selector label rather than by readiness, because the marker-clear bug lets it
+> report Ready while degraded.
+>
+> ```bash
+> # what is in force
+> kubectl -n tidaldb-cluster get svc tidaldb -o jsonpath='{.spec.selector}'
+> #   => includes tidaldb.orchard9.ai/serving: "true"
+> # only tidaldb-1 and tidaldb-2 carry that label (applied to the PODS, not the template)
+> ```
+>
+> **FOOTGUN — this label does NOT survive pod recreation.** It is on the pods, not
+> the StatefulSet template (adding it to the template would trigger a rolling
+> update, and with only two healthy voters that loses quorum). If `tidaldb-1` or
+> `tidaldb-2` is recreated it silently drops out of the client Service. If BOTH
+> are recreated the Service has ZERO endpoints and all reads fail.
+>
+> ```bash
+> # SYMPTOM: reads fail / Service has no endpoints
+> kubectl -n tidaldb-cluster get endpoints tidaldb
+> # REPAIR: re-label whichever healthy pods lost it
+> kubectl -n tidaldb-cluster label pod tidaldb-1 tidaldb-2 tidaldb.orchard9.ai/serving=true --overwrite
+> ```
+>
+> **REVERT once tidaldb-0 genuinely holds every shard** (per-shard `applied_events`
+> matching each shard's leader AND a real query agreeing with a healthy peer — a
+> frontier number alone is what misled us here):
+>
+> ```bash
+> kubectl -n tidaldb-cluster patch svc tidaldb --type=merge \
+>   -p '{"spec":{"selector":{"app.kubernetes.io/name":"tidaldb","app.kubernetes.io/component":"cluster-node"}}}'
+> kubectl -n tidaldb-cluster label pod tidaldb-1 tidaldb-2 tidaldb.orchard9.ai/serving-
+> ```
+
+---
+
 ## 0. First 60 seconds — orient
 
 ```bash