The 2026-08-20 livelock ran 21h with no page. TidalDBClusterReseedPending is
`tidaldb_cluster_reseed_required == 1 for 10m`, written for exactly this, but the
defect cleared the marker ~200ms after each latch, so the gauge flapped 1->0
every ~30s and never held 1 for 10m. Add TidalDBClusterReseedFlapping, which
keys off `changes(...[15m]) > 2` instead of a hold duration, so a latch/clear
loop pages even when the gauge reads 0 at both ends of the window and even if a
future clear path reintroduces the spurious clear.
Also add an on-call banner for the operator intervention now in force: the
client Service selector is narrowed to keep tidaldb-0 out of the read path,
because its shard-1 frontier was cross-seeded from shard 2's snapshot artifact
and its reads are untrustworthy even at lag_events: 0. The banner carries the
footgun (the label is on the pods, not the template, so it does not survive pod
recreation) plus repair and revert commands.
The runbooks had drifted to the retired m8/m11p5 design while all m12 production
reality (topology, perf, fixes, DR) sat only in a profiling doc no operator opens.
This promotes that reality into the runbooks and fixes the contradictions.
Contradictions fixed:
- runbooks/cluster.md: the "NEITHER IS QUORUM-ACKED HA YET" status banner was FALSE
(quorum-ack + automatic election have been live since m11p3/p4). Rewritten to
state the deployed reality (single-StatefulSet full-placement RF3, rc7).
- README.md: the cluster section called the HA cluster a "built-in simulated
cluster / multi-region fabric" demo and showed promote-by-region as failover.
Rewritten — real quorum HA, automatic failover, /cluster/promote is a maintenance
verb. Kept the honest caveats (experimental gate, global-signals-only).
Reality promoted into the runbooks:
- Live topology (single STS, ns tidaldb-cluster, 3 voters, full-placement RF3,
gRPC 9601/9602/9603, HTTPS+mTLS :9500), the five shipped fixes, and the real
build+digest-pin procedure (cross-compile -> trixie -> amd64 PLATFORM manifest,
not the index/attestation digest) in cluster.md + kubernetes.md.
- Stale constants: soak ramp 3900 -> 200 rps; cluster grace 60 -> 600s; the
pre-m12 4.5k/s signal-write perf table annotated + the 1536-D read reality added.
- ops/capacity-planning.md: new "Ref-A 3-node fleet — measured capacity" section
(read p99/ceiling, ~250 rps write knee, 1M needs >16GiB nodes, pod resources).
- ops/recovery.md: new cluster-recovery routing section + scoped the quiesce-and-
copy note to standalone (the cluster uses tidalctl + the DR runbook).
New docs:
- runbooks/disaster-recovery.md: the proven S3/R2 backup -> restore -> byte-verify
-> query-proof procedure, full-cluster rebuild, PITR posture (previously
undocumented despite being proven against real S3).
- runbooks/on-call.md: incident response — symptom -> golden signal -> runbook,
severity, escalation, and the open alert-wiring step.
- runbooks/README.md: the runbook index + current production facts.
Open follow-up (infra, not docs): ops/prometheus-alerts.yaml is accurate but
design-reference; promoting it to a live PrometheusRule is the one unwired step.
