tidaldb/docs/ops
jordan c58b18b994 docs(ops): un-blind the reseed alert and record the active VIP intervention
The 2026-08-20 livelock ran 21h with no page. TidalDBClusterReseedPending is
`tidaldb_cluster_reseed_required == 1 for 10m`, written for exactly this, but the
defect cleared the marker ~200ms after each latch, so the gauge flapped 1->0
every ~30s and never held 1 for 10m. Add TidalDBClusterReseedFlapping, which
keys off `changes(...[15m]) > 2` instead of a hold duration, so a latch/clear
loop pages even when the gauge reads 0 at both ends of the window and even if a
future clear path reintroduces the spurious clear.

Also add an on-call banner for the operator intervention now in force: the
client Service selector is narrowed to keep tidaldb-0 out of the read path,
because its shard-1 frontier was cross-seeded from shard 2's snapshot artifact
and its reads are untrustworthy even at lag_events: 0. The banner carries the
footgun (the label is on the pods, not the template, so it does not survive pod
recreation) plus repair and revert commands.
2026-08-20 22:32:50 -06:00
..
benchmark-1536-peach.md bench(1536): production-shape capacity — read path is cheap, quorum write is the ceiling 2026-06-13 21:40:58 -06:00
capacity-planning.md fleet remediation: make the workspace gate runnable, then fix what it caught 2026-08-16 12:38:14 -06:00
grafana-dashboard.json feat(m11): observability+ops (m11p8) + perf-sweep wave 2 T2 2026-06-13 09:17:49 -06:00
monitoring.md feat(m11): observability+ops (m11p8) + perf-sweep wave 2 T2 2026-06-13 09:17:49 -06:00
prometheus-alerts.yaml docs(ops): un-blind the reseed alert and record the active VIP intervention 2026-08-20 22:32:50 -06:00
recovery.md docs(m12): refresh API, specs, ops, and roadmap to the shipped M12 reality 2026-06-23 21:39:55 -06:00
stress-test-p3-t2.md feat(m11): catch-up timer retry + TSEG segment version header (m11p4) 2026-06-11 17:05:20 -06:00
stress-test-p4-regression.md feat(m11): Raft leader election over WAL stream (m11p4) 2026-06-11 23:30:24 -06:00
stress-test-thepeach.md feat(tidal-stress): open-loop capacity load generator (thepeach feed workload) 2026-06-10 21:54:21 -06:00