Every command in it was run against the live orchard9-k3sf deployment and its
output recorded before commit. Nothing is aspirational, and the three defects
found while dogfooding it are fixed rather than left for the reader:
* the backup check sorted ALL backups by timestamp and selected a
restore-canary run (20 items, 1 volume) — it would have "passed" while
telling you nothing about the fleet. Now filters on the schedule label.
* the certificate check dialled the hostname, which fails on a workstation
behind a split-DNS resolver. Now connects by IP with SNI.
* a prose line was sitting inside a bash fence.
Sections 1-8 verify what is deployed today. Section 9 is deliberately separate:
HTTP metrics, the operator/data credential split, and structured logs are
committed and tested but INERT until an image roll, so their absence is not
mistaken for a regression. Five dashboard panels are legitimately empty for the
same reason and the doc says which.
Carries the two measurement traps this deploy actually produced, because both
generated false alarms: port-forward needs sleep 8 (a shorter wait races the bind
and reads like a dead node), and pod-to-pod reachability must not be probed with
/dev/tcp under sh (dash has no /dev/tcp, so an OPEN port reports refused - that
briefly looked like a cluster partition).
Also records the known-red reseed tests as environmental rather than regressions:
bisect against the preceding commit shows all three fail identically there, on
the first ack=quorum write ~1s after the gRPC listeners bind and before peer ship
channels exist, against the harness's own 3s client timeout.
The runbooks had drifted to the retired m8/m11p5 design while all m12 production
reality (topology, perf, fixes, DR) sat only in a profiling doc no operator opens.
This promotes that reality into the runbooks and fixes the contradictions.
Contradictions fixed:
- runbooks/cluster.md: the "NEITHER IS QUORUM-ACKED HA YET" status banner was FALSE
(quorum-ack + automatic election have been live since m11p3/p4). Rewritten to
state the deployed reality (single-StatefulSet full-placement RF3, rc7).
- README.md: the cluster section called the HA cluster a "built-in simulated
cluster / multi-region fabric" demo and showed promote-by-region as failover.
Rewritten — real quorum HA, automatic failover, /cluster/promote is a maintenance
verb. Kept the honest caveats (experimental gate, global-signals-only).
Reality promoted into the runbooks:
- Live topology (single STS, ns tidaldb-cluster, 3 voters, full-placement RF3,
gRPC 9601/9602/9603, HTTPS+mTLS :9500), the five shipped fixes, and the real
build+digest-pin procedure (cross-compile -> trixie -> amd64 PLATFORM manifest,
not the index/attestation digest) in cluster.md + kubernetes.md.
- Stale constants: soak ramp 3900 -> 200 rps; cluster grace 60 -> 600s; the
pre-m12 4.5k/s signal-write perf table annotated + the 1536-D read reality added.
- ops/capacity-planning.md: new "Ref-A 3-node fleet — measured capacity" section
(read p99/ceiling, ~250 rps write knee, 1M needs >16GiB nodes, pod resources).
- ops/recovery.md: new cluster-recovery routing section + scoped the quiesce-and-
copy note to standalone (the cluster uses tidalctl + the DR runbook).
New docs:
- runbooks/disaster-recovery.md: the proven S3/R2 backup -> restore -> byte-verify
-> query-proof procedure, full-cluster rebuild, PITR posture (previously
undocumented despite being proven against real S3).
- runbooks/on-call.md: incident response — symptom -> golden signal -> runbook,
severity, escalation, and the open alert-wiring step.
- runbooks/README.md: the runbook index + current production facts.
Open follow-up (infra, not docs): ops/prometheus-alerts.yaml is accurate but
design-reference; promoting it to a live PrometheusRule is the one unwired step.
