Commit Graph

1 Commits

Author SHA1 Message Date
jordan
d21a202a56 docs(runbooks): add an executable deploy verification checklist
Every command in it was run against the live orchard9-k3sf deployment and its
output recorded before commit. Nothing is aspirational, and the three defects
found while dogfooding it are fixed rather than left for the reader:

  * the backup check sorted ALL backups by timestamp and selected a
    restore-canary run (20 items, 1 volume) — it would have "passed" while
    telling you nothing about the fleet. Now filters on the schedule label.
  * the certificate check dialled the hostname, which fails on a workstation
    behind a split-DNS resolver. Now connects by IP with SNI.
  * a prose line was sitting inside a bash fence.

Sections 1-8 verify what is deployed today. Section 9 is deliberately separate:
HTTP metrics, the operator/data credential split, and structured logs are
committed and tested but INERT until an image roll, so their absence is not
mistaken for a regression. Five dashboard panels are legitimately empty for the
same reason and the doc says which.

Carries the two measurement traps this deploy actually produced, because both
generated false alarms: port-forward needs sleep 8 (a shorter wait races the bind
and reads like a dead node), and pod-to-pod reachability must not be probed with
/dev/tcp under sh (dash has no /dev/tcp, so an OPEN port reports refused - that
briefly looked like a cluster partition).

Also records the known-red reseed tests as environmental rather than regressions:
bisect against the preceding commit shows all three fail identically there, on
the first ack=quorum write ~1s after the gRPC listeners bind and before peer ship
channels exist, against the harness's own 3s client timeout.
2026-08-23 11:17:01 -06:00