docs(runbooks): add an executable deploy verification checklist

Every command in it was run against the live orchard9-k3sf deployment and its output recorded before commit. Nothing is aspirational, and the three defects found while dogfooding it are fixed rather than left for the reader: * the backup check sorted ALL backups by timestamp and selected a restore-canary run (20 items, 1 volume) — it would have "passed" while telling you nothing about the fleet. Now filters on the schedule label. * the certificate check dialled the hostname, which fails on a workstation behind a split-DNS resolver. Now connects by IP with SNI. * a prose line was sitting inside a bash fence. Sections 1-8 verify what is deployed today. Section 9 is deliberately separate: HTTP metrics, the operator/data credential split, and structured logs are committed and tested but INERT until an image roll, so their absence is not mistaken for a regression. Five dashboard panels are legitimately empty for the same reason and the doc says which. Carries the two measurement traps this deploy actually produced, because both generated false alarms: port-forward needs sleep 8 (a shorter wait races the bind and reads like a dead node), and pod-to-pod reachability must not be probed with /dev/tcp under sh (dash has no /dev/tcp, so an OPEN port reports refused - that briefly looked like a cluster partition). Also records the known-red reseed tests as environmental rather than regressions: bisect against the preceding commit shows all three fail identically there, on the first ack=quorum write ~1s after the gRPC listeners bind and before peer ship channels exist, against the harness's own 3s client timeout.
2026-08-23 11:17:01 -06:00 · 2026-08-23 11:17:01 -06:00 · d21a202a56
commit d21a202a56
parent 4766f566de
2 changed files with 470 additions and 1 deletions
--- a/docs/runbooks/README.md
+++ b/docs/runbooks/README.md
@ -7,13 +7,14 @@ and history live in the `orchard9-k3sf` repo (`cluster-state.yaml`,

 | When you need to… | Open |
 |---|---|
+| **Verify a deploy end to end** — every layer, with the command and its expected output | [`deploy-verification.md`](deploy-verification.md) |
 | **Respond to an incident / page** — symptom → signal → fix | [`on-call.md`](on-call.md) |
 | Operate the cluster — API, failover, partition, membership, scale, rolling upgrade | [`cluster.md`](cluster.md) |
 | Run it on Kubernetes — deploy, digest-pin, probes, scale up/down, troubleshoot pods | [`kubernetes.md`](kubernetes.md) |
 | Back up / restore / DR — object-store export, restore, byte-verify, query-proof, R2, rebuild | [`disaster-recovery.md`](disaster-recovery.md) |
 | Recover a single node / standalone engine — corrupt keyspace, WAL, lock, schema | [`../ops/recovery.md`](../ops/recovery.md) |
 | Size a deployment — single-node tables + the measured Ref-A cluster envelope | [`../ops/capacity-planning.md`](../ops/capacity-planning.md) |
-| Read the metrics / wire dashboards & alerts | [`../ops/monitoring.md`](../ops/monitoring.md), [`../ops/grafana-dashboard.json`](../ops/grafana-dashboard.json), [`../ops/prometheus-alerts.yaml`](../ops/prometheus-alerts.yaml) |
+| Read the metrics / wire dashboards & alerts | [`../ops/observability.md`](../ops/observability.md), [`../ops/grafana-tidaldb.json`](../ops/grafana-tidaldb.json), [`../ops/prometheus-alerts.yaml`](../ops/prometheus-alerts.yaml) |
 | Understand the live perf/topology findings (dev handoff) | [`../profiling/m12-cluster-deploy-findings.md`](../profiling/m12-cluster-deploy-findings.md) |

 ## Current production facts (2026-06-19)
--- a/docs/runbooks/deploy-verification.md
+++ b/docs/runbooks/deploy-verification.md
@ -0,0 +1,468 @@
+# Deploy verification
+
+Walk this top to bottom. Every command here was executed against the live
+`orchard9-k3sf` deployment and its output recorded — nothing is aspirational.
+
+Each check states **what it proves**, the command, and what you should see. If a
+check fails, its **If it fails** line says where to look.
+
+Sections 1–8 verify what is deployed **now**. Section 9 is deliberately separate:
+those checks cannot pass until a new image is rolled, and saying so up front is
+the point.
+
+---
+
+## 0. Setup
+
+```bash
+export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
+export TIDAL_API_KEY=$(kubectl -n tidaldb-cluster get secret tidaldb-credentials \
+  -o jsonpath='{.data.TIDAL_API_KEY}' | base64 -d)
+
+# Confirm you are pointed at the right cluster before anything else.
+kubectl config current-context
+```
+
+`TIDAL_API_KEY` is the **data-plane** bearer. It is not an operator credential —
+see §9.2.
+
+---
+
+## 1. Cluster is up and converged
+
+**Proves:** all three voters are serving, and no node is behind or reseeding.
+
+```bash
+kubectl -n tidaldb-cluster get pods -l app.kubernetes.io/name=tidaldb \
+  -o custom-columns=NAME:.metadata.name,READY:.status.containerStatuses[0].ready,RESTARTS:.status.containerStatuses[0].restartCount
+```
+
+Expect three pods, `READY=true`. Restart counts are cumulative since the last
+roll — a *stable* non-zero value is fine, a *climbing* one is not.
+
+```
+NAME        READY   RESTARTS
+tidaldb-0   true    1
+tidaldb-1   true    0
+tidaldb-2   true    1
+```
+
+Then the authoritative per-node view. **Query each pod directly** — see §1.1 for
+why the aggregated view is not trustworthy here:
+
+```bash
+for i in 0 1 2; do
+  PORT=$((19700+i))
+  kubectl -n tidaldb-cluster port-forward tidaldb-$i $PORT:9500 >/dev/null 2>&1 &
+  PF=$!; sleep 8
+  printf 'tidaldb-%s: ' "$i"
+  curl -sk --max-time 10 -H "Authorization: Bearer $TIDAL_API_KEY" \
+    "https://127.0.0.1:$PORT/cluster/status/local" \
+  | python3 -c "
+import json,sys
+d=json.load(sys.stdin)
+print('region=%-11s reseed=%-5s' % (d.get('region'), d.get('reseed_required')), end='')
+for r in d.get('shards') or []:
+    print(' g%s[app=%s lag=%s ldr=%s]' % (r.get('shard'), r.get('applied_events'),
+          r.get('lag_events'), r.get('leader')), end='')
+print()"
+  kill $PF 2>/dev/null; wait $PF 2>/dev/null
+done
+```
+
+**Pass:** every pod reports `reseed=False` and `lag=0` on every group, all naming
+the same leader. Observed:
+
+```
+tidaldb-0: region=tidaldb-0   reseed=False g0[app=13324724 lag=0 ldr=tidaldb-2] g1[...lag=0...] g2[...lag=0...]
+tidaldb-1: region=tidaldb-1   reseed=False g0[app=13324724 lag=0 ldr=tidaldb-2] g1[...lag=0...] g2[...lag=0...]
+tidaldb-2: region=tidaldb-2   reseed=False g0[app=13322237 lag=0 ldr=tidaldb-2] g1[...lag=0...] g2[...lag=0...]
+```
+
+The 8-event difference on `g0` between the leader and its followers is the
+leader's own un-shipped tail, not lag.
+
+**If it fails:** `reseed=True` that survives a restart is the m11p5 livelock
+signature — see `docs/runbooks/cluster.md §8`. `lag` climbing on one node means
+that follower is not keeping up.
+
+> **Sleep 8, not 4.** A shorter wait races `port-forward`'s bind and produces an
+> empty body that reads exactly like a dead node. Two separate false alarms
+> during this deploy came from exactly that.
+
+### 1.1 Known caveat: the aggregated view under-reports peers
+
+`GET /cluster/status` can report a peer it holds no frontier report for as
+`applied_events: 0`, then derive `lag` against that zero — so a **converged**
+peer appears to be the leader's entire history behind, sometimes labelled
+`reachable: false` / `partitioned: true`.
+
+Observed on this deployment: two healthy nodes shown as `UNREACHABLE
+PARTITIONED` at 13.3M lag, while every node's own `/cluster/status/local`
+reported `lag=0` and pod-to-pod connectivity was open with nothing logged.
+
+`tidalctl cluster-status` detects that signature and prints `NO REPORT
+(aggregated view; query the node directly)` instead of repeating it as lag. **Do
+not open an incident on `NO REPORT` before running the §1 per-pod loop.** This is
+an open reporting defect, tracked in `docs/ops/observability.md §4`.
+
+---
+
+## 2. Public endpoint and TLS
+
+**Proves:** the hostname resolves publicly and serves a certificate a normal
+client will accept.
+
+```bash
+dig +short tidaldb.threesix.ai
+```
+
+Expect all three node IPs (order varies):
+
+```
+208.122.204.172
+208.122.204.173
+208.122.204.174
+```
+
+If your workstation runs a split-DNS resolver (Tailscale MagicDNS will do this),
+`dig` may succeed while `curl` cannot resolve. Confirm public resolution
+independently:
+
+```bash
+curl -s -H 'accept: application/dns-json' \
+  'https://cloudflare-dns.com/dns-query?name=tidaldb.threesix.ai&type=A' \
+| python3 -c "import json,sys; print([a['data'] for a in json.load(sys.stdin).get('Answer',[])])"
+```
+
+Certificate identity:
+
+```bash
+# Connect by IP with SNI, so this works even when the local resolver lags.
+echo | openssl s_client -connect 208.122.204.172:443 \
+  -servername tidaldb.threesix.ai 2>/dev/null \
+| openssl x509 -noout -subject -issuer -enddate
+```
+
+**Pass:** `subject=CN=tidaldb.threesix.ai`, issuer `Let's Encrypt`, `notAfter` at
+least 30 days out. Observed `notAfter=Nov 20 04:03:50 2026 GMT`.
+
+**If it fails:** cert-manager uses the **dns01** solver (`letsencrypt-prod`), not
+http01 — http01 cannot work behind a gateway gate that rejects unknown callers,
+because it rejects the ACME challenge too. Check
+`kubectl -n tidaldb-cluster get certificate tidaldb-public-tls`.
+
+---
+
+## 3. Authentication behaves correctly
+
+**Proves:** the data surface is credential-gated and the admin surface is not
+published at all.
+
+Use `--resolve` if your local resolver lags; it still validates the certificate.
+
+```bash
+R="--resolve tidaldb.threesix.ai:443:208.122.204.172"
+B="https://tidaldb.threesix.ai"
+
+printf 'health (open)      %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 $R $B/health)"
+printf 'search no bearer   %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 $R "$B/search?query=a&limit=1")"
+printf 'search bad bearer  %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 -H 'Authorization: Bearer wrong' $R "$B/search?query=a&limit=1")"
+printf 'search good bearer %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 20 -H "Authorization: Bearer $TIDAL_API_KEY" $R "$B/search?query=a&limit=1")"
+```
+
+**Pass:** `200`, `401`, `401`, `200`.
+
+Admin and metrics surfaces must be **unroutable** from the internet:
+
+```bash
+for p in /cluster/status /cluster/members /metrics /openapi.json; do
+  printf '%-18s %s\n' "$p" "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 $R "$B$p")"
+done
+```
+
+**Pass:** `404` for all four. A `200` on `/cluster/status` would be leaking
+leader identity, membership and seqnos to the internet; a `200` on `/metrics`
+would be leaking corpus size unauthenticated.
+
+**If it fails:** the path allowlist lives in `k8s/cluster/ingress.yaml`. Only the
+data paths are published, deliberately.
+
+### 3.1 A real write, end to end
+
+```bash
+curl -s --max-time 25 -X POST \
+  -H 'content-type: application/json' \
+  -H "Authorization: Bearer $TIDAL_API_KEY" \
+  -H 'x-tidal-ack: quorum' \
+  -d '{"entity_id":999000099,"metadata":{"title":"deploy verification","category":"probe"}}' \
+  $R "$B/items" -o /dev/null -w 'quorum write %{http_code}\n'
+```
+
+**Pass:** `201`. This is the strongest single check in the document — it proves
+DNS, TLS, the gateway, the backend TLS hop, authentication, and **quorum
+replication** in one request.
+
+---
+
+## 4. Network isolation
+
+**Proves:** the unauthenticated metrics port and the peer plane are not reachable
+from arbitrary pods.
+
+```bash
+kubectl -n tidaldb-cluster get networkpolicy tidaldb
+PIP=$(kubectl -n tidaldb-cluster get pod tidaldb-0 -o jsonpath='{.status.podIP}')
+
+# A pod in an unrelated namespace must NOT reach :9091.
+kubectl -n threesix exec gitea-0 -- \
+  sh -c "wget -qO- --timeout=5 http://$PIP:9091/metrics 2>&1 | head -1"
+```
+
+**Pass:** `Connection refused`. Before the policy existed this returned
+`# HELP tidaldb_uptime_seconds …` from any pod in the cluster.
+
+The scraper must still get through:
+
+```bash
+kubectl -n observability exec deploy/vmagent -- \
+  sh -c "wget -qO- --timeout=6 http://$PIP:9091/metrics 2>/dev/null | grep -c '^tidaldb_'"
+```
+
+**Pass:** a few hundred series (observed `353`).
+
+> **Do not test pod-to-pod reachability with `/dev/tcp` under `sh`.** The
+> container's `sh` is dash, which has no `/dev/tcp`, so it reports failure for a
+> port that is open. That false negative briefly looked like a cluster partition
+> during this deploy. Use `bash -c` explicitly.
+
+---
+
+## 5. Metrics and dashboard
+
+**Proves:** the series exist with the labels the dashboard queries, and the
+dashboard is loaded.
+
+```bash
+kubectl -n observability port-forward svc/vmsingle 8428:8428 >/dev/null 2>&1 &
+sleep 6
+curl -s -G 'http://127.0.0.1:8428/api/v1/series' \
+  --data-urlencode 'match[]=tidaldb_health_ok' \
+| python3 -c "import json,sys; d=json.load(sys.stdin)['data']; print(len(d),'series'); print(sorted(d[0]))"
+```
+
+**Pass:** non-zero series, labels including `namespace`, `pod`, `container`,
+`partition_id`. The dashboard's `$namespace`/`$pod` variables depend on those
+exact names.
+
+Dashboard presence (needs the Grafana admin password):
+
+```bash
+PW=$(kubectl -n observability get secret grafana-admin -o jsonpath='{.data.password}' | base64 -d)
+kubectl -n observability port-forward deploy/grafana 3000:3000 >/dev/null 2>&1 &
+sleep 6
+curl -s -u "admin:$PW" 'http://127.0.0.1:3000/api/dashboards/uid/tidaldb-overview' \
+| python3 -c "
+import json,sys
+d=json.load(sys.stdin); db=d['dashboard']
+print('LOADED:', db['title'], '| folder:', d['meta'].get('folderTitle'),
+      '| panels:', sum(1 for p in db['panels'] if p['type']!='row'))"
+```
+
+**Pass:** `LOADED: tidalDB — usage, errors, and cluster health | folder: Databases | panels: 13`.
+
+**If it fails:** the dashboard is a key on the `grafana-database-dashboards`
+ConfigMap; Grafana's file provider rescans every 30s. Source of truth is
+`docs/ops/grafana-tidaldb.json`.
+
+---
+
+## 6. Logs are queryable
+
+**Proves:** log lines are reaching the platform.
+
+```bash
+kubectl -n tidaldb-cluster logs tidaldb-0 --tail=20
+```
+
+**Pass:** readable lines, no raw `\x1b[` escape fragments.
+
+> Today's deployed image emits **plain text**, so `level:error` filtering in
+> VictoriaLogs does **not** work yet — the collector cannot parse the line, keeps
+> the text, and stamps every entry `level=info`. Structured logging is built and
+> tested but requires the image roll and `JSON_LOGS=1`; see §9.3.
+
+Until then, filter at the source:
+
+```bash
+kubectl -n tidaldb-cluster logs tidaldb-0 --since=15m | grep -iE 'ERROR|WARN'
+```
+
+---
+
+## 7. Live debugging tools work
+
+**Proves:** you can interrogate a running cluster without hand-rolling curl.
+
+```bash
+cargo build -p tidalctl        # or use a released binary
+kubectl -n tidaldb-cluster port-forward svc/tidaldb 9500:9500 >/dev/null 2>&1 &
+sleep 9
+
+# --insecure is required: the client port is served with the INTERNAL cluster CA,
+# whose leaf is issued for in-cluster DNS names.
+./target/debug/tidalctl cluster-status \
+  --url https://127.0.0.1:9500 --key "$TIDAL_API_KEY" --insecure
+echo "exit=$?"
+```
+
+**Pass:** a leader line, a region table, a shard table. Exit `0` when converged,
+`2` when not — so `tidalctl cluster-status && deploy` is a safe gate. Remember
+§1.1: `NO REPORT` means the aggregated view lacks a peer report, not that the
+peer is down.
+
+```bash
+# Convergence monitor, bounded so it terminates.
+./target/debug/tidalctl watch --url https://127.0.0.1:9500 \
+  --key "$TIDAL_API_KEY" --insecure --interval 5 --count 3
+```
+
+**Pass:** one line per tick, each ending `[ok]` or `[DEGRADED]`.
+
+Exit-code contract (verified):
+
+| Situation | Exit |
+|---|---|
+| converged | 0 |
+| bad/absent credential | 2 |
+| server unreachable | 2 |
+| `--url` without a scheme | 1 |
+| missing `--url` / `--path` | 1 |
+
+---
+
+## 8. Backups
+
+**Proves:** a whole-fleet backup completed recently and captured every volume.
+
+```bash
+# Select the latest backup FROM THE FLEET SCHEDULE. Sorting all backups by
+# timestamp picks up restore-canary runs (20 items, 1 volume), which would
+# "pass" while telling you nothing about the fleet.
+B=$(kubectl -n backup-system get backup.velero.io \
+      -l velero.io/schedule-name=velero-fleet-daily \
+      --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1].metadata.name}')
+echo "checking $B"
+kubectl -n backup-system get backup.velero.io "$B" \
+  -o jsonpath='phase={.status.phase} errors={.status.errors} items={.status.progress.itemsBackedUp}/{.status.progress.totalItems}{"\n"}'
+kubectl -n backup-system get podvolumebackups -l velero.io/backup-name="$B" \
+  --no-headers | awk '{print $2}' | sort | uniq -c
+```
+
+**Pass:** `phase=Completed`, no errors, and **every** PodVolumeBackup
+`Completed`. Observed on `velero-fleet-daily-20260823033025`: `3708/3708 items,
+48/48 Completed`.
+
+**Careful:** a single failed PVB marks the whole backup `PartiallyFailed` and
+freezes the alert gauge, even when every volume that matters was captured. If you
+trigger a manual backup to satisfy the alert, it **must** carry the schedule
+label or the metric will not advance:
+
+The alert reads
+`velero_backup_last_successful_timestamp{schedule="velero-fleet-daily"}`, so a
+manual Backup **must** carry that schedule label or the gauge never moves. A
+manual run without it completed 48/48 and the alert stayed critical:
+
+```yaml
+metadata:
+  labels:
+    velero.io/schedule-name: velero-fleet-daily
+```
+
+---
+
+## 9. Not active yet — requires an image roll
+
+The cluster runs `m12-boot-pull-fix-20260821`. The following are committed and
+covered by tests but **inert in production** until a newer image is deployed.
+They are listed so their absence is not mistaken for a regression.
+
+### 9.1 HTTP request/error metrics
+
+```bash
+PIP=$(kubectl -n tidaldb-cluster get pod tidaldb-0 -o jsonpath='{.status.podIP}')
+kubectl -n observability exec deploy/vmagent -- \
+  sh -c "wget -qO- --timeout=6 http://$PIP:9091/metrics 2>/dev/null | grep -c tidaldb_http_requests_total"
+```
+
+Currently `0`. After the roll: non-zero, and the dashboard's *Request rate by
+route*, *Requests by status*, *5xx ratio*, *Auth rejections* and *HTTP p99*
+panels populate. Until then those five panels are legitimately empty.
+
+### 9.2 Operator/data credential split
+
+After the roll, add the key (picked up by the credential poller, **no restart**):
+
+```bash
+kubectl -n tidaldb-cluster patch secret tidaldb-credentials --type=merge \
+  -p "{\"stringData\":{\"TIDAL_ADMIN_KEY\":\"$(openssl rand -hex 32)\"}}"
+```
+
+Then verify the split, port-forwarded (the admin routes are not published):
+
+```bash
+# data bearer on an operator verb -> 403 (authenticated, not authorized)
+curl -sk -o /dev/null -w '%{http_code}\n' -X POST \
+  -H "Authorization: Bearer $TIDAL_API_KEY" -d '{"region":"tidaldb-1"}' \
+  https://127.0.0.1:9500/cluster/promote
+
+# admin key -> anything except 401/403
+ADMIN=$(kubectl -n tidaldb-cluster get secret tidaldb-credentials \
+  -o jsonpath='{.data.TIDAL_ADMIN_KEY}' | base64 -d)
+curl -sk -o /dev/null -w '%{http_code}\n' -X POST \
+  -H "Authorization: Bearer $ADMIN" -d '{"region":"tidaldb-1"}' \
+  https://127.0.0.1:9500/cluster/heal
+```
+
+**Pass:** `403` then not-`403`. Before this key exists, the data bearer can
+remove members and transfer shards — that is the exposure the split closes.
+
+### 9.3 Structured logs
+
+Add `JSON_LOGS=1` to the StatefulSet, then:
+
+```bash
+kubectl -n tidaldb-cluster logs tidaldb-0 --tail=5
+```
+
+**Pass:** one JSON object per line carrying `ts`, `level`, `service`, `msg`, and
+`request_id` inside a request span. `level:error` then works in VictoriaLogs.
+
+---
+
+## Known-red, not deploy blockers
+
+Verified by bisect against the preceding commit — these are **not** caused by the
+observability or credential work:
+
+| Item | State |
+|---|---|
+| `mp_follower_reseeds_via_snapshot_after_compaction` | red on this workstation |
+| `mp_quarantined_node_reseeds_without_wipe` | red on this workstation |
+| `mp_multi_group_node_converges_after_reseeding_several_groups` | red on this workstation |
+| `mp_graceful_rolling_restart_under_load_no_reseed` | red on baseline `main` |
+
+The first three fail on the *first* `ack=quorum` write, roughly a second after
+the gRPC listeners bind and before peer ship channels are established, against
+the harness's own 3s client timeout — a startup race that loses on a loaded
+machine. All three fail identically at the commit before this work. The
+workstation they were run on is at 100% disk capacity, which is the most likely
+aggravating factor. Re-run on a machine with headroom before treating them as a
+code defect.
+
+Also open, unrelated to this deploy:
+
+- `kubectl apply -k k8s/cluster/` **downgrades** live limits (`cpu:3/7Gi` in the
+  cluster vs `2/6Gi` in the manifest). Fix the manifest or apply selectively.
+- The `tidaldb` namespace standalone Deployment is live `1/1` while the fleet
+  record calls it superseded. Nothing routes to it.