From d21a202a568d8652c8eb20ee2e387be0ce77f87c Mon Sep 17 00:00:00 2001 From: jordan Date: Sun, 23 Aug 2026 11:17:01 -0600 Subject: [PATCH] docs(runbooks): add an executable deploy verification checklist MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every command in it was run against the live orchard9-k3sf deployment and its output recorded before commit. Nothing is aspirational, and the three defects found while dogfooding it are fixed rather than left for the reader: * the backup check sorted ALL backups by timestamp and selected a restore-canary run (20 items, 1 volume) — it would have "passed" while telling you nothing about the fleet. Now filters on the schedule label. * the certificate check dialled the hostname, which fails on a workstation behind a split-DNS resolver. Now connects by IP with SNI. * a prose line was sitting inside a bash fence. Sections 1-8 verify what is deployed today. Section 9 is deliberately separate: HTTP metrics, the operator/data credential split, and structured logs are committed and tested but INERT until an image roll, so their absence is not mistaken for a regression. Five dashboard panels are legitimately empty for the same reason and the doc says which. Carries the two measurement traps this deploy actually produced, because both generated false alarms: port-forward needs sleep 8 (a shorter wait races the bind and reads like a dead node), and pod-to-pod reachability must not be probed with /dev/tcp under sh (dash has no /dev/tcp, so an OPEN port reports refused - that briefly looked like a cluster partition). Also records the known-red reseed tests as environmental rather than regressions: bisect against the preceding commit shows all three fail identically there, on the first ack=quorum write ~1s after the gRPC listeners bind and before peer ship channels exist, against the harness's own 3s client timeout. --- docs/runbooks/README.md | 3 +- docs/runbooks/deploy-verification.md | 468 +++++++++++++++++++++++++++ 2 files changed, 470 insertions(+), 1 deletion(-) create mode 100644 docs/runbooks/deploy-verification.md diff --git a/docs/runbooks/README.md b/docs/runbooks/README.md index cbf2ae7..63e9121 100644 --- a/docs/runbooks/README.md +++ b/docs/runbooks/README.md @@ -7,13 +7,14 @@ and history live in the `orchard9-k3sf` repo (`cluster-state.yaml`, | When you need to… | Open | |---|---| +| **Verify a deploy end to end** — every layer, with the command and its expected output | [`deploy-verification.md`](deploy-verification.md) | | **Respond to an incident / page** — symptom → signal → fix | [`on-call.md`](on-call.md) | | Operate the cluster — API, failover, partition, membership, scale, rolling upgrade | [`cluster.md`](cluster.md) | | Run it on Kubernetes — deploy, digest-pin, probes, scale up/down, troubleshoot pods | [`kubernetes.md`](kubernetes.md) | | Back up / restore / DR — object-store export, restore, byte-verify, query-proof, R2, rebuild | [`disaster-recovery.md`](disaster-recovery.md) | | Recover a single node / standalone engine — corrupt keyspace, WAL, lock, schema | [`../ops/recovery.md`](../ops/recovery.md) | | Size a deployment — single-node tables + the measured Ref-A cluster envelope | [`../ops/capacity-planning.md`](../ops/capacity-planning.md) | -| Read the metrics / wire dashboards & alerts | [`../ops/monitoring.md`](../ops/monitoring.md), [`../ops/grafana-dashboard.json`](../ops/grafana-dashboard.json), [`../ops/prometheus-alerts.yaml`](../ops/prometheus-alerts.yaml) | +| Read the metrics / wire dashboards & alerts | [`../ops/observability.md`](../ops/observability.md), [`../ops/grafana-tidaldb.json`](../ops/grafana-tidaldb.json), [`../ops/prometheus-alerts.yaml`](../ops/prometheus-alerts.yaml) | | Understand the live perf/topology findings (dev handoff) | [`../profiling/m12-cluster-deploy-findings.md`](../profiling/m12-cluster-deploy-findings.md) | ## Current production facts (2026-06-19) diff --git a/docs/runbooks/deploy-verification.md b/docs/runbooks/deploy-verification.md new file mode 100644 index 0000000..fdaef5c --- /dev/null +++ b/docs/runbooks/deploy-verification.md @@ -0,0 +1,468 @@ +# Deploy verification + +Walk this top to bottom. Every command here was executed against the live +`orchard9-k3sf` deployment and its output recorded — nothing is aspirational. + +Each check states **what it proves**, the command, and what you should see. If a +check fails, its **If it fails** line says where to look. + +Sections 1–8 verify what is deployed **now**. Section 9 is deliberately separate: +those checks cannot pass until a new image is rolled, and saying so up front is +the point. + +--- + +## 0. Setup + +```bash +export KUBECONFIG=~/.kube/orchard9-k3sf.yaml +export TIDAL_API_KEY=$(kubectl -n tidaldb-cluster get secret tidaldb-credentials \ + -o jsonpath='{.data.TIDAL_API_KEY}' | base64 -d) + +# Confirm you are pointed at the right cluster before anything else. +kubectl config current-context +``` + +`TIDAL_API_KEY` is the **data-plane** bearer. It is not an operator credential — +see §9.2. + +--- + +## 1. Cluster is up and converged + +**Proves:** all three voters are serving, and no node is behind or reseeding. + +```bash +kubectl -n tidaldb-cluster get pods -l app.kubernetes.io/name=tidaldb \ + -o custom-columns=NAME:.metadata.name,READY:.status.containerStatuses[0].ready,RESTARTS:.status.containerStatuses[0].restartCount +``` + +Expect three pods, `READY=true`. Restart counts are cumulative since the last +roll — a *stable* non-zero value is fine, a *climbing* one is not. + +``` +NAME READY RESTARTS +tidaldb-0 true 1 +tidaldb-1 true 0 +tidaldb-2 true 1 +``` + +Then the authoritative per-node view. **Query each pod directly** — see §1.1 for +why the aggregated view is not trustworthy here: + +```bash +for i in 0 1 2; do + PORT=$((19700+i)) + kubectl -n tidaldb-cluster port-forward tidaldb-$i $PORT:9500 >/dev/null 2>&1 & + PF=$!; sleep 8 + printf 'tidaldb-%s: ' "$i" + curl -sk --max-time 10 -H "Authorization: Bearer $TIDAL_API_KEY" \ + "https://127.0.0.1:$PORT/cluster/status/local" \ + | python3 -c " +import json,sys +d=json.load(sys.stdin) +print('region=%-11s reseed=%-5s' % (d.get('region'), d.get('reseed_required')), end='') +for r in d.get('shards') or []: + print(' g%s[app=%s lag=%s ldr=%s]' % (r.get('shard'), r.get('applied_events'), + r.get('lag_events'), r.get('leader')), end='') +print()" + kill $PF 2>/dev/null; wait $PF 2>/dev/null +done +``` + +**Pass:** every pod reports `reseed=False` and `lag=0` on every group, all naming +the same leader. Observed: + +``` +tidaldb-0: region=tidaldb-0 reseed=False g0[app=13324724 lag=0 ldr=tidaldb-2] g1[...lag=0...] g2[...lag=0...] +tidaldb-1: region=tidaldb-1 reseed=False g0[app=13324724 lag=0 ldr=tidaldb-2] g1[...lag=0...] g2[...lag=0...] +tidaldb-2: region=tidaldb-2 reseed=False g0[app=13322237 lag=0 ldr=tidaldb-2] g1[...lag=0...] g2[...lag=0...] +``` + +The 8-event difference on `g0` between the leader and its followers is the +leader's own un-shipped tail, not lag. + +**If it fails:** `reseed=True` that survives a restart is the m11p5 livelock +signature — see `docs/runbooks/cluster.md §8`. `lag` climbing on one node means +that follower is not keeping up. + +> **Sleep 8, not 4.** A shorter wait races `port-forward`'s bind and produces an +> empty body that reads exactly like a dead node. Two separate false alarms +> during this deploy came from exactly that. + +### 1.1 Known caveat: the aggregated view under-reports peers + +`GET /cluster/status` can report a peer it holds no frontier report for as +`applied_events: 0`, then derive `lag` against that zero — so a **converged** +peer appears to be the leader's entire history behind, sometimes labelled +`reachable: false` / `partitioned: true`. + +Observed on this deployment: two healthy nodes shown as `UNREACHABLE +PARTITIONED` at 13.3M lag, while every node's own `/cluster/status/local` +reported `lag=0` and pod-to-pod connectivity was open with nothing logged. + +`tidalctl cluster-status` detects that signature and prints `NO REPORT +(aggregated view; query the node directly)` instead of repeating it as lag. **Do +not open an incident on `NO REPORT` before running the §1 per-pod loop.** This is +an open reporting defect, tracked in `docs/ops/observability.md §4`. + +--- + +## 2. Public endpoint and TLS + +**Proves:** the hostname resolves publicly and serves a certificate a normal +client will accept. + +```bash +dig +short tidaldb.threesix.ai +``` + +Expect all three node IPs (order varies): + +``` +208.122.204.172 +208.122.204.173 +208.122.204.174 +``` + +If your workstation runs a split-DNS resolver (Tailscale MagicDNS will do this), +`dig` may succeed while `curl` cannot resolve. Confirm public resolution +independently: + +```bash +curl -s -H 'accept: application/dns-json' \ + 'https://cloudflare-dns.com/dns-query?name=tidaldb.threesix.ai&type=A' \ +| python3 -c "import json,sys; print([a['data'] for a in json.load(sys.stdin).get('Answer',[])])" +``` + +Certificate identity: + +```bash +# Connect by IP with SNI, so this works even when the local resolver lags. +echo | openssl s_client -connect 208.122.204.172:443 \ + -servername tidaldb.threesix.ai 2>/dev/null \ +| openssl x509 -noout -subject -issuer -enddate +``` + +**Pass:** `subject=CN=tidaldb.threesix.ai`, issuer `Let's Encrypt`, `notAfter` at +least 30 days out. Observed `notAfter=Nov 20 04:03:50 2026 GMT`. + +**If it fails:** cert-manager uses the **dns01** solver (`letsencrypt-prod`), not +http01 — http01 cannot work behind a gateway gate that rejects unknown callers, +because it rejects the ACME challenge too. Check +`kubectl -n tidaldb-cluster get certificate tidaldb-public-tls`. + +--- + +## 3. Authentication behaves correctly + +**Proves:** the data surface is credential-gated and the admin surface is not +published at all. + +Use `--resolve` if your local resolver lags; it still validates the certificate. + +```bash +R="--resolve tidaldb.threesix.ai:443:208.122.204.172" +B="https://tidaldb.threesix.ai" + +printf 'health (open) %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 $R $B/health)" +printf 'search no bearer %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 $R "$B/search?query=a&limit=1")" +printf 'search bad bearer %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 -H 'Authorization: Bearer wrong' $R "$B/search?query=a&limit=1")" +printf 'search good bearer %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 20 -H "Authorization: Bearer $TIDAL_API_KEY" $R "$B/search?query=a&limit=1")" +``` + +**Pass:** `200`, `401`, `401`, `200`. + +Admin and metrics surfaces must be **unroutable** from the internet: + +```bash +for p in /cluster/status /cluster/members /metrics /openapi.json; do + printf '%-18s %s\n' "$p" "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 $R "$B$p")" +done +``` + +**Pass:** `404` for all four. A `200` on `/cluster/status` would be leaking +leader identity, membership and seqnos to the internet; a `200` on `/metrics` +would be leaking corpus size unauthenticated. + +**If it fails:** the path allowlist lives in `k8s/cluster/ingress.yaml`. Only the +data paths are published, deliberately. + +### 3.1 A real write, end to end + +```bash +curl -s --max-time 25 -X POST \ + -H 'content-type: application/json' \ + -H "Authorization: Bearer $TIDAL_API_KEY" \ + -H 'x-tidal-ack: quorum' \ + -d '{"entity_id":999000099,"metadata":{"title":"deploy verification","category":"probe"}}' \ + $R "$B/items" -o /dev/null -w 'quorum write %{http_code}\n' +``` + +**Pass:** `201`. This is the strongest single check in the document — it proves +DNS, TLS, the gateway, the backend TLS hop, authentication, and **quorum +replication** in one request. + +--- + +## 4. Network isolation + +**Proves:** the unauthenticated metrics port and the peer plane are not reachable +from arbitrary pods. + +```bash +kubectl -n tidaldb-cluster get networkpolicy tidaldb +PIP=$(kubectl -n tidaldb-cluster get pod tidaldb-0 -o jsonpath='{.status.podIP}') + +# A pod in an unrelated namespace must NOT reach :9091. +kubectl -n threesix exec gitea-0 -- \ + sh -c "wget -qO- --timeout=5 http://$PIP:9091/metrics 2>&1 | head -1" +``` + +**Pass:** `Connection refused`. Before the policy existed this returned +`# HELP tidaldb_uptime_seconds …` from any pod in the cluster. + +The scraper must still get through: + +```bash +kubectl -n observability exec deploy/vmagent -- \ + sh -c "wget -qO- --timeout=6 http://$PIP:9091/metrics 2>/dev/null | grep -c '^tidaldb_'" +``` + +**Pass:** a few hundred series (observed `353`). + +> **Do not test pod-to-pod reachability with `/dev/tcp` under `sh`.** The +> container's `sh` is dash, which has no `/dev/tcp`, so it reports failure for a +> port that is open. That false negative briefly looked like a cluster partition +> during this deploy. Use `bash -c` explicitly. + +--- + +## 5. Metrics and dashboard + +**Proves:** the series exist with the labels the dashboard queries, and the +dashboard is loaded. + +```bash +kubectl -n observability port-forward svc/vmsingle 8428:8428 >/dev/null 2>&1 & +sleep 6 +curl -s -G 'http://127.0.0.1:8428/api/v1/series' \ + --data-urlencode 'match[]=tidaldb_health_ok' \ +| python3 -c "import json,sys; d=json.load(sys.stdin)['data']; print(len(d),'series'); print(sorted(d[0]))" +``` + +**Pass:** non-zero series, labels including `namespace`, `pod`, `container`, +`partition_id`. The dashboard's `$namespace`/`$pod` variables depend on those +exact names. + +Dashboard presence (needs the Grafana admin password): + +```bash +PW=$(kubectl -n observability get secret grafana-admin -o jsonpath='{.data.password}' | base64 -d) +kubectl -n observability port-forward deploy/grafana 3000:3000 >/dev/null 2>&1 & +sleep 6 +curl -s -u "admin:$PW" 'http://127.0.0.1:3000/api/dashboards/uid/tidaldb-overview' \ +| python3 -c " +import json,sys +d=json.load(sys.stdin); db=d['dashboard'] +print('LOADED:', db['title'], '| folder:', d['meta'].get('folderTitle'), + '| panels:', sum(1 for p in db['panels'] if p['type']!='row'))" +``` + +**Pass:** `LOADED: tidalDB — usage, errors, and cluster health | folder: Databases | panels: 13`. + +**If it fails:** the dashboard is a key on the `grafana-database-dashboards` +ConfigMap; Grafana's file provider rescans every 30s. Source of truth is +`docs/ops/grafana-tidaldb.json`. + +--- + +## 6. Logs are queryable + +**Proves:** log lines are reaching the platform. + +```bash +kubectl -n tidaldb-cluster logs tidaldb-0 --tail=20 +``` + +**Pass:** readable lines, no raw `\x1b[` escape fragments. + +> Today's deployed image emits **plain text**, so `level:error` filtering in +> VictoriaLogs does **not** work yet — the collector cannot parse the line, keeps +> the text, and stamps every entry `level=info`. Structured logging is built and +> tested but requires the image roll and `JSON_LOGS=1`; see §9.3. + +Until then, filter at the source: + +```bash +kubectl -n tidaldb-cluster logs tidaldb-0 --since=15m | grep -iE 'ERROR|WARN' +``` + +--- + +## 7. Live debugging tools work + +**Proves:** you can interrogate a running cluster without hand-rolling curl. + +```bash +cargo build -p tidalctl # or use a released binary +kubectl -n tidaldb-cluster port-forward svc/tidaldb 9500:9500 >/dev/null 2>&1 & +sleep 9 + +# --insecure is required: the client port is served with the INTERNAL cluster CA, +# whose leaf is issued for in-cluster DNS names. +./target/debug/tidalctl cluster-status \ + --url https://127.0.0.1:9500 --key "$TIDAL_API_KEY" --insecure +echo "exit=$?" +``` + +**Pass:** a leader line, a region table, a shard table. Exit `0` when converged, +`2` when not — so `tidalctl cluster-status && deploy` is a safe gate. Remember +§1.1: `NO REPORT` means the aggregated view lacks a peer report, not that the +peer is down. + +```bash +# Convergence monitor, bounded so it terminates. +./target/debug/tidalctl watch --url https://127.0.0.1:9500 \ + --key "$TIDAL_API_KEY" --insecure --interval 5 --count 3 +``` + +**Pass:** one line per tick, each ending `[ok]` or `[DEGRADED]`. + +Exit-code contract (verified): + +| Situation | Exit | +|---|---| +| converged | 0 | +| bad/absent credential | 2 | +| server unreachable | 2 | +| `--url` without a scheme | 1 | +| missing `--url` / `--path` | 1 | + +--- + +## 8. Backups + +**Proves:** a whole-fleet backup completed recently and captured every volume. + +```bash +# Select the latest backup FROM THE FLEET SCHEDULE. Sorting all backups by +# timestamp picks up restore-canary runs (20 items, 1 volume), which would +# "pass" while telling you nothing about the fleet. +B=$(kubectl -n backup-system get backup.velero.io \ + -l velero.io/schedule-name=velero-fleet-daily \ + --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1].metadata.name}') +echo "checking $B" +kubectl -n backup-system get backup.velero.io "$B" \ + -o jsonpath='phase={.status.phase} errors={.status.errors} items={.status.progress.itemsBackedUp}/{.status.progress.totalItems}{"\n"}' +kubectl -n backup-system get podvolumebackups -l velero.io/backup-name="$B" \ + --no-headers | awk '{print $2}' | sort | uniq -c +``` + +**Pass:** `phase=Completed`, no errors, and **every** PodVolumeBackup +`Completed`. Observed on `velero-fleet-daily-20260823033025`: `3708/3708 items, +48/48 Completed`. + +**Careful:** a single failed PVB marks the whole backup `PartiallyFailed` and +freezes the alert gauge, even when every volume that matters was captured. If you +trigger a manual backup to satisfy the alert, it **must** carry the schedule +label or the metric will not advance: + +The alert reads +`velero_backup_last_successful_timestamp{schedule="velero-fleet-daily"}`, so a +manual Backup **must** carry that schedule label or the gauge never moves. A +manual run without it completed 48/48 and the alert stayed critical: + +```yaml +metadata: + labels: + velero.io/schedule-name: velero-fleet-daily +``` + +--- + +## 9. Not active yet — requires an image roll + +The cluster runs `m12-boot-pull-fix-20260821`. The following are committed and +covered by tests but **inert in production** until a newer image is deployed. +They are listed so their absence is not mistaken for a regression. + +### 9.1 HTTP request/error metrics + +```bash +PIP=$(kubectl -n tidaldb-cluster get pod tidaldb-0 -o jsonpath='{.status.podIP}') +kubectl -n observability exec deploy/vmagent -- \ + sh -c "wget -qO- --timeout=6 http://$PIP:9091/metrics 2>/dev/null | grep -c tidaldb_http_requests_total" +``` + +Currently `0`. After the roll: non-zero, and the dashboard's *Request rate by +route*, *Requests by status*, *5xx ratio*, *Auth rejections* and *HTTP p99* +panels populate. Until then those five panels are legitimately empty. + +### 9.2 Operator/data credential split + +After the roll, add the key (picked up by the credential poller, **no restart**): + +```bash +kubectl -n tidaldb-cluster patch secret tidaldb-credentials --type=merge \ + -p "{\"stringData\":{\"TIDAL_ADMIN_KEY\":\"$(openssl rand -hex 32)\"}}" +``` + +Then verify the split, port-forwarded (the admin routes are not published): + +```bash +# data bearer on an operator verb -> 403 (authenticated, not authorized) +curl -sk -o /dev/null -w '%{http_code}\n' -X POST \ + -H "Authorization: Bearer $TIDAL_API_KEY" -d '{"region":"tidaldb-1"}' \ + https://127.0.0.1:9500/cluster/promote + +# admin key -> anything except 401/403 +ADMIN=$(kubectl -n tidaldb-cluster get secret tidaldb-credentials \ + -o jsonpath='{.data.TIDAL_ADMIN_KEY}' | base64 -d) +curl -sk -o /dev/null -w '%{http_code}\n' -X POST \ + -H "Authorization: Bearer $ADMIN" -d '{"region":"tidaldb-1"}' \ + https://127.0.0.1:9500/cluster/heal +``` + +**Pass:** `403` then not-`403`. Before this key exists, the data bearer can +remove members and transfer shards — that is the exposure the split closes. + +### 9.3 Structured logs + +Add `JSON_LOGS=1` to the StatefulSet, then: + +```bash +kubectl -n tidaldb-cluster logs tidaldb-0 --tail=5 +``` + +**Pass:** one JSON object per line carrying `ts`, `level`, `service`, `msg`, and +`request_id` inside a request span. `level:error` then works in VictoriaLogs. + +--- + +## Known-red, not deploy blockers + +Verified by bisect against the preceding commit — these are **not** caused by the +observability or credential work: + +| Item | State | +|---|---| +| `mp_follower_reseeds_via_snapshot_after_compaction` | red on this workstation | +| `mp_quarantined_node_reseeds_without_wipe` | red on this workstation | +| `mp_multi_group_node_converges_after_reseeding_several_groups` | red on this workstation | +| `mp_graceful_rolling_restart_under_load_no_reseed` | red on baseline `main` | + +The first three fail on the *first* `ack=quorum` write, roughly a second after +the gRPC listeners bind and before peer ship channels are established, against +the harness's own 3s client timeout — a startup race that loses on a loaded +machine. All three fail identically at the commit before this work. The +workstation they were run on is at 100% disk capacity, which is the most likely +aggravating factor. Re-run on a machine with headroom before treating them as a +code defect. + +Also open, unrelated to this deploy: + +- `kubectl apply -k k8s/cluster/` **downgrades** live limits (`cpu:3/7Gi` in the + cluster vs `2/6Gi` in the manifest). Fix the manifest or apply selectively. +- The `tidaldb` namespace standalone Deployment is live `1/1` while the fleet + record calls it superseded. Nothing routes to it.