tidaldb/docs/runbooks/deploy-verification.md

Deploy verification

Walk this top to bottom. Every command here was executed against the live orchard9-k3sf deployment and its output recorded — nothing is aspirational.

Each check states what it proves, the command, and what you should see. If a check fails, its If it fails line says where to look.

Sections 1–8 verify what is deployed now. Section 9 covers operator authority plus the two features that are committed but inert on the running image — kept separate so their absence is not mistaken for a regression.

Run it automatically first

Every check below is also a Playwright test that asserts the same thing and records what it observed. Run that first; walk the manual steps when something fails, or when you want to see a layer for yourself.

export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
npm install && npx playwright install chromium   # first time only
cargo build -p tidalctl                          # section 7 needs the binary

npm run test:e2e:list     # discovery: syntax, imports, registration
npm run test:e2e:smoke    # cluster plane + public plane + a quorum write
npm run test:e2e          # the whole runbook, ~40 s

Credentials are read from the cluster by globalSetup, so nothing is pasted into a shell. A missing prerequisite fails loudly rather than skipping a check. The HTML report at playwright-report/ carries the transcript of every command the suite ran, which is the evidence trail this document used to describe in prose.

There is one deliberate asymmetry. The manual steps tell you to sleep 8 after a kubectl port-forward; the harness polls the port until it accepts a connection instead. Same fix, deterministic rather than empirical — and it is why the automated pass takes 40 s where the manual walk takes several minutes.

A stakeholder walkthrough of the same evidence lives in demo/ — see demo/storyboard.md for what it shows and demo/visual-audit.md for how each frame was reviewed.

---

0. Setup

export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
export TIDAL_API_KEY=$(kubectl -n tidaldb-cluster get secret tidaldb-credentials \
  -o jsonpath='{.data.TIDAL_API_KEY}' | base64 -d)

# Confirm you are pointed at the right cluster before anything else.
kubectl config current-context

TIDAL_API_KEY is the data-plane bearer. It is not an operator credential — see §9.2.

---

1. Cluster is up and converged

Proves: all three voters are serving, and no node is behind or reseeding.

kubectl -n tidaldb-cluster get pods -l app.kubernetes.io/name=tidaldb \
  -o custom-columns=NAME:.metadata.name,READY:.status.containerStatuses[0].ready,RESTARTS:.status.containerStatuses[0].restartCount

Expect three pods, READY=true. Restart counts are cumulative since the last roll — a stable non-zero value is fine, a climbing one is not.

NAME        READY   RESTARTS
tidaldb-0   true    1
tidaldb-1   true    0
tidaldb-2   true    1

Then the authoritative per-node view. Query each pod directly — see §1.1 for why the aggregated view is not trustworthy here:

for i in 0 1 2; do
  PORT=$((19700+i))
  kubectl -n tidaldb-cluster port-forward tidaldb-$i $PORT:9500 >/dev/null 2>&1 &
  PF=$!; sleep 8
  printf 'tidaldb-%s: ' "$i"
  curl -sk --max-time 10 -H "Authorization: Bearer $TIDAL_API_KEY" \
    "https://127.0.0.1:$PORT/cluster/status/local" \
  | python3 -c "
import json,sys
d=json.load(sys.stdin)
print('region=%-11s reseed=%-5s' % (d.get('region'), d.get('reseed_required')), end='')
for r in d.get('shards') or []:
    print(' g%s[app=%s lag=%s ldr=%s]' % (r.get('shard'), r.get('applied_events'),
          r.get('lag_events'), r.get('leader')), end='')
print()"
  kill $PF 2>/dev/null; wait $PF 2>/dev/null
done

Pass: every pod reports reseed=False and lag=0 on every group, all naming the same leader. Observed:

tidaldb-0: region=tidaldb-0   reseed=False g0[app=13324724 lag=0 ldr=tidaldb-2] g1[...lag=0...] g2[...lag=0...]
tidaldb-1: region=tidaldb-1   reseed=False g0[app=13324724 lag=0 ldr=tidaldb-2] g1[...lag=0...] g2[...lag=0...]
tidaldb-2: region=tidaldb-2   reseed=False g0[app=13322237 lag=0 ldr=tidaldb-2] g1[...lag=0...] g2[...lag=0...]

The 8-event difference on g0 between the leader and its followers is the leader's own un-shipped tail, not lag.

If it fails: reseed=True that survives a restart is the m11p5 livelock signature — see docs/runbooks/cluster.md §8. lag climbing on one node means that follower is not keeping up.

Sleep 8, not 4. A shorter wait races port-forward's bind and produces an empty body that reads exactly like a dead node. Two separate false alarms during this deploy came from exactly that.

1.1 Known caveat: the aggregated view under-reports peers

GET /cluster/status can report a peer it holds no frontier report for as applied_events: 0, then derive lag against that zero — so a converged peer appears to be the leader's entire history behind, sometimes labelled reachable: false / partitioned: true.

Observed on this deployment: two healthy nodes shown as UNREACHABLE PARTITIONED at 13.3M lag, while every node's own /cluster/status/local reported lag=0 and pod-to-pod connectivity was open with nothing logged.

tidalctl cluster-status detects that signature and prints NO REPORT (aggregated view; query the node directly) instead of repeating it as lag. Do not open an incident on NO REPORT before running the §1 per-pod loop. This is an open reporting defect, tracked in docs/ops/observability.md §4.

---

2. Public endpoint and TLS

Proves: the hostname resolves publicly and serves a certificate a normal client will accept.

dig +short tidaldb.threesix.ai

Expect all three node IPs (order varies):

208.122.204.172
208.122.204.173
208.122.204.174

If your workstation runs a split-DNS resolver (Tailscale MagicDNS will do this), dig may succeed while curl cannot resolve. Confirm public resolution independently:

curl -s -H 'accept: application/dns-json' \
  'https://cloudflare-dns.com/dns-query?name=tidaldb.threesix.ai&type=A' \
| python3 -c "import json,sys; print([a['data'] for a in json.load(sys.stdin).get('Answer',[])])"

Certificate identity:

# Connect by IP with SNI, so this works even when the local resolver lags.
echo | openssl s_client -connect 208.122.204.172:443 \
  -servername tidaldb.threesix.ai 2>/dev/null \
| openssl x509 -noout -subject -issuer -enddate

Pass: subject=CN=tidaldb.threesix.ai, issuer Let's Encrypt, notAfter at least 30 days out. Observed notAfter=Nov 20 04:03:50 2026 GMT.

If it fails: cert-manager uses the dns01 solver (letsencrypt-prod), not http01 — http01 cannot work behind a gateway gate that rejects unknown callers, because it rejects the ACME challenge too. Check kubectl -n tidaldb-cluster get certificate tidaldb-public-tls.

---

3. Authentication behaves correctly

Proves: the data surface is credential-gated and the admin surface is not published at all.

Use --resolve if your local resolver lags; it still validates the certificate.

R="--resolve tidaldb.threesix.ai:443:208.122.204.172"
B="https://tidaldb.threesix.ai"

printf 'health (open)      %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 $R $B/health)"
printf 'search no bearer   %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 $R "$B/search?query=a&limit=1")"
printf 'search bad bearer  %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 -H 'Authorization: Bearer wrong' $R "$B/search?query=a&limit=1")"
printf 'search good bearer %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 20 -H "Authorization: Bearer $TIDAL_API_KEY" $R "$B/search?query=a&limit=1")"

Pass: 200, 401, 401, 200.

Admin and metrics surfaces must be unroutable from the internet:

for p in /cluster/status /cluster/members /metrics /openapi.json; do
  printf '%-18s %s\n' "$p" "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 $R "$B$p")"
done

Pass: 404 for all four. A 200 on /cluster/status would be leaking leader identity, membership and seqnos to the internet; a 200 on /metrics would be leaking corpus size unauthenticated.

If it fails: the path allowlist lives in k8s/cluster/ingress.yaml. Only the data paths are published, deliberately.

3.1 A real write, end to end

curl -s --max-time 25 -X POST \
  -H 'content-type: application/json' \
  -H "Authorization: Bearer $TIDAL_API_KEY" \
  -H 'x-tidal-ack: quorum' \
  -d '{"entity_id":999000099,"metadata":{"title":"deploy verification","category":"probe"}}' \
  $R "$B/items" -o /dev/null -w 'quorum write %{http_code}\n'

Pass: 201. This is the strongest single check in the document — it proves DNS, TLS, the gateway, the backend TLS hop, authentication, and quorum replication in one request.

---

4. Network isolation

Proves: the unauthenticated metrics port and the peer plane are not reachable from arbitrary pods.

kubectl -n tidaldb-cluster get networkpolicy tidaldb
PIP=$(kubectl -n tidaldb-cluster get pod tidaldb-0 -o jsonpath='{.status.podIP}')

# A pod in an unrelated namespace must NOT reach :9091.
kubectl -n threesix exec gitea-0 -- \
  sh -c "wget -qO- --timeout=5 http://$PIP:9091/metrics 2>&1 | head -1"

Pass: Connection refused. Before the policy existed this returned # HELP tidaldb_uptime_seconds … from any pod in the cluster.

The scraper must still get through:

kubectl -n observability exec deploy/vmagent -- \
  sh -c "wget -qO- --timeout=6 http://$PIP:9091/metrics 2>/dev/null | grep -c '^tidaldb_'"

Pass: a few hundred series (observed 353).

Do not test pod-to-pod reachability with /dev/tcp under sh. The container's sh is dash, which has no /dev/tcp, so it reports failure for a port that is open. That false negative briefly looked like a cluster partition during this deploy. Use bash -c explicitly.

---

5. Metrics and dashboard

Proves: the series exist with the labels the dashboard queries, and the dashboard is loaded.

kubectl -n observability port-forward svc/vmsingle 8428:8428 >/dev/null 2>&1 &
sleep 6
curl -s -G 'http://127.0.0.1:8428/api/v1/series' \
  --data-urlencode 'match[]=tidaldb_health_ok' \
| python3 -c "import json,sys; d=json.load(sys.stdin)['data']; print(len(d),'series'); print(sorted(d[0]))"

Pass: non-zero series, labels including namespace, pod, container, partition_id. The dashboard's $namespace/$pod variables depend on those exact names.

Dashboard presence (needs the Grafana admin password):

PW=$(kubectl -n observability get secret grafana-admin -o jsonpath='{.data.password}' | base64 -d)
kubectl -n observability port-forward deploy/grafana 3000:3000 >/dev/null 2>&1 &
sleep 6
curl -s -u "admin:$PW" 'http://127.0.0.1:3000/api/dashboards/uid/tidaldb-overview' \
| python3 -c "
import json,sys
d=json.load(sys.stdin); db=d['dashboard']
print('LOADED:', db['title'], '| folder:', d['meta'].get('folderTitle'),
      '| panels:', sum(1 for p in db['panels'] if p['type']!='row'))"

Pass: LOADED: tidalDB — usage, errors, and cluster health | folder: Databases | panels: 13.

If it fails: the dashboard is a key on the grafana-database-dashboards ConfigMap; Grafana's file provider rescans every 30s. Source of truth is docs/ops/grafana-tidaldb.json.

---

6. Logs are queryable

Proves: log lines are reaching the platform.

kubectl -n tidaldb-cluster logs tidaldb-0 --tail=20

Pass: readable lines, no raw \x1b[ escape fragments.

Today's deployed image emits plain text, so level:error filtering in VictoriaLogs does not work yet — the collector cannot parse the line, keeps the text, and stamps every entry level=info. Structured logging is built and tested but requires the image roll and JSON_LOGS=1; see §9.3.

Until then, filter at the source:

kubectl -n tidaldb-cluster logs tidaldb-0 --since=15m | grep -iE 'ERROR|WARN'

---

7. Live debugging tools work

Proves: you can interrogate a running cluster without hand-rolling curl.

cargo build -p tidalctl        # or use a released binary
kubectl -n tidaldb-cluster port-forward svc/tidaldb 9500:9500 >/dev/null 2>&1 &
sleep 9

# --insecure is required: the client port is served with the INTERNAL cluster CA,
# whose leaf is issued for in-cluster DNS names.
./target/debug/tidalctl cluster-status \
  --url https://127.0.0.1:9500 --key "$TIDAL_API_KEY" --insecure
echo "exit=$?"

Pass: a leader line, a region table, a shard table. Exit 0 when converged, 2 when not — so tidalctl cluster-status && deploy is a safe gate. Remember §1.1: NO REPORT means the aggregated view lacks a peer report, not that the peer is down.

# Convergence monitor, bounded so it terminates.
./target/debug/tidalctl watch --url https://127.0.0.1:9500 \
  --key "$TIDAL_API_KEY" --insecure --interval 5 --count 3

Pass: one line per tick, each ending [ok] or [DEGRADED].

Exit-code contract (verified):

Situation	Exit
converged	0
bad/absent credential	2
server unreachable	2
--url without a scheme	1
missing --url / --path	1

---

8. Backups

Proves: a whole-fleet backup completed recently and captured every volume.

# Select the latest backup FROM THE FLEET SCHEDULE. Sorting all backups by
# timestamp picks up restore-canary runs (20 items, 1 volume), which would
# "pass" while telling you nothing about the fleet.
B=$(kubectl -n backup-system get backup.velero.io \
      -l velero.io/schedule-name=velero-fleet-daily \
      --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1].metadata.name}')
echo "checking $B"
kubectl -n backup-system get backup.velero.io "$B" \
  -o jsonpath='phase={.status.phase} errors={.status.errors} items={.status.progress.itemsBackedUp}/{.status.progress.totalItems}{"\n"}'
kubectl -n backup-system get podvolumebackups -l velero.io/backup-name="$B" \
  --no-headers | awk '{print $2}' | sort | uniq -c

Pass: phase=Completed, no errors, and every PodVolumeBackup Completed. Observed on velero-fleet-daily-20260823033025: 3708/3708 items, 48/48 Completed.

Careful: a single failed PVB marks the whole backup PartiallyFailed and freezes the alert gauge, even when every volume that matters was captured. If you trigger a manual backup to satisfy the alert, it must carry the schedule label or the metric will not advance:

The alert reads velero_backup_last_successful_timestamp{schedule="velero-fleet-daily"}, so a manual Backup must carry that schedule label or the gauge never moves. A manual run without it completed 48/48 and the alert stayed critical:

metadata:
  labels:
    velero.io/schedule-name: velero-fleet-daily

---

9. Operator authority, and what is still inert

The cluster runs registry.threesix.ai/tidal/server:m12-admin-gate-20260823@sha256:6e220060… (pods started 2026-08-23 05:32–05:41 UTC). That image carries the operator/data credential split but predates the observability commit, so §9.1 and §9.3 below are still inert. They are listed so their absence is not mistaken for a regression.

9.2 Operator/data credential split — LIVE, verify it stays that way

Confirmed working on this deployment. The admin routes are not published, so port-forward first:

kubectl -n tidaldb-cluster port-forward svc/tidaldb 9500:9500 >/dev/null 2>&1 &
sleep 8
ADMIN=$(kubectl -n tidaldb-cluster get secret tidaldb-credentials \
  -o jsonpath='{.data.TIDAL_ADMIN_KEY}' | base64 -d)

# data bearer on an operator verb -> 403 (authenticated, NOT authorized)
curl -sk -o /dev/null -w 'data  -> promote %{http_code}\n' -X POST \
  -H "Authorization: Bearer $TIDAL_API_KEY" -H 'content-type: application/json' \
  -d '{"region":"tidaldb-1"}' https://127.0.0.1:9500/cluster/promote

# admin bearer -> authorized
curl -sk -o /dev/null -w 'admin -> promote %{http_code}\n' -X POST \
  -H "Authorization: Bearer $ADMIN" -H 'content-type: application/json' \
  -d '{"region":"tidaldb-1"}' https://127.0.0.1:9500/cluster/promote

Pass: 403 then 200. Observed exactly that. Before this key existed the data bearer could remove members and transfer shards — that is the exposure the split closes.

The key was hot-loaded with no restart, and the startup log still says it is missing. The pod booted 05:41 and logged TIDAL_ADMIN_KEY is not set; kubelet materialized the projected secret file at 05:51:43; the credential poller picked it up and the gate went live. So a TIDAL_ADMIN_KEY is not set WARN in the boot log does not mean the gate is open — test the behavior, which is why the two curls above are the actual check.

9.1 HTTP request/error metrics — still inert

PIP=$(kubectl -n tidaldb-cluster get pod tidaldb-0 -o jsonpath='{.status.podIP}')
kubectl -n observability exec deploy/vmagent -- \
  sh -c "wget -qO- --timeout=15 http://$PIP:9091/metrics | grep -c tidaldb_http_requests_total"

Currently 0 on all three pods. After the observability image is rolled: non-zero, and the dashboard's Request rate by route, Requests by status, 5xx ratio, Auth rejections and HTTP p99 panels populate. Until then those five panels are legitimately empty.

Use --timeout=15, not 6. A 6-second wget truncates this scrape on a busy node and returns zero lines, which reads exactly like "the metric is missing" — it briefly looked like one pod had stopped exporting entirely.

9.3 Structured logs

Add JSON_LOGS=1 to the StatefulSet, then:

kubectl -n tidaldb-cluster logs tidaldb-0 --tail=5

Pass: one JSON object per line carrying ts, level, service, msg, and request_id inside a request span. level:error then works in VictoriaLogs.

---

Known-red, not deploy blockers

Verified by bisect against the preceding commit — these are not caused by the observability or credential work:

Item	State
mp_follower_reseeds_via_snapshot_after_compaction	red on this workstation
mp_quarantined_node_reseeds_without_wipe	red on this workstation
mp_multi_group_node_converges_after_reseeding_several_groups	red on this workstation
mp_graceful_rolling_restart_under_load_no_reseed	red on baseline main

The first three fail on the first ack=quorum write, roughly a second after the gRPC listeners bind and before peer ship channels are established, against the harness's own 3s client timeout — a startup race that loses on a loaded machine. All three fail identically at the commit before this work. The workstation they were run on is at 100% disk capacity, which is the most likely aggravating factor. Re-run on a machine with headroom before treating them as a code defect.

Also open, unrelated to this deploy:

• kubectl apply -k k8s/cluster/ downgrades live limits (cpu:3/7Gi in the cluster vs 2/6Gi in the manifest). Fix the manifest or apply selectively.
• The tidaldb namespace standalone Deployment is live 1/1 while the fleet record calls it superseded. Nothing routes to it.