tidaldb/docs/runbooks/deploy-verification.md

Deploy verification

Walk this top to bottom. Every command here was executed against the live orchard9-k3sf deployment and its output recorded — nothing is aspirational.

Each check states what it proves, the command, and what you should see. If a check fails, its If it fails line says where to look.

Sections 1–8 verify what is deployed now. Section 9 is deliberately separate: those checks cannot pass until a new image is rolled, and saying so up front is the point.

---

0. Setup

export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
export TIDAL_API_KEY=$(kubectl -n tidaldb-cluster get secret tidaldb-credentials \
  -o jsonpath='{.data.TIDAL_API_KEY}' | base64 -d)

# Confirm you are pointed at the right cluster before anything else.
kubectl config current-context

TIDAL_API_KEY is the data-plane bearer. It is not an operator credential — see §9.2.

---

1. Cluster is up and converged

Proves: all three voters are serving, and no node is behind or reseeding.

kubectl -n tidaldb-cluster get pods -l app.kubernetes.io/name=tidaldb \
  -o custom-columns=NAME:.metadata.name,READY:.status.containerStatuses[0].ready,RESTARTS:.status.containerStatuses[0].restartCount

Expect three pods, READY=true. Restart counts are cumulative since the last roll — a stable non-zero value is fine, a climbing one is not.

NAME        READY   RESTARTS
tidaldb-0   true    1
tidaldb-1   true    0
tidaldb-2   true    1

Then the authoritative per-node view. Query each pod directly — see §1.1 for why the aggregated view is not trustworthy here:

for i in 0 1 2; do
  PORT=$((19700+i))
  kubectl -n tidaldb-cluster port-forward tidaldb-$i $PORT:9500 >/dev/null 2>&1 &
  PF=$!; sleep 8
  printf 'tidaldb-%s: ' "$i"
  curl -sk --max-time 10 -H "Authorization: Bearer $TIDAL_API_KEY" \
    "https://127.0.0.1:$PORT/cluster/status/local" \
  | python3 -c "
import json,sys
d=json.load(sys.stdin)
print('region=%-11s reseed=%-5s' % (d.get('region'), d.get('reseed_required')), end='')
for r in d.get('shards') or []:
    print(' g%s[app=%s lag=%s ldr=%s]' % (r.get('shard'), r.get('applied_events'),
          r.get('lag_events'), r.get('leader')), end='')
print()"
  kill $PF 2>/dev/null; wait $PF 2>/dev/null
done

Pass: every pod reports reseed=False and lag=0 on every group, all naming the same leader. Observed:

tidaldb-0: region=tidaldb-0   reseed=False g0[app=13324724 lag=0 ldr=tidaldb-2] g1[...lag=0...] g2[...lag=0...]
tidaldb-1: region=tidaldb-1   reseed=False g0[app=13324724 lag=0 ldr=tidaldb-2] g1[...lag=0...] g2[...lag=0...]
tidaldb-2: region=tidaldb-2   reseed=False g0[app=13322237 lag=0 ldr=tidaldb-2] g1[...lag=0...] g2[...lag=0...]

The 8-event difference on g0 between the leader and its followers is the leader's own un-shipped tail, not lag.

If it fails: reseed=True that survives a restart is the m11p5 livelock signature — see docs/runbooks/cluster.md §8. lag climbing on one node means that follower is not keeping up.

Sleep 8, not 4. A shorter wait races port-forward's bind and produces an empty body that reads exactly like a dead node. Two separate false alarms during this deploy came from exactly that.

1.1 Known caveat: the aggregated view under-reports peers

GET /cluster/status can report a peer it holds no frontier report for as applied_events: 0, then derive lag against that zero — so a converged peer appears to be the leader's entire history behind, sometimes labelled reachable: false / partitioned: true.

Observed on this deployment: two healthy nodes shown as UNREACHABLE PARTITIONED at 13.3M lag, while every node's own /cluster/status/local reported lag=0 and pod-to-pod connectivity was open with nothing logged.

tidalctl cluster-status detects that signature and prints NO REPORT (aggregated view; query the node directly) instead of repeating it as lag. Do not open an incident on NO REPORT before running the §1 per-pod loop. This is an open reporting defect, tracked in docs/ops/observability.md §4.

---

2. Public endpoint and TLS

Proves: the hostname resolves publicly and serves a certificate a normal client will accept.

dig +short tidaldb.threesix.ai

Expect all three node IPs (order varies):

208.122.204.172
208.122.204.173
208.122.204.174

If your workstation runs a split-DNS resolver (Tailscale MagicDNS will do this), dig may succeed while curl cannot resolve. Confirm public resolution independently:

curl -s -H 'accept: application/dns-json' \
  'https://cloudflare-dns.com/dns-query?name=tidaldb.threesix.ai&type=A' \
| python3 -c "import json,sys; print([a['data'] for a in json.load(sys.stdin).get('Answer',[])])"

Certificate identity:

# Connect by IP with SNI, so this works even when the local resolver lags.
echo | openssl s_client -connect 208.122.204.172:443 \
  -servername tidaldb.threesix.ai 2>/dev/null \
| openssl x509 -noout -subject -issuer -enddate

Pass: subject=CN=tidaldb.threesix.ai, issuer Let's Encrypt, notAfter at least 30 days out. Observed notAfter=Nov 20 04:03:50 2026 GMT.

If it fails: cert-manager uses the dns01 solver (letsencrypt-prod), not http01 — http01 cannot work behind a gateway gate that rejects unknown callers, because it rejects the ACME challenge too. Check kubectl -n tidaldb-cluster get certificate tidaldb-public-tls.

---

3. Authentication behaves correctly

Proves: the data surface is credential-gated and the admin surface is not published at all.

Use --resolve if your local resolver lags; it still validates the certificate.

R="--resolve tidaldb.threesix.ai:443:208.122.204.172"
B="https://tidaldb.threesix.ai"

printf 'health (open)      %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 $R $B/health)"
printf 'search no bearer   %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 $R "$B/search?query=a&limit=1")"
printf 'search bad bearer  %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 -H 'Authorization: Bearer wrong' $R "$B/search?query=a&limit=1")"
printf 'search good bearer %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 20 -H "Authorization: Bearer $TIDAL_API_KEY" $R "$B/search?query=a&limit=1")"

Pass: 200, 401, 401, 200.

Admin and metrics surfaces must be unroutable from the internet:

for p in /cluster/status /cluster/members /metrics /openapi.json; do
  printf '%-18s %s\n' "$p" "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 $R "$B$p")"
done

Pass: 404 for all four. A 200 on /cluster/status would be leaking leader identity, membership and seqnos to the internet; a 200 on /metrics would be leaking corpus size unauthenticated.

If it fails: the path allowlist lives in k8s/cluster/ingress.yaml. Only the data paths are published, deliberately.

3.1 A real write, end to end

curl -s --max-time 25 -X POST \
  -H 'content-type: application/json' \
  -H "Authorization: Bearer $TIDAL_API_KEY" \
  -H 'x-tidal-ack: quorum' \
  -d '{"entity_id":999000099,"metadata":{"title":"deploy verification","category":"probe"}}' \
  $R "$B/items" -o /dev/null -w 'quorum write %{http_code}\n'

Pass: 201. This is the strongest single check in the document — it proves DNS, TLS, the gateway, the backend TLS hop, authentication, and quorum replication in one request.

---

4. Network isolation

Proves: the unauthenticated metrics port and the peer plane are not reachable from arbitrary pods.

kubectl -n tidaldb-cluster get networkpolicy tidaldb
PIP=$(kubectl -n tidaldb-cluster get pod tidaldb-0 -o jsonpath='{.status.podIP}')

# A pod in an unrelated namespace must NOT reach :9091.
kubectl -n threesix exec gitea-0 -- \
  sh -c "wget -qO- --timeout=5 http://$PIP:9091/metrics 2>&1 | head -1"

Pass: Connection refused. Before the policy existed this returned # HELP tidaldb_uptime_seconds … from any pod in the cluster.

The scraper must still get through:

kubectl -n observability exec deploy/vmagent -- \
  sh -c "wget -qO- --timeout=6 http://$PIP:9091/metrics 2>/dev/null | grep -c '^tidaldb_'"

Pass: a few hundred series (observed 353).

Do not test pod-to-pod reachability with /dev/tcp under sh. The container's sh is dash, which has no /dev/tcp, so it reports failure for a port that is open. That false negative briefly looked like a cluster partition during this deploy. Use bash -c explicitly.

---

5. Metrics and dashboard

Proves: the series exist with the labels the dashboard queries, and the dashboard is loaded.

kubectl -n observability port-forward svc/vmsingle 8428:8428 >/dev/null 2>&1 &
sleep 6
curl -s -G 'http://127.0.0.1:8428/api/v1/series' \
  --data-urlencode 'match[]=tidaldb_health_ok' \
| python3 -c "import json,sys; d=json.load(sys.stdin)['data']; print(len(d),'series'); print(sorted(d[0]))"

Pass: non-zero series, labels including namespace, pod, container, partition_id. The dashboard's $namespace/$pod variables depend on those exact names.

Dashboard presence (needs the Grafana admin password):

PW=$(kubectl -n observability get secret grafana-admin -o jsonpath='{.data.password}' | base64 -d)
kubectl -n observability port-forward deploy/grafana 3000:3000 >/dev/null 2>&1 &
sleep 6
curl -s -u "admin:$PW" 'http://127.0.0.1:3000/api/dashboards/uid/tidaldb-overview' \
| python3 -c "
import json,sys
d=json.load(sys.stdin); db=d['dashboard']
print('LOADED:', db['title'], '| folder:', d['meta'].get('folderTitle'),
      '| panels:', sum(1 for p in db['panels'] if p['type']!='row'))"

Pass: LOADED: tidalDB — usage, errors, and cluster health | folder: Databases | panels: 13.

If it fails: the dashboard is a key on the grafana-database-dashboards ConfigMap; Grafana's file provider rescans every 30s. Source of truth is docs/ops/grafana-tidaldb.json.

---

6. Logs are queryable

Proves: log lines are reaching the platform.

kubectl -n tidaldb-cluster logs tidaldb-0 --tail=20

Pass: readable lines, no raw \x1b[ escape fragments.

Today's deployed image emits plain text, so level:error filtering in VictoriaLogs does not work yet — the collector cannot parse the line, keeps the text, and stamps every entry level=info. Structured logging is built and tested but requires the image roll and JSON_LOGS=1; see §9.3.

Until then, filter at the source:

kubectl -n tidaldb-cluster logs tidaldb-0 --since=15m | grep -iE 'ERROR|WARN'

---

7. Live debugging tools work

Proves: you can interrogate a running cluster without hand-rolling curl.

cargo build -p tidalctl        # or use a released binary
kubectl -n tidaldb-cluster port-forward svc/tidaldb 9500:9500 >/dev/null 2>&1 &
sleep 9

# --insecure is required: the client port is served with the INTERNAL cluster CA,
# whose leaf is issued for in-cluster DNS names.
./target/debug/tidalctl cluster-status \
  --url https://127.0.0.1:9500 --key "$TIDAL_API_KEY" --insecure
echo "exit=$?"

Pass: a leader line, a region table, a shard table. Exit 0 when converged, 2 when not — so tidalctl cluster-status && deploy is a safe gate. Remember §1.1: NO REPORT means the aggregated view lacks a peer report, not that the peer is down.

# Convergence monitor, bounded so it terminates.
./target/debug/tidalctl watch --url https://127.0.0.1:9500 \
  --key "$TIDAL_API_KEY" --insecure --interval 5 --count 3

Pass: one line per tick, each ending [ok] or [DEGRADED].

Exit-code contract (verified):

Situation	Exit
converged	0
bad/absent credential	2
server unreachable	2
--url without a scheme	1
missing --url / --path	1

---

8. Backups

Proves: a whole-fleet backup completed recently and captured every volume.

# Select the latest backup FROM THE FLEET SCHEDULE. Sorting all backups by
# timestamp picks up restore-canary runs (20 items, 1 volume), which would
# "pass" while telling you nothing about the fleet.
B=$(kubectl -n backup-system get backup.velero.io \
      -l velero.io/schedule-name=velero-fleet-daily \
      --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1].metadata.name}')
echo "checking $B"
kubectl -n backup-system get backup.velero.io "$B" \
  -o jsonpath='phase={.status.phase} errors={.status.errors} items={.status.progress.itemsBackedUp}/{.status.progress.totalItems}{"\n"}'
kubectl -n backup-system get podvolumebackups -l velero.io/backup-name="$B" \
  --no-headers | awk '{print $2}' | sort | uniq -c

Pass: phase=Completed, no errors, and every PodVolumeBackup Completed. Observed on velero-fleet-daily-20260823033025: 3708/3708 items, 48/48 Completed.

Careful: a single failed PVB marks the whole backup PartiallyFailed and freezes the alert gauge, even when every volume that matters was captured. If you trigger a manual backup to satisfy the alert, it must carry the schedule label or the metric will not advance:

The alert reads velero_backup_last_successful_timestamp{schedule="velero-fleet-daily"}, so a manual Backup must carry that schedule label or the gauge never moves. A manual run without it completed 48/48 and the alert stayed critical:

metadata:
  labels:
    velero.io/schedule-name: velero-fleet-daily

---

9. Not active yet — requires an image roll

The cluster runs m12-boot-pull-fix-20260821. The following are committed and covered by tests but inert in production until a newer image is deployed. They are listed so their absence is not mistaken for a regression.

9.1 HTTP request/error metrics

PIP=$(kubectl -n tidaldb-cluster get pod tidaldb-0 -o jsonpath='{.status.podIP}')
kubectl -n observability exec deploy/vmagent -- \
  sh -c "wget -qO- --timeout=6 http://$PIP:9091/metrics 2>/dev/null | grep -c tidaldb_http_requests_total"

Currently 0. After the roll: non-zero, and the dashboard's Request rate by route, Requests by status, 5xx ratio, Auth rejections and HTTP p99 panels populate. Until then those five panels are legitimately empty.

9.2 Operator/data credential split

After the roll, add the key (picked up by the credential poller, no restart):

kubectl -n tidaldb-cluster patch secret tidaldb-credentials --type=merge \
  -p "{\"stringData\":{\"TIDAL_ADMIN_KEY\":\"$(openssl rand -hex 32)\"}}"

Then verify the split, port-forwarded (the admin routes are not published):

# data bearer on an operator verb -> 403 (authenticated, not authorized)
curl -sk -o /dev/null -w '%{http_code}\n' -X POST \
  -H "Authorization: Bearer $TIDAL_API_KEY" -d '{"region":"tidaldb-1"}' \
  https://127.0.0.1:9500/cluster/promote

# admin key -> anything except 401/403
ADMIN=$(kubectl -n tidaldb-cluster get secret tidaldb-credentials \
  -o jsonpath='{.data.TIDAL_ADMIN_KEY}' | base64 -d)
curl -sk -o /dev/null -w '%{http_code}\n' -X POST \
  -H "Authorization: Bearer $ADMIN" -d '{"region":"tidaldb-1"}' \
  https://127.0.0.1:9500/cluster/heal

Pass: 403 then not-403. Before this key exists, the data bearer can remove members and transfer shards — that is the exposure the split closes.

9.3 Structured logs

Add JSON_LOGS=1 to the StatefulSet, then:

kubectl -n tidaldb-cluster logs tidaldb-0 --tail=5

Pass: one JSON object per line carrying ts, level, service, msg, and request_id inside a request span. level:error then works in VictoriaLogs.

---

Known-red, not deploy blockers

Verified by bisect against the preceding commit — these are not caused by the observability or credential work:

Item	State
mp_follower_reseeds_via_snapshot_after_compaction	red on this workstation
mp_quarantined_node_reseeds_without_wipe	red on this workstation
mp_multi_group_node_converges_after_reseeding_several_groups	red on this workstation
mp_graceful_rolling_restart_under_load_no_reseed	red on baseline main

The first three fail on the first ack=quorum write, roughly a second after the gRPC listeners bind and before peer ship channels are established, against the harness's own 3s client timeout — a startup race that loses on a loaded machine. All three fail identically at the commit before this work. The workstation they were run on is at 100% disk capacity, which is the most likely aggravating factor. Re-run on a machine with headroom before treating them as a code defect.

Also open, unrelated to this deploy:

• kubectl apply -k k8s/cluster/ downgrades live limits (cpu:3/7Gi in the cluster vs 2/6Gi in the manifest). Fix the manifest or apply selectively.
• The tidaldb namespace standalone Deployment is live 1/1 while the fleet record calls it superseded. Nothing routes to it.