tidaldb/docs/runbooks/deploy-verification.md

# Deploy verification

Walk this top to bottom. Every command here was executed against the live
`orchard9-k3sf` deployment and its output recorded — nothing is aspirational.

Each check states **what it proves**, the command, and what you should see. If a
check fails, its **If it fails** line says where to look.

Sections 1–8 verify what is deployed **now**. Section 9 covers operator
authority plus the two features that are committed but inert on the running
image — kept separate so their absence is not mistaken for a regression.

## Run it automatically first

Every check below is also a Playwright test that asserts the same thing and
records what it observed. Run that first; walk the manual steps when something
fails, or when you want to see a layer for yourself.

```bash
export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
npm install && npx playwright install chromium   # first time only
cargo build -p tidalctl                          # section 7 needs the binary

npm run test:e2e:list     # discovery: syntax, imports, registration
npm run test:e2e:smoke    # cluster plane + public plane + a quorum write
npm run test:e2e          # the whole runbook, ~40 s
```

Credentials are read from the cluster by `globalSetup`, so nothing is pasted
into a shell. A missing prerequisite fails loudly rather than skipping a check.
The HTML report at `playwright-report/` carries the transcript of every command
the suite ran, which is the evidence trail this document used to describe in
prose.

There is one deliberate asymmetry. The manual steps tell you to `sleep 8` after
a `kubectl port-forward`; the harness polls the port until it accepts a
connection instead. Same fix, deterministic rather than empirical — and it is
why the automated pass takes 40 s where the manual walk takes several minutes.

A stakeholder walkthrough of the same evidence lives in `demo/` — see
`demo/storyboard.md` for what it shows and `demo/visual-audit.md` for how each
frame was reviewed.

---

## 0. Setup

```bash
export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
export TIDAL_API_KEY=$(kubectl -n tidaldb-cluster get secret tidaldb-credentials \
  -o jsonpath='{.data.TIDAL_API_KEY}' | base64 -d)

# Confirm you are pointed at the right cluster before anything else.
kubectl config current-context
```

`TIDAL_API_KEY` is the **data-plane** bearer. It is not an operator credential —
see §9.2.

---

## 1. Cluster is up and converged

**Proves:** all three voters are serving, and no node is behind or reseeding.

```bash
kubectl -n tidaldb-cluster get pods -l app.kubernetes.io/name=tidaldb \
  -o custom-columns=NAME:.metadata.name,READY:.status.containerStatuses[0].ready,RESTARTS:.status.containerStatuses[0].restartCount
```

Expect three pods, `READY=true`. Restart counts are cumulative since the last
roll — a *stable* non-zero value is fine, a *climbing* one is not.

```
NAME        READY   RESTARTS
tidaldb-0   true    1
tidaldb-1   true    0
tidaldb-2   true    1
```

Then the authoritative per-node view. **Query each pod directly** — see §1.1 for
why the aggregated view is not trustworthy here:

```bash
for i in 0 1 2; do
  PORT=$((19700+i))
  kubectl -n tidaldb-cluster port-forward tidaldb-$i $PORT:9500 >/dev/null 2>&1 &
  PF=$!; sleep 8
  printf 'tidaldb-%s: ' "$i"
  curl -sk --max-time 10 -H "Authorization: Bearer $TIDAL_API_KEY" \
    "https://127.0.0.1:$PORT/cluster/status/local" \
  | python3 -c "
import json,sys
d=json.load(sys.stdin)
print('region=%-11s reseed=%-5s' % (d.get('region'), d.get('reseed_required')), end='')
for r in d.get('shards') or []:
    print(' g%s[app=%s lag=%s ldr=%s]' % (r.get('shard'), r.get('applied_events'),
          r.get('lag_events'), r.get('leader')), end='')
print()"
  kill $PF 2>/dev/null; wait $PF 2>/dev/null
done
```

**Pass:** every pod reports `reseed=False` and `lag=0` on every group, all naming
the same leader. Observed:

```
tidaldb-0: region=tidaldb-0   reseed=False g0[app=13324724 lag=0 ldr=tidaldb-2] g1[...lag=0...] g2[...lag=0...]
tidaldb-1: region=tidaldb-1   reseed=False g0[app=13324724 lag=0 ldr=tidaldb-2] g1[...lag=0...] g2[...lag=0...]
tidaldb-2: region=tidaldb-2   reseed=False g0[app=13322237 lag=0 ldr=tidaldb-2] g1[...lag=0...] g2[...lag=0...]
```

The 8-event difference on `g0` between the leader and its followers is the
leader's own un-shipped tail, not lag.

**If it fails:** `reseed=True` that survives a restart is the m11p5 livelock
signature — see `docs/runbooks/cluster.md §8`. `lag` climbing on one node means
that follower is not keeping up.

> **Sleep 8, not 4.** A shorter wait races `port-forward`'s bind and produces an
> empty body that reads exactly like a dead node. Two separate false alarms
> during this deploy came from exactly that.

### 1.1 Known caveat: the aggregated view under-reports peers

`GET /cluster/status` can report a peer it holds no frontier report for as
`applied_events: 0`, then derive `lag` against that zero — so a **converged**
peer appears to be the leader's entire history behind, sometimes labelled
`reachable: false` / `partitioned: true`.

Observed on this deployment: two healthy nodes shown as `UNREACHABLE
PARTITIONED` at 13.3M lag, while every node's own `/cluster/status/local`
reported `lag=0` and pod-to-pod connectivity was open with nothing logged.

`tidalctl cluster-status` detects that signature and prints `NO REPORT
(aggregated view; query the node directly)` instead of repeating it as lag. **Do
not open an incident on `NO REPORT` before running the §1 per-pod loop.** This is
an open reporting defect, tracked in `docs/ops/observability.md §4`.

---

## 2. Public endpoint and TLS

**Proves:** the hostname resolves publicly and serves a certificate a normal
client will accept.

```bash
dig +short tidaldb.threesix.ai
```

Expect all three node IPs (order varies):

```
208.122.204.172
208.122.204.173
208.122.204.174
```

If your workstation runs a split-DNS resolver (Tailscale MagicDNS will do this),
`dig` may succeed while `curl` cannot resolve. Confirm public resolution
independently:

```bash
curl -s -H 'accept: application/dns-json' \
  'https://cloudflare-dns.com/dns-query?name=tidaldb.threesix.ai&type=A' \
| python3 -c "import json,sys; print([a['data'] for a in json.load(sys.stdin).get('Answer',[])])"
```

Certificate identity:

```bash
# Connect by IP with SNI, so this works even when the local resolver lags.
echo | openssl s_client -connect 208.122.204.172:443 \
  -servername tidaldb.threesix.ai 2>/dev/null \
| openssl x509 -noout -subject -issuer -enddate
```

**Pass:** `subject=CN=tidaldb.threesix.ai`, issuer `Let's Encrypt`, `notAfter` at
least 30 days out. Observed `notAfter=Nov 20 04:03:50 2026 GMT`.

**If it fails:** cert-manager uses the **dns01** solver (`letsencrypt-prod`), not
http01 — http01 cannot work behind a gateway gate that rejects unknown callers,
because it rejects the ACME challenge too. Check
`kubectl -n tidaldb-cluster get certificate tidaldb-public-tls`.

---

## 3. Authentication behaves correctly

**Proves:** the data surface is credential-gated and the admin surface is not
published at all.

Use `--resolve` if your local resolver lags; it still validates the certificate.

```bash
R="--resolve tidaldb.threesix.ai:443:208.122.204.172"
B="https://tidaldb.threesix.ai"

printf 'health (open)      %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 $R $B/health)"
printf 'search no bearer   %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 $R "$B/search?query=a&limit=1")"
printf 'search bad bearer  %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 -H 'Authorization: Bearer wrong' $R "$B/search?query=a&limit=1")"
printf 'search good bearer %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 20 -H "Authorization: Bearer $TIDAL_API_KEY" $R "$B/search?query=a&limit=1")"
```

**Pass:** `200`, `401`, `401`, `200`.

Admin and metrics surfaces must be **unroutable** from the internet:

```bash
for p in /cluster/status /cluster/members /metrics /openapi.json; do
  printf '%-18s %s\n' "$p" "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 $R "$B$p")"
done
```

**Pass:** `404` for all four. A `200` on `/cluster/status` would be leaking
leader identity, membership and seqnos to the internet; a `200` on `/metrics`
would be leaking corpus size unauthenticated.

**If it fails:** the path allowlist lives in `k8s/cluster/ingress.yaml`. Only the
data paths are published, deliberately.

### 3.1 A real write, end to end

```bash
curl -s --max-time 25 -X POST \
  -H 'content-type: application/json' \
  -H "Authorization: Bearer $TIDAL_API_KEY" \
  -H 'x-tidal-ack: quorum' \
  -d '{"entity_id":999000099,"metadata":{"title":"deploy verification","category":"probe"}}' \
  $R "$B/items" -o /dev/null -w 'quorum write %{http_code}\n'
```

**Pass:** `201`. This is the strongest single check in the document — it proves
DNS, TLS, the gateway, the backend TLS hop, authentication, and **quorum
replication** in one request.

---

## 4. Network isolation

**Proves:** the unauthenticated metrics port and the peer plane are not reachable
from arbitrary pods.

```bash
kubectl -n tidaldb-cluster get networkpolicy tidaldb
PIP=$(kubectl -n tidaldb-cluster get pod tidaldb-0 -o jsonpath='{.status.podIP}')

# A pod in an unrelated namespace must NOT reach :9091.
kubectl -n threesix exec gitea-0 -- \
  sh -c "wget -qO- --timeout=5 http://$PIP:9091/metrics 2>&1 | head -1"
```

**Pass:** `Connection refused`. Before the policy existed this returned
`# HELP tidaldb_uptime_seconds …` from any pod in the cluster.

The scraper must still get through:

```bash
kubectl -n observability exec deploy/vmagent -- \
  sh -c "wget -qO- --timeout=6 http://$PIP:9091/metrics 2>/dev/null | grep -c '^tidaldb_'"
```

**Pass:** a few hundred series (observed `353`).

> **Do not test pod-to-pod reachability with `/dev/tcp` under `sh`.** The
> container's `sh` is dash, which has no `/dev/tcp`, so it reports failure for a
> port that is open. That false negative briefly looked like a cluster partition
> during this deploy. Use `bash -c` explicitly.

---

## 5. Metrics and dashboard

**Proves:** the series exist with the labels the dashboard queries, and the
dashboard is loaded.

```bash
kubectl -n observability port-forward svc/vmsingle 8428:8428 >/dev/null 2>&1 &
sleep 6
curl -s -G 'http://127.0.0.1:8428/api/v1/series' \
  --data-urlencode 'match[]=tidaldb_health_ok' \
| python3 -c "import json,sys; d=json.load(sys.stdin)['data']; print(len(d),'series'); print(sorted(d[0]))"
```

**Pass:** non-zero series, labels including `namespace`, `pod`, `container`,
`partition_id`. The dashboard's `$namespace`/`$pod` variables depend on those
exact names.

Dashboard presence (needs the Grafana admin password):

```bash
PW=$(kubectl -n observability get secret grafana-admin -o jsonpath='{.data.password}' | base64 -d)
kubectl -n observability port-forward deploy/grafana 3000:3000 >/dev/null 2>&1 &
sleep 6
curl -s -u "admin:$PW" 'http://127.0.0.1:3000/api/dashboards/uid/tidaldb-overview' \
| python3 -c "
import json,sys
d=json.load(sys.stdin); db=d['dashboard']
print('LOADED:', db['title'], '| folder:', d['meta'].get('folderTitle'),
      '| panels:', sum(1 for p in db['panels'] if p['type']!='row'))"
```

**Pass:** `LOADED: tidalDB — usage, errors, and cluster health | folder: Databases | panels: 13`.

**If it fails:** the dashboard is a key on the `grafana-database-dashboards`
ConfigMap; Grafana's file provider rescans every 30s. Source of truth is
`docs/ops/grafana-tidaldb.json`.

---

## 6. Logs are queryable

**Proves:** log lines are reaching the platform.

```bash
kubectl -n tidaldb-cluster logs tidaldb-0 --tail=20
```

**Pass:** readable lines, no raw `\x1b[` escape fragments.

> Today's deployed image emits **plain text**, so `level:error` filtering in
> VictoriaLogs does **not** work yet — the collector cannot parse the line, keeps
> the text, and stamps every entry `level=info`. Structured logging is built and
> tested but requires the image roll and `JSON_LOGS=1`; see §9.3.

Until then, filter at the source:

```bash
kubectl -n tidaldb-cluster logs tidaldb-0 --since=15m | grep -iE 'ERROR|WARN'
```

---

## 7. Live debugging tools work

**Proves:** you can interrogate a running cluster without hand-rolling curl.

```bash
cargo build -p tidalctl        # or use a released binary
kubectl -n tidaldb-cluster port-forward svc/tidaldb 9500:9500 >/dev/null 2>&1 &
sleep 9

# --insecure is required: the client port is served with the INTERNAL cluster CA,
# whose leaf is issued for in-cluster DNS names.
./target/debug/tidalctl cluster-status \
  --url https://127.0.0.1:9500 --key "$TIDAL_API_KEY" --insecure
echo "exit=$?"
```

**Pass:** a leader line, a region table, a shard table. Exit `0` when converged,
`2` when not — so `tidalctl cluster-status && deploy` is a safe gate. Remember
§1.1: `NO REPORT` means the aggregated view lacks a peer report, not that the
peer is down.

```bash
# Convergence monitor, bounded so it terminates.
./target/debug/tidalctl watch --url https://127.0.0.1:9500 \
  --key "$TIDAL_API_KEY" --insecure --interval 5 --count 3
```

**Pass:** one line per tick, each ending `[ok]` or `[DEGRADED]`.

Exit-code contract (verified):

| Situation | Exit |
|---|---|
| converged | 0 |
| bad/absent credential | 2 |
| server unreachable | 2 |
| `--url` without a scheme | 1 |
| missing `--url` / `--path` | 1 |

---

## 8. Backups

**Proves:** a whole-fleet backup completed recently and captured every volume.

```bash
# Select the latest backup FROM THE FLEET SCHEDULE. Sorting all backups by
# timestamp picks up restore-canary runs (20 items, 1 volume), which would
# "pass" while telling you nothing about the fleet.
B=$(kubectl -n backup-system get backup.velero.io \
      -l velero.io/schedule-name=velero-fleet-daily \
      --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1].metadata.name}')
echo "checking $B"
kubectl -n backup-system get backup.velero.io "$B" \
  -o jsonpath='phase={.status.phase} errors={.status.errors} items={.status.progress.itemsBackedUp}/{.status.progress.totalItems}{"\n"}'
kubectl -n backup-system get podvolumebackups -l velero.io/backup-name="$B" \
  --no-headers | awk '{print $2}' | sort | uniq -c
```

**Pass:** `phase=Completed`, no errors, and **every** PodVolumeBackup
`Completed`. Observed on `velero-fleet-daily-20260823033025`: `3708/3708 items,
48/48 Completed`.

**Careful:** a single failed PVB marks the whole backup `PartiallyFailed` and
freezes the alert gauge, even when every volume that matters was captured. If you
trigger a manual backup to satisfy the alert, it **must** carry the schedule
label or the metric will not advance:

The alert reads
`velero_backup_last_successful_timestamp{schedule="velero-fleet-daily"}`, so a
manual Backup **must** carry that schedule label or the gauge never moves. A
manual run without it completed 48/48 and the alert stayed critical:

```yaml
metadata:
  labels:
    velero.io/schedule-name: velero-fleet-daily
```

---

## 9. Operator authority, and what is still inert

The cluster runs
`registry.threesix.ai/tidal/server:m12-admin-gate-20260823@sha256:6e220060…`
(pods started 2026-08-23 05:32–05:41 UTC). That image carries the
operator/data credential split but **predates** the observability commit, so
§9.1 and §9.3 below are still inert. They are listed so their absence is not
mistaken for a regression.

### 9.2 Operator/data credential split — LIVE, verify it stays that way

Confirmed working on this deployment. The admin routes are not published, so
port-forward first:

```bash
kubectl -n tidaldb-cluster port-forward svc/tidaldb 9500:9500 >/dev/null 2>&1 &
sleep 8
ADMIN=$(kubectl -n tidaldb-cluster get secret tidaldb-credentials \
  -o jsonpath='{.data.TIDAL_ADMIN_KEY}' | base64 -d)

# data bearer on an operator verb -> 403 (authenticated, NOT authorized)
curl -sk -o /dev/null -w 'data  -> promote %{http_code}\n' -X POST \
  -H "Authorization: Bearer $TIDAL_API_KEY" -H 'content-type: application/json' \
  -d '{"region":"tidaldb-1"}' https://127.0.0.1:9500/cluster/promote

# admin bearer -> authorized
curl -sk -o /dev/null -w 'admin -> promote %{http_code}\n' -X POST \
  -H "Authorization: Bearer $ADMIN" -H 'content-type: application/json' \
  -d '{"region":"tidaldb-1"}' https://127.0.0.1:9500/cluster/promote
```

**Pass:** `403` then `200`. Observed exactly that. Before this key existed the
data bearer could remove members and transfer shards — that is the exposure the
split closes.

> **The key was hot-loaded with no restart, and the startup log still says it is
> missing.** The pod booted 05:41 and logged `TIDAL_ADMIN_KEY is not set`;
> kubelet materialized the projected secret file at 05:51:43; the credential
> poller picked it up and the gate went live. So a `TIDAL_ADMIN_KEY is not set`
> WARN in the boot log does **not** mean the gate is open — test the behavior,
> which is why the two curls above are the actual check.

### 9.1 HTTP request/error metrics — still inert

```bash
PIP=$(kubectl -n tidaldb-cluster get pod tidaldb-0 -o jsonpath='{.status.podIP}')
kubectl -n observability exec deploy/vmagent -- \
  sh -c "wget -qO- --timeout=15 http://$PIP:9091/metrics | grep -c tidaldb_http_requests_total"
```

Currently `0` on all three pods. After the observability image is rolled:
non-zero, and the dashboard's *Request rate by route*, *Requests by status*,
*5xx ratio*, *Auth rejections* and *HTTP p99* panels populate. Until then those
five panels are legitimately empty.

> **Use `--timeout=15`, not 6.** A 6-second `wget` truncates this scrape on a
> busy node and returns zero lines, which reads exactly like "the metric is
> missing" — it briefly looked like one pod had stopped exporting entirely.

### 9.3 Structured logs

Add `JSON_LOGS=1` to the StatefulSet, then:

```bash
kubectl -n tidaldb-cluster logs tidaldb-0 --tail=5
```

**Pass:** one JSON object per line carrying `ts`, `level`, `service`, `msg`, and
`request_id` inside a request span. `level:error` then works in VictoriaLogs.

---

## Known-red, not deploy blockers

Verified by bisect against the preceding commit — these are **not** caused by the
observability or credential work:

| Item | State |
|---|---|
| `mp_follower_reseeds_via_snapshot_after_compaction` | red on this workstation |
| `mp_quarantined_node_reseeds_without_wipe` | red on this workstation |
| `mp_multi_group_node_converges_after_reseeding_several_groups` | red on this workstation |
| `mp_graceful_rolling_restart_under_load_no_reseed` | red on baseline `main` |

The first three fail on the *first* `ack=quorum` write, roughly a second after
the gRPC listeners bind and before peer ship channels are established, against
the harness's own 3s client timeout — a startup race that loses on a loaded
machine. All three fail identically at the commit before this work. The
workstation they were run on is at 100% disk capacity, which is the most likely
aggravating factor. Re-run on a machine with headroom before treating them as a
code defect.

Also open, unrelated to this deploy:

- `kubectl apply -k k8s/cluster/` **downgrades** live limits (`cpu:3/7Gi` in the
  cluster vs `2/6Gi` in the manifest). Fix the manifest or apply selectively.
- The `tidaldb` namespace standalone Deployment is live `1/1` while the fleet
  record calls it superseded. Nothing routes to it.