tidaldb/docs/runbooks/deploy-verification.md

# Deploy verification

Walk this top to bottom. Every command here was executed against the live
`orchard9-k3sf` deployment and its output recorded — nothing is aspirational.

Each check states **what it proves**, the command, and what you should see. If a
check fails, its **If it fails** line says where to look.

Sections 1–8 verify what is deployed **now**. Section 9 is deliberately separate:
those checks cannot pass until a new image is rolled, and saying so up front is
the point.

---

## 0. Setup

```bash
export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
export TIDAL_API_KEY=$(kubectl -n tidaldb-cluster get secret tidaldb-credentials \
  -o jsonpath='{.data.TIDAL_API_KEY}' | base64 -d)

# Confirm you are pointed at the right cluster before anything else.
kubectl config current-context
```

`TIDAL_API_KEY` is the **data-plane** bearer. It is not an operator credential —
see §9.2.

---

## 1. Cluster is up and converged

**Proves:** all three voters are serving, and no node is behind or reseeding.

```bash
kubectl -n tidaldb-cluster get pods -l app.kubernetes.io/name=tidaldb \
  -o custom-columns=NAME:.metadata.name,READY:.status.containerStatuses[0].ready,RESTARTS:.status.containerStatuses[0].restartCount
```

Expect three pods, `READY=true`. Restart counts are cumulative since the last
roll — a *stable* non-zero value is fine, a *climbing* one is not.

```
NAME        READY   RESTARTS
tidaldb-0   true    1
tidaldb-1   true    0
tidaldb-2   true    1
```

Then the authoritative per-node view. **Query each pod directly** — see §1.1 for
why the aggregated view is not trustworthy here:

```bash
for i in 0 1 2; do
  PORT=$((19700+i))
  kubectl -n tidaldb-cluster port-forward tidaldb-$i $PORT:9500 >/dev/null 2>&1 &
  PF=$!; sleep 8
  printf 'tidaldb-%s: ' "$i"
  curl -sk --max-time 10 -H "Authorization: Bearer $TIDAL_API_KEY" \
    "https://127.0.0.1:$PORT/cluster/status/local" \
  | python3 -c "
import json,sys
d=json.load(sys.stdin)
print('region=%-11s reseed=%-5s' % (d.get('region'), d.get('reseed_required')), end='')
for r in d.get('shards') or []:
    print(' g%s[app=%s lag=%s ldr=%s]' % (r.get('shard'), r.get('applied_events'),
          r.get('lag_events'), r.get('leader')), end='')
print()"
  kill $PF 2>/dev/null; wait $PF 2>/dev/null
done
```

**Pass:** every pod reports `reseed=False` and `lag=0` on every group, all naming
the same leader. Observed:

```
tidaldb-0: region=tidaldb-0   reseed=False g0[app=13324724 lag=0 ldr=tidaldb-2] g1[...lag=0...] g2[...lag=0...]
tidaldb-1: region=tidaldb-1   reseed=False g0[app=13324724 lag=0 ldr=tidaldb-2] g1[...lag=0...] g2[...lag=0...]
tidaldb-2: region=tidaldb-2   reseed=False g0[app=13322237 lag=0 ldr=tidaldb-2] g1[...lag=0...] g2[...lag=0...]
```

The 8-event difference on `g0` between the leader and its followers is the
leader's own un-shipped tail, not lag.

**If it fails:** `reseed=True` that survives a restart is the m11p5 livelock
signature — see `docs/runbooks/cluster.md §8`. `lag` climbing on one node means
that follower is not keeping up.

> **Sleep 8, not 4.** A shorter wait races `port-forward`'s bind and produces an
> empty body that reads exactly like a dead node. Two separate false alarms
> during this deploy came from exactly that.

### 1.1 Known caveat: the aggregated view under-reports peers

`GET /cluster/status` can report a peer it holds no frontier report for as
`applied_events: 0`, then derive `lag` against that zero — so a **converged**
peer appears to be the leader's entire history behind, sometimes labelled
`reachable: false` / `partitioned: true`.

Observed on this deployment: two healthy nodes shown as `UNREACHABLE
PARTITIONED` at 13.3M lag, while every node's own `/cluster/status/local`
reported `lag=0` and pod-to-pod connectivity was open with nothing logged.

`tidalctl cluster-status` detects that signature and prints `NO REPORT
(aggregated view; query the node directly)` instead of repeating it as lag. **Do
not open an incident on `NO REPORT` before running the §1 per-pod loop.** This is
an open reporting defect, tracked in `docs/ops/observability.md §4`.

---

## 2. Public endpoint and TLS

**Proves:** the hostname resolves publicly and serves a certificate a normal
client will accept.

```bash
dig +short tidaldb.threesix.ai
```

Expect all three node IPs (order varies):

```
208.122.204.172
208.122.204.173
208.122.204.174
```

If your workstation runs a split-DNS resolver (Tailscale MagicDNS will do this),
`dig` may succeed while `curl` cannot resolve. Confirm public resolution
independently:

```bash
curl -s -H 'accept: application/dns-json' \
  'https://cloudflare-dns.com/dns-query?name=tidaldb.threesix.ai&type=A' \
| python3 -c "import json,sys; print([a['data'] for a in json.load(sys.stdin).get('Answer',[])])"
```

Certificate identity:

```bash
# Connect by IP with SNI, so this works even when the local resolver lags.
echo | openssl s_client -connect 208.122.204.172:443 \
  -servername tidaldb.threesix.ai 2>/dev/null \
| openssl x509 -noout -subject -issuer -enddate
```

**Pass:** `subject=CN=tidaldb.threesix.ai`, issuer `Let's Encrypt`, `notAfter` at
least 30 days out. Observed `notAfter=Nov 20 04:03:50 2026 GMT`.

**If it fails:** cert-manager uses the **dns01** solver (`letsencrypt-prod`), not
http01 — http01 cannot work behind a gateway gate that rejects unknown callers,
because it rejects the ACME challenge too. Check
`kubectl -n tidaldb-cluster get certificate tidaldb-public-tls`.

---

## 3. Authentication behaves correctly

**Proves:** the data surface is credential-gated and the admin surface is not
published at all.

Use `--resolve` if your local resolver lags; it still validates the certificate.

```bash
R="--resolve tidaldb.threesix.ai:443:208.122.204.172"
B="https://tidaldb.threesix.ai"

printf 'health (open)      %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 $R $B/health)"
printf 'search no bearer   %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 $R "$B/search?query=a&limit=1")"
printf 'search bad bearer  %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 -H 'Authorization: Bearer wrong' $R "$B/search?query=a&limit=1")"
printf 'search good bearer %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 20 -H "Authorization: Bearer $TIDAL_API_KEY" $R "$B/search?query=a&limit=1")"
```

**Pass:** `200`, `401`, `401`, `200`.

Admin and metrics surfaces must be **unroutable** from the internet:

```bash
for p in /cluster/status /cluster/members /metrics /openapi.json; do
  printf '%-18s %s\n' "$p" "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 $R "$B$p")"
done
```

**Pass:** `404` for all four. A `200` on `/cluster/status` would be leaking
leader identity, membership and seqnos to the internet; a `200` on `/metrics`
would be leaking corpus size unauthenticated.

**If it fails:** the path allowlist lives in `k8s/cluster/ingress.yaml`. Only the
data paths are published, deliberately.

### 3.1 A real write, end to end

```bash
curl -s --max-time 25 -X POST \
  -H 'content-type: application/json' \
  -H "Authorization: Bearer $TIDAL_API_KEY" \
  -H 'x-tidal-ack: quorum' \
  -d '{"entity_id":999000099,"metadata":{"title":"deploy verification","category":"probe"}}' \
  $R "$B/items" -o /dev/null -w 'quorum write %{http_code}\n'
```

**Pass:** `201`. This is the strongest single check in the document — it proves
DNS, TLS, the gateway, the backend TLS hop, authentication, and **quorum
replication** in one request.

---

## 4. Network isolation

**Proves:** the unauthenticated metrics port and the peer plane are not reachable
from arbitrary pods.

```bash
kubectl -n tidaldb-cluster get networkpolicy tidaldb
PIP=$(kubectl -n tidaldb-cluster get pod tidaldb-0 -o jsonpath='{.status.podIP}')

# A pod in an unrelated namespace must NOT reach :9091.
kubectl -n threesix exec gitea-0 -- \
  sh -c "wget -qO- --timeout=5 http://$PIP:9091/metrics 2>&1 | head -1"
```

**Pass:** `Connection refused`. Before the policy existed this returned
`# HELP tidaldb_uptime_seconds …` from any pod in the cluster.

The scraper must still get through:

```bash
kubectl -n observability exec deploy/vmagent -- \
  sh -c "wget -qO- --timeout=6 http://$PIP:9091/metrics 2>/dev/null | grep -c '^tidaldb_'"
```

**Pass:** a few hundred series (observed `353`).

> **Do not test pod-to-pod reachability with `/dev/tcp` under `sh`.** The
> container's `sh` is dash, which has no `/dev/tcp`, so it reports failure for a
> port that is open. That false negative briefly looked like a cluster partition
> during this deploy. Use `bash -c` explicitly.

---

## 5. Metrics and dashboard

**Proves:** the series exist with the labels the dashboard queries, and the
dashboard is loaded.

```bash
kubectl -n observability port-forward svc/vmsingle 8428:8428 >/dev/null 2>&1 &
sleep 6
curl -s -G 'http://127.0.0.1:8428/api/v1/series' \
  --data-urlencode 'match[]=tidaldb_health_ok' \
| python3 -c "import json,sys; d=json.load(sys.stdin)['data']; print(len(d),'series'); print(sorted(d[0]))"
```

**Pass:** non-zero series, labels including `namespace`, `pod`, `container`,
`partition_id`. The dashboard's `$namespace`/`$pod` variables depend on those
exact names.

Dashboard presence (needs the Grafana admin password):

```bash
PW=$(kubectl -n observability get secret grafana-admin -o jsonpath='{.data.password}' | base64 -d)
kubectl -n observability port-forward deploy/grafana 3000:3000 >/dev/null 2>&1 &
sleep 6
curl -s -u "admin:$PW" 'http://127.0.0.1:3000/api/dashboards/uid/tidaldb-overview' \
| python3 -c "
import json,sys
d=json.load(sys.stdin); db=d['dashboard']
print('LOADED:', db['title'], '| folder:', d['meta'].get('folderTitle'),
      '| panels:', sum(1 for p in db['panels'] if p['type']!='row'))"
```

**Pass:** `LOADED: tidalDB — usage, errors, and cluster health | folder: Databases | panels: 13`.

**If it fails:** the dashboard is a key on the `grafana-database-dashboards`
ConfigMap; Grafana's file provider rescans every 30s. Source of truth is
`docs/ops/grafana-tidaldb.json`.

---

## 6. Logs are queryable

**Proves:** log lines are reaching the platform.

```bash
kubectl -n tidaldb-cluster logs tidaldb-0 --tail=20
```

**Pass:** readable lines, no raw `\x1b[` escape fragments.

> Today's deployed image emits **plain text**, so `level:error` filtering in
> VictoriaLogs does **not** work yet — the collector cannot parse the line, keeps
> the text, and stamps every entry `level=info`. Structured logging is built and
> tested but requires the image roll and `JSON_LOGS=1`; see §9.3.

Until then, filter at the source:

```bash
kubectl -n tidaldb-cluster logs tidaldb-0 --since=15m | grep -iE 'ERROR|WARN'
```

---

## 7. Live debugging tools work

**Proves:** you can interrogate a running cluster without hand-rolling curl.

```bash
cargo build -p tidalctl        # or use a released binary
kubectl -n tidaldb-cluster port-forward svc/tidaldb 9500:9500 >/dev/null 2>&1 &
sleep 9

# --insecure is required: the client port is served with the INTERNAL cluster CA,
# whose leaf is issued for in-cluster DNS names.
./target/debug/tidalctl cluster-status \
  --url https://127.0.0.1:9500 --key "$TIDAL_API_KEY" --insecure
echo "exit=$?"
```

**Pass:** a leader line, a region table, a shard table. Exit `0` when converged,
`2` when not — so `tidalctl cluster-status && deploy` is a safe gate. Remember
§1.1: `NO REPORT` means the aggregated view lacks a peer report, not that the
peer is down.

```bash
# Convergence monitor, bounded so it terminates.
./target/debug/tidalctl watch --url https://127.0.0.1:9500 \
  --key "$TIDAL_API_KEY" --insecure --interval 5 --count 3
```

**Pass:** one line per tick, each ending `[ok]` or `[DEGRADED]`.

Exit-code contract (verified):

| Situation | Exit |
|---|---|
| converged | 0 |
| bad/absent credential | 2 |
| server unreachable | 2 |
| `--url` without a scheme | 1 |
| missing `--url` / `--path` | 1 |

---

## 8. Backups

**Proves:** a whole-fleet backup completed recently and captured every volume.

```bash
# Select the latest backup FROM THE FLEET SCHEDULE. Sorting all backups by
# timestamp picks up restore-canary runs (20 items, 1 volume), which would
# "pass" while telling you nothing about the fleet.
B=$(kubectl -n backup-system get backup.velero.io \
      -l velero.io/schedule-name=velero-fleet-daily \
      --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1].metadata.name}')
echo "checking $B"
kubectl -n backup-system get backup.velero.io "$B" \
  -o jsonpath='phase={.status.phase} errors={.status.errors} items={.status.progress.itemsBackedUp}/{.status.progress.totalItems}{"\n"}'
kubectl -n backup-system get podvolumebackups -l velero.io/backup-name="$B" \
  --no-headers | awk '{print $2}' | sort | uniq -c
```

**Pass:** `phase=Completed`, no errors, and **every** PodVolumeBackup
`Completed`. Observed on `velero-fleet-daily-20260823033025`: `3708/3708 items,
48/48 Completed`.

**Careful:** a single failed PVB marks the whole backup `PartiallyFailed` and
freezes the alert gauge, even when every volume that matters was captured. If you
trigger a manual backup to satisfy the alert, it **must** carry the schedule
label or the metric will not advance:

The alert reads
`velero_backup_last_successful_timestamp{schedule="velero-fleet-daily"}`, so a
manual Backup **must** carry that schedule label or the gauge never moves. A
manual run without it completed 48/48 and the alert stayed critical:

```yaml
metadata:
  labels:
    velero.io/schedule-name: velero-fleet-daily
```

---

## 9. Not active yet — requires an image roll

The cluster runs `m12-boot-pull-fix-20260821`. The following are committed and
covered by tests but **inert in production** until a newer image is deployed.
They are listed so their absence is not mistaken for a regression.

### 9.1 HTTP request/error metrics

```bash
PIP=$(kubectl -n tidaldb-cluster get pod tidaldb-0 -o jsonpath='{.status.podIP}')
kubectl -n observability exec deploy/vmagent -- \
  sh -c "wget -qO- --timeout=6 http://$PIP:9091/metrics 2>/dev/null | grep -c tidaldb_http_requests_total"
```

Currently `0`. After the roll: non-zero, and the dashboard's *Request rate by
route*, *Requests by status*, *5xx ratio*, *Auth rejections* and *HTTP p99*
panels populate. Until then those five panels are legitimately empty.

### 9.2 Operator/data credential split

After the roll, add the key (picked up by the credential poller, **no restart**):

```bash
kubectl -n tidaldb-cluster patch secret tidaldb-credentials --type=merge \
  -p "{\"stringData\":{\"TIDAL_ADMIN_KEY\":\"$(openssl rand -hex 32)\"}}"
```

Then verify the split, port-forwarded (the admin routes are not published):

```bash
# data bearer on an operator verb -> 403 (authenticated, not authorized)
curl -sk -o /dev/null -w '%{http_code}\n' -X POST \
  -H "Authorization: Bearer $TIDAL_API_KEY" -d '{"region":"tidaldb-1"}' \
  https://127.0.0.1:9500/cluster/promote

# admin key -> anything except 401/403
ADMIN=$(kubectl -n tidaldb-cluster get secret tidaldb-credentials \
  -o jsonpath='{.data.TIDAL_ADMIN_KEY}' | base64 -d)
curl -sk -o /dev/null -w '%{http_code}\n' -X POST \
  -H "Authorization: Bearer $ADMIN" -d '{"region":"tidaldb-1"}' \
  https://127.0.0.1:9500/cluster/heal
```

**Pass:** `403` then not-`403`. Before this key exists, the data bearer can
remove members and transfer shards — that is the exposure the split closes.

### 9.3 Structured logs

Add `JSON_LOGS=1` to the StatefulSet, then:

```bash
kubectl -n tidaldb-cluster logs tidaldb-0 --tail=5
```

**Pass:** one JSON object per line carrying `ts`, `level`, `service`, `msg`, and
`request_id` inside a request span. `level:error` then works in VictoriaLogs.

---

## Known-red, not deploy blockers

Verified by bisect against the preceding commit — these are **not** caused by the
observability or credential work:

| Item | State |
|---|---|
| `mp_follower_reseeds_via_snapshot_after_compaction` | red on this workstation |
| `mp_quarantined_node_reseeds_without_wipe` | red on this workstation |
| `mp_multi_group_node_converges_after_reseeding_several_groups` | red on this workstation |
| `mp_graceful_rolling_restart_under_load_no_reseed` | red on baseline `main` |

The first three fail on the *first* `ack=quorum` write, roughly a second after
the gRPC listeners bind and before peer ship channels are established, against
the harness's own 3s client timeout — a startup race that loses on a loaded
machine. All three fail identically at the commit before this work. The
workstation they were run on is at 100% disk capacity, which is the most likely
aggravating factor. Re-run on a machine with headroom before treating them as a
code defect.

Also open, unrelated to this deploy:

- `kubectl apply -k k8s/cluster/` **downgrades** live limits (`cpu:3/7Gi` in the
  cluster vs `2/6Gi` in the manifest). Fix the manifest or apply selectively.
- The `tidaldb` namespace standalone Deployment is live `1/1` while the fleet
  record calls it superseded. Nothing routes to it.