tidaldb/docs/runbooks/deploy-verification.md

# Deploy verification

> ## Do NOT deploy between 03:30 and 04:00 UTC
>
> `velero-fleet-daily` fires at **03:30 UTC** and takes **9–25 min** (measured over 15 successful runs).
> A rolling restart inside that window **cancels the in-flight volume backup for every pod it restarts**, and
> the parent Velero `Backup` then stalls indefinitely — it does not fail, it sits `InProgress` and blocks the
> next scheduled run.
>
> Observed 2026-08-31: a roll at 03:50/03:55/03:56 cancelled
> `podvolumebackup …-wgtkt` (`pod=tidaldb-0 volume=data`) at 2.7 GB of 5.4 GB. The Backup froze at
> **3529/3907 items** and was still frozen 67 minutes later, heading for the 240-minute timeout that produces
> `PartiallyFailed`. The same fingerprint — Canceled/Failed PVBs clustered on one date — explains the
> `PartiallyFailed` runs on 2026-08-17, 08-18, 08-19 and 08-25.
>
> If you have already done it: confirm the previous day's backup is `Completed` with full item coverage, then
> delete the stalled Backup so the schedule unblocks. Do not wait for the timeout.


Walk this top to bottom. Every command here was executed against the live
`orchard9-k3sf` deployment and its output recorded — nothing is aspirational.

Each check states **what it proves**, the command, and what you should see. If a
check fails, its **If it fails** line says where to look.

Sections 1–8 verify what is deployed **now**. Section 9 covers operator authority
plus the one feature still inert on the running **pods** — §9.3, structured logs,
which is now requested by the manifest and waits only on a restart. §9.1 and §9.2
were in that list and are now live; the header stays because a section that
quietly drops its closed items teaches you nothing about how they closed.

## Run it automatically first

Every check below is also a Playwright test that asserts the same thing and
records what it observed. Run that first; walk the manual steps when something
fails, or when you want to see a layer for yourself.

```bash
export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
npm install && npx playwright install chromium   # first time only
cargo build -p tidalctl                          # section 7 needs the binary

npm run test:e2e:list     # discovery: syntax, imports, registration
npm run test:e2e:smoke    # cluster plane + public plane + a quorum write
npm run test:e2e          # the whole runbook, 34 checks, ~55 s
```

Credentials are read from the cluster by `globalSetup`, so nothing is pasted
into a shell. A missing prerequisite fails loudly rather than skipping a check.

### The product, not just the deployment

Everything in this runbook verifies that the deployment *answers*. None of it
verifies that the database does the thing it exists to do, so there is a second,
separate suite for ranking semantics:

```bash
npm run test:e2e:semantics   # 5 checks, ~14 s, NO cluster required
npm run app:dev              # open the same app by hand and click Like
```

It boots a throwaway standalone node, seeds a 60-item fixture catalog with
deterministic vectors from `tidal-stress`'s own generator, and asserts that a
signal write reorders the next query, that decay is applied at the declared
half-life, and that ANN results equal brute-force cosine. It has its own config
(`playwright.semantics.config.ts`) precisely because it must run with no cluster
and no credentials — `npm run test:all` runs both suites.

### Capturing the walkthrough (evidence a human can watch)

The regression suite answers "is the deployment correct?". The capture suite
answers "can someone watch the proof?" — it photographs states the regression
suite has *already asserted*, then Remotion assembles them into one video.

```bash
npm run test:demo      # 10 captures, each asserts before it photographs
npm run demo:promote   # copy captures in + stamp the LIVE image/revision
npm run demo:preflight # gate: every capture present, hash-stable, audited
npm run demo:render    # -> demo/out/deploy-verification.mp4 (90 s, 1920x1080)
```

**Run these in order, and never skip `demo:promote`.** The capture suite writes to
`test-results/demo-captures/`, not into `demo/public/captures/`, because a capture
becomes a published frame only after someone looks at it. Promotion is the step
that copies frames across and re-stamps the manifest from the live StatefulSet.

That step did not exist until 2026-08-30 — it was done by hand once — and the
consequence is the exact failure this runbook exists to prevent: the promoted
frames and the rendered video still described `m12-admin-gate-20260823` after two
image rolls, while `demo:preflight` cheerfully reported "audited perfect" about
week-old evidence. `demo:render` will happily encode stale PNGs; nothing else
notices.

`demo:promote` leaves every verdict `pending`, so `demo:preflight` **fails** until
the frames are audited. That failure is correct — clear it by opening the images
at full resolution against `demo/audience-brief.md` and then:

```bash
npm run demo:promote -- --audited "<what you actually checked>"
```

The note is stored in the manifest beside the verdict. An audit with no statement
of what was looked at is not an audit.

It deliberately never touches the deployed cluster: `skip` is declared
`permanent: true`, so seeding signals into the live corpus would be
irreversible. Two of its five checks report a product gap rather than a success
(`skip` is inert under all 27 built-in profiles; see `demo/capability-inventory.md`
BUG-018), and one cluster-targeted tripwire pins the duplicated `rank`
(BUG-020) with its root cause in `scatter_merge`.
The HTML report at `playwright-report/` carries the transcript of every command
the suite ran, which is the evidence trail this document used to describe in
prose.

There is one deliberate asymmetry. The manual steps tell you to `sleep 8` after
a `kubectl port-forward`; the harness polls the port until it accepts a
connection instead. Same fix, deterministic rather than empirical — and it is
why the automated pass takes 40 s where the manual walk takes several minutes.

A stakeholder walkthrough of the same evidence lives in `demo/` — see
`demo/storyboard.md` for what it shows and `demo/visual-audit.md` for how each
frame was reviewed.

---

## 0. Setup

```bash
export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
export TIDAL_API_KEY=$(kubectl -n tidaldb-cluster get secret tidaldb-credentials \
  -o jsonpath='{.data.TIDAL_API_KEY}' | base64 -d)

# Confirm you are pointed at the right cluster before anything else.
kubectl config current-context
```

`TIDAL_API_KEY` is the **data-plane** bearer. It is not an operator credential —
see §9.2.

---

## 1. Cluster is up and converged

**Proves:** all three voters are serving, and no node is behind or reseeding.

```bash
kubectl -n tidaldb-cluster get pods -l app.kubernetes.io/name=tidaldb \
  -o custom-columns=NAME:.metadata.name,READY:.status.containerStatuses[0].ready,RESTARTS:.status.containerStatuses[0].restartCount
```

Expect three pods, `READY=true`. Restart counts are cumulative since the last
roll — a *stable* non-zero value is fine, a *climbing* one is not.

```
NAME        READY   RESTARTS
tidaldb-0   true    1
tidaldb-1   true    0
tidaldb-2   true    1
```

Then the authoritative per-node view. **Query each pod directly** — see §1.1 for
why the aggregated view is not trustworthy here:

```bash
for i in 0 1 2; do
  PORT=$((19700+i))
  kubectl -n tidaldb-cluster port-forward tidaldb-$i $PORT:9500 >/dev/null 2>&1 &
  PF=$!; sleep 8
  printf 'tidaldb-%s: ' "$i"
  curl -sk --max-time 10 -H "Authorization: Bearer $TIDAL_API_KEY" \
    "https://127.0.0.1:$PORT/cluster/status/local" \
  | python3 -c "
import json,sys
d=json.load(sys.stdin)
print('region=%-11s reseed=%-5s' % (d.get('region'), d.get('reseed_required')), end='')
for r in d.get('shards') or []:
    print(' g%s[app=%s lag=%s ldr=%s]' % (r.get('shard'), r.get('applied_events'),
          r.get('lag_events'), r.get('leader')), end='')
print()"
  kill $PF 2>/dev/null; wait $PF 2>/dev/null
done
```

**Pass:** every pod reports `reseed=False` and `lag=0` on every group, all naming
the same leader. Observed:

```
tidaldb-0: region=tidaldb-0   reseed=False g0[app=13324724 lag=0 ldr=tidaldb-2] g1[...lag=0...] g2[...lag=0...]
tidaldb-1: region=tidaldb-1   reseed=False g0[app=13324724 lag=0 ldr=tidaldb-2] g1[...lag=0...] g2[...lag=0...]
tidaldb-2: region=tidaldb-2   reseed=False g0[app=13322237 lag=0 ldr=tidaldb-2] g1[...lag=0...] g2[...lag=0...]
```

The 8-event difference on `g0` between the leader and its followers is the
leader's own un-shipped tail, not lag.

**If it fails:** `reseed=True` that survives a restart is the m11p5 livelock
signature — see `docs/runbooks/cluster.md §8`. `lag` climbing on one node means
that follower is not keeping up.

> **Sleep 8, not 4.** A shorter wait races `port-forward`'s bind and produces an
> empty body that reads exactly like a dead node. Two separate false alarms
> during this deploy came from exactly that.

### 1.1 Known caveat: the aggregated view under-reports peers

`GET /cluster/status` can report a peer it holds no frontier report for as
`applied_events: 0`, then derive `lag` against that zero — so a **converged**
peer appears to be the leader's entire history behind, sometimes labelled
`reachable: false` / `partitioned: true`.

Observed on this deployment: two healthy nodes shown as `UNREACHABLE
PARTITIONED` at 13.3M lag, while every node's own `/cluster/status/local`
reported `lag=0` and pod-to-pod connectivity was open with nothing logged.

`tidalctl cluster-status` detects that signature and prints `NO REPORT
(aggregated view; query the node directly)` instead of repeating it as lag. **Do
not open an incident on `NO REPORT` before running the §1 per-pod loop.** This is
an open reporting defect, tracked in `docs/ops/observability.md §4`.

---

## 2. Public endpoint and TLS

**Proves:** the hostname resolves publicly and serves a certificate a normal
client will accept.

```bash
dig +short tidaldb.threesix.ai
```

Expect all three node IPs (order varies):

```
208.122.204.172
208.122.204.173
208.122.204.174
```

If your workstation runs a split-DNS resolver (Tailscale MagicDNS will do this),
`dig` may succeed while `curl` cannot resolve. Confirm public resolution
independently:

```bash
curl -s -H 'accept: application/dns-json' \
  'https://cloudflare-dns.com/dns-query?name=tidaldb.threesix.ai&type=A' \
| python3 -c "import json,sys; print([a['data'] for a in json.load(sys.stdin).get('Answer',[])])"
```

Certificate identity:

```bash
# Connect by IP with SNI, so this works even when the local resolver lags.
echo | openssl s_client -connect 208.122.204.172:443 \
  -servername tidaldb.threesix.ai 2>/dev/null \
| openssl x509 -noout -subject -issuer -enddate
```

**Pass:** `subject=CN=tidaldb.threesix.ai`, issuer `Let's Encrypt`, `notAfter` at
least 30 days out. Observed `notAfter=Nov 20 04:03:50 2026 GMT`.

**If it fails:** cert-manager uses the **dns01** solver (`letsencrypt-prod`), not
http01 — http01 cannot work behind a gateway gate that rejects unknown callers,
because it rejects the ACME challenge too. Check
`kubectl -n tidaldb-cluster get certificate tidaldb-public-tls`.

---

## 3. Authentication behaves correctly

**Proves:** the data surface is credential-gated and the admin surface is not
published at all.

Use `--resolve` if your local resolver lags; it still validates the certificate.

```bash
R="--resolve tidaldb.threesix.ai:443:208.122.204.172"
B="https://tidaldb.threesix.ai"

printf 'health (open)      %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 $R $B/health)"
printf 'search no bearer   %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 $R "$B/search?query=a&limit=1")"
printf 'search bad bearer  %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 -H 'Authorization: Bearer wrong' $R "$B/search?query=a&limit=1")"
printf 'search good bearer %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 20 -H "Authorization: Bearer $TIDAL_API_KEY" $R "$B/search?query=a&limit=1")"
```

**Pass:** `200`, `401`, `401`, `200`.

Admin and metrics surfaces must be **unroutable** from the internet:

```bash
for p in /cluster/status /cluster/members /metrics /openapi.json; do
  printf '%-18s %s\n' "$p" "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 $R "$B$p")"
done
```

**Pass:** `404` for all four. A `200` on `/cluster/status` would be leaking
leader identity, membership and seqnos to the internet; a `200` on `/metrics`
would be leaking corpus size unauthenticated.

**If it fails:** the path allowlist lives in `k8s/cluster/ingress.yaml`. Only the
data paths are published, deliberately.

### 3.1 A real write, end to end

```bash
curl -s --max-time 25 -X POST \
  -H 'content-type: application/json' \
  -H "Authorization: Bearer $TIDAL_API_KEY" \
  -H 'x-tidal-ack: quorum' \
  -d '{"entity_id":999000099,"metadata":{"title":"deploy verification","category":"probe"}}' \
  $R "$B/items" -o /dev/null -w 'quorum write %{http_code}\n'
```

**Pass:** `201`. This is the strongest single check in the document — it proves
DNS, TLS, the gateway, the backend TLS hop, authentication, and **quorum
replication** in one request.

---

## 4. Network isolation

**Proves:** the unauthenticated metrics port and the peer plane are not reachable
from arbitrary pods.

```bash
kubectl -n tidaldb-cluster get networkpolicy tidaldb
PIP=$(kubectl -n tidaldb-cluster get pod tidaldb-0 -o jsonpath='{.status.podIP}')

# A pod in an unrelated namespace must NOT reach :9091.
kubectl -n threesix exec gitea-0 -- \
  sh -c "wget -qO- --timeout=5 http://$PIP:9091/metrics 2>&1 | head -1"
```

**Pass:** `Connection refused`. Before the policy existed this returned
`# HELP tidaldb_uptime_seconds …` from any pod in the cluster.

The scraper must still get through:

```bash
kubectl -n observability exec deploy/vmagent -- \
  sh -c "wget -qO- --timeout=6 http://$PIP:9091/metrics 2>/dev/null | grep -c '^tidaldb_'"
```

**Pass:** a few hundred series (observed `353`).

> **Do not test pod-to-pod reachability with `/dev/tcp` under `sh`.** The
> container's `sh` is dash, which has no `/dev/tcp`, so it reports failure for a
> port that is open. That false negative briefly looked like a cluster partition
> during this deploy. Use `bash -c` explicitly.

---

## 5. Metrics and dashboard

**Proves:** the series exist with the labels the dashboard queries, and the
dashboard is loaded.

```bash
kubectl -n observability port-forward svc/vmsingle 8428:8428 >/dev/null 2>&1 &
sleep 6
curl -s -G 'http://127.0.0.1:8428/api/v1/series' \
  --data-urlencode 'match[]=tidaldb_health_ok' \
| python3 -c "import json,sys; d=json.load(sys.stdin)['data']; print(len(d),'series'); print(sorted(d[0]))"
```

**Pass:** non-zero series, labels including `namespace`, `pod`, `container`,
`partition_id`. The dashboard's `$namespace`/`$pod` variables depend on those
exact names.

Dashboard presence (needs the Grafana admin password):

```bash
PW=$(kubectl -n observability get secret grafana-admin -o jsonpath='{.data.password}' | base64 -d)
kubectl -n observability port-forward deploy/grafana 3000:3000 >/dev/null 2>&1 &
sleep 6
curl -s -u "admin:$PW" 'http://127.0.0.1:3000/api/dashboards/uid/tidaldb-overview' \
| python3 -c "
import json,sys
d=json.load(sys.stdin); db=d['dashboard']
print('LOADED:', db['title'], '| folder:', d['meta'].get('folderTitle'),
      '| panels:', sum(1 for p in db['panels'] if p['type']!='row'))"
```

**Pass:** `LOADED: tidalDB — usage, errors, and cluster health | folder: Databases | panels: 13`.

**If it fails:** the dashboard is a key on the `grafana-database-dashboards`
ConfigMap; Grafana's file provider rescans every 30s. Source of truth is
`docs/ops/grafana-tidaldb.json`.

---

## 6. Logs are queryable

**Proves:** log lines are reaching the platform.

```bash
kubectl -n tidaldb-cluster logs tidaldb-0 --tail=20
```

**Pass:** readable lines, no raw `\x1b[` escape fragments.

> **Today the pods still emit plain text**, so the collector cannot parse the
> line, keeps the text, and stamps every tidalDB entry `level=info` — *including
> its WARNs*. Measured 2026-08-30: `unit:tidaldb-cluster` faceted over 30m
> returned exactly one row, `{"service":"tidaldb","level":"info","n":"457"}`,
> while `level:error` matched fine for services that do emit JSON (`relay`,
> `external-secrets`). The collector is not broken. tidalDB was not speaking to
> it.
>
> The fix is a deployment change, not a code change: `JSON_LOGS=1` is now on the
> StatefulSet and takes effect on the next pod restart. See §9.3.

Until that roll lands, filter at the source:

```bash
kubectl -n tidaldb-cluster logs tidaldb-0 --since=15m | grep -iE 'ERROR|WARN'
```

---

## 7. Live debugging tools work

**Proves:** you can interrogate a running cluster without hand-rolling curl.

```bash
cargo build -p tidalctl        # or use a released binary
kubectl -n tidaldb-cluster port-forward svc/tidaldb 9500:9500 >/dev/null 2>&1 &
sleep 9

# --insecure is required: the client port is served with the INTERNAL cluster CA,
# whose leaf is issued for in-cluster DNS names.
./target/debug/tidalctl cluster-status \
  --url https://127.0.0.1:9500 --key "$TIDAL_API_KEY" --insecure
echo "exit=$?"
```

**Pass:** a leader line, a region table, a shard table. Exit `0` when converged,
`2` when not — so `tidalctl cluster-status && deploy` is a safe gate. Remember
§1.1: `NO REPORT` means the aggregated view lacks a peer report, not that the
peer is down.

```bash
# Convergence monitor, bounded so it terminates.
./target/debug/tidalctl watch --url https://127.0.0.1:9500 \
  --key "$TIDAL_API_KEY" --insecure --interval 5 --count 3
```

**Pass:** one line per tick, each ending `[ok]` or `[DEGRADED]`.

Exit-code contract (verified):

| Situation | Exit |
|---|---|
| converged | 0 |
| bad/absent credential | 2 |
| server unreachable | 2 |
| `--url` without a scheme | 1 |
| missing `--url` / `--path` | 1 |

---

## 8. Backups

**Proves:** a whole-fleet backup completed recently and captured every volume.

```bash
# Select the latest backup FROM THE FLEET SCHEDULE. Sorting all backups by
# timestamp picks up restore-canary runs (20 items, 1 volume), which would
# "pass" while telling you nothing about the fleet.
B=$(kubectl -n backup-system get backup.velero.io \
      -l velero.io/schedule-name=velero-fleet-daily \
      --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1].metadata.name}')
echo "checking $B"
kubectl -n backup-system get backup.velero.io "$B" \
  -o jsonpath='phase={.status.phase} errors={.status.errors} items={.status.progress.itemsBackedUp}/{.status.progress.totalItems}{"\n"}'
kubectl -n backup-system get podvolumebackups -l velero.io/backup-name="$B" \
  --no-headers | awk '{print $2}' | sort | uniq -c
```

**Pass:** `phase=Completed`, no errors, and **every** PodVolumeBackup
`Completed`. Observed on `velero-fleet-daily-20260823033025`: `3708/3708 items,
48/48 Completed`.

**Careful:** a single failed PVB marks the whole backup `PartiallyFailed` and
freezes the alert gauge, even when every volume that matters was captured. If you
trigger a manual backup to satisfy the alert, it **must** carry the schedule
label or the metric will not advance:

The alert reads
`velero_backup_last_successful_timestamp{schedule="velero-fleet-daily"}`, so a
manual Backup **must** carry that schedule label or the gauge never moves. A
manual run without it completed 48/48 and the alert stayed critical:

```yaml
metadata:
  labels:
    velero.io/schedule-name: velero-fleet-daily
```

---

## 9. Operator authority, and what is still inert

The cluster runs
`registry.threesix.ai/tidal/server:m12-vsc-20260830@sha256:5c18d2b1…`
(rolled 2026-08-30, built from `8aa1fbb`). That image carries the operator/data
credential split (§9.2), the HTTP request metrics (§9.1), the blob-replication
ledger, and the structured-log emitter (§9.3).

**§9.1 is LIVE.** Both `09-operator-authority.spec.ts` and the `CAP-015` capture
asserted `tidaldb_http_* = 0`; that became false and they failed, which is how
the drift surfaced within minutes instead of rotting here. Both assertions are
now inverted, so a rollback that removes the HTTP metrics fails them again.

One correction to the earlier wording, forced by the revision table below: the
HTTP metrics and the structured-log emitter came from the **same commit**,
`4766f56`, so they reached production together at **rev 30**, not at rev 31. The
`0` readings recorded in this section were taken while rev 29 was running; rev 30
held the cluster for only 2h21m and was never measured. "Went live with the
`m12-vsc` roll" is where it was *noticed*, not where it started.

Live reading after the roll: `tidaldb_http_* = 185` against a baseline of
`tidaldb_* = 552` — the baseline is what proves the scrape worked, so "present"
is distinguishable from "unscraped".

**§9.3's label was stale, and it is corrected here — with the dates, because the
earlier wording was right when written.** "Requires the image roll and
`JSON_LOGS=1`" was true up to 2026-08-30T17:38Z. It stopped being true at that
moment and nobody updated it. The StatefulSet's own revision history is the
record:

| rev | image | live from | carries the emitter? |
|---|---|---|---|
| 29 | `m12-admin-gate-20260823` | 2026-08-23T05:32Z | no — built before `4766f56` |
| 30 | `m12-vector-grow-20260830` | 2026-08-30T17:38Z | **yes** — `7c1c80d`, which descends from `4766f56` |
| 31 | `m12-vsc-20260830` | 2026-08-30T19:59Z | **yes** — `8aa1fbb` |

```bash
kubectl -n tidaldb-cluster get controllerrevision -l app.kubernetes.io/name=tidaldb \
  -o custom-columns=REV:.revision,WHEN:.metadata.creationTimestamp,IMAGE:'.data.spec.template.spec.containers[0].image'
git merge-base --is-ancestor 4766f56 8aa1fbb && echo "emitter is in the running image"
```

So the image half of the requirement has been satisfied through **two** rolls,
and the env half was never satisfied at all — neither roll added `JSON_LOGS`, and
this section kept naming the image as the blocker after the image stopped being
one. The lesson is not "the doc lied"; it is that a gap described by its
*prerequisite* rather than its *current state* survives the prerequisite being
met.

`JSON_LOGS=1` and `TIDAL_SERVICE_NAME=tidaldb` are now in
`k8s/cluster/statefulset.yaml`. **An env change does nothing to already-running
pods**, so §9.3 stays inert until the next roll restarts them — that is the one
item here still genuinely pending, and it is pending on a restart, not on code.
Container logs carry no ANSI escapes (BUG-006 resolved on this image) and
`06-logs.spec.ts` pins that in the other direction — a regression to coloured
output fails there.

### 9.2 Operator/data credential split — LIVE, verify it stays that way

Confirmed working on this deployment. The admin routes are not published, so
port-forward first:

```bash
kubectl -n tidaldb-cluster port-forward svc/tidaldb 9500:9500 >/dev/null 2>&1 &
sleep 8
ADMIN=$(kubectl -n tidaldb-cluster get secret tidaldb-credentials \
  -o jsonpath='{.data.TIDAL_ADMIN_KEY}' | base64 -d)

# data bearer on an operator verb -> 403 (authenticated, NOT authorized)
curl -sk -o /dev/null -w 'data  -> promote %{http_code}\n' -X POST \
  -H "Authorization: Bearer $TIDAL_API_KEY" -H 'content-type: application/json' \
  -d '{"region":"tidaldb-1"}' https://127.0.0.1:9500/cluster/promote

# admin bearer -> authorized
curl -sk -o /dev/null -w 'admin -> promote %{http_code}\n' -X POST \
  -H "Authorization: Bearer $ADMIN" -H 'content-type: application/json' \
  -d '{"region":"tidaldb-1"}' https://127.0.0.1:9500/cluster/promote
```

**Pass:** `403` then `200`. Observed exactly that. Before this key existed the
data bearer could remove members and transfer shards — that is the exposure the
split closes.

> **The key was hot-loaded with no restart, and the startup log still says it is
> missing.** The pod booted 05:41 and logged `TIDAL_ADMIN_KEY is not set`;
> kubelet materialized the projected secret file at 05:51:43; the credential
> poller picked it up and the gate went live. So a `TIDAL_ADMIN_KEY is not set`
> WARN in the boot log does **not** mean the gate is open — test the behavior,
> which is why the two curls above are the actual check.

### 9.1 HTTP request/error metrics — LIVE

Live since **rev 30** (`m12-vector-grow-20260830`, 2026-08-30T17:38Z) — the same
commit `4766f56` that carried the structured-log emitter. This section claimed
"still inert" for two images after that stopped being true, the same failure mode
as §9.3, in the same document. Verify it stays live:

```bash
for p in tidaldb-0 tidaldb-1 tidaldb-2; do
  PIP=$(kubectl -n tidaldb-cluster get pod "$p" -o jsonpath='{.status.podIP}')
  HTTP=$(kubectl -n observability exec deploy/vmagent -- \
    sh -c "wget -qO- --timeout=15 http://$PIP:9091/metrics | grep -c tidaldb_http_requests_total")
  ALL=$(kubectl -n observability exec deploy/vmagent -- \
    sh -c "wget -qO- --timeout=15 http://$PIP:9091/metrics | grep -c tidaldb_")
  echo "$p http=$HTTP all=$ALL"
done
```

**Pass:** non-zero `http` on all three. Observed 2026-08-31T02:19Z:
`http=19/24/18` against `all=821/878/812`. (The preamble's `185` counts every
`tidaldb_http_*` line, a wider grep — same conclusion, different denominator.)
Always read the baseline too — it is what
distinguishes "the metric is missing" from "the scrape returned nothing". The
dashboard's *Request rate by route*, *Requests by status*, *5xx ratio*, *Auth
rejections* and *HTTP p99* panels populate from these.

> **Use `--timeout=15`, not 6.** A 6-second `wget` truncates this scrape on a
> busy node and returns zero lines, which reads exactly like "the metric is
> missing" — it briefly looked like one pod had stopped exporting entirely.

### 9.3 Structured logs — requested in the manifest, live after the next roll

The capability is not missing and never was; see the §9 preamble. What was
missing is the request for it, which is now in source:

```bash
kubectl -n tidaldb-cluster get statefulset tidaldb \
  -o jsonpath='{range .spec.template.spec.containers[*].env[*]}{.name}={.value}{"\n"}{end}' \
  | grep -E 'JSON_LOGS|TIDAL_SERVICE_NAME'
```

**Pass:** `JSON_LOGS=1` and `TIDAL_SERVICE_NAME=tidaldb`. Until the roll this
prints nothing on the live object while `kubectl kustomize k8s/cluster/` shows
both — that divergence *is* the pending state, not a defect.

**After the roll — emitter side:**

```bash
kubectl -n tidaldb-cluster logs tidaldb-0 -c tidaldb --tail=5 | python3 -c "
import json,sys
for line in sys.stdin:
    json.loads(line)
print('all sampled lines are JSON')"
```

**Pass:** one JSON object per line carrying `ts`, `level`, `service`, `target`,
`msg`, and `request_id` inside a request span.

**After the roll — consumer side, which is the check that actually matters:**

```bash
kubectl -n observability exec deploy/vmagent -- sh -c \
  'wget -qO- --timeout=25 \
   --post-data="query=unit:tidaldb-cluster | stats by (service, level) count() n&start=15m" \
   http://victoria-logs:9428/select/logsql/query'
```

**Pass:** `service:"tidaldb"` now appears under more than one `level`, and
`level:error` / `level:warn` return tidalDB lines. The pre-roll reading was a
single row — `{"service":"tidaldb","level":"info","n":"457"}` — with every WARN
flattened onto info. **Do not accept the emitter check alone.** The emitter has
looked correct for weeks while the store still could not filter; that asymmetry
is precisely how this section stayed wrong.

> **Why `TIDAL_SERVICE_NAME` is set alongside it.** Vector's `normalize`
> transform (`cm/vector-cluster-config`, ns `observability`) substitutes the
> container name only `when !exists(.service)`. Plain-text lines have no
> `service`, so today's land as `tidaldb`; the JSON emitter sets its own, default
> `tidal-server`. Verified by piping a wire-format line through the cluster's own
> `vector vrl` running that exact program: with `JSON_LOGS` alone the field flips
> `tidaldb` → `tidal-server`. `service` is one of four `_stream_fields`, so its
> value is index identity — an unannounced rename would split tidalDB's log
> history at the very moment its format changed. Pinning it moves one variable
> instead of two.

---

## Known-red, not deploy blockers

Verified by bisect against the preceding commit — these are **not** caused by the
observability or credential work:

| Item | State |
|---|---|
| `mp_follower_reseeds_via_snapshot_after_compaction` | red on this workstation |
| `mp_quarantined_node_reseeds_without_wipe` | red on this workstation |
| `mp_multi_group_node_converges_after_reseeding_several_groups` | red on this workstation |
| `mp_graceful_rolling_restart_under_load_no_reseed` | red on baseline `main` |

The first three fail on the *first* `ack=quorum` write, roughly a second after
the gRPC listeners bind and before peer ship channels are established, against
the harness's own 3s client timeout — a startup race that loses on a loaded
machine. All three fail identically at the commit before this work. The
workstation they were run on is at 100% disk capacity, which is the most likely
aggravating factor. Re-run on a machine with headroom before treating them as a
code defect.

Also open, unrelated to this deploy:

- `kubectl apply -k k8s/cluster/` **downgrades** live limits (`cpu:3/7Gi` in the
  cluster vs `2/6Gi` in the manifest). Fix the manifest or apply selectively.
- The `tidaldb` namespace standalone Deployment is live `1/1` while the fleet
  record calls it superseded. Nothing routes to it.