tidaldb/docs/runbooks/deploy-verification.md

Deploy verification

Run the rolling-upgrade gate before you tag a release

mp_rolling_upgrade_no_loss_no_stall proves a rolling upgrade under load loses no acknowledged write and never stalls. It gated the CI image build from m11p8 until 2026-08-31, when it was measured as marginal on the CI hardware: pipelines #5 and #6 passed, #9 and #10 failed, all four identically at cluster_lifecycle.rs:362 timed out: WAL relay alone must reconverge all three nodes to 1e-6. The same test on the same commit passes locally in 19.61s.

It spawns three real tidal-server processes and asks them to reconverge to 1e-6 within 180s, on a node with ~1700m of free CPU shared with production tidalDB. Two passes and two failures is a coin flip, and a coin flip that blocks image builds trains everyone to re-run until green — which is how a gate stops being one. The budget was NOT raised again; that would hide the hardware behind a loosened threshold.

So run it by hand, on a machine where its result means something:

cargo test -p tidal-server --features cluster-e2e --test cluster_lifecycle \
  mp_rolling_upgrade_no_loss_no_stall -- --nocapture --test-threads 1
# expect: 1 passed, ~20s on a developer workstation

It also still runs nightly inside cluster_lifecycle, where a flake costs a re-read of the morning report rather than a blocked release. The push path is now gated by fast-suites — 51 tests across nine deterministic in-process suites with no spawned processes and no convergence budget to starve.

Do NOT deploy between 03:30 and 04:00 UTC

velero-fleet-daily fires at 03:30 UTC and takes 9–25 min (measured over 15 successful runs). A rolling restart inside that window cancels the in-flight volume backup for every pod it restarts, and the parent Velero Backup then stalls indefinitely — it does not fail, it sits InProgress and blocks the next scheduled run.

Observed 2026-08-31: a roll at 03:50/03:55/03:56 cancelled podvolumebackup …-wgtkt (pod=tidaldb-0 volume=data) at 2.7 GB of 5.4 GB. The Backup froze at 3529/3907 items and was still frozen 67 minutes later, heading for the 240-minute timeout that produces PartiallyFailed. The same fingerprint — Canceled/Failed PVBs clustered on one date — explains the PartiallyFailed runs on 2026-08-17, 08-18, 08-19 and 08-25.

If you have already done it: confirm the previous day's backup is Completed with full item coverage, then delete the stalled Backup so the schedule unblocks. Do not wait for the timeout.

Walk this top to bottom. Every command here was executed against the live orchard9-k3sf deployment and its output recorded — nothing is aspirational.

Each check states what it proves, the command, and what you should see. If a check fails, its If it fails line says where to look.

Sections 1–8 verify what is deployed now. Section 9 covers operator authority plus the one feature still inert on the running pods — §9.3, structured logs, which is now requested by the manifest and waits only on a restart. §9.1 and §9.2 were in that list and are now live; the header stays because a section that quietly drops its closed items teaches you nothing about how they closed.

Run it automatically first

Every check below is also a Playwright test that asserts the same thing and records what it observed. Run that first; walk the manual steps when something fails, or when you want to see a layer for yourself.

export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
npm install && npx playwright install chromium   # first time only
cargo build -p tidalctl                          # section 7 needs the binary

npm run test:e2e:list     # discovery: syntax, imports, registration
npm run test:e2e:smoke    # cluster plane + public plane + a quorum write
npm run test:e2e          # the whole runbook, 34 checks, ~55 s

Credentials are read from the cluster by globalSetup, so nothing is pasted into a shell. A missing prerequisite fails loudly rather than skipping a check.

The product, not just the deployment

Everything in this runbook verifies that the deployment answers. None of it verifies that the database does the thing it exists to do, so there is a second, separate suite for ranking semantics:

npm run test:e2e:semantics   # 5 checks, ~14 s, NO cluster required
npm run app:dev              # open the same app by hand and click Like

It boots a throwaway standalone node, seeds a 60-item fixture catalog with deterministic vectors from tidal-stress's own generator, and asserts that a signal write reorders the next query, that decay is applied at the declared half-life, and that ANN results equal brute-force cosine. It has its own config (playwright.semantics.config.ts) precisely because it must run with no cluster and no credentials — npm run test:all runs both suites.

Capturing the walkthrough (evidence a human can watch)

The regression suite answers "is the deployment correct?". The capture suite answers "can someone watch the proof?" — it photographs states the regression suite has already asserted, then Remotion assembles them into one video.

npm run test:demo      # 10 captures, each asserts before it photographs
npm run demo:promote   # copy captures in + stamp the LIVE image/revision
npm run demo:preflight # gate: every capture present, hash-stable, audited
npm run demo:render    # -> demo/out/deploy-verification.mp4 (90 s, 1920x1080)

Run these in order, and never skip demo:promote. The capture suite writes to test-results/demo-captures/, not into demo/public/captures/, because a capture becomes a published frame only after someone looks at it. Promotion is the step that copies frames across and re-stamps the manifest from the live StatefulSet.

That step did not exist until 2026-08-30 — it was done by hand once — and the consequence is the exact failure this runbook exists to prevent: the promoted frames and the rendered video still described m12-admin-gate-20260823 after two image rolls, while demo:preflight cheerfully reported "audited perfect" about week-old evidence. demo:render will happily encode stale PNGs; nothing else notices.

demo:promote leaves every verdict pending, so demo:preflight fails until the frames are audited. That failure is correct — clear it by opening the images at full resolution against demo/audience-brief.md and then:

npm run demo:promote -- --audited "<what you actually checked>"

The note is stored in the manifest beside the verdict. An audit with no statement of what was looked at is not an audit.

It deliberately never touches the deployed cluster: skip is declared permanent: true, so seeding signals into the live corpus would be irreversible. Two of its five checks report a product gap rather than a success (skip is inert under all 27 built-in profiles; see demo/capability-inventory.md BUG-018), and one cluster-targeted tripwire pins the duplicated rank (BUG-020) with its root cause in scatter_merge. The HTML report at playwright-report/ carries the transcript of every command the suite ran, which is the evidence trail this document used to describe in prose.

There is one deliberate asymmetry. The manual steps tell you to sleep 8 after a kubectl port-forward; the harness polls the port until it accepts a connection instead. Same fix, deterministic rather than empirical — and it is why the automated pass takes 40 s where the manual walk takes several minutes.

A stakeholder walkthrough of the same evidence lives in demo/ — see demo/storyboard.md for what it shows and demo/visual-audit.md for how each frame was reviewed.

---

0. Setup

export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
export TIDAL_API_KEY=$(kubectl -n tidaldb-cluster get secret tidaldb-credentials \
  -o jsonpath='{.data.TIDAL_API_KEY}' | base64 -d)

# Confirm you are pointed at the right cluster before anything else.
kubectl config current-context

TIDAL_API_KEY is the data-plane bearer. It is not an operator credential — see §9.2.

---

1. Cluster is up and converged

Proves: all three voters are serving, and no node is behind or reseeding.

kubectl -n tidaldb-cluster get pods -l app.kubernetes.io/name=tidaldb \
  -o custom-columns=NAME:.metadata.name,READY:.status.containerStatuses[0].ready,RESTARTS:.status.containerStatuses[0].restartCount

Expect three pods, READY=true. Restart counts are cumulative since the last roll — a stable non-zero value is fine, a climbing one is not.

NAME        READY   RESTARTS
tidaldb-0   true    1
tidaldb-1   true    0
tidaldb-2   true    1

Then the authoritative per-node view. Query each pod directly — see §1.1 for why the aggregated view is not trustworthy here:

for i in 0 1 2; do
  PORT=$((19700+i))
  kubectl -n tidaldb-cluster port-forward tidaldb-$i $PORT:9500 >/dev/null 2>&1 &
  PF=$!; sleep 8
  printf 'tidaldb-%s: ' "$i"
  curl -sk --max-time 10 -H "Authorization: Bearer $TIDAL_API_KEY" \
    "https://127.0.0.1:$PORT/cluster/status/local" \
  | python3 -c "
import json,sys
d=json.load(sys.stdin)
print('region=%-11s reseed=%-5s' % (d.get('region'), d.get('reseed_required')), end='')
for r in d.get('shards') or []:
    print(' g%s[app=%s lag=%s ldr=%s]' % (r.get('shard'), r.get('applied_events'),
          r.get('lag_events'), r.get('leader')), end='')
print()"
  kill $PF 2>/dev/null; wait $PF 2>/dev/null
done

Pass: every pod reports reseed=False and lag=0 on every group, all naming the same leader. Observed:

tidaldb-0: region=tidaldb-0   reseed=False g0[app=13324724 lag=0 ldr=tidaldb-2] g1[...lag=0...] g2[...lag=0...]
tidaldb-1: region=tidaldb-1   reseed=False g0[app=13324724 lag=0 ldr=tidaldb-2] g1[...lag=0...] g2[...lag=0...]
tidaldb-2: region=tidaldb-2   reseed=False g0[app=13322237 lag=0 ldr=tidaldb-2] g1[...lag=0...] g2[...lag=0...]

The 8-event difference on g0 between the leader and its followers is the leader's own un-shipped tail, not lag.

If it fails: reseed=True that survives a restart is the m11p5 livelock signature — see docs/runbooks/cluster.md §8. lag climbing on one node means that follower is not keeping up.

Sleep 8, not 4. A shorter wait races port-forward's bind and produces an empty body that reads exactly like a dead node. Two separate false alarms during this deploy came from exactly that.

1.1 Known caveat: the aggregated view under-reports peers

GET /cluster/status can report a peer it holds no frontier report for as applied_events: 0, then derive lag against that zero — so a converged peer appears to be the leader's entire history behind, sometimes labelled reachable: false / partitioned: true.

Observed on this deployment: two healthy nodes shown as UNREACHABLE PARTITIONED at 13.3M lag, while every node's own /cluster/status/local reported lag=0 and pod-to-pod connectivity was open with nothing logged.

tidalctl cluster-status detects that signature and prints NO REPORT (aggregated view; query the node directly) instead of repeating it as lag. Do not open an incident on NO REPORT before running the §1 per-pod loop. This is an open reporting defect, tracked in docs/ops/observability.md §4.

---

2. Public endpoint and TLS

Proves: the hostname resolves publicly and serves a certificate a normal client will accept.

dig +short tidaldb.threesix.ai

Expect all three node IPs (order varies):

208.122.204.172
208.122.204.173
208.122.204.174

If your workstation runs a split-DNS resolver (Tailscale MagicDNS will do this), dig may succeed while curl cannot resolve. Confirm public resolution independently:

curl -s -H 'accept: application/dns-json' \
  'https://cloudflare-dns.com/dns-query?name=tidaldb.threesix.ai&type=A' \
| python3 -c "import json,sys; print([a['data'] for a in json.load(sys.stdin).get('Answer',[])])"

Certificate identity:

# Connect by IP with SNI, so this works even when the local resolver lags.
echo | openssl s_client -connect 208.122.204.172:443 \
  -servername tidaldb.threesix.ai 2>/dev/null \
| openssl x509 -noout -subject -issuer -enddate

Pass: subject=CN=tidaldb.threesix.ai, issuer Let's Encrypt, notAfter at least 30 days out. Observed notAfter=Nov 20 04:03:50 2026 GMT.

If it fails: cert-manager uses the dns01 solver (letsencrypt-prod), not http01 — http01 cannot work behind a gateway gate that rejects unknown callers, because it rejects the ACME challenge too. Check kubectl -n tidaldb-cluster get certificate tidaldb-public-tls.

---

3. Authentication behaves correctly

Proves: the data surface is credential-gated and the admin surface is not published at all.

Use --resolve if your local resolver lags; it still validates the certificate.

R="--resolve tidaldb.threesix.ai:443:208.122.204.172"
B="https://tidaldb.threesix.ai"

printf 'health (open)      %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 $R $B/health)"
printf 'search no bearer   %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 $R "$B/search?query=a&limit=1")"
printf 'search bad bearer  %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 -H 'Authorization: Bearer wrong' $R "$B/search?query=a&limit=1")"
printf 'search good bearer %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 20 -H "Authorization: Bearer $TIDAL_API_KEY" $R "$B/search?query=a&limit=1")"

Pass: 200, 401, 401, 200.

Admin and metrics surfaces must be unroutable from the internet:

for p in /cluster/status /cluster/members /metrics /openapi.json; do
  printf '%-18s %s\n' "$p" "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 $R "$B$p")"
done

Pass: 404 for all four. A 200 on /cluster/status would be leaking leader identity, membership and seqnos to the internet; a 200 on /metrics would be leaking corpus size unauthenticated.

If it fails: the path allowlist lives in k8s/cluster/ingress.yaml. Only the data paths are published, deliberately.

3.1 A real write, end to end

curl -s --max-time 25 -X POST \
  -H 'content-type: application/json' \
  -H "Authorization: Bearer $TIDAL_API_KEY" \
  -H 'x-tidal-ack: quorum' \
  -d '{"entity_id":999000099,"metadata":{"title":"deploy verification","category":"probe"}}' \
  $R "$B/items" -o /dev/null -w 'quorum write %{http_code}\n'

Pass: 201. This is the strongest single check in the document — it proves DNS, TLS, the gateway, the backend TLS hop, authentication, and quorum replication in one request.

---

4. Network isolation

Proves: the unauthenticated metrics port and the peer plane are not reachable from arbitrary pods.

kubectl -n tidaldb-cluster get networkpolicy tidaldb
PIP=$(kubectl -n tidaldb-cluster get pod tidaldb-0 -o jsonpath='{.status.podIP}')

# A pod in an unrelated namespace must NOT reach :9091.
kubectl -n threesix exec gitea-0 -- \
  sh -c "wget -qO- --timeout=5 http://$PIP:9091/metrics 2>&1 | head -1"

Pass: Connection refused. Before the policy existed this returned # HELP tidaldb_uptime_seconds … from any pod in the cluster.

The scraper must still get through:

kubectl -n observability exec deploy/vmagent -- \
  sh -c "wget -qO- --timeout=6 http://$PIP:9091/metrics 2>/dev/null | grep -c '^tidaldb_'"

Pass: a few hundred series (observed 353).

Do not test pod-to-pod reachability with /dev/tcp under sh. The container's sh is dash, which has no /dev/tcp, so it reports failure for a port that is open. That false negative briefly looked like a cluster partition during this deploy. Use bash -c explicitly.

---

5. Metrics and dashboard

Proves: the series exist with the labels the dashboard queries, and the dashboard is loaded.

kubectl -n observability port-forward svc/vmsingle 8428:8428 >/dev/null 2>&1 &
sleep 6
curl -s -G 'http://127.0.0.1:8428/api/v1/series' \
  --data-urlencode 'match[]=tidaldb_health_ok' \
| python3 -c "import json,sys; d=json.load(sys.stdin)['data']; print(len(d),'series'); print(sorted(d[0]))"

Pass: non-zero series, labels including namespace, pod, container, partition_id. The dashboard's $namespace/$pod variables depend on those exact names.

Dashboard presence (needs the Grafana admin password):

PW=$(kubectl -n observability get secret grafana-admin -o jsonpath='{.data.password}' | base64 -d)
kubectl -n observability port-forward deploy/grafana 3000:3000 >/dev/null 2>&1 &
sleep 6
curl -s -u "admin:$PW" 'http://127.0.0.1:3000/api/dashboards/uid/tidaldb-overview' \
| python3 -c "
import json,sys
d=json.load(sys.stdin); db=d['dashboard']
print('LOADED:', db['title'], '| folder:', d['meta'].get('folderTitle'),
      '| panels:', sum(1 for p in db['panels'] if p['type']!='row'))"

Pass: LOADED: tidalDB — usage, errors, and cluster health | folder: Databases | panels: 13.

If it fails: the dashboard is a key on the grafana-database-dashboards ConfigMap; Grafana's file provider rescans every 30s. Source of truth is docs/ops/grafana-tidaldb.json.

---

6. Logs are queryable

Proves: log lines are reaching the platform.

kubectl -n tidaldb-cluster logs tidaldb-0 --tail=20

Pass: readable lines, no raw \x1b[ escape fragments.

Today the pods still emit plain text, so the collector cannot parse the line, keeps the text, and stamps every tidalDB entry level=info — including its WARNs. Measured 2026-08-30: unit:tidaldb-cluster faceted over 30m returned exactly one row, {"service":"tidaldb","level":"info","n":"457"}, while level:error matched fine for services that do emit JSON (relay, external-secrets). The collector is not broken. tidalDB was not speaking to it.

The fix is a deployment change, not a code change: JSON_LOGS=1 is now on the StatefulSet and takes effect on the next pod restart. See §9.3.

Until that roll lands, filter at the source:

kubectl -n tidaldb-cluster logs tidaldb-0 --since=15m | grep -iE 'ERROR|WARN'

---

7. Live debugging tools work

Proves: you can interrogate a running cluster without hand-rolling curl.

cargo build -p tidalctl        # or use a released binary
kubectl -n tidaldb-cluster port-forward svc/tidaldb 9500:9500 >/dev/null 2>&1 &
sleep 9

# --insecure is required: the client port is served with the INTERNAL cluster CA,
# whose leaf is issued for in-cluster DNS names.
./target/debug/tidalctl cluster-status \
  --url https://127.0.0.1:9500 --key "$TIDAL_API_KEY" --insecure
echo "exit=$?"

Pass: a leader line, a region table, a shard table. Exit 0 when converged, 2 when not — so tidalctl cluster-status && deploy is a safe gate. Remember §1.1: NO REPORT means the aggregated view lacks a peer report, not that the peer is down.

# Convergence monitor, bounded so it terminates.
./target/debug/tidalctl watch --url https://127.0.0.1:9500 \
  --key "$TIDAL_API_KEY" --insecure --interval 5 --count 3

Pass: one line per tick, each ending [ok] or [DEGRADED].

Exit-code contract (verified):

Situation	Exit
converged	0
bad/absent credential	2
server unreachable	2
--url without a scheme	1
missing --url / --path	1

---

8. Backups

Proves: a whole-fleet backup completed recently and captured every volume.

# Select the latest backup FROM THE FLEET SCHEDULE. Sorting all backups by
# timestamp picks up restore-canary runs (20 items, 1 volume), which would
# "pass" while telling you nothing about the fleet.
B=$(kubectl -n backup-system get backup.velero.io \
      -l velero.io/schedule-name=velero-fleet-daily \
      --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1].metadata.name}')
echo "checking $B"
kubectl -n backup-system get backup.velero.io "$B" \
  -o jsonpath='phase={.status.phase} errors={.status.errors} items={.status.progress.itemsBackedUp}/{.status.progress.totalItems}{"\n"}'
kubectl -n backup-system get podvolumebackups -l velero.io/backup-name="$B" \
  --no-headers | awk '{print $2}' | sort | uniq -c

Pass: phase=Completed, no errors, and every PodVolumeBackup Completed. Observed on velero-fleet-daily-20260823033025: 3708/3708 items, 48/48 Completed.

Careful: a single failed PVB marks the whole backup PartiallyFailed and freezes the alert gauge, even when every volume that matters was captured. If you trigger a manual backup to satisfy the alert, it must carry the schedule label or the metric will not advance:

The alert reads velero_backup_last_successful_timestamp{schedule="velero-fleet-daily"}, so a manual Backup must carry that schedule label or the gauge never moves. A manual run without it completed 48/48 and the alert stayed critical:

metadata:
  labels:
    velero.io/schedule-name: velero-fleet-daily

---

9. Operator authority, and what is still inert

The cluster runs registry.threesix.ai/tidal/server:m12-vsc-20260830@sha256:5c18d2b1… (rolled 2026-08-30, built from 8aa1fbb). That image carries the operator/data credential split (§9.2), the HTTP request metrics (§9.1), the blob-replication ledger, and the structured-log emitter (§9.3).

§9.1 is LIVE. Both 09-operator-authority.spec.ts and the CAP-015 capture asserted tidaldb_http_* = 0; that became false and they failed, which is how the drift surfaced within minutes instead of rotting here. Both assertions are now inverted, so a rollback that removes the HTTP metrics fails them again.

One correction to the earlier wording, forced by the revision table below: the HTTP metrics and the structured-log emitter came from the same commit, 4766f56, so they reached production together at rev 30, not at rev 31. The 0 readings recorded in this section were taken while rev 29 was running; rev 30 held the cluster for only 2h21m and was never measured. "Went live with the m12-vsc roll" is where it was noticed, not where it started.

Live reading after the roll: tidaldb_http_* = 185 against a baseline of tidaldb_* = 552 — the baseline is what proves the scrape worked, so "present" is distinguishable from "unscraped".

§9.3's label was stale, and it is corrected here — with the dates, because the earlier wording was right when written. "Requires the image roll and JSON_LOGS=1" was true up to 2026-08-30T17:38Z. It stopped being true at that moment and nobody updated it. The StatefulSet's own revision history is the record:

rev	image	live from	carries the emitter?
29	m12-admin-gate-20260823	2026-08-23T05:32Z	no — built before 4766f56
30	m12-vector-grow-20260830	2026-08-30T17:38Z	yes — 7c1c80d, which descends from 4766f56
31	m12-vsc-20260830	2026-08-30T19:59Z	yes — 8aa1fbb

kubectl -n tidaldb-cluster get controllerrevision -l app.kubernetes.io/name=tidaldb \
  -o custom-columns=REV:.revision,WHEN:.metadata.creationTimestamp,IMAGE:'.data.spec.template.spec.containers[0].image'
git merge-base --is-ancestor 4766f56 8aa1fbb && echo "emitter is in the running image"

So the image half of the requirement has been satisfied through two rolls, and the env half was never satisfied at all — neither roll added JSON_LOGS, and this section kept naming the image as the blocker after the image stopped being one. The lesson is not "the doc lied"; it is that a gap described by its prerequisite rather than its current state survives the prerequisite being met.

JSON_LOGS=1 and TIDAL_SERVICE_NAME=tidaldb are now in k8s/cluster/statefulset.yaml. An env change does nothing to already-running pods, so §9.3 stays inert until the next roll restarts them — that is the one item here still genuinely pending, and it is pending on a restart, not on code. Container logs carry no ANSI escapes (BUG-006 resolved on this image) and 06-logs.spec.ts pins that in the other direction — a regression to coloured output fails there.

9.2 Operator/data credential split — LIVE, verify it stays that way

Confirmed working on this deployment. The admin routes are not published, so port-forward first:

kubectl -n tidaldb-cluster port-forward svc/tidaldb 9500:9500 >/dev/null 2>&1 &
sleep 8
ADMIN=$(kubectl -n tidaldb-cluster get secret tidaldb-credentials \
  -o jsonpath='{.data.TIDAL_ADMIN_KEY}' | base64 -d)

# data bearer on an operator verb -> 403 (authenticated, NOT authorized)
curl -sk -o /dev/null -w 'data  -> promote %{http_code}\n' -X POST \
  -H "Authorization: Bearer $TIDAL_API_KEY" -H 'content-type: application/json' \
  -d '{"region":"tidaldb-1"}' https://127.0.0.1:9500/cluster/promote

# admin bearer -> authorized
curl -sk -o /dev/null -w 'admin -> promote %{http_code}\n' -X POST \
  -H "Authorization: Bearer $ADMIN" -H 'content-type: application/json' \
  -d '{"region":"tidaldb-1"}' https://127.0.0.1:9500/cluster/promote

Pass: 403 then 200. Observed exactly that. Before this key existed the data bearer could remove members and transfer shards — that is the exposure the split closes.

The key was hot-loaded with no restart, and the startup log still says it is missing. The pod booted 05:41 and logged TIDAL_ADMIN_KEY is not set; kubelet materialized the projected secret file at 05:51:43; the credential poller picked it up and the gate went live. So a TIDAL_ADMIN_KEY is not set WARN in the boot log does not mean the gate is open — test the behavior, which is why the two curls above are the actual check.

9.1 HTTP request/error metrics — LIVE

Live since rev 30 (m12-vector-grow-20260830, 2026-08-30T17:38Z) — the same commit 4766f56 that carried the structured-log emitter. This section claimed "still inert" for two images after that stopped being true, the same failure mode as §9.3, in the same document. Verify it stays live:

for p in tidaldb-0 tidaldb-1 tidaldb-2; do
  PIP=$(kubectl -n tidaldb-cluster get pod "$p" -o jsonpath='{.status.podIP}')
  HTTP=$(kubectl -n observability exec deploy/vmagent -- \
    sh -c "wget -qO- --timeout=15 http://$PIP:9091/metrics | grep -c tidaldb_http_requests_total")
  ALL=$(kubectl -n observability exec deploy/vmagent -- \
    sh -c "wget -qO- --timeout=15 http://$PIP:9091/metrics | grep -c tidaldb_")
  echo "$p http=$HTTP all=$ALL"
done

Pass: non-zero http on all three. Observed 2026-08-31T02:19Z: http=19/24/18 against all=821/878/812. (The preamble's 185 counts every tidaldb_http_* line, a wider grep — same conclusion, different denominator.) Always read the baseline too — it is what distinguishes "the metric is missing" from "the scrape returned nothing". The dashboard's Request rate by route, Requests by status, 5xx ratio, Auth rejections and HTTP p99 panels populate from these.

Use --timeout=15, not 6. A 6-second wget truncates this scrape on a busy node and returns zero lines, which reads exactly like "the metric is missing" — it briefly looked like one pod had stopped exporting entirely.

9.3 Structured logs — requested in the manifest, live after the next roll

The capability is not missing and never was; see the §9 preamble. What was missing is the request for it, which is now in source:

kubectl -n tidaldb-cluster get statefulset tidaldb \
  -o jsonpath='{range .spec.template.spec.containers[*].env[*]}{.name}={.value}{"\n"}{end}' \
  | grep -E 'JSON_LOGS|TIDAL_SERVICE_NAME'

Pass: JSON_LOGS=1 and TIDAL_SERVICE_NAME=tidaldb. Until the roll this prints nothing on the live object while kubectl kustomize k8s/cluster/ shows both — that divergence is the pending state, not a defect.

After the roll — emitter side:

kubectl -n tidaldb-cluster logs tidaldb-0 -c tidaldb --tail=5 | python3 -c "
import json,sys
for line in sys.stdin:
    json.loads(line)
print('all sampled lines are JSON')"

Pass: one JSON object per line carrying ts, level, service, target, msg, and request_id inside a request span.

After the roll — consumer side, which is the check that actually matters:

kubectl -n observability exec deploy/vmagent -- sh -c \
  'wget -qO- --timeout=25 \
   --post-data="query=unit:tidaldb-cluster | stats by (service, level) count() n&start=15m" \
   http://victoria-logs:9428/select/logsql/query'

Pass: service:"tidaldb" now appears under more than one level, and level:error / level:warn return tidalDB lines. The pre-roll reading was a single row — {"service":"tidaldb","level":"info","n":"457"} — with every WARN flattened onto info. Do not accept the emitter check alone. The emitter has looked correct for weeks while the store still could not filter; that asymmetry is precisely how this section stayed wrong.

Why TIDAL_SERVICE_NAME is set alongside it. Vector's normalize transform (cm/vector-cluster-config, ns observability) substitutes the container name only when !exists(.service). Plain-text lines have no service, so today's land as tidaldb; the JSON emitter sets its own, default tidal-server. Verified by piping a wire-format line through the cluster's own vector vrl running that exact program: with JSON_LOGS alone the field flips tidaldb → tidal-server. service is one of four _stream_fields, so its value is index identity — an unannounced rename would split tidalDB's log history at the very moment its format changed. Pinning it moves one variable instead of two.

---

Known-red, not deploy blockers

Verified by bisect against the preceding commit — these are not caused by the observability or credential work:

Item	State
mp_follower_reseeds_via_snapshot_after_compaction	red on this workstation
mp_quarantined_node_reseeds_without_wipe	red on this workstation
mp_multi_group_node_converges_after_reseeding_several_groups	red on this workstation
mp_graceful_rolling_restart_under_load_no_reseed	red on baseline main

The first three fail on the first ack=quorum write, roughly a second after the gRPC listeners bind and before peer ship channels are established, against the harness's own 3s client timeout — a startup race that loses on a loaded machine. All three fail identically at the commit before this work. The workstation they were run on is at 100% disk capacity, which is the most likely aggravating factor. Re-run on a machine with headroom before treating them as a code defect.

Also open, unrelated to this deploy:

• kubectl apply -k k8s/cluster/ downgrades live limits (cpu:3/7Gi in the cluster vs 2/6Gi in the manifest). Fix the manifest or apply selectively.
• The tidaldb namespace standalone Deployment is live 1/1 while the fleet record calls it superseded. Nothing routes to it.