15f6b11187
Turns docs/runbooks/deploy-verification.md from prose into 32 executable checks
against the live orchard9-k3sf cluster, and it found real defects on its first
run — including in the runbook it verifies.
WHY PLAYWRIGHT, HONESTLY
tidalDB serves zero HTML (no text/html, no Html(), 10 JSON routes), so this uses
Playwright in three distinct roles rather than pretending there is a UI:
* request fixture as a real HTTP client for DNS/TLS/auth/quorum/404;
* a browser for the only genuine screens in the chain, Grafana;
* a test harness for cluster-plane checks with no HTTP surface, shelling out
to kubectl and attaching the real transcript as evidence.
WHAT IT CAUGHT
* The runbook asserted the operator/data credential split was "not active yet
- requires an image roll". globalSetup read the live image and the live
secret; a probe returned data->403, admin->200. It had been enforcing the
whole time. Section 9 rewritten. (BUG-001)
* docs/ops/grafana-tidaldb.json shipped datasource uid ${DS_PROMETHEUS} - a
Grafana export-for-sharing placeholder with no __inputs block to resolve it.
Under ConfigMap provisioning every panel queried a datasource that did not
exist, so the whole board was blank. The API said "loaded" and I had only
ever checked the API. 41 refs fixed here, 58 across the fleet ConfigMap,
which was also blanking the postgres and redis dashboards. (BUG-007)
* Stat panels used calcs "lastNonNull". Grafana's reducer is "lastNotNull", so
no value was ever computed and Cluster health / Reseed pending / Indexed
vectors rendered as empty boxes. I chased panel width and then panel height
before comparing against a working stat panel elsewhere in the same Grafana.
A spelling error wearing a layout bug's clothes. (BUG-009)
* The namespace variable defaulted to All, so cluster panels silently included
tidaldb-586b544c8-vpkmw from the superseded standalone deployment. Latency
legends read "p50 p50 p50" with no way to tell the nodes apart. Both fixed.
* "5xx ratio" rendered "No data" as large green text - at a glance a healthy
value. And Fleet state gave three fields one shared green threshold, so
reseed_required=1 would have shown GREEN during the exact incident the panel
exists to surface. Split into three panels with per-field mappings.
* tidalctl cluster-status exits 2 on a FULLY CONVERGED cluster, because the
aggregated endpoint reports healthy peers as region=null applied=0
reachable=false. The runbook claimed `cluster-status && deploy` was a safe
gate; that claim came from an exit code masked by a shell pipeline. The gate
can never pass here. Documented, test pins it, engine defect recorded.
(BUG-005)
* The deployed image writes ANSI colour into container logs, which the
collector stores verbatim. Already fixed in logging.rs, not yet rolled;
pinned as a tripwire. (BUG-006)
* The runbook's own backup command sorted ALL backups by timestamp and
selected a restore-canary run: 20 items, one volume, a meaningless pass.
Now filters on the schedule label the freshness alert actually watches.
DEFECTS FOUND BY LOOKING AT THE SCREENS
Six of the first eight captures were slop and were fixed, not promoted:
230-350px of dead space; a verdict that rendered "exit code 2" in green; the
1600x1800 dashboard scaled into 16:9 until illegible (now clipped to the
evidence band using real element bounds); the dream beat whose caption described
a contradiction the image did not show (now a purpose-built capture holding the
committed doc text, the running image, and the live 403/200 side by side); and a
one-frame blink to bare background at every scene boundary, because Remotion
Sequences do not overlap and both scenes sat at opacity 0 on the boundary frame.
TRIPWIRES IN THE HONEST DIRECTION
Three tests assert what is ABSENT - zero tidaldb_http_* families, JSON_LOGS
unset, plain-text logs - and each carries the message "good news, roll the
runbook section from pending to live". The metric-absence test also asserts the
baseline family count, so "absent" cannot pass for "the scrape failed". That is
the drift that made section 9 stale in the first place.
Regression config uses workers:1 and retries:0 deliberately: a live-cluster
check that only passes on the second attempt has told you something true.
Verified: 32 passed (46.8s); 9 demo captures each asserting before photographing;
tsc clean; render 82.05s 1920x1080 h264, 0 empty frames across 10 boundaries;
every promoted image inspected individually and judged perfect; walk-the-render
ledger complete with no fails.
6.3 KiB
6.3 KiB
Storyboard — tidalDB deploy verification
Ordered by the audience ladder in audience-brief.md, not by runbook section
order. Needs first to buy credibility, wants in the middle, the single dream beat
where attention peaks, then a recap that restates the needs as met.
Every functional beat cites a capability, a green test, a promoted capture, and the rung it serves. A beat serving no rung is cut.
30 fps. Durations are seconds.
| Beat | Capability | Rung | Persona | Business purpose | Proof state | Test | Capture | Caption | Secs |
|---|---|---|---|---|---|---|---|---|---|
| B1 opening | — | need | operator | Orientation: what this is and what it will prove | Title card | — | — | "tidalDB deploy verification. Three voters on orchard9-k3sf, one public endpoint, and a runbook an operator can walk. Every number that follows came from a command that ran against the live cluster." |
9 |
| B2 convergence | CAP-002 | need | operator | Quorum with fault tolerance actually exists, rather than being inferred from pod readiness | Per-node lag=0, no reseed, agreed leader per shard group |
01-cluster-convergence.spec.ts :: every node reports zero lag… |
CAP-002-convergence |
"Each node asked for its own view. A pod can be Ready while its replication is stalled — that is what the reseed livelock exploited." | 7 |
| B3 boundary + write | CAP-005 CAP-006 | need | operator | The strongest single proof: full stack works AND the data plane is closed | 401, 401, 200, then 201 quorum-acked | 03-auth-boundary.spec.ts :: a quorum-acked write is committed… |
CAP-006-quorum-write |
"Same path, three credentials. Then a write a majority of nodes acknowledged — the one claim a health endpoint cannot fake." | 8 |
| B4 isolation | CAP-008 | need | operator | Least-privilege network access without blinding monitoring | Refused from a foreign namespace; 332 series to the scraper | 04-network-isolation.spec.ts :: a pod in an unrelated namespace is refused… |
CAP-008-network-isolation |
"The refusal is the point. A policy that blocks everything is an outage; one that blocks nothing is theatre." | 7 |
| B5 authority | CAP-014 | want | operator | Blast radius of a leaked application key stops at data, not cluster topology | Data bearer 403, admin bearer authorised | 09-operator-authority.spec.ts :: the data-plane credential is refused… |
CAP-014-authority |
"403, not 401 — the key is valid, it just cannot remove a cluster member." | 6 |
| B6 dashboard | CAP-010 | want | operator | The first surface opened during an incident actually shows the cluster | Cluster health OK, reseed none, 33.3K vectors, populated latency charts | 05-metrics-dashboard.spec.ts :: an operator opening the dashboard sees populated charts |
CAP-010-dashboard |
"The board an operator opens at 3am. Health, reseed state, corpus size, and per-node latency — scoped to the cluster." | 8 |
| B7 recovery | CAP-013 | want | operator | The cluster can actually be restored and the alert is trustworthy | Completed, 3708/3708 items, 48/48 volumes | 08-backups.spec.ts :: the newest fleet-schedule backup completed… |
CAP-013-backup |
"Selected by the schedule label the freshness alert watches — not merely the newest backup object." | 6 |
| B8 blind spot | CAP-012 | want | operator | A tool that reports what it cannot see is trustworthy about what it can | NO REPORT markers instead of fabricated lag; exit 2 |
07-tidalctl.spec.ts :: the aggregated-status gap is reported as NO REPORT… |
CAP-012-tidalctl |
"It refuses to invent a number. Two healthy peers would otherwise read as 13.3 million events behind." | 7 |
| B9a inert | CAP-015 | need | operator | A verification that hides its gaps cannot be trusted about what it claims | Running image plus zero tidaldb_http_* families against a working baseline scrape |
09-operator-authority.spec.ts :: HTTP request metrics are absent… |
CAP-015-inert |
"Two committed features are absent from the running image. The suite asserts that absence, so the day it changes it fails and says so." | 7 |
| B9 the drift catch | CAP-014 | dream | operator | Verification that audits its own documentation instead of drifting from it | The committed doc text, the actually-running image, and the live 403/200 probe, side by side | deploy-verification.demo.spec.ts :: the harness corrected its own runbook |
CAP-014-drift |
"The runbook said the credential split was 'not active yet — requires an image roll'. On its first run the harness read the live image and the live secret, and proved the gate was already enforcing. The document was wrong. The harness said so, before anyone noticed." | 10 |
| B10 recap | — | need | operator | Restate the needs as met, including the gaps | Closing card | — | — | "32 checks green against the live deployment. Two committed features are absent from the running image, and the suite asserts that absence deliberately — so the day it changes, it says so." | 7 |
Total: 82 s across 11 beats.
Ladder coverage
Two-way mapping, per the audience protocol.
| Brief entry | Served by |
|---|---|
| Need — evidence is real, commands visible | B1, B2, B3, B4 (every capture shows the command that produced it) |
| Need — write proven committed by quorum | B3 |
| Need — every boundary shows its denial | B3 (401×2), B4 (refused), B5 (403) |
| Need — what is not verified is stated | B9a (evidence on screen), B10 (recap) |
| Want — one command a different person can run | B1, B10 |
| Want — operator/data authority visibly enforced | B5 |
| Dream — the harness catches a stale claim in the runbook it verifies | B9 (exactly one dream beat) |
No need rung is unserved, and no need-serving beat was cut in response to an
audience-bar failure — the five weak-design failures found in the first capture
pass were fixed at the capture layer and re-audited.
Deliberately not beats
- A "green wall" summary of all 32 tests. Counting tests is not evidence, and this viewer has seen a livelocked cluster report itself healthy.
- The three red
cluster_reseedtests. They belong in the written record where the bisect evidence can be read, not in a 75-second walkthrough where a red frame would read as a deploy failure. - Clicking, scrolling, or navigating. The dashboard beat shows the board, not the act of opening it.