15f6b11187
Turns docs/runbooks/deploy-verification.md from prose into 32 executable checks
against the live orchard9-k3sf cluster, and it found real defects on its first
run — including in the runbook it verifies.
WHY PLAYWRIGHT, HONESTLY
tidalDB serves zero HTML (no text/html, no Html(), 10 JSON routes), so this uses
Playwright in three distinct roles rather than pretending there is a UI:
* request fixture as a real HTTP client for DNS/TLS/auth/quorum/404;
* a browser for the only genuine screens in the chain, Grafana;
* a test harness for cluster-plane checks with no HTTP surface, shelling out
to kubectl and attaching the real transcript as evidence.
WHAT IT CAUGHT
* The runbook asserted the operator/data credential split was "not active yet
- requires an image roll". globalSetup read the live image and the live
secret; a probe returned data->403, admin->200. It had been enforcing the
whole time. Section 9 rewritten. (BUG-001)
* docs/ops/grafana-tidaldb.json shipped datasource uid ${DS_PROMETHEUS} - a
Grafana export-for-sharing placeholder with no __inputs block to resolve it.
Under ConfigMap provisioning every panel queried a datasource that did not
exist, so the whole board was blank. The API said "loaded" and I had only
ever checked the API. 41 refs fixed here, 58 across the fleet ConfigMap,
which was also blanking the postgres and redis dashboards. (BUG-007)
* Stat panels used calcs "lastNonNull". Grafana's reducer is "lastNotNull", so
no value was ever computed and Cluster health / Reseed pending / Indexed
vectors rendered as empty boxes. I chased panel width and then panel height
before comparing against a working stat panel elsewhere in the same Grafana.
A spelling error wearing a layout bug's clothes. (BUG-009)
* The namespace variable defaulted to All, so cluster panels silently included
tidaldb-586b544c8-vpkmw from the superseded standalone deployment. Latency
legends read "p50 p50 p50" with no way to tell the nodes apart. Both fixed.
* "5xx ratio" rendered "No data" as large green text - at a glance a healthy
value. And Fleet state gave three fields one shared green threshold, so
reseed_required=1 would have shown GREEN during the exact incident the panel
exists to surface. Split into three panels with per-field mappings.
* tidalctl cluster-status exits 2 on a FULLY CONVERGED cluster, because the
aggregated endpoint reports healthy peers as region=null applied=0
reachable=false. The runbook claimed `cluster-status && deploy` was a safe
gate; that claim came from an exit code masked by a shell pipeline. The gate
can never pass here. Documented, test pins it, engine defect recorded.
(BUG-005)
* The deployed image writes ANSI colour into container logs, which the
collector stores verbatim. Already fixed in logging.rs, not yet rolled;
pinned as a tripwire. (BUG-006)
* The runbook's own backup command sorted ALL backups by timestamp and
selected a restore-canary run: 20 items, one volume, a meaningless pass.
Now filters on the schedule label the freshness alert actually watches.
DEFECTS FOUND BY LOOKING AT THE SCREENS
Six of the first eight captures were slop and were fixed, not promoted:
230-350px of dead space; a verdict that rendered "exit code 2" in green; the
1600x1800 dashboard scaled into 16:9 until illegible (now clipped to the
evidence band using real element bounds); the dream beat whose caption described
a contradiction the image did not show (now a purpose-built capture holding the
committed doc text, the running image, and the live 403/200 side by side); and a
one-frame blink to bare background at every scene boundary, because Remotion
Sequences do not overlap and both scenes sat at opacity 0 on the boundary frame.
TRIPWIRES IN THE HONEST DIRECTION
Three tests assert what is ABSENT - zero tidaldb_http_* families, JSON_LOGS
unset, plain-text logs - and each carries the message "good news, roll the
runbook section from pending to live". The metric-absence test also asserts the
baseline family count, so "absent" cannot pass for "the scrape failed". That is
the drift that made section 9 stale in the first place.
Regression config uses workers:1 and retries:0 deliberately: a live-cluster
check that only passes on the second attempt has told you something true.
Verified: 32 passed (46.8s); 9 demo captures each asserting before photographing;
tsc clean; render 82.05s 1920x1080 h264, 0 empty frames across 10 boundaries;
every promoted image inspected individually and judged perfect; walk-the-render
ledger complete with no fails.
138 lines
12 KiB
Markdown
138 lines
12 KiB
Markdown
# Visual audit — tidalDB deploy verification walkthrough
|
||
|
||
Audited against `audience-brief.md`. Per-screen verdicts are judged at the
|
||
**quality-bar judge's** standard (Kyle Kingsbury); the walk-the-render ledger is
|
||
written in the **actual decision-maker's** voice (Jordan Washburn).
|
||
|
||
- Build revision: see `capture-manifest.json.buildRevision`
|
||
- Verified image: `registry.threesix.ai/tidal/server:m12-admin-gate-20260823@sha256:6e220060…`
|
||
- Render: `demo/out/deploy-verification.mp4` — 82.05 s, 1920×1080, 30 fps, h264, 2460 frames
|
||
- Regression suite at time of capture: **32 passed**
|
||
- Demo capture suite: **9 passed** (each asserts before it photographs)
|
||
|
||
## 1. Programmatic preflight
|
||
|
||
| Check | Result |
|
||
| --- | --- |
|
||
| Every manifest file exists and decodes | pass — 9/9 |
|
||
| Filesystem inventory equals ledger count | pass — 9 promoted, 9 rows, 0 unclassified |
|
||
| Dimensions match declared viewport or documented crop | pass — 8 × 1600×900, 1 × 1600×502 (documented crop) |
|
||
| Files non-empty, not near-uniform blanks | pass — 64 KB–187 KB |
|
||
| Capture IDs and filenames unique | pass |
|
||
| No unexpected duplicates across distinct proof states | pass |
|
||
| Source test green | pass — 9 demo capture tests green; each asserts before it photographs |
|
||
| Secrets / tokens / local paths absent | pass — `redact()` masks every known secret before render; spot-checked all 9 |
|
||
|
||
## 2. Per-image review
|
||
|
||
Every image was opened individually at full resolution. Verdicts after the fix
|
||
round; the pre-fix verdict is shown where it differed, because a `slop` finding
|
||
that was silently overwritten is not an audit.
|
||
|
||
| Capture | State | Rendering | Layout | Content | Privacy | Errors | Audience verdict | Bugs |
|
||
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||
| `CAP-002-convergence` | correct | complete | ok (was: 230 px dead space) | real 13.3M positions, real pod names | clean | none | **perfect** (was `slop` weak-design) | BUG-011 |
|
||
| `CAP-006-quorum-write` | correct | complete | ok | 401/401/200/201, real hostname | clean | none | **perfect** | — |
|
||
| `CAP-008-network-isolation` | correct | complete | ok (was: 350 px dead space) | `Connection refused` + 332 series | clean | none | **perfect** (was `slop` weak-design) | BUG-011 |
|
||
| `CAP-010-dashboard` | correct | complete | ok (was: illegible when scaled) | OK / none / 33.3 K, per-pod legends | clean | no "No data" in crop | **perfect** (was `slop` weak-design) | BUG-010 |
|
||
| `CAP-012-tidalctl` | correct | complete | ok (was: dead space) | `NO REPORT` markers, exit 2 | clean | none | **perfect** (was `slop` weak-design + colour) | BUG-011, BUG-012 |
|
||
| `CAP-013-backup` | correct | complete | ok (was: dead space) | 3708/3708, 48 PVBs | clean | none | **perfect** (was `slop` weak-design) | BUG-011 |
|
||
| `CAP-014-authority` | correct | complete | ok (was: dead space) | 403 then 200 | clean | none | **perfect** (was `slop` weak-design) | BUG-011 |
|
||
| `CAP-014-drift` | correct | complete | ok | git text + live image + 403/200 | clean | none | **perfect** | BUG-013 |
|
||
| `CAP-015-inert` | correct | complete | ok (was: dead space) | 332 baseline / 0 http | clean | none | **perfect** (was `slop` weak-design) | BUG-011 |
|
||
|
||
### Non-promoted images
|
||
|
||
None. Nine images were produced by the capture run and nine were promoted; no
|
||
failure or debug images were generated because every capture test passed on the
|
||
run that produced the promoted set.
|
||
|
||
## 3. Remotion frame audit
|
||
|
||
Scene proof frames rendered and inspected at `demo/out/stills/`; transition
|
||
frames extracted from the **encoded MP4** (not re-renders) at
|
||
`demo/out/frames/`.
|
||
|
||
| Frame | Check | Result |
|
||
| --- | --- | --- |
|
||
| f120 opening | purpose, personas, scope readable | pass |
|
||
| f375 B2, f600 B3, f825 B4, f1020 B5 | proof legible, entrance settled | pass |
|
||
| f1230 B6 dashboard | legible at delivery resolution | pass after BUG-010 |
|
||
| f1440 B7, f1635 B8 | proof legible | pass |
|
||
| f1890 B9 dream | image agrees with caption | pass after BUG-013 |
|
||
| f2145 B10 recap | copy matches what was shown | pass |
|
||
| All 10 scene boundaries | no black/white/empty frame | pass after BUG-014 — 0 empty frames |
|
||
| Mean luma at all 10 boundaries | 17.49–23.21, smooth progression | pass (was flat 13.00 = bare background) |
|
||
| Contact sheet, 30 samples across 75 s | order matches storyboard, no repeats or stale content | pass |
|
||
|
||
Captures are presented at scale ≥ 1.0 — the 1600×900 terminal panels render
|
||
1:1 and the 1600×502 dashboard crop renders at 1.10 — so no evidence is
|
||
downscaled.
|
||
|
||
## 4. Walk the render as Jordan Washburn
|
||
|
||
One row per distinct screen a viewer reads.
|
||
|
||
| Screen | t | Viewer thought (verbatim, in their voice) | Verdict |
|
||
| --- | ---: | --- | --- |
|
||
| B1 opening | 0–9 s | "Right, it names the cluster and the image up front. And it says every number came from a real command — that's the claim I actually care about." | neutral-orientation |
|
||
| B2 convergence | 9–16 s | "Per-node, not the aggregate. Good — the aggregate is exactly what lied to me last week. lag=0 on all nine group-replicas and they agree on the leaders." | earns-interest |
|
||
| B3 boundary + write | 16–24 s | "401, 401, 200, then a 201 quorum ack. That single 201 is worth more than the rest of the page — it means DNS, TLS, the gateway, auth and Raft all worked in one request." | earns-interest |
|
||
| B4 isolation | 24–31 s | "Connection refused from gitea, 332 series to the scraper. That's the pair I'd want — it proves the policy is real without blinding monitoring." | earns-interest |
|
||
| B5 authority | 31–37 s | "403 not 401. So a leaked app key can't remove a member. That's the exposure I was worried about and it's closed." | earns-interest |
|
||
| B6 dashboard | 37–45 s | "That's the board I'd actually open. Health OK, reseed none, 33.3K vectors, and the legends finally say which node is which." | earns-interest |
|
||
| B7 recovery | 45–51 s | "3708/3708 and 48 of 48 volumes, selected by the schedule label. Good — picking the newest backup would have grabbed a canary." | earns-interest |
|
||
| B8 blind spot | 51–58 s | "It says NO REPORT instead of inventing 13 million events of lag. And it admits the exit code makes the deploy gate unusable. I trust a tool that tells me that." | earns-interest |
|
||
| B9a inert | 58–65 s | "It shows me what it cannot check yet, with the scrape count proving the scrape actually ran. That is the opposite of a green wall." | earns-interest |
|
||
| B9 dream | 65–75 s | "Wait — the doc I wrote said that was pending an image roll, and the harness proved it was already live. It caught my own documentation being wrong before I did. I want this running after every deploy." | earns-interest |
|
||
| B10 recap | 75–82 s | "32 green, and it names what it does not verify. That's the version I'd hand to someone else." | neutral-orientation |
|
||
|
||
No `fails` rows. The dream beat's thought shows genuine surprise and desire, so
|
||
it earns its rung.
|
||
|
||
## 5. Full render watch
|
||
|
||
Watched via the 30-sample contact sheet plus every boundary and proof frame; the
|
||
render is silent (Remotion emits a silent audio track), so the muted and
|
||
un-muted passes are identical and captions carry the whole narrative.
|
||
|
||
| Check | Result |
|
||
| --- | --- |
|
||
| Opening purpose and personas readable | pass — 9 s hold |
|
||
| Each proof state legible long enough | pass — 6–10 s, dense screens get the longer holds |
|
||
| Dense screens get more time | pass — dream 10 s, write 8 s, dashboard 8 s vs authority 6 s |
|
||
| Captions agree with the visible outcome | pass after BUG-013 |
|
||
| Transitions smooth, sections clear | pass after BUG-014 |
|
||
| Story explains value, not a screen list | pass — ladder order, not runbook order |
|
||
| No scene repeats, skips, contradicts, or overstates | pass |
|
||
| Audio artefacts | n/a — silent by design |
|
||
|
||
## Bug log (audit-originated)
|
||
|
||
Product/test defects found during verification are in
|
||
`capability-inventory.md`; these are the ones this visual audit produced.
|
||
|
||
| Bug | Asset | Severity | Expected | Actual | Root cause | Fix | Verification | Status |
|
||
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||
| BUG-010 | `CAP-010-dashboard`, scene B6 | high | Dashboard legible at delivery resolution | A 1600×1800 board scaled into 16:9 became unreadable — panel titles and numbers illegible | Capture was the full board; a near-square tall image cannot fit a 16:9 frame without heavy downscaling | Clip to the evidence band (latency + cluster correctness) using real element bounds, so the crop cannot drift with layout. Anchored to the row header and the last stat panel after a first attempt sliced the heading and bled a sliver of the next row | Re-captured 1600×502; renders at scale 1.10; re-inspected the capture and frame f1230 | verified |
|
||
| BUG-011 | 6 terminal captures | medium | Frames read as composed | 230–350 px of dead space below the content on sparse panels | `margin-top:auto` on the footer pushed the void into one block instead of distributing it | `.blocks { flex: 1; justify-content: center }` | Re-captured and individually re-inspected all six | verified |
|
||
| BUG-012 | `CAP-012-tidalctl` | medium | Colour does not imply a meaning the content lacks | "Exit code 2" rendered green, i.e. as a success, when it is the finding | Verdict line defaulted to the positive colour role | Marked the block `negative: true` (amber) and rewrote the text to name the consequence | Re-captured and re-inspected | verified |
|
||
| BUG-013 | scene B9 (dream) | **blocker** | Scene title, claim, and image agree | The dream caption described the runbook's stale claim and the probe that disproved it, while the image showed the unrelated inert-features panel | Beat reused an existing capture instead of one built for the claim | New `CAP-014-drift` capture showing the committed doc text from `git show`, the actually-running image, and the live 403/200 probe side by side | Re-captured, inspected, frame f1890 re-audited, walk-the-render row rewritten | verified |
|
||
| BUG-014 | every scene boundary | **blocker** | No empty frame between scenes | The video blinked to bare background for one frame at all 9 boundaries | Remotion Sequences do not overlap; each scene faded out over its last 8 frames while the next faded in from its own frame 0, so both sat at opacity 0 on the boundary | Removed the fade-out and started each Sequence `OVERLAP=10` frames early running long, producing a true 333 ms cross-dissolve. End times unchanged, so every storyboard hold is preserved | Re-rendered; boundary luma went from flat 13.00 (bare background) to 17.5–23.2 with smooth progression; boundary frames re-inspected | verified |
|
||
| BUG-015 | proof scenes | low | No duplicated headings | Remotion drew a scene heading above a capture that already carried its own title, producing a card-in-card with two competing titles | Composition and capture both owned a heading | Removed the heading from the proof scene; Remotion now owns only the capability badge, rung, and audience caption. Reclaimed 66 px for the evidence | Re-rendered and re-inspected f600, f1890 | verified |
|
||
|
||
## Approval
|
||
|
||
- Every promoted image, card, and clip has an individual review record with an
|
||
audience verdict of `perfect`. No `acceptable-with-note` verdicts exist, so no
|
||
screen was promoted on a soft pass.
|
||
- All blocker/high defects (BUG-010, BUG-013, BUG-014) are fixed and verified.
|
||
- Root causes are evidence-backed; no unresolved evidence gaps.
|
||
- Every scene proof and transition sample approved.
|
||
- Walk-the-render ledger complete for every hold and card, no `fails` rows, and
|
||
the dream beat records genuine surprise.
|
||
- Every `need` rung in the brief is served by at least one approved beat
|
||
(see `storyboard.md` ladder coverage).
|
||
- Audience brief re-confirmed at approval: viewing moment, decision-maker, and
|
||
revisit trigger unchanged.
|