15f6b11187
Turns docs/runbooks/deploy-verification.md from prose into 32 executable checks
against the live orchard9-k3sf cluster, and it found real defects on its first
run — including in the runbook it verifies.
WHY PLAYWRIGHT, HONESTLY
tidalDB serves zero HTML (no text/html, no Html(), 10 JSON routes), so this uses
Playwright in three distinct roles rather than pretending there is a UI:
* request fixture as a real HTTP client for DNS/TLS/auth/quorum/404;
* a browser for the only genuine screens in the chain, Grafana;
* a test harness for cluster-plane checks with no HTTP surface, shelling out
to kubectl and attaching the real transcript as evidence.
WHAT IT CAUGHT
* The runbook asserted the operator/data credential split was "not active yet
- requires an image roll". globalSetup read the live image and the live
secret; a probe returned data->403, admin->200. It had been enforcing the
whole time. Section 9 rewritten. (BUG-001)
* docs/ops/grafana-tidaldb.json shipped datasource uid ${DS_PROMETHEUS} - a
Grafana export-for-sharing placeholder with no __inputs block to resolve it.
Under ConfigMap provisioning every panel queried a datasource that did not
exist, so the whole board was blank. The API said "loaded" and I had only
ever checked the API. 41 refs fixed here, 58 across the fleet ConfigMap,
which was also blanking the postgres and redis dashboards. (BUG-007)
* Stat panels used calcs "lastNonNull". Grafana's reducer is "lastNotNull", so
no value was ever computed and Cluster health / Reseed pending / Indexed
vectors rendered as empty boxes. I chased panel width and then panel height
before comparing against a working stat panel elsewhere in the same Grafana.
A spelling error wearing a layout bug's clothes. (BUG-009)
* The namespace variable defaulted to All, so cluster panels silently included
tidaldb-586b544c8-vpkmw from the superseded standalone deployment. Latency
legends read "p50 p50 p50" with no way to tell the nodes apart. Both fixed.
* "5xx ratio" rendered "No data" as large green text - at a glance a healthy
value. And Fleet state gave three fields one shared green threshold, so
reseed_required=1 would have shown GREEN during the exact incident the panel
exists to surface. Split into three panels with per-field mappings.
* tidalctl cluster-status exits 2 on a FULLY CONVERGED cluster, because the
aggregated endpoint reports healthy peers as region=null applied=0
reachable=false. The runbook claimed `cluster-status && deploy` was a safe
gate; that claim came from an exit code masked by a shell pipeline. The gate
can never pass here. Documented, test pins it, engine defect recorded.
(BUG-005)
* The deployed image writes ANSI colour into container logs, which the
collector stores verbatim. Already fixed in logging.rs, not yet rolled;
pinned as a tripwire. (BUG-006)
* The runbook's own backup command sorted ALL backups by timestamp and
selected a restore-canary run: 20 items, one volume, a meaningless pass.
Now filters on the schedule label the freshness alert actually watches.
DEFECTS FOUND BY LOOKING AT THE SCREENS
Six of the first eight captures were slop and were fixed, not promoted:
230-350px of dead space; a verdict that rendered "exit code 2" in green; the
1600x1800 dashboard scaled into 16:9 until illegible (now clipped to the
evidence band using real element bounds); the dream beat whose caption described
a contradiction the image did not show (now a purpose-built capture holding the
committed doc text, the running image, and the live 403/200 side by side); and a
one-frame blink to bare background at every scene boundary, because Remotion
Sequences do not overlap and both scenes sat at opacity 0 on the boundary frame.
TRIPWIRES IN THE HONEST DIRECTION
Three tests assert what is ABSENT - zero tidaldb_http_* families, JSON_LOGS
unset, plain-text logs - and each carries the message "good news, roll the
runbook section from pending to live". The metric-absence test also asserts the
baseline family count, so "absent" cannot pass for "the scrape failed". That is
the drift that made section 9 stale in the first place.
Regression config uses workers:1 and retries:0 deliberately: a live-cluster
check that only passes on the second attempt has told you something true.
Verified: 32 passed (46.8s); 9 demo captures each asserting before photographing;
tsc clean; render 82.05s 1920x1080 h264, 0 empty frames across 10 boundaries;
every promoted image inspected individually and judged perfect; walk-the-render
ledger complete with no fails.
111 lines
6.4 KiB
Markdown
111 lines
6.4 KiB
Markdown
# Audience brief — tidalDB deploy verification
|
|
|
|
Required input to the capability inventory's prioritization, the storyboard, and
|
|
every per-screen verdict. Written before the storyboard, per the audience
|
|
protocol.
|
|
|
|
## Viewing moment
|
|
|
|
- **Room / delivery.** Sent as a link, watched alone, no narrator present. The
|
|
viewer opens it after a deploy — either because they are about to trust the
|
|
cluster with something, or because something already looks wrong and they want
|
|
to know which layer to suspect. There is nobody to ask "wait, what was that
|
|
screen?", so every frame must survive being read cold.
|
|
- **Decision sought.** "Do I believe this deployment is verified, and can I hand
|
|
this procedure to someone else and expect the same answer?" Not a purchase —
|
|
an *adoption* decision about a verification procedure.
|
|
- **Prior exposure.** The viewer has already seen this cluster lie to them. In
|
|
the last week: a reseed livelock where every node reported healthy while the
|
|
cluster could not make progress; an aggregated status endpoint that reported
|
|
two converged peers as `UNREACHABLE PARTITIONED` at 13.3M events of lag; a
|
|
monitoring alert that outlived the DNS record it watched. They have also seen
|
|
four separate false alarms produced by the *measurement*, not the system.
|
|
- **First-30-seconds judgment.** Whether the evidence is real or narrated. A
|
|
screen that says "all systems healthy" earns instant distrust, because that is
|
|
exactly what the livelocked cluster said. Numbers, exit codes, and a visible
|
|
negative result buy credibility. Adjectives spend it.
|
|
|
|
## Viewers
|
|
|
|
- **Actual decision-maker: Jordan Washburn** — the operator who owns this
|
|
cluster and asked for a procedure they can walk. Judges whether each step is
|
|
*runnable* and whether the claim matches what the command actually printed.
|
|
Has personally been burned by inferring health from the absence of bad news,
|
|
so an unproven assertion reads as a defect rather than an optimism.
|
|
|
|
- **Quality-bar judge: Kyle Kingsbury (Aphyr, author of Jepsen)** — the hardest
|
|
plausible judge for a distributed-database verification claim, and already
|
|
this project's `@tidal-distributed` persona. He has spent a career
|
|
demonstrating that databases which pass their own test suites still lose
|
|
committed writes, and his standard method is to distrust the system's
|
|
self-report and check for the property from outside.
|
|
|
|
What he would demand:
|
|
1. **A real acknowledged write, not a health endpoint.** `/health` returning
|
|
200 proves a process is running. He would ask whether a write was
|
|
acknowledged by a *quorum* and survived, and would treat any verification
|
|
that stops at readiness probes as unserious.
|
|
2. **The negative case, executed.** Every claim of a boundary — auth, network
|
|
isolation, operator authority — must show the denial actually happening.
|
|
An untested boundary is an assumption wearing a boundary's clothes.
|
|
3. **Honesty about what is not proven.** He would rather see "these three
|
|
tests are red and here is the bisect that shows why" than a green wall. A
|
|
demo that hides its gaps fails his bar no matter how polished the frames.
|
|
|
|
## Ladder
|
|
|
|
- **Need.** The evidence is real: every number on screen came from a command
|
|
that ran against the live cluster, and the commands are visible.
|
|
- **Need.** A write is proven *committed by quorum*, not merely accepted — the
|
|
one claim a database cannot fake with a status endpoint.
|
|
- **Need.** Every security boundary shows its denial: unauthenticated request
|
|
refused, wrong credential refused, foreign pod refused at the network layer,
|
|
data credential refused on an operator verb.
|
|
- **Need.** What is *not* verified is stated on screen, with the reason — no
|
|
green wall.
|
|
- **Want.** The whole procedure is one command a different person can run, and
|
|
it fails loudly rather than skipping when a prerequisite is missing.
|
|
- **Want.** The operator/data authority split is visibly enforced, so an
|
|
application key cannot remove a cluster member.
|
|
- **Dream (exactly one).** The harness catches a stale claim in the runbook it
|
|
is verifying — on its first run, before any human notices. Verification that
|
|
audits its own documentation, rather than drifting away from it. This actually
|
|
happened during the build: the doc asserted the admin credential split was
|
|
"not active yet, requires an image roll", and the harness's own startup probe
|
|
read the live image and the live secret and proved the gate was already
|
|
enforcing. The document was wrong; the harness said so.
|
|
- *Runners-up (recorded for future demos):* the port-forward poller that
|
|
replaces the runbook's empirical `sleep 8` with a deterministic wait; the
|
|
credential poller hot-loading an admin key with no restart while the boot
|
|
log still claims it is missing.
|
|
|
|
## Vocabulary and realism notes
|
|
|
|
- **Use:** quorum-acked, converged, applied/lag, frontier, reseed, NetworkPolicy,
|
|
bearer, 401/403, PodVolumeBackup, exit code. This viewer's domain is
|
|
operations and distributed systems; these are the correct words.
|
|
- **Avoid:** "seamless", "robust", "enterprise-grade", "fully verified", any
|
|
adjective substituting for a number. Avoid "all systems healthy" outright —
|
|
that phrasing is what the livelocked cluster reported.
|
|
- **Data realism.** All data is the real production corpus: 33,331 vectors,
|
|
applied positions above 13.3M, real pod names (`tidaldb-0..2`), real
|
|
namespaces, real node IPs, a real Let's Encrypt certificate. Nothing is
|
|
seeded for the demo. The one write this suite performs uses entity ids in a
|
|
reserved `999_000_0xx` band so a verification write is never mistaken for
|
|
corpus data — that band is visible on screen and explained.
|
|
- **Secrets.** Bearer tokens, the admin key, and the Grafana password are
|
|
redacted to `«redacted»` by the harness before any attachment or capture is
|
|
written. A screen showing a live bearer would be an immediate `slop` verdict.
|
|
|
|
## Revisit trigger
|
|
|
|
Rewrite this brief when any of the following change:
|
|
|
|
- The observability image is rolled (§9.1/§9.3 stop being inert, and the dream
|
|
beat's specific stale claim is no longer the live example).
|
|
- The three red `cluster_reseed` tests go green, or are proven to be a product
|
|
defect rather than a harness/environment race.
|
|
- The audience shifts from the operator who owns the cluster to an external
|
|
party (a customer, an auditor) — that is a different room, a different
|
|
decision, and a different ladder.
|