jordan/tidaldb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test(e2e): Playwright evidence harness for the deploy-verification runbook

Browse Source 
Turns docs/runbooks/deploy-verification.md from prose into 32 executable checks
against the live orchard9-k3sf cluster, and it found real defects on its first
run — including in the runbook it verifies.

WHY PLAYWRIGHT, HONESTLY
tidalDB serves zero HTML (no text/html, no Html(), 10 JSON routes), so this uses
Playwright in three distinct roles rather than pretending there is a UI:
  * request fixture as a real HTTP client for DNS/TLS/auth/quorum/404;
  * a browser for the only genuine screens in the chain, Grafana;
  * a test harness for cluster-plane checks with no HTTP surface, shelling out
    to kubectl and attaching the real transcript as evidence.

WHAT IT CAUGHT
  * The runbook asserted the operator/data credential split was "not active yet
    - requires an image roll". globalSetup read the live image and the live
    secret; a probe returned data->403, admin->200. It had been enforcing the
    whole time. Section 9 rewritten. (BUG-001)
  * docs/ops/grafana-tidaldb.json shipped datasource uid ${DS_PROMETHEUS} - a
    Grafana export-for-sharing placeholder with no __inputs block to resolve it.
    Under ConfigMap provisioning every panel queried a datasource that did not
    exist, so the whole board was blank. The API said "loaded" and I had only
    ever checked the API. 41 refs fixed here, 58 across the fleet ConfigMap,
    which was also blanking the postgres and redis dashboards. (BUG-007)
  * Stat panels used calcs "lastNonNull". Grafana's reducer is "lastNotNull", so
    no value was ever computed and Cluster health / Reseed pending / Indexed
    vectors rendered as empty boxes. I chased panel width and then panel height
    before comparing against a working stat panel elsewhere in the same Grafana.
    A spelling error wearing a layout bug's clothes. (BUG-009)
  * The namespace variable defaulted to All, so cluster panels silently included
    tidaldb-586b544c8-vpkmw from the superseded standalone deployment. Latency
    legends read "p50 p50 p50" with no way to tell the nodes apart. Both fixed.
  * "5xx ratio" rendered "No data" as large green text - at a glance a healthy
    value. And Fleet state gave three fields one shared green threshold, so
    reseed_required=1 would have shown GREEN during the exact incident the panel
    exists to surface. Split into three panels with per-field mappings.
  * tidalctl cluster-status exits 2 on a FULLY CONVERGED cluster, because the
    aggregated endpoint reports healthy peers as region=null applied=0
    reachable=false. The runbook claimed `cluster-status && deploy` was a safe
    gate; that claim came from an exit code masked by a shell pipeline. The gate
    can never pass here. Documented, test pins it, engine defect recorded.
    (BUG-005)
  * The deployed image writes ANSI colour into container logs, which the
    collector stores verbatim. Already fixed in logging.rs, not yet rolled;
    pinned as a tripwire. (BUG-006)
  * The runbook's own backup command sorted ALL backups by timestamp and
    selected a restore-canary run: 20 items, one volume, a meaningless pass.
    Now filters on the schedule label the freshness alert actually watches.

DEFECTS FOUND BY LOOKING AT THE SCREENS
Six of the first eight captures were slop and were fixed, not promoted:
230-350px of dead space; a verdict that rendered "exit code 2" in green; the
1600x1800 dashboard scaled into 16:9 until illegible (now clipped to the
evidence band using real element bounds); the dream beat whose caption described
a contradiction the image did not show (now a purpose-built capture holding the
committed doc text, the running image, and the live 403/200 side by side); and a
one-frame blink to bare background at every scene boundary, because Remotion
Sequences do not overlap and both scenes sat at opacity 0 on the boundary frame.

TRIPWIRES IN THE HONEST DIRECTION
Three tests assert what is ABSENT - zero tidaldb_http_* families, JSON_LOGS
unset, plain-text logs - and each carries the message "good news, roll the
runbook section from pending to live". The metric-absence test also asserts the
baseline family count, so "absent" cannot pass for "the scrape failed". That is
the drift that made section 9 stale in the first place.

Regression config uses workers:1 and retries:0 deliberately: a live-cluster
check that only passes on the second attempt has told you something true.

Verified: 32 passed (46.8s); 9 demo captures each asserting before photographing;
tsc clean; render 82.05s 1920x1080 h264, 0 empty frames across 10 boundaries;
every promoted image inspected individually and judged perfect; walk-the-render
ledger complete with no fails.
This commit is contained in:
jordan 2026-08-23 14:03:29 -06:00
parent d21a202a56
commit 15f6b11187
46 changed files with 10047 additions and 179 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
.gitignore vendored
View File

@ -39,3 +39,11 @@ tmp/
.claude/worktrees/
.sdlc/tools/*/index/
.sdlc/telemetry.redb
# Playwright deploy-verification harness artifacts
test-results/
playwright-report/
playwright-report-demo/
playwright/.auth/
demo/out/
demo/.cache/

110
demo/audience-brief.md Normal file
View File

@ -0,0 +1,110 @@
# Audience brief — tidalDB deploy verification
Required input to the capability inventory's prioritization, the storyboard, and
every per-screen verdict. Written before the storyboard, per the audience
protocol.
## Viewing moment
- **Room / delivery.** Sent as a link, watched alone, no narrator present. The
viewer opens it after a deploy — either because they are about to trust the
cluster with something, or because something already looks wrong and they want
to know which layer to suspect. There is nobody to ask "wait, what was that
screen?", so every frame must survive being read cold.
- **Decision sought.** "Do I believe this deployment is verified, and can I hand
this procedure to someone else and expect the same answer?" Not a purchase —
an *adoption* decision about a verification procedure.
- **Prior exposure.** The viewer has already seen this cluster lie to them. In
the last week: a reseed livelock where every node reported healthy while the
cluster could not make progress; an aggregated status endpoint that reported
two converged peers as `UNREACHABLE PARTITIONED` at 13.3M events of lag; a
monitoring alert that outlived the DNS record it watched. They have also seen
four separate false alarms produced by the *measurement*, not the system.
- **First-30-seconds judgment.** Whether the evidence is real or narrated. A
screen that says "all systems healthy" earns instant distrust, because that is
exactly what the livelocked cluster said. Numbers, exit codes, and a visible
negative result buy credibility. Adjectives spend it.
## Viewers
- **Actual decision-maker: Jordan Washburn** — the operator who owns this
cluster and asked for a procedure they can walk. Judges whether each step is
*runnable* and whether the claim matches what the command actually printed.
Has personally been burned by inferring health from the absence of bad news,
so an unproven assertion reads as a defect rather than an optimism.
- **Quality-bar judge: Kyle Kingsbury (Aphyr, author of Jepsen)** — the hardest
plausible judge for a distributed-database verification claim, and already
this project's `@tidal-distributed` persona. He has spent a career
demonstrating that databases which pass their own test suites still lose
committed writes, and his standard method is to distrust the system's
self-report and check for the property from outside.
What he would demand:
1. **A real acknowledged write, not a health endpoint.** `/health` returning
200 proves a process is running. He would ask whether a write was
acknowledged by a *quorum* and survived, and would treat any verification
that stops at readiness probes as unserious.
2. **The negative case, executed.** Every claim of a boundary — auth, network
isolation, operator authority — must show the denial actually happening.
An untested boundary is an assumption wearing a boundary's clothes.
3. **Honesty about what is not proven.** He would rather see "these three
tests are red and here is the bisect that shows why" than a green wall. A
demo that hides its gaps fails his bar no matter how polished the frames.
## Ladder
- **Need.** The evidence is real: every number on screen came from a command
that ran against the live cluster, and the commands are visible.
- **Need.** A write is proven *committed by quorum*, not merely accepted — the
one claim a database cannot fake with a status endpoint.
- **Need.** Every security boundary shows its denial: unauthenticated request
refused, wrong credential refused, foreign pod refused at the network layer,
data credential refused on an operator verb.
- **Need.** What is *not* verified is stated on screen, with the reason — no
green wall.
- **Want.** The whole procedure is one command a different person can run, and
it fails loudly rather than skipping when a prerequisite is missing.
- **Want.** The operator/data authority split is visibly enforced, so an
application key cannot remove a cluster member.
- **Dream (exactly one).** The harness catches a stale claim in the runbook it
is verifying — on its first run, before any human notices. Verification that
audits its own documentation, rather than drifting away from it. This actually
happened during the build: the doc asserted the admin credential split was
"not active yet, requires an image roll", and the harness's own startup probe
read the live image and the live secret and proved the gate was already
enforcing. The document was wrong; the harness said so.
- *Runners-up (recorded for future demos):* the port-forward poller that
replaces the runbook's empirical `sleep 8` with a deterministic wait; the
credential poller hot-loading an admin key with no restart while the boot
log still claims it is missing.
## Vocabulary and realism notes
- **Use:** quorum-acked, converged, applied/lag, frontier, reseed, NetworkPolicy,
bearer, 401/403, PodVolumeBackup, exit code. This viewer's domain is
operations and distributed systems; these are the correct words.
- **Avoid:** "seamless", "robust", "enterprise-grade", "fully verified", any
adjective substituting for a number. Avoid "all systems healthy" outright —
that phrasing is what the livelocked cluster reported.
- **Data realism.** All data is the real production corpus: 33,331 vectors,
applied positions above 13.3M, real pod names (`tidaldb-0..2`), real
namespaces, real node IPs, a real Let's Encrypt certificate. Nothing is
seeded for the demo. The one write this suite performs uses entity ids in a
reserved `999_000_0xx` band so a verification write is never mistaken for
corpus data — that band is visible on screen and explained.
- **Secrets.** Bearer tokens, the admin key, and the Grafana password are
redacted to `«redacted»` by the harness before any attachment or capture is
written. A screen showing a live bearer would be an immediate `slop` verdict.
## Revisit trigger
Rewrite this brief when any of the following change:
- The observability image is rolled (§9.1/§9.3 stop being inert, and the dream
beat's specific stale claim is no longer the live example).
- The three red `cluster_reseed` tests go green, or are proven to be a product
defect rather than a harness/environment race.
- The audience shifts from the operator who owns the cluster to an external
party (a customer, an auditor) — that is a different room, a different
decision, and a different ladder.

353
demo/capability-inventory.md Normal file
View File

@ -0,0 +1,353 @@
# Capability inventory — tidalDB deploy verification
Scope: every claim `docs/runbooks/deploy-verification.md` makes about the live
`orchard9-k3sf` deployment. One row per *capability* (a verifiable property),
not per command — several commands can serve one property, and one command can
touch several.
## Sources reconciled
| Source | What it contributed |
| --- | --- |
| `docs/runbooks/deploy-verification.md` | The nine sections and their pass criteria. |
| `tidal-server/src/router.rs` | The 10 registered routes; which are open vs bearer-gated vs admin-gated. |
| `k8s/cluster/ingress.yaml` | The published path allowlist — what is reachable from the internet at all. |
| `k8s/cluster/networkpolicy.yaml` | Ingress rules for `:9091`/`:9601` and the deliberate `:9500` exception. |
| `k8s/cluster/statefulset.yaml` | Probe ports, env, resource requests (and the live-vs-manifest drift). |
| `docs/ops/observability.md` | The four observability surfaces and the aggregated-status caveat. |
| `docs/ops/grafana-tidaldb.json` | The 13 dashboard panels and their PromQL. |
| Live cluster (`kubectl`, `curl`) | Actual image digest, actual secret keys, actual behavior. |
**Verified against.** Image
`registry.threesix.ai/tidal/server:m12-admin-gate-20260823@sha256:6e220060…`,
pods started 2026-08-23 05:32–05:41 UTC, namespace `tidaldb-cluster`,
repo commit recorded per-run in `E2E_BUILD_REVISION`.
**Environment gate discovered during inventory.** The running image predates the
observability commit (`4766f56`), so HTTP request metrics and structured logs are
absent by construction, while the operator/data credential split from `388e445`
*is* live. The runbook asserted all three were pending; that was stale. See
`BUG-001`.
## Inventory
### CAP-001 — Cluster membership is complete and every voter is Ready
- **Area.** `kubectl -n tidaldb-cluster get pods`; StatefulSet `tidaldb`.
- **Business purpose.** Three voters is the minimum for quorum with one-node
fault tolerance. Two Ready pods still serve reads and writes but tolerate no
further loss — a state that looks fine and is one failure from unavailable.
- **Personas.** Operator.
- **Primary workflow.** List pods by label; assert every known pod is Ready and
that the set is exactly the expected three.
- **Edge cases.** A fourth unexpected pod (mid-scale or orphan); a pod Running
but not Ready; a climbing restart count.
- **Permission boundary.** None — cluster-plane read.
- **Dependencies.** kubeconfig; the StatefulSet.
- **Observability.** Pod `ready` condition, `restartCount`, `startTime`.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/01-cluster-convergence.spec.ts`
### CAP-002 — Every node has converged: zero lag, no reseed, one agreed leader
- **Area.** `GET /cluster/status/local` on each pod via port-forward.
- **Business purpose.** The property that actually matters. A pod can be Ready
while its replication is stalled or it is rebuilding from a snapshot. This is
the check the 2026-08-20 reseed livelock defeated, which is why it is asked of
each node individually rather than of the aggregate.
- **Personas.** Operator.
- **Primary workflow.** For each pod: port-forward `:9500`, GET
`/cluster/status/local` with the data bearer, assert `reseed_required=false`
and `lag_events=0` on every shard group, and that all pods name the same
leader.
- **Edge cases.** `reseed_required=true` surviving a restart (livelock
signature); non-zero lag on one follower; disagreeing leaders (split brain);
the leader's own un-shipped tail showing as a small applied-position
difference, which is *not* lag.
- **Permission boundary.** Requires the data bearer since `388e445` moved
`/cluster/status*` behind auth; an unauthenticated request must be refused.
- **Dependencies.** port-forward; the data bearer.
- **Observability.** `applied_events`, `lag_events`, `leader`, `term`,
`reseed_required` per shard.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/01-cluster-convergence.spec.ts`
### CAP-003 — Public DNS resolves to every node
- **Area.** `tidaldb.threesix.ai` A records.
- **Business purpose.** Three A records is the load distribution and the
failover story. One missing record silently concentrates all traffic on two
nodes. A *retired* record that monitoring still watches produces a permanent
false alarm — which is exactly what happened to the previous hostname.
- **Personas.** Operator; any external API consumer.
- **Primary workflow.** Resolve the hostname for real and assert the address set
equals the three known node IPs.
- **Edge cases.** Split-DNS resolver returning a different set than public DNS;
partial record set; a stale record for a retired host.
- **Permission boundary.** None.
- **Dependencies.** Cloudflare zone `threesix.ai`.
- **Observability.** Resolved address list, attached per run.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/smoke.spec.ts`, `tests/e2e/features/02-public-endpoint.spec.ts`
### CAP-004 — TLS identity is correct and not near expiry
- **Area.** TLS handshake on `:443` for `tidaldb.threesix.ai`.
- **Business purpose.** A client that cannot validate the certificate either
fails closed (outage) or is configured to skip validation (silent downgrade to
no transport security). Expiry is the most common self-inflicted outage in this
class.
- **Personas.** External API consumer.
- **Primary workflow.** Complete a real handshake with validation enabled;
assert subject CN, a Let's Encrypt issuer, and at least 30 days of remaining
validity.
- **Edge cases.** Wrong SAN; self-signed fallback; cert valid but chain
incomplete; expiry inside the renewal window.
- **Permission boundary.** None.
- **Dependencies.** cert-manager `letsencrypt-prod` (Cloudflare dns01), Traefik.
- **Observability.** Subject, issuer, `notAfter`, days remaining.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/02-public-endpoint.spec.ts`
### CAP-005 — The data plane refuses unauthenticated and wrong credentials
- **Area.** `GET /search` (and the other published data paths) via the ingress.
- **Business purpose.** This endpoint is on the open internet. The bearer is the
only thing between the corpus and anyone who finds the hostname.
- **Personas.** External API consumer (allowed); anonymous internet (denied).
- **Primary workflow.** Same path three ways: no credential → 401, wrong
credential → 401, correct credential → 200 with a body.
- **Edge cases.** A 200 for the wrong bearer would be the maximum-severity
finding in this suite. A 500 on a bad credential would leak that the key was
parsed but mishandled.
- **Permission boundary.** `Authorization: Bearer` compared with
`subtle::ConstantTimeEq`.
- **Dependencies.** Secret `tidaldb-credentials`; Traefik.
- **Observability.** Status codes for all three attempts, recorded together.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/03-auth-boundary.spec.ts`
### CAP-006 — A quorum-acked write commits end to end
- **Area.** `POST /items` with `x-tidal-ack: quorum` via the ingress.
- **Business purpose.** The single strongest check available. One request
traverses DNS, TLS, Traefik, the backend TLS hop, authentication, and Raft
replication; `201` means a quorum acknowledged the write, not that one node
accepted it. This is the claim the quality-bar judge would ask for first and
the one a health endpoint cannot fake.
- **Personas.** External API consumer.
- **Primary workflow.** POST an item in the reserved `999_000_0xx` id band with
quorum ack; assert `201`.
- **Edge cases.** `202` (accepted, not quorum-acked) must fail. A timeout under
quorum loss. `503` when the leader is mid-election.
- **Permission boundary.** Requires the data bearer.
- **Dependencies.** Whole stack, plus a live leader and two reachable followers.
- **Observability.** Status, body, and the pre/post applied position.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/03-auth-boundary.spec.ts`, `tests/e2e/smoke.spec.ts`
### CAP-007 — Operator and metrics surfaces are unroutable from the internet
- **Area.** `/cluster/status`, `/cluster/members`, `/metrics`, `/openapi.json`
through the public ingress.
- **Business purpose.** `/cluster/status` leaks leader identity, membership and
sequence positions; `/metrics` leaks corpus size unauthenticated. Neither is
in the published allowlist, so both must 404 at the gateway — a defence that
holds even if a future build were to forget an auth check.
- **Personas.** Anonymous internet (denied).
- **Primary workflow.** Request each path publicly; assert 404 (gateway
rejection), not 401 (reached the app).
- **Edge cases.** A 401 here is a *finding*: it means the request reached the
application and only the credential check stopped it, so the allowlist has
drifted.
- **Permission boundary.** Traefik path allowlist in `k8s/cluster/ingress.yaml`.
- **Dependencies.** Ingress.
- **Observability.** Status per path.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/03-auth-boundary.spec.ts`
### CAP-008 — The metrics port is not reachable from arbitrary pods
- **Area.** NetworkPolicy `tidaldb`; pod `:9091` and `:9601`.
- **Business purpose.** Before the policy existed, any pod anywhere in the
cluster could read the unauthenticated metrics endpoint. The policy must deny
a foreign namespace while still admitting the scraper — a rule that blocks
both is an outage of observability, and a rule that blocks neither is
theatre.
- **Personas.** Foreign workload (denied); vmagent scraper (allowed).
- **Primary workflow.** From `threesix/gitea-0`, expect connection refused on
`:9091`. From `observability/vmagent`, expect a few hundred `tidaldb_` series.
- **Edge cases.** `:9500` is deliberately left open because all three probes
originate from the node — a wrong rule there fails liveness at 6×10s and
restarts every pod. A too-short client timeout returns zero lines and reads
exactly like "the metric is gone" (see `BUG-002`).
- **Permission boundary.** The NetworkPolicy itself is the boundary under test.
- **Dependencies.** CNI policy enforcement; the foreign pod existing.
- **Observability.** Both command transcripts, attached side by side.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/04-network-isolation.spec.ts`
### CAP-009 — Metrics reach the platform with the labels the dashboard queries
- **Area.** vmsingle `/api/v1/series`.
- **Business purpose.** The dashboard's `$namespace`/`$pod` template variables
depend on exact label names. A renamed or dropped label leaves every panel
blank while the underlying series still exists — a monitoring outage that
looks like a product outage.
- **Personas.** Operator.
- **Primary workflow.** Query series for `tidaldb_health_ok`; assert non-zero
and that `namespace`, `pod`, `container`, `partition_id` are all present.
- **Edge cases.** Stale series from deleted pods inflating the count; a label
present but empty.
- **Permission boundary.** None (in-cluster, port-forwarded).
- **Dependencies.** vmagent scrape config; vmsingle.
- **Observability.** Series count and the label set.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/05-metrics-dashboard.spec.ts`
### CAP-010 — The Grafana dashboard is loaded and its panels render
- **Area.** Grafana, dashboard uid `tidaldb-overview`, folder Databases.
- **Business purpose.** The dashboard is the operator's first stop during an
incident. "Loaded" is not enough — a loaded dashboard whose panels all read
"No data" is worse than no dashboard, because it implies the system is idle.
**This is the only genuine browser surface in the entire evidence chain.**
- **Personas.** Operator.
- **Primary workflow.** Log in, open the dashboard, wait for panels to settle,
screenshot the whole board and each populated panel, and classify every panel
as populated or empty.
- **Edge cases.** Five panels are *legitimately* empty pre-roll because they
query `tidaldb_http_*`, which the running image does not emit. That must be
asserted as expected-empty, not silently tolerated — otherwise a real future
regression to those panels is invisible.
- **Permission boundary.** Grafana admin credential; the dashboard must not be
anonymously readable.
- **Dependencies.** Grafana provisioning ConfigMap; vmsingle datasource.
- **Observability.** Panel-by-panel populated/empty classification.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/05-metrics-dashboard.spec.ts`
### CAP-011 — Container logs are readable and free of unexpected errors
- **Area.** `kubectl logs` for each pod.
- **Business purpose.** Logs are the first diagnostic surface. The deployed
image emits plain text, so `level:error` filtering in VictoriaLogs does *not*
work yet — filtering must happen at the source until the observability image
is rolled. Saying so prevents an operator concluding "no errors" from a query
that cannot match.
- **Personas.** Operator.
- **Primary workflow.** Read recent logs per pod; assert no ANSI escape
fragments; classify WARN/ERROR lines and assert the only WARNs are the known,
documented set.
- **Edge cases.** The `TIDAL_ADMIN_KEY is not set` WARN is present *and stale* —
the key was hot-loaded later and the gate is live. A boot-log WARN is not
proof of current state (see `BUG-003`).
- **Permission boundary.** None — cluster-plane read.
- **Dependencies.** kubelet log retention.
- **Observability.** The log lines themselves.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/06-logs.spec.ts`
### CAP-012 — `tidalctl` interrogates a live cluster and its exit codes gate
- **Area.** `tidalctl cluster-status`, `watch`.
- **Business purpose.** Gives an operator a live view without hand-rolling curl,
and an exit code a deploy script can branch on. `cluster-status && deploy` is
only safe if a non-converged cluster really exits non-zero.
- **Personas.** Operator; deploy automation.
- **Primary workflow.** Run `cluster-status` against a port-forwarded node with
the internal CA (`--insecure` required); assert a leader line, a region table,
a shard table. Run `watch --count` and assert one line per tick, terminating.
- **Edge cases.** The aggregated-status gap: a peer with no frontier report is
reported `applied=0` with lag derived against that zero, so a converged peer
can look 13.3M behind. `tidalctl` must print `NO REPORT`, not lag. Bad
credential → exit 2; malformed `--url` → exit 1.
- **Permission boundary.** Data bearer accepted; a wrong one must exit 2.
- **Dependencies.** Built `target/debug/tidalctl`; port-forward.
- **Observability.** stdout plus exit code.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/07-tidalctl.spec.ts`
### CAP-013 — The fleet backup completed and captured every volume
- **Area.** Velero `Backup` and `PodVolumeBackup` in `backup-system`.
- **Business purpose.** The recovery story. A single failed PodVolumeBackup
marks the whole backup `PartiallyFailed` and freezes
`velero_backup_last_successful_timestamp`, so the alert fires even though the
volumes that matter were captured.
- **Personas.** Operator.
- **Primary workflow.** Select the newest backup **carrying the fleet schedule
label**, assert `Completed`, no errors, `itemsBackedUp == totalItems`, and
every PVB `Completed`.
- **Edge cases.** Sorting all backups by timestamp selects a `restore-canary`
run (20 items, 1 volume) which would "pass" while proving nothing about the
fleet — the defect this inventory step exists to prevent (`BUG-004`).
- **Permission boundary.** None — cluster-plane read.
- **Dependencies.** Velero; the `velero-fleet-daily` schedule; node-agent.
- **Observability.** Phase, item counts, per-PVB phase histogram.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/08-backups.spec.ts`
### CAP-014 — Operator authority is separated from data-plane access
- **Area.** `POST /cluster/promote` and siblings, port-forwarded.
- **Business purpose.** Without the split, the application key that any client
holds can remove a cluster member, force a partition, or transfer a shard.
**Live on this deployment**, contrary to what the runbook said.
- **Personas.** Operator (allowed); application/data client (denied).
- **Primary workflow.** Data bearer on `/cluster/promote` → 403 (authenticated,
not authorized). Admin bearer → not 401/403.
- **Edge cases.** The boot log still WARNs that the key is unset because it was
materialized after startup and hot-loaded by the credential poller — the log
is stale, the behavior is authoritative. When no admin key exists at all the
gate degrades to previous behavior by design, so the test must distinguish
"absent" from "broken".
- **Permission boundary.** The `admin_gate` itself. Peer-callable verbs
(`/cluster/catchup`, `/join`, `/members`, `/reconcile*`) stay on the plain
bearer deliberately — gating them would break replication.
- **Dependencies.** `TIDAL_ADMIN_KEY` in `tidaldb-credentials`; the projected
secret volume; the credential poller.
- **Observability.** Both status codes, plus the presence of the mounted key file.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/09-operator-authority.spec.ts`
### CAP-015 — Inert observability features are inert for a known reason
- **Area.** `tidaldb_http_requests_total`; `JSON_LOGS`.
- **Business purpose.** A tripwire in the honest direction. These are committed
and unit-tested but absent from the running image. Asserting their *current
absence* means the day someone rolls the observability image, this test fails
and tells them to update the runbook — instead of the runbook quietly rotting,
which is precisely what happened to §9.2.
- **Personas.** Operator.
- **Primary workflow.** Scrape each pod's `:9091` with a generous timeout;
assert zero `tidaldb_http_*` series and record it as expected-for-this-image.
Assert `JSON_LOGS` is absent from the StatefulSet env.
- **Edge cases.** A short scrape timeout returns zero lines for *every* metric
and would make this test pass for the wrong reason — so it also asserts the
ordinary `tidaldb_` series are present, proving the scrape actually worked.
- **Permission boundary.** None.
- **Dependencies.** vmagent exec; the StatefulSet spec.
- **Observability.** Both counts, so "absent" is distinguishable from "unscraped".
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/09-operator-authority.spec.ts`
## Intentionally excluded
| Capability | Why lower leverage | Alternate evidence | Owner / revisit |
| --- | --- | --- | --- |
| Failover under induced node loss | Deliberately destructive against the production cluster this suite verifies. Killing a voter to watch election is a game-day exercise, not a post-deploy check. | `tidal-server` cluster e2e suite (`cluster_runbook`, `cluster_reseed`) exercises election and reseed against ephemeral multi-process clusters. | Operator; revisit when a staging cluster exists. |
| Restore from the Velero backup | Restoring over live data is unacceptable; a restore drill needs an isolated target namespace. | `docs/runbooks/disaster-recovery.md` manual drill; `restore-canary-*` Backup objects prove the restore path independently. | Operator; revisit at the next DR drill. |
| Read/write throughput and latency SLA | Load generation against production would distort the very metrics the dashboard checks assert on. | `docs/ops/stress-test-*.md`, `tidal-stress`; nightly soak. | Operator; not a deploy gate. |
| VictoriaLogs LogsQL query surface | `/select/logsql/query` returns "unsupported path requested" on this build, and the deployed image emits plain text so `level:error` cannot match anyway. | `kubectl logs` filtering at source (CAP-011), documented as the current method. | Operator; revisit when the observability image is rolled. |
| Traefik rate-limit thresholds (200/400) | Proving the limit requires deliberately flooding a production ingress shared with 35 other services. | Middleware config asserted declaratively in `k8s/cluster/ingress.yaml`. | Operator; revisit with a dedicated test host. |
| The `tidaldb` namespace standalone Deployment | Live `1/1` but nothing routes to it; the fleet record calls it superseded. Verifying it would legitimise a surface that should be retired. | Noted as open drift in the runbook's closing section. | Operator; revisit by retiring it. |
## Bug log
| Bug ID | Capability | Severity | Expected | Actual | Evidence | Root cause | Fix | Verification | Status |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BUG-001 | CAP-014 | High | The runbook describes the live state of the credential split. | Runbook §9 said the split was "not active yet — requires an image roll"; it was already live and enforcing. | Harness global setup read image `m12-admin-gate-20260823` and found `TIDAL_ADMIN_KEY` in the secret; behavioural probe returned data→403, admin→200. | The runbook was written against the previously deployed image and not revisited after the roll. Documentation drift, not a product defect. | Rewrote §9: 9.2 marked LIVE with the working probe; §9.1/§9.3 kept as inert with the reason (image predates `4766f56`). | Re-read §9; `09-operator-authority.spec.ts` now asserts the live gate and would fail if it regressed. | fixed |
| BUG-002 | CAP-008, CAP-015 | Medium | A metrics scrape either returns series or fails loudly. | `wget --timeout=6` returned zero lines for `tidaldb-2`, indistinguishable from "this pod exports nothing". | Three retries at `--timeout=10` returned metrics every time; peers returned 332/344 series at 6s. | Client timeout too short for the payload on a loaded node — a measurement artifact, not a product defect. | Harness uses a generous timeout and asserts baseline `tidaldb_` series are present before concluding a specific metric is absent. Runbook §9.1 now specifies `--timeout=15` and explains why. | `04-network-isolation.spec.ts` and `09-operator-authority.spec.ts` both green with the baseline-present guard. | fixed |
| BUG-003 | CAP-011, CAP-014 | Low | The boot log reflects the current credential state. | Boot log WARNs `TIDAL_ADMIN_KEY is not set` while the gate is live and enforcing. | Pod started 05:41; WARN at 05:42; kubelet materialized the projected secret at 05:51:43; poller loaded it. | Working as designed — the credential poller exists so a key can be added without a restart. The log is a point-in-time record, not current state. | Documented in runbook §9.2 as an explicit "do not trust the boot WARN, test the behaviour" note; the harness allowlists this WARN with that reason. | `06-logs.spec.ts` allowlists it with the rationale; `09-operator-authority.spec.ts` proves the behaviour. | fixed |
| BUG-004 | CAP-013 | Medium | The backup check verifies the fleet backup. | Selecting the newest Backup by timestamp picked `restore-canary-longhorn-rwx-*` (20 items, 1 PVB) — it would report "pass" having verified nothing about the fleet. | Ran the runbook's own command as written; it returned the canary. | The selector sorted all Backups instead of filtering to the schedule that the freshness alert actually watches. | Runbook and harness both now filter on `velero.io/schedule-name=velero-fleet-daily`. | Re-ran: selects `velero-fleet-daily-20260823033025`, 3708/3708 items, 48/48 PVBs Completed. | fixed |

156
demo/capture-manifest.json Normal file
View File

@ -0,0 +1,156 @@
{
"schemaVersion": 1,
"buildRevision": "d21a202",
"verifiedImage": "registry.threesix.ai/tidal/server:m12-admin-gate-20260823@sha256:6e220060a342658b734d258245b20f6233d96e26415b3a44956b1c3bceebe48c",
"capturedEnvironment": "orchard9-k3sf / namespace tidaldb-cluster (live)",
"viewport": {
"width": 1600,
"height": 900
},
"captures": [
{
"id": "CAP-002-convergence",
"capabilityId": "CAP-002",
"testId": "workflows/deploy-verification.demo.spec.ts :: demo capture \u2014 deploy verification :: CAP-002 every node is converged",
"file": "captures/CAP-002-convergence.png",
"expected": "Three nodes, zero lag on every shard group, no reseed pending",
"businessPurpose": "Quorum with one-node fault tolerance actually exists, rather than being assumed from pod readiness",
"personas": [
"cluster operator"
],
"width": 1600,
"height": 900,
"contentHash": "sha256:69da2d8601585268be468d049afc4747aa2e6ae4c7c1f46b927e0de7ab5ca1b9",
"audienceVerdict": "perfect",
"auditStatus": "pass"
},
{
"id": "CAP-006-quorum-write",
"capabilityId": "CAP-006",
"testId": "workflows/deploy-verification.demo.spec.ts :: demo capture \u2014 deploy verification :: CAP-005 CAP-006 the boundary refuses, then a quorum write commits",
"file": "captures/CAP-006-quorum-write.png",
"expected": "401, 401, 200, then 201 for a quorum-acked write",
"businessPurpose": "The single strongest available proof: the full stack works and the data plane is closed to strangers",
"personas": [
"cluster operator"
],
"width": 1600,
"height": 900,
"contentHash": "sha256:f1fe76f19643382349006988d5bbddb68a501c1bf3037720c07e98a5a27ddfea",
"audienceVerdict": "perfect",
"auditStatus": "pass"
},
{
"id": "CAP-008-network-isolation",
"capabilityId": "CAP-008",
"testId": "workflows/deploy-verification.demo.spec.ts :: demo capture \u2014 deploy verification :: CAP-008 the metrics port is closed to foreign pods but open to the scraper",
"file": "captures/CAP-008-network-isolation.png",
"expected": "Connection refused from a foreign namespace; hundreds of series to the scraper",
"businessPurpose": "Least-privilege network access without blinding the monitoring stack",
"personas": [
"cluster operator"
],
"width": 1600,
"height": 900,
"contentHash": "sha256:247fce3324c519319176a20095a1c7a7b2b5eb5ad6745ff9046e8b6886271c6a",
"audienceVerdict": "perfect",
"auditStatus": "pass"
},
{
"id": "CAP-010-dashboard",
"capabilityId": "CAP-010",
"testId": "workflows/deploy-verification.demo.spec.ts :: demo capture \u2014 deploy verification :: CAP-010 the operator dashboard renders live data",
"file": "captures/CAP-010-dashboard.png",
"expected": "Cluster health OK, reseed none, corpus size, and per-node latency charts \u2014 legible at delivery resolution",
"businessPurpose": "The first surface an operator opens during an incident actually shows the cluster",
"personas": [
"cluster operator"
],
"width": 1600,
"height": 502,
"contentHash": "sha256:296570addb43426d4fbb1f8bb69b8a1fac9c691d9cdccf2e927a5db81e4407fe",
"audienceVerdict": "perfect",
"auditStatus": "pass"
},
{
"id": "CAP-014-authority",
"capabilityId": "CAP-014",
"testId": "workflows/deploy-verification.demo.spec.ts :: demo capture \u2014 deploy verification :: CAP-014 operator authority is separate from data access",
"file": "captures/CAP-014-authority.png",
"expected": "403 for the data credential, not-403 for the admin credential",
"businessPurpose": "Blast radius of a leaked application key is bounded to data, not cluster topology",
"personas": [
"cluster operator"
],
"width": 1600,
"height": 900,
"contentHash": "sha256:9897670f83c5c408b57a9c88c0097d3fcc43f983ad463be7bfbcabc49e3522b1",
"audienceVerdict": "perfect",
"auditStatus": "pass"
},
{
"id": "CAP-014-drift",
"capabilityId": "CAP-014",
"testId": "workflows/deploy-verification.demo.spec.ts :: demo capture \u2014 deploy verification :: CAP-014 CAP-015 the harness corrected its own runbook",
"file": "captures/CAP-014-drift.png",
"expected": "The superseded claim beside the live probe that contradicts it",
"businessPurpose": "Verification that audits its own documentation instead of drifting away from it",
"personas": [
"cluster operator"
],
"width": 1600,
"height": 900,
"contentHash": "sha256:7901e70f8df834a91f1399d1ad8fcf4398546bef14fa46ad23ec605e9d12a344",
"audienceVerdict": "perfect",
"auditStatus": "pass"
},
{
"id": "CAP-013-backup",
"capabilityId": "CAP-013",
"testId": "workflows/deploy-verification.demo.spec.ts :: demo capture \u2014 deploy verification :: CAP-013 the fleet backup captured every volume",
"file": "captures/CAP-013-backup.png",
"expected": "Completed, all items, every PodVolumeBackup Completed",
"businessPurpose": "The cluster can actually be restored, and the alert is trustworthy",
"personas": [
"cluster operator"
],
"width": 1600,
"height": 900,
"contentHash": "sha256:e570917de51ac352a18949117dbf6c9e990fc75127b1bc6c71d0473e2f21371f",
"audienceVerdict": "perfect",
"auditStatus": "pass"
},
{
"id": "CAP-015-inert",
"capabilityId": "CAP-015",
"testId": "workflows/deploy-verification.demo.spec.ts :: demo capture \u2014 deploy verification :: CAP-015 what is NOT verified is stated",
"file": "captures/CAP-015-inert.png",
"expected": "Zero tidaldb_http_* families while baseline tidaldb_* families are present",
"businessPurpose": "A verification that hides its gaps cannot be trusted about the parts it claims",
"personas": [
"cluster operator"
],
"width": 1600,
"height": 900,
"contentHash": "sha256:671ce44f8e3a629f88cab8f807c9389e068660f51a00e68232c9f92c87b55a76",
"audienceVerdict": "perfect",
"auditStatus": "pass"
},
{
"id": "CAP-012-tidalctl",
"capabilityId": "CAP-012",
"testId": "workflows/deploy-verification.demo.spec.ts :: demo capture \u2014 deploy verification :: CAP-012 tidalctl gives an operator a live view and an exit code",
"file": "captures/CAP-012-tidalctl.png",
"expected": "Leader, region table with NO REPORT markers, shard table, exit 2",
"businessPurpose": "An operator can interrogate the cluster without hand-rolling curl, and is told what the tool cannot see",
"personas": [
"cluster operator"
],
"width": 1600,
"height": 900,
"contentHash": "sha256:e13e0bfecc766d02f839334487cf0364472c43bf0f1fbe1a11159e6312a58aa6",
"audienceVerdict": "perfect",
"auditStatus": "pass"
}
]
}

98
demo/preflight.ts Normal file
View File

@ -0,0 +1,98 @@
/**
* Render gate. Refuses to let a stale, missing, or unreviewed asset reach the
* composition.
*
* Run before `demo:render`. Remotion also rejects a bad capture at render time
* via requireCapture(), but failing here gives a readable list instead of one
* React error, and it catches drift the composition cannot see — a file whose
* bytes changed after it was approved.
*/
import { createHash } from 'node:crypto';
import { readFile, readdir } from 'node:fs/promises';
import { join } from 'node:path';
type Capture = {
id: string;
file: string;
width: number;
height: number;
contentHash: string;
audienceVerdict: string;
auditStatus: string;
};
const CAPTURE_ROOT = 'demo/public';
async function main(): Promise<void> {
const manifest = JSON.parse(await readFile('demo/capture-manifest.json', 'utf8')) as {
buildRevision: string;
captures: Capture[];
};
const scenes = await readFile('demo/src/scenes.ts', 'utf8');
const problems: string[] = [];
for (const capture of manifest.captures) {
if (capture.auditStatus !== 'pass') {
problems.push(`${capture.id}: auditStatus is '${capture.auditStatus}', expected 'pass'`);
}
if (capture.audienceVerdict !== 'perfect') {
problems.push(
`${capture.id}: audienceVerdict is '${capture.audienceVerdict}'. Only 'perfect' promotes — ` +
`fix the screen at its owning layer or cut the beat honestly.`,
);
}
let bytes: Buffer;
try {
bytes = await readFile(join(CAPTURE_ROOT, capture.file));
} catch {
problems.push(`${capture.id}: file missing at ${join(CAPTURE_ROOT, capture.file)}`);
continue;
}
const actual = `sha256:${createHash('sha256').update(bytes).digest('hex')}`;
if (actual !== capture.contentHash) {
problems.push(
`${capture.id}: content hash changed since approval — the file was re-captured without ` +
`being re-audited. Re-inspect it, then update the manifest.`,
);
}
}
// Every promoted capture should be used, and every referenced capture promoted.
const referenced = [...scenes.matchAll(/captureId: '([^']+)'/g)].map((m) => m[1]);
const promoted = manifest.captures.map((c) => c.id);
for (const id of referenced) {
if (!promoted.includes(id)) {
problems.push(`scene references '${id}', which is not a promoted capture`);
}
}
const orphans = promoted.filter((id) => !referenced.includes(id));
// Nothing may sit in the promoted directory without a manifest row.
const onDisk = (await readdir(join(CAPTURE_ROOT, 'captures'))).filter((f) => f.endsWith('.png'));
for (const file of onDisk) {
if (!promoted.includes(file.replace(/\.png$/, ''))) {
problems.push(`${file} is in demo/public/captures but has no manifest row`);
}
}
process.stdout.write(
`preflight: ${manifest.captures.length} captures, revision ${manifest.buildRevision}\n`,
);
if (orphans.length > 0) {
// Not fatal: a capture can be a standalone artefact. Reported so it is a
// decision rather than an accident.
process.stdout.write(` note: promoted but unused by any scene: ${orphans.join(', ')}\n`);
}
if (problems.length > 0) {
process.stderr.write(`\npreflight FAILED:\n${problems.map((p) => ` ✗ ${p}`).join('\n')}\n`);
process.exit(1);
}
process.stdout.write(' all captures present, hash-stable, and audited perfect\n');
}
await main();

BIN
demo/public/captures/CAP-002-convergence.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 107 KiB

BIN
demo/public/captures/CAP-006-quorum-write.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 94 KiB

BIN
demo/public/captures/CAP-008-network-isolation.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 73 KiB

BIN
demo/public/captures/CAP-010-dashboard.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 56 KiB

BIN
demo/public/captures/CAP-012-tidalctl.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 105 KiB

BIN
demo/public/captures/CAP-013-backup.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 81 KiB

BIN
demo/public/captures/CAP-014-authority.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 63 KiB

BIN
demo/public/captures/CAP-014-drift.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 118 KiB

BIN
demo/public/captures/CAP-015-inert.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 82 KiB

15
demo/src/Root.tsx Normal file
View File

@ -0,0 +1,15 @@
import React from 'react';
import { Composition } from 'remotion';
import { Walkthrough } from './Walkthrough';
import { FPS, HEIGHT, WIDTH, durationInFrames } from './scenes';
export const RemotionRoot: React.FC = () => (
<Composition
id="DeployVerification"
component={Walkthrough}
durationInFrames={durationInFrames}
fps={FPS}
width={WIDTH}
height={HEIGHT}
/>
);

282
demo/src/Walkthrough.tsx Normal file
View File

@ -0,0 +1,282 @@
import React from 'react';
import {
AbsoluteFill,
Img,
Sequence,
interpolate,
staticFile,
useCurrentFrame,
} from 'remotion';
import manifest from '../capture-manifest.json';
import { FPS, HEIGHT, WIDTH, timeline, type Scene } from './scenes';
/**
* Presentation only. Every pixel of evidence in here is a promoted Playwright
* capture; this file controls pacing, framing, and captions and nothing else.
* It must never composite, crop, or annotate in a way that changes what the
* underlying screen said.
*/
const COLORS = {
bg: '#0a0d13',
panel: '#121821',
line: '#243044',
text: '#e3e9f2',
dim: '#8797ad',
accent: '#4da3ff',
good: '#56d364',
dream: '#d29922',
} as const;
const SANS =
'ui-sans-serif, -apple-system, "Segoe UI", system-ui, "Helvetica Neue", sans-serif';
const MONO = 'ui-monospace, SFMono-Regular, "SF Mono", Menlo, monospace';
type CaptureEntry = {
id: string;
file: string;
width: number;
height: number;
audienceVerdict: string;
auditStatus: string;
};
const captures: Record<string, CaptureEntry> = Object.fromEntries(
(manifest.captures as CaptureEntry[]).map((c) => [c.id, c]),
);
/** Reject at render time rather than shipping a scene with no evidence. */
function requireCapture(captureId: string): CaptureEntry {
const entry = captures[captureId];
if (!entry) throw new Error(`capture ${captureId} is not in the manifest`);
if (entry.auditStatus !== 'pass' || entry.audienceVerdict !== 'perfect') {
throw new Error(
`capture ${captureId} is ${entry.auditStatus}/${entry.audienceVerdict}; only a ` +
`passing, perfect capture may be rendered`,
);
}
return entry;
}
/**
* Frames of cross-dissolve between scenes — 10 at 30fps is 333ms, inside the
* 200-400ms micro-transition range.
*/
export const OVERLAP = 10;
/**
* Fade IN only, never out.
*
* The first cut of this composition faded each scene out over its last 8 frames
* while the next faded in from its own frame 0. Because Remotion Sequences do
* not overlap by default, both sat at opacity 0 on the boundary frame and the
* video blinked to bare background between every scene. Scenes now start
* OVERLAP frames early and rise on top of the outgoing one, which is a real
* cross-dissolve with no empty frame.
*/
function useEntrance() {
const frame = useCurrentFrame();
const opacity = interpolate(frame, [0, OVERLAP], [0, 1], {
extrapolateRight: 'clamp',
});
const lift = interpolate(frame, [0, OVERLAP + 4], [12, 0], {
extrapolateRight: 'clamp',
});
return { opacity, lift };
}
const RungBadge: React.FC<{ rung: Scene['rung'] }> = ({ rung }) => (
<span
style={{
fontFamily: MONO,
fontSize: 20,
letterSpacing: 1.6,
textTransform: 'uppercase',
color: rung === 'dream' ? COLORS.dream : COLORS.dim,
border: `1px solid ${rung === 'dream' ? COLORS.dream : COLORS.line}`,
borderRadius: 999,
padding: '6px 16px',
}}
>
{rung}
</span>
);
const TitleScene: React.FC<{ scene: Extract<Scene, { kind: 'title' | 'recap' }> }> = ({
scene,
}) => {
const { opacity, lift } = useEntrance();
return (
<AbsoluteFill
style={{
backgroundColor: COLORS.bg,
justifyContent: 'center',
padding: '0 150px',
opacity,
transform: `translateY(${lift}px)`,
}}
>
<div
style={{
width: 96,
height: 5,
backgroundColor: scene.kind === 'recap' ? COLORS.good : COLORS.accent,
marginBottom: 44,
}}
/>
<h1
style={{
margin: 0,
fontFamily: SANS,
fontSize: 78,
lineHeight: 1.08,
fontWeight: 680,
letterSpacing: -1.6,
color: COLORS.text,
}}
>
{scene.heading}
</h1>
<div style={{ marginTop: 40, display: 'flex', flexDirection: 'column', gap: 20 }}>
{scene.lines.map((line) => (
<p
key={line}
style={{
margin: 0,
fontFamily: SANS,
fontSize: 34,
lineHeight: 1.42,
color: COLORS.dim,
maxWidth: 1500,
}}
>
{line}
</p>
))}
</div>
{scene.footer ? (
<p
style={{
marginTop: 56,
fontFamily: MONO,
fontSize: 24,
color: COLORS.dim,
opacity: 0.82,
}}
>
{scene.footer}
</p>
) : null}
</AbsoluteFill>
);
};
const ProofScene: React.FC<{ scene: Extract<Scene, { kind: 'proof' }> }> = ({ scene }) => {
const { opacity, lift } = useEntrance();
const capture = requireCapture(scene.captureId);
// Reserve fixed bands for the heading and caption, then fit the capture into
// whatever is left, preserving its aspect ratio. Tall captures (the 1600x1800
// dashboard) letterbox instead of being squashed or cropped.
const headerBand = 76;
const captionBand = 104;
const availableWidth = WIDTH - 160;
const availableHeight = HEIGHT - headerBand - captionBand;
const scale = Math.min(availableWidth / capture.width, availableHeight / capture.height);
const renderWidth = Math.round(capture.width * scale);
const renderHeight = Math.round(capture.height * scale);
return (
<AbsoluteFill style={{ backgroundColor: COLORS.bg, opacity }}>
<div
style={{
height: headerBand,
padding: '30px 80px 0',
display: 'flex',
alignItems: 'center',
justifyContent: 'flex-end',
gap: 18,
transform: `translateY(${lift}px)`,
}}
>
<span style={{ fontFamily: MONO, fontSize: 22, color: COLORS.dim }}>
{scene.capabilityId}
</span>
<RungBadge rung={scene.rung} />
</div>
<div
style={{
flex: 1,
display: 'flex',
alignItems: 'center',
justifyContent: 'center',
}}
>
<div
style={{
width: renderWidth,
height: renderHeight,
borderRadius: 10,
overflow: 'hidden',
border: `1px solid ${COLORS.line}`,
boxShadow: '0 24px 70px rgba(0,0,0,0.55)',
}}
>
<Img
src={staticFile(capture.file)}
style={{ width: '100%', height: '100%', display: 'block' }}
/>
</div>
</div>
<div
style={{
height: captionBand,
padding: '0 80px 30px',
display: 'flex',
alignItems: 'flex-end',
}}
>
<p
style={{
margin: 0,
fontFamily: SANS,
fontSize: 27,
lineHeight: 1.42,
color: scene.rung === 'dream' ? COLORS.text : COLORS.dim,
maxWidth: 1660,
borderLeft: `3px solid ${scene.rung === 'dream' ? COLORS.dream : COLORS.line}`,
paddingLeft: 22,
}}
>
{scene.caption}
</p>
</div>
</AbsoluteFill>
);
};
export const Walkthrough: React.FC = () => (
<AbsoluteFill style={{ backgroundColor: COLORS.bg }}>
{timeline.map(({ scene, from, frames }) => (
// Start early and run long by OVERLAP so the incoming scene dissolves
// over the outgoing one. End times are unchanged, so total duration and
// every storyboard hold stay exactly as specified.
<Sequence
key={scene.id}
from={Math.max(0, from - (from > 0 ? OVERLAP : 0))}
durationInFrames={frames + (from > 0 ? OVERLAP : 0)}
name={scene.id}
>
{scene.kind === 'proof' ? (
<ProofScene scene={scene} />
) : (
<TitleScene scene={scene} />
)}
</Sequence>
))}
</AbsoluteFill>
);
export const config = { FPS, WIDTH, HEIGHT };

4
demo/src/index.ts Normal file
View File

@ -0,0 +1,4 @@
import { registerRoot } from 'remotion';
import { RemotionRoot } from './Root';
registerRoot(RemotionRoot);

184
demo/src/scenes.ts Normal file
View File

@ -0,0 +1,184 @@
/**
* Scene metadata — the single source of truth for order, duration, and which
* promoted capture each beat presents. Mirrors demo/storyboard.md; a beat here
* with no storyboard row, or a capture not marked `perfect` in the manifest, is
* a defect the preflight rejects.
*/
export const FPS = 30;
export const WIDTH = 1920;
export const HEIGHT = 1080;
export type AudienceRung = 'need' | 'want' | 'dream';
export type Scene =
| {
kind: 'title';
id: string;
seconds: number;
rung: AudienceRung;
heading: string;
lines: string[];
footer?: string;
}
| {
kind: 'proof';
id: string;
seconds: number;
rung: AudienceRung;
capabilityId: string;
captureId: string;
heading: string;
caption: string;
}
| {
kind: 'recap';
id: string;
seconds: number;
rung: AudienceRung;
heading: string;
lines: string[];
footer?: string;
};
export const scenes: Scene[] = [
{
kind: 'title',
id: 'B1-opening',
seconds: 9,
rung: 'need',
heading: 'tidalDB deploy verification',
lines: [
'Three voters on orchard9-k3sf. One public endpoint. A runbook an operator can walk.',
'Every number that follows came from a command that ran against the live cluster.',
],
footer: 'namespace tidaldb-cluster · image m12-admin-gate-20260823 · 32 checks green',
},
{
kind: 'proof',
id: 'B2-convergence',
seconds: 7,
rung: 'need',
capabilityId: 'CAP-002',
captureId: 'CAP-002-convergence',
heading: 'Every node agrees, and none is behind',
caption:
'Each node asked for its own view. A pod can be Ready while its replication is stalled — that is what the reseed livelock exploited.',
},
{
kind: 'proof',
id: 'B3-boundary-write',
seconds: 8,
rung: 'need',
capabilityId: 'CAP-006',
captureId: 'CAP-006-quorum-write',
heading: 'The boundary holds, and the write commits',
caption:
'Same path, three credentials. Then a write a majority of nodes acknowledged — the one claim a health endpoint cannot fake.',
},
{
kind: 'proof',
id: 'B4-isolation',
seconds: 7,
rung: 'need',
capabilityId: 'CAP-008',
captureId: 'CAP-008-network-isolation',
heading: 'The refusal is the point',
caption:
'A NetworkPolicy that blocks everything is an observability outage. One that blocks nothing is theatre. Both directions proven.',
},
{
kind: 'proof',
id: 'B5-authority',
seconds: 6,
rung: 'want',
capabilityId: 'CAP-014',
captureId: 'CAP-014-authority',
heading: 'An application key cannot remove a member',
caption:
'403, not 401 — the credential is valid, it simply has no operator authority.',
},
{
kind: 'proof',
id: 'B6-dashboard',
seconds: 8,
rung: 'want',
capabilityId: 'CAP-010',
captureId: 'CAP-010-dashboard',
heading: 'The board an operator opens at 3am',
caption:
'Health, reseed state, corpus size, and per-node latency — scoped to the cluster, not to whatever else shares the metric name.',
},
{
kind: 'proof',
id: 'B7-recovery',
seconds: 6,
rung: 'want',
capabilityId: 'CAP-013',
captureId: 'CAP-013-backup',
heading: 'The recovery story is intact',
caption:
'Selected by the schedule label the freshness alert actually watches — not merely the newest backup object.',
},
{
kind: 'proof',
id: 'B8-blind-spot',
seconds: 7,
rung: 'want',
capabilityId: 'CAP-012',
captureId: 'CAP-012-tidalctl',
heading: 'It refuses to invent a number',
caption:
'Two healthy peers would otherwise read as 13.3 million events behind. NO REPORT is an honest "I do not know".',
},
{
kind: 'proof',
id: 'B9a-inert',
seconds: 7,
rung: 'need',
capabilityId: 'CAP-015',
captureId: 'CAP-015-inert',
heading: 'What this deployment does not yet do',
caption:
'Two committed features are absent from the running image. The suite asserts that absence, so the day it changes it fails and says so.',
},
{
kind: 'proof',
id: 'B9-drift-catch',
seconds: 10,
rung: 'dream',
capabilityId: 'CAP-014',
captureId: 'CAP-014-drift',
heading: 'The harness corrected its own runbook',
caption:
'The runbook said the credential split was "not active yet — requires an image roll". On its first run the harness read the live image and the live secret, and proved the gate was already enforcing. The document was wrong; the harness said so before anyone noticed.',
},
{
kind: 'recap',
id: 'B10-recap',
seconds: 7,
rung: 'need',
heading: 'Verified, including the gaps',
lines: [
'32 checks green against the live deployment.',
'Two committed features are absent from the running image — and the suite asserts that absence deliberately.',
'The day it changes, the suite fails and says so.',
],
footer: 'docs/runbooks/deploy-verification.md · npm run test:e2e',
},
];
export const durationInFrames = scenes.reduce(
(total, scene) => total + Math.round(scene.seconds * FPS),
0,
);
/** Frame offset of each scene, derived once so nothing drifts. */
export const timeline = scenes.reduce<{ scene: Scene; from: number; frames: number }[]>(
(acc, scene) => {
const from = acc.length === 0 ? 0 : acc[acc.length - 1].from + acc[acc.length - 1].frames;
acc.push({ scene, from, frames: Math.round(scene.seconds * FPS) });
return acc;
},
[],
);

54
demo/storyboard.md Normal file
View File

@ -0,0 +1,54 @@
# Storyboard — tidalDB deploy verification
Ordered by the audience ladder in `audience-brief.md`, not by runbook section
order. Needs first to buy credibility, wants in the middle, the single dream beat
where attention peaks, then a recap that restates the needs as met.
Every functional beat cites a capability, a green test, a promoted capture, and
the rung it serves. A beat serving no rung is cut.
30 fps. Durations are seconds.
| Beat | Capability | Rung | Persona | Business purpose | Proof state | Test | Capture | Caption | Secs |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | ---: |
| B1 opening | — | need | operator | Orientation: what this is and what it will prove | Title card | — | — | "tidalDB deploy verification. Three voters on `orchard9-k3sf`, one public endpoint, and a runbook an operator can walk. Every number that follows came from a command that ran against the live cluster." | 9 |
| B2 convergence | CAP-002 | need | operator | Quorum with fault tolerance actually exists, rather than being inferred from pod readiness | Per-node `lag=0`, no reseed, agreed leader per shard group | `01-cluster-convergence.spec.ts :: every node reports zero lag…` | `CAP-002-convergence` | "Each node asked for its own view. A pod can be Ready while its replication is stalled — that is what the reseed livelock exploited." | 7 |
| B3 boundary + write | CAP-005 CAP-006 | need | operator | The strongest single proof: full stack works AND the data plane is closed | 401, 401, 200, then 201 quorum-acked | `03-auth-boundary.spec.ts :: a quorum-acked write is committed…` | `CAP-006-quorum-write` | "Same path, three credentials. Then a write a majority of nodes acknowledged — the one claim a health endpoint cannot fake." | 8 |
| B4 isolation | CAP-008 | need | operator | Least-privilege network access without blinding monitoring | Refused from a foreign namespace; 332 series to the scraper | `04-network-isolation.spec.ts :: a pod in an unrelated namespace is refused…` | `CAP-008-network-isolation` | "The refusal is the point. A policy that blocks everything is an outage; one that blocks nothing is theatre." | 7 |
| B5 authority | CAP-014 | want | operator | Blast radius of a leaked application key stops at data, not cluster topology | Data bearer 403, admin bearer authorised | `09-operator-authority.spec.ts :: the data-plane credential is refused…` | `CAP-014-authority` | "403, not 401 — the key is valid, it just cannot remove a cluster member." | 6 |
| B6 dashboard | CAP-010 | want | operator | The first surface opened during an incident actually shows the cluster | Cluster health OK, reseed none, 33.3K vectors, populated latency charts | `05-metrics-dashboard.spec.ts :: an operator opening the dashboard sees populated charts` | `CAP-010-dashboard` | "The board an operator opens at 3am. Health, reseed state, corpus size, and per-node latency — scoped to the cluster." | 8 |
| B7 recovery | CAP-013 | want | operator | The cluster can actually be restored and the alert is trustworthy | Completed, 3708/3708 items, 48/48 volumes | `08-backups.spec.ts :: the newest fleet-schedule backup completed…` | `CAP-013-backup` | "Selected by the schedule label the freshness alert watches — not merely the newest backup object." | 6 |
| B8 blind spot | CAP-012 | want | operator | A tool that reports what it cannot see is trustworthy about what it can | `NO REPORT` markers instead of fabricated lag; exit 2 | `07-tidalctl.spec.ts :: the aggregated-status gap is reported as NO REPORT…` | `CAP-012-tidalctl` | "It refuses to invent a number. Two healthy peers would otherwise read as 13.3 million events behind." | 7 |
| B9a inert | CAP-015 | need | operator | A verification that hides its gaps cannot be trusted about what it claims | Running image plus zero `tidaldb_http_*` families against a working baseline scrape | `09-operator-authority.spec.ts :: HTTP request metrics are absent…` | `CAP-015-inert` | "Two committed features are absent from the running image. The suite asserts that absence, so the day it changes it fails and says so." | 7 |
| **B9 the drift catch** | CAP-014 | **dream** | operator | Verification that audits its own documentation instead of drifting from it | The committed doc text, the actually-running image, and the live 403/200 probe, side by side | `deploy-verification.demo.spec.ts :: the harness corrected its own runbook` | `CAP-014-drift` | "The runbook said the credential split was 'not active yet — requires an image roll'. On its first run the harness read the live image and the live secret, and proved the gate was already enforcing. The document was wrong. The harness said so, before anyone noticed." | 10 |
| B10 recap | — | need | operator | Restate the needs as met, including the gaps | Closing card | — | — | "32 checks green against the live deployment. Two committed features are absent from the running image, and the suite asserts that absence deliberately — so the day it changes, it says so." | 7 |
**Total: 82 s** across 11 beats.
## Ladder coverage
Two-way mapping, per the audience protocol.
| Brief entry | Served by |
| --- | --- |
| Need — evidence is real, commands visible | B1, B2, B3, B4 (every capture shows the command that produced it) |
| Need — write proven committed by quorum | B3 |
| Need — every boundary shows its denial | B3 (401×2), B4 (refused), B5 (403) |
| Need — what is not verified is stated | B9a (evidence on screen), B10 (recap) |
| Want — one command a different person can run | B1, B10 |
| Want — operator/data authority visibly enforced | B5 |
| Dream — the harness catches a stale claim in the runbook it verifies | B9 (exactly one dream beat) |
No `need` rung is unserved, and no `need`-serving beat was cut in response to an
audience-bar failure — the five weak-design failures found in the first capture
pass were fixed at the capture layer and re-audited.
## Deliberately not beats
- **A "green wall" summary of all 32 tests.** Counting tests is not evidence, and
this viewer has seen a livelocked cluster report itself healthy.
- **The three red `cluster_reseed` tests.** They belong in the written record
where the bisect evidence can be read, not in a 75-second walkthrough where a
red frame would read as a deploy failure.
- **Clicking, scrolling, or navigating.** The dashboard beat shows the board, not
the act of opening it.

15
demo/tsconfig.json Normal file
View File

@ -0,0 +1,15 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "ESNext",
"moduleResolution": "bundler",
"jsx": "react-jsx",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"resolveJsonModule": true,
"noEmit": true,
"types": ["node"]
},
"include": ["src/**/*.ts", "src/**/*.tsx", "capture-manifest.json"]
}

137
demo/visual-audit.md Normal file
View File

@ -0,0 +1,137 @@
# Visual audit — tidalDB deploy verification walkthrough
Audited against `audience-brief.md`. Per-screen verdicts are judged at the
**quality-bar judge's** standard (Kyle Kingsbury); the walk-the-render ledger is
written in the **actual decision-maker's** voice (Jordan Washburn).
- Build revision: see `capture-manifest.json.buildRevision`
- Verified image: `registry.threesix.ai/tidal/server:m12-admin-gate-20260823@sha256:6e220060…`
- Render: `demo/out/deploy-verification.mp4` — 82.05 s, 1920×1080, 30 fps, h264, 2460 frames
- Regression suite at time of capture: **32 passed**
- Demo capture suite: **9 passed** (each asserts before it photographs)
## 1. Programmatic preflight
| Check | Result |
| --- | --- |
| Every manifest file exists and decodes | pass — 9/9 |
| Filesystem inventory equals ledger count | pass — 9 promoted, 9 rows, 0 unclassified |
| Dimensions match declared viewport or documented crop | pass — 8 × 1600×900, 1 × 1600×502 (documented crop) |
| Files non-empty, not near-uniform blanks | pass — 64 KB–187 KB |
| Capture IDs and filenames unique | pass |
| No unexpected duplicates across distinct proof states | pass |
| Source test green | pass — 9 demo capture tests green; each asserts before it photographs |
| Secrets / tokens / local paths absent | pass — `redact()` masks every known secret before render; spot-checked all 9 |
## 2. Per-image review
Every image was opened individually at full resolution. Verdicts after the fix
round; the pre-fix verdict is shown where it differed, because a `slop` finding
that was silently overwritten is not an audit.
| Capture | State | Rendering | Layout | Content | Privacy | Errors | Audience verdict | Bugs |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `CAP-002-convergence` | correct | complete | ok (was: 230 px dead space) | real 13.3M positions, real pod names | clean | none | **perfect** (was `slop` weak-design) | BUG-011 |
| `CAP-006-quorum-write` | correct | complete | ok | 401/401/200/201, real hostname | clean | none | **perfect** | — |
| `CAP-008-network-isolation` | correct | complete | ok (was: 350 px dead space) | `Connection refused` + 332 series | clean | none | **perfect** (was `slop` weak-design) | BUG-011 |
| `CAP-010-dashboard` | correct | complete | ok (was: illegible when scaled) | OK / none / 33.3 K, per-pod legends | clean | no "No data" in crop | **perfect** (was `slop` weak-design) | BUG-010 |
| `CAP-012-tidalctl` | correct | complete | ok (was: dead space) | `NO REPORT` markers, exit 2 | clean | none | **perfect** (was `slop` weak-design + colour) | BUG-011, BUG-012 |
| `CAP-013-backup` | correct | complete | ok (was: dead space) | 3708/3708, 48 PVBs | clean | none | **perfect** (was `slop` weak-design) | BUG-011 |
| `CAP-014-authority` | correct | complete | ok (was: dead space) | 403 then 200 | clean | none | **perfect** (was `slop` weak-design) | BUG-011 |
| `CAP-014-drift` | correct | complete | ok | git text + live image + 403/200 | clean | none | **perfect** | BUG-013 |
| `CAP-015-inert` | correct | complete | ok (was: dead space) | 332 baseline / 0 http | clean | none | **perfect** (was `slop` weak-design) | BUG-011 |
### Non-promoted images
None. Nine images were produced by the capture run and nine were promoted; no
failure or debug images were generated because every capture test passed on the
run that produced the promoted set.
## 3. Remotion frame audit
Scene proof frames rendered and inspected at `demo/out/stills/`; transition
frames extracted from the **encoded MP4** (not re-renders) at
`demo/out/frames/`.
| Frame | Check | Result |
| --- | --- | --- |
| f120 opening | purpose, personas, scope readable | pass |
| f375 B2, f600 B3, f825 B4, f1020 B5 | proof legible, entrance settled | pass |
| f1230 B6 dashboard | legible at delivery resolution | pass after BUG-010 |
| f1440 B7, f1635 B8 | proof legible | pass |
| f1890 B9 dream | image agrees with caption | pass after BUG-013 |
| f2145 B10 recap | copy matches what was shown | pass |
| All 10 scene boundaries | no black/white/empty frame | pass after BUG-014 — 0 empty frames |
| Mean luma at all 10 boundaries | 17.49–23.21, smooth progression | pass (was flat 13.00 = bare background) |
| Contact sheet, 30 samples across 75 s | order matches storyboard, no repeats or stale content | pass |
Captures are presented at scale ≥ 1.0 — the 1600×900 terminal panels render
1:1 and the 1600×502 dashboard crop renders at 1.10 — so no evidence is
downscaled.
## 4. Walk the render as Jordan Washburn
One row per distinct screen a viewer reads.
| Screen | t | Viewer thought (verbatim, in their voice) | Verdict |
| --- | ---: | --- | --- |
| B1 opening | 0–9 s | "Right, it names the cluster and the image up front. And it says every number came from a real command — that's the claim I actually care about." | neutral-orientation |
| B2 convergence | 9–16 s | "Per-node, not the aggregate. Good — the aggregate is exactly what lied to me last week. lag=0 on all nine group-replicas and they agree on the leaders." | earns-interest |
| B3 boundary + write | 16–24 s | "401, 401, 200, then a 201 quorum ack. That single 201 is worth more than the rest of the page — it means DNS, TLS, the gateway, auth and Raft all worked in one request." | earns-interest |
| B4 isolation | 24–31 s | "Connection refused from gitea, 332 series to the scraper. That's the pair I'd want — it proves the policy is real without blinding monitoring." | earns-interest |
| B5 authority | 31–37 s | "403 not 401. So a leaked app key can't remove a member. That's the exposure I was worried about and it's closed." | earns-interest |
| B6 dashboard | 37–45 s | "That's the board I'd actually open. Health OK, reseed none, 33.3K vectors, and the legends finally say which node is which." | earns-interest |
| B7 recovery | 45–51 s | "3708/3708 and 48 of 48 volumes, selected by the schedule label. Good — picking the newest backup would have grabbed a canary." | earns-interest |
| B8 blind spot | 51–58 s | "It says NO REPORT instead of inventing 13 million events of lag. And it admits the exit code makes the deploy gate unusable. I trust a tool that tells me that." | earns-interest |
| B9a inert | 58–65 s | "It shows me what it cannot check yet, with the scrape count proving the scrape actually ran. That is the opposite of a green wall." | earns-interest |
| B9 dream | 65–75 s | "Wait — the doc I wrote said that was pending an image roll, and the harness proved it was already live. It caught my own documentation being wrong before I did. I want this running after every deploy." | earns-interest |
| B10 recap | 75–82 s | "32 green, and it names what it does not verify. That's the version I'd hand to someone else." | neutral-orientation |
No `fails` rows. The dream beat's thought shows genuine surprise and desire, so
it earns its rung.
## 5. Full render watch
Watched via the 30-sample contact sheet plus every boundary and proof frame; the
render is silent (Remotion emits a silent audio track), so the muted and
un-muted passes are identical and captions carry the whole narrative.
| Check | Result |
| --- | --- |
| Opening purpose and personas readable | pass — 9 s hold |
| Each proof state legible long enough | pass — 6–10 s, dense screens get the longer holds |
| Dense screens get more time | pass — dream 10 s, write 8 s, dashboard 8 s vs authority 6 s |
| Captions agree with the visible outcome | pass after BUG-013 |
| Transitions smooth, sections clear | pass after BUG-014 |
| Story explains value, not a screen list | pass — ladder order, not runbook order |
| No scene repeats, skips, contradicts, or overstates | pass |
| Audio artefacts | n/a — silent by design |
## Bug log (audit-originated)
Product/test defects found during verification are in
`capability-inventory.md`; these are the ones this visual audit produced.
| Bug | Asset | Severity | Expected | Actual | Root cause | Fix | Verification | Status |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BUG-010 | `CAP-010-dashboard`, scene B6 | high | Dashboard legible at delivery resolution | A 1600×1800 board scaled into 16:9 became unreadable — panel titles and numbers illegible | Capture was the full board; a near-square tall image cannot fit a 16:9 frame without heavy downscaling | Clip to the evidence band (latency + cluster correctness) using real element bounds, so the crop cannot drift with layout. Anchored to the row header and the last stat panel after a first attempt sliced the heading and bled a sliver of the next row | Re-captured 1600×502; renders at scale 1.10; re-inspected the capture and frame f1230 | verified |
| BUG-011 | 6 terminal captures | medium | Frames read as composed | 230–350 px of dead space below the content on sparse panels | `margin-top:auto` on the footer pushed the void into one block instead of distributing it | `.blocks { flex: 1; justify-content: center }` | Re-captured and individually re-inspected all six | verified |
| BUG-012 | `CAP-012-tidalctl` | medium | Colour does not imply a meaning the content lacks | "Exit code 2" rendered green, i.e. as a success, when it is the finding | Verdict line defaulted to the positive colour role | Marked the block `negative: true` (amber) and rewrote the text to name the consequence | Re-captured and re-inspected | verified |
| BUG-013 | scene B9 (dream) | **blocker** | Scene title, claim, and image agree | The dream caption described the runbook's stale claim and the probe that disproved it, while the image showed the unrelated inert-features panel | Beat reused an existing capture instead of one built for the claim | New `CAP-014-drift` capture showing the committed doc text from `git show`, the actually-running image, and the live 403/200 probe side by side | Re-captured, inspected, frame f1890 re-audited, walk-the-render row rewritten | verified |
| BUG-014 | every scene boundary | **blocker** | No empty frame between scenes | The video blinked to bare background for one frame at all 9 boundaries | Remotion Sequences do not overlap; each scene faded out over its last 8 frames while the next faded in from its own frame 0, so both sat at opacity 0 on the boundary | Removed the fade-out and started each Sequence `OVERLAP=10` frames early running long, producing a true 333 ms cross-dissolve. End times unchanged, so every storyboard hold is preserved | Re-rendered; boundary luma went from flat 13.00 (bare background) to 17.5–23.2 with smooth progression; boundary frames re-inspected | verified |
| BUG-015 | proof scenes | low | No duplicated headings | Remotion drew a scene heading above a capture that already carried its own title, producing a card-in-card with two competing titles | Composition and capture both owned a heading | Removed the heading from the proof scene; Remotion now owns only the capability badge, rung, and audience caption. Reclaimed 66 px for the evidence | Re-rendered and re-inspected f600, f1890 | verified |
## Approval
- Every promoted image, card, and clip has an individual review record with an
audience verdict of `perfect`. No `acceptable-with-note` verdicts exist, so no
screen was promoted on a soft pass.
- All blocker/high defects (BUG-010, BUG-013, BUG-014) are fixed and verified.
- Root causes are evidence-backed; no unresolved evidence gaps.
- Every scene proof and transition sample approved.
- Walk-the-render ledger complete for every hold and card, no `fails` rows, and
the dream beat records genuine surprise.
- Every `need` rung in the brief is served by at least one approved beat
(see `storyboard.md` ladder coverage).
- Audience brief re-confirmed at approval: viewing moment, decision-maker, and
revisit trigger unchanged.

418
docs/ops/grafana-tidaldb.json
View File

@ -28,7 +28,7 @@
"description": "Requests per second by matched route TEMPLATE. Before tidaldb_http_requests_total existed there was no metric anywhere that could answer 'how much traffic are we serving'.",
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"gridPos": {
"h": 8,
@ -81,7 +81,7 @@
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"editorMode": "code",
"expr": "sum by (route) (rate(tidaldb_http_requests_total{namespace=~\"$namespace\",pod=~\"$pod\"}[$__rate_interval]))",
@ -98,7 +98,7 @@
"description": "Exact status codes, not classes: 401 (no/bad credential) vs 403 (valid data bearer, not an operator) vs 429 (rate limited) vs 503 (quorum timeout) are different incidents.",
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"gridPos": {
"h": 8,
@ -151,7 +151,7 @@
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"editorMode": "code",
"expr": "sum by (status) (rate(tidaldb_http_requests_total{namespace=~\"$namespace\",pod=~\"$pod\"}[$__rate_interval]))",
@ -168,7 +168,7 @@
"description": "Server-fault share of served requests. clamp_min keeps an idle cluster from dividing by zero and painting red.",
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"gridPos": {
"h": 6,
@ -200,7 +200,8 @@
},
"mappings": [],
"unit": "percentunit",
"decimals": 2
"decimals": 2,
"noValue": "no requests"
},
"overrides": []
},
@ -211,7 +212,7 @@
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNonNull"
"lastNotNull"
],
"fields": "",
"values": false
@ -222,7 +223,7 @@
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"editorMode": "code",
"expr": "sum(rate(tidaldb_http_requests_total{namespace=~\"$namespace\",pod=~\"$pod\",status=~\"5..\"}[$__rate_interval])) / clamp_min(sum(rate(tidaldb_http_requests_total{namespace=~\"$namespace\",pod=~\"$pod\"}[$__rate_interval])), 0.001)",
@ -239,7 +240,7 @@
"description": "401 = missing/invalid credential. 403 = a valid DATA bearer used against an operator verb, i.e. a client trying to reach /cluster/*. A sustained 403 rate is a client misconfigured with the wrong key, or someone probing.",
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"gridPos": {
"h": 6,
@ -292,7 +293,7 @@
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"editorMode": "code",
"expr": "sum by (status, route) (rate(tidaldb_http_requests_total{namespace=~\"$namespace\",pod=~\"$pod\",status=~\"401|403\"}[$__rate_interval]))",
@ -309,7 +310,7 @@
"description": "End-to-end latency as the CLIENT sees it, including auth, queueing behind the concurrency cap, and the leader forward. The engine's own latency series exclude all of that.",
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"gridPos": {
"h": 6,
@ -362,7 +363,7 @@
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"editorMode": "code",
"expr": "histogram_quantile(0.99, sum by (route, le) (rate(tidaldb_http_request_duration_us_bucket{namespace=~\"$namespace\",pod=~\"$pod\"}[$__rate_interval]))) / 1000",
@ -392,7 +393,7 @@
"description": "Search percentiles published directly by the engine.",
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"gridPos": {
"h": 7,
@ -445,33 +446,33 @@
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"editorMode": "code",
"expr": "tidaldb_search_latency_us_p50{namespace=~\"$namespace\",pod=~\"$pod\"} / 1000",
"legendFormat": "p50",
"legendFormat": "{{pod}} p50",
"range": true,
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"editorMode": "code",
"expr": "tidaldb_search_latency_us_p95{namespace=~\"$namespace\",pod=~\"$pod\"} / 1000",
"legendFormat": "p95",
"legendFormat": "{{pod}} p95",
"range": true,
"refId": "B"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"editorMode": "code",
"expr": "tidaldb_search_latency_us_p99{namespace=~\"$namespace\",pod=~\"$pod\"} / 1000",
"legendFormat": "p99",
"legendFormat": "{{pod}} p99",
"range": true,
"refId": "C"
}
@ -484,7 +485,7 @@
"description": "Retrieve percentiles published directly by the engine.",
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"gridPos": {
"h": 7,
@ -537,33 +538,33 @@
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"editorMode": "code",
"expr": "tidaldb_retrieve_latency_us_p50{namespace=~\"$namespace\",pod=~\"$pod\"} / 1000",
"legendFormat": "p50",
"legendFormat": "{{pod}} p50",
"range": true,
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"editorMode": "code",
"expr": "tidaldb_retrieve_latency_us_p95{namespace=~\"$namespace\",pod=~\"$pod\"} / 1000",
"legendFormat": "p95",
"legendFormat": "{{pod}} p95",
"range": true,
"refId": "B"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"editorMode": "code",
"expr": "tidaldb_retrieve_latency_us_p99{namespace=~\"$namespace\",pod=~\"$pod\"} / 1000",
"legendFormat": "p99",
"legendFormat": "{{pod}} p99",
"range": true,
"refId": "C"
}
@ -576,7 +577,7 @@
"description": "Write path. WAL group-commit fsync is plotted alongside because it is usually the reason a write p99 moves.",
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"gridPos": {
"h": 7,
@ -629,7 +630,7 @@
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"editorMode": "code",
"expr": "histogram_quantile(0.99, sum by (le) (rate(tidaldb_signal_write_latency_us_bucket{namespace=~\"$namespace\",pod=~\"$pod\"}[$__rate_interval]))) / 1000",
@ -640,7 +641,7 @@
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"editorMode": "code",
"expr": "histogram_quantile(0.99, sum by (le) (rate(tidaldb_cluster_wal_fsync_us_bucket{namespace=~\"$namespace\",pod=~\"$pod\"}[$__rate_interval]))) / 1000",
@ -663,6 +664,224 @@
},
"panels": []
},
{
"id": 14,
"type": "stat",
"title": "Cluster health",
"description": "min(tidaldb_health_ok) across the selected pods. DOWN if any node is unhealthy.",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"h": 4,
"w": 8,
"x": 0,
"y": 24
},
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto",
"textMode": "value",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
}
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "red",
"value": null
},
{
"color": "green",
"value": 1
}
]
},
"mappings": [
{
"type": "value",
"options": {
"0": {
"text": "DOWN",
"color": "red",
"index": 0
},
"1": {
"text": "OK",
"color": "green",
"index": 1
}
}
}
]
},
"overrides": []
},
"targets": [
{
"refId": "A",
"expr": "min(tidaldb_health_ok{namespace=~\"$namespace\",pod=~\"$pod\"})",
"legendFormat": "Cluster health",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
}
}
]
},
{
"id": 18,
"type": "stat",
"title": "Reseed pending",
"description": "max(tidaldb_cluster_reseed_required). REQUIRED means a node is rebuilding or must.",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"h": 4,
"w": 8,
"x": 8,
"y": 24
},
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto",
"textMode": "value",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
}
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 1
}
]
},
"mappings": [
{
"type": "value",
"options": {
"0": {
"text": "none",
"color": "green",
"index": 0
},
"1": {
"text": "REQUIRED",
"color": "red",
"index": 1
}
}
}
]
},
"overrides": []
},
"targets": [
{
"refId": "A",
"expr": "max(tidaldb_cluster_reseed_required{namespace=~\"$namespace\",pod=~\"$pod\"})",
"legendFormat": "Reseed pending",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
}
}
]
},
{
"id": 19,
"type": "stat",
"title": "Indexed vectors",
"description": "max(tidaldb_usearch_vector_count) \u2014 corpus size on the fullest node.",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"h": 4,
"w": 8,
"x": 16,
"y": 24
},
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto",
"textMode": "value",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
}
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "text",
"value": null
}
]
},
"mappings": [],
"unit": "short"
},
"overrides": []
},
"targets": [
{
"refId": "A",
"expr": "max(tidaldb_usearch_vector_count{namespace=~\"$namespace\",pod=~\"$pod\"})",
"legendFormat": "Indexed vectors",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
}
}
]
},
{
"id": 12,
"type": "timeseries",
@ -670,13 +889,13 @@
"description": "Events this replica is behind the leader. Sustained non-zero means a follower is not keeping up; a step change usually follows a leader change.",
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"gridPos": {
"h": 7,
"w": 8,
"w": 12,
"x": 0,
"y": 24
"y": 28
},
"fieldConfig": {
"defaults": {
@ -722,7 +941,7 @@
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"editorMode": "code",
"expr": "tidaldb_replication_lag_seqno{namespace=~\"$namespace\",pod=~\"$pod\"}",
@ -739,13 +958,13 @@
"description": "scatter degraded is the one to watch: a 200 answered with PARTIAL results because a shard errored or missed its deadline. It is a CORRECTNESS signal that looks like success to the client.",
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"gridPos": {
"h": 7,
"w": 8,
"x": 8,
"y": 24
"w": 12,
"x": 12,
"y": 28
},
"fieldConfig": {
"defaults": {
@ -791,7 +1010,7 @@
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"editorMode": "code",
"expr": "sum(rate(tidaldb_cluster_quorum_timeouts_total{namespace=~\"$namespace\",pod=~\"$pod\"}[$__rate_interval]))",
@ -802,7 +1021,7 @@
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"editorMode": "code",
"expr": "sum(rate(tidaldb_cluster_forward_failures_total{namespace=~\"$namespace\",pod=~\"$pod\"}[$__rate_interval]))",
@ -813,7 +1032,7 @@
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"editorMode": "code",
"expr": "sum(rate(tidaldb_cluster_scatter_degraded_total{namespace=~\"$namespace\",pod=~\"$pod\"}[$__rate_interval]))",
@ -824,7 +1043,7 @@
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"editorMode": "code",
"expr": "sum(rate(tidaldb_cluster_leader_changes_total{namespace=~\"$namespace\",pod=~\"$pod\"}[$__rate_interval]))",
@ -834,89 +1053,6 @@
}
]
},
{
"id": 14,
"type": "stat",
"title": "Fleet state",
"description": "health 0 or reseed pending 1 means a node is not serving normally. A reseed marker that persists past a restart is the m11p5 livelock signature.",
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"gridPos": {
"h": 7,
"w": 8,
"x": 16,
"y": 24
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"mappings": []
},
"overrides": []
},
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNonNull"
],
"fields": "",
"values": false
},
"textMode": "auto"
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "min(tidaldb_health_ok{namespace=~\"$namespace\",pod=~\"$pod\"})",
"legendFormat": "health",
"range": true,
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "max(tidaldb_cluster_reseed_required{namespace=~\"$namespace\",pod=~\"$pod\"})",
"legendFormat": "reseed pending",
"range": true,
"refId": "B"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "max(tidaldb_usearch_vector_count{namespace=~\"$namespace\",pod=~\"$pod\"})",
"legendFormat": "vectors",
"range": true,
"refId": "C"
}
]
},
{
"id": 15,
"type": "row",
@ -926,7 +1062,7 @@
"h": 1,
"w": 24,
"x": 0,
"y": 31
"y": 35
},
"panels": []
},
@ -937,13 +1073,13 @@
"description": "Load actively shed. NOTE: tidalDB's own limiter is UNLIMITED unless TIDAL_RATE_LIMIT_RPS is set, and it keys per PRINCIPAL \u2014 with one shared bearer every caller is one principal, so this is an aggregate cap, not fairness.",
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"gridPos": {
"h": 7,
"w": 12,
"x": 0,
"y": 32
"y": 36
},
"fieldConfig": {
"defaults": {
@ -990,7 +1126,7 @@
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"editorMode": "code",
"expr": "sum(rate(tidaldb_rate_limited_total{namespace=~\"$namespace\",pod=~\"$pod\"}[$__rate_interval]))",
@ -1001,7 +1137,7 @@
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"editorMode": "code",
"expr": "sum(rate(tidaldb_cluster_write_pool_rejections_total{namespace=~\"$namespace\",pod=~\"$pod\"}[$__rate_interval]))",
@ -1012,7 +1148,7 @@
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"editorMode": "code",
"expr": "sum by (status) (rate(tidaldb_http_requests_total{namespace=~\"$namespace\",pod=~\"$pod\",status=~\"429|408\"}[$__rate_interval]))",
@ -1029,13 +1165,13 @@
"description": "Uncompacted WAL and time since the last successful checkpoint. A checkpoint age that only climbs means the checkpoint thread is wedged \u2014 recovery time grows with it.",
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"gridPos": {
"h": 7,
"w": 12,
"x": 12,
"y": 32
"y": 36
},
"fieldConfig": {
"defaults": {
@ -1081,7 +1217,7 @@
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"editorMode": "code",
"expr": "tidaldb_wal_lag_bytes{namespace=~\"$namespace\",pod=~\"$pod\"}",
@ -1092,7 +1228,7 @@
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"editorMode": "code",
"expr": "tidaldb_checkpoint_age_seconds{namespace=~\"$namespace\",pod=~\"$pod\"}",
@ -1112,10 +1248,18 @@
"templating": {
"list": [
{
"current": {},
"current": {
"selected": true,
"text": [
"tidaldb-cluster"
],
"value": [
"tidaldb-cluster"
]
},
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"definition": "label_values(tidaldb_health_ok, namespace)",
"hide": 0,
@ -1136,7 +1280,7 @@
"current": {},
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
"uid": "victoriametrics"
},
"definition": "label_values(tidaldb_health_ok{namespace=~\"$namespace\"}, pod)",
"hide": 0,

122
docs/runbooks/deploy-verification.md
View File

@ -6,9 +6,40 @@ Walk this top to bottom. Every command here was executed against the live
Each check states **what it proves**, the command, and what you should see. If a
check fails, its **If it fails** line says where to look.
Sections 1–8 verify what is deployed **now**. Section 9 is deliberately separate:
those checks cannot pass until a new image is rolled, and saying so up front is
the point.
Sections 1–8 verify what is deployed **now**. Section 9 covers operator
authority plus the two features that are committed but inert on the running
image — kept separate so their absence is not mistaken for a regression.
## Run it automatically first
Every check below is also a Playwright test that asserts the same thing and
records what it observed. Run that first; walk the manual steps when something
fails, or when you want to see a layer for yourself.
```bash
export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
npm install && npx playwright install chromium # first time only
cargo build -p tidalctl # section 7 needs the binary
npm run test:e2e:list # discovery: syntax, imports, registration
npm run test:e2e:smoke # cluster plane + public plane + a quorum write
npm run test:e2e # the whole runbook, ~40 s
```
Credentials are read from the cluster by `globalSetup`, so nothing is pasted
into a shell. A missing prerequisite fails loudly rather than skipping a check.
The HTML report at `playwright-report/` carries the transcript of every command
the suite ran, which is the evidence trail this document used to describe in
prose.
There is one deliberate asymmetry. The manual steps tell you to `sleep 8` after
a `kubectl port-forward`; the harness polls the port until it accepts a
connection instead. Same fix, deterministic rather than empirical — and it is
why the automated pass takes 40 s where the manual walk takes several minutes.
A stakeholder walkthrough of the same evidence lives in `demo/` — see
`demo/storyboard.md` for what it shows and `demo/visual-audit.md` for how each
frame was reviewed.
---
@ -381,51 +412,64 @@ metadata:
---
## 9. Not active yet — requires an image roll
## 9. Operator authority, and what is still inert
The cluster runs `m12-boot-pull-fix-20260821`. The following are committed and
covered by tests but **inert in production** until a newer image is deployed.
They are listed so their absence is not mistaken for a regression.
The cluster runs
`registry.threesix.ai/tidal/server:m12-admin-gate-20260823@sha256:6e220060…`
(pods started 2026-08-23 05:32–05:41 UTC). That image carries the
operator/data credential split but **predates** the observability commit, so
§9.1 and §9.3 below are still inert. They are listed so their absence is not
mistaken for a regression.
### 9.1 HTTP request/error metrics
### 9.2 Operator/data credential split — LIVE, verify it stays that way
Confirmed working on this deployment. The admin routes are not published, so
port-forward first:
```bash
kubectl -n tidaldb-cluster port-forward svc/tidaldb 9500:9500 >/dev/null 2>&1 &
sleep 8
ADMIN=$(kubectl -n tidaldb-cluster get secret tidaldb-credentials \
-o jsonpath='{.data.TIDAL_ADMIN_KEY}' | base64 -d)
# data bearer on an operator verb -> 403 (authenticated, NOT authorized)
curl -sk -o /dev/null -w 'data -> promote %{http_code}\n' -X POST \
-H "Authorization: Bearer $TIDAL_API_KEY" -H 'content-type: application/json' \
-d '{"region":"tidaldb-1"}' https://127.0.0.1:9500/cluster/promote
# admin bearer -> authorized
curl -sk -o /dev/null -w 'admin -> promote %{http_code}\n' -X POST \
-H "Authorization: Bearer $ADMIN" -H 'content-type: application/json' \
-d '{"region":"tidaldb-1"}' https://127.0.0.1:9500/cluster/promote
```
**Pass:** `403` then `200`. Observed exactly that. Before this key existed the
data bearer could remove members and transfer shards — that is the exposure the
split closes.
> **The key was hot-loaded with no restart, and the startup log still says it is
> missing.** The pod booted 05:41 and logged `TIDAL_ADMIN_KEY is not set`;
> kubelet materialized the projected secret file at 05:51:43; the credential
> poller picked it up and the gate went live. So a `TIDAL_ADMIN_KEY is not set`
> WARN in the boot log does **not** mean the gate is open — test the behavior,
> which is why the two curls above are the actual check.
### 9.1 HTTP request/error metrics — still inert
```bash
PIP=$(kubectl -n tidaldb-cluster get pod tidaldb-0 -o jsonpath='{.status.podIP}')
kubectl -n observability exec deploy/vmagent -- \
sh -c "wget -qO- --timeout=6 http://$PIP:9091/metrics 2>/dev/null | grep -c tidaldb_http_requests_total"
sh -c "wget -qO- --timeout=15 http://$PIP:9091/metrics | grep -c tidaldb_http_requests_total"
```
Currently `0`. After the roll: non-zero, and the dashboard's *Request rate by
route*, *Requests by status*, *5xx ratio*, *Auth rejections* and *HTTP p99*
panels populate. Until then those five panels are legitimately empty.
Currently `0` on all three pods. After the observability image is rolled:
non-zero, and the dashboard's *Request rate by route*, *Requests by status*,
*5xx ratio*, *Auth rejections* and *HTTP p99* panels populate. Until then those
five panels are legitimately empty.
### 9.2 Operator/data credential split
After the roll, add the key (picked up by the credential poller, **no restart**):
```bash
kubectl -n tidaldb-cluster patch secret tidaldb-credentials --type=merge \
-p "{\"stringData\":{\"TIDAL_ADMIN_KEY\":\"$(openssl rand -hex 32)\"}}"
```
Then verify the split, port-forwarded (the admin routes are not published):
```bash
# data bearer on an operator verb -> 403 (authenticated, not authorized)
curl -sk -o /dev/null -w '%{http_code}\n' -X POST \
-H "Authorization: Bearer $TIDAL_API_KEY" -d '{"region":"tidaldb-1"}' \
https://127.0.0.1:9500/cluster/promote
# admin key -> anything except 401/403
ADMIN=$(kubectl -n tidaldb-cluster get secret tidaldb-credentials \
-o jsonpath='{.data.TIDAL_ADMIN_KEY}' | base64 -d)
curl -sk -o /dev/null -w '%{http_code}\n' -X POST \
-H "Authorization: Bearer $ADMIN" -d '{"region":"tidaldb-1"}' \
https://127.0.0.1:9500/cluster/heal
```
**Pass:** `403` then not-`403`. Before this key exists, the data bearer can
remove members and transfer shards — that is the exposure the split closes.
> **Use `--timeout=15`, not 6.** A 6-second `wget` truncates this scrape on a
> busy node and returns zero lines, which reads exactly like "the metric is
> missing" — it briefly looked like one pod had stopped exporting entirely.
### 9.3 Structured logs

4522
package-lock.json generated
View File

File diff suppressed because it is too large Load Diff

30
package.json
View File

@ -1 +1,29 @@
{}
{
"name": "tidaldb-deploy-verification",
"private": true,
"version": "0.0.0",
"description": "Playwright evidence harness for docs/runbooks/deploy-verification.md",
"scripts": {
"test:e2e": "playwright test --config playwright.config.ts",
"test:e2e:smoke": "playwright test --config playwright.config.ts tests/e2e/smoke.spec.ts",
"test:e2e:list": "playwright test --config playwright.config.ts --reporter=list --list",
"test:e2e:ui": "playwright test --config playwright.config.ts --ui",
"test:demo": "playwright test --config playwright.demo.config.ts",
"test:demo:list": "playwright test --config playwright.demo.config.ts --reporter=list --list",
"demo:preflight": "node --experimental-strip-types demo/preflight.ts",
"demo:studio": "remotion studio demo/src/index.ts",
"demo:still": "remotion still demo/src/index.ts DeployVerification demo/out/still.png",
"demo:render": "remotion render demo/src/index.ts DeployVerification demo/out/deploy-verification.mp4"
},
"devDependencies": {
"@playwright/test": "^1.56.1",
"@remotion/bundler": "^4.0.515",
"@remotion/cli": "^4.0.515",
"@remotion/renderer": "^4.0.515",
"@types/react": "^19.2.18",
"react": "^19.2.8",
"react-dom": "^19.2.8",
"remotion": "^4.0.515",
"typescript": "^7.0.2"
}
}

67
playwright.config.ts Normal file
View File

@ -0,0 +1,67 @@
import { defineConfig, devices } from '@playwright/test';
/**
* Regression config for the deploy-verification suite.
*
* This suite verifies a LIVE deployment (`orchard9-k3sf`, namespace
* `tidaldb-cluster`) against docs/runbooks/deploy-verification.md. It is not a
* unit suite and it does not boot an application — there is no `webServer`,
* because the thing under test is already running in a cluster.
*
* Consequences of testing a live system, all deliberate:
*
* - `workers: 1`. The suite opens kubectl port-forwards and writes to a real
* quorum. Parallel workers would fight over local ports and interleave
* writes into shared state, turning a real regression into a flake.
* - `retries: 0`, even on CI. A retry here would mask exactly the kind of
* intermittent cluster fault this suite exists to catch.
* - Long timeouts. Port-forward establishment plus a quorum-acked write across
* three nodes is legitimately slower than a local HTTP call.
*
* CLI hazard: passing `--reporter=...` REPLACES this reporter array rather than
* merging with it. Never add a `test:*` script that passes `--reporter` if a
* custom reporter is introduced later.
*/
export default defineConfig({
testDir: './tests/e2e',
testIgnore: ['**/demo/**'],
globalSetup: './tests/e2e/support/env-bootstrap.ts',
timeout: 120_000,
expect: { timeout: 20_000 },
forbidOnly: !!process.env.CI,
// A live-cluster check that only passes on the second attempt has told you
// something true about the cluster. Do not hide it.
retries: 0,
workers: 1,
fullyParallel: false,
reporter: process.env.CI
? [
['github'],
['html', { open: 'never', outputFolder: 'playwright-report' }],
['junit', { outputFile: 'test-results/playwright-junit.xml' }],
]
: [['list'], ['html', { open: 'never', outputFolder: 'playwright-report' }]],
outputDir: 'test-results/playwright',
use: {
// No global baseURL: this suite deliberately talks to several distinct
// origins (public ingress, port-forwarded pod, Grafana, Alertmanager) and a
// default would make which-origin bugs invisible.
actionTimeout: 20_000,
navigationTimeout: 45_000,
trace: 'retain-on-failure',
screenshot: 'only-on-failure',
video: 'retain-on-failure',
},
projects: [
{
name: 'verification',
use: { ...devices['Desktop Chrome'] },
},
],
});

66
playwright.demo.config.ts Normal file
View File

@ -0,0 +1,66 @@
import { defineConfig, devices } from '@playwright/test';
/**
* Demo capture config — separate from regression on purpose.
*
* Regression answers "is the deployment correct?". This config answers "can a
* human watch the proof?". Mixing them trains people to skip the fast suite, and
* a slowMo'd, video-recording regression run is nobody's PR gate.
*
* What differs from playwright.config.ts:
* - scoped to tests/e2e/demo, which only captures states the regression suite
* has already asserted;
* - a fixed 1600x900 viewport and pinned locale/timezone, so a capture is
* reproducible and a re-capture diffs cleanly;
* - video on, because "record the entire process" is the deliverable;
* - screenshots written explicitly at named proof points rather than
* on-failure, since the images ARE the artifact here;
* - its own outputDir so a demo run never clobbers regression artifacts.
*
* slowMo is deliberately NOT set. Global slowMo stretches every action equally,
* including the ones nobody needs to watch. Presentation pacing belongs to
* Remotion, which can hold a proof state for as long as it needs to be read.
*/
export default defineConfig({
testDir: './tests/e2e/demo',
globalSetup: './tests/e2e/support/env-bootstrap.ts',
timeout: 240_000,
expect: { timeout: 30_000 },
forbidOnly: !!process.env.CI,
retries: 0,
workers: 1,
fullyParallel: false,
reporter: [
['list'],
['html', { open: 'never', outputFolder: 'playwright-report-demo' }],
['json', { outputFile: 'test-results/playwright-demo/capture-run.json' }],
],
outputDir: 'test-results/playwright-demo',
use: {
actionTimeout: 30_000,
navigationTimeout: 60_000,
trace: 'retain-on-failure',
screenshot: 'only-on-failure',
video: { mode: 'on', size: { width: 1600, height: 900 } },
// Deterministic presentation surface. A capture that shifts with the
// reviewer's locale cannot be re-audited by content hash.
viewport: { width: 1600, height: 900 },
colorScheme: 'dark',
locale: 'en-US',
timezoneId: 'UTC',
deviceScaleFactor: 2,
},
projects: [
{
name: 'demo-capture',
use: { ...devices['Desktop Chrome'], viewport: { width: 1600, height: 900 } },
},
],
});

12
remotion.config.ts Normal file
View File

@ -0,0 +1,12 @@
/**
* Remotion config for the deploy-verification walkthrough.
*
* The captures live under demo/public/captures per the demo directory contract,
* so the public dir has to be pointed there — otherwise staticFile() resolves
* against a repo-root public/ that does not exist and every proof scene 404s.
*/
import { Config } from '@remotion/cli/config';
Config.setPublicDir('demo/public');
Config.setVideoImageFormat('jpeg');
Config.setOverwriteOutput(true);

241
tests/e2e/demo/support/proof-panel.ts Normal file
View File

@ -0,0 +1,241 @@
/**
* Renders REAL captured command output into a legible proof image.
*
* This is a presentation layer over evidence, never a substitute for it. Every
* line rendered here was produced by a command that actually ran against the
* live cluster in this same test run — the text is passed through verbatim and
* HTML-escaped, never composed or edited. The header states the exact command
* so a viewer can re-run it.
*
* Why render terminal output at all: tidalDB has no operator web UI, so most of
* the runbook's evidence is stdout. A screenshot of a terminal is the honest way
* to show that to someone, and for this audience — an operator, judged at Kyle
* Kingsbury's bar — real command output IS the credibility signal. The
* alternative would be inventing a UI that does not exist.
*/
import { createHash } from 'node:crypto';
import { mkdir, writeFile } from 'node:fs/promises';
import { dirname, join } from 'node:path';
import type { Page, TestInfo } from '@playwright/test';
import { redact } from '../../support/env';
export const CAPTURE_WIDTH = 1600;
export const CAPTURE_HEIGHT = 900;
/** Where the demo capture run writes its images before review. */
export const CAPTURE_DIR = join('test-results', 'demo-captures');
export type ProofLine = {
/** The command exactly as executed. */
command: string;
/** Its verbatim output. */
output: string;
/** What this line proves, in the viewer's vocabulary. */
verdict?: string;
/** Marks a deliberately-negative result (a refusal that should happen). */
negative?: boolean;
};
export type ProofPanel = {
captureId: string;
capabilityId: string;
title: string;
/** One sentence: what an operator learns from this screen. */
subtitle: string;
blocks: ProofLine[];
/** Optional footer, e.g. an explicit caveat. */
footnote?: string;
};
export type CaptureRecord = {
id: string;
capabilityId: string;
testId: string;
file: string;
expected: string;
businessPurpose: string;
personas: string[];
width: number;
height: number;
contentHash: string;
audienceVerdict: 'pending' | 'perfect' | 'acceptable-with-note' | 'slop';
auditStatus: 'pending' | 'pass' | 'fail';
};
const escapeHtml = (value: string): string =>
value
.replace(/&/g, '&amp;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;');
/**
* Terminal-styled document. Colours are limited to three roles — command,
* output, verdict — so nothing on screen implies a meaning it does not have. A
* refusal that is SUPPOSED to happen is marked as expected rather than red,
* because a red screen in a verification demo reads as a failure.
*/
function panelHtml(panel: ProofPanel): string {
const blocks = panel.blocks
.map((block) => {
const verdict = block.verdict
? `<div class="verdict ${block.negative ? 'expected' : 'good'}">${escapeHtml(
block.verdict,
)}</div>`
: '';
return `
<section class="block">
<div class="cmd"><span class="prompt">$</span> ${escapeHtml(block.command)}</div>
<pre class="out">${escapeHtml(block.output.trimEnd())}</pre>
${verdict}
</section>`;
})
.join('\n');
return `<!doctype html>
<html><head><meta charset="utf-8"><style>
:root {
--bg: #0d1117; --panel: #131922; --line: #222c3a;
--text: #d6deeb; --dim: #7c8ba1;
--cmd: #79c0ff; --good: #56d364; --expected: #d29922;
}
* { box-sizing: border-box; }
html, body { margin: 0; width: ${CAPTURE_WIDTH}px; height: ${CAPTURE_HEIGHT}px; }
body {
background: var(--bg); color: var(--text);
font: 15px/1.55 ui-monospace, SFMono-Regular, "SF Mono", Menlo, monospace;
padding: 44px 56px; display: flex; flex-direction: column;
}
header { border-bottom: 1px solid var(--line); padding-bottom: 18px; margin-bottom: 24px; }
h1 {
margin: 0 0 8px; font-size: 30px; letter-spacing: -0.2px; font-weight: 650;
font-family: ui-sans-serif, -apple-system, "Segoe UI", system-ui, sans-serif;
}
.sub {
margin: 0; color: var(--dim); font-size: 16px; max-width: 1150px;
font-family: ui-sans-serif, -apple-system, "Segoe UI", system-ui, sans-serif;
}
/* flex:1 + centred: a sparse panel reads as a composed slide instead of a
page that ran out of content, which is how 230-350px of dead space looked
on five of the first eight captures. */
.blocks {
display: flex; flex-direction: column; gap: 18px; overflow: hidden;
flex: 1; justify-content: center;
}
.block { background: var(--panel); border: 1px solid var(--line); border-radius: 8px; padding: 16px 18px; }
.cmd { color: var(--cmd); font-size: 14px; margin-bottom: 10px; word-break: break-all; }
.prompt { color: var(--dim); margin-right: 8px; }
pre.out {
margin: 0; white-space: pre-wrap; word-break: break-word;
font-size: 14.5px; line-height: 1.5; color: var(--text);
}
.verdict {
margin-top: 12px; padding-top: 10px; border-top: 1px dashed var(--line);
font-size: 14.5px; font-weight: 600;
font-family: ui-sans-serif, -apple-system, "Segoe UI", system-ui, sans-serif;
}
.verdict.good { color: var(--good); }
.verdict.expected { color: var(--expected); }
footer {
margin-top: auto; padding-top: 18px; border-top: 1px solid var(--line);
color: var(--dim); font-size: 13.5px;
font-family: ui-sans-serif, -apple-system, "Segoe UI", system-ui, sans-serif;
}
</style></head>
<body>
<header>
<h1>${escapeHtml(panel.title)}</h1>
<p class="sub">${escapeHtml(panel.subtitle)}</p>
</header>
<div class="blocks">${blocks}</div>
${panel.footnote ? `<footer>${escapeHtml(panel.footnote)}</footer>` : ''}
</body></html>`;
}
/**
* Render a proof panel and write it as a named capture.
*
* Secrets are redacted on the way in, so a bearer token can never reach an
* image even if a caller passes raw output through.
*/
export async function captureProofPanel(
page: Page,
testInfo: TestInfo,
panel: ProofPanel,
meta: { expected: string; businessPurpose: string; personas: string[] },
): Promise<CaptureRecord> {
const safe: ProofPanel = {
...panel,
blocks: panel.blocks.map((block) => ({
...block,
command: redact(block.command),
output: redact(block.output),
})),
};
await page.setViewportSize({ width: CAPTURE_WIDTH, height: CAPTURE_HEIGHT });
await page.setContent(panelHtml(safe), { waitUntil: 'load' });
await page.evaluate(() => document.fonts.ready);
const file = join(CAPTURE_DIR, `${panel.captureId}.png`);
await mkdir(dirname(file), { recursive: true });
const buffer = await page.screenshot({ path: file });
return {
id: panel.captureId,
capabilityId: panel.capabilityId,
testId: `${testInfo.titlePath.join(' :: ')}`,
file: `captures/${panel.captureId}.png`,
expected: meta.expected,
businessPurpose: meta.businessPurpose,
personas: meta.personas,
width: CAPTURE_WIDTH,
height: CAPTURE_HEIGHT,
contentHash: `sha256:${createHash('sha256').update(buffer).digest('hex')}`,
audienceVerdict: 'pending',
auditStatus: 'pending',
};
}
/** Record an already-taken screenshot (e.g. a real browser surface). */
export async function recordScreenshot(
buffer: Buffer,
testInfo: TestInfo,
spec: {
captureId: string;
capabilityId: string;
expected: string;
businessPurpose: string;
personas: string[];
width: number;
height: number;
},
): Promise<CaptureRecord> {
const file = join(CAPTURE_DIR, `${spec.captureId}.png`);
await mkdir(dirname(file), { recursive: true });
await writeFile(file, buffer);
return {
id: spec.captureId,
capabilityId: spec.capabilityId,
testId: `${testInfo.titlePath.join(' :: ')}`,
file: `captures/${spec.captureId}.png`,
expected: spec.expected,
businessPurpose: spec.businessPurpose,
personas: spec.personas,
width: spec.width,
height: spec.height,
contentHash: `sha256:${createHash('sha256').update(buffer).digest('hex')}`,
audienceVerdict: 'pending',
auditStatus: 'pending',
};
}
/** Append capture records to the run manifest for later promotion. */
export async function writeManifestFragment(
name: string,
records: CaptureRecord[],
): Promise<void> {
const file = join(CAPTURE_DIR, `manifest-${name}.json`);
await mkdir(dirname(file), { recursive: true });
await writeFile(file, `${JSON.stringify(records, null, 2)}\n`);
}

745
tests/e2e/demo/workflows/deploy-verification.demo.spec.ts Normal file
View File

@ -0,0 +1,745 @@
/**
* Demo capture — the stakeholder walkthrough of the deploy verification.
*
* Every capture in here re-runs the SAME assertion the regression suite makes,
* then photographs the asserted state. Nothing is captured that has not just
* been proven in the same test: if the assertion fails, no image is written.
* That is the whole contract — Playwright establishes truth, Remotion presents
* it, and this file is the seam.
*
* Beats map to demo/storyboard.md and capability IDs to
* demo/capability-inventory.md.
*/
import { expect, test } from '@playwright/test';
import { kubectl, run, tidalctl, withPortForward } from '../../support/cluster';
import {
BACKUP_NAMESPACE,
BACKUP_SCHEDULE,
DASHBOARD_UID,
FOREIGN_NAMESPACE,
FOREIGN_POD,
NAMESPACE,
OBS_NAMESPACE,
POD_NAMES,
PORT_CLIENT,
PORT_METRICS,
PUBLIC_BASE_URL,
PUBLIC_HOST,
adminKey,
apiKey,
grafanaPassword,
} from '../../support/env';
import {
CAPTURE_HEIGHT,
CAPTURE_WIDTH,
captureProofPanel,
recordScreenshot,
writeManifestFragment,
type CaptureRecord,
} from '../support/proof-panel';
const OPERATOR = ['cluster operator'];
const records: CaptureRecord[] = [];
test.afterAll(async () => {
await writeManifestFragment('deploy-verification', records);
});
test.describe('demo capture — deploy verification', () => {
test('CAP-002 every node is converged', async ({ page, playwright }, testInfo) => {
const key = apiKey();
const rows: string[] = [];
for (const pod of POD_NAMES) {
const status = await withPortForward(NAMESPACE, pod, PORT_CLIENT, async (forward) => {
const context = await playwright.request.newContext({
ignoreHTTPSErrors: true,
extraHTTPHeaders: { authorization: `Bearer ${key}` },
});
try {
const response = await context.get(
`https://127.0.0.1:${forward.localPort}/cluster/status/local`,
);
expect(response.status()).toBe(200);
return await response.json();
} finally {
await context.dispose();
}
});
// Assert before capturing. A capture is only ever a photograph of an
// already-proven state.
expect(status.reseed_required, `${pod} reseed`).toBe(false);
for (const shard of status.shards) {
expect(shard.lag_events, `${pod} shard ${shard.shard} lag`).toBe(0);
}
const shards = status.shards
.map(
(s: { shard: number; applied_events: number; lag_events: number; leader: string }) =>
`group ${s.shard}: applied=${s.applied_events} lag=${s.lag_events} leader=${s.leader}`,
)
.join('\n ');
rows.push(`${status.region} reseed=${status.reseed_required}\n ${shards}`);
}
records.push(
await captureProofPanel(
page,
testInfo,
{
captureId: 'CAP-002-convergence',
capabilityId: 'CAP-002',
title: 'Every node agrees, and none is behind',
subtitle:
'Each node is asked for its own view. The aggregated endpoint under-reports peers, ' +
'so the per-node answer is the authoritative one.',
blocks: [
{
command: 'GET /cluster/status/local (each pod, via port-forward)',
output: rows.join('\n\n'),
verdict:
'lag=0 on all 3 shard groups of all 3 nodes, no reseed pending, one agreed leader per group.',
},
],
footnote:
'A pod can be Ready while its replication is stalled — this is the check the ' +
'2026-08-20 reseed livelock defeated.',
},
{
expected: 'Three nodes, zero lag on every shard group, no reseed pending',
businessPurpose:
'Quorum with one-node fault tolerance actually exists, rather than being assumed from pod readiness',
personas: OPERATOR,
},
),
);
});
test('CAP-005 CAP-006 the boundary refuses, then a quorum write commits', async ({
page,
playwright,
}, testInfo) => {
const anonymous = await playwright.request.newContext();
const wrong = await playwright.request.newContext({
extraHTTPHeaders: { authorization: 'Bearer definitely-not-the-key' },
});
const authorized = await playwright.request.newContext({
extraHTTPHeaders: { authorization: `Bearer ${apiKey()}` },
});
try {
const target = `${PUBLIC_BASE_URL}/search?query=verification&limit=1`;
const none = await anonymous.get(target);
const bad = await wrong.get(target);
const good = await authorized.get(target);
expect(none.status()).toBe(401);
expect(bad.status()).toBe(401);
expect(good.status()).toBe(200);
const write = await authorized.post(`${PUBLIC_BASE_URL}/items`, {
headers: { 'content-type': 'application/json', 'x-tidal-ack': 'quorum' },
data: {
entity_id: 999_000_099,
metadata: { title: 'deploy verification probe', category: 'verification' },
},
});
expect(write.status()).toBe(201);
records.push(
await captureProofPanel(
page,
testInfo,
{
captureId: 'CAP-006-quorum-write',
capabilityId: 'CAP-006',
title: 'The boundary holds, and a write is committed by quorum',
subtitle:
'One hostname on the open internet. The same path, three credentials — then a real ' +
'write that a majority of nodes acknowledged.',
blocks: [
{
command: `curl -s -o /dev/null -w '%{http_code}' ${PUBLIC_HOST}/search # no credential`,
output: String(none.status()),
verdict: 'Refused. Expected — the corpus is not public.',
negative: true,
},
{
command: `curl -H 'Authorization: Bearer <wrong>' ${PUBLIC_HOST}/search`,
output: String(bad.status()),
verdict: 'Refused. A wrong key is rejected in constant time.',
negative: true,
},
{
command: `curl -H 'Authorization: Bearer <key>' ${PUBLIC_HOST}/search`,
output: String(good.status()),
verdict: 'Served.',
},
{
command: `curl -X POST -H 'x-tidal-ack: quorum' ${PUBLIC_HOST}/items`,
output: `${write.status()} Created`,
verdict:
'201 means a QUORUM acknowledged the write — not that one node accepted it. ' +
'DNS, TLS, gateway, auth and Raft replication, proven in one request.',
},
],
footnote:
'Verification writes use entity ids in a reserved 999_000_0xx band so a probe is ' +
'never mistaken for corpus data.',
},
{
expected: '401, 401, 200, then 201 for a quorum-acked write',
businessPurpose:
'The single strongest available proof: the full stack works and the data plane is closed to strangers',
personas: OPERATOR,
},
),
);
} finally {
await anonymous.dispose();
await wrong.dispose();
await authorized.dispose();
}
});
test('CAP-008 the metrics port is closed to foreign pods but open to the scraper', async ({
page,
}, testInfo) => {
const ip = await kubectl([
'-n',
NAMESPACE,
'get',
'pod',
'tidaldb-0',
'-o',
'jsonpath={.status.podIP}',
]);
const target = ip.stdout.trim();
const foreign = await kubectl([
'-n',
FOREIGN_NAMESPACE,
'exec',
FOREIGN_POD,
'--',
'sh',
'-c',
`wget -qO- --timeout=5 http://${target}:${PORT_METRICS}/metrics 2>&1 | head -2`,
]);
const scraper = await kubectl(
[
'-n',
OBS_NAMESPACE,
'exec',
'deploy/vmagent',
'--',
'sh',
'-c',
`wget -qO- --timeout=15 http://${target}:${PORT_METRICS}/metrics | grep -c '^tidaldb_'`,
],
{ timeoutMs: 60_000 },
);
const foreignOutput = `${foreign.stdout}${foreign.stderr}`.trim();
expect(foreignOutput).not.toContain('tidaldb_');
const series = Number.parseInt(scraper.stdout.trim(), 10);
expect(series).toBeGreaterThan(100);
records.push(
await captureProofPanel(
page,
testInfo,
{
captureId: 'CAP-008-network-isolation',
capabilityId: 'CAP-008',
title: 'The unauthenticated metrics port is not cluster-wide',
subtitle:
'A NetworkPolicy that blocks everything is an observability outage; one that blocks ' +
'nothing is theatre. Both directions are shown.',
blocks: [
{
command: `kubectl -n ${FOREIGN_NAMESPACE} exec ${FOREIGN_POD} -- wget http://${target}:9091/metrics`,
output: foreignOutput.slice(0, 220),
verdict:
'Refused. Before the policy existed, any pod in the cluster could read the corpus size.',
negative: true,
},
{
command: `kubectl -n ${OBS_NAMESPACE} exec deploy/vmagent -- wget … | grep -c '^tidaldb_'`,
output: `${series}`,
verdict: `The scraper still collects ${series} series — monitoring is intact.`,
},
],
footnote:
'Port 9500 is deliberately left open: all three probes originate from the node, and a ' +
'wrong rule there restarts every pod.',
},
{
expected: 'Connection refused from a foreign namespace; hundreds of series to the scraper',
businessPurpose: 'Least-privilege network access without blinding the monitoring stack',
personas: OPERATOR,
},
),
);
});
test('CAP-010 the operator dashboard renders live data', async ({ page }, testInfo) => {
await withPortForward(OBS_NAMESPACE, 'deploy/grafana', 3000, async (forward) => {
await page.setViewportSize({ width: CAPTURE_WIDTH, height: 1800 });
await page.setExtraHTTPHeaders({
authorization: `Basic ${Buffer.from(`admin:${grafanaPassword()}`).toString('base64')}`,
});
await page.goto(
`http://127.0.0.1:${forward.localPort}/d/${DASHBOARD_UID}/?from=now-6h&to=now&refresh=`,
{ waitUntil: 'networkidle' },
);
await page
.waitForFunction(() => document.querySelectorAll('canvas, .uplot').length > 3, {
timeout: 90_000,
})
.catch(() => undefined);
await page.waitForTimeout(6_000);
const health = await page.locator('[data-panelid="14"]').innerText();
expect(health, 'the health stat must render a value, not an empty box').toMatch(/OK|DOWN/);
// Clip to the evidence band rather than shipping the whole 1600x1800
// board. Scaled into a 16:9 frame the full board became illegible — the
// audit protocol's rule is to crop to the relevant region, not to shrink
// a dense page until nobody can read it. Bounds come from the real
// elements so a layout change cannot silently mis-crop. See BUG-010.
// Anchor the top to the ROW header (panel 7) so its title is not sliced,
// and the bottom tight to the last stat panel so the next row does not
// bleed in as a sliver. Both were visible crop artefacts on the first
// attempt.
const rowHeader = await page.locator('[data-panelid="7"]').boundingBox();
const last = await page.locator('[data-panelid="19"]').boundingBox();
expect(rowHeader, 'latency row header must anchor the top of the crop').not.toBeNull();
expect(last, 'indexed vectors panel must anchor the bottom of the crop').not.toBeNull();
const top = Math.max(0, Math.floor(rowHeader!.y - 8));
const bottom = Math.ceil(last!.y + last!.height + 8);
const clip = {
x: 0,
y: top,
width: CAPTURE_WIDTH,
height: bottom - top,
};
records.push(
await recordScreenshot(await page.screenshot({ clip }), testInfo, {
captureId: 'CAP-010-dashboard',
capabilityId: 'CAP-010',
expected:
'Cluster health OK, reseed none, corpus size, and per-node latency charts — legible at delivery resolution',
businessPurpose:
'The first surface an operator opens during an incident actually shows the cluster',
personas: OPERATOR,
width: clip.width,
height: clip.height,
}),
);
});
});
test('CAP-014 operator authority is separate from data access', async ({
page,
playwright,
}, testInfo) => {
const admin = adminKey();
expect(admin, 'admin key must be present').toBeTruthy();
await withPortForward(NAMESPACE, 'svc/tidaldb', PORT_CLIENT, async (forward) => {
const base = `https://127.0.0.1:${forward.localPort}`;
const body = { region: 'tidaldb-1' };
const dataContext = await playwright.request.newContext({
ignoreHTTPSErrors: true,
extraHTTPHeaders: {
authorization: `Bearer ${apiKey()}`,
'content-type': 'application/json',
},
});
const adminContext = await playwright.request.newContext({
ignoreHTTPSErrors: true,
extraHTTPHeaders: {
authorization: `Bearer ${admin}`,
'content-type': 'application/json',
},
});
try {
const dataAttempt = await dataContext.post(`${base}/cluster/promote`, { data: body });
const adminAttempt = await adminContext.post(`${base}/cluster/promote`, { data: body });
expect(dataAttempt.status()).toBe(403);
expect(adminAttempt.status()).not.toBe(403);
records.push(
await captureProofPanel(
page,
testInfo,
{
captureId: 'CAP-014-authority',
capabilityId: 'CAP-014',
title: 'An application key cannot remove a cluster member',
subtitle:
'The data credential authenticates but is not authorised for destructive ' +
'operator verbs. Only the admin key is.',
blocks: [
{
command: "POST /cluster/promote Authorization: Bearer <data key>",
output: `${dataAttempt.status()} Forbidden`,
verdict:
'403, not 401 — the key is valid, it simply lacks operator authority.',
negative: true,
},
{
command: 'POST /cluster/promote Authorization: Bearer <admin key>',
output: `${adminAttempt.status()}`,
verdict: 'Authorised. Operator authority is a separate credential.',
},
],
footnote:
'Before this split, the key every client holds could remove a member, force a ' +
'partition, or transfer a shard.',
},
{
expected: '403 for the data credential, not-403 for the admin credential',
businessPurpose:
'Blast radius of a leaked application key is bounded to data, not cluster topology',
personas: OPERATOR,
},
),
);
} finally {
await dataContext.dispose();
await adminContext.dispose();
}
});
});
test('CAP-014 CAP-015 the harness corrected its own runbook', async ({
page,
playwright,
}, testInfo) => {
// The dream beat needs a capture that SHOWS the contradiction, not one that
// merely sits next to a caption describing it. Left: the claim exactly as it
// was committed. Right: the live probe that disproved it. Both are real —
// the doc text comes out of git, the status codes out of the cluster.
const staleClaim = await run('git', [
'show',
'd21a202:docs/runbooks/deploy-verification.md',
]);
expect(staleClaim.code, 'the original runbook revision must be readable').toBe(0);
const claimLines = staleClaim.stdout
.split('\n')
.slice(383, 388)
.join('\n')
.trimEnd();
expect(claimLines, 'expected the superseded section heading').toContain(
'Not active yet',
);
const admin = adminKey();
expect(admin, 'admin key must be present').toBeTruthy();
const probe = await withPortForward(
NAMESPACE,
'svc/tidaldb',
PORT_CLIENT,
async (forward) => {
const base = `https://127.0.0.1:${forward.localPort}`;
const body = { region: 'tidaldb-1' };
const dataContext = await playwright.request.newContext({
ignoreHTTPSErrors: true,
extraHTTPHeaders: {
authorization: `Bearer ${apiKey()}`,
'content-type': 'application/json',
},
});
const adminContext = await playwright.request.newContext({
ignoreHTTPSErrors: true,
extraHTTPHeaders: {
authorization: `Bearer ${admin}`,
'content-type': 'application/json',
},
});
try {
const dataStatus = (await dataContext.post(`${base}/cluster/promote`, { data: body })).status();
const adminStatus = (await adminContext.post(`${base}/cluster/promote`, { data: body })).status();
return { dataStatus, adminStatus };
} finally {
await dataContext.dispose();
await adminContext.dispose();
}
},
);
expect(probe.dataStatus, 'the gate must be enforcing').toBe(403);
expect(probe.adminStatus, 'the admin key must be authorised').not.toBe(403);
const image = await kubectl([
'-n',
NAMESPACE,
'get',
'statefulset',
'tidaldb',
'-o',
'jsonpath={.spec.template.spec.containers[0].image}',
]);
records.push(
await captureProofPanel(
page,
testInfo,
{
captureId: 'CAP-014-drift',
capabilityId: 'CAP-014',
title: 'The document was wrong, and the harness said so',
subtitle:
'The runbook described a security control as not yet active. The harness read the ' +
'live image and the live secret, and found it already enforcing.',
blocks: [
{
command: 'git show d21a202:docs/runbooks/deploy-verification.md # as committed',
output: claimLines,
verdict: 'The claim: inert, pending an image roll.',
negative: true,
},
{
command: 'kubectl get statefulset tidaldb -o jsonpath={..image}',
output: image.stdout.trim(),
verdict: 'A different image is running than the one the runbook described.',
},
{
command: 'POST /cluster/promote with the data key, then the admin key',
output: `data key -> ${probe.dataStatus} Forbidden\nadmin key -> ${probe.adminStatus}`,
verdict:
'The gate was live the whole time. Documentation drift, caught by the thing it documents.',
},
],
footnote:
'Recorded as BUG-001. Section 9 of the runbook was rewritten in the same pass that ' +
'found this.',
},
{
expected: 'The superseded claim beside the live probe that contradicts it',
businessPurpose:
'Verification that audits its own documentation instead of drifting away from it',
personas: OPERATOR,
},
),
);
});
test('CAP-013 the fleet backup captured every volume', async ({ page }, testInfo) => {
const list = await kubectl([
'-n',
BACKUP_NAMESPACE,
'get',
'backup.velero.io',
'-l',
`velero.io/schedule-name=${BACKUP_SCHEDULE}`,
'--sort-by=.metadata.creationTimestamp',
'-o',
'jsonpath={range .items[*]}{.metadata.name}{"\\n"}{end}',
]);
const names = list.stdout.trim().split('\n').filter(Boolean);
const newest = names[names.length - 1];
expect(newest).not.toMatch(/restore-canary/);
const detail = await kubectl([
'-n',
BACKUP_NAMESPACE,
'get',
'backup.velero.io',
newest,
'-o',
'jsonpath=phase={.status.phase} items={.status.progress.itemsBackedUp}/{.status.progress.totalItems}',
]);
const volumes = await kubectl([
'-n',
BACKUP_NAMESPACE,
'get',
'podvolumebackups',
'-l',
`velero.io/backup-name=${newest}`,
'-o',
'jsonpath={range .items[*]}{.status.phase}{"\\n"}{end}',
]);
const phases = volumes.stdout.trim().split('\n').filter(Boolean);
expect(detail.stdout).toContain('phase=Completed');
expect([...new Set(phases)]).toEqual(['Completed']);
records.push(
await captureProofPanel(
page,
testInfo,
{
captureId: 'CAP-013-backup',
capabilityId: 'CAP-013',
title: 'The recovery story is intact',
subtitle:
'Selected by the schedule label the freshness alert actually watches — not simply the ' +
'newest backup object.',
blocks: [
{
command: `kubectl -n ${BACKUP_NAMESPACE} get backup.velero.io -l velero.io/schedule-name=${BACKUP_SCHEDULE} | tail -1`,
output: `${newest}\n${detail.stdout.trim()}`,
verdict: 'Completed with every discovered item captured.',
},
{
command: `kubectl -n ${BACKUP_NAMESPACE} get podvolumebackups -l velero.io/backup-name=${newest}`,
output: `${phases.length} PodVolumeBackups, all ${[...new Set(phases)].join(', ')}`,
verdict:
'One failed volume marks the whole backup PartiallyFailed and freezes the ' +
'freshness alert — so every volume must be clean.',
},
],
footnote:
'Sorting all backups by timestamp instead would have selected a restore-canary run: ' +
'20 items, one volume, and a meaningless pass.',
},
{
expected: 'Completed, all items, every PodVolumeBackup Completed',
businessPurpose: 'The cluster can actually be restored, and the alert is trustworthy',
personas: OPERATOR,
},
),
);
});
test('CAP-015 what is NOT verified is stated', async ({ page }, testInfo) => {
const ip = await kubectl([
'-n',
NAMESPACE,
'get',
'pod',
'tidaldb-0',
'-o',
'jsonpath={.status.podIP}',
]);
const counts = await kubectl(
[
'-n',
OBS_NAMESPACE,
'exec',
'deploy/vmagent',
'--',
'sh',
'-c',
`wget -qO- --timeout=15 http://${ip.stdout.trim()}:${PORT_METRICS}/metrics ` +
`| awk '/^tidaldb_http_/{h++} /^tidaldb_/{t++} END{print "tidaldb_* = "(t+0)"\\ntidaldb_http_* = "(h+0)}'`,
],
{ timeoutMs: 60_000 },
);
const image = await kubectl([
'-n',
NAMESPACE,
'get',
'statefulset',
'tidaldb',
'-o',
'jsonpath={.spec.template.spec.containers[0].image}',
]);
expect(counts.stdout).toMatch(/tidaldb_http_\* = 0/);
records.push(
await captureProofPanel(
page,
testInfo,
{
captureId: 'CAP-015-inert',
capabilityId: 'CAP-015',
title: 'What this deployment does not yet do',
subtitle:
'Two committed features are absent from the running image. Stating that is part of ' +
'the verification, not a footnote to it.',
blocks: [
{
command: 'kubectl get statefulset tidaldb -o jsonpath={..image}',
output: image.stdout.trim(),
verdict: 'This image carries the credential split but predates the observability commit.',
},
{
command: 'scrape :9091 and count metric families',
output: counts.stdout.trim(),
verdict:
'Zero HTTP metrics — and the baseline count proves the scrape WORKED, so "absent" ' +
'is distinguishable from "unscraped". Five dashboard panels are legitimately empty.',
negative: true,
},
],
footnote:
'The suite asserts this absence deliberately: the day the observability image is ' +
'rolled, these tests fail and say so — instead of the runbook silently rotting.',
},
{
expected: 'Zero tidaldb_http_* families while baseline tidaldb_* families are present',
businessPurpose:
'A verification that hides its gaps cannot be trusted about the parts it claims',
personas: OPERATOR,
},
),
);
});
test('CAP-012 tidalctl gives an operator a live view and an exit code', async ({
page,
}, testInfo) => {
await withPortForward(NAMESPACE, 'svc/tidaldb', PORT_CLIENT, async (forward) => {
const result = await tidalctl(
[
'cluster-status',
'--url',
`https://127.0.0.1:${forward.localPort}`,
'--key',
apiKey(),
'--insecure',
],
{ timeoutMs: 45_000 },
);
expect(result.stdout).toContain('regions:');
records.push(
await captureProofPanel(
page,
testInfo,
{
captureId: 'CAP-012-tidalctl',
capabilityId: 'CAP-012',
title: 'One command, and it names its own blind spot',
subtitle:
'The aggregated endpoint reports peers it holds no frontier report for as zero. ' +
'tidalctl labels that instead of repeating it as lag.',
blocks: [
{
command: 'tidalctl cluster-status --url https://… --insecure',
output: result.stdout.trim(),
// negative: the exit code is the FINDING, not a success. Rendering
// it green would have colour implying "good" for a defect.
negative: true,
verdict:
'NO REPORT is an honest "I do not know", not a fabricated 13.3M-event deficit. ' +
`But exit code ${result.code} on a converged cluster makes the documented ` +
'`cluster-status && deploy` gate unusable.',
},
],
footnote:
'Consequence, recorded as BUG-005: because a converged cluster still exits 2, ' +
'`tidalctl cluster-status && deploy` is NOT a usable gate on this deployment.',
},
{
expected: 'Leader, region table with NO REPORT markers, shard table, exit 2',
businessPurpose:
'An operator can interrogate the cluster without hand-rolling curl, and is told what the tool cannot see',
personas: OPERATOR,
},
),
);
});
});
});

193
tests/e2e/features/01-cluster-convergence.spec.ts Normal file
View File

@ -0,0 +1,193 @@
/**
* Runbook section 1 — the cluster is up and converged.
*
* CAP-001 membership is complete and every voter is Ready
* CAP-002 every node has converged: zero lag, no reseed, one agreed leader
*
* Why each node is asked individually rather than asking the aggregate:
* `GET /cluster/status` can report a peer it holds no frontier report for as
* `applied_events: 0` and then derive lag against that zero, so a fully
* converged peer shows up as the leader's entire history behind. The per-node
* `/cluster/status/local` is the authoritative view. See CAP-012 and
* docs/ops/observability.md section 4.
*/
import { expect, test } from '@playwright/test';
import { kubectl, withPortForward } from '../support/cluster';
import { observed, recordJson } from '../support/evidence';
import { NAMESPACE, POD_NAMES, PORT_CLIENT, apiKey } from '../support/env';
/** One shard group's replication position on one node. */
type ShardStatus = {
shard: number;
is_leader: boolean;
leader: string;
term: number;
role: string;
applied_events: number;
leader_seqno: number;
lag_events: number;
reseed_required: boolean;
reseeding: boolean;
};
type LocalStatus = {
region: string;
leader: string;
reseed_required: boolean;
reseeding: boolean;
quarantined: boolean;
partitioned: string[];
applied_events: number;
lag_events: number;
shards: ShardStatus[];
};
test.describe('section 1 — cluster convergence', () => {
test('membership is exactly the known voter set and every pod is Ready', async ({}, testInfo) => {
const result = await observed(testInfo, 'get pods wide', () =>
kubectl([
'-n',
NAMESPACE,
'get',
'pods',
'-l',
'app.kubernetes.io/name=tidaldb',
'-o',
'jsonpath={range .items[*]}{.metadata.name}{"\\t"}{.status.containerStatuses[0].ready}{"\\t"}{.status.containerStatuses[0].restartCount}{"\\t"}{.status.phase}{"\\n"}{end}',
]),
);
expect(result.code, result.stderr).toBe(0);
const pods = result.stdout
.trim()
.split('\n')
.filter((line) => line.trim() !== '')
.map((line) => {
const [name, ready, restarts, phase] = line.split('\t');
return {
name,
ready: ready === 'true',
restarts: Number.parseInt(restarts, 10),
phase,
};
});
await recordJson(testInfo, 'pod-inventory', pods);
// An unexpected extra pod means a scale operation is mid-flight or an
// orphan survived — either way the voter set is not what the runbook
// assumes, so assert the exact set rather than a minimum count.
expect(
pods.map((p) => p.name).sort(),
'voter set must be exactly the documented pods',
).toEqual([...POD_NAMES].sort());
for (const pod of pods) {
expect(pod.phase, `${pod.name} phase`).toBe('Running');
expect(pod.ready, `${pod.name} must be Ready`).toBe(true);
}
});
test('every node reports zero lag, no reseed, and agrees on one leader', async ({
playwright,
}, testInfo) => {
const key = apiKey();
const statuses: LocalStatus[] = [];
for (const pod of POD_NAMES) {
const status = await test.step(`read ${pod} /cluster/status/local`, async () =>
withPortForward(NAMESPACE, pod, PORT_CLIENT, async (forward) => {
// The pod serves TLS with the internal cluster CA, whose leaf is
// issued for in-cluster DNS names — a 127.0.0.1 tunnel cannot
// validate it. ignoreHTTPSErrors is scoped to this one context: a
// local port-forward to a named pod, never the public endpoint.
const context = await playwright.request.newContext({
ignoreHTTPSErrors: true,
extraHTTPHeaders: { authorization: `Bearer ${key}` },
});
try {
const response = await context.get(
`https://127.0.0.1:${forward.localPort}/cluster/status/local`,
);
expect(response.status(), `${pod} status endpoint`).toBe(200);
return (await response.json()) as LocalStatus;
} finally {
await context.dispose();
}
}));
statuses.push(status);
}
await recordJson(
testInfo,
'per-node-convergence',
statuses.map((status) => ({
region: status.region,
leader: status.leader,
reseed_required: status.reseed_required,
reseeding: status.reseeding,
quarantined: status.quarantined,
partitioned: status.partitioned,
shards: status.shards.map((shard) => ({
shard: shard.shard,
leader: shard.leader,
term: shard.term,
role: shard.role,
applied_events: shard.applied_events,
lag_events: shard.lag_events,
reseed_required: shard.reseed_required,
})),
})),
);
expect(statuses.length, 'one status per pod').toBe(POD_NAMES.length);
for (const status of statuses) {
expect(status.region, 'each pod reports its own region').toBeTruthy();
// reseed_required surviving a restart is the m11p5 livelock signature.
expect(status.reseed_required, `${status.region} must not require reseed`).toBe(false);
expect(status.reseeding, `${status.region} must not be reseeding`).toBe(false);
expect(status.quarantined, `${status.region} must not be quarantined`).toBe(false);
expect(status.partitioned, `${status.region} must see no partitions`).toEqual([]);
expect(status.shards.length, `${status.region} shard groups`).toBe(3);
for (const shard of status.shards) {
expect(
shard.lag_events,
`${status.region} shard ${shard.shard} must have zero lag`,
).toBe(0);
expect(
shard.reseed_required,
`${status.region} shard ${shard.shard} must not require reseed`,
).toBe(false);
expect(
shard.applied_events,
`${status.region} shard ${shard.shard} should have applied real events`,
).toBeGreaterThan(0);
}
}
// Agreement, not identity: leadership legitimately moves between runs (it
// moved from tidaldb-2 to tidaldb-1 during this suite's development), so
// pinning a node name would produce a test that fails on a healthy
// election. What must hold is that every node names the SAME leader for
// each shard group — disagreement is split brain.
for (let shardIndex = 0; shardIndex < 3; shardIndex += 1) {
const leaders = [
...new Set(
statuses.map(
(status) => status.shards.find((s) => s.shard === shardIndex)?.leader,
),
),
];
expect(
leaders,
`all nodes must agree on the leader of shard ${shardIndex}`,
).toHaveLength(1);
expect(leaders[0], `shard ${shardIndex} must have a leader`).toBeTruthy();
}
});
});

152
tests/e2e/features/02-public-endpoint.spec.ts Normal file
View File

@ -0,0 +1,152 @@
/**
* Runbook section 2 — the public endpoint and its TLS identity.
*
* CAP-003 public DNS resolves to every node
* CAP-004 TLS identity is correct and not near expiry
*
* The certificate is inspected through a real validated handshake rather than
* by reading the cert-manager Certificate object. A Certificate resource can
* report Ready while Traefik serves a stale or default certificate — the only
* claim that matters is what an external client is actually handed.
*/
import { resolve4 } from 'node:dns/promises';
import { connect, type PeerCertificate } from 'node:tls';
import { expect, test } from '@playwright/test';
import { recordJson } from '../support/evidence';
import { EXPECTED_NODE_IPS, PUBLIC_BASE_URL, PUBLIC_HOST } from '../support/env';
/** Minimum remaining certificate validity before this is a finding. */
const MIN_CERT_DAYS = 30;
type CertificateFacts = {
subjectCN: string;
issuer: string;
validTo: string;
daysRemaining: number;
subjectAltName?: string;
authorized: boolean;
protocol: string | null;
};
/**
* Complete a validated TLS handshake against one node IP with SNI set, and
* report what the peer presented.
*
* Connecting by IP with an explicit servername is deliberate: it proves the
* certificate is correct on *that specific node* rather than on whichever node
* DNS happened to return, which is how a partially-deployed certificate hides.
*/
async function inspectCertificate(host: string): Promise<CertificateFacts> {
const { promise, resolve, reject } = Promise.withResolvers<CertificateFacts>();
const socket = connect({
host,
port: 443,
servername: PUBLIC_HOST,
rejectUnauthorized: true,
timeout: 15_000,
});
socket.once('secureConnect', () => {
const certificate: PeerCertificate = socket.getPeerCertificate();
const validTo = new Date(certificate.valid_to);
// X.509 RDNs can legitimately repeat, so Node types these as
// `string | string[]`. Take the first value rather than stringifying an
// array into the assertion message.
const first = (value: string | string[] | undefined): string =>
Array.isArray(value) ? (value[0] ?? '') : (value ?? '');
resolve({
subjectCN: first(certificate.subject?.CN),
issuer: [first(certificate.issuer?.O), first(certificate.issuer?.CN)]
.filter(Boolean)
.join(' '),
validTo: validTo.toISOString(),
daysRemaining: Math.floor((validTo.getTime() - Date.now()) / 86_400_000),
subjectAltName: certificate.subjectaltname,
authorized: socket.authorized,
protocol: socket.getProtocol(),
});
socket.end();
});
socket.once('timeout', () => {
socket.destroy();
reject(new Error(`TLS handshake to ${host} timed out`));
});
socket.once('error', (error) => reject(error));
return promise;
}
test.describe('section 2 — public endpoint and TLS', () => {
test('the public hostname resolves to every node IP', async ({}, testInfo) => {
const addresses = await resolve4(PUBLIC_HOST);
await recordJson(testInfo, 'dns-a-records', {
host: PUBLIC_HOST,
resolved: [...addresses].sort(),
expected: [...EXPECTED_NODE_IPS].sort(),
});
// A missing record silently concentrates traffic; an extra record points
// somewhere undocumented. Assert the exact set.
expect([...addresses].sort(), 'A records must match the known node set').toEqual(
[...EXPECTED_NODE_IPS].sort(),
);
});
test('every node presents a valid, correctly-named, unexpired certificate', async ({}, testInfo) => {
const facts: Record<string, CertificateFacts> = {};
for (const ip of EXPECTED_NODE_IPS) {
facts[ip] = await test.step(`TLS handshake to ${ip}`, () => inspectCertificate(ip));
}
await recordJson(testInfo, 'tls-certificates', facts);
for (const [ip, certificate] of Object.entries(facts)) {
// rejectUnauthorized was true, so reaching here already proves the chain
// validated. Assert it explicitly so the intent survives a refactor.
expect(certificate.authorized, `${ip} chain must validate`).toBe(true);
expect(certificate.subjectCN, `${ip} certificate CN`).toBe(PUBLIC_HOST);
expect(certificate.issuer, `${ip} issuer should be Let's Encrypt`).toContain(
"Let's Encrypt",
);
expect(certificate.subjectAltName, `${ip} SAN must cover the hostname`).toContain(
PUBLIC_HOST,
);
expect(
certificate.daysRemaining,
`${ip} certificate expires in ${certificate.daysRemaining}d — renewal is overdue`,
).toBeGreaterThan(MIN_CERT_DAYS);
expect(certificate.protocol, `${ip} should negotiate modern TLS`).toMatch(
/TLSv1\.[23]/,
);
}
// All three nodes are fronted by the same Traefik and must present the same
// certificate. A divergence means one node did not pick up a renewal.
const distinctExpiries = [
...new Set(Object.values(facts).map((certificate) => certificate.validTo)),
];
expect(
distinctExpiries,
'every node must serve the same certificate generation',
).toHaveLength(1);
});
test('health is publicly reachable without a credential', async ({
playwright,
}, testInfo) => {
const context = await playwright.request.newContext({ ignoreHTTPSErrors: false });
try {
const response = await context.get(`${PUBLIC_BASE_URL}/health`);
const body = await response.text();
await recordJson(testInfo, 'health', { status: response.status(), body: body.slice(0, 300) });
// /health is intentionally open — it is what the probes and the uptime
// monitor call. It must not require the bearer.
expect(response.status(), '/health must be open and 200').toBe(200);
} finally {
await context.dispose();
}
});
});

156
tests/e2e/features/03-auth-boundary.spec.ts Normal file
View File

@ -0,0 +1,156 @@
/**
* Runbook section 3 — the authentication boundary and a real committed write.
*
* CAP-005 the data plane refuses unauthenticated and wrong credentials
* CAP-006 a quorum-acked write commits end to end
* CAP-007 operator and metrics surfaces are unroutable from the internet
*
* The negative cases are the point. A suite that only proves the happy path has
* proved that the endpoint works, not that it is protected — and this endpoint
* is on the open internet.
*/
import { expect, test } from '@playwright/test';
import { recordJson } from '../support/evidence';
import { PUBLIC_BASE_URL, apiKey } from '../support/env';
/**
* Entity ids for verification writes live in a reserved band so a probe is
* never mistaken for corpus data by someone reading the store later.
*/
const VERIFICATION_ENTITY_BASE = 999_000_000;
/**
* Paths published through the ingress allowlist in k8s/cluster/ingress.yaml.
* Everything else must be rejected by the gateway before it reaches the app.
*/
const UNPUBLISHED_PATHS = [
'/cluster/status',
'/cluster/members',
'/metrics',
'/openapi.json',
] as const;
test.describe('section 3 — authentication boundary', () => {
test('the same path is refused without a credential, refused with a wrong one, and served with the right one', async ({
playwright,
}, testInfo) => {
const anonymous = await playwright.request.newContext();
const wrong = await playwright.request.newContext({
extraHTTPHeaders: { authorization: 'Bearer definitely-not-the-key' },
});
const authorized = await playwright.request.newContext({
extraHTTPHeaders: { authorization: `Bearer ${apiKey()}` },
});
try {
const target = `${PUBLIC_BASE_URL}/search?query=verification&limit=1`;
const [anonymousResponse, wrongResponse, authorizedResponse] = [
await anonymous.get(target),
await wrong.get(target),
await authorized.get(target),
];
const observedStatuses = {
noCredential: anonymousResponse.status(),
wrongCredential: wrongResponse.status(),
correctCredential: authorizedResponse.status(),
};
await recordJson(testInfo, 'auth-matrix', observedStatuses);
// 401 for both denials. A 200 on the wrong bearer would be the highest
// severity finding this suite can produce; a 500 would mean the key was
// parsed and then mishandled.
expect(observedStatuses.noCredential, 'anonymous read must be refused').toBe(401);
expect(observedStatuses.wrongCredential, 'wrong bearer must be refused').toBe(401);
expect(observedStatuses.correctCredential, 'correct bearer must be served').toBe(200);
// A 200 with an empty body would mean the credential was accepted but the
// engine returned nothing — proof of auth, not proof of service.
const payload = await authorizedResponse.json();
await recordJson(testInfo, 'authorized-search-shape', {
keys: Object.keys(payload as Record<string, unknown>),
});
expect(
Object.keys(payload as Record<string, unknown>).length,
'an authorized read must return a real payload',
).toBeGreaterThan(0);
} finally {
await anonymous.dispose();
await wrong.dispose();
await authorized.dispose();
}
});
test('a quorum-acked write is committed, not merely accepted', async ({
playwright,
}, testInfo) => {
const context = await playwright.request.newContext({
baseURL: PUBLIC_BASE_URL,
extraHTTPHeaders: { authorization: `Bearer ${apiKey()}` },
});
try {
const entityId = VERIFICATION_ENTITY_BASE + 99;
const response = await context.post('/items', {
headers: { 'content-type': 'application/json', 'x-tidal-ack': 'quorum' },
data: {
entity_id: entityId,
metadata: {
title: 'deploy verification probe',
category: 'verification',
},
},
});
const body = await response.text();
await recordJson(testInfo, 'quorum-write', {
entityId,
status: response.status(),
body: body.slice(0, 300),
});
// 201 specifically. 202 would mean the write was accepted for later
// replication — that is exactly the weaker guarantee this check exists to
// rule out, so it must not be tolerated as "close enough".
expect(
response.status(),
'a quorum-ack write must return 201 (committed by quorum), not 202 (accepted)',
).toBe(201);
} finally {
await context.dispose();
}
});
test('operator and metrics surfaces are rejected by the gateway, not merely unauthorized', async ({
playwright,
}, testInfo) => {
// Deliberately carrying a VALID credential. If a path answered 401 that
// would be tolerable; answering 200 to a valid key on /cluster/status would
// mean the allowlist has drifted and the operator surface is public.
const context = await playwright.request.newContext({
baseURL: PUBLIC_BASE_URL,
extraHTTPHeaders: { authorization: `Bearer ${apiKey()}` },
});
try {
const statuses: Record<string, number> = {};
for (const path of UNPUBLISHED_PATHS) {
statuses[path] = (await context.get(path)).status();
}
await recordJson(testInfo, 'unpublished-path-statuses', statuses);
for (const path of UNPUBLISHED_PATHS) {
// 404 means Traefik never routed it. 401 would mean the request reached
// the application and only the credential check stopped it — the
// gateway allowlist would have drifted.
expect(
statuses[path],
`${path} must be unroutable (404) from the internet, not merely unauthorized`,
).toBe(404);
}
} finally {
await context.dispose();
}
});
});

170
tests/e2e/features/04-network-isolation.spec.ts Normal file
View File

@ -0,0 +1,170 @@
/**
* Runbook section 4 — NetworkPolicy enforcement.
*
* CAP-008 the metrics port is not reachable from arbitrary pods
* CAP-009 metrics reach the platform with the labels the dashboard queries
*
* A NetworkPolicy that blocks everything is an observability outage and one
* that blocks nothing is theatre, so both directions are proven in the same
* test: the foreign pod is refused AND the scraper still collects.
*/
import { expect, test } from '@playwright/test';
import { kubectl, withPortForward } from '../support/cluster';
import { observed, recordJson } from '../support/evidence';
import {
FOREIGN_NAMESPACE,
FOREIGN_POD,
NAMESPACE,
OBS_NAMESPACE,
PORT_METRICS,
} from '../support/env';
/**
* Generous, because a 6-second client timeout truncates this scrape on a loaded
* node and returns zero lines — indistinguishable from "this pod exports
* nothing". That false negative briefly looked like a pod had stopped
* exporting entirely (BUG-002).
*/
const SCRAPE_TIMEOUT_SECONDS = 15;
/** Minimum series a healthy node exports, well below the observed ~330. */
const MIN_EXPECTED_SERIES = 100;
async function podIp(pod: string): Promise<string> {
const result = await kubectl([
'-n',
NAMESPACE,
'get',
'pod',
pod,
'-o',
'jsonpath={.status.podIP}',
]);
expect(result.code, `could not read ${pod} IP: ${result.stderr}`).toBe(0);
expect(result.stdout.trim(), `${pod} should have an IP`).toMatch(/^\d+\.\d+\.\d+\.\d+$/);
return result.stdout.trim();
}
test.describe('section 4 — network isolation', () => {
test('the NetworkPolicy exists and selects the cluster pods', async ({}, testInfo) => {
const result = await observed(testInfo, 'get networkpolicy', () =>
kubectl(['-n', NAMESPACE, 'get', 'networkpolicy', 'tidaldb', '-o', 'json']),
);
expect(result.code, result.stderr).toBe(0);
const policy = JSON.parse(result.stdout) as {
spec: { podSelector: { matchLabels: Record<string, string> }; ingress: unknown[] };
};
await recordJson(testInfo, 'policy-selector', policy.spec.podSelector);
expect(
policy.spec.podSelector.matchLabels['app.kubernetes.io/name'],
'the policy must select the tidaldb pods',
).toBe('tidaldb');
expect(
policy.spec.ingress.length,
'the policy must declare at least one ingress allowance',
).toBeGreaterThan(0);
});
test('a pod in an unrelated namespace is refused on the metrics port', async ({}, testInfo) => {
const target = await podIp('tidaldb-0');
const result = await observed(testInfo, 'foreign pod probes metrics port', () =>
kubectl([
'-n',
FOREIGN_NAMESPACE,
'exec',
FOREIGN_POD,
'--',
'sh',
'-c',
`wget -qO- --timeout=5 http://${target}:${PORT_METRICS}/metrics 2>&1 | head -3`,
]),
);
const output = `${result.stdout}${result.stderr}`;
await recordJson(testInfo, 'foreign-probe', {
from: `${FOREIGN_NAMESPACE}/${FOREIGN_POD}`,
to: `${target}:${PORT_METRICS}`,
exitCode: result.code,
output: output.trim().slice(0, 400),
});
// Before the policy existed this returned '# HELP tidaldb_uptime_seconds…'
// from any pod in the cluster. The refusal is the whole point.
expect(
output,
'a foreign pod must NOT receive metrics — this is the exposure the policy closes',
).not.toContain('tidaldb_');
expect(output.toLowerCase()).toMatch(/refused|timed out|no route|unreachable/);
});
test('the scraper still collects metrics through the policy', async ({}, testInfo) => {
const target = await podIp('tidaldb-0');
const result = await observed(testInfo, 'scraper collects metrics', () =>
kubectl(
[
'-n',
OBS_NAMESPACE,
'exec',
'deploy/vmagent',
'--',
'sh',
'-c',
`wget -qO- --timeout=${SCRAPE_TIMEOUT_SECONDS} http://${target}:${PORT_METRICS}/metrics | grep -c '^tidaldb_'`,
],
{ timeoutMs: 45_000 },
),
);
expect(result.code, `scrape failed: ${result.stderr}`).toBe(0);
const seriesCount = Number.parseInt(result.stdout.trim(), 10);
await recordJson(testInfo, 'scrape-series-count', { target, seriesCount });
// A policy that blocked the scraper too would be an observability outage
// dressed as a security win.
expect(
seriesCount,
'the scraper must still reach the metrics port through the policy',
).toBeGreaterThan(MIN_EXPECTED_SERIES);
});
test('metrics carry the labels the dashboard templates depend on', async ({
playwright,
}, testInfo) => {
await withPortForward(OBS_NAMESPACE, 'svc/vmsingle', 8428, async (forward) => {
const context = await playwright.request.newContext();
try {
const response = await context.get(
`http://127.0.0.1:${forward.localPort}/api/v1/series`,
{ params: { 'match[]': 'tidaldb_health_ok' } },
);
expect(response.status(), 'vmsingle series query').toBe(200);
const payload = (await response.json()) as { data: Record<string, string>[] };
const labelSets = payload.data ?? [];
await recordJson(testInfo, 'series-labels', {
seriesCount: labelSets.length,
labels: labelSets.length > 0 ? Object.keys(labelSets[0]).sort() : [],
});
expect(labelSets.length, 'tidaldb_health_ok must exist in the store').toBeGreaterThan(0);
// The dashboard's $namespace / $pod template variables resolve against
// these exact label names. A rename leaves every panel blank while the
// series still exists — a monitoring outage that looks like an outage.
for (const label of ['namespace', 'pod', 'container', 'partition_id']) {
expect(
Object.keys(labelSets[0]),
`series must carry the '${label}' label the dashboard queries`,
).toContain(label);
}
} finally {
await context.dispose();
}
});
});
});

312
tests/e2e/features/05-metrics-dashboard.spec.ts Normal file
View File

@ -0,0 +1,312 @@
/**
* Runbook section 5 — the Grafana dashboard.
*
* CAP-010 the dashboard is loaded and its panels render
*
* This is the only genuine browser surface in the whole evidence chain, and it
* is the reason the visual pass matters. An API check proves the dashboard
* OBJECT exists; it does not prove an operator opening it sees anything. The
* dashboard shipped referencing datasource uid `${DS_PROMETHEUS}` — a Grafana
* export-for-sharing placeholder with no `__inputs` block to resolve it — so
* every panel queried a datasource that did not exist and the whole board was
* blank. The API said "loaded". See BUG-007.
*
* Grafana here runs behind an auth proxy with the login form disabled
* (GF_AUTH_DISABLE_LOGIN_FORM=true), so the suite authenticates with the
* cluster's own grafana-admin basic credential rather than inventing an
* identity or driving a form that does not exist.
*/
import { expect, test, type Page } from '@playwright/test';
import { withPortForward } from '../support/cluster';
import { recordJson } from '../support/evidence';
import { DASHBOARD_UID, OBS_NAMESPACE, grafanaPassword } from '../support/env';
const GRAFANA_PORT = 3000;
/** Panels whose queries need metrics the running image does not emit. */
const EXPECTED_EMPTY_PANEL_IDS = [2, 3, 4, 5, 6];
function basicAuth(): string {
return `Basic ${Buffer.from(`admin:${grafanaPassword()}`).toString('base64')}`;
}
type PanelState = {
id: number;
title: string;
populated: boolean;
noData: boolean;
};
/**
* Read every panel's rendered state.
*
* Grafana virtualizes: a panel below the fold is never queried and never draws.
* So this scrolls the whole board first, then classifies — otherwise every
* off-screen panel looks broken.
*/
async function readPanelStates(page: Page): Promise<PanelState[]> {
await page.evaluate(async () => {
const scroller =
document.querySelector('[class*="scrollbar-view"]') ??
document.scrollingElement ??
document.body;
const step = 400;
for (let y = 0; y < scroller.scrollHeight; y += step) {
scroller.scrollTop = y;
await new Promise((resolve) => setTimeout(resolve, 250));
}
scroller.scrollTop = 0;
await new Promise((resolve) => setTimeout(resolve, 500));
});
return page.evaluate(() => {
const panels: PanelState[] = [];
for (const element of Array.from(document.querySelectorAll('[data-panelid]'))) {
const id = Number.parseInt(element.getAttribute('data-panelid') ?? '0', 10);
const text = element instanceof HTMLElement ? element.innerText : '';
const header = element.querySelector('h2, h6, [class*="panel-title"]');
// Everything below the panel title.
const body = text.replace(/^[^\n]*\n?/, '').trim();
const noData = /no data/i.test(body);
// A populated panel drew a chart OR rendered a value. Requiring a DIGIT
// would be wrong: a stat panel with value mappings legitimately renders
// words — "OK" for health, "none" for reseed pending — and those are the
// most important readings on the board.
const populated =
!!element.querySelector('canvas') ||
!!element.querySelector('.uplot') ||
(body.length > 0 && !noData);
panels.push({
id,
title: (header?.textContent ?? text.split('\n')[0] ?? '').trim(),
populated,
noData,
});
}
return panels;
});
}
/**
* Every `datasource.uid` anywhere in a dashboard model — panels, targets,
* template variables, annotations. Only these are datasource references; the
* dashboard's own `uid` and any panel `libraryPanel.uid` are not.
*/
function collectDatasourceUids(node: unknown, found: string[] = []): string[] {
if (Array.isArray(node)) {
for (const item of node) collectDatasourceUids(item, found);
return found;
}
if (node === null || typeof node !== 'object') return found;
for (const [key, value] of Object.entries(node)) {
if (key === 'datasource' && value !== null && typeof value === 'object' && 'uid' in value) {
const uid = value.uid;
if (typeof uid === 'string') found.push(uid);
}
collectDatasourceUids(value, found);
}
return found;
}
test.describe('section 5 — metrics dashboard', () => {
test('the dashboard is provisioned with a resolvable datasource', async ({
playwright,
}, testInfo) => {
await withPortForward(OBS_NAMESPACE, 'deploy/grafana', GRAFANA_PORT, async (forward) => {
const context = await playwright.request.newContext({
extraHTTPHeaders: { authorization: basicAuth() },
});
try {
const base = `http://127.0.0.1:${forward.localPort}`;
const datasources = await context.get(`${base}/api/datasources`);
expect(datasources.status(), 'datasource list').toBe(200);
const provisioned = (await datasources.json()) as { uid: string; type: string }[];
const knownUids = provisioned.map((ds) => ds.uid);
const dashboard = await context.get(`${base}/api/dashboards/uid/${DASHBOARD_UID}`);
expect(dashboard.status(), `dashboard ${DASHBOARD_UID} must be provisioned`).toBe(200);
const payload = (await dashboard.json()) as {
dashboard: { title: string; panels: { id: number; type: string; title?: string }[] };
meta: { folderTitle?: string };
};
const serialized = JSON.stringify(payload.dashboard);
// Walk for `datasource.uid` specifically. A blanket /"uid":"…"/ scan
// also catches the dashboard's OWN uid and every library-panel uid,
// and reports them as missing datasources.
const referencedUids = [...new Set(collectDatasourceUids(payload.dashboard))];
const unresolvable = referencedUids.filter((uid) => !knownUids.includes(uid));
await recordJson(testInfo, 'datasource-wiring', {
provisioned: knownUids,
referenced: referencedUids,
unresolvable,
});
// The defect that made every panel blank. `${DS_PROMETHEUS}` is the
// placeholder Grafana writes when you "export for sharing"; under
// ConfigMap file-provisioning there is no import step to substitute it.
expect(
serialized,
'the dashboard must not ship an unresolved export placeholder',
).not.toContain('DS_PROMETHEUS');
expect(
unresolvable,
`dashboard references datasource uids that do not exist: ${unresolvable.join(', ')}`,
).toEqual([]);
expect(payload.meta.folderTitle, 'dashboard folder').toBe('Databases');
const nonRowPanels = payload.dashboard.panels.filter((panel) => panel.type !== 'row');
// 15, not 13: the single "Fleet state" panel was split into three
// single-query stat panels (Cluster health / Reseed pending / Indexed
// vectors) so each gets its own thresholds and value mappings. See
// BUG-008 and BUG-009.
expect(nonRowPanels.length, 'expected the full 15-panel board').toBe(15);
} finally {
await context.dispose();
}
});
});
test('every panel query returns data from the live store', async ({
playwright,
}, testInfo) => {
await withPortForward(OBS_NAMESPACE, 'deploy/grafana', GRAFANA_PORT, async (forward) => {
const context = await playwright.request.newContext({
extraHTTPHeaders: { authorization: basicAuth(), 'content-type': 'application/json' },
});
try {
const base = `http://127.0.0.1:${forward.localPort}`;
const dashboard = await context.get(`${base}/api/dashboards/uid/${DASHBOARD_UID}`);
const payload = (await dashboard.json()) as {
dashboard: {
panels: {
id: number;
type: string;
title?: string;
targets?: { expr?: string; refId?: string }[];
datasource?: { type?: string; uid?: string };
}[];
};
};
const results: Record<string, { expr: string; frames: number; status: number }> = {};
for (const panel of payload.dashboard.panels) {
if (panel.type === 'row' || !panel.targets) continue;
for (const [index, target] of panel.targets.entries()) {
if (!target.expr) continue;
const response = await context.post(`${base}/api/ds/query`, {
data: {
queries: [
{
refId: 'A',
datasource: panel.datasource ?? { type: 'prometheus', uid: 'victoriametrics' },
expr: target.expr,
instant: true,
},
],
from: 'now-6h',
to: 'now',
},
});
const body = (await response.json()) as {
results?: { A?: { frames?: unknown[]; status?: number; error?: string } };
};
results[`panel-${panel.id}-${index}`] = {
expr: target.expr.slice(0, 90),
frames: body.results?.A?.frames?.length ?? 0,
status: response.status(),
};
}
}
await recordJson(testInfo, 'panel-query-results', results);
// Every query must at least EXECUTE. A query that errors is a broken
// panel regardless of whether the underlying metric exists yet.
const failed = Object.entries(results).filter(([, r]) => r.status !== 200);
expect(
failed.map(([k, r]) => `${k}: HTTP ${r.status}`),
'every panel query must execute against a real datasource',
).toEqual([]);
} finally {
await context.dispose();
}
});
});
test('an operator opening the dashboard sees populated charts', async ({ page }, testInfo) => {
await withPortForward(OBS_NAMESPACE, 'deploy/grafana', GRAFANA_PORT, async (forward) => {
const consoleErrors: string[] = [];
page.on('console', (message) => {
if (message.type() !== 'error') return;
const text = message.text();
// Grafana Live opens a WebSocket; a browser cannot attach a basic-auth
// header to a WS upgrade, so this retry loop is an artefact of how the
// suite authenticates and not a dashboard defect.
if (text.includes('/api/live/ws')) return;
consoleErrors.push(text.slice(0, 200));
});
await page.setExtraHTTPHeaders({ authorization: basicAuth() });
await page.goto(
`http://127.0.0.1:${forward.localPort}/d/${DASHBOARD_UID}/?from=now-6h&to=now&refresh=`,
{ waitUntil: 'networkidle' },
);
// Wait for real plotted output, not merely for the shell to mount.
await page
.waitForFunction(() => document.querySelectorAll('canvas, .uplot').length > 3, {
timeout: 90_000,
})
.catch(() => undefined);
const panels = await readPanelStates(page);
const populated = panels.filter((panel) => panel.populated);
const dataPanels = panels.filter((panel) => panel.id > 0 && !isRow(panel));
await recordJson(testInfo, 'rendered-panel-states', {
total: panels.length,
populated: populated.length,
consoleErrors,
panels,
});
await testInfo.attach('dashboard.png', {
body: await page.screenshot({ fullPage: true }),
contentType: 'image/png',
});
expect(consoleErrors, `dashboard raised console errors: ${consoleErrors.join(' | ')}`).toEqual(
[],
);
// The whole point: charts must actually be drawn. Before the datasource
// fix this was zero while the API happily reported the dashboard loaded.
expect(
populated.length,
`no panel drew anything — the board is blank to an operator. Populated: ` +
`${populated.map((p) => p.id).join(',')}`,
).toBeGreaterThan(0);
// The five HTTP panels legitimately have no data on this image; the rest
// must render. Asserting the split means a real regression in the other
// eight is visible instead of being absorbed into "some panels are empty".
const blankNonHttp = dataPanels.filter(
(panel) => !panel.populated && !EXPECTED_EMPTY_PANEL_IDS.includes(panel.id),
);
expect(
blankNonHttp.map((panel) => `${panel.id} ${panel.title}`),
'these panels should have data but drew nothing',
).toEqual([]);
});
});
});
/** Row headers carry no data and must not be counted as blank panels. */
function isRow(panel: PanelState): boolean {
return [1, 7, 11, 15].includes(panel.id);
}

217
tests/e2e/features/06-logs.spec.ts Normal file
View File

@ -0,0 +1,217 @@
/**
* Runbook section 6 — logs are readable and replication is currently healthy.
*
* CAP-011 container logs are readable and free of unexpected errors
*
* Design note. An earlier draft of this spec allowlisted every WARN it found,
* which would have permanently hidden `batch ship failing … transport channel
* closed` — a real replication failure that ran for 115 consecutive seconds
* after an election. A blanket allowlist is a mute button.
*
* So this spec splits the question in two:
* 1. Is replication healthy RIGHT NOW? → asserted against a recent window.
* 2. What has the pod been saying? → recorded as evidence, and only
* genuinely benign, explained lines are tolerated.
*/
import { expect, test } from '@playwright/test';
import { kubectl } from '../support/cluster';
import { observed, recordJson } from '../support/evidence';
import { NAMESPACE, POD_NAMES } from '../support/env';
/** Window that must be quiet for replication to count as currently healthy. */
const HEALTH_WINDOW = '5m';
/** Longer window recorded as evidence, not asserted clean. */
const HISTORY_WINDOW = '30m';
/** Strip ANSI so patterns match; the deployed image colours its output. */
function decolour(text: string): string {
return text.replace(/\x1b\[[0-9;]*m/g, '');
}
/**
* Warnings that are genuinely benign on this deployment, each with the reason.
* An allowlist entry without a rationale is not allowed — if the reason cannot
* be written down, the line is not understood and must not be silenced.
*/
const BENIGN_WARNINGS: { match: RegExp; why: string }[] = [
{
match: /TIDAL_ADMIN_KEY is not set/i,
why:
'Stale boot-time WARN. kubelet materialized the projected secret at 05:51:43, after the ' +
'pod started at 05:41, and the credential poller hot-loaded it. The gate IS live — ' +
'proven behaviourally in 09-operator-authority.spec.ts. See BUG-003.',
},
{
match: /reading credential file failed.*admin-key/i,
why:
'Same cause: the admin-key volume is mounted optional:true, so each poller pass logged a ' +
'miss until the key existed.',
},
{
match: /TIDAL_CLUSTER_KEY not set/i,
why: 'Peer authentication uses the internal CA; the shared cluster key is unused here.',
},
{
match: /Multi-process cluster mode enabled/i,
why: 'Informational notice on every cluster-mode boot — this IS the production HA shape.',
},
{
match: /metrics server bound to non-loopback address/i,
why:
'Intentional: :9091 must be reachable by the vmagent scraper. Exposure is contained by ' +
'the NetworkPolicy, which 04-network-isolation.spec.ts proves refuses foreign pods.',
},
{
match: /catch-up stream open failed; will retry/i,
why:
'Self-healing by design — the stream reopens on the next detected gap or retry timer. ' +
'Tolerated only in the history window; the health window must be clean.',
},
];
/** Replication distress that must be absent in the health window. */
const REPLICATION_DISTRESS = /batch ship failing|transport channel closed|quarantin/i;
test.describe('section 6 — logs', () => {
test('replication is quiet right now on every pod', async ({}, testInfo) => {
const perPod: Record<string, { distressLines: string[]; sampled: number }> = {};
for (const pod of POD_NAMES) {
const result = await observed(testInfo, `recent logs ${pod}`, () =>
kubectl(['-n', NAMESPACE, 'logs', pod, `--since=${HEALTH_WINDOW}`], {
timeoutMs: 60_000,
}),
);
expect(result.code, `could not read ${pod} logs: ${result.stderr}`).toBe(0);
const lines = decolour(result.stdout)
.split('\n')
.filter((line) => line.trim() !== '');
perPod[pod] = {
sampled: lines.length,
distressLines: lines.filter((line) => REPLICATION_DISTRESS.test(line)).slice(0, 5),
};
}
await recordJson(testInfo, 'replication-health-window', { window: HEALTH_WINDOW, perPod });
for (const pod of POD_NAMES) {
// A pod shipping batches into a closed transport channel is not
// replicating, even while /cluster/status/local still reports lag=0
// because the leader has not yet advanced past the stuck position.
expect(
perPod[pod].distressLines,
`${pod} is reporting replication distress in the last ${HEALTH_WINDOW}: ` +
`${perPod[pod].distressLines.join(' | ')}`,
).toEqual([]);
}
});
test('no pod logged an ERROR, and every WARN is explained', async ({}, testInfo) => {
const perPod: Record<
string,
{ lines: number; errors: string[]; unexplained: string[]; benignCounts: Record<string, number> }
> = {};
for (const pod of POD_NAMES) {
const result = await observed(testInfo, `history logs ${pod}`, () =>
kubectl(['-n', NAMESPACE, 'logs', pod, `--since=${HISTORY_WINDOW}`, '--tail=400'], {
timeoutMs: 60_000,
}),
);
expect(result.code, `could not read ${pod} logs: ${result.stderr}`).toBe(0);
const lines = decolour(result.stdout)
.split('\n')
.filter((line) => line.trim() !== '');
const warnings = lines.filter((line) => /\bWARN\b/.test(line));
const benignCounts: Record<string, number> = {};
const unexplained: string[] = [];
for (const line of warnings) {
const matched = BENIGN_WARNINGS.find((entry) => entry.match.test(line));
if (matched) {
benignCounts[matched.match.source] = (benignCounts[matched.match.source] ?? 0) + 1;
} else if (REPLICATION_DISTRESS.test(line)) {
// Recorded, not failed: the previous test owns the health verdict and
// a historical, recovered episode is legitimate history.
benignCounts['recovered-replication-episode'] =
(benignCounts['recovered-replication-episode'] ?? 0) + 1;
} else {
unexplained.push(line.slice(0, 200));
}
}
perPod[pod] = {
lines: lines.length,
errors: lines.filter((line) => /\bERROR\b/.test(line)).slice(0, 10),
unexplained: unexplained.slice(0, 10),
benignCounts,
};
}
await recordJson(testInfo, 'log-classification', {
window: HISTORY_WINDOW,
allowlist: BENIGN_WARNINGS.map((entry) => ({ pattern: entry.match.source, why: entry.why })),
perPod,
});
for (const pod of POD_NAMES) {
expect(perPod[pod].lines, `${pod} should be logging at all`).toBeGreaterThan(0);
expect(
perPod[pod].errors,
`${pod} logged ERROR lines: ${perPod[pod].errors.join(' | ')}`,
).toEqual([]);
expect(
perPod[pod].unexplained,
`${pod} logged WARNs with no recorded rationale — investigate and either fix or ` +
`add an explained allowlist entry: ${perPod[pod].unexplained.join(' | ')}`,
).toEqual([]);
}
});
test('the deployed image still emits coloured plain text, so level filtering must happen at the source', async ({}, testInfo) => {
const result = await observed(testInfo, 'log format sample', () =>
kubectl(['-n', NAMESPACE, 'logs', 'tidaldb-0', '--tail=5'], { timeoutMs: 45_000 }),
);
expect(result.code, result.stderr).toBe(0);
const rawLines = result.stdout.split('\n').filter((line) => line.trim() !== '');
expect(rawLines.length, 'expected log lines to classify').toBeGreaterThan(0);
const jsonLines = rawLines.filter((line) => {
try {
return typeof JSON.parse(decolour(line)) === 'object';
} catch {
return false;
}
});
const ansiLines = rawLines.filter((line) => /\x1b\[/.test(line));
await recordJson(testInfo, 'log-format', {
sampled: rawLines.length,
jsonLines: jsonLines.length,
ansiLines: ansiLines.length,
conclusion:
'Coloured plain text. VictoriaLogs level:error cannot match, so filter with ' +
'kubectl logs | grep until the observability image is rolled (runbook 9.3).',
});
// Two tripwires in the honest direction. Both are CORRECT for the running
// image, and both fail the day the observability image lands — which is
// when the runbook needs updating. That is the drift that made section 9.2
// stale in the first place.
expect(
jsonLines.length,
'logs became structured JSON — roll runbook section 9.3 from pending to live and ' +
'invert this assertion',
).toBe(0);
expect(
ansiLines.length,
'ANSI escapes are gone — the observability image has been rolled; mark BUG-006 ' +
'verified and invert this assertion',
).toBeGreaterThan(0);
});
});

168
tests/e2e/features/07-tidalctl.spec.ts Normal file
View File

@ -0,0 +1,168 @@
/**
* Runbook section 7 — the live debugging tool.
*
* CAP-012 tidalctl interrogates a live cluster and its exit codes gate
*
* The exit-code contract is the part that matters operationally: an operator is
* expected to write `tidalctl cluster-status && deploy`, which is only safe if a
* non-converged cluster really exits non-zero and a bad credential does not
* silently exit 0.
*/
import { expect, test } from '@playwright/test';
import { tidalctl, withPortForward } from '../support/cluster';
import { observed, recordJson } from '../support/evidence';
import { NAMESPACE, PORT_CLIENT, apiKey } from '../support/env';
test.describe('section 7 — tidalctl live interrogation', () => {
test('cluster-status reports leadership, regions, and shards from a live node', async ({}, testInfo) => {
await withPortForward(NAMESPACE, 'svc/tidaldb', PORT_CLIENT, async (forward) => {
const result = await observed(testInfo, 'tidalctl cluster-status', () =>
tidalctl(
[
'cluster-status',
'--url',
`https://127.0.0.1:${forward.localPort}`,
'--key',
apiKey(),
// Required: the client port serves the INTERNAL cluster CA, whose
// leaf is issued for in-cluster DNS names, so a localhost tunnel
// cannot validate it.
'--insecure',
],
{ timeoutMs: 45_000 },
),
);
// Exit 2, not 0, on a FULLY CONVERGED cluster. This is not a tidalctl
// bug in isolation — the aggregated /cluster/status endpoint reports two
// of three healthy peers as `region: null, applied_events: 0,
// lag_events: 13322235, reachable: false`, while each of those peers'
// own /cluster/status/local reports lag=0 and all three agree on the
// leader (proven in 01-cluster-convergence.spec.ts). tidalctl correctly
// labels the gap `NO REPORT` but still folds it into its degraded
// verdict, so the exit code is 2.
//
// Consequence: `tidalctl cluster-status && deploy` can NEVER pass on this
// deployment. The runbook claimed it was a safe gate; that claim was
// written from an exit code masked by a shell pipeline. See BUG-005.
expect(
result.code,
'expected exit 2 — the aggregated-status gap makes a converged cluster report ' +
'degraded. If this is now 0, the engine-side peer reporting was fixed: ' +
'update runbook section 7 and mark BUG-005 verified.',
).toBe(2);
// The output must still be correct and complete even though the verdict
// is degraded — an operator reads these tables to locate the problem.
expect(result.stdout, 'must name the leader').toMatch(/leader:/);
expect(result.stdout, 'must list regions').toContain('regions:');
expect(result.stdout, 'must list shards').toContain('shards:');
});
});
test('the aggregated-status gap is reported as NO REPORT, never as fabricated lag', async ({}, testInfo) => {
await withPortForward(NAMESPACE, 'svc/tidaldb', PORT_CLIENT, async (forward) => {
const result = await observed(testInfo, 'tidalctl cluster-status regions', () =>
tidalctl(
[
'cluster-status',
'--url',
`https://127.0.0.1:${forward.localPort}`,
'--key',
apiKey(),
'--insecure',
],
{ timeoutMs: 45_000 },
),
);
// Degraded verdict for the reason pinned in the previous test (BUG-005);
// what matters here is HOW the gap is reported, not the exit code.
expect(result.code, result.stderr).toBe(2);
// The aggregated endpoint reports a peer it holds no frontier report for
// as applied=0 and derives lag against that zero, so a converged peer can
// read as the leader's entire history behind. tidalctl must name that
// condition rather than repeat it as lag.
const regionLines = result.stdout
.split('\n')
.filter((line) => /applied=/.test(line));
await recordJson(testInfo, 'region-lines', regionLines);
expect(regionLines.length, 'expected a line per region').toBeGreaterThan(0);
for (const line of regionLines) {
if (/applied=0\b/.test(line)) {
expect(
line,
'a peer with no frontier report must be labelled NO REPORT, not shown as real lag',
).toContain('NO REPORT');
}
}
});
});
test('watch emits one line per tick and terminates on the requested count', async ({}, testInfo) => {
await withPortForward(NAMESPACE, 'svc/tidaldb', PORT_CLIENT, async (forward) => {
const result = await observed(testInfo, 'tidalctl watch', () =>
tidalctl(
[
'watch',
'--url',
`https://127.0.0.1:${forward.localPort}`,
'--key',
apiKey(),
'--insecure',
'--interval',
'2',
'--count',
'3',
],
{ timeoutMs: 60_000 },
),
);
const ticks = result.stdout.split('\n').filter((line) => line.includes('leader='));
await recordJson(testInfo, 'watch-ticks', ticks);
// A bounded --count must terminate on its own. An unbounded watch here
// would hang the suite, which is why the flag exists.
expect(ticks.length, 'watch must emit exactly the requested number of ticks').toBe(3);
for (const tick of ticks) {
expect(tick, 'each tick must carry a verdict').toMatch(/\[(ok|DEGRADED)\]/);
}
});
});
test('exit codes gate correctly for a bad credential and a malformed url', async ({}, testInfo) => {
await withPortForward(NAMESPACE, 'svc/tidaldb', PORT_CLIENT, async (forward) => {
const badCredential = await observed(testInfo, 'tidalctl with wrong key', () =>
tidalctl(
[
'cluster-status',
'--url',
`https://127.0.0.1:${forward.localPort}`,
'--key',
'definitely-not-the-key',
'--insecure',
],
{ timeoutMs: 45_000 },
),
);
// Exit 2, not 0. A tool that exits 0 on a rejected credential would make
// `tidalctl cluster-status && deploy` deploy against an unverified cluster.
expect(
badCredential.code,
'a rejected credential must exit 2, never 0',
).toBe(2);
});
const malformedUrl = await observed(testInfo, 'tidalctl with schemeless url', () =>
tidalctl(
['cluster-status', '--url', '127.0.0.1:9500', '--key', apiKey(), '--insecure'],
{ timeoutMs: 20_000 },
),
);
// Exit 1 = usage error, distinct from exit 2 = reachable but unhealthy.
expect(malformedUrl.code, 'a malformed --url must exit 1 (usage)').toBe(1);
});
});

182
tests/e2e/features/08-backups.spec.ts Normal file
View File

@ -0,0 +1,182 @@
/**
* Runbook section 8 — the fleet backup completed and captured every volume.
*
* CAP-013 the fleet backup completed and captured every volume
*
* The selector is the whole lesson here. Sorting every Backup by creation
* timestamp and taking the newest returns a `restore-canary-*` run — 20 items,
* one volume — which reports Completed and proves nothing about the fleet. The
* check must filter on the schedule label that the freshness alert actually
* watches. See BUG-004.
*/
import { expect, test } from '@playwright/test';
import { kubectl } from '../support/cluster';
import { observed, recordJson } from '../support/evidence';
import { BACKUP_NAMESPACE, BACKUP_SCHEDULE } from '../support/env';
/** A fleet backup covering thousands of objects, not a canary's handful. */
const MIN_FLEET_ITEMS = 1_000;
/** Age beyond which a "most recent" backup is itself the finding. */
const MAX_BACKUP_AGE_HOURS = 48;
type BackupStatus = {
phase?: string;
errors?: number;
startTimestamp?: string;
completionTimestamp?: string;
progress?: { itemsBackedUp?: number; totalItems?: number };
};
test.describe('section 8 — backups', () => {
test('the newest fleet-schedule backup completed with every item and every volume', async ({}, testInfo) => {
const list = await observed(testInfo, 'list fleet-schedule backups', () =>
kubectl(
[
'-n',
BACKUP_NAMESPACE,
'get',
'backup.velero.io',
'-l',
`velero.io/schedule-name=${BACKUP_SCHEDULE}`,
'--sort-by=.metadata.creationTimestamp',
'-o',
'jsonpath={range .items[*]}{.metadata.name}{"\\n"}{end}',
],
{ timeoutMs: 45_000 },
),
);
expect(list.code, list.stderr).toBe(0);
const names = list.stdout
.trim()
.split('\n')
.filter((line) => line.trim() !== '');
expect(
names.length,
`no Backup carries label velero.io/schedule-name=${BACKUP_SCHEDULE} — either the ` +
`schedule was renamed or it has never run, and the freshness alert watches this label`,
).toBeGreaterThan(0);
const newest = names[names.length - 1];
// Guard the selector itself. A canary backup slipping through means the
// filter regressed and this whole test would be verifying the wrong object.
expect(
newest,
'the selected backup must come from the fleet schedule, not a restore canary',
).not.toMatch(/restore-canary/);
const detail = await observed(testInfo, `describe ${newest}`, () =>
kubectl(
['-n', BACKUP_NAMESPACE, 'get', 'backup.velero.io', newest, '-o', 'jsonpath={.status}'],
{ timeoutMs: 45_000 },
),
);
expect(detail.code, detail.stderr).toBe(0);
const status = JSON.parse(detail.stdout) as BackupStatus;
const ageHours = status.completionTimestamp
? (Date.now() - new Date(status.completionTimestamp).getTime()) / 3_600_000
: Number.POSITIVE_INFINITY;
await recordJson(testInfo, 'backup-status', {
name: newest,
phase: status.phase,
errors: status.errors ?? 0,
itemsBackedUp: status.progress?.itemsBackedUp,
totalItems: status.progress?.totalItems,
completedAt: status.completionTimestamp,
ageHours: Number.isFinite(ageHours) ? Number(ageHours.toFixed(1)) : null,
});
expect(status.phase, `${newest} must have completed cleanly`).toBe('Completed');
expect(status.errors ?? 0, `${newest} reported errors`).toBe(0);
expect(
status.progress?.itemsBackedUp,
'every discovered item must be backed up',
).toBe(status.progress?.totalItems);
expect(
status.progress?.totalItems ?? 0,
'a fleet backup should cover thousands of items — a small count means the wrong object',
).toBeGreaterThan(MIN_FLEET_ITEMS);
expect(
ageHours,
`the newest fleet backup is ${ageHours.toFixed(1)}h old — the schedule may have stopped`,
).toBeLessThan(MAX_BACKUP_AGE_HOURS);
const volumes = await observed(testInfo, `pod volume backups for ${newest}`, () =>
kubectl(
[
'-n',
BACKUP_NAMESPACE,
'get',
'podvolumebackups',
'-l',
`velero.io/backup-name=${newest}`,
'-o',
'jsonpath={range .items[*]}{.status.phase}{"\\n"}{end}',
],
{ timeoutMs: 45_000 },
),
);
expect(volumes.code, volumes.stderr).toBe(0);
const phases: Record<string, number> = {};
for (const phase of volumes.stdout.trim().split('\n')) {
if (phase.trim() === '') continue;
phases[phase.trim()] = (phases[phase.trim()] ?? 0) + 1;
}
await recordJson(testInfo, 'pod-volume-backup-phases', phases);
expect(
Object.keys(phases).length,
'the fleet backup must have captured pod volumes',
).toBeGreaterThan(0);
// One failed PVB marks the whole Backup PartiallyFailed and freezes
// velero_backup_last_successful_timestamp, so the fleet alert fires even
// when everything that matters was captured. Every volume must be clean.
expect(
Object.keys(phases).sort(),
`not every PodVolumeBackup completed: ${JSON.stringify(phases)}`,
).toEqual(['Completed']);
});
test('no Velero backup in the namespace is stuck in progress', async ({}, testInfo) => {
const result = await observed(testInfo, 'all backup phases', () =>
kubectl(
[
'-n',
BACKUP_NAMESPACE,
'get',
'backup.velero.io',
'-o',
'jsonpath={range .items[*]}{.metadata.name}{"\\t"}{.status.phase}{"\\n"}{end}',
],
{ timeoutMs: 45_000 },
),
);
expect(result.code, result.stderr).toBe(0);
const stuck = result.stdout
.trim()
.split('\n')
.filter((line) => line.trim() !== '')
.map((line) => {
const [name, phase] = line.split('\t');
return { name, phase };
})
.filter((backup) => backup.phase === 'InProgress' || backup.phase === 'Deleting');
await recordJson(testInfo, 'in-flight-backups', stuck);
// A backup wedged InProgress blocks the next scheduled run and silently
// stops the whole fleet from being protected.
expect(
stuck,
`backups stuck in flight: ${stuck.map((b) => `${b.name}=${b.phase}`).join(', ')}`,
).toEqual([]);
});
});

233
tests/e2e/features/09-operator-authority.spec.ts Normal file
View File

@ -0,0 +1,233 @@
/**
* Runbook section 9 — operator authority, and what is still inert.
*
* CAP-014 operator authority is separated from data-plane access
* CAP-015 inert observability features are inert for a known reason
*
* This spec exists because the runbook was WRONG about its own subject. It
* claimed the credential split was "not active yet — requires an image roll".
* The harness read the live image and the live secret and found the gate
* already enforcing. Documentation drift, caught by the thing it documents.
* See BUG-001.
*
* The inert-feature tests are tripwires in the honest direction: they assert
* ABSENCE today, so the day someone rolls the observability image they fail and
* say "update the runbook" — instead of the runbook rotting again.
*/
import { expect, test } from '@playwright/test';
import { kubectl, withPortForward } from '../support/cluster';
import { observed, recordJson } from '../support/evidence';
import {
NAMESPACE,
OBS_NAMESPACE,
PORT_CLIENT,
PORT_METRICS,
POD_NAMES,
adminKey,
apiKey,
} from '../support/env';
/** Generous: a short scrape timeout returns zero lines for everything (BUG-002). */
const SCRAPE_TIMEOUT_SECONDS = 15;
test.describe('section 9 — operator authority', () => {
test('the data-plane credential is refused on a destructive operator verb', async ({
playwright,
}, testInfo) => {
const admin = adminKey();
expect(
admin,
'TIDAL_ADMIN_KEY is absent from the secret. Without it the gate degrades to previous ' +
'behaviour by design and any client key can remove a member — add the key before ' +
'treating this deployment as verified.',
).toBeTruthy();
await withPortForward(NAMESPACE, 'svc/tidaldb', PORT_CLIENT, async (forward) => {
const base = `https://127.0.0.1:${forward.localPort}`;
const body = { region: 'tidaldb-1' };
const dataContext = await playwright.request.newContext({
ignoreHTTPSErrors: true,
extraHTTPHeaders: {
authorization: `Bearer ${apiKey()}`,
'content-type': 'application/json',
},
});
const adminContext = await playwright.request.newContext({
ignoreHTTPSErrors: true,
extraHTTPHeaders: {
authorization: `Bearer ${admin}`,
'content-type': 'application/json',
},
});
try {
const dataAttempt = await dataContext.post(`${base}/cluster/promote`, { data: body });
const adminAttempt = await adminContext.post(`${base}/cluster/promote`, { data: body });
await recordJson(testInfo, 'authority-split', {
verb: 'POST /cluster/promote',
dataCredential: dataAttempt.status(),
adminCredential: adminAttempt.status(),
});
// 403, not 401: the data bearer IS a valid credential, it simply lacks
// operator authority. A 401 here would mean the admin gate rejected it
// before authenticating, which would also break peer-callable verbs.
expect(
dataAttempt.status(),
'the data bearer must be authenticated but NOT authorized (403) on an operator verb',
).toBe(403);
// The admin key must clear both gates. It is deliberately a superset
// credential — one Authorization header per request means it has to
// authenticate as well as authorize, or operators get 401 before the
// admin gate ever runs.
expect(
adminAttempt.status(),
'the admin key must clear both authentication and the admin gate',
).not.toBe(401);
expect(adminAttempt.status(), 'the admin key must not be forbidden').not.toBe(403);
} finally {
await dataContext.dispose();
await adminContext.dispose();
}
});
});
test('cluster status requires a credential', async ({ playwright }, testInfo) => {
await withPortForward(NAMESPACE, 'svc/tidaldb', PORT_CLIENT, async (forward) => {
const anonymous = await playwright.request.newContext({ ignoreHTTPSErrors: true });
try {
const response = await anonymous.get(
`https://127.0.0.1:${forward.localPort}/cluster/status`,
);
await recordJson(testInfo, 'anonymous-cluster-status', { status: response.status() });
// Moved behind auth by 388e445. Unauthenticated access would leak
// leader identity, membership, and sequence positions.
expect(
response.status(),
'/cluster/status must require a credential even inside the cluster',
).toBe(401);
} finally {
await anonymous.dispose();
}
});
});
test('the admin key is mounted as a projected secret file', async ({}, testInfo) => {
const result = await observed(testInfo, 'list admin-key mount', () =>
kubectl(
['-n', NAMESPACE, 'exec', 'tidaldb-0', '-c', 'tidaldb', '--', 'ls', '/etc/tidaldb/admin-key/'],
{ timeoutMs: 45_000 },
),
);
await recordJson(testInfo, 'admin-key-mount', {
exitCode: result.code,
entries: result.stdout.trim().split('\n').filter(Boolean),
});
// The mount is optional:true on purpose — a required mount would prevent
// the pod from starting at all when the key is absent. Its presence here
// is what let the credential poller hot-load the key without a restart,
// which is why the boot log's "not set" WARN is stale (BUG-003).
expect(result.code, `admin-key mount unreadable: ${result.stderr}`).toBe(0);
expect(
result.stdout,
'the projected admin-key file must be present for the poller to load',
).toContain('admin-key');
});
test('HTTP request metrics are absent, and the scrape that proves it actually worked', async ({}, testInfo) => {
const counts: Record<string, { baseline: number; http: number }> = {};
for (const pod of POD_NAMES) {
const ip = await kubectl([
'-n',
NAMESPACE,
'get',
'pod',
pod,
'-o',
'jsonpath={.status.podIP}',
]);
expect(ip.code, ip.stderr).toBe(0);
const result = await observed(testInfo, `scrape ${pod}`, () =>
kubectl(
[
'-n',
OBS_NAMESPACE,
'exec',
'deploy/vmagent',
'--',
'sh',
'-c',
`wget -qO- --timeout=${SCRAPE_TIMEOUT_SECONDS} http://${ip.stdout.trim()}:${PORT_METRICS}/metrics ` +
`| awk '/^tidaldb_http_/{h++} /^tidaldb_/{t++} END{print (t+0)" "(h+0)}'`,
],
{ timeoutMs: 60_000 },
),
);
expect(result.code, `scrape of ${pod} failed: ${result.stderr}`).toBe(0);
const [baseline, http] = result.stdout.trim().split(/\s+/).map(Number);
counts[pod] = { baseline, http };
}
await recordJson(testInfo, 'metric-family-counts', counts);
for (const pod of POD_NAMES) {
// Prove the scrape WORKED before concluding a metric is missing. A short
// timeout returns zero for everything, which would make the assertion
// below pass for entirely the wrong reason — the exact trap that made one
// healthy pod look like it had stopped exporting (BUG-002).
expect(
counts[pod].baseline,
`${pod} returned no metrics at all — the scrape failed, so its http-metric count ` +
`proves nothing`,
).toBeGreaterThan(100);
expect(
counts[pod].http,
`${pod} now exports tidaldb_http_* metrics — good news: the observability image has ` +
`been rolled. Move runbook section 9.1 from inert to live, confirm the five HTTP ` +
`dashboard panels populate, and invert this assertion.`,
).toBe(0);
}
});
test('structured logging is not yet enabled on the StatefulSet', async ({}, testInfo) => {
const result = await observed(testInfo, 'statefulset env', () =>
kubectl(
[
'-n',
NAMESPACE,
'get',
'statefulset',
'tidaldb',
'-o',
'jsonpath={range .spec.template.spec.containers[0].env[*]}{.name}={.value}{"\\n"}{end}',
],
{ timeoutMs: 45_000 },
),
);
expect(result.code, result.stderr).toBe(0);
const env = result.stdout
.trim()
.split('\n')
.filter((line) => line.trim() !== '');
await recordJson(testInfo, 'statefulset-env', env);
const jsonLogs = env.find((line) => line.startsWith('JSON_LOGS='));
expect(
jsonLogs,
'JSON_LOGS is now set — roll runbook section 9.3 from pending to live, verify one JSON ' +
'object per line, and invert this assertion',
).toBeUndefined();
});
});

118
tests/e2e/smoke.spec.ts Normal file
View File

@ -0,0 +1,118 @@
/**
* Smoke: the shortest path that proves the harness can reach every plane it
* needs, and that the deployment is fundamentally alive.
*
* If this fails, nothing else in the suite is worth reading. It covers:
* - the cluster plane (kubectl can list the StatefulSet pods)
* - public DNS + TLS (the hostname resolves and the ingress serves health)
* - the credential path (the sourced bearer is accepted)
* - the write path (a quorum-acked write really commits)
*/
import { resolve4 } from 'node:dns/promises';
import { expect, test } from '@playwright/test';
import { kubectl } from './support/cluster';
import { observed, recordJson } from './support/evidence';
import {
EXPECTED_NODE_IPS,
NAMESPACE,
POD_NAMES,
PUBLIC_BASE_URL,
PUBLIC_HOST,
apiKey,
} from './support/env';
test.describe('smoke', () => {
test('all three cluster pods are Ready', async ({}, testInfo) => {
const result = await observed(testInfo, 'get pods', () =>
kubectl([
'-n',
NAMESPACE,
'get',
'pods',
'-l',
'app.kubernetes.io/name=tidaldb',
'-o',
'jsonpath={range .items[*]}{.metadata.name}{" "}{.status.containerStatuses[0].ready}{"\\n"}{end}',
]),
);
expect(result.code, `kubectl failed: ${result.stderr}`).toBe(0);
const readyByPod: Record<string, boolean> = {};
for (const line of result.stdout.trim().split('\n')) {
if (line.trim() === '') continue;
const [name, ready] = line.trim().split(/\s+/);
readyByPod[name] = ready === 'true';
}
await recordJson(testInfo, 'pod-readiness', readyByPod);
for (const pod of POD_NAMES) {
expect(readyByPod[pod], `${pod} should be Ready`).toBe(true);
}
expect(
Object.keys(readyByPod).sort(),
'expected exactly the known pod set',
).toEqual([...POD_NAMES].sort());
});
test('public hostname resolves to every node and serves health over valid TLS', async ({
playwright,
}, testInfo) => {
// Real public resolution is part of the claim, so resolve it for real
// rather than pinning an IP. A pinned IP would still pass if DNS were
// broken — which is precisely the failure that retired the old hostname.
const addresses = await resolve4(PUBLIC_HOST);
await recordJson(testInfo, 'dns-resolution', { host: PUBLIC_HOST, addresses });
expect(
[...addresses].sort(),
'every node IP should answer for the public hostname',
).toEqual([...EXPECTED_NODE_IPS].sort());
// ignoreHTTPSErrors stays false: an invalid or expired certificate must
// fail this test, not be tolerated.
const context = await playwright.request.newContext({ ignoreHTTPSErrors: false });
try {
const response = await context.get(`${PUBLIC_BASE_URL}/health`);
await recordJson(testInfo, 'health-response', {
status: response.status(),
body: (await response.text()).slice(0, 500),
});
expect(response.status(), 'health must be publicly reachable').toBe(200);
} finally {
await context.dispose();
}
});
test('sourced bearer is accepted and a quorum write commits', async ({
playwright,
}, testInfo) => {
const context = await playwright.request.newContext({
baseURL: PUBLIC_BASE_URL,
extraHTTPHeaders: { authorization: `Bearer ${apiKey()}` },
});
try {
const read = await context.get('/search', { params: { query: 'smoke', limit: 1 } });
expect(read.status(), 'authenticated read must succeed').toBe(200);
const write = await context.post('/items', {
headers: { 'content-type': 'application/json', 'x-tidal-ack': 'quorum' },
data: {
entity_id: 999_000_001,
metadata: { title: 'harness smoke', category: 'verification' },
},
});
await recordJson(testInfo, 'quorum-write', {
status: write.status(),
body: (await write.text()).slice(0, 500),
});
// 201 means the write was acknowledged by a quorum, not merely accepted
// by one node. This is the single strongest signal in the suite.
expect(write.status(), 'quorum-acked write must commit').toBe(201);
} finally {
await context.dispose();
}
});
});

227
tests/e2e/support/cluster.ts Normal file
View File

@ -0,0 +1,227 @@
/**
* Cluster-plane access for checks that have no HTTP surface.
*
* tidalDB serves JSON only — it has no operator web UI — and pod readiness,
* NetworkPolicy enforcement, container logs, and Velero state are not reachable
* over HTTP at all. Those checks shell out to kubectl. That is a deliberate,
* documented choice: the alternative is inventing an HTTP surface that does not
* exist and asserting against a fiction.
*
* Everything here returns real captured output for attachment as evidence, with
* secrets redacted on the way out.
*/
import { execFile, spawn, type ChildProcess } from 'node:child_process';
import { createConnection } from 'node:net';
import { setTimeout as sleep } from 'node:timers/promises';
import { promisify } from 'node:util';
import { KUBECONFIG, TIDALCTL_BIN, redact } from './env';
const execFileAsync = promisify(execFile);
export type CommandResult = {
/** argv as run, for the evidence record. */
command: string;
code: number;
stdout: string;
stderr: string;
};
const DEFAULT_TIMEOUT_MS = 30_000;
/**
* Run a binary with no shell. Never throws on a non-zero exit — the caller
* asserts on `code`, so a failing command is evidence rather than a crash.
*/
export async function run(
file: string,
args: string[],
options: { timeoutMs?: number; env?: Record<string, string> } = {},
): Promise<CommandResult> {
const command = redact([file, ...args].join(' '));
try {
const { stdout, stderr } = await execFileAsync(file, args, {
timeout: options.timeoutMs ?? DEFAULT_TIMEOUT_MS,
maxBuffer: 32 * 1024 * 1024,
env: { ...process.env, KUBECONFIG, ...options.env },
});
return { command, code: 0, stdout: redact(stdout), stderr: redact(stderr) };
} catch (error) {
const err = error as {
code?: number | string;
stdout?: string;
stderr?: string;
message: string;
};
return {
command,
code: typeof err.code === 'number' ? err.code : 1,
stdout: redact(err.stdout ?? ''),
stderr: redact(err.stderr ?? err.message),
};
}
}
/** Run kubectl with the pinned kubeconfig. */
export async function kubectl(
args: string[],
options: { timeoutMs?: number } = {},
): Promise<CommandResult> {
return run('kubectl', args, options);
}
/** Run kubectl and fail loudly — for prerequisites, where a failure is fatal. */
export async function kubectlOrThrow(args: string[]): Promise<string> {
const result = await kubectl(args);
if (result.code !== 0) {
throw new Error(
`kubectl failed (exit ${result.code}): ${result.command}\n${result.stderr}`,
);
}
return result.stdout;
}
/** Run the built tidalctl binary. */
export async function tidalctl(
args: string[],
options: { timeoutMs?: number } = {},
): Promise<CommandResult> {
return run(TIDALCTL_BIN, args, options);
}
/**
* Resolve one Secret key to plaintext. Used only to source credentials into the
* suite; the value is registered for redaction and never attached.
*/
export async function secretValue(
namespace: string,
secret: string,
key: string,
): Promise<string | undefined> {
const result = await kubectl([
'-n',
namespace,
'get',
'secret',
secret,
'-o',
`jsonpath={.data.${key}}`,
]);
if (result.code !== 0 || result.stdout.trim() === '') return undefined;
return Buffer.from(result.stdout.trim(), 'base64').toString('utf8');
}
/** True once something accepts a TCP connection on the port. */
async function portAccepts(port: number, host = '127.0.0.1'): Promise<boolean> {
const { promise, resolve } = Promise.withResolvers<boolean>();
const socket = createConnection({ port, host });
const settle = (ok: boolean) => {
socket.destroy();
resolve(ok);
};
socket.setTimeout(1_000);
socket.once('connect', () => settle(true));
socket.once('timeout', () => settle(false));
socket.once('error', () => settle(false));
return promise;
}
export type PortForward = {
localPort: number;
target: string;
close: () => Promise<void>;
};
/**
* Ports are allocated from a high base and bumped per acquisition so parallel
* workers never collide. Playwright workers are separate processes, so the
* worker index is folded into the base.
*/
let portCursor = 0;
const PORT_BASE = 19_600;
const WORKER_STRIDE = 40;
/**
* Open a port-forward and wait until the local port actually accepts a
* connection.
*
* The manual runbook tells a human to `sleep 8` because a shorter wait races
* kubectl's bind and returns an empty body that reads exactly like a dead node.
* A harness can do better than a fixed sleep: poll until the port answers, then
* proceed. Faster when the bind is quick, and it cannot produce that false
* negative when the cluster is slow.
*/
export async function portForward(
namespace: string,
target: string,
remotePort: number,
options: { timeoutMs?: number } = {},
): Promise<PortForward> {
const workerIndex = Number.parseInt(process.env.TEST_PARALLEL_INDEX ?? '0', 10);
const localPort = PORT_BASE + workerIndex * WORKER_STRIDE + portCursor;
portCursor = (portCursor + 1) % WORKER_STRIDE;
const timeoutMs = options.timeoutMs ?? 30_000;
const child: ChildProcess = spawn(
'kubectl',
['-n', namespace, 'port-forward', target, `${localPort}:${remotePort}`],
{ env: { ...process.env, KUBECONFIG }, stdio: ['ignore', 'pipe', 'pipe'] },
);
let exitInfo: string | undefined;
let stderr = '';
child.stderr?.on('data', (chunk: Buffer) => {
stderr += chunk.toString();
});
child.once('exit', (code, signal) => {
exitInfo = `kubectl port-forward exited early (code=${code} signal=${signal}): ${stderr.trim()}`;
});
const close = async () => {
if (child.exitCode !== null || child.signalCode !== null) return;
const { promise, resolve } = Promise.withResolvers<void>();
const hardKill = setTimeout(() => {
child.kill('SIGKILL');
resolve();
}, 3_000);
child.once('exit', () => {
clearTimeout(hardKill);
resolve();
});
child.kill('SIGTERM');
await promise;
};
const deadline = Date.now() + timeoutMs;
while (Date.now() < deadline) {
if (exitInfo) {
await close();
throw new Error(`${exitInfo}\ntarget: ${namespace}/${target}:${remotePort}`);
}
if (await portAccepts(localPort)) {
return { localPort, target: `${namespace}/${target}:${remotePort}`, close };
}
await sleep(250);
}
await close();
throw new Error(
`port-forward to ${namespace}/${target}:${remotePort} never accepted a connection ` +
`on 127.0.0.1:${localPort} within ${timeoutMs}ms.\nkubectl stderr: ${stderr.trim()}`,
);
}
/** Run `body` against an open port-forward and always tear it down. */
export async function withPortForward<T>(
namespace: string,
target: string,
remotePort: number,
body: (forward: PortForward) => Promise<T>,
): Promise<T> {
const forward = await portForward(namespace, target, remotePort);
try {
return await body(forward);
} finally {
await forward.close();
}
}

142
tests/e2e/support/env-bootstrap.ts Normal file
View File

@ -0,0 +1,142 @@
/**
* Global setup: source live credentials from the cluster once, before any test
* runs, and fail loudly if a prerequisite is missing.
*
* The suite must not silently skip a check because a credential was absent —
* that is the "false green" failure mode this whole harness exists to prevent.
* Every prerequisite either resolves here or the run stops with an actionable
* message.
*/
import { kubectl, secretValue } from './cluster';
import {
BACKUP_NAMESPACE,
NAMESPACE,
OBS_NAMESPACE,
TIDALCTL_BIN,
} from './env';
import { access } from 'node:fs/promises';
type Prerequisite = { name: string; check: () => Promise<string | undefined> };
async function firstFailure(prerequisites: Prerequisite[]): Promise<string[]> {
const failures: string[] = [];
for (const prerequisite of prerequisites) {
const problem = await prerequisite.check();
if (problem) failures.push(` ✗ ${prerequisite.name}: ${problem}`);
}
return failures;
}
export default async function globalSetup(): Promise<void> {
const failures = await firstFailure([
{
name: 'kubectl reaches the cluster',
check: async () => {
const result = await kubectl(['get', '--raw', '/readyz'], { timeoutMs: 15_000 });
if (result.code !== 0) return `kubectl not usable — ${result.stderr.trim()}`;
return undefined;
},
},
{
name: `namespace ${NAMESPACE} exists`,
check: async () => {
const result = await kubectl(['get', 'namespace', NAMESPACE, '-o', 'name']);
return result.code === 0 ? undefined : result.stderr.trim();
},
},
{
name: `namespace ${OBS_NAMESPACE} exists`,
check: async () => {
const result = await kubectl(['get', 'namespace', OBS_NAMESPACE, '-o', 'name']);
return result.code === 0 ? undefined : result.stderr.trim();
},
},
{
name: `namespace ${BACKUP_NAMESPACE} exists`,
check: async () => {
const result = await kubectl(['get', 'namespace', BACKUP_NAMESPACE, '-o', 'name']);
return result.code === 0 ? undefined : result.stderr.trim();
},
},
{
name: `${TIDALCTL_BIN} is built`,
check: async () => {
try {
await access(TIDALCTL_BIN);
return undefined;
} catch {
return `not found — run 'cargo build -p tidalctl'`;
}
},
},
]);
if (failures.length > 0) {
throw new Error(
`Deploy-verification prerequisites failed:\n${failures.join('\n')}\n\n` +
`See docs/runbooks/deploy-verification.md section 0.`,
);
}
// Source credentials from the cluster so no operator has to paste a secret
// into their shell. Written into process.env for the workers to inherit.
if (!process.env.E2E_TIDAL_API_KEY) {
const key = await secretValue(NAMESPACE, 'tidaldb-credentials', 'TIDAL_API_KEY');
if (!key) {
throw new Error(
`Could not read TIDAL_API_KEY from secret ${NAMESPACE}/tidaldb-credentials. ` +
`The data-plane bearer is required for every auth check.`,
);
}
process.env.E2E_TIDAL_API_KEY = key;
}
// Optional: present only after the admin-key image is rolled (section 9.2).
if (!process.env.E2E_TIDAL_ADMIN_KEY) {
const adminKey = await secretValue(
NAMESPACE,
'tidaldb-credentials',
'TIDAL_ADMIN_KEY',
);
if (adminKey) process.env.E2E_TIDAL_ADMIN_KEY = adminKey;
}
if (!process.env.E2E_GRAFANA_PASSWORD) {
const password = await secretValue(OBS_NAMESPACE, 'grafana-admin', 'password');
if (!password) {
throw new Error(
`Could not read password from secret ${OBS_NAMESPACE}/grafana-admin. ` +
`The dashboard checks in section 5 need it.`,
);
}
process.env.E2E_GRAFANA_PASSWORD = password;
}
// Record the image actually running, so every artifact says what was verified
// rather than what the repo happened to contain.
if (!process.env.E2E_BUILD_REVISION) {
const image = await kubectl([
'-n',
NAMESPACE,
'get',
'statefulset',
'tidaldb',
'-o',
'jsonpath={.spec.template.spec.containers[0].image}',
]);
process.env.E2E_BUILD_REVISION =
image.code === 0 && image.stdout.trim() !== '' ? image.stdout.trim() : 'unknown';
}
if (!process.env.E2E_RUN_ID) {
process.env.E2E_RUN_ID = `verify-${new Date().toISOString().replace(/[:.]/g, '-')}`;
}
process.stdout.write(
`\ndeploy-verification prerequisites OK\n` +
` image: ${process.env.E2E_BUILD_REVISION}\n` +
` run-id: ${process.env.E2E_RUN_ID}\n` +
` admin key present: ${process.env.E2E_TIDAL_ADMIN_KEY ? 'yes' : 'no (pre-roll, section 9.2)'}\n\n`,
);
}

127
tests/e2e/support/env.ts Normal file
View File

@ -0,0 +1,127 @@
/**
* Single source of truth for every URL, namespace, credential, and path this
* suite touches. Nothing else in the suite may read process.env directly.
*
* Defaults target the live orchard9-k3sf deployment described in
* docs/runbooks/deploy-verification.md. Override any value via env var to point
* the same suite at another cluster.
*/
function required(name: string, value: string | undefined): string {
if (!value || value.trim() === '') {
throw new Error(
`${name} is required but empty.\n` +
`Run 'source scripts/verify-env.sh' or export it manually.\n` +
`See docs/runbooks/deploy-verification.md section 0.`,
);
}
return value.trim();
}
/** Public hostname under test. */
export const PUBLIC_HOST = process.env.E2E_PUBLIC_HOST ?? 'tidaldb.threesix.ai';
/** Public base URL. */
export const PUBLIC_BASE_URL = `https://${PUBLIC_HOST}`;
/**
* Node IP used to pin the TLS connection when the local resolver lags behind
* public DNS (split-DNS resolvers such as Tailscale MagicDNS do this).
* Certificate validation still applies — only name resolution is bypassed.
*/
export const RESOLVE_IP = process.env.E2E_RESOLVE_IP ?? '208.122.204.172';
/** All node IPs that should answer for PUBLIC_HOST. */
export const EXPECTED_NODE_IPS = (
process.env.E2E_NODE_IPS ?? '208.122.204.172,208.122.204.173,208.122.204.174'
)
.split(',')
.map((ip) => ip.trim())
.filter(Boolean);
/** Kubernetes namespace holding the cluster StatefulSet. */
export const NAMESPACE = process.env.E2E_NAMESPACE ?? 'tidaldb-cluster';
/** Namespace holding Grafana / vmagent / vmsingle / Alertmanager. */
export const OBS_NAMESPACE = process.env.E2E_OBS_NAMESPACE ?? 'observability';
/** Namespace holding Velero. */
export const BACKUP_NAMESPACE = process.env.E2E_BACKUP_NAMESPACE ?? 'backup-system';
/** Velero schedule whose freshness the fleet alert watches. */
export const BACKUP_SCHEDULE =
process.env.E2E_BACKUP_SCHEDULE ?? 'velero-fleet-daily';
/** StatefulSet pod names, in ordinal order. */
export const POD_NAMES = (process.env.E2E_POD_NAMES ?? 'tidaldb-0,tidaldb-1,tidaldb-2')
.split(',')
.map((p) => p.trim())
.filter(Boolean);
/** Namespace + pod used to prove the metrics port is NOT reachable cluster-wide. */
export const FOREIGN_NAMESPACE = process.env.E2E_FOREIGN_NAMESPACE ?? 'threesix';
export const FOREIGN_POD = process.env.E2E_FOREIGN_POD ?? 'gitea-0';
/** Grafana dashboard uid under test. */
export const DASHBOARD_UID = process.env.E2E_DASHBOARD_UID ?? 'tidaldb-overview';
/** Ports inside the tidalDB container. */
export const PORT_CLIENT = 9500;
export const PORT_METRICS = 9091;
/** kubeconfig used for every kubectl invocation. */
export const KUBECONFIG =
process.env.KUBECONFIG ?? `${process.env.HOME}/.kube/orchard9-k3sf.yaml`;
/** Path to the built tidalctl binary. */
export const TIDALCTL_BIN = process.env.E2E_TIDALCTL_BIN ?? 'target/debug/tidalctl';
/** Revision recorded in capture metadata. */
export const BUILD_REVISION = process.env.E2E_BUILD_REVISION ?? 'unset';
/** Correlates this run across browser, CLI, and cluster evidence. */
export const RUN_ID = process.env.E2E_RUN_ID ?? `local-${Date.now()}`;
/**
* Data-plane bearer. Read lazily: a test that does not need it must not fail
* at import time, and the value must never be written into an artifact.
*/
export function apiKey(): string {
return required('E2E_TIDAL_API_KEY', process.env.E2E_TIDAL_API_KEY);
}
/** Operator bearer. Absent until the admin-key image is rolled (section 9.2). */
export function adminKey(): string | undefined {
const value = process.env.E2E_TIDAL_ADMIN_KEY;
return value && value.trim() !== '' ? value.trim() : undefined;
}
/** Grafana admin password, read lazily for the same reason as apiKey(). */
export function grafanaPassword(): string {
return required('E2E_GRAFANA_PASSWORD', process.env.E2E_GRAFANA_PASSWORD);
}
/**
* Every secret this suite can hold, for redaction before anything is written to
* an attachment, log line, or capture. Order matters: longest first, so a key
* that contains another as a substring is masked completely.
*/
export function secretsForRedaction(): string[] {
return [
process.env.E2E_TIDAL_API_KEY,
process.env.E2E_TIDAL_ADMIN_KEY,
process.env.E2E_GRAFANA_PASSWORD,
]
.filter((v): v is string => !!v && v.trim().length >= 8)
.map((v) => v.trim())
.sort((a, b) => b.length - a.length);
}
/** Replace every known secret with a stable placeholder. */
export function redact(text: string): string {
let out = text;
for (const secret of secretsForRedaction()) {
out = out.split(secret).join('«redacted»');
}
return out;
}

64
tests/e2e/support/evidence.ts Normal file
View File

@ -0,0 +1,64 @@
/**
* Evidence recording. Every check attaches what it actually observed, so a
* green run is readable afterwards and a red run is diagnosable without a
* re-run.
*
* This is the harness half of the project's observability rule: a process that
* discards its own output is a defect, and a test that asserts without
* recording what it saw is the same defect wearing a different hat.
*/
import { test, type TestInfo } from '@playwright/test';
import { redact } from './env';
import type { CommandResult } from './cluster';
/** Human-readable transcript of one command, for attachment. */
function transcript(result: CommandResult): string {
const lines = [`$ ${result.command}`, `exit: ${result.code}`];
if (result.stdout.trim() !== '') lines.push('--- stdout ---', result.stdout.trimEnd());
if (result.stderr.trim() !== '') lines.push('--- stderr ---', result.stderr.trimEnd());
return `${lines.join('\n')}\n`;
}
/** Attach one command's full transcript under a stable name. */
export async function recordCommand(
testInfo: TestInfo,
name: string,
result: CommandResult,
): Promise<void> {
await testInfo.attach(`${name}.txt`, {
body: Buffer.from(transcript(result)),
contentType: 'text/plain',
});
}
/** Attach an arbitrary observation as JSON, redacted. */
export async function recordJson(
testInfo: TestInfo,
name: string,
value: unknown,
): Promise<void> {
await testInfo.attach(`${name}.json`, {
body: Buffer.from(redact(JSON.stringify(value, null, 2))),
contentType: 'application/json',
});
}
/**
* Run a command inside a named `test.step` and attach its transcript, whether
* it succeeded or not. Returns the result so the caller can assert on it.
*
* Steps are what make the HTML report read as a walkthrough of the runbook
* rather than a flat list of assertions.
*/
export async function observed<T extends CommandResult>(
testInfo: TestInfo,
label: string,
invoke: () => Promise<T>,
): Promise<T> {
return test.step(label, async () => {
const result = await invoke();
await recordCommand(testInfo, label.replace(/[^a-z0-9]+/gi, '-').toLowerCase(), result);
return result;
});
}

101
tests/e2e/support/observability.ts Normal file
View File

@ -0,0 +1,101 @@
import {
expect,
type ConsoleMessage,
type Page,
type Request,
type Response,
type TestInfo,
} from '@playwright/test';
export type BrowserSignal = {
kind: 'console' | 'pageerror' | 'requestfailed' | 'response';
summary: string;
url?: string;
status?: number;
};
export type SignalGuardOptions = {
inspectResponse?: (response: Response) => boolean;
allow?: (signal: BrowserSignal) => boolean;
};
export type BrowserSignalGuard = {
signals: BrowserSignal[];
assertClean: () => Promise<void>;
attach: (testInfo: TestInfo) => Promise<void>;
};
/**
* Capture browser failures for the entire workflow. Install immediately after
* the page is created, before navigation or app actions.
*/
export function installBrowserSignalGuard(
page: Page,
options: SignalGuardOptions = {},
): BrowserSignalGuard {
const signals: BrowserSignal[] = [];
const record = (signal: BrowserSignal) => {
if (!options.allow?.(signal)) signals.push(signal);
};
page.on('console', (message: ConsoleMessage) => {
if (message.type() === 'error') {
record({
kind: 'console',
summary: message.text(),
url: message.location().url,
});
}
});
page.on('pageerror', (error: Error) => {
record({ kind: 'pageerror', summary: error.stack ?? error.message });
});
page.on('requestfailed', (request: Request) => {
record({
kind: 'requestfailed',
summary: request.failure()?.errorText ?? 'Request failed',
url: request.url(),
});
});
page.on('response', (response: Response) => {
const shouldInspect = options.inspectResponse
? options.inspectResponse(response)
: sameOrigin(response.url(), page.url());
if (shouldInspect && response.status() >= 400) {
record({
kind: 'response',
summary: `${response.request().method()} ${response.status()}`,
url: response.url(),
status: response.status(),
});
}
});
return {
signals,
async assertClean() {
await expect(
signals,
`Unexpected browser signals:\n${JSON.stringify(signals, null, 2)}`,
).toEqual([]);
},
async attach(testInfo: TestInfo) {
await testInfo.attach('browser-signals.json', {
body: Buffer.from(JSON.stringify(signals, null, 2)),
contentType: 'application/json',
});
},
};
}
function sameOrigin(candidate: string, currentPage: string): boolean {
try {
const pageOrigin = new URL(currentPage).origin;
return pageOrigin !== 'null' && new URL(candidate).origin === pageOrigin;
} catch {
return false;
}
}

25
tsconfig.json Normal file
View File

@ -0,0 +1,25 @@
{
"compilerOptions": {
"target": "ES2023",
"lib": ["ES2024", "DOM", "DOM.Iterable"],
"module": "ESNext",
"moduleResolution": "bundler",
"jsx": "react-jsx",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"resolveJsonModule": true,
"noEmit": true,
"types": ["node"]
},
"include": [
"playwright.config.ts",
"playwright.demo.config.ts",
"remotion.config.ts",
"tests/**/*.ts",
"demo/src/**/*.ts",
"demo/src/**/*.tsx",
"demo/capture-manifest.json"
],
"exclude": ["node_modules", "site", "target", "applications"]
}