15f6b11187
Turns docs/runbooks/deploy-verification.md from prose into 32 executable checks
against the live orchard9-k3sf cluster, and it found real defects on its first
run — including in the runbook it verifies.
WHY PLAYWRIGHT, HONESTLY
tidalDB serves zero HTML (no text/html, no Html(), 10 JSON routes), so this uses
Playwright in three distinct roles rather than pretending there is a UI:
* request fixture as a real HTTP client for DNS/TLS/auth/quorum/404;
* a browser for the only genuine screens in the chain, Grafana;
* a test harness for cluster-plane checks with no HTTP surface, shelling out
to kubectl and attaching the real transcript as evidence.
WHAT IT CAUGHT
* The runbook asserted the operator/data credential split was "not active yet
- requires an image roll". globalSetup read the live image and the live
secret; a probe returned data->403, admin->200. It had been enforcing the
whole time. Section 9 rewritten. (BUG-001)
* docs/ops/grafana-tidaldb.json shipped datasource uid ${DS_PROMETHEUS} - a
Grafana export-for-sharing placeholder with no __inputs block to resolve it.
Under ConfigMap provisioning every panel queried a datasource that did not
exist, so the whole board was blank. The API said "loaded" and I had only
ever checked the API. 41 refs fixed here, 58 across the fleet ConfigMap,
which was also blanking the postgres and redis dashboards. (BUG-007)
* Stat panels used calcs "lastNonNull". Grafana's reducer is "lastNotNull", so
no value was ever computed and Cluster health / Reseed pending / Indexed
vectors rendered as empty boxes. I chased panel width and then panel height
before comparing against a working stat panel elsewhere in the same Grafana.
A spelling error wearing a layout bug's clothes. (BUG-009)
* The namespace variable defaulted to All, so cluster panels silently included
tidaldb-586b544c8-vpkmw from the superseded standalone deployment. Latency
legends read "p50 p50 p50" with no way to tell the nodes apart. Both fixed.
* "5xx ratio" rendered "No data" as large green text - at a glance a healthy
value. And Fleet state gave three fields one shared green threshold, so
reseed_required=1 would have shown GREEN during the exact incident the panel
exists to surface. Split into three panels with per-field mappings.
* tidalctl cluster-status exits 2 on a FULLY CONVERGED cluster, because the
aggregated endpoint reports healthy peers as region=null applied=0
reachable=false. The runbook claimed `cluster-status && deploy` was a safe
gate; that claim came from an exit code masked by a shell pipeline. The gate
can never pass here. Documented, test pins it, engine defect recorded.
(BUG-005)
* The deployed image writes ANSI colour into container logs, which the
collector stores verbatim. Already fixed in logging.rs, not yet rolled;
pinned as a tripwire. (BUG-006)
* The runbook's own backup command sorted ALL backups by timestamp and
selected a restore-canary run: 20 items, one volume, a meaningless pass.
Now filters on the schedule label the freshness alert actually watches.
DEFECTS FOUND BY LOOKING AT THE SCREENS
Six of the first eight captures were slop and were fixed, not promoted:
230-350px of dead space; a verdict that rendered "exit code 2" in green; the
1600x1800 dashboard scaled into 16:9 until illegible (now clipped to the
evidence band using real element bounds); the dream beat whose caption described
a contradiction the image did not show (now a purpose-built capture holding the
committed doc text, the running image, and the live 403/200 side by side); and a
one-frame blink to bare background at every scene boundary, because Remotion
Sequences do not overlap and both scenes sat at opacity 0 on the boundary frame.
TRIPWIRES IN THE HONEST DIRECTION
Three tests assert what is ABSENT - zero tidaldb_http_* families, JSON_LOGS
unset, plain-text logs - and each carries the message "good news, roll the
runbook section from pending to live". The metric-absence test also asserts the
baseline family count, so "absent" cannot pass for "the scrape failed". That is
the drift that made section 9 stale in the first place.
Regression config uses workers:1 and retries:0 deliberately: a live-cluster
check that only passes on the second attempt has told you something true.
Verified: 32 passed (46.8s); 9 demo captures each asserting before photographing;
tsc clean; render 82.05s 1920x1080 h264, 0 empty frames across 10 boundaries;
every promoted image inspected individually and judged perfect; walk-the-render
ledger complete with no fails.
354 lines
23 KiB
Markdown
354 lines
23 KiB
Markdown
# Capability inventory — tidalDB deploy verification
|
||
|
||
Scope: every claim `docs/runbooks/deploy-verification.md` makes about the live
|
||
`orchard9-k3sf` deployment. One row per *capability* (a verifiable property),
|
||
not per command — several commands can serve one property, and one command can
|
||
touch several.
|
||
|
||
## Sources reconciled
|
||
|
||
| Source | What it contributed |
|
||
| --- | --- |
|
||
| `docs/runbooks/deploy-verification.md` | The nine sections and their pass criteria. |
|
||
| `tidal-server/src/router.rs` | The 10 registered routes; which are open vs bearer-gated vs admin-gated. |
|
||
| `k8s/cluster/ingress.yaml` | The published path allowlist — what is reachable from the internet at all. |
|
||
| `k8s/cluster/networkpolicy.yaml` | Ingress rules for `:9091`/`:9601` and the deliberate `:9500` exception. |
|
||
| `k8s/cluster/statefulset.yaml` | Probe ports, env, resource requests (and the live-vs-manifest drift). |
|
||
| `docs/ops/observability.md` | The four observability surfaces and the aggregated-status caveat. |
|
||
| `docs/ops/grafana-tidaldb.json` | The 13 dashboard panels and their PromQL. |
|
||
| Live cluster (`kubectl`, `curl`) | Actual image digest, actual secret keys, actual behavior. |
|
||
|
||
**Verified against.** Image
|
||
`registry.threesix.ai/tidal/server:m12-admin-gate-20260823@sha256:6e220060…`,
|
||
pods started 2026-08-23 05:32–05:41 UTC, namespace `tidaldb-cluster`,
|
||
repo commit recorded per-run in `E2E_BUILD_REVISION`.
|
||
|
||
**Environment gate discovered during inventory.** The running image predates the
|
||
observability commit (`4766f56`), so HTTP request metrics and structured logs are
|
||
absent by construction, while the operator/data credential split from `388e445`
|
||
*is* live. The runbook asserted all three were pending; that was stale. See
|
||
`BUG-001`.
|
||
|
||
## Inventory
|
||
|
||
### CAP-001 — Cluster membership is complete and every voter is Ready
|
||
|
||
- **Area.** `kubectl -n tidaldb-cluster get pods`; StatefulSet `tidaldb`.
|
||
- **Business purpose.** Three voters is the minimum for quorum with one-node
|
||
fault tolerance. Two Ready pods still serve reads and writes but tolerate no
|
||
further loss — a state that looks fine and is one failure from unavailable.
|
||
- **Personas.** Operator.
|
||
- **Primary workflow.** List pods by label; assert every known pod is Ready and
|
||
that the set is exactly the expected three.
|
||
- **Edge cases.** A fourth unexpected pod (mid-scale or orphan); a pod Running
|
||
but not Ready; a climbing restart count.
|
||
- **Permission boundary.** None — cluster-plane read.
|
||
- **Dependencies.** kubeconfig; the StatefulSet.
|
||
- **Observability.** Pod `ready` condition, `restartCount`, `startTime`.
|
||
- **Coverage.** `existing-green`
|
||
- **Evidence.** `tests/e2e/features/01-cluster-convergence.spec.ts`
|
||
|
||
### CAP-002 — Every node has converged: zero lag, no reseed, one agreed leader
|
||
|
||
- **Area.** `GET /cluster/status/local` on each pod via port-forward.
|
||
- **Business purpose.** The property that actually matters. A pod can be Ready
|
||
while its replication is stalled or it is rebuilding from a snapshot. This is
|
||
the check the 2026-08-20 reseed livelock defeated, which is why it is asked of
|
||
each node individually rather than of the aggregate.
|
||
- **Personas.** Operator.
|
||
- **Primary workflow.** For each pod: port-forward `:9500`, GET
|
||
`/cluster/status/local` with the data bearer, assert `reseed_required=false`
|
||
and `lag_events=0` on every shard group, and that all pods name the same
|
||
leader.
|
||
- **Edge cases.** `reseed_required=true` surviving a restart (livelock
|
||
signature); non-zero lag on one follower; disagreeing leaders (split brain);
|
||
the leader's own un-shipped tail showing as a small applied-position
|
||
difference, which is *not* lag.
|
||
- **Permission boundary.** Requires the data bearer since `388e445` moved
|
||
`/cluster/status*` behind auth; an unauthenticated request must be refused.
|
||
- **Dependencies.** port-forward; the data bearer.
|
||
- **Observability.** `applied_events`, `lag_events`, `leader`, `term`,
|
||
`reseed_required` per shard.
|
||
- **Coverage.** `existing-green`
|
||
- **Evidence.** `tests/e2e/features/01-cluster-convergence.spec.ts`
|
||
|
||
### CAP-003 — Public DNS resolves to every node
|
||
|
||
- **Area.** `tidaldb.threesix.ai` A records.
|
||
- **Business purpose.** Three A records is the load distribution and the
|
||
failover story. One missing record silently concentrates all traffic on two
|
||
nodes. A *retired* record that monitoring still watches produces a permanent
|
||
false alarm — which is exactly what happened to the previous hostname.
|
||
- **Personas.** Operator; any external API consumer.
|
||
- **Primary workflow.** Resolve the hostname for real and assert the address set
|
||
equals the three known node IPs.
|
||
- **Edge cases.** Split-DNS resolver returning a different set than public DNS;
|
||
partial record set; a stale record for a retired host.
|
||
- **Permission boundary.** None.
|
||
- **Dependencies.** Cloudflare zone `threesix.ai`.
|
||
- **Observability.** Resolved address list, attached per run.
|
||
- **Coverage.** `existing-green`
|
||
- **Evidence.** `tests/e2e/smoke.spec.ts`, `tests/e2e/features/02-public-endpoint.spec.ts`
|
||
|
||
### CAP-004 — TLS identity is correct and not near expiry
|
||
|
||
- **Area.** TLS handshake on `:443` for `tidaldb.threesix.ai`.
|
||
- **Business purpose.** A client that cannot validate the certificate either
|
||
fails closed (outage) or is configured to skip validation (silent downgrade to
|
||
no transport security). Expiry is the most common self-inflicted outage in this
|
||
class.
|
||
- **Personas.** External API consumer.
|
||
- **Primary workflow.** Complete a real handshake with validation enabled;
|
||
assert subject CN, a Let's Encrypt issuer, and at least 30 days of remaining
|
||
validity.
|
||
- **Edge cases.** Wrong SAN; self-signed fallback; cert valid but chain
|
||
incomplete; expiry inside the renewal window.
|
||
- **Permission boundary.** None.
|
||
- **Dependencies.** cert-manager `letsencrypt-prod` (Cloudflare dns01), Traefik.
|
||
- **Observability.** Subject, issuer, `notAfter`, days remaining.
|
||
- **Coverage.** `existing-green`
|
||
- **Evidence.** `tests/e2e/features/02-public-endpoint.spec.ts`
|
||
|
||
### CAP-005 — The data plane refuses unauthenticated and wrong credentials
|
||
|
||
- **Area.** `GET /search` (and the other published data paths) via the ingress.
|
||
- **Business purpose.** This endpoint is on the open internet. The bearer is the
|
||
only thing between the corpus and anyone who finds the hostname.
|
||
- **Personas.** External API consumer (allowed); anonymous internet (denied).
|
||
- **Primary workflow.** Same path three ways: no credential → 401, wrong
|
||
credential → 401, correct credential → 200 with a body.
|
||
- **Edge cases.** A 200 for the wrong bearer would be the maximum-severity
|
||
finding in this suite. A 500 on a bad credential would leak that the key was
|
||
parsed but mishandled.
|
||
- **Permission boundary.** `Authorization: Bearer` compared with
|
||
`subtle::ConstantTimeEq`.
|
||
- **Dependencies.** Secret `tidaldb-credentials`; Traefik.
|
||
- **Observability.** Status codes for all three attempts, recorded together.
|
||
- **Coverage.** `existing-green`
|
||
- **Evidence.** `tests/e2e/features/03-auth-boundary.spec.ts`
|
||
|
||
### CAP-006 — A quorum-acked write commits end to end
|
||
|
||
- **Area.** `POST /items` with `x-tidal-ack: quorum` via the ingress.
|
||
- **Business purpose.** The single strongest check available. One request
|
||
traverses DNS, TLS, Traefik, the backend TLS hop, authentication, and Raft
|
||
replication; `201` means a quorum acknowledged the write, not that one node
|
||
accepted it. This is the claim the quality-bar judge would ask for first and
|
||
the one a health endpoint cannot fake.
|
||
- **Personas.** External API consumer.
|
||
- **Primary workflow.** POST an item in the reserved `999_000_0xx` id band with
|
||
quorum ack; assert `201`.
|
||
- **Edge cases.** `202` (accepted, not quorum-acked) must fail. A timeout under
|
||
quorum loss. `503` when the leader is mid-election.
|
||
- **Permission boundary.** Requires the data bearer.
|
||
- **Dependencies.** Whole stack, plus a live leader and two reachable followers.
|
||
- **Observability.** Status, body, and the pre/post applied position.
|
||
- **Coverage.** `existing-green`
|
||
- **Evidence.** `tests/e2e/features/03-auth-boundary.spec.ts`, `tests/e2e/smoke.spec.ts`
|
||
|
||
### CAP-007 — Operator and metrics surfaces are unroutable from the internet
|
||
|
||
- **Area.** `/cluster/status`, `/cluster/members`, `/metrics`, `/openapi.json`
|
||
through the public ingress.
|
||
- **Business purpose.** `/cluster/status` leaks leader identity, membership and
|
||
sequence positions; `/metrics` leaks corpus size unauthenticated. Neither is
|
||
in the published allowlist, so both must 404 at the gateway — a defence that
|
||
holds even if a future build were to forget an auth check.
|
||
- **Personas.** Anonymous internet (denied).
|
||
- **Primary workflow.** Request each path publicly; assert 404 (gateway
|
||
rejection), not 401 (reached the app).
|
||
- **Edge cases.** A 401 here is a *finding*: it means the request reached the
|
||
application and only the credential check stopped it, so the allowlist has
|
||
drifted.
|
||
- **Permission boundary.** Traefik path allowlist in `k8s/cluster/ingress.yaml`.
|
||
- **Dependencies.** Ingress.
|
||
- **Observability.** Status per path.
|
||
- **Coverage.** `existing-green`
|
||
- **Evidence.** `tests/e2e/features/03-auth-boundary.spec.ts`
|
||
|
||
### CAP-008 — The metrics port is not reachable from arbitrary pods
|
||
|
||
- **Area.** NetworkPolicy `tidaldb`; pod `:9091` and `:9601`.
|
||
- **Business purpose.** Before the policy existed, any pod anywhere in the
|
||
cluster could read the unauthenticated metrics endpoint. The policy must deny
|
||
a foreign namespace while still admitting the scraper — a rule that blocks
|
||
both is an outage of observability, and a rule that blocks neither is
|
||
theatre.
|
||
- **Personas.** Foreign workload (denied); vmagent scraper (allowed).
|
||
- **Primary workflow.** From `threesix/gitea-0`, expect connection refused on
|
||
`:9091`. From `observability/vmagent`, expect a few hundred `tidaldb_` series.
|
||
- **Edge cases.** `:9500` is deliberately left open because all three probes
|
||
originate from the node — a wrong rule there fails liveness at 6×10s and
|
||
restarts every pod. A too-short client timeout returns zero lines and reads
|
||
exactly like "the metric is gone" (see `BUG-002`).
|
||
- **Permission boundary.** The NetworkPolicy itself is the boundary under test.
|
||
- **Dependencies.** CNI policy enforcement; the foreign pod existing.
|
||
- **Observability.** Both command transcripts, attached side by side.
|
||
- **Coverage.** `existing-green`
|
||
- **Evidence.** `tests/e2e/features/04-network-isolation.spec.ts`
|
||
|
||
### CAP-009 — Metrics reach the platform with the labels the dashboard queries
|
||
|
||
- **Area.** vmsingle `/api/v1/series`.
|
||
- **Business purpose.** The dashboard's `$namespace`/`$pod` template variables
|
||
depend on exact label names. A renamed or dropped label leaves every panel
|
||
blank while the underlying series still exists — a monitoring outage that
|
||
looks like a product outage.
|
||
- **Personas.** Operator.
|
||
- **Primary workflow.** Query series for `tidaldb_health_ok`; assert non-zero
|
||
and that `namespace`, `pod`, `container`, `partition_id` are all present.
|
||
- **Edge cases.** Stale series from deleted pods inflating the count; a label
|
||
present but empty.
|
||
- **Permission boundary.** None (in-cluster, port-forwarded).
|
||
- **Dependencies.** vmagent scrape config; vmsingle.
|
||
- **Observability.** Series count and the label set.
|
||
- **Coverage.** `existing-green`
|
||
- **Evidence.** `tests/e2e/features/05-metrics-dashboard.spec.ts`
|
||
|
||
### CAP-010 — The Grafana dashboard is loaded and its panels render
|
||
|
||
- **Area.** Grafana, dashboard uid `tidaldb-overview`, folder Databases.
|
||
- **Business purpose.** The dashboard is the operator's first stop during an
|
||
incident. "Loaded" is not enough — a loaded dashboard whose panels all read
|
||
"No data" is worse than no dashboard, because it implies the system is idle.
|
||
**This is the only genuine browser surface in the entire evidence chain.**
|
||
- **Personas.** Operator.
|
||
- **Primary workflow.** Log in, open the dashboard, wait for panels to settle,
|
||
screenshot the whole board and each populated panel, and classify every panel
|
||
as populated or empty.
|
||
- **Edge cases.** Five panels are *legitimately* empty pre-roll because they
|
||
query `tidaldb_http_*`, which the running image does not emit. That must be
|
||
asserted as expected-empty, not silently tolerated — otherwise a real future
|
||
regression to those panels is invisible.
|
||
- **Permission boundary.** Grafana admin credential; the dashboard must not be
|
||
anonymously readable.
|
||
- **Dependencies.** Grafana provisioning ConfigMap; vmsingle datasource.
|
||
- **Observability.** Panel-by-panel populated/empty classification.
|
||
- **Coverage.** `existing-green`
|
||
- **Evidence.** `tests/e2e/features/05-metrics-dashboard.spec.ts`
|
||
|
||
### CAP-011 — Container logs are readable and free of unexpected errors
|
||
|
||
- **Area.** `kubectl logs` for each pod.
|
||
- **Business purpose.** Logs are the first diagnostic surface. The deployed
|
||
image emits plain text, so `level:error` filtering in VictoriaLogs does *not*
|
||
work yet — filtering must happen at the source until the observability image
|
||
is rolled. Saying so prevents an operator concluding "no errors" from a query
|
||
that cannot match.
|
||
- **Personas.** Operator.
|
||
- **Primary workflow.** Read recent logs per pod; assert no ANSI escape
|
||
fragments; classify WARN/ERROR lines and assert the only WARNs are the known,
|
||
documented set.
|
||
- **Edge cases.** The `TIDAL_ADMIN_KEY is not set` WARN is present *and stale* —
|
||
the key was hot-loaded later and the gate is live. A boot-log WARN is not
|
||
proof of current state (see `BUG-003`).
|
||
- **Permission boundary.** None — cluster-plane read.
|
||
- **Dependencies.** kubelet log retention.
|
||
- **Observability.** The log lines themselves.
|
||
- **Coverage.** `existing-green`
|
||
- **Evidence.** `tests/e2e/features/06-logs.spec.ts`
|
||
|
||
### CAP-012 — `tidalctl` interrogates a live cluster and its exit codes gate
|
||
|
||
- **Area.** `tidalctl cluster-status`, `watch`.
|
||
- **Business purpose.** Gives an operator a live view without hand-rolling curl,
|
||
and an exit code a deploy script can branch on. `cluster-status && deploy` is
|
||
only safe if a non-converged cluster really exits non-zero.
|
||
- **Personas.** Operator; deploy automation.
|
||
- **Primary workflow.** Run `cluster-status` against a port-forwarded node with
|
||
the internal CA (`--insecure` required); assert a leader line, a region table,
|
||
a shard table. Run `watch --count` and assert one line per tick, terminating.
|
||
- **Edge cases.** The aggregated-status gap: a peer with no frontier report is
|
||
reported `applied=0` with lag derived against that zero, so a converged peer
|
||
can look 13.3M behind. `tidalctl` must print `NO REPORT`, not lag. Bad
|
||
credential → exit 2; malformed `--url` → exit 1.
|
||
- **Permission boundary.** Data bearer accepted; a wrong one must exit 2.
|
||
- **Dependencies.** Built `target/debug/tidalctl`; port-forward.
|
||
- **Observability.** stdout plus exit code.
|
||
- **Coverage.** `existing-green`
|
||
- **Evidence.** `tests/e2e/features/07-tidalctl.spec.ts`
|
||
|
||
### CAP-013 — The fleet backup completed and captured every volume
|
||
|
||
- **Area.** Velero `Backup` and `PodVolumeBackup` in `backup-system`.
|
||
- **Business purpose.** The recovery story. A single failed PodVolumeBackup
|
||
marks the whole backup `PartiallyFailed` and freezes
|
||
`velero_backup_last_successful_timestamp`, so the alert fires even though the
|
||
volumes that matter were captured.
|
||
- **Personas.** Operator.
|
||
- **Primary workflow.** Select the newest backup **carrying the fleet schedule
|
||
label**, assert `Completed`, no errors, `itemsBackedUp == totalItems`, and
|
||
every PVB `Completed`.
|
||
- **Edge cases.** Sorting all backups by timestamp selects a `restore-canary`
|
||
run (20 items, 1 volume) which would "pass" while proving nothing about the
|
||
fleet — the defect this inventory step exists to prevent (`BUG-004`).
|
||
- **Permission boundary.** None — cluster-plane read.
|
||
- **Dependencies.** Velero; the `velero-fleet-daily` schedule; node-agent.
|
||
- **Observability.** Phase, item counts, per-PVB phase histogram.
|
||
- **Coverage.** `existing-green`
|
||
- **Evidence.** `tests/e2e/features/08-backups.spec.ts`
|
||
|
||
### CAP-014 — Operator authority is separated from data-plane access
|
||
|
||
- **Area.** `POST /cluster/promote` and siblings, port-forwarded.
|
||
- **Business purpose.** Without the split, the application key that any client
|
||
holds can remove a cluster member, force a partition, or transfer a shard.
|
||
**Live on this deployment**, contrary to what the runbook said.
|
||
- **Personas.** Operator (allowed); application/data client (denied).
|
||
- **Primary workflow.** Data bearer on `/cluster/promote` → 403 (authenticated,
|
||
not authorized). Admin bearer → not 401/403.
|
||
- **Edge cases.** The boot log still WARNs that the key is unset because it was
|
||
materialized after startup and hot-loaded by the credential poller — the log
|
||
is stale, the behavior is authoritative. When no admin key exists at all the
|
||
gate degrades to previous behavior by design, so the test must distinguish
|
||
"absent" from "broken".
|
||
- **Permission boundary.** The `admin_gate` itself. Peer-callable verbs
|
||
(`/cluster/catchup`, `/join`, `/members`, `/reconcile*`) stay on the plain
|
||
bearer deliberately — gating them would break replication.
|
||
- **Dependencies.** `TIDAL_ADMIN_KEY` in `tidaldb-credentials`; the projected
|
||
secret volume; the credential poller.
|
||
- **Observability.** Both status codes, plus the presence of the mounted key file.
|
||
- **Coverage.** `existing-green`
|
||
- **Evidence.** `tests/e2e/features/09-operator-authority.spec.ts`
|
||
|
||
### CAP-015 — Inert observability features are inert for a known reason
|
||
|
||
- **Area.** `tidaldb_http_requests_total`; `JSON_LOGS`.
|
||
- **Business purpose.** A tripwire in the honest direction. These are committed
|
||
and unit-tested but absent from the running image. Asserting their *current
|
||
absence* means the day someone rolls the observability image, this test fails
|
||
and tells them to update the runbook — instead of the runbook quietly rotting,
|
||
which is precisely what happened to §9.2.
|
||
- **Personas.** Operator.
|
||
- **Primary workflow.** Scrape each pod's `:9091` with a generous timeout;
|
||
assert zero `tidaldb_http_*` series and record it as expected-for-this-image.
|
||
Assert `JSON_LOGS` is absent from the StatefulSet env.
|
||
- **Edge cases.** A short scrape timeout returns zero lines for *every* metric
|
||
and would make this test pass for the wrong reason — so it also asserts the
|
||
ordinary `tidaldb_` series are present, proving the scrape actually worked.
|
||
- **Permission boundary.** None.
|
||
- **Dependencies.** vmagent exec; the StatefulSet spec.
|
||
- **Observability.** Both counts, so "absent" is distinguishable from "unscraped".
|
||
- **Coverage.** `existing-green`
|
||
- **Evidence.** `tests/e2e/features/09-operator-authority.spec.ts`
|
||
|
||
## Intentionally excluded
|
||
|
||
| Capability | Why lower leverage | Alternate evidence | Owner / revisit |
|
||
| --- | --- | --- | --- |
|
||
| Failover under induced node loss | Deliberately destructive against the production cluster this suite verifies. Killing a voter to watch election is a game-day exercise, not a post-deploy check. | `tidal-server` cluster e2e suite (`cluster_runbook`, `cluster_reseed`) exercises election and reseed against ephemeral multi-process clusters. | Operator; revisit when a staging cluster exists. |
|
||
| Restore from the Velero backup | Restoring over live data is unacceptable; a restore drill needs an isolated target namespace. | `docs/runbooks/disaster-recovery.md` manual drill; `restore-canary-*` Backup objects prove the restore path independently. | Operator; revisit at the next DR drill. |
|
||
| Read/write throughput and latency SLA | Load generation against production would distort the very metrics the dashboard checks assert on. | `docs/ops/stress-test-*.md`, `tidal-stress`; nightly soak. | Operator; not a deploy gate. |
|
||
| VictoriaLogs LogsQL query surface | `/select/logsql/query` returns "unsupported path requested" on this build, and the deployed image emits plain text so `level:error` cannot match anyway. | `kubectl logs` filtering at source (CAP-011), documented as the current method. | Operator; revisit when the observability image is rolled. |
|
||
| Traefik rate-limit thresholds (200/400) | Proving the limit requires deliberately flooding a production ingress shared with 35 other services. | Middleware config asserted declaratively in `k8s/cluster/ingress.yaml`. | Operator; revisit with a dedicated test host. |
|
||
| The `tidaldb` namespace standalone Deployment | Live `1/1` but nothing routes to it; the fleet record calls it superseded. Verifying it would legitimise a surface that should be retired. | Noted as open drift in the runbook's closing section. | Operator; revisit by retiring it. |
|
||
|
||
## Bug log
|
||
|
||
| Bug ID | Capability | Severity | Expected | Actual | Evidence | Root cause | Fix | Verification | Status |
|
||
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||
| BUG-001 | CAP-014 | High | The runbook describes the live state of the credential split. | Runbook §9 said the split was "not active yet — requires an image roll"; it was already live and enforcing. | Harness global setup read image `m12-admin-gate-20260823` and found `TIDAL_ADMIN_KEY` in the secret; behavioural probe returned data→403, admin→200. | The runbook was written against the previously deployed image and not revisited after the roll. Documentation drift, not a product defect. | Rewrote §9: 9.2 marked LIVE with the working probe; §9.1/§9.3 kept as inert with the reason (image predates `4766f56`). | Re-read §9; `09-operator-authority.spec.ts` now asserts the live gate and would fail if it regressed. | fixed |
|
||
| BUG-002 | CAP-008, CAP-015 | Medium | A metrics scrape either returns series or fails loudly. | `wget --timeout=6` returned zero lines for `tidaldb-2`, indistinguishable from "this pod exports nothing". | Three retries at `--timeout=10` returned metrics every time; peers returned 332/344 series at 6s. | Client timeout too short for the payload on a loaded node — a measurement artifact, not a product defect. | Harness uses a generous timeout and asserts baseline `tidaldb_` series are present before concluding a specific metric is absent. Runbook §9.1 now specifies `--timeout=15` and explains why. | `04-network-isolation.spec.ts` and `09-operator-authority.spec.ts` both green with the baseline-present guard. | fixed |
|
||
| BUG-003 | CAP-011, CAP-014 | Low | The boot log reflects the current credential state. | Boot log WARNs `TIDAL_ADMIN_KEY is not set` while the gate is live and enforcing. | Pod started 05:41; WARN at 05:42; kubelet materialized the projected secret at 05:51:43; poller loaded it. | Working as designed — the credential poller exists so a key can be added without a restart. The log is a point-in-time record, not current state. | Documented in runbook §9.2 as an explicit "do not trust the boot WARN, test the behaviour" note; the harness allowlists this WARN with that reason. | `06-logs.spec.ts` allowlists it with the rationale; `09-operator-authority.spec.ts` proves the behaviour. | fixed |
|
||
| BUG-004 | CAP-013 | Medium | The backup check verifies the fleet backup. | Selecting the newest Backup by timestamp picked `restore-canary-longhorn-rwx-*` (20 items, 1 PVB) — it would report "pass" having verified nothing about the fleet. | Ran the runbook's own command as written; it returned the canary. | The selector sorted all Backups instead of filtering to the schedule that the freshness alert actually watches. | Runbook and harness both now filter on `velero.io/schedule-name=velero-fleet-daily`. | Re-ran: selects `velero-fleet-daily-20260823033025`, 3708/3708 items, 48/48 PVBs Completed. | fixed |
|