9523f6da43
The existing 32 checks prove the deployment answers -- TLS, auth, quorum commit,
convergence, isolation, dashboards, backups. Not one wrote a signal and observed
an order change, so VISION.md:17 "Ranking is not a feature. It is a primitive."
was unverified. This adds a 60-item content-feed app and five assertions that
verify the product's semantics, on a hermetic standalone node.
Added
- tests/e2e/app/: fixture contract (60 items, 4 categories, each owning one
unoccupied 100-id embedding cluster), a deep-module harness owning the whole
lifecycle behind startApp(), the product page, and an app:dev entry point.
- tidal-stress/src/bin/feed-fixture.rs: seeds the catalog and emits brute-force
ground truth, reusing recall::embedding_for rather than adding a third copy of
the corpus generator (tidal/src/db/items.rs already holds a second).
- GroundTruth::from_ids: the oracle now serves sparse id sets. build() delegates,
so there is no transient copy even at 1M, and top_k indexes positionally.
- 10-ranking-semantics.spec.ts (5 hermetic checks) and
11-ranking-integrity.spec.ts (2 cluster tripwires).
- playwright.semantics.config.ts + CAP-016 demo beat (walkthrough 82s -> 90s).
Measured, not merely green
- like: index 59 -> 0, like_boost 2.0, with no sleep between write and read.
- decay: implied half-lives 7.0007 d and 14.0014 d against a schema declaring
7 d and 14 d, recovered from a 4-second window via H = t*ln2 / -ln(v2/v1) and
compared against the schema the node actually loaded, not a hardcoded copy.
- ANN: top-10 identical to brute-force cosine on all four probes; self-distance
0.0148-0.0197 against a 0.05 tolerance.
- rank: dense 1..60 on standalone vs [1,1,1,2,2,3,4,3,4,5,6,5] on the cluster.
Three product findings, pinned and routed to @tidal-engineer
- BUG-018 (High) skip is durably accepted and query-time inert. Penalty is fully
implemented (ranking/profile.rs:227 -> executor/signal_values.rs:183, labelled
{signal}_penalty at executor/mod.rs:65) but skeleton() sets penalties: vec![]
(ranking/builtins.rs:62) and none of the 27 built-ins overrides it. So
VISION.md:187 "negative signals are equal citizens" holds for no shipped
profile. Same anti-pattern as the reseed defects and scatter_merge: a guard
present on one path, absent on its sibling.
- BUG-019 (Medium) three built-ins read signals this schema does not declare --
trending/share_velocity, hidden_gems/completion, controversial/dislike -- so
those terms are permanently 0 and trending ranks on view_velocity alone.
- BUG-020 (Low) for_you declares Scan{sort_field:"created_at"} but ignores a
created_at metadata value; an order matching neither id-asc nor
created_at-desc came back strictly id-ascending.
Two assertions therefore report a gap rather than a success, written as tripwires
whose failure message says what to do when the gap closes. The rank defect is
localised, not fixed: scatter_merge (cluster/node.rs:7542) returns a merged slice
without re-stamping rank while scores stay correctly ordered, so the fault is the
missing stamp and not the merge's sort.
Notes
- Hermetic by construction: its own config, because FullConfig.projects is not
filtered by --project and globalSetup publishes credentials into the main
process that forked workers inherit -- so a setup project cannot replace it,
and weakening globalSetup would destroy the fail-loud behaviour that is its
purpose. Verified with KUBECONFIG=/nonexistent and all E2E_* unset.
- Never touches the deployed corpus: skip is permanent: true, so seeding it into
production would be irreversible.
- The page contains no sort, no hostname and no credential; the harness proxy
injects auth server-side so no bearer reaches a browser or a capture.
- Schema comes from k8s/cluster/schema-configmap.yaml, asserted at 1536 dims;
tidal-server/config/default-schema.yaml declares 128 and would 422 every write.
Verification: 5 semantics + 34 regression + 10 demo captures green; tsc clean;
tidal-stress clippy clean under clippy::all=deny with unwrap_used=deny; 2101
tidaldb lib tests; preflight 10/10 perfect; render 90.05s/2700 frames with zero
empty boundary frames; zero orphan processes or temp dirs after teardown.
64 lines
7.6 KiB
Markdown
64 lines
7.6 KiB
Markdown
# Storyboard — tidalDB deploy verification
|
||
|
||
Ordered by the audience ladder in `audience-brief.md`, not by runbook section
|
||
order. Needs first to buy credibility, wants in the middle, the single dream beat
|
||
where attention peaks, then a recap that restates the needs as met.
|
||
|
||
Every functional beat cites a capability, a green test, a promoted capture, and
|
||
the rung it serves. A beat serving no rung is cut.
|
||
|
||
30 fps. Durations are seconds.
|
||
|
||
| Beat | Capability | Rung | Persona | Business purpose | Proof state | Test | Capture | Caption | Secs |
|
||
| --- | --- | --- | --- | --- | --- | --- | --- | --- | ---: |
|
||
| B1 opening | — | need | operator | Orientation: what this is and what it will prove | Title card | — | — | "tidalDB deploy verification. Three voters on `orchard9-k3sf`, one public endpoint, and a runbook an operator can walk. Every number that follows came from a command that ran against the live cluster." | 9 |
|
||
| B2 convergence | CAP-002 | need | operator | Quorum with fault tolerance actually exists, rather than being inferred from pod readiness | Per-node `lag=0`, no reseed, agreed leader per shard group | `01-cluster-convergence.spec.ts :: every node reports zero lag…` | `CAP-002-convergence` | "Each node asked for its own view. A pod can be Ready while its replication is stalled — that is what the reseed livelock exploited." | 7 |
|
||
| B3 boundary + write | CAP-005 CAP-006 | need | operator | The strongest single proof: full stack works AND the data plane is closed | 401, 401, 200, then 201 quorum-acked | `03-auth-boundary.spec.ts :: a quorum-acked write is committed…` | `CAP-006-quorum-write` | "Same path, three credentials. Then a write a majority of nodes acknowledged — the one claim a health endpoint cannot fake." | 8 |
|
||
| B3a feed reorder | CAP-016 | need | developer, operator | The product thesis: a signal write changes the order of the next query | Same 5-item feed before and after one `like`; the liked item goes 5 -> 1 with `like_boost 2.000` | `feed-app.demo.spec.ts :: CAP-016 a signal write reorders the feed immediately` | `CAP-016-feed-reorder` | "Everything before this proves the deployment answers. This is what it is for: one like, the same query again, and the item is first. No ETL between the write and the read — a local node on fixture data, so the claim is the mechanism, not the corpus." | 8 |
|
||
| B4 isolation | CAP-008 | need | operator | Least-privilege network access without blinding monitoring | Refused from a foreign namespace; 332 series to the scraper | `04-network-isolation.spec.ts :: a pod in an unrelated namespace is refused…` | `CAP-008-network-isolation` | "The refusal is the point. A policy that blocks everything is an outage; one that blocks nothing is theatre." | 7 |
|
||
| B5 authority | CAP-014 | want | operator | Blast radius of a leaked application key stops at data, not cluster topology | Data bearer 403, admin bearer authorised | `09-operator-authority.spec.ts :: the data-plane credential is refused…` | `CAP-014-authority` | "403, not 401 — the key is valid, it just cannot remove a cluster member." | 6 |
|
||
| B6 dashboard | CAP-010 | want | operator | The first surface opened during an incident actually shows the cluster | Cluster health OK, reseed none, 33.3K vectors, populated latency charts | `05-metrics-dashboard.spec.ts :: an operator opening the dashboard sees populated charts` | `CAP-010-dashboard` | "The board an operator opens at 3am. Health, reseed state, corpus size, and per-node latency — scoped to the cluster." | 8 |
|
||
| B7 recovery | CAP-013 | want | operator | The cluster can actually be restored and the alert is trustworthy | Completed, 3708/3708 items, 48/48 volumes | `08-backups.spec.ts :: the newest fleet-schedule backup completed…` | `CAP-013-backup` | "Selected by the schedule label the freshness alert watches — not merely the newest backup object." | 6 |
|
||
| B8 blind spot | CAP-012 | want | operator | A tool that reports what it cannot see is trustworthy about what it can | `NO REPORT` markers instead of fabricated lag; exit 2 | `07-tidalctl.spec.ts :: the aggregated-status gap is reported as NO REPORT…` | `CAP-012-tidalctl` | "It refuses to invent a number. Two healthy peers would otherwise read as 13.3 million events behind." | 7 |
|
||
| B9a inert | CAP-015 | need | operator | A verification that hides its gaps cannot be trusted about what it claims | Running image plus zero `tidaldb_http_*` families against a working baseline scrape | `09-operator-authority.spec.ts :: HTTP request metrics are absent…` | `CAP-015-inert` | "Two committed features are absent from the running image. The suite asserts that absence, so the day it changes it fails and says so." | 7 |
|
||
| **B9 the drift catch** | CAP-014 | **dream** | operator | Verification that audits its own documentation instead of drifting from it | The committed doc text, the actually-running image, and the live 403/200 probe, side by side | `deploy-verification.demo.spec.ts :: the harness corrected its own runbook` | `CAP-014-drift` | "The runbook said the credential split was 'not active yet — requires an image roll'. On its first run the harness read the live image and the live secret, and proved the gate was already enforcing. The document was wrong. The harness said so, before anyone noticed." | 10 |
|
||
| B10 recap | — | need | operator | Restate the needs as met, including the gaps | Closing card | — | — | "32 checks green against the live deployment. Two committed features are absent from the running image, and the suite asserts that absence deliberately — so the day it changes, it says so." | 7 |
|
||
|
||
**Total: 90 s** across 12 beats.
|
||
|
||
## Ladder coverage
|
||
|
||
Two-way mapping, per the audience protocol.
|
||
|
||
| Brief entry | Served by |
|
||
| --- | --- |
|
||
| Need — evidence is real, commands visible | B1, B2, B3, B3a, B4 (every capture shows the command or the surface that produced it) |
|
||
| Need — write proven committed by quorum | B3 |
|
||
| Need — every boundary shows its denial | B3 (401×2), B4 (refused), B5 (403) |
|
||
| Need — the database does the thing it exists to do | B3a (one signal write reorders the feed, immediately) |
|
||
| Need — what is not verified is stated | B9a (evidence on screen), B10 (recap) |
|
||
| Want — one command a different person can run | B1, B10 |
|
||
| Want — operator/data authority visibly enforced | B5 |
|
||
| Dream — the harness catches a stale claim in the runbook it verifies | B9 (exactly one dream beat) |
|
||
|
||
No `need` rung is unserved, and no `need`-serving beat was cut in response to an
|
||
audience-bar failure — the five weak-design failures found in the first capture
|
||
pass were fixed at the capture layer and re-audited.
|
||
|
||
## Deliberately not beats
|
||
|
||
- **A "green wall" summary of all 32 tests.** Counting tests is not evidence, and
|
||
this viewer has seen a livelocked cluster report itself healthy.
|
||
- **The three red `cluster_reseed` tests.** They belong in the written record
|
||
where the bisect evidence can be read, not in a 75-second walkthrough where a
|
||
red frame would read as a deploy failure.
|
||
- **Clicking, scrolling, or navigating.** The dashboard beat shows the board, not
|
||
the act of opening it. B3a is the one place a click happens, and even there the
|
||
frame shows the two resulting states rather than the gesture.
|
||
- **A `skip` beat.** The suite proves `skip` is durably accepted and then ignored
|
||
by all 27 built-in ranking profiles (no profile populates `penalties`), so there
|
||
is no order change to show. That finding belongs in the written record, not in a
|
||
frame that would imply a working demotion.
|
||
- **A search or vector-search beat.** Both work, but neither is the thesis, and a
|
||
75-second walkthrough that shows four surfaces shows none of them.
|