tidaldb/tests/e2e/app/fixture-contract.ts
jordan 9523f6da43 test(e2e): verify ranking semantics with a content-feed app, and route three product findings
The existing 32 checks prove the deployment answers -- TLS, auth, quorum commit,
convergence, isolation, dashboards, backups. Not one wrote a signal and observed
an order change, so VISION.md:17 "Ranking is not a feature. It is a primitive."
was unverified. This adds a 60-item content-feed app and five assertions that
verify the product's semantics, on a hermetic standalone node.

Added
- tests/e2e/app/: fixture contract (60 items, 4 categories, each owning one
  unoccupied 100-id embedding cluster), a deep-module harness owning the whole
  lifecycle behind startApp(), the product page, and an app:dev entry point.
- tidal-stress/src/bin/feed-fixture.rs: seeds the catalog and emits brute-force
  ground truth, reusing recall::embedding_for rather than adding a third copy of
  the corpus generator (tidal/src/db/items.rs already holds a second).
- GroundTruth::from_ids: the oracle now serves sparse id sets. build() delegates,
  so there is no transient copy even at 1M, and top_k indexes positionally.
- 10-ranking-semantics.spec.ts (5 hermetic checks) and
  11-ranking-integrity.spec.ts (2 cluster tripwires).
- playwright.semantics.config.ts + CAP-016 demo beat (walkthrough 82s -> 90s).

Measured, not merely green
- like: index 59 -> 0, like_boost 2.0, with no sleep between write and read.
- decay: implied half-lives 7.0007 d and 14.0014 d against a schema declaring
  7 d and 14 d, recovered from a 4-second window via H = t*ln2 / -ln(v2/v1) and
  compared against the schema the node actually loaded, not a hardcoded copy.
- ANN: top-10 identical to brute-force cosine on all four probes; self-distance
  0.0148-0.0197 against a 0.05 tolerance.
- rank: dense 1..60 on standalone vs [1,1,1,2,2,3,4,3,4,5,6,5] on the cluster.

Three product findings, pinned and routed to @tidal-engineer
- BUG-018 (High) skip is durably accepted and query-time inert. Penalty is fully
  implemented (ranking/profile.rs:227 -> executor/signal_values.rs:183, labelled
  {signal}_penalty at executor/mod.rs:65) but skeleton() sets penalties: vec![]
  (ranking/builtins.rs:62) and none of the 27 built-ins overrides it. So
  VISION.md:187 "negative signals are equal citizens" holds for no shipped
  profile. Same anti-pattern as the reseed defects and scatter_merge: a guard
  present on one path, absent on its sibling.
- BUG-019 (Medium) three built-ins read signals this schema does not declare --
  trending/share_velocity, hidden_gems/completion, controversial/dislike -- so
  those terms are permanently 0 and trending ranks on view_velocity alone.
- BUG-020 (Low) for_you declares Scan{sort_field:"created_at"} but ignores a
  created_at metadata value; an order matching neither id-asc nor
  created_at-desc came back strictly id-ascending.

Two assertions therefore report a gap rather than a success, written as tripwires
whose failure message says what to do when the gap closes. The rank defect is
localised, not fixed: scatter_merge (cluster/node.rs:7542) returns a merged slice
without re-stamping rank while scores stay correctly ordered, so the fault is the
missing stamp and not the merge's sort.

Notes
- Hermetic by construction: its own config, because FullConfig.projects is not
  filtered by --project and globalSetup publishes credentials into the main
  process that forked workers inherit -- so a setup project cannot replace it,
  and weakening globalSetup would destroy the fail-loud behaviour that is its
  purpose. Verified with KUBECONFIG=/nonexistent and all E2E_* unset.
- Never touches the deployed corpus: skip is permanent: true, so seeding it into
  production would be irreversible.
- The page contains no sort, no hostname and no credential; the harness proxy
  injects auth server-side so no bearer reaches a browser or a capture.
- Schema comes from k8s/cluster/schema-configmap.yaml, asserted at 1536 dims;
  tidal-server/config/default-schema.yaml declares 128 and would 422 every write.

Verification: 5 semantics + 34 regression + 10 demo captures green; tsc clean;
tidal-stress clippy clean under clippy::all=deny with unwrap_used=deny; 2101
tidaldb lib tests; preflight 10/10 perfect; render 90.05s/2700 frames with zero
empty boundary frames; zero orphan processes or temp dirs after teardown.
2026-08-23 22:42:02 -06:00

187 lines
6.7 KiB
TypeScript

/**
* The content-feed app's fixture contract — the single home for the catalog and
* for the shape of the ground truth the Rust fixture binary emits.
*
* Everything else imports this: the harness (to seed and to serve
* `/catalog.json`), the page (via that endpoint), and the specs (to pick probe
* items and to read the oracle). Without one contract, five consumers each
* decide independently what an item is and which ids exist.
*
* Data and types only. No I/O, no HTTP, and — deliberately — no ranking
* arithmetic: `CODING_GUIDELINES.md:88` puts scoring in a named ranking profile
* inside the database, and this app exists to demonstrate exactly that.
*/
/** The embedding width the schema declares (`k8s/cluster/schema-configmap.yaml:51`). */
export const EMBEDDING_DIM = 1536;
/**
* Items per category. 15 keeps a whole category inside one 100-id embedding
* cluster; crossing 100 would spill into the next cluster and silently destroy
* the ground-truth separation the oracle depends on.
*/
export const ITEMS_PER_CATEGORY = 15;
/**
* Each category owns ONE embedding cluster.
*
* `embedding_for` assigns cluster = id / 100 (`tidal-stress/src/recall.rs:104`),
* so a category's ids must share a 100-id band. These bands were probed against
* the live corpus and are unoccupied: the nearest existing vector sits at
* distance ~2.04 while intra-cluster neighbours sit at ~0.435. That 4.7x
* separation is what makes a brute-force top-k over just these 60 items the
* *global* top-k — no 6 GB oracle required.
*
* The `999_000_0xx` band is deliberately avoided: earlier verification probes
* already wrote items there, so its cluster is polluted.
*/
export const CATEGORIES = [
{ name: 'Field recordings', idBase: 900_000_000 },
{ name: 'Analog synthesis', idBase: 900_000_100 },
{ name: 'Choral & sacred', idBase: 900_000_200 },
{ name: 'Free jazz', idBase: 900_000_300 },
] as const;
export type CatalogItem = {
entityId: number;
title: string;
category: string;
};
/**
* Titles per category, in id order. These are the only prose a viewer reads on
* screen, so they are real and human rather than `post-1` — a bare id would make
* the recording read as a developer artifact rather than a product.
*/
const TITLES: Record<string, readonly string[]> = {
'Field recordings': [
'Harbour Ice at Thaw',
'Nightjars, Dungeness Shingle',
'Tram Depot, 04:40',
'Rain on a Zinc Roof',
'Cicadas Before the Storm',
'Understory, Monsoon Week',
'Fog Signal, Outer Channel',
'Beehive Interior, Midsummer',
'Grain Elevator, Idling',
'Salt Flats, Wind Only',
'Cathedral Steps at Dusk',
'Snowmelt Under Rock',
'Ferry Wake, North Passage',
'Sparrows in a Bus Shelter',
'Powerlines, Dry Heat',
],
'Analog synthesis': [
'Ladder Filter Sketch No. 4',
'Two Oscillators, Slight Detune',
'Ring Modulator Study',
'Tape Delay, Self-Oscillating',
'Sample and Hold Lullaby',
'Patchbay at Low Voltage',
'Sawtooth Descending',
'Envelope Follower Duet',
'Noise Source, Filtered Slowly',
'Sequencer Drift',
'Bucket Brigade Chorus',
'Sync Lead, Held Open',
'Resonance at the Edge',
'Pulse Width Breathing',
'Cold Start, Warm Bias',
],
'Choral & sacred': [
'Vespers for a Small Room',
'Antiphon in Two Voices',
'Kyrie, Winter Setting',
'Plainchant, Reconstructed',
'Nunc Dimittis at Compline',
'Motet for Eight Parts',
'Requiem Fragment, Anonymous',
'Magnificat in the Old Style',
'Litany with Drone',
'Alleluia, Second Mode',
'Lament for Holy Saturday',
'Te Deum, Village Choir',
'Canticle of the Three',
'Hymn at the Lighting of Lamps',
'Psalm 130, Unaccompanied',
],
'Free jazz': [
'Ashfall Quartet, Take 3',
'Blindfold Duet',
'Circular Breathing Suite',
'Downtown Loft, Second Set',
'Extended Technique No. 9',
'Fractured Standard',
'Glass Reeds',
'Horns Against a Wall',
'Inside the Piano',
'Junk Percussion Trio',
'Kinetic Sculpture Session',
'Long Tones, No Meter',
'Multiphonic Conversation',
'Nine Bells and a Bass',
'Overblown Ballad',
],
};
/**
* The 60-item catalog: 4 categories x 15 items, ids inside each category's band.
*
* Insertion order is category-grouped, and the feed's initial order is
* id-ascending, so the first page shows one category. That is not a layout bug:
* a freshly seeded corpus has no signals, every item ties on score, and
* `for_you`'s tie-break is the entity id. Interleaving was tried and reverted —
* the `for_you` candidate scan declares `sort_field: "created_at"`
* (`tidal/src/ranking/builtins.rs:49`) but ignores a `created_at` metadata value
* entirely (measured: an order matching neither id-ascending nor
* created_at-descending came back strictly id-ascending), so insertion order and
* timestamps have no observable effect. The page stops looking uniform the
* instant a signal is written, which is the point.
*/
export const CATALOG: readonly CatalogItem[] = CATEGORIES.flatMap(({ name, idBase }) =>
(TITLES[name] ?? []).map((title, offset) => ({
entityId: idBase + offset,
title,
category: name,
})),
);
/** Id -> item, for the id→title join the page performs (`/feed` returns no metadata). */
export const CATALOG_BY_ID: Record<string, CatalogItem> = Object.fromEntries(
CATALOG.map((item) => [String(item.entityId), item]),
);
/**
* Items the spec probes for the ANN assertion — one per category, so every
* cluster is covered rather than just the first.
*/
export const PROBE_IDS = [900_000_007, 900_000_107, 900_000_207, 900_000_307] as const;
/**
* What `tidal-stress`'s `feed-fixture` binary writes. A generated artifact, never
* committed: a checked-in oracle drifts from its generator the first time either
* one changes, and the drift is silent.
*/
export type FixtureGroundTruth = {
dim: number;
/** entityId -> its true cosine-ranked neighbour ids, nearest first. */
neighboursByItem: Record<string, number[]>;
/**
* entityId -> its raw 1536-float vector, for the handful of items the spec
* probes. The ANN assertion has to POST a query vector, and the vectors
* otherwise exist only inside the Rust binary. Emitting a few probes rather
* than all 60 keeps this artifact ~100 KB instead of ~1 MB.
*/
probeVectors: Record<string, number[]>;
/**
* How far an item's own vector may land from itself and still count as "zero".
* Non-zero purely because the vector round-trips through f32 JSON and the
* engine re-normalises on write.
*/
selfDistanceTolerance: number;
};
/** The three signal names the schema declares. Nothing else exists. */
export const SIGNALS = ['view', 'like', 'skip'] as const;
export type SignalName = (typeof SIGNALS)[number];