9523f6da43
The existing 32 checks prove the deployment answers -- TLS, auth, quorum commit,
convergence, isolation, dashboards, backups. Not one wrote a signal and observed
an order change, so VISION.md:17 "Ranking is not a feature. It is a primitive."
was unverified. This adds a 60-item content-feed app and five assertions that
verify the product's semantics, on a hermetic standalone node.
Added
- tests/e2e/app/: fixture contract (60 items, 4 categories, each owning one
unoccupied 100-id embedding cluster), a deep-module harness owning the whole
lifecycle behind startApp(), the product page, and an app:dev entry point.
- tidal-stress/src/bin/feed-fixture.rs: seeds the catalog and emits brute-force
ground truth, reusing recall::embedding_for rather than adding a third copy of
the corpus generator (tidal/src/db/items.rs already holds a second).
- GroundTruth::from_ids: the oracle now serves sparse id sets. build() delegates,
so there is no transient copy even at 1M, and top_k indexes positionally.
- 10-ranking-semantics.spec.ts (5 hermetic checks) and
11-ranking-integrity.spec.ts (2 cluster tripwires).
- playwright.semantics.config.ts + CAP-016 demo beat (walkthrough 82s -> 90s).
Measured, not merely green
- like: index 59 -> 0, like_boost 2.0, with no sleep between write and read.
- decay: implied half-lives 7.0007 d and 14.0014 d against a schema declaring
7 d and 14 d, recovered from a 4-second window via H = t*ln2 / -ln(v2/v1) and
compared against the schema the node actually loaded, not a hardcoded copy.
- ANN: top-10 identical to brute-force cosine on all four probes; self-distance
0.0148-0.0197 against a 0.05 tolerance.
- rank: dense 1..60 on standalone vs [1,1,1,2,2,3,4,3,4,5,6,5] on the cluster.
Three product findings, pinned and routed to @tidal-engineer
- BUG-018 (High) skip is durably accepted and query-time inert. Penalty is fully
implemented (ranking/profile.rs:227 -> executor/signal_values.rs:183, labelled
{signal}_penalty at executor/mod.rs:65) but skeleton() sets penalties: vec![]
(ranking/builtins.rs:62) and none of the 27 built-ins overrides it. So
VISION.md:187 "negative signals are equal citizens" holds for no shipped
profile. Same anti-pattern as the reseed defects and scatter_merge: a guard
present on one path, absent on its sibling.
- BUG-019 (Medium) three built-ins read signals this schema does not declare --
trending/share_velocity, hidden_gems/completion, controversial/dislike -- so
those terms are permanently 0 and trending ranks on view_velocity alone.
- BUG-020 (Low) for_you declares Scan{sort_field:"created_at"} but ignores a
created_at metadata value; an order matching neither id-asc nor
created_at-desc came back strictly id-ascending.
Two assertions therefore report a gap rather than a success, written as tripwires
whose failure message says what to do when the gap closes. The rank defect is
localised, not fixed: scatter_merge (cluster/node.rs:7542) returns a merged slice
without re-stamping rank while scores stay correctly ordered, so the fault is the
missing stamp and not the merge's sort.
Notes
- Hermetic by construction: its own config, because FullConfig.projects is not
filtered by --project and globalSetup publishes credentials into the main
process that forked workers inherit -- so a setup project cannot replace it,
and weakening globalSetup would destroy the fail-loud behaviour that is its
purpose. Verified with KUBECONFIG=/nonexistent and all E2E_* unset.
- Never touches the deployed corpus: skip is permanent: true, so seeding it into
production would be irreversible.
- The page contains no sort, no hostname and no credential; the harness proxy
injects auth server-side so no bearer reaches a browser or a capture.
- Schema comes from k8s/cluster/schema-configmap.yaml, asserted at 1536 dims;
tidal-server/config/default-schema.yaml declares 128 and would 422 every write.
Verification: 5 semantics + 34 regression + 10 demo captures green; tsc clean;
tidal-stress clippy clean under clippy::all=deny with unwrap_used=deny; 2101
tidaldb lib tests; preflight 10/10 perfect; render 90.05s/2700 frames with zero
empty boundary frames; zero orphan processes or temp dirs after teardown.
71 lines
2.6 KiB
TypeScript
71 lines
2.6 KiB
TypeScript
import { defineConfig, devices } from '@playwright/test';
|
|
|
|
/**
|
|
* Regression config for the deploy-verification suite.
|
|
*
|
|
* This suite verifies a LIVE deployment (`orchard9-k3sf`, namespace
|
|
* `tidaldb-cluster`) against docs/runbooks/deploy-verification.md. It is not a
|
|
* unit suite and it does not boot an application — there is no `webServer`,
|
|
* because the thing under test is already running in a cluster.
|
|
*
|
|
* Consequences of testing a live system, all deliberate:
|
|
*
|
|
* - `workers: 1`. The suite opens kubectl port-forwards and writes to a real
|
|
* quorum. Parallel workers would fight over local ports and interleave
|
|
* writes into shared state, turning a real regression into a flake.
|
|
* - `retries: 0`, even on CI. A retry here would mask exactly the kind of
|
|
* intermittent cluster fault this suite exists to catch.
|
|
* - Long timeouts. Port-forward establishment plus a quorum-acked write across
|
|
* three nodes is legitimately slower than a local HTTP call.
|
|
*
|
|
* CLI hazard: passing `--reporter=...` REPLACES this reporter array rather than
|
|
* merging with it. Never add a `test:*` script that passes `--reporter` if a
|
|
* custom reporter is introduced later.
|
|
*/
|
|
|
|
export default defineConfig({
|
|
testDir: './tests/e2e',
|
|
// The demo suite has its own config; the ranking-semantics suite has its own
|
|
// too (playwright.semantics.config.ts) because it is hermetic and must not be
|
|
// gated on this config's cluster prerequisites.
|
|
testIgnore: ['**/demo/**', '**/features/10-ranking-semantics.spec.ts'],
|
|
globalSetup: './tests/e2e/support/env-bootstrap.ts',
|
|
timeout: 120_000,
|
|
expect: { timeout: 20_000 },
|
|
forbidOnly: !!process.env.CI,
|
|
|
|
// A live-cluster check that only passes on the second attempt has told you
|
|
// something true about the cluster. Do not hide it.
|
|
retries: 0,
|
|
workers: 1,
|
|
fullyParallel: false,
|
|
|
|
reporter: process.env.CI
|
|
? [
|
|
['github'],
|
|
['html', { open: 'never', outputFolder: 'playwright-report' }],
|
|
['junit', { outputFile: 'test-results/playwright-junit.xml' }],
|
|
]
|
|
: [['list'], ['html', { open: 'never', outputFolder: 'playwright-report' }]],
|
|
|
|
outputDir: 'test-results/playwright',
|
|
|
|
use: {
|
|
// No global baseURL: this suite deliberately talks to several distinct
|
|
// origins (public ingress, port-forwarded pod, Grafana, Alertmanager) and a
|
|
// default would make which-origin bugs invisible.
|
|
actionTimeout: 20_000,
|
|
navigationTimeout: 45_000,
|
|
trace: 'retain-on-failure',
|
|
screenshot: 'only-on-failure',
|
|
video: 'retain-on-failure',
|
|
},
|
|
|
|
projects: [
|
|
{
|
|
name: 'verification',
|
|
use: { ...devices['Desktop Chrome'] },
|
|
},
|
|
],
|
|
});
|