tidaldb/demo/capability-inventory.md
jordan 9523f6da43 test(e2e): verify ranking semantics with a content-feed app, and route three product findings
The existing 32 checks prove the deployment answers -- TLS, auth, quorum commit,
convergence, isolation, dashboards, backups. Not one wrote a signal and observed
an order change, so VISION.md:17 "Ranking is not a feature. It is a primitive."
was unverified. This adds a 60-item content-feed app and five assertions that
verify the product's semantics, on a hermetic standalone node.

Added
- tests/e2e/app/: fixture contract (60 items, 4 categories, each owning one
  unoccupied 100-id embedding cluster), a deep-module harness owning the whole
  lifecycle behind startApp(), the product page, and an app:dev entry point.
- tidal-stress/src/bin/feed-fixture.rs: seeds the catalog and emits brute-force
  ground truth, reusing recall::embedding_for rather than adding a third copy of
  the corpus generator (tidal/src/db/items.rs already holds a second).
- GroundTruth::from_ids: the oracle now serves sparse id sets. build() delegates,
  so there is no transient copy even at 1M, and top_k indexes positionally.
- 10-ranking-semantics.spec.ts (5 hermetic checks) and
  11-ranking-integrity.spec.ts (2 cluster tripwires).
- playwright.semantics.config.ts + CAP-016 demo beat (walkthrough 82s -> 90s).

Measured, not merely green
- like: index 59 -> 0, like_boost 2.0, with no sleep between write and read.
- decay: implied half-lives 7.0007 d and 14.0014 d against a schema declaring
  7 d and 14 d, recovered from a 4-second window via H = t*ln2 / -ln(v2/v1) and
  compared against the schema the node actually loaded, not a hardcoded copy.
- ANN: top-10 identical to brute-force cosine on all four probes; self-distance
  0.0148-0.0197 against a 0.05 tolerance.
- rank: dense 1..60 on standalone vs [1,1,1,2,2,3,4,3,4,5,6,5] on the cluster.

Three product findings, pinned and routed to @tidal-engineer
- BUG-018 (High) skip is durably accepted and query-time inert. Penalty is fully
  implemented (ranking/profile.rs:227 -> executor/signal_values.rs:183, labelled
  {signal}_penalty at executor/mod.rs:65) but skeleton() sets penalties: vec![]
  (ranking/builtins.rs:62) and none of the 27 built-ins overrides it. So
  VISION.md:187 "negative signals are equal citizens" holds for no shipped
  profile. Same anti-pattern as the reseed defects and scatter_merge: a guard
  present on one path, absent on its sibling.
- BUG-019 (Medium) three built-ins read signals this schema does not declare --
  trending/share_velocity, hidden_gems/completion, controversial/dislike -- so
  those terms are permanently 0 and trending ranks on view_velocity alone.
- BUG-020 (Low) for_you declares Scan{sort_field:"created_at"} but ignores a
  created_at metadata value; an order matching neither id-asc nor
  created_at-desc came back strictly id-ascending.

Two assertions therefore report a gap rather than a success, written as tripwires
whose failure message says what to do when the gap closes. The rank defect is
localised, not fixed: scatter_merge (cluster/node.rs:7542) returns a merged slice
without re-stamping rank while scores stay correctly ordered, so the fault is the
missing stamp and not the merge's sort.

Notes
- Hermetic by construction: its own config, because FullConfig.projects is not
  filtered by --project and globalSetup publishes credentials into the main
  process that forked workers inherit -- so a setup project cannot replace it,
  and weakening globalSetup would destroy the fail-loud behaviour that is its
  purpose. Verified with KUBECONFIG=/nonexistent and all E2E_* unset.
- Never touches the deployed corpus: skip is permanent: true, so seeding it into
  production would be irreversible.
- The page contains no sort, no hostname and no credential; the harness proxy
  injects auth server-side so no bearer reaches a browser or a capture.
- Schema comes from k8s/cluster/schema-configmap.yaml, asserted at 1536 dims;
  tidal-server/config/default-schema.yaml declares 128 and would 422 every write.

Verification: 5 semantics + 34 regression + 10 demo captures green; tsc clean;
tidal-stress clippy clean under clippy::all=deny with unwrap_used=deny; 2101
tidaldb lib tests; preflight 10/10 perfect; render 90.05s/2700 frames with zero
empty boundary frames; zero orphan processes or temp dirs after teardown.
2026-08-23 22:42:02 -06:00

34 KiB
Raw Blame History

Capability inventory — tidalDB deploy verification

Scope: every claim docs/runbooks/deploy-verification.md makes about the live orchard9-k3sf deployment. One row per capability (a verifiable property), not per command — several commands can serve one property, and one command can touch several.

Sources reconciled

Source What it contributed
docs/runbooks/deploy-verification.md The nine sections and their pass criteria.
tidal-server/src/router.rs The 10 registered routes; which are open vs bearer-gated vs admin-gated.
k8s/cluster/ingress.yaml The published path allowlist — what is reachable from the internet at all.
k8s/cluster/networkpolicy.yaml Ingress rules for :9091/:9601 and the deliberate :9500 exception.
k8s/cluster/statefulset.yaml Probe ports, env, resource requests (and the live-vs-manifest drift).
docs/ops/observability.md The four observability surfaces and the aggregated-status caveat.
docs/ops/grafana-tidaldb.json The 13 dashboard panels and their PromQL.
Live cluster (kubectl, curl) Actual image digest, actual secret keys, actual behavior.
k8s/cluster/schema-configmap.yaml The three declared signals and their decay, the two text fields, and the 1536-dim embedding slot.
tidal/src/ranking/builtins.rs, profile.rs The 27 built-in ranking profiles and the boost/penalty/gate/exclude mechanisms they do and do not populate.
Local standalone node (tidal-server standalone + feed-fixture) Actual ranking, decay, and ANN behaviour on a corpus with known ground truth.

Verified against. Image registry.threesix.ai/tidal/server:m12-admin-gate-20260823@sha256:6e220060…, pods started 2026-08-23 05:3205:41 UTC, namespace tidaldb-cluster, repo commit recorded per-run in E2E_BUILD_REVISION.

Environment gate discovered during inventory. The running image predates the observability commit (4766f56), so HTTP request metrics and structured logs are absent by construction, while the operator/data credential split from 388e445 is live. The runbook asserted all three were pending; that was stale. See BUG-001.

Inventory

CAP-001 — Cluster membership is complete and every voter is Ready

  • Area. kubectl -n tidaldb-cluster get pods; StatefulSet tidaldb.
  • Business purpose. Three voters is the minimum for quorum with one-node fault tolerance. Two Ready pods still serve reads and writes but tolerate no further loss — a state that looks fine and is one failure from unavailable.
  • Personas. Operator.
  • Primary workflow. List pods by label; assert every known pod is Ready and that the set is exactly the expected three.
  • Edge cases. A fourth unexpected pod (mid-scale or orphan); a pod Running but not Ready; a climbing restart count.
  • Permission boundary. None — cluster-plane read.
  • Dependencies. kubeconfig; the StatefulSet.
  • Observability. Pod ready condition, restartCount, startTime.
  • Coverage. existing-green
  • Evidence. tests/e2e/features/01-cluster-convergence.spec.ts

CAP-002 — Every node has converged: zero lag, no reseed, one agreed leader

  • Area. GET /cluster/status/local on each pod via port-forward.
  • Business purpose. The property that actually matters. A pod can be Ready while its replication is stalled or it is rebuilding from a snapshot. This is the check the 2026-08-20 reseed livelock defeated, which is why it is asked of each node individually rather than of the aggregate.
  • Personas. Operator.
  • Primary workflow. For each pod: port-forward :9500, GET /cluster/status/local with the data bearer, assert reseed_required=false and lag_events=0 on every shard group, and that all pods name the same leader.
  • Edge cases. reseed_required=true surviving a restart (livelock signature); non-zero lag on one follower; disagreeing leaders (split brain); the leader's own un-shipped tail showing as a small applied-position difference, which is not lag.
  • Permission boundary. Requires the data bearer since 388e445 moved /cluster/status* behind auth; an unauthenticated request must be refused.
  • Dependencies. port-forward; the data bearer.
  • Observability. applied_events, lag_events, leader, term, reseed_required per shard.
  • Coverage. existing-green
  • Evidence. tests/e2e/features/01-cluster-convergence.spec.ts

CAP-003 — Public DNS resolves to every node

  • Area. tidaldb.threesix.ai A records.
  • Business purpose. Three A records is the load distribution and the failover story. One missing record silently concentrates all traffic on two nodes. A retired record that monitoring still watches produces a permanent false alarm — which is exactly what happened to the previous hostname.
  • Personas. Operator; any external API consumer.
  • Primary workflow. Resolve the hostname for real and assert the address set equals the three known node IPs.
  • Edge cases. Split-DNS resolver returning a different set than public DNS; partial record set; a stale record for a retired host.
  • Permission boundary. None.
  • Dependencies. Cloudflare zone threesix.ai.
  • Observability. Resolved address list, attached per run.
  • Coverage. existing-green
  • Evidence. tests/e2e/smoke.spec.ts, tests/e2e/features/02-public-endpoint.spec.ts

CAP-004 — TLS identity is correct and not near expiry

  • Area. TLS handshake on :443 for tidaldb.threesix.ai.
  • Business purpose. A client that cannot validate the certificate either fails closed (outage) or is configured to skip validation (silent downgrade to no transport security). Expiry is the most common self-inflicted outage in this class.
  • Personas. External API consumer.
  • Primary workflow. Complete a real handshake with validation enabled; assert subject CN, a Let's Encrypt issuer, and at least 30 days of remaining validity.
  • Edge cases. Wrong SAN; self-signed fallback; cert valid but chain incomplete; expiry inside the renewal window.
  • Permission boundary. None.
  • Dependencies. cert-manager letsencrypt-prod (Cloudflare dns01), Traefik.
  • Observability. Subject, issuer, notAfter, days remaining.
  • Coverage. existing-green
  • Evidence. tests/e2e/features/02-public-endpoint.spec.ts

CAP-005 — The data plane refuses unauthenticated and wrong credentials

  • Area. GET /search (and the other published data paths) via the ingress.
  • Business purpose. This endpoint is on the open internet. The bearer is the only thing between the corpus and anyone who finds the hostname.
  • Personas. External API consumer (allowed); anonymous internet (denied).
  • Primary workflow. Same path three ways: no credential → 401, wrong credential → 401, correct credential → 200 with a body.
  • Edge cases. A 200 for the wrong bearer would be the maximum-severity finding in this suite. A 500 on a bad credential would leak that the key was parsed but mishandled.
  • Permission boundary. Authorization: Bearer compared with subtle::ConstantTimeEq.
  • Dependencies. Secret tidaldb-credentials; Traefik.
  • Observability. Status codes for all three attempts, recorded together.
  • Coverage. existing-green
  • Evidence. tests/e2e/features/03-auth-boundary.spec.ts

CAP-006 — A quorum-acked write commits end to end

  • Area. POST /items with x-tidal-ack: quorum via the ingress.
  • Business purpose. The single strongest check available. One request traverses DNS, TLS, Traefik, the backend TLS hop, authentication, and Raft replication; 201 means a quorum acknowledged the write, not that one node accepted it. This is the claim the quality-bar judge would ask for first and the one a health endpoint cannot fake.
  • Personas. External API consumer.
  • Primary workflow. POST an item in the reserved 999_000_0xx id band with quorum ack; assert 201.
  • Edge cases. 202 (accepted, not quorum-acked) must fail. A timeout under quorum loss. 503 when the leader is mid-election.
  • Permission boundary. Requires the data bearer.
  • Dependencies. Whole stack, plus a live leader and two reachable followers.
  • Observability. Status, body, and the pre/post applied position.
  • Coverage. existing-green
  • Evidence. tests/e2e/features/03-auth-boundary.spec.ts, tests/e2e/smoke.spec.ts

CAP-007 — Operator and metrics surfaces are unroutable from the internet

  • Area. /cluster/status, /cluster/members, /metrics, /openapi.json through the public ingress.
  • Business purpose. /cluster/status leaks leader identity, membership and sequence positions; /metrics leaks corpus size unauthenticated. Neither is in the published allowlist, so both must 404 at the gateway — a defence that holds even if a future build were to forget an auth check.
  • Personas. Anonymous internet (denied).
  • Primary workflow. Request each path publicly; assert 404 (gateway rejection), not 401 (reached the app).
  • Edge cases. A 401 here is a finding: it means the request reached the application and only the credential check stopped it, so the allowlist has drifted.
  • Permission boundary. Traefik path allowlist in k8s/cluster/ingress.yaml.
  • Dependencies. Ingress.
  • Observability. Status per path.
  • Coverage. existing-green
  • Evidence. tests/e2e/features/03-auth-boundary.spec.ts

CAP-008 — The metrics port is not reachable from arbitrary pods

  • Area. NetworkPolicy tidaldb; pod :9091 and :9601.
  • Business purpose. Before the policy existed, any pod anywhere in the cluster could read the unauthenticated metrics endpoint. The policy must deny a foreign namespace while still admitting the scraper — a rule that blocks both is an outage of observability, and a rule that blocks neither is theatre.
  • Personas. Foreign workload (denied); vmagent scraper (allowed).
  • Primary workflow. From threesix/gitea-0, expect connection refused on :9091. From observability/vmagent, expect a few hundred tidaldb_ series.
  • Edge cases. :9500 is deliberately left open because all three probes originate from the node — a wrong rule there fails liveness at 6×10s and restarts every pod. A too-short client timeout returns zero lines and reads exactly like "the metric is gone" (see BUG-002).
  • Permission boundary. The NetworkPolicy itself is the boundary under test.
  • Dependencies. CNI policy enforcement; the foreign pod existing.
  • Observability. Both command transcripts, attached side by side.
  • Coverage. existing-green
  • Evidence. tests/e2e/features/04-network-isolation.spec.ts

CAP-009 — Metrics reach the platform with the labels the dashboard queries

  • Area. vmsingle /api/v1/series.
  • Business purpose. The dashboard's $namespace/$pod template variables depend on exact label names. A renamed or dropped label leaves every panel blank while the underlying series still exists — a monitoring outage that looks like a product outage.
  • Personas. Operator.
  • Primary workflow. Query series for tidaldb_health_ok; assert non-zero and that namespace, pod, container, partition_id are all present.
  • Edge cases. Stale series from deleted pods inflating the count; a label present but empty.
  • Permission boundary. None (in-cluster, port-forwarded).
  • Dependencies. vmagent scrape config; vmsingle.
  • Observability. Series count and the label set.
  • Coverage. existing-green
  • Evidence. tests/e2e/features/05-metrics-dashboard.spec.ts

CAP-010 — The Grafana dashboard is loaded and its panels render

  • Area. Grafana, dashboard uid tidaldb-overview, folder Databases.
  • Business purpose. The dashboard is the operator's first stop during an incident. "Loaded" is not enough — a loaded dashboard whose panels all read "No data" is worse than no dashboard, because it implies the system is idle. This is the only genuine browser surface in the entire evidence chain.
  • Personas. Operator.
  • Primary workflow. Log in, open the dashboard, wait for panels to settle, screenshot the whole board and each populated panel, and classify every panel as populated or empty.
  • Edge cases. Five panels are legitimately empty pre-roll because they query tidaldb_http_*, which the running image does not emit. That must be asserted as expected-empty, not silently tolerated — otherwise a real future regression to those panels is invisible.
  • Permission boundary. Grafana admin credential; the dashboard must not be anonymously readable.
  • Dependencies. Grafana provisioning ConfigMap; vmsingle datasource.
  • Observability. Panel-by-panel populated/empty classification.
  • Coverage. existing-green
  • Evidence. tests/e2e/features/05-metrics-dashboard.spec.ts

CAP-011 — Container logs are readable and free of unexpected errors

  • Area. kubectl logs for each pod.
  • Business purpose. Logs are the first diagnostic surface. The deployed image emits plain text, so level:error filtering in VictoriaLogs does not work yet — filtering must happen at the source until the observability image is rolled. Saying so prevents an operator concluding "no errors" from a query that cannot match.
  • Personas. Operator.
  • Primary workflow. Read recent logs per pod; assert no ANSI escape fragments; classify WARN/ERROR lines and assert the only WARNs are the known, documented set.
  • Edge cases. The TIDAL_ADMIN_KEY is not set WARN is present and stale — the key was hot-loaded later and the gate is live. A boot-log WARN is not proof of current state (see BUG-003).
  • Permission boundary. None — cluster-plane read.
  • Dependencies. kubelet log retention.
  • Observability. The log lines themselves.
  • Coverage. existing-green
  • Evidence. tests/e2e/features/06-logs.spec.ts

CAP-012 — tidalctl interrogates a live cluster and its exit codes gate

  • Area. tidalctl cluster-status, watch.
  • Business purpose. Gives an operator a live view without hand-rolling curl, and an exit code a deploy script can branch on. cluster-status && deploy is only safe if a non-converged cluster really exits non-zero.
  • Personas. Operator; deploy automation.
  • Primary workflow. Run cluster-status against a port-forwarded node with the internal CA (--insecure required); assert a leader line, a region table, a shard table. Run watch --count and assert one line per tick, terminating.
  • Edge cases. The aggregated-status gap: a peer with no frontier report is reported applied=0 with lag derived against that zero, so a converged peer can look 13.3M behind. tidalctl must print NO REPORT, not lag. Bad credential → exit 2; malformed --url → exit 1.
  • Permission boundary. Data bearer accepted; a wrong one must exit 2.
  • Dependencies. Built target/debug/tidalctl; port-forward.
  • Observability. stdout plus exit code.
  • Coverage. existing-green
  • Evidence. tests/e2e/features/07-tidalctl.spec.ts

CAP-013 — The fleet backup completed and captured every volume

  • Area. Velero Backup and PodVolumeBackup in backup-system.
  • Business purpose. The recovery story. A single failed PodVolumeBackup marks the whole backup PartiallyFailed and freezes velero_backup_last_successful_timestamp, so the alert fires even though the volumes that matter were captured.
  • Personas. Operator.
  • Primary workflow. Select the newest backup carrying the fleet schedule label, assert Completed, no errors, itemsBackedUp == totalItems, and every PVB Completed.
  • Edge cases. Sorting all backups by timestamp selects a restore-canary run (20 items, 1 volume) which would "pass" while proving nothing about the fleet — the defect this inventory step exists to prevent (BUG-004).
  • Permission boundary. None — cluster-plane read.
  • Dependencies. Velero; the velero-fleet-daily schedule; node-agent.
  • Observability. Phase, item counts, per-PVB phase histogram.
  • Coverage. existing-green
  • Evidence. tests/e2e/features/08-backups.spec.ts

CAP-014 — Operator authority is separated from data-plane access

  • Area. POST /cluster/promote and siblings, port-forwarded.
  • Business purpose. Without the split, the application key that any client holds can remove a cluster member, force a partition, or transfer a shard. Live on this deployment, contrary to what the runbook said.
  • Personas. Operator (allowed); application/data client (denied).
  • Primary workflow. Data bearer on /cluster/promote → 403 (authenticated, not authorized). Admin bearer → not 401/403.
  • Edge cases. The boot log still WARNs that the key is unset because it was materialized after startup and hot-loaded by the credential poller — the log is stale, the behavior is authoritative. When no admin key exists at all the gate degrades to previous behavior by design, so the test must distinguish "absent" from "broken".
  • Permission boundary. The admin_gate itself. Peer-callable verbs (/cluster/catchup, /join, /members, /reconcile*) stay on the plain bearer deliberately — gating them would break replication.
  • Dependencies. TIDAL_ADMIN_KEY in tidaldb-credentials; the projected secret volume; the credential poller.
  • Observability. Both status codes, plus the presence of the mounted key file.
  • Coverage. existing-green
  • Evidence. tests/e2e/features/09-operator-authority.spec.ts

CAP-015 — Inert observability features are inert for a known reason

  • Area. tidaldb_http_requests_total; JSON_LOGS.
  • Business purpose. A tripwire in the honest direction. These are committed and unit-tested but absent from the running image. Asserting their current absence means the day someone rolls the observability image, this test fails and tells them to update the runbook — instead of the runbook quietly rotting, which is precisely what happened to §9.2.
  • Personas. Operator.
  • Primary workflow. Scrape each pod's :9091 with a generous timeout; assert zero tidaldb_http_* series and record it as expected-for-this-image. Assert JSON_LOGS is absent from the StatefulSet env.
  • Edge cases. A short scrape timeout returns zero lines for every metric and would make this test pass for the wrong reason — so it also asserts the ordinary tidaldb_ series are present, proving the scrape actually worked.
  • Permission boundary. None.
  • Dependencies. vmagent exec; the StatefulSet spec.
  • Observability. Both counts, so "absent" is distinguishable from "unscraped".
  • Coverage. existing-green
  • Evidence. tests/e2e/features/09-operator-authority.spec.ts

CAP-016 — A signal write changes the order of the next query

  • Area. POST /signals then GET /feed; the 60-item fixture catalog on a local standalone node (tests/e2e/app/).
  • Business purpose. VISION.md:17 — "Ranking is not a feature. It is a primitive." Every other capability here proves the deployment answers; none of them proves the database does the thing it exists to do. A ranking database whose signals do not reorder anything is a document store with extra steps.
  • Personas. Application developer; operator.
  • Primary workflow. Seed 60 items with deterministic 1536-dim vectors from tidal-stress's own embedding_for; read /feed?profile=for_you&limit=60; write one unit-weight like on the last-ranked item; re-read immediately with no sleep and no retry; assert the item's index decreased.
  • Edge cases. Position is asserted, never score — the profile re-normalises scores across the candidate set (measured: unliked items move 0.5 → 0.0 once one item is boosted), so a score comparison would be brittle and would drag ranking arithmetic into the test. A freshly seeded corpus has no signals, so every item ties and the order is the entity-id tie-break; that is correct, and pre-seeding engagement to make it look livelier would destroy the baseline.
  • Permission boundary. None on standalone (tidal-server/tests/standalone.rs:32). Against an authenticated target the harness proxy injects the bearer server-side, so no credential reaches the browser.
  • Dependencies. tidal-server standalone; the feed-fixture binary; the cluster's own schema, extracted from k8s/cluster/schema-configmap.yaml (dimensions: 1536) rather than tidal-server/config/default-schema.yaml, which declares 128 and would reject every embedding with a 422.
  • Observability. /feed returns each item's decayed signal values, so the assertion records why the order changed, not just that it did.
  • Coverage. new-test
  • Evidence. tests/e2e/features/10-ranking-semantics.spec.ts; tests/e2e/demo/workflows/feed-app.demo.spec.ts (capture CAP-016-feed-reorder)

CAP-017 — Decay is computed at query time from the declared half-life

  • Area. signals[].value on /feed; decay in the loaded schema.
  • Business purpose. CODING_GUIDELINES.md:88 — "Decay is a type, not a formula you call." If decay were a batch job, a value would be stale between runs; if it were applied at write time, it could not change without a write.
  • Personas. Application developer.
  • Primary workflow. Write view and like concurrently (so both carry the same elapsed time), read the feed twice separated by a known interval, and recover each half-life from the two readings via H = t·ln2 / ln(v₂/v₁). Compare against the half-life declared by the schema the node actually loaded.
  • Edge cases. The interval is the independent variable of the property under test, not a sleep that makes an assertion pass — the comment says so. The declared value is parsed from the loaded schema rather than hardcoded, so the assertion cannot degenerate into a tautology. skip is declared permanent, and permanence is the one decay claim this surface cannot show, because no built-in profile reports a skip term at all (CAP-018) — recorded, not faked.
  • Permission boundary. None.
  • Dependencies. CAP-016's fixture.
  • Observability. Both readings, the elapsed time, and both implied half-lives are attached.
  • Coverage. new-test
  • Evidence. tests/e2e/features/10-ranking-semantics.spec.ts — measured 7.0007 d and 14.0014 d against declared 7 d and 14 d, from a 4-second window.

CAP-018 — A negative signal is durably accepted and then ignored

  • Area. POST /signals skip; all seven resolvable ranking profiles.
  • Business purpose. A tripwire in the honest direction, in the same spirit as CAP-015. VISION.md:187 claims "Negative signals are equal citizens". They are not, in any shipped profile — and a walkthrough that implied otherwise would be the exact dishonesty this inventory exists to prevent.
  • Personas. Application developer; operator.
  • Primary workflow. Record the target's position under all seven profiles, write five unit-weight skips, re-read all seven, assert every position is unchanged and no profile reports a skip_penalty term.
  • Edge cases. Five writes rather than one, so the result cannot be dismissed as falling below a rounding threshold. The earlier draft of this check skipped an item that had just been liked and concluded nothing, because the like_boost dominated — the target must be signal-neutral.
  • Permission boundary. None.
  • Dependencies. CAP-016's fixture.
  • Observability. Positions before/after per profile, plus the signal-name vocabulary each profile reports — which is how the undeclared-signal finding (BUG-019) surfaced.
  • Coverage. new-test
  • Evidence. tests/e2e/features/10-ranking-semantics.spec.ts

CAP-019 — Vector search returns the true nearest neighbours

  • Area. POST /vector_search; the usearch ANN index.
  • Business purpose. An approximate index that is quietly wrong is worse than a slow exact one: recommendations degrade with no error and no alert.
  • Personas. Application developer.
  • Primary workflow. For one probe per category, POST the item's own raw vector and assert it returns itself first at ≈0 distance, then assert the whole top-k equals brute-force cosine computed by tidal-stress's GroundTruth over the same corpus, from the same generator that produced the indexed vectors.
  • Edge cases. The request field is vector, not values — a wrong name is answered 422, which reads like a malformed body. The response field is items, not matches. Each category owns an otherwise-unoccupied 100-id embedding cluster so the corpus has real neighbour structure: recall.rs:94-101 warns that uniform-random high-dimensional vectors make recall@k measure "impossible tie-breaking, not index quality".
  • Permission boundary. None.
  • Dependencies. CAP-016's fixture; GroundTruth::from_ids.
  • Observability. Returned ids, truth ids, and the self-distance per probe.
  • Coverage. new-test
  • Evidence. tests/e2e/features/10-ranking-semantics.spec.ts — 10/10 exact match on all four probes; self-distances 0.01480.0197 against a 0.05 tolerance.

CAP-020 — Rank is a dense sequence on standalone, and duplicated on the cluster

  • Area. rank on /feed and /search; scatter_merge.
  • Business purpose. rank is the field a client paginates and displays on. Duplicated ranks silently corrupt any consumer that keys on it.
  • Personas. Application developer; operator.
  • Primary workflow. Assert rank is exactly 1..n on a standalone node. Separately, assert the deployed cluster still returns duplicates, with the root cause cited and a "delete this tripwire" instruction in the message.
  • Edge cases. A limit of 12 can be answered from one shard group and would show no duplicate; the cluster check uses 12. The cluster check also asserts scores are still correctly ordered — that is what localises the fault to the missing rank stamp rather than to the merge's sort.
  • Permission boundary. Read-only against the cluster; nothing is written.
  • Dependencies. CAP-016's fixture (standalone half); the public ingress and data bearer (cluster half).
  • Observability. Both rank arrays and both score arrays are attached.
  • Coverage. new-test
  • Evidence. tests/e2e/features/10-ranking-semantics.spec.ts (dense 1..60); tests/e2e/features/11-ranking-integrity.spec.ts (cluster [1,1,1,2,2,3,4,3,4,5,6,5])

Intentionally excluded

Capability Why lower leverage Alternate evidence Owner / revisit
Failover under induced node loss Deliberately destructive against the production cluster this suite verifies. Killing a voter to watch election is a game-day exercise, not a post-deploy check. tidal-server cluster e2e suite (cluster_runbook, cluster_reseed) exercises election and reseed against ephemeral multi-process clusters. Operator; revisit when a staging cluster exists.
Restore from the Velero backup Restoring over live data is unacceptable; a restore drill needs an isolated target namespace. docs/runbooks/disaster-recovery.md manual drill; restore-canary-* Backup objects prove the restore path independently. Operator; revisit at the next DR drill.
Read/write throughput and latency SLA Load generation against production would distort the very metrics the dashboard checks assert on. docs/ops/stress-test-*.md, tidal-stress; nightly soak. Operator; not a deploy gate.
VictoriaLogs LogsQL query surface /select/logsql/query returns "unsupported path requested" on this build, and the deployed image emits plain text so level:error cannot match anyway. kubectl logs filtering at source (CAP-011), documented as the current method. Operator; revisit when the observability image is rolled.
Traefik rate-limit thresholds (200/400) Proving the limit requires deliberately flooding a production ingress shared with 35 other services. Middleware config asserted declaratively in k8s/cluster/ingress.yaml. Operator; revisit with a dedicated test host.
The tidaldb namespace standalone Deployment Live 1/1 but nothing routes to it; the fleet record calls it superseded. Verifying it would legitimise a surface that should be retired. Noted as open drift in the runbook's closing section. Operator; revisit by retiring it.

Bug log

Bug ID Capability Severity Expected Actual Evidence Root cause Fix Verification Status
BUG-001 CAP-014 High The runbook describes the live state of the credential split. Runbook §9 said the split was "not active yet — requires an image roll"; it was already live and enforcing. Harness global setup read image m12-admin-gate-20260823 and found TIDAL_ADMIN_KEY in the secret; behavioural probe returned data→403, admin→200. The runbook was written against the previously deployed image and not revisited after the roll. Documentation drift, not a product defect. Rewrote §9: 9.2 marked LIVE with the working probe; §9.1/§9.3 kept as inert with the reason (image predates 4766f56). Re-read §9; 09-operator-authority.spec.ts now asserts the live gate and would fail if it regressed. fixed
BUG-002 CAP-008, CAP-015 Medium A metrics scrape either returns series or fails loudly. wget --timeout=6 returned zero lines for tidaldb-2, indistinguishable from "this pod exports nothing". Three retries at --timeout=10 returned metrics every time; peers returned 332/344 series at 6s. Client timeout too short for the payload on a loaded node — a measurement artifact, not a product defect. Harness uses a generous timeout and asserts baseline tidaldb_ series are present before concluding a specific metric is absent. Runbook §9.1 now specifies --timeout=15 and explains why. 04-network-isolation.spec.ts and 09-operator-authority.spec.ts both green with the baseline-present guard. fixed
BUG-003 CAP-011, CAP-014 Low The boot log reflects the current credential state. Boot log WARNs TIDAL_ADMIN_KEY is not set while the gate is live and enforcing. Pod started 05:41; WARN at 05:42; kubelet materialized the projected secret at 05:51:43; poller loaded it. Working as designed — the credential poller exists so a key can be added without a restart. The log is a point-in-time record, not current state. Documented in runbook §9.2 as an explicit "do not trust the boot WARN, test the behaviour" note; the harness allowlists this WARN with that reason. 06-logs.spec.ts allowlists it with the rationale; 09-operator-authority.spec.ts proves the behaviour. fixed
BUG-004 CAP-013 Medium The backup check verifies the fleet backup. Selecting the newest Backup by timestamp picked restore-canary-longhorn-rwx-* (20 items, 1 PVB) — it would report "pass" having verified nothing about the fleet. Ran the runbook's own command as written; it returned the canary. The selector sorted all Backups instead of filtering to the schedule that the freshness alert actually watches. Runbook and harness both now filter on velero.io/schedule-name=velero-fleet-daily. Re-ran: selects velero-fleet-daily-20260823033025, 3708/3708 items, 48/48 PVBs Completed. fixed
BUG-018 CAP-018 High A declared negative signal can demote an item. Five skip writes changed nothing under any of the seven resolvable profiles, and no profile reports a skip term. 10-ranking-semantics.spec.ts records positions before/after for all seven profiles: identical. The Penalty mechanism is fully implemented (tidal/src/ranking/profile.rs:227, applied at tidal/src/ranking/executor/signal_values.rs:183, labelled {signal}_penalty at tidal/src/ranking/executor/mod.rs:65) but never populated: every built-in profile is built from skeleton(), which sets penalties: vec![] (tidal/src/ranking/builtins.rs:62), and none of the 27 overrides it. Same shape as the scatter_merge defect — a guard present on one path and absent on its sibling. Not fixed here. Adding a skip penalty to a shipped profile changes ranking behaviour for every consumer and is a product decision, not a test fix. Routed to @tidal-engineer with the inertness pinned by CAP-018. 10-ranking-semantics.spec.ts asserts the current inertness and fails with an instruction the day a profile applies a penalty. routed
BUG-019 CAP-018 Medium Every signal a profile reads is declared by the schema. Three built-in profiles read signals this schema does not declare, so those terms are permanently 0: trending reads share_velocity, hidden_gems reads completion, controversial reads dislike. Signal-name vocabulary captured per profile in skip-is-inert.json. Built-in profiles reference a richer signal set than the deployed schema declares; the engine reports the term rather than rejecting the profile, so the degradation is silent. trending effectively ranks on view_velocity alone. Not fixed here. Either the profiles should validate against the loaded schema at registration, or the schema should declare the full set — a product decision. Routed to @tidal-engineer. The vocabulary is recorded as evidence on every run, so a change is visible. routed
BUG-020 CAP-016 Low A candidate scan honours the sort field it declares. for_you declares CandidateStrategy::Scan { sort_field: "created_at" } (tidal/src/ranking/builtins.rs:49) but a created_at metadata value has no effect: an order chosen to match neither id-ascending nor created_at-descending came back strictly id-ascending. Probed four items with interleaved timestamps; observed order was id-ascending in every case, including numeric-epoch values. The scan appears to ignore the metadata string entirely and fall back to entity id. Not fixed here. Recorded so nobody "fixes" a monotonous fixture feed by writing timestamps that cannot work; the fixture contract documents it at the point of temptation. Documented in tests/e2e/app/fixture-contract.ts. routed