tidaldb/demo/capability-inventory.md

# Capability inventory — tidalDB deploy verification

Scope: every claim `docs/runbooks/deploy-verification.md` makes about the live
`orchard9-k3sf` deployment. One row per *capability* (a verifiable property),
not per command — several commands can serve one property, and one command can
touch several.

## Sources reconciled

| Source | What it contributed |
| --- | --- |
| `docs/runbooks/deploy-verification.md` | The nine sections and their pass criteria. |
| `tidal-server/src/router.rs` | The 10 registered routes; which are open vs bearer-gated vs admin-gated. |
| `k8s/cluster/ingress.yaml` | The published path allowlist — what is reachable from the internet at all. |
| `k8s/cluster/networkpolicy.yaml` | Ingress rules for `:9091`/`:9601` and the deliberate `:9500` exception. |
| `k8s/cluster/statefulset.yaml` | Probe ports, env, resource requests (and the live-vs-manifest drift). |
| `docs/ops/observability.md` | The four observability surfaces and the aggregated-status caveat. |
| `docs/ops/grafana-tidaldb.json` | The 13 dashboard panels and their PromQL. |
| Live cluster (`kubectl`, `curl`) | Actual image digest, actual secret keys, actual behavior. |
| `k8s/cluster/schema-configmap.yaml` | The three declared signals and their decay, the two text fields, and the 1536-dim embedding slot. |
| `tidal/src/ranking/builtins.rs`, `profile.rs` | The 27 built-in ranking profiles and the boost/penalty/gate/exclude mechanisms they do and do not populate. |
| Local standalone node (`tidal-server standalone` + `feed-fixture`) | Actual ranking, decay, and ANN behaviour on a corpus with known ground truth. |

**Verified against.** Image
`registry.threesix.ai/tidal/server:m12-admin-gate-20260823@sha256:6e220060…`,
pods started 2026-08-23 05:32–05:41 UTC, namespace `tidaldb-cluster`,
repo commit recorded per-run in `E2E_BUILD_REVISION`.

**Environment gate discovered during inventory — CORRECTED 2026-08-31.** As
written on 2026-08-23 this read: *the running image predates the observability
commit (`4766f56`), so HTTP request metrics and structured logs are absent by
construction.* That was true of the image this inventory was taken against and
**is false now.** Both capabilities from `4766f56` are in the running image:

| rev | image | live from | carries `4766f56`? |
|---|---|---|---|
| 29 | `m12-admin-gate-20260823` | 2026-08-23T05:32Z | no — the image this inventory was taken against |
| 30 | `m12-vector-grow-20260830` | 2026-08-30T17:38Z | **yes** — `7c1c80d` descends from `4766f56` |
| 31 | `m12-vsc-20260830` | 2026-08-30T19:59Z | **yes** — `8aa1fbb` |

```bash
kubectl -n tidaldb-cluster get controllerrevision -l app.kubernetes.io/name=tidaldb \
  -o custom-columns=REV:.revision,WHEN:.metadata.creationTimestamp,IMAGE:'.data.spec.template.spec.containers[0].image'
git merge-base --is-ancestor 4766f56 8aa1fbb   # succeeds
```

Current state of the three:

- **Operator/data credential split** (`388e445`) — live since 2026-08-23, as
  originally recorded.
- **HTTP request metrics** — **live.** Measured 2026-08-31T02:19Z:
  `tidaldb_http_requests_total` present on all three pods, 19/24/18 series
  against `tidaldb_*` baselines of 821/878/812.
- **Structured logs** — in the image since rev 30. The StatefulSet did not ask
  for them: `JSON_LOGS` was added to `k8s/cluster/statefulset.yaml` on
  2026-08-31 and an env change takes effect only on the next pod restart. The
  restart is the sole outstanding item; no code and no image roll is needed.

See `BUG-001` for the original finding and `docs/runbooks/deploy-verification.md`
§9.1/§9.3 for live state.

## Inventory

### CAP-001 — Cluster membership is complete and every voter is Ready

- **Area.** `kubectl -n tidaldb-cluster get pods`; StatefulSet `tidaldb`.
- **Business purpose.** Three voters is the minimum for quorum with one-node
  fault tolerance. Two Ready pods still serve reads and writes but tolerate no
  further loss — a state that looks fine and is one failure from unavailable.
- **Personas.** Operator.
- **Primary workflow.** List pods by label; assert every known pod is Ready and
  that the set is exactly the expected three.
- **Edge cases.** A fourth unexpected pod (mid-scale or orphan); a pod Running
  but not Ready; a climbing restart count.
- **Permission boundary.** None — cluster-plane read.
- **Dependencies.** kubeconfig; the StatefulSet.
- **Observability.** Pod `ready` condition, `restartCount`, `startTime`.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/01-cluster-convergence.spec.ts`

### CAP-002 — Every node has converged: zero lag, no reseed, one agreed leader

- **Area.** `GET /cluster/status/local` on each pod via port-forward.
- **Business purpose.** The property that actually matters. A pod can be Ready
  while its replication is stalled or it is rebuilding from a snapshot. This is
  the check the 2026-08-20 reseed livelock defeated, which is why it is asked of
  each node individually rather than of the aggregate.
- **Personas.** Operator.
- **Primary workflow.** For each pod: port-forward `:9500`, GET
  `/cluster/status/local` with the data bearer, assert `reseed_required=false`
  and `lag_events=0` on every shard group, and that all pods name the same
  leader.
- **Edge cases.** `reseed_required=true` surviving a restart (livelock
  signature); non-zero lag on one follower; disagreeing leaders (split brain);
  the leader's own un-shipped tail showing as a small applied-position
  difference, which is *not* lag.
- **Permission boundary.** Requires the data bearer since `388e445` moved
  `/cluster/status*` behind auth; an unauthenticated request must be refused.
- **Dependencies.** port-forward; the data bearer.
- **Observability.** `applied_events`, `lag_events`, `leader`, `term`,
  `reseed_required` per shard.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/01-cluster-convergence.spec.ts`

### CAP-003 — Public DNS resolves to every node

- **Area.** `tidaldb.threesix.ai` A records.
- **Business purpose.** Three A records is the load distribution and the
  failover story. One missing record silently concentrates all traffic on two
  nodes. A *retired* record that monitoring still watches produces a permanent
  false alarm — which is exactly what happened to the previous hostname.
- **Personas.** Operator; any external API consumer.
- **Primary workflow.** Resolve the hostname for real and assert the address set
  equals the three known node IPs.
- **Edge cases.** Split-DNS resolver returning a different set than public DNS;
  partial record set; a stale record for a retired host.
- **Permission boundary.** None.
- **Dependencies.** Cloudflare zone `threesix.ai`.
- **Observability.** Resolved address list, attached per run.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/smoke.spec.ts`, `tests/e2e/features/02-public-endpoint.spec.ts`

### CAP-004 — TLS identity is correct and not near expiry

- **Area.** TLS handshake on `:443` for `tidaldb.threesix.ai`.
- **Business purpose.** A client that cannot validate the certificate either
  fails closed (outage) or is configured to skip validation (silent downgrade to
  no transport security). Expiry is the most common self-inflicted outage in this
  class.
- **Personas.** External API consumer.
- **Primary workflow.** Complete a real handshake with validation enabled;
  assert subject CN, a Let's Encrypt issuer, and at least 30 days of remaining
  validity.
- **Edge cases.** Wrong SAN; self-signed fallback; cert valid but chain
  incomplete; expiry inside the renewal window.
- **Permission boundary.** None.
- **Dependencies.** cert-manager `letsencrypt-prod` (Cloudflare dns01), Traefik.
- **Observability.** Subject, issuer, `notAfter`, days remaining.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/02-public-endpoint.spec.ts`

### CAP-005 — The data plane refuses unauthenticated and wrong credentials

- **Area.** `GET /search` (and the other published data paths) via the ingress.
- **Business purpose.** This endpoint is on the open internet. The bearer is the
  only thing between the corpus and anyone who finds the hostname.
- **Personas.** External API consumer (allowed); anonymous internet (denied).
- **Primary workflow.** Same path three ways: no credential → 401, wrong
  credential → 401, correct credential → 200 with a body.
- **Edge cases.** A 200 for the wrong bearer would be the maximum-severity
  finding in this suite. A 500 on a bad credential would leak that the key was
  parsed but mishandled.
- **Permission boundary.** `Authorization: Bearer` compared with
  `subtle::ConstantTimeEq`.
- **Dependencies.** Secret `tidaldb-credentials`; Traefik.
- **Observability.** Status codes for all three attempts, recorded together.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/03-auth-boundary.spec.ts`

### CAP-006 — A quorum-acked write commits end to end

- **Area.** `POST /items` with `x-tidal-ack: quorum` via the ingress.
- **Business purpose.** The single strongest check available. One request
  traverses DNS, TLS, Traefik, the backend TLS hop, authentication, and Raft
  replication; `201` means a quorum acknowledged the write, not that one node
  accepted it. This is the claim the quality-bar judge would ask for first and
  the one a health endpoint cannot fake.
- **Personas.** External API consumer.
- **Primary workflow.** POST an item in the reserved `999_000_0xx` id band with
  quorum ack; assert `201`.
- **Edge cases.** `202` (accepted, not quorum-acked) must fail. A timeout under
  quorum loss. `503` when the leader is mid-election.
- **Permission boundary.** Requires the data bearer.
- **Dependencies.** Whole stack, plus a live leader and two reachable followers.
- **Observability.** Status, body, and the pre/post applied position.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/03-auth-boundary.spec.ts`, `tests/e2e/smoke.spec.ts`

### CAP-007 — Operator and metrics surfaces are unroutable from the internet

- **Area.** `/cluster/status`, `/cluster/members`, `/metrics`, `/openapi.json`
  through the public ingress.
- **Business purpose.** `/cluster/status` leaks leader identity, membership and
  sequence positions; `/metrics` leaks corpus size unauthenticated. Neither is
  in the published allowlist, so both must 404 at the gateway — a defence that
  holds even if a future build were to forget an auth check.
- **Personas.** Anonymous internet (denied).
- **Primary workflow.** Request each path publicly; assert 404 (gateway
  rejection), not 401 (reached the app).
- **Edge cases.** A 401 here is a *finding*: it means the request reached the
  application and only the credential check stopped it, so the allowlist has
  drifted.
- **Permission boundary.** Traefik path allowlist in `k8s/cluster/ingress.yaml`.
- **Dependencies.** Ingress.
- **Observability.** Status per path.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/03-auth-boundary.spec.ts`

### CAP-008 — The metrics port is not reachable from arbitrary pods

- **Area.** NetworkPolicy `tidaldb`; pod `:9091` and `:9601`.
- **Business purpose.** Before the policy existed, any pod anywhere in the
  cluster could read the unauthenticated metrics endpoint. The policy must deny
  a foreign namespace while still admitting the scraper — a rule that blocks
  both is an outage of observability, and a rule that blocks neither is
  theatre.
- **Personas.** Foreign workload (denied); vmagent scraper (allowed).
- **Primary workflow.** From `threesix/gitea-0`, expect connection refused on
  `:9091`. From `observability/vmagent`, expect a few hundred `tidaldb_` series.
- **Edge cases.** `:9500` is deliberately left open because all three probes
  originate from the node — a wrong rule there fails liveness at 6×10s and
  restarts every pod. A too-short client timeout returns zero lines and reads
  exactly like "the metric is gone" (see `BUG-002`).
- **Permission boundary.** The NetworkPolicy itself is the boundary under test.
- **Dependencies.** CNI policy enforcement; the foreign pod existing.
- **Observability.** Both command transcripts, attached side by side.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/04-network-isolation.spec.ts`

### CAP-009 — Metrics reach the platform with the labels the dashboard queries

- **Area.** vmsingle `/api/v1/series`.
- **Business purpose.** The dashboard's `$namespace`/`$pod` template variables
  depend on exact label names. A renamed or dropped label leaves every panel
  blank while the underlying series still exists — a monitoring outage that
  looks like a product outage.
- **Personas.** Operator.
- **Primary workflow.** Query series for `tidaldb_health_ok`; assert non-zero
  and that `namespace`, `pod`, `container`, `partition_id` are all present.
- **Edge cases.** Stale series from deleted pods inflating the count; a label
  present but empty.
- **Permission boundary.** None (in-cluster, port-forwarded).
- **Dependencies.** vmagent scrape config; vmsingle.
- **Observability.** Series count and the label set.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/05-metrics-dashboard.spec.ts`

### CAP-010 — The Grafana dashboard is loaded and its panels render

- **Area.** Grafana, dashboard uid `tidaldb-overview`, folder Databases.
- **Business purpose.** The dashboard is the operator's first stop during an
  incident. "Loaded" is not enough — a loaded dashboard whose panels all read
  "No data" is worse than no dashboard, because it implies the system is idle.
  **This is the only genuine browser surface in the entire evidence chain.**
- **Personas.** Operator.
- **Primary workflow.** Log in, open the dashboard, wait for panels to settle,
  screenshot the whole board and each populated panel, and classify every panel
  as populated or empty.
- **Edge cases.** Five panels are *legitimately* empty pre-roll because they
  query `tidaldb_http_*`, which the running image does not emit. That must be
  asserted as expected-empty, not silently tolerated — otherwise a real future
  regression to those panels is invisible.
- **Permission boundary.** Grafana admin credential; the dashboard must not be
  anonymously readable.
- **Dependencies.** Grafana provisioning ConfigMap; vmsingle datasource.
- **Observability.** Panel-by-panel populated/empty classification.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/05-metrics-dashboard.spec.ts`

### CAP-011 — Container logs are readable and free of unexpected errors

- **Area.** `kubectl logs` for each pod.
- **Business purpose.** Logs are the first diagnostic surface. The deployed
  image emits plain text, so `level:error` filtering in VictoriaLogs does *not*
  work yet — filtering must happen at the source until the observability image
  is rolled. Saying so prevents an operator concluding "no errors" from a query
  that cannot match.
- **Personas.** Operator.
- **Primary workflow.** Read recent logs per pod; assert no ANSI escape
  fragments; classify WARN/ERROR lines and assert the only WARNs are the known,
  documented set.
- **Edge cases.** The `TIDAL_ADMIN_KEY is not set` WARN is present *and stale* —
  the key was hot-loaded later and the gate is live. A boot-log WARN is not
  proof of current state (see `BUG-003`).
- **Permission boundary.** None — cluster-plane read.
- **Dependencies.** kubelet log retention.
- **Observability.** The log lines themselves.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/06-logs.spec.ts`

### CAP-012 — `tidalctl` interrogates a live cluster and its exit codes gate

- **Area.** `tidalctl cluster-status`, `watch`.
- **Business purpose.** Gives an operator a live view without hand-rolling curl,
  and an exit code a deploy script can branch on. `cluster-status && deploy` is
  only safe if a non-converged cluster really exits non-zero.
- **Personas.** Operator; deploy automation.
- **Primary workflow.** Run `cluster-status` against a port-forwarded node with
  the internal CA (`--insecure` required); assert a leader line, a region table,
  a shard table. Run `watch --count` and assert one line per tick, terminating.
- **Edge cases.** The aggregated-status gap: a peer with no frontier report is
  reported `applied=0` with lag derived against that zero, so a converged peer
  can look 13.3M behind. `tidalctl` must print `NO REPORT`, not lag. Bad
  credential → exit 2; malformed `--url` → exit 1.
- **Permission boundary.** Data bearer accepted; a wrong one must exit 2.
- **Dependencies.** Built `target/debug/tidalctl`; port-forward.
- **Observability.** stdout plus exit code.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/07-tidalctl.spec.ts`

### CAP-013 — The fleet backup completed and captured every volume

- **Area.** Velero `Backup` and `PodVolumeBackup` in `backup-system`.
- **Business purpose.** The recovery story. A single failed PodVolumeBackup
  marks the whole backup `PartiallyFailed` and freezes
  `velero_backup_last_successful_timestamp`, so the alert fires even though the
  volumes that matter were captured.
- **Personas.** Operator.
- **Primary workflow.** Select the newest backup **carrying the fleet schedule
  label**, assert `Completed`, no errors, `itemsBackedUp == totalItems`, and
  every PVB `Completed`.
- **Edge cases.** Sorting all backups by timestamp selects a `restore-canary`
  run (20 items, 1 volume) which would "pass" while proving nothing about the
  fleet — the defect this inventory step exists to prevent (`BUG-004`).
- **Permission boundary.** None — cluster-plane read.
- **Dependencies.** Velero; the `velero-fleet-daily` schedule; node-agent.
- **Observability.** Phase, item counts, per-PVB phase histogram.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/08-backups.spec.ts`

### CAP-014 — Operator authority is separated from data-plane access

- **Area.** `POST /cluster/promote` and siblings, port-forwarded.
- **Business purpose.** Without the split, the application key that any client
  holds can remove a cluster member, force a partition, or transfer a shard.
  **Live on this deployment**, contrary to what the runbook said.
- **Personas.** Operator (allowed); application/data client (denied).
- **Primary workflow.** Data bearer on `/cluster/promote` → 403 (authenticated,
  not authorized). Admin bearer → not 401/403.
- **Edge cases.** The boot log still WARNs that the key is unset because it was
  materialized after startup and hot-loaded by the credential poller — the log
  is stale, the behavior is authoritative. When no admin key exists at all the
  gate degrades to previous behavior by design, so the test must distinguish
  "absent" from "broken".
- **Permission boundary.** The `admin_gate` itself. Peer-callable verbs
  (`/cluster/catchup`, `/join`, `/members`, `/reconcile*`) stay on the plain
  bearer deliberately — gating them would break replication.
- **Dependencies.** `TIDAL_ADMIN_KEY` in `tidaldb-credentials`; the projected
  secret volume; the credential poller.
- **Observability.** Both status codes, plus the presence of the mounted key file.
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/09-operator-authority.spec.ts`

### CAP-015 — Observability features are asserted at their real current state

> **Corrected 2026-08-31.** This entry described both features as "absent from
> the running image". Half of that is now false and the other half was never an
> image problem. Title and bullets updated; the spec itself was already inverted
> for the metrics half on 2026-08-30.

- **Area.** `tidaldb_http_requests_total`; `JSON_LOGS`.
- **Business purpose.** A tripwire in the honest direction. Pinning what is
  *currently* true means the day reality moves, the test fails and names the
  runbook section to update — instead of the runbook quietly rotting, which is
  precisely what happened to §9.2, then to §9.1, then to §9.3.
- **Personas.** Operator.
- **Primary workflow.** Scrape each pod's `:9091` with a generous timeout and
  assert `tidaldb_http_*` series are **present** (inverted 2026-08-30 when the
  metrics went live). Assert `JSON_LOGS` is **absent from the live StatefulSet**
  — still true, and still the correct assertion until the pods restart with the
  env var now in the manifest, at which point it inverts too.
- **Edge cases.** A short scrape timeout returns zero lines for *every* metric
  and would make this test pass for the wrong reason — so it also asserts the
  ordinary `tidaldb_` series are present (>100), proving the scrape actually
  worked.
- **Permission boundary.** None.
- **Dependencies.** vmagent exec; the StatefulSet spec.
- **Observability.** Both counts, so "absent" is distinguishable from "unscraped".
- **Coverage.** `existing-green`
- **Evidence.** `tests/e2e/features/09-operator-authority.spec.ts`

### CAP-016 — A signal write changes the order of the next query

- **Area.** `POST /signals` then `GET /feed`; the 60-item fixture catalog on a
  local standalone node (`tests/e2e/app/`).
- **Business purpose.** `VISION.md:17` — "Ranking is not a feature. It is a
  primitive." Every other capability here proves the *deployment* answers; none
  of them proves the database does the thing it exists to do. A ranking database
  whose signals do not reorder anything is a document store with extra steps.
- **Personas.** Application developer; operator.
- **Primary workflow.** Seed 60 items with deterministic 1536-dim vectors from
  `tidal-stress`'s own `embedding_for`; read `/feed?profile=for_you&limit=60`;
  write one unit-weight `like` on the last-ranked item; re-read immediately with
  no sleep and no retry; assert the item's index decreased.
- **Edge cases.** Position is asserted, never score — the profile re-normalises
  scores across the candidate set (measured: unliked items move 0.5 → 0.0 once
  one item is boosted), so a score comparison would be brittle *and* would drag
  ranking arithmetic into the test. A freshly seeded corpus has no signals, so
  every item ties and the order is the entity-id tie-break; that is correct, and
  pre-seeding engagement to make it look livelier would destroy the baseline.
- **Permission boundary.** None on standalone (`tidal-server/tests/standalone.rs:32`).
  Against an authenticated target the harness proxy injects the bearer
  server-side, so no credential reaches the browser.
- **Dependencies.** `tidal-server standalone`; the `feed-fixture` binary; the
  cluster's own schema, extracted from `k8s/cluster/schema-configmap.yaml`
  (`dimensions: 1536`) rather than `tidal-server/config/default-schema.yaml`,
  which declares 128 and would reject every embedding with a 422.
- **Observability.** `/feed` returns each item's decayed signal values, so the
  assertion records *why* the order changed, not just that it did.
- **Coverage.** `new-test`
- **Evidence.** `tests/e2e/features/10-ranking-semantics.spec.ts`;
  `tests/e2e/demo/workflows/feed-app.demo.spec.ts` (capture `CAP-016-feed-reorder`)

### CAP-017 — Decay is computed at query time from the declared half-life

- **Area.** `signals[].value` on `/feed`; `decay` in the loaded schema.
- **Business purpose.** `CODING_GUIDELINES.md:88` — "Decay is a type, not a
  formula you call." If decay were a batch job, a value would be stale between
  runs; if it were applied at write time, it could not change without a write.
- **Personas.** Application developer.
- **Primary workflow.** Write `view` and `like` concurrently (so both carry the
  same elapsed time), read the feed twice separated by a known interval, and
  recover each half-life from the two readings via `H = t·ln2 / −ln(v₂/v₁)`.
  Compare against the half-life declared by the schema the node actually loaded.
- **Edge cases.** The interval is the independent variable of the property under
  test, not a sleep that makes an assertion pass — the comment says so. The
  declared value is parsed from the loaded schema rather than hardcoded, so the
  assertion cannot degenerate into a tautology. `skip` is declared permanent, and
  permanence is the one decay claim this surface *cannot* show, because no
  built-in profile reports a skip term at all (CAP-018) — recorded, not faked.
- **Permission boundary.** None.
- **Dependencies.** CAP-016's fixture.
- **Observability.** Both readings, the elapsed time, and both implied
  half-lives are attached.
- **Coverage.** `new-test`
- **Evidence.** `tests/e2e/features/10-ranking-semantics.spec.ts` — measured
  7.0007 d and 14.0014 d against declared 7 d and 14 d, from a 4-second window.

### CAP-018 — A negative signal is durably accepted and then ignored

- **Area.** `POST /signals` `skip`; all seven resolvable ranking profiles.
- **Business purpose.** A tripwire in the honest direction, in the same spirit as
  CAP-015. `VISION.md:187` claims "Negative signals are equal citizens". They are
  not, in any shipped profile — and a walkthrough that implied otherwise would be
  the exact dishonesty this inventory exists to prevent.
- **Personas.** Application developer; operator.
- **Primary workflow.** Record the target's position under all seven profiles,
  write five unit-weight `skip`s, re-read all seven, assert every position is
  unchanged and no profile reports a `skip_penalty` term.
- **Edge cases.** Five writes rather than one, so the result cannot be dismissed
  as falling below a rounding threshold. The earlier draft of this check skipped
  an item that had just been *liked* and concluded nothing, because the
  `like_boost` dominated — the target must be signal-neutral.
- **Permission boundary.** None.
- **Dependencies.** CAP-016's fixture.
- **Observability.** Positions before/after per profile, plus the signal-name
  vocabulary each profile reports — which is how the undeclared-signal finding
  (BUG-019) surfaced.
- **Coverage.** `new-test`
- **Evidence.** `tests/e2e/features/10-ranking-semantics.spec.ts`

### CAP-019 — Vector search returns the true nearest neighbours

- **Area.** `POST /vector_search`; the usearch ANN index.
- **Business purpose.** An approximate index that is quietly *wrong* is worse
  than a slow exact one: recommendations degrade with no error and no alert.
- **Personas.** Application developer.
- **Primary workflow.** For one probe per category, POST the item's own raw
  vector and assert it returns itself first at ≈0 distance, then assert the whole
  top-k equals brute-force cosine computed by `tidal-stress`'s `GroundTruth`
  over the same corpus, from the same generator that produced the indexed vectors.
- **Edge cases.** The request field is `vector`, not `values` — a wrong name is
  answered 422, which reads like a malformed body. The response field is `items`,
  not `matches`. Each category owns an otherwise-unoccupied 100-id embedding
  cluster so the corpus has real neighbour structure: `recall.rs:94-101` warns
  that uniform-random high-dimensional vectors make recall@k measure "impossible
  tie-breaking, not index quality".
- **Permission boundary.** None.
- **Dependencies.** CAP-016's fixture; `GroundTruth::from_ids`.
- **Observability.** Returned ids, truth ids, and the self-distance per probe.
- **Coverage.** `new-test`
- **Evidence.** `tests/e2e/features/10-ranking-semantics.spec.ts` — 10/10 exact
  match on all four probes; self-distances 0.0148–0.0197 against a 0.05 tolerance.

### CAP-020 — Rank is a dense sequence on standalone AND on the cluster

- **Area.** `rank` on `/feed` and `/search`; `scatter_merge`.
- **Business purpose.** `rank` is the field a client paginates and displays on.
  Duplicated ranks silently corrupt any consumer that keys on it.
- **Personas.** Application developer; operator.
- **Primary workflow.** Assert `rank` is exactly `1..n` on a standalone node, on
  both cluster merge shapes, and on the deployed cluster.
- **Edge cases.** A limit of 1–2 can be answered from one shard group and would
  show no duplicate; the cluster checks use 12. They also assert scores are still
  correctly ordered — the stamp is a renumbering, never a re-sort, so an ordering
  change would be a different and worse defect. The single-group `[only]` fast
  path bypasses the merge entirely and is asserted separately rather than assumed.
- **Permission boundary.** Read-only against the cluster; nothing is written.
- **Dependencies.** CAP-016's fixture (standalone half); the public ingress and
  data bearer (deployed-cluster half).
- **Observability.** Both rank arrays and both score arrays are attached.
- **Coverage.** `new-test`
- **Evidence.** `tests/e2e/features/10-ranking-semantics.spec.ts` (engine, dense
  1..60); `tidal-server/tests/cluster_sharding.rs`
  `mp_ranked_reads_stamp_dense_rank_on_both_merge_shapes` (dense 1..12 on `/feed`
  and `/search`, all 3 nodes, at 3 groups AND 1 group);
  `tests/e2e/features/11-ranking-integrity.spec.ts` (deployed cluster).

  **History.** This capability was first recorded as *"dense on standalone, and
  duplicated on the cluster"*: under full placement `/search` returned
  `[1,1,1,2,2,3,4,3,4,5,6,5]` because `scatter_merge` sorted and truncated the
  concatenated per-group slices without re-stamping `rank`. `scatter_merge` now
  takes the same `set_rank` closure as `merge_cross_shard`, and the tripwires that
  pinned the defect were replaced by the positive assertions above.

## Intentionally excluded

| Capability | Why lower leverage | Alternate evidence | Owner / revisit |
| --- | --- | --- | --- |
| Failover under induced node loss | Deliberately destructive against the production cluster this suite verifies. Killing a voter to watch election is a game-day exercise, not a post-deploy check. | `tidal-server` cluster e2e suite (`cluster_runbook`, `cluster_reseed`) exercises election and reseed against ephemeral multi-process clusters. | Operator; revisit when a staging cluster exists. |
| Restore from the Velero backup | Restoring over live data is unacceptable; a restore drill needs an isolated target namespace. | `docs/runbooks/disaster-recovery.md` manual drill; `restore-canary-*` Backup objects prove the restore path independently. | Operator; revisit at the next DR drill. |
| Read/write throughput and latency SLA | Load generation against production would distort the very metrics the dashboard checks assert on. | `docs/ops/stress-test-*.md`, `tidal-stress`; nightly soak. | Operator; not a deploy gate. |
| VictoriaLogs LogsQL query surface | **Justification corrected 2026-08-31 — the original was factually wrong.** It claimed `/select/logsql/query` returns "unsupported path requested" on this build; that path answers normally (`_time:15m AND unit:tidaldb-cluster \| stats count() n` → `{"n":"248"}`). The real reason to defer is that tidalDB's lines are not classified yet: pre-`JSON_LOGS` every line is stamped `level:info`, so a query surface test would assert a shape that is about to change. | `kubectl logs` filtering at source (CAP-011); `docs/ops/observability.md` §2 documents the working queries and their pre/post-roll meaning. | Operator; revisit immediately after the `JSON_LOGS` restart, not "when the image is rolled" — the image is already rolled. |
| Traefik rate-limit thresholds (200/400) | Proving the limit requires deliberately flooding a production ingress shared with 35 other services. | Middleware config asserted declaratively in `k8s/cluster/ingress.yaml`. | Operator; revisit with a dedicated test host. |
| The `tidaldb` namespace standalone Deployment | Live `1/1` but nothing routes to it; the fleet record calls it superseded. Verifying it would legitimise a surface that should be retired. | Noted as open drift in the runbook's closing section. | Operator; revisit by retiring it. |

## Bug log

| Bug ID | Capability | Severity | Expected | Actual | Evidence | Root cause | Fix | Verification | Status |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BUG-001 | CAP-014 | High | The runbook describes the live state of the credential split. | Runbook §9 said the split was "not active yet — requires an image roll"; it was already live and enforcing. | Harness global setup read image `m12-admin-gate-20260823` and found `TIDAL_ADMIN_KEY` in the secret; behavioural probe returned data→403, admin→200. | The runbook was written against the previously deployed image and not revisited after the roll. Documentation drift, not a product defect. | Rewrote §9: 9.2 marked LIVE with the working probe; §9.1/§9.3 kept as inert with the reason (image predates `4766f56`). | Re-read §9; `09-operator-authority.spec.ts` now asserts the live gate and would fail if it regressed. | fixed |
| BUG-002 | CAP-008, CAP-015 | Medium | A metrics scrape either returns series or fails loudly. | `wget --timeout=6` returned zero lines for `tidaldb-2`, indistinguishable from "this pod exports nothing". | Three retries at `--timeout=10` returned metrics every time; peers returned 332/344 series at 6s. | Client timeout too short for the payload on a loaded node — a measurement artifact, not a product defect. | Harness uses a generous timeout and asserts baseline `tidaldb_` series are present before concluding a specific metric is absent. Runbook §9.1 now specifies `--timeout=15` and explains why. | `04-network-isolation.spec.ts` and `09-operator-authority.spec.ts` both green with the baseline-present guard. | fixed |
| BUG-003 | CAP-011, CAP-014 | Low | The boot log reflects the current credential state. | Boot log WARNs `TIDAL_ADMIN_KEY is not set` while the gate is live and enforcing. | Pod started 05:41; WARN at 05:42; kubelet materialized the projected secret at 05:51:43; poller loaded it. | Working as designed — the credential poller exists so a key can be added without a restart. The log is a point-in-time record, not current state. | Documented in runbook §9.2 as an explicit "do not trust the boot WARN, test the behaviour" note; the harness allowlists this WARN with that reason. | `06-logs.spec.ts` allowlists it with the rationale; `09-operator-authority.spec.ts` proves the behaviour. | fixed |
| BUG-004 | CAP-013 | Medium | The backup check verifies the fleet backup. | Selecting the newest Backup by timestamp picked `restore-canary-longhorn-rwx-*` (20 items, 1 PVB) — it would report "pass" having verified nothing about the fleet. | Ran the runbook's own command as written; it returned the canary. | The selector sorted all Backups instead of filtering to the schedule that the freshness alert actually watches. | Runbook and harness both now filter on `velero.io/schedule-name=velero-fleet-daily`. | Re-ran: selects `velero-fleet-daily-20260823033025`, 3708/3708 items, 48/48 PVBs Completed. | fixed |
| BUG-018 | CAP-018 | High | A declared negative signal can demote an item. | Five `skip` writes changed nothing under any of the seven resolvable profiles, and no profile reports a skip term. | `10-ranking-semantics.spec.ts` records positions before/after for all seven profiles: identical. | The `Penalty` mechanism is fully implemented (`tidal/src/ranking/profile.rs:227`, applied at `tidal/src/ranking/executor/signal_values.rs:183`, labelled `{signal}_penalty` at `tidal/src/ranking/executor/mod.rs:65`) but never populated: every built-in profile is built from `skeleton()`, which sets `penalties: vec![]` (`tidal/src/ranking/builtins.rs:62`), and none of the 27 overrides it. Same shape as the `scatter_merge` defect — a guard present on one path and absent on its sibling. | Not fixed here. Adding a skip penalty to a shipped profile changes ranking behaviour for every consumer and is a product decision, not a test fix. Routed to `@tidal-engineer` with the inertness pinned by CAP-018. | `10-ranking-semantics.spec.ts` asserts the current inertness and fails with an instruction the day a profile applies a penalty. | routed |
| BUG-019 | CAP-018 | Medium | Every signal a profile reads is declared by the schema. | Three built-in profiles read signals this schema does not declare, so those terms are permanently 0: `trending` reads `share_velocity`, `hidden_gems` reads `completion`, `controversial` reads `dislike`. | Signal-name vocabulary captured per profile in `skip-is-inert.json`. | Built-in profiles reference a richer signal set than the deployed schema declares; the engine reports the term rather than rejecting the profile, so the degradation is silent. `trending` effectively ranks on `view_velocity` alone. | Not fixed here. Either the profiles should validate against the loaded schema at registration, or the schema should declare the full set — a product decision. Routed to `@tidal-engineer`. | The vocabulary is recorded as evidence on every run, so a change is visible. | routed |
| BUG-020 | CAP-016 | Low | A candidate scan honours the sort field it declares. | `for_you` declares `CandidateStrategy::Scan { sort_field: "created_at" }` (`tidal/src/ranking/builtins.rs:49`) but a `created_at` metadata value has no effect: an order chosen to match neither id-ascending nor created_at-descending came back strictly id-ascending. | Probed four items with interleaved timestamps; observed order was id-ascending in every case, including numeric-epoch values. | The scan appears to ignore the metadata string entirely and fall back to entity id. | Not fixed here. Recorded so nobody "fixes" a monotonous fixture feed by writing timestamps that cannot work; the fixture contract documents it at the point of temptation. | Documented in `tests/e2e/app/fixture-contract.ts`. | routed |