tidaldb

jordan/tidaldb

Fork 0

Commit Graph

Author	SHA1	Message	Date
jordan	fe8d0c87e7	harden: restore CI verification, remove four wire-level fabrications, instrument the 401 path Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement). Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release gate past its budget. Both push-path steps now declare backend_options.kubernetes.resources as two YAML anchors declared once on their first consuming step. The values are CALIBRATED against measured free node capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's original figure) fits on NO node and would sit Pending forever, because `ci-build-bounds` grants permission and the nodes supply capacity, and those are not the same thing. The `nightly` cron described in this file for 216 days was never created, so tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero signal while reading like standing coverage. nightly-chaos and nightly-security-ops now alias the anchors and have budgets matching the gate (their 120/90 were TIGHTER on the same runner, so they would have failed nightly for a budget reason, not a correctness one). nightly-soak is REMOVED, not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best node has 1700m free CPU, so it would fail on starvation rather than regression — manufacturing a nightly false alarm. Its commands move verbatim to docs/runbooks/nightly-soak.md. Ring 1 — four fabrications removed from the wire. - scatter_merge sorted and truncated without re-stamping rank, so /feed and /search returned 1,1,2 under full placement. Reuses merge_cross_shard's existing stamp; asserted on BOTH the multi-group merge path and the single-group [only] fast path that bypasses it. - aggregate_region_row's None arm invented `applied_events: 0` plus a deficit derived from it. applied_events/lag_events are now Option<u64>, null on the wire. leader_last_seq was also unwrap_or(0), so a node that could not reach the LEADER computed 0 - applied = 0 for every region and reported a converged cluster it had never measured — a fabrication pointing the dangerous way. - tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was actively hiding the PVC-wipe shape: a measured zero with a real deficit rendered as "no report" instead of BEHIND. Now read off the wire; converged exits 0, partitioned still exits nonzero. - /sharded/* answered 201/204 for single-copy writes with nothing anywhere saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the existing invalid_input path. Six call sites migrated, not the two this roadmap predicted — including docs/runbooks/cluster.md §16.3, which told operators to run a quorum-write probe via POST /sharded/items. That probe cannot verify quorum: the surface applies locally with no WAL append. It was used as the safety check between every step of a staged deploy earlier today. Ring 2 — observability. JSON_LOGS was already implemented and the deployment simply never asked for it; the StatefulSet now sets it, plus TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the VictoriaLogs `service` stream field and would have blinded every query keyed on it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only the origin would mean each vector lands on exactly one node, replicas never agree, and the alert built on it pages forever. Found by measurement, not planned: the 401 path discarded every fact about every rejection. Traefik has served 101,858 rejected requests to the public ingress — 87.6% of all its traffic — with no record of who or why anywhere. unauthorized_response now emits reason (missing_token vs invalid_token, the distinction that separates a scanner from a rotation that missed a consumer) and the forwarded client. The token is never logged. Also: scripts/restore-fleet.sh --cluster started the soak monitor while deliberately leaving its gate suspended, orphaning a watcher that has reported "0/30 green nights" for 13 days. The pair now moves together. Doc-guard's three-warning backlog is cleared with real backfill for M4/M6/M12. Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline, counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5; cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with the two remaining failures asserting the rank fix against the not-yet-rolled image — they are the post-deploy proof.	2026-08-30 20:55:58 -06:00
jordan	9523f6da43	test(e2e): verify ranking semantics with a content-feed app, and route three product findings The existing 32 checks prove the deployment answers -- TLS, auth, quorum commit, convergence, isolation, dashboards, backups. Not one wrote a signal and observed an order change, so VISION.md:17 "Ranking is not a feature. It is a primitive." was unverified. This adds a 60-item content-feed app and five assertions that verify the product's semantics, on a hermetic standalone node. Added - tests/e2e/app/: fixture contract (60 items, 4 categories, each owning one unoccupied 100-id embedding cluster), a deep-module harness owning the whole lifecycle behind startApp(), the product page, and an app:dev entry point. - tidal-stress/src/bin/feed-fixture.rs: seeds the catalog and emits brute-force ground truth, reusing recall::embedding_for rather than adding a third copy of the corpus generator (tidal/src/db/items.rs already holds a second). - GroundTruth::from_ids: the oracle now serves sparse id sets. build() delegates, so there is no transient copy even at 1M, and top_k indexes positionally. - 10-ranking-semantics.spec.ts (5 hermetic checks) and 11-ranking-integrity.spec.ts (2 cluster tripwires). - playwright.semantics.config.ts + CAP-016 demo beat (walkthrough 82s -> 90s). Measured, not merely green - like: index 59 -> 0, like_boost 2.0, with no sleep between write and read. - decay: implied half-lives 7.0007 d and 14.0014 d against a schema declaring 7 d and 14 d, recovered from a 4-second window via H = tln2 / -ln(v2/v1) and compared against the schema the node actually loaded, not a hardcoded copy. - ANN: top-10 identical to brute-force cosine on all four probes; self-distance 0.0148-0.0197 against a 0.05 tolerance. - rank: dense 1..60 on standalone vs [1,1,1,2,2,3,4,3,4,5,6,5] on the cluster. Three product findings, pinned and routed to @tidal-engineer - BUG-018 (High) skip is durably accepted and query-time inert. Penalty is fully implemented (ranking/profile.rs:227 -> executor/signal_values.rs:183, labelled {signal}_penalty at executor/mod.rs:65) but skeleton() sets penalties: vec![] (ranking/builtins.rs:62) and none of the 27 built-ins overrides it. So VISION.md:187 "negative signals are equal citizens" holds for no shipped profile. Same anti-pattern as the reseed defects and scatter_merge: a guard present on one path, absent on its sibling. - BUG-019 (Medium) three built-ins read signals this schema does not declare -- trending/share_velocity, hidden_gems/completion, controversial/dislike -- so those terms are permanently 0 and trending ranks on view_velocity alone. - BUG-020 (Low) for_you declares Scan{sort_field:"created_at"} but ignores a created_at metadata value; an order matching neither id-asc nor created_at-desc came back strictly id-ascending. Two assertions therefore report a gap rather than a success, written as tripwires whose failure message says what to do when the gap closes. The rank defect is localised, not fixed: scatter_merge (cluster/node.rs:7542) returns a merged slice without re-stamping rank while scores stay correctly ordered, so the fault is the missing stamp and not the merge's sort. Notes - Hermetic by construction: its own config, because FullConfig.projects is not filtered by --project and globalSetup publishes credentials into the main process that forked workers inherit -- so a setup project cannot replace it, and weakening globalSetup would destroy the fail-loud behaviour that is its purpose. Verified with KUBECONFIG=/nonexistent and all E2E_ unset. - Never touches the deployed corpus: skip is permanent: true, so seeding it into production would be irreversible. - The page contains no sort, no hostname and no credential; the harness proxy injects auth server-side so no bearer reaches a browser or a capture. - Schema comes from k8s/cluster/schema-configmap.yaml, asserted at 1536 dims; tidal-server/config/default-schema.yaml declares 128 and would 422 every write. Verification: 5 semantics + 34 regression + 10 demo captures green; tsc clean; tidal-stress clippy clean under clippy::all=deny with unwrap_used=deny; 2101 tidaldb lib tests; preflight 10/10 perfect; render 90.05s/2700 frames with zero empty boundary frames; zero orphan processes or temp dirs after teardown.	2026-08-23 22:42:02 -06:00

Author

SHA1

Message

Date

jordan

fe8d0c87e7

harden: restore CI verification, remove four wire-level fabrications, instrument the 401 path

Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement).

Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace
default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release
gate past its budget. Both push-path steps now declare
backend_options.kubernetes.resources as two YAML anchors declared once on their
first consuming step. The values are CALIBRATED against measured free node
capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's
original figure) fits on NO node and would sit Pending forever, because
`ci-build-bounds` grants permission and the nodes supply capacity, and those are
not the same thing.

The `nightly` cron described in this file for 216 days was never created, so
tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero
signal while reading like standing coverage. nightly-chaos and
nightly-security-ops now alias the anchors and have budgets matching the gate
(their 120/90 were TIGHTER on the same runner, so they would have failed
nightly for a budget reason, not a correctness one). nightly-soak is REMOVED,
not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best
node has 1700m free CPU, so it would fail on starvation rather than regression —
manufacturing a nightly false alarm. Its commands move verbatim to
docs/runbooks/nightly-soak.md.

Ring 1 — four fabrications removed from the wire.
- scatter_merge sorted and truncated without re-stamping rank, so /feed and
  /search returned 1,1,2 under full placement. Reuses merge_cross_shard's
  existing stamp; asserted on BOTH the multi-group merge path and the
  single-group [only] fast path that bypasses it.
- aggregate_region_row's None arm invented `applied_events: 0` plus a deficit
  derived from it. applied_events/lag_events are now Option<u64>, null on the
  wire. leader_last_seq was also unwrap_or(0), so a node that could not reach
  the LEADER computed 0 - applied = 0 for every region and reported a converged
  cluster it had never measured — a fabrication pointing the dangerous way.
- tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was
  actively hiding the PVC-wipe shape: a measured zero with a real deficit
  rendered as "no report" instead of BEHIND. Now read off the wire; converged
  exits 0, partitioned still exits nonzero.
- /sharded/* answered 201/204 for single-copy writes with nothing anywhere
  saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the
  existing invalid_input path. Six call sites migrated, not the two this
  roadmap predicted — including docs/runbooks/cluster.md §16.3, which told
  operators to run a quorum-write probe via POST /sharded/items. That probe
  cannot verify quorum: the surface applies locally with no WAL append. It was
  used as the safety check between every step of a staged deploy earlier today.

Ring 2 — observability. JSON_LOGS was already implemented and the deployment
simply never asked for it; the StatefulSet now sets it, plus
TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the
VictoriaLogs `service` stream field and would have blinded every query keyed on
it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the
origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only
the origin would mean each vector lands on exactly one node, replicas never
agree, and the alert built on it pages forever.

Found by measurement, not planned: the 401 path discarded every fact about
every rejection. Traefik has served 101,858 rejected requests to the public
ingress — 87.6% of all its traffic — with no record of who or why anywhere.
unauthorized_response now emits reason (missing_token vs invalid_token, the
distinction that separates a scanner from a rotation that missed a consumer)
and the forwarded client. The token is never logged.

Also: scripts/restore-fleet.sh --cluster started the soak monitor while
deliberately leaving its gate suspended, orphaning a watcher that has reported
"0/30 green nights" for 13 days. The pair now moves together. Doc-guard's
three-warning backlog is cleared with real backfill for M4/M6/M12.

Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline,
counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5;
cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with
the two remaining failures asserting the rank fix against the not-yet-rolled
image — they are the post-deploy proof.

2026-08-30 20:55:58 -06:00

jordan

9523f6da43

test(e2e): verify ranking semantics with a content-feed app, and route three product findings

The existing 32 checks prove the deployment answers -- TLS, auth, quorum commit,
convergence, isolation, dashboards, backups. Not one wrote a signal and observed
an order change, so VISION.md:17 "Ranking is not a feature. It is a primitive."
was unverified. This adds a 60-item content-feed app and five assertions that
verify the product's semantics, on a hermetic standalone node.

Added
- tests/e2e/app/: fixture contract (60 items, 4 categories, each owning one
  unoccupied 100-id embedding cluster), a deep-module harness owning the whole
  lifecycle behind startApp(), the product page, and an app:dev entry point.
- tidal-stress/src/bin/feed-fixture.rs: seeds the catalog and emits brute-force
  ground truth, reusing recall::embedding_for rather than adding a third copy of
  the corpus generator (tidal/src/db/items.rs already holds a second).
- GroundTruth::from_ids: the oracle now serves sparse id sets. build() delegates,
  so there is no transient copy even at 1M, and top_k indexes positionally.
- 10-ranking-semantics.spec.ts (5 hermetic checks) and
  11-ranking-integrity.spec.ts (2 cluster tripwires).
- playwright.semantics.config.ts + CAP-016 demo beat (walkthrough 82s -> 90s).

Measured, not merely green
- like: index 59 -> 0, like_boost 2.0, with no sleep between write and read.
- decay: implied half-lives 7.0007 d and 14.0014 d against a schema declaring
  7 d and 14 d, recovered from a 4-second window via H = t*ln2 / -ln(v2/v1) and
  compared against the schema the node actually loaded, not a hardcoded copy.
- ANN: top-10 identical to brute-force cosine on all four probes; self-distance
  0.0148-0.0197 against a 0.05 tolerance.
- rank: dense 1..60 on standalone vs [1,1,1,2,2,3,4,3,4,5,6,5] on the cluster.

Three product findings, pinned and routed to @tidal-engineer
- BUG-018 (High) skip is durably accepted and query-time inert. Penalty is fully
  implemented (ranking/profile.rs:227 -> executor/signal_values.rs:183, labelled
  {signal}_penalty at executor/mod.rs:65) but skeleton() sets penalties: vec![]
  (ranking/builtins.rs:62) and none of the 27 built-ins overrides it. So
  VISION.md:187 "negative signals are equal citizens" holds for no shipped
  profile. Same anti-pattern as the reseed defects and scatter_merge: a guard
  present on one path, absent on its sibling.
- BUG-019 (Medium) three built-ins read signals this schema does not declare --
  trending/share_velocity, hidden_gems/completion, controversial/dislike -- so
  those terms are permanently 0 and trending ranks on view_velocity alone.
- BUG-020 (Low) for_you declares Scan{sort_field:"created_at"} but ignores a
  created_at metadata value; an order matching neither id-asc nor
  created_at-desc came back strictly id-ascending.

Two assertions therefore report a gap rather than a success, written as tripwires
whose failure message says what to do when the gap closes. The rank defect is
localised, not fixed: scatter_merge (cluster/node.rs:7542) returns a merged slice
without re-stamping rank while scores stay correctly ordered, so the fault is the
missing stamp and not the merge's sort.

Notes
- Hermetic by construction: its own config, because FullConfig.projects is not
  filtered by --project and globalSetup publishes credentials into the main
  process that forked workers inherit -- so a setup project cannot replace it,
  and weakening globalSetup would destroy the fail-loud behaviour that is its
  purpose. Verified with KUBECONFIG=/nonexistent and all E2E_* unset.
- Never touches the deployed corpus: skip is permanent: true, so seeding it into
  production would be irreversible.
- The page contains no sort, no hostname and no credential; the harness proxy
  injects auth server-side so no bearer reaches a browser or a capture.
- Schema comes from k8s/cluster/schema-configmap.yaml, asserted at 1536 dims;
  tidal-server/config/default-schema.yaml declares 128 and would 422 every write.

Verification: 5 semantics + 34 regression + 10 demo captures green; tsc clean;
tidal-stress clippy clean under clippy::all=deny with unwrap_used=deny; 2101
tidaldb lib tests; preflight 10/10 perfect; render 90.05s/2700 frames with zero
empty boundary frames; zero orphan processes or temp dirs after teardown.

2026-08-23 22:42:02 -06:00

2 Commits