jordan fe8d0c87e7 harden: restore CI verification, remove four wire-level fabrications, instrument the 401 path

Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement).

Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace
default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release
gate past its budget. Both push-path steps now declare
backend_options.kubernetes.resources as two YAML anchors declared once on their
first consuming step. The values are CALIBRATED against measured free node
capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's
original figure) fits on NO node and would sit Pending forever, because
`ci-build-bounds` grants permission and the nodes supply capacity, and those are
not the same thing.

The `nightly` cron described in this file for 216 days was never created, so
tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero
signal while reading like standing coverage. nightly-chaos and
nightly-security-ops now alias the anchors and have budgets matching the gate
(their 120/90 were TIGHTER on the same runner, so they would have failed
nightly for a budget reason, not a correctness one). nightly-soak is REMOVED,
not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best
node has 1700m free CPU, so it would fail on starvation rather than regression —
manufacturing a nightly false alarm. Its commands move verbatim to
docs/runbooks/nightly-soak.md.

Ring 1 — four fabrications removed from the wire.
- scatter_merge sorted and truncated without re-stamping rank, so /feed and
  /search returned 1,1,2 under full placement. Reuses merge_cross_shard's
  existing stamp; asserted on BOTH the multi-group merge path and the
  single-group [only] fast path that bypasses it.
- aggregate_region_row's None arm invented `applied_events: 0` plus a deficit
  derived from it. applied_events/lag_events are now Option<u64>, null on the
  wire. leader_last_seq was also unwrap_or(0), so a node that could not reach
  the LEADER computed 0 - applied = 0 for every region and reported a converged
  cluster it had never measured — a fabrication pointing the dangerous way.
- tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was
  actively hiding the PVC-wipe shape: a measured zero with a real deficit
  rendered as "no report" instead of BEHIND. Now read off the wire; converged
  exits 0, partitioned still exits nonzero.
- /sharded/* answered 201/204 for single-copy writes with nothing anywhere
  saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the
  existing invalid_input path. Six call sites migrated, not the two this
  roadmap predicted — including docs/runbooks/cluster.md §16.3, which told
  operators to run a quorum-write probe via POST /sharded/items. That probe
  cannot verify quorum: the surface applies locally with no WAL append. It was
  used as the safety check between every step of a staged deploy earlier today.

Ring 2 — observability. JSON_LOGS was already implemented and the deployment
simply never asked for it; the StatefulSet now sets it, plus
TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the
VictoriaLogs `service` stream field and would have blinded every query keyed on
it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the
origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only
the origin would mean each vector lands on exactly one node, replicas never
agree, and the alert built on it pages forever.

Found by measurement, not planned: the 401 path discarded every fact about
every rejection. Traefik has served 101,858 rejected requests to the public
ingress — 87.6% of all its traffic — with no record of who or why anywhere.
unauthorized_response now emits reason (missing_token vs invalid_token, the
distinction that separates a scanner from a rotation that missed a consumer)
and the forwarded client. The token is never logged.

Also: scripts/restore-fleet.sh --cluster started the soak monitor while
deliberately leaving its gate suspended, orphaning a watcher that has reported
"0/30 green nights" for 13 days. The pair now moves together. Doc-guard's
three-warning backlog is cleared with real backfill for M4/M6/M12.

Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline,
counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5;
cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with
the two remaining failures asserting the rank fix against the not-yet-rolled
image — they are the post-deploy proof.

2026-08-30 20:55:58 -06:00

2.2 KiB

Raw Blame History

m6p5 — Query Composition + SUGGEST Autocomplete (✅ COMPLETE 2026-02-23)

Phase spec and acceptance criteria: ROADMAP · Milestone 6 · Phase 5. Milestone index: README.md. Backfilled record.

What shipped

WithinScope on SEARCH (tidal/src/query/search/types.rs:33-44) — five variants, exactly as specified:

Variant	Candidate set
`Trending { window_hours }`	items above the velocity threshold in the window, from the global ledger
`CohortTrending { cohort, window_hours }`	same, scoped to a defined cohort (m6p1)
`Following`	items from creators the querying user follows; requires `for_user`
`Category { name }`	items matching the category metadata value
`Collection { id }`	items in the collection's bitmap (m6p4)

The scope resolves to a RoaringBitmap applied before candidate scoring, so it reaches both retrieval arms: a post-filter for Tantivy BM25 and a predicate callback for the USearch ANN search. Scoping before scoring is the point — scoping after would rank the wrong universe and then discard the answer.

db.suggest() (tidal/src/query/suggest.rs, tidal/src/db/query_ops.rs). Prefix autocomplete over terms extracted from item titles at write time, updated incrementally on write_item_with_metadata; an empty prefix returns trending search terms by recent query frequency, counted on each db.search() call.

Evidence

tidal/tests/m6p5_scope.rs — 13 #[test] functions: each WithinScope variant independently, the composition case (search within cohort trending), and SUGGEST with both a prefix and an empty prefix.
tidal/src/query/search/types/tests.rs — WithinScope unit coverage.

Dependencies this phase actually had

CohortTrending needs m6p1's cohort ledger and Collection needs m6p4's collection bitmap, so despite the ROADMAP DAG showing phases 1–4 as broadly parallel, m6p5 was genuinely gated on both.

Note on the "< 50 ms SEARCH / < 20 ms SUGGEST" criteria

Marked [x] with no recorded measurement in docs/. Functional behaviour is proven by m6p5_scope.rs; the two latency figures are not evidenced.

2.2 KiB Raw Blame History Unescape Escape