fe8d0c87e7
Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement). Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release gate past its budget. Both push-path steps now declare backend_options.kubernetes.resources as two YAML anchors declared once on their first consuming step. The values are CALIBRATED against measured free node capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's original figure) fits on NO node and would sit Pending forever, because `ci-build-bounds` grants permission and the nodes supply capacity, and those are not the same thing. The `nightly` cron described in this file for 216 days was never created, so tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero signal while reading like standing coverage. nightly-chaos and nightly-security-ops now alias the anchors and have budgets matching the gate (their 120/90 were TIGHTER on the same runner, so they would have failed nightly for a budget reason, not a correctness one). nightly-soak is REMOVED, not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best node has 1700m free CPU, so it would fail on starvation rather than regression — manufacturing a nightly false alarm. Its commands move verbatim to docs/runbooks/nightly-soak.md. Ring 1 — four fabrications removed from the wire. - scatter_merge sorted and truncated without re-stamping rank, so /feed and /search returned 1,1,2 under full placement. Reuses merge_cross_shard's existing stamp; asserted on BOTH the multi-group merge path and the single-group [only] fast path that bypasses it. - aggregate_region_row's None arm invented `applied_events: 0` plus a deficit derived from it. applied_events/lag_events are now Option<u64>, null on the wire. leader_last_seq was also unwrap_or(0), so a node that could not reach the LEADER computed 0 - applied = 0 for every region and reported a converged cluster it had never measured — a fabrication pointing the dangerous way. - tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was actively hiding the PVC-wipe shape: a measured zero with a real deficit rendered as "no report" instead of BEHIND. Now read off the wire; converged exits 0, partitioned still exits nonzero. - /sharded/* answered 201/204 for single-copy writes with nothing anywhere saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the existing invalid_input path. Six call sites migrated, not the two this roadmap predicted — including docs/runbooks/cluster.md §16.3, which told operators to run a quorum-write probe via POST /sharded/items. That probe cannot verify quorum: the surface applies locally with no WAL append. It was used as the safety check between every step of a staged deploy earlier today. Ring 2 — observability. JSON_LOGS was already implemented and the deployment simply never asked for it; the StatefulSet now sets it, plus TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the VictoriaLogs `service` stream field and would have blinded every query keyed on it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only the origin would mean each vector lands on exactly one node, replicas never agree, and the alert built on it pages forever. Found by measurement, not planned: the 401 path discarded every fact about every rejection. Traefik has served 101,858 rejected requests to the public ingress — 87.6% of all its traffic — with no record of who or why anywhere. unauthorized_response now emits reason (missing_token vs invalid_token, the distinction that separates a scanner from a rotation that missed a consumer) and the forwarded client. The token is never logged. Also: scripts/restore-fleet.sh --cluster started the soak monitor while deliberately leaving its gate suspended, orphaning a watcher that has reported "0/30 green nights" for 13 days. The pair now moves together. Doc-guard's three-warning backlog is cleared with real backfill for M4/M6/M12. Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline, counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5; cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with the two remaining failures asserting the rank fix against the not-yet-rolled image — they are the post-deploy proof.
52 lines
3.1 KiB
Markdown
52 lines
3.1 KiB
Markdown
# m12p2 — ANN candidate generation in RETRIEVE (✅ COMPLETE 2026-06-14)
|
||
|
||
Landed in `bb21e69`; cache-invalidation fix `da5d2d4`. Changelog:
|
||
[CHANGELOG.md](../../../CHANGELOG.md). Milestone index: [README.md](README.md).
|
||
Backfilled record. This is the **G1 unblock**.
|
||
|
||
## What shipped
|
||
|
||
1. **`CandidateStrategy::Ann` is real.** It previously fell back to a scan with a
|
||
warning. The db layer now resolves the query vector — the user's **preference
|
||
vector** for `for_you`, the **seed item's embedding** (`similar_to`) for
|
||
`related` — and Stage 1 runs an `O(ef_search)` HNSW search over the item
|
||
content slot instead of scanning an arbitrary low-id slice of the universe
|
||
(`tidal/src/query/executor/candidate_gen.rs`,
|
||
`tidal/src/query/executor/pipeline.rs`). `for_you` and `related` are now `Ann`
|
||
profiles.
|
||
2. **Graceful, not conditional-on-luck.** No vector registry, no preference
|
||
vector, or no seed ⇒ it degrades to a scan. Anonymous reads and cold-start
|
||
users still serve; every embedding-less schema and every pre-m12p2 caller is
|
||
unchanged.
|
||
3. **`trending` stopped being an O(N) ledger scan.** `SignalRanked` is now a
|
||
cached per-signal-type top-K (`tidal/src/signals/ledger/hot_top_k.rs`): O(K)
|
||
on the served path, with a bounded O(N) rebuild only when stale. The cache is
|
||
sound because decay preserves relative order at a fixed λ, so it stays valid
|
||
until the next write. Small ledgers rebuild on any write (always fresh); large
|
||
ledgers throttle the rebuild off the read hot path (1 s). `trending` now uses
|
||
`SignalRanked(view)`, so it ranks the actually-viewed corpus at any id — not
|
||
whatever happened to sit in the low-id scan slice.
|
||
4. **`related` over HTTP.** `GET /feed?profile=related&similar_to=<id>` resolves
|
||
the seed's embedding and runs ANN — "more like this" on the read surface
|
||
(`similar_to` on `FeedQuery`, threaded through all three feed handlers).
|
||
5. **Harness support for measuring it.** `tidal-stress --feed-profile <name>`
|
||
forces every feed read to one profile (per-profile retrieve p99) and
|
||
`--seed-preferences` builds a preference vector per user so `for_you`
|
||
exercises ANN rather than the scan fallback.
|
||
6. **Cache correctness under replication** (`da5d2d4`): the `SignalRanked` top-K
|
||
cache is invalidated on CRDT reconciliation. Without that, a follower could
|
||
serve a top-K assembled before reconciliation merged remote signal state.
|
||
|
||
## Evidence
|
||
|
||
- **trending retrieve p99 = 3.5–7.7 ms** under concurrent writes against a real
|
||
1536-D standalone server — inside the 10 ms G1 target — via the cached top-K.
|
||
- `for_you` retrieve is ANN-backed at p99 ≈ 24 ms, dominated by the Stage-3
|
||
preference-boost recompute (a per-candidate embedding read). Recorded, not
|
||
hidden, and flagged forward to m12p3's index/score tuning as the risk
|
||
register's "materialized-score layer".
|
||
- ANN candidate recall is the m12p1 `/vector_search` probe: 0.9997.
|
||
- `tidal/tests/m12p2_ann_retrieve.rs` (5) — proves ANN, `related`, and `trending`
|
||
reach relevant items at high ids that a scan can never reach, plus cache
|
||
freshness.
|