fe8d0c87e7
Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement). Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release gate past its budget. Both push-path steps now declare backend_options.kubernetes.resources as two YAML anchors declared once on their first consuming step. The values are CALIBRATED against measured free node capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's original figure) fits on NO node and would sit Pending forever, because `ci-build-bounds` grants permission and the nodes supply capacity, and those are not the same thing. The `nightly` cron described in this file for 216 days was never created, so tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero signal while reading like standing coverage. nightly-chaos and nightly-security-ops now alias the anchors and have budgets matching the gate (their 120/90 were TIGHTER on the same runner, so they would have failed nightly for a budget reason, not a correctness one). nightly-soak is REMOVED, not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best node has 1700m free CPU, so it would fail on starvation rather than regression — manufacturing a nightly false alarm. Its commands move verbatim to docs/runbooks/nightly-soak.md. Ring 1 — four fabrications removed from the wire. - scatter_merge sorted and truncated without re-stamping rank, so /feed and /search returned 1,1,2 under full placement. Reuses merge_cross_shard's existing stamp; asserted on BOTH the multi-group merge path and the single-group [only] fast path that bypasses it. - aggregate_region_row's None arm invented `applied_events: 0` plus a deficit derived from it. applied_events/lag_events are now Option<u64>, null on the wire. leader_last_seq was also unwrap_or(0), so a node that could not reach the LEADER computed 0 - applied = 0 for every region and reported a converged cluster it had never measured — a fabrication pointing the dangerous way. - tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was actively hiding the PVC-wipe shape: a measured zero with a real deficit rendered as "no report" instead of BEHIND. Now read off the wire; converged exits 0, partitioned still exits nonzero. - /sharded/* answered 201/204 for single-copy writes with nothing anywhere saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the existing invalid_input path. Six call sites migrated, not the two this roadmap predicted — including docs/runbooks/cluster.md §16.3, which told operators to run a quorum-write probe via POST /sharded/items. That probe cannot verify quorum: the surface applies locally with no WAL append. It was used as the safety check between every step of a staged deploy earlier today. Ring 2 — observability. JSON_LOGS was already implemented and the deployment simply never asked for it; the StatefulSet now sets it, plus TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the VictoriaLogs `service` stream field and would have blinded every query keyed on it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only the origin would mean each vector lands on exactly one node, replicas never agree, and the alert built on it pages forever. Found by measurement, not planned: the 401 path discarded every fact about every rejection. Traefik has served 101,858 rejected requests to the public ingress — 87.6% of all its traffic — with no record of who or why anywhere. unauthorized_response now emits reason (missing_token vs invalid_token, the distinction that separates a scanner from a rotation that missed a consumer) and the forwarded client. The token is never logged. Also: scripts/restore-fleet.sh --cluster started the soak monitor while deliberately leaving its gate suspended, orphaning a watcher that has reported "0/30 green nights" for 13 days. The pair now moves together. Doc-guard's three-warning backlog is cleared with real backfill for M4/M6/M12. Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline, counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5; cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with the two remaining failures asserting the rank fix against the not-yet-rolled image — they are the post-deploy proof.
81 lines
4.3 KiB
Markdown
81 lines
4.3 KiB
Markdown
# Multi-vector user preference modelling (✅ COMPLETE 2026-06-23)
|
||
|
||
Landed in `6a937fc`, with the docs/spec/API refresh in `4051077`. Design:
|
||
[docs/research/multi-vector-preference.md](../../research/multi-vector-preference.md).
|
||
Changelog: [CHANGELOG.md](../../../CHANGELOG.md). Milestone index:
|
||
[README.md](README.md). Backfilled record.
|
||
|
||
> **Deliberately not numbered.** The ROADMAP's M12 row lists this work as a peer
|
||
> of m12p1–m12p6 with no phase number, and no commit or document ever called it
|
||
> `m12p7`. It is filed under its own name rather than given an invented id.
|
||
|
||
## The premise
|
||
|
||
A warm user is many interests, not one averaged vector. Averaging jazz and
|
||
powerlifting into a single centroid produces a vector that retrieves neither —
|
||
and ANN candidate generation (m12p2) made that failure load-bearing, because the
|
||
preference vector *is* the query.
|
||
|
||
## What shipped
|
||
|
||
1. **Online preference clustering** (`tidal/src/entities/multi_preference.rs`).
|
||
A warm user (≥ `COLD_START_N = 5` interactions) maintains up to `K_MAX`
|
||
preference clusters, built by online sequential k-means with a DP-means
|
||
threshold split: a new engagement updates its nearest cluster via a per-cluster
|
||
adaptive EMA, or — past the split threshold and under the cap — opens a new
|
||
cluster. At the cap the nearest cluster absorbs the engagement, so the
|
||
structure is bounded rather than unboundedly growing.
|
||
2. **Interests that fade.** Per-cluster *importance* composes the canonical
|
||
forward-decay kernel anchored to each engagement's timestamp, so a stale
|
||
interest decays instead of persisting at full strength forever. Reusing the
|
||
canonical kernel is the same discipline `SessionHotState` follows: one decay
|
||
implementation, not per-tier copies.
|
||
3. **Top-M ANN fan-out.** At query time `for_you` selects the top-`M` clusters by
|
||
current importance, issues `M` ANN queries
|
||
(`candidate_gen::ann_candidates_multi`), and merges by best (minimum)
|
||
distance; the personalization boost is the **max** cosine over all clusters —
|
||
a candidate that matches any one interest strongly is not diluted by the
|
||
interests it does not match.
|
||
4. **Cold-start fallback preserved.** Users below the cold-start threshold keep
|
||
the single adaptive-LR vector (`tidal/src/entities/preference.rs`), so nothing
|
||
about the pre-existing behaviour changes for a new user.
|
||
5. **Zero-migration persistence.** `MultiPreferenceVectors::checkpoint` /
|
||
`restore` (`multi_preference.rs:642,703`) serialize per-cluster
|
||
`[update_count:8 LE][importance_at_anchor:4 LE][anchor_ts:8 LE]` behind a
|
||
`FORMAT_VERSION` byte, with a first-byte discrimination trick so a **legacy
|
||
single-vector row whose `update_count` low byte happens to equal the format
|
||
version** is still read correctly as a cold-start user rather than
|
||
misparsed. That edge is covered by
|
||
`restore_rescues_legacy_row_whose_count_low_byte_equals_format_version`.
|
||
|
||
## Evidence
|
||
|
||
- Unit coverage in `multi_preference.rs`:
|
||
`checkpoint_restore_roundtrip_multi_cluster`,
|
||
`restore_reads_legacy_single_vector_rows_as_cold_start`,
|
||
`restore_renormalizes_torn_cluster_to_unit_length`,
|
||
`restore_drops_torn_tail_cluster_keeps_prefix`,
|
||
`restore_skips_dimension_mismatch`,
|
||
`restore_skips_dim_mismatched_multi_row_not_loaded_as_garbage`,
|
||
`checkpoint_skips_cold_row_for_user_also_in_clusters_no_demotion`.
|
||
The torn-row and dimension-mismatch cases matter: a checkpoint is read after a
|
||
crash, so "garbage in the tail" is the expected input, not the exceptional one.
|
||
- `tidal/tests/m12_preference_event_time.rs` (1) — event-time anchoring.
|
||
- `tidal/benches/multi_preference.rs`.
|
||
|
||
## Side effect worth recording
|
||
|
||
This work incidentally closed a limitation M6 had booked against M7: per-user
|
||
preference `update_count` is now persisted and restored (see
|
||
[milestone-6/phase-4.md](../milestone-6/phase-4.md) and
|
||
[milestone-6/phase-6.md](../milestone-6/phase-6.md)). The `# Known Limitation`
|
||
comment in `tidal/src/entities/preference.rs:37-45` still claims otherwise and is
|
||
stale.
|
||
|
||
## Deferred
|
||
|
||
The **offline medoid-recluster tier** — periodically re-deriving cluster centroids
|
||
from stored engagements rather than only updating them online — is recorded in the
|
||
ROADMAP as an open follow-up. Online sequential k-means is order-dependent; a
|
||
recluster pass is what would remove that dependence. Not built.
|