jordan fe8d0c87e7 harden: restore CI verification, remove four wire-level fabrications, instrument the 401 path

Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement).

Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace
default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release
gate past its budget. Both push-path steps now declare
backend_options.kubernetes.resources as two YAML anchors declared once on their
first consuming step. The values are CALIBRATED against measured free node
capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's
original figure) fits on NO node and would sit Pending forever, because
`ci-build-bounds` grants permission and the nodes supply capacity, and those are
not the same thing.

The `nightly` cron described in this file for 216 days was never created, so
tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero
signal while reading like standing coverage. nightly-chaos and
nightly-security-ops now alias the anchors and have budgets matching the gate
(their 120/90 were TIGHTER on the same runner, so they would have failed
nightly for a budget reason, not a correctness one). nightly-soak is REMOVED,
not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best
node has 1700m free CPU, so it would fail on starvation rather than regression —
manufacturing a nightly false alarm. Its commands move verbatim to
docs/runbooks/nightly-soak.md.

Ring 1 — four fabrications removed from the wire.
- scatter_merge sorted and truncated without re-stamping rank, so /feed and
  /search returned 1,1,2 under full placement. Reuses merge_cross_shard's
  existing stamp; asserted on BOTH the multi-group merge path and the
  single-group [only] fast path that bypasses it.
- aggregate_region_row's None arm invented `applied_events: 0` plus a deficit
  derived from it. applied_events/lag_events are now Option<u64>, null on the
  wire. leader_last_seq was also unwrap_or(0), so a node that could not reach
  the LEADER computed 0 - applied = 0 for every region and reported a converged
  cluster it had never measured — a fabrication pointing the dangerous way.
- tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was
  actively hiding the PVC-wipe shape: a measured zero with a real deficit
  rendered as "no report" instead of BEHIND. Now read off the wire; converged
  exits 0, partitioned still exits nonzero.
- /sharded/* answered 201/204 for single-copy writes with nothing anywhere
  saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the
  existing invalid_input path. Six call sites migrated, not the two this
  roadmap predicted — including docs/runbooks/cluster.md §16.3, which told
  operators to run a quorum-write probe via POST /sharded/items. That probe
  cannot verify quorum: the surface applies locally with no WAL append. It was
  used as the safety check between every step of a staged deploy earlier today.

Ring 2 — observability. JSON_LOGS was already implemented and the deployment
simply never asked for it; the StatefulSet now sets it, plus
TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the
VictoriaLogs `service` stream field and would have blinded every query keyed on
it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the
origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only
the origin would mean each vector lands on exactly one node, replicas never
agree, and the alert built on it pages forever.

Found by measurement, not planned: the 401 path discarded every fact about
every rejection. Traefik has served 101,858 rejected requests to the public
ingress — 87.6% of all its traffic — with no record of who or why anywhere.
unauthorized_response now emits reason (missing_token vs invalid_token, the
distinction that separates a scanner from a rotation that missed a consumer)
and the forwarded client. The token is never logged.

Also: scripts/restore-fleet.sh --cluster started the soak monitor while
deliberately leaving its gate suspended, orphaning a watcher that has reported
"0/30 green nights" for 13 days. The pair now moves together. Doc-guard's
three-warning backlog is cleared with real backfill for M4/M6/M12.

Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline,
counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5;
cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with
the two remaining failures asserting the rank fix against the not-yet-rolled
image — they are the post-deploy proof.

2026-08-30 20:55:58 -06:00

4.3 KiB

Raw Blame History

Multi-vector user preference modelling (✅ COMPLETE 2026-06-23)

Landed in 6a937fc, with the docs/spec/API refresh in 4051077. Design: docs/research/multi-vector-preference.md. Changelog: CHANGELOG.md. Milestone index: README.md. Backfilled record.

Deliberately not numbered. The ROADMAP's M12 row lists this work as a peer of m12p1–m12p6 with no phase number, and no commit or document ever called it m12p7. It is filed under its own name rather than given an invented id.

The premise

A warm user is many interests, not one averaged vector. Averaging jazz and powerlifting into a single centroid produces a vector that retrieves neither — and ANN candidate generation (m12p2) made that failure load-bearing, because the preference vector is the query.

What shipped

Online preference clustering (tidal/src/entities/multi_preference.rs). A warm user (≥ COLD_START_N = 5 interactions) maintains up to K_MAX preference clusters, built by online sequential k-means with a DP-means threshold split: a new engagement updates its nearest cluster via a per-cluster adaptive EMA, or — past the split threshold and under the cap — opens a new cluster. At the cap the nearest cluster absorbs the engagement, so the structure is bounded rather than unboundedly growing.
Interests that fade. Per-cluster importance composes the canonical forward-decay kernel anchored to each engagement's timestamp, so a stale interest decays instead of persisting at full strength forever. Reusing the canonical kernel is the same discipline SessionHotState follows: one decay implementation, not per-tier copies.
Top-M ANN fan-out. At query time for_you selects the top-M clusters by current importance, issues M ANN queries (candidate_gen::ann_candidates_multi), and merges by best (minimum) distance; the personalization boost is the max cosine over all clusters — a candidate that matches any one interest strongly is not diluted by the interests it does not match.
Cold-start fallback preserved. Users below the cold-start threshold keep the single adaptive-LR vector (tidal/src/entities/preference.rs), so nothing about the pre-existing behaviour changes for a new user.
Zero-migration persistence. MultiPreferenceVectors::checkpoint / restore (multi_preference.rs:642,703) serialize per-cluster [update_count:8 LE][importance_at_anchor:4 LE][anchor_ts:8 LE] behind a FORMAT_VERSION byte, with a first-byte discrimination trick so a legacy single-vector row whose update_count low byte happens to equal the format version is still read correctly as a cold-start user rather than misparsed. That edge is covered by restore_rescues_legacy_row_whose_count_low_byte_equals_format_version.

Evidence

Unit coverage in multi_preference.rs: checkpoint_restore_roundtrip_multi_cluster, restore_reads_legacy_single_vector_rows_as_cold_start, restore_renormalizes_torn_cluster_to_unit_length, restore_drops_torn_tail_cluster_keeps_prefix, restore_skips_dimension_mismatch, restore_skips_dim_mismatched_multi_row_not_loaded_as_garbage, checkpoint_skips_cold_row_for_user_also_in_clusters_no_demotion. The torn-row and dimension-mismatch cases matter: a checkpoint is read after a crash, so "garbage in the tail" is the expected input, not the exceptional one.
tidal/tests/m12_preference_event_time.rs (1) — event-time anchoring.
tidal/benches/multi_preference.rs.

Side effect worth recording

This work incidentally closed a limitation M6 had booked against M7: per-user preference update_count is now persisted and restored (see milestone-6/phase-4.md and milestone-6/phase-6.md). The # Known Limitation comment in tidal/src/entities/preference.rs:37-45 still claims otherwise and is stale.

Deferred

The offline medoid-recluster tier — periodically re-deriving cluster centroids from stored engagements rather than only updating them online — is recorded in the ROADMAP as an open follow-up. Online sequential k-means is order-dependent; a recluster pass is what would remove that dependence. Not built.

4.3 KiB Raw Blame History Unescape Escape