fe8d0c87e7
Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement). Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release gate past its budget. Both push-path steps now declare backend_options.kubernetes.resources as two YAML anchors declared once on their first consuming step. The values are CALIBRATED against measured free node capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's original figure) fits on NO node and would sit Pending forever, because `ci-build-bounds` grants permission and the nodes supply capacity, and those are not the same thing. The `nightly` cron described in this file for 216 days was never created, so tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero signal while reading like standing coverage. nightly-chaos and nightly-security-ops now alias the anchors and have budgets matching the gate (their 120/90 were TIGHTER on the same runner, so they would have failed nightly for a budget reason, not a correctness one). nightly-soak is REMOVED, not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best node has 1700m free CPU, so it would fail on starvation rather than regression — manufacturing a nightly false alarm. Its commands move verbatim to docs/runbooks/nightly-soak.md. Ring 1 — four fabrications removed from the wire. - scatter_merge sorted and truncated without re-stamping rank, so /feed and /search returned 1,1,2 under full placement. Reuses merge_cross_shard's existing stamp; asserted on BOTH the multi-group merge path and the single-group [only] fast path that bypasses it. - aggregate_region_row's None arm invented `applied_events: 0` plus a deficit derived from it. applied_events/lag_events are now Option<u64>, null on the wire. leader_last_seq was also unwrap_or(0), so a node that could not reach the LEADER computed 0 - applied = 0 for every region and reported a converged cluster it had never measured — a fabrication pointing the dangerous way. - tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was actively hiding the PVC-wipe shape: a measured zero with a real deficit rendered as "no report" instead of BEHIND. Now read off the wire; converged exits 0, partitioned still exits nonzero. - /sharded/* answered 201/204 for single-copy writes with nothing anywhere saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the existing invalid_input path. Six call sites migrated, not the two this roadmap predicted — including docs/runbooks/cluster.md §16.3, which told operators to run a quorum-write probe via POST /sharded/items. That probe cannot verify quorum: the surface applies locally with no WAL append. It was used as the safety check between every step of a staged deploy earlier today. Ring 2 — observability. JSON_LOGS was already implemented and the deployment simply never asked for it; the StatefulSet now sets it, plus TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the VictoriaLogs `service` stream field and would have blinded every query keyed on it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only the origin would mean each vector lands on exactly one node, replicas never agree, and the alert built on it pages forever. Found by measurement, not planned: the 401 path discarded every fact about every rejection. Traefik has served 101,858 rejected requests to the public ingress — 87.6% of all its traffic — with no record of who or why anywhere. unauthorized_response now emits reason (missing_token vs invalid_token, the distinction that separates a scanner from a rotation that missed a consumer) and the forwarded client. The token is never logged. Also: scripts/restore-fleet.sh --cluster started the soak monitor while deliberately leaving its gate suspended, orphaning a watcher that has reported "0/30 green nights" for 13 days. The pair now moves together. Doc-guard's three-warning backlog is cleared with real backfill for M4/M6/M12. Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline, counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5; cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with the two remaining failures asserting the rank fix against the not-yet-rolled image — they are the post-deploy proof.
60 lines
3.5 KiB
Markdown
60 lines
3.5 KiB
Markdown
# m6p2 — Social Graph Extension + Collaborative Filtering (✅ COMPLETE 2026-02-23)
|
||
|
||
Phase spec and acceptance criteria: [ROADMAP · Milestone 6 · Phase 2](../ROADMAP.md).
|
||
Milestone index: [README.md](README.md). Backfilled record.
|
||
|
||
## What shipped
|
||
|
||
1. **Reverse relationship index.** Given a creator, retrieve their inbound
|
||
follower set (`RoaringBitmap`-backed), maintained on every relationship write
|
||
and persisted so it survives restart.
|
||
2. **`FilterExpr::social_graph(user_id, depth)`**
|
||
(`tidal/src/query/executor/social_filter.rs`,
|
||
`tidal/src/storage/indexes/filter/expr.rs`). Depth 1 constrains candidates to
|
||
items from followed creators; depth 2 expands through the resolved follow
|
||
graph. When combined with a trending profile, velocity reads are scoped to the
|
||
resolved subgraph (`tidal/src/ranking/executor/scoring.rs`).
|
||
3. **Co-engagement index** (`tidal/src/entities/co_engagement.rs`). On a positive
|
||
engagement (like, or completion ≥ 0.8) pairwise edges are recorded between the
|
||
engaged item and the user's last `USER_RECENT_CAPACITY = 50` positively
|
||
engaged items; edge weight increments per co-occurrence. Edges are
|
||
**asymmetric** — `(A,B)` and `(B,A)` are separate entries, and scoring keys on
|
||
`(seed_item, candidate)`. Bounded at `DEFAULT_CO_ENGAGEMENT_CAPACITY = 50_000`
|
||
pairs.
|
||
4. **Collaborative-filtering boost in `related`.** The ranking executor takes an
|
||
optional `CoEngagementIndex` (`with_co_engagement`,
|
||
`tidal/src/ranking/executor/mod.rs:212`) and folds a co-engagement term into
|
||
the score for `related`-style queries, with a `co_engagement` entry in the
|
||
reason snapshot so the contribution is explainable.
|
||
|
||
## Evidence
|
||
|
||
- `tidal/tests/m6_social.rs` — 8 `#[test]` functions.
|
||
- `tidal/benches/social.rs` — social-graph benchmarks added by the same commit;
|
||
numbers recorded in [docs/profiling/social-scale.md](../../profiling/social-scale.md).
|
||
- `tidal/tests/m7p3_social_scale.rs` — M7 re-tests this surface at scale.
|
||
|
||
## Divergence from the plan — two, both self-documented in source
|
||
|
||
1. **Eviction is minimum-weight, not LRU.** The ROADMAP (and the `0.1.0`
|
||
CHANGELOG entry) say "LRU eviction". The shipped policy is a weight-based
|
||
batch eviction: when the edge count exceeds capacity, edges are removed in
|
||
ascending weight order until the count is back at capacity. The module header
|
||
states the reasoning explicitly and says it "is NOT a true LRU policy" —
|
||
strongest co-occurrence edges carry the most recommendation value
|
||
(Sarwar et al., 2001), so signal strength beats recency here. The behaviour is
|
||
deliberate and better justified than the plan; only the plan's wording is
|
||
wrong.
|
||
2. **The `related` blend is additive, not a convex combination.** The ROADMAP
|
||
criterion reads `final = embedding_sim × 0.6 + co_engagement × 0.3 +
|
||
signal_score × 0.1`. The effective formula is
|
||
`base_signal_score + boost_sum + co_eng_score × 0.3`
|
||
(`tidal/src/ranking/executor/mod.rs:775-796`). The comment there explains why:
|
||
embedding similarity is not available as a per-candidate scoring signal
|
||
because ANN retrieval uses the embedding for candidate *selection*, not
|
||
re-scoring. Folding `embedding_sim` in multiplicatively needs per-candidate ANN
|
||
distances plumbed through the executor context. That plumbing arrived with M12
|
||
(`for_you` / `related` ANN candidate generation, see
|
||
[milestone-12/phase-2.md](../milestone-12/phase-2.md)), so the blend is now
|
||
revisitable — it has not been revisited.
|