fe8d0c87e7
Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement). Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release gate past its budget. Both push-path steps now declare backend_options.kubernetes.resources as two YAML anchors declared once on their first consuming step. The values are CALIBRATED against measured free node capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's original figure) fits on NO node and would sit Pending forever, because `ci-build-bounds` grants permission and the nodes supply capacity, and those are not the same thing. The `nightly` cron described in this file for 216 days was never created, so tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero signal while reading like standing coverage. nightly-chaos and nightly-security-ops now alias the anchors and have budgets matching the gate (their 120/90 were TIGHTER on the same runner, so they would have failed nightly for a budget reason, not a correctness one). nightly-soak is REMOVED, not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best node has 1700m free CPU, so it would fail on starvation rather than regression — manufacturing a nightly false alarm. Its commands move verbatim to docs/runbooks/nightly-soak.md. Ring 1 — four fabrications removed from the wire. - scatter_merge sorted and truncated without re-stamping rank, so /feed and /search returned 1,1,2 under full placement. Reuses merge_cross_shard's existing stamp; asserted on BOTH the multi-group merge path and the single-group [only] fast path that bypasses it. - aggregate_region_row's None arm invented `applied_events: 0` plus a deficit derived from it. applied_events/lag_events are now Option<u64>, null on the wire. leader_last_seq was also unwrap_or(0), so a node that could not reach the LEADER computed 0 - applied = 0 for every region and reported a converged cluster it had never measured — a fabrication pointing the dangerous way. - tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was actively hiding the PVC-wipe shape: a measured zero with a real deficit rendered as "no report" instead of BEHIND. Now read off the wire; converged exits 0, partitioned still exits nonzero. - /sharded/* answered 201/204 for single-copy writes with nothing anywhere saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the existing invalid_input path. Six call sites migrated, not the two this roadmap predicted — including docs/runbooks/cluster.md §16.3, which told operators to run a quorum-write probe via POST /sharded/items. That probe cannot verify quorum: the surface applies locally with no WAL append. It was used as the safety check between every step of a staged deploy earlier today. Ring 2 — observability. JSON_LOGS was already implemented and the deployment simply never asked for it; the StatefulSet now sets it, plus TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the VictoriaLogs `service` stream field and would have blinded every query keyed on it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only the origin would mean each vector lands on exactly one node, replicas never agree, and the alert built on it pages forever. Found by measurement, not planned: the 401 path discarded every fact about every rejection. Traefik has served 101,858 rejected requests to the public ingress — 87.6% of all its traffic — with no record of who or why anywhere. unauthorized_response now emits reason (missing_token vs invalid_token, the distinction that separates a scanner from a rotation that missed a consumer) and the forwarded client. The token is never logged. Also: scripts/restore-fleet.sh --cluster started the soak monitor while deliberately leaving its gate suspended, orphaning a watcher that has reported "0/30 green nights" for 13 days. The pair now moves together. Doc-guard's three-warning backlog is cleared with real backfill for M4/M6/M12. Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline, counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5; cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with the two remaining failures asserting the rank fix against the not-yet-rolled image — they are the post-deploy proof.
3.5 KiB
3.5 KiB
m6p2 — Social Graph Extension + Collaborative Filtering (✅ COMPLETE 2026-02-23)
Phase spec and acceptance criteria: ROADMAP · Milestone 6 · Phase 2. Milestone index: README.md. Backfilled record.
What shipped
- Reverse relationship index. Given a creator, retrieve their inbound
follower set (
RoaringBitmap-backed), maintained on every relationship write and persisted so it survives restart. FilterExpr::social_graph(user_id, depth)(tidal/src/query/executor/social_filter.rs,tidal/src/storage/indexes/filter/expr.rs). Depth 1 constrains candidates to items from followed creators; depth 2 expands through the resolved follow graph. When combined with a trending profile, velocity reads are scoped to the resolved subgraph (tidal/src/ranking/executor/scoring.rs).- Co-engagement index (
tidal/src/entities/co_engagement.rs). On a positive engagement (like, or completion ≥ 0.8) pairwise edges are recorded between the engaged item and the user's lastUSER_RECENT_CAPACITY = 50positively engaged items; edge weight increments per co-occurrence. Edges are asymmetric —(A,B)and(B,A)are separate entries, and scoring keys on(seed_item, candidate). Bounded atDEFAULT_CO_ENGAGEMENT_CAPACITY = 50_000pairs. - Collaborative-filtering boost in
related. The ranking executor takes an optionalCoEngagementIndex(with_co_engagement,tidal/src/ranking/executor/mod.rs:212) and folds a co-engagement term into the score forrelated-style queries, with aco_engagemententry in the reason snapshot so the contribution is explainable.
Evidence
tidal/tests/m6_social.rs— 8#[test]functions.tidal/benches/social.rs— social-graph benchmarks added by the same commit; numbers recorded in docs/profiling/social-scale.md.tidal/tests/m7p3_social_scale.rs— M7 re-tests this surface at scale.
Divergence from the plan — two, both self-documented in source
- Eviction is minimum-weight, not LRU. The ROADMAP (and the
0.1.0CHANGELOG entry) say "LRU eviction". The shipped policy is a weight-based batch eviction: when the edge count exceeds capacity, edges are removed in ascending weight order until the count is back at capacity. The module header states the reasoning explicitly and says it "is NOT a true LRU policy" — strongest co-occurrence edges carry the most recommendation value (Sarwar et al., 2001), so signal strength beats recency here. The behaviour is deliberate and better justified than the plan; only the plan's wording is wrong. - The
relatedblend is additive, not a convex combination. The ROADMAP criterion readsfinal = embedding_sim × 0.6 + co_engagement × 0.3 + signal_score × 0.1. The effective formula isbase_signal_score + boost_sum + co_eng_score × 0.3(tidal/src/ranking/executor/mod.rs:775-796). The comment there explains why: embedding similarity is not available as a per-candidate scoring signal because ANN retrieval uses the embedding for candidate selection, not re-scoring. Foldingembedding_simin multiplicatively needs per-candidate ANN distances plumbed through the executor context. That plumbing arrived with M12 (for_you/relatedANN candidate generation, see milestone-12/phase-2.md), so the blend is now revisitable — it has not been revisited.