tidaldb/docs/planning/milestone-3/phase-3/OVERVIEW.md

Milestone 3, Phase 3: Personalized Ranking Profiles

Phase Deliverable

Four personalized ranking profiles (for_you, following, related, notification) that incorporate user context into scoring, plus cold-start handling for new users and new items. The FOR USER @user_id clause in the query parser is parsed and resolved into a UserContext that loads the user's preference vector, interaction weights, followed creators, and blocked state. The profile executor uses this context to score candidates with personalization factors.

Before m3p3, all ranking profiles operate on population-level signals (trending, hot, new, etc.). After m3p3, profiles can weight candidates by how well they match the user's learned preferences, boost items from creators the user frequently engages with, and inject exploration candidates from unfollowed creators to prevent filter bubbles.

Acceptance Criteria

• FOR USER @user_id clause parsed by the query parser
• UserContext loaded from UserStateIndex, InteractionWeightLedger, preference vector
• for_you profile: ANN retrieval using user preference vector, scoring = preference_match * engagement * recency * social_proof, 10% exploration budget
• following profile: candidates restricted to followed creators, sorted by created_at DESC
• related profile: ANN retrieval using source item embedding, re-ranked by user preference
• notification profile: candidates from followed creators' recent items, scored by relationship_strength * item_quality
• Cold-start users (no preference vector): fall back to population-level signals (trending/quality)
• Cold-start items (no signals): exploration window -- appear in ~2% of for_you feeds
• Exploration budget: ~5 of 50 for_you results from unfollowed creators
• ProfileExecutor extended with score_with_user_context() method
• Profile registration: for_you, following, related, notification added to ProfileRegistry as builtins

Dependencies

• Requires: m3p2 (feedback loop: preference vectors populated, interaction weights updated, user state bitmaps maintained), m2p3 (ranking profile engine, ProfileExecutor), m2p5 (query parser, RETRIEVE executor), m2p1 (vector index for ANN retrieval with user preference vector)
• Blocks: m3p4 (User State Filters need FOR USER parsing to inject user context into filter evaluation)

Research References

• docs/research/ann_for_tidaldb.md -- ANN retrieval with user preference vector as query
• VISION.md -- Ranking profiles, personalization factors
• USE_CASES.md -- UC-01 (For You), UC-04 (Following), UC-05 (Related), UC-07 (Notifications)

Task Index

#	Task	Delivers	Depends On	Complexity
01	FOR USER Query Context	UserContext loader, query parser extension for FOR USER, planner integration	None	M
02	Personalized Profiles	for_you, following, related, notification profiles, executor extensions	Task 01	L
03	Cold Start and Exploration	Cold-start fallback, exploration budget injection, new-item exploration window	Task 02	M

Task Dependency DAG

Task 01: FOR USER Query Context
    |
    v
Task 02: Personalized Profiles
    |
    v
Task 03: Cold Start and Exploration

All three tasks are sequential: Task 02 needs the user context from Task 01, and Task 03 needs the profiles from Task 02 to inject exploration candidates.

File Layout

tidal/src/
  db/
    user_context.rs  -- UserContext loader, query context resolution (Task 01)
  query/
    mod.rs           -- Extended parser with FOR USER clause (Task 01)
  ranking/
    personalized.rs  -- Personalized scoring functions, profile definitions (Task 02)
    exploration.rs   -- Cold-start fallback, exploration budget (Task 03)
    builtins.rs      -- Extended with for_you, following, related, notification (Task 02)
tidal/tests/
  m3p3_personalized.rs -- Phase integration tests

Open Questions

1. Exploration budget implementation: Should exploration candidates be selected randomly from the full corpus, or from a "new items" pool? Recommendation: random selection from the full corpus minus followed creators' items. This maximizes serendipity. New-item exploration is a separate budget within the exploration slice.

2. Social proof signal: How should "social proof" (engagement from followed creators' followers) be implemented? For M3, social proof is approximated by the item's population-level engagement signals (view velocity, like count). True social graph traversal ("trending among my follows' follows") is deferred to M6.

3. SIMILAR TO clause: The related profile needs a source item for ANN retrieval. Should SIMILAR TO @item_id be a separate parser clause, or embedded in the profile configuration? Recommendation: separate clause (RETRIEVE items SIMILAR TO @item_abc USING PROFILE related ...). This keeps the profile generic and the source item explicit.

4. Notification frequency capping: Should the notification profile enforce per-creator notification limits (e.g., max 3 per creator per day)? Recommendation: deferred to M6. For M3, the notification profile ranks by recency * relationship strength without capping.