tidaldb/docs/planning/milestone-3/phase-4/OVERVIEW.md

Milestone 3, Phase 4: User State Filters + M3 UAT Integration Test

Phase Deliverable

Composable user-state filters (unseen, unblocked, saved, liked, in_progress) that integrate with the existing FilterExpr / FilterResult system from m2p2, plus the end-to-end M3 UAT integration test that proves the full "For You" query works. User-state filters require the FOR USER clause (from m3p3 Task 01) to resolve user context and are evaluated alongside metadata filters during the RETRIEVE pipeline.

This phase is the final deliverable of Milestone 3. After m3p4, the query RETRIEVE items FOR USER @user_id USING PROFILE for_you FILTER unseen, unblocked DIVERSITY max_per_creator:2 LIMIT 50 executes correctly, returns personalized results, excludes seen/blocked/hidden items, enforces diversity, includes exploration candidates, and reflects signal events written 100ms ago.

Acceptance Criteria

• FILTER unseen excludes items the user has viewed
• FILTER unblocked excludes items from blocked creators and hidden items
• FILTER saved returns only items the user has saved
• FILTER liked returns only items the user has liked
• FILTER in_progress returns items with partial completion signal
• User-state filters compose with metadata filters: FILTER unseen, category:jazz, format:video
• User-state filters require FOR USER clause; used without it returns an error
• FilterExpr extended with Unseen, Unblocked, Saved, Liked, InProgress variants
• Filter evaluation produces FilterResult::Predicate for user-state filters (not bitmap)
• The RETRIEVE executor intersects user-state predicates with metadata filter bitmaps
• Full M3 UAT integration test passes (see Task 02)

Dependencies

• Requires: m3p3 (personalized profiles, FOR USER query context, cold-start handling), m3p2 (feedback loop: seen bitmaps populated, hard negatives enforced), m3p1 (user state index, relationships), m2p2 (filter engine, FilterExpr, FilterResult)
• Blocks: Nothing (this is the final M3 phase)

Research References

• VISION.md -- User-state filters as first-class query primitives
• USE_CASES.md -- Filter reference (Appendix A: unseen, unblocked, saved, liked)
• API.md -- FILTER clause syntax

Task Index

#	Task	Delivers	Depends On	Complexity
01	User State Filters	Unseen, Unblocked, Saved, Liked, InProgress filter variants, executor integration	None	M
02	M3 UAT Integration Test	End-to-end test proving the full M3 scenario	Task 01	L

Task Dependency DAG

Task 01: User State Filters
    |
    v
Task 02: M3 UAT Integration Test

Task 02 depends on Task 01 because the UAT test exercises user-state filters as part of the full query.

File Layout

tidal/src/
  storage/
    indexes/
      filter.rs      -- Extended with Unseen, Unblocked, Saved, Liked, InProgress variants
      user_filter.rs  -- User-state filter evaluation logic (new file)
  query/
    mod.rs           -- Parser extended to recognize user-state filter keywords
tidal/tests/
  m3p4_user_filters.rs -- Task 01 integration tests
  m3_uat.rs            -- Task 02 full M3 UAT integration test

Open Questions

1. Saved/liked bitmap maintenance: The saved and liked filters require per-user bitmaps of saved/liked item IDs. These bitmaps are maintained by signal dispatch (m3p2) when "save" and "like" signals are written. Should a "save" signal also mark the item as "seen"? Recommendation: yes -- saving an item implies the user has seen it. The mark_seen call is made alongside the save bitmap update.

2. in_progress threshold: What constitutes "in progress"? Recommendation: an item is in_progress if it has a completion signal with value > 0.0 and < 0.8 (i.e., started but not finished). The threshold is configurable. Items with completion >= 0.8 are considered "completed" and excluded from in_progress.

3. Filter error for missing FOR USER: When FILTER unseen is used without FOR USER, should the query fail with an error or silently ignore the filter? Recommendation: return a LumenError::Query error with message "FILTER unseen requires FOR USER clause". Silent ignoring would produce confusing results.