39ada28c6e
M2: RETRIEVE query pipeline with 5-stage execution (candidate → filter → score → diversify → limit),
usearch HNSW vector index, bitmap/range/universe filters, ranking profiles with signal scoring,
MMR diversity enforcement, and m2_uat integration tests.
M3: Entity system with typed metadata, relationship graph (follows/blocks/interactions),
creator entities, session tracking, and m3_uat integration tests.
M4: Advanced ranking with builtin functions (freshness, trending, controversy, wilson),
ranking executor with explain mode, query executor integration, benchmarks for
query/ranking/vector/filters/diversity, and m4_uat integration tests.
Includes: 9 new blog posts, marketing site updates, updated roadmap, and updated vision doc.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
74 lines
4.3 KiB
Markdown
74 lines
4.3 KiB
Markdown
# Milestone 3, Phase 4: User State Filters + M3 UAT Integration Test
|
|
|
|
## Phase Deliverable
|
|
|
|
Composable user-state filters (`unseen`, `unblocked`, `saved`, `liked`, `in_progress`) that integrate with the existing `FilterExpr` / `FilterResult` system from m2p2, plus the end-to-end M3 UAT integration test that proves the full "For You" query works. User-state filters require the `FOR USER` clause (from m3p3 Task 01) to resolve user context and are evaluated alongside metadata filters during the RETRIEVE pipeline.
|
|
|
|
This phase is the final deliverable of Milestone 3. After m3p4, the query `RETRIEVE items FOR USER @user_id USING PROFILE for_you FILTER unseen, unblocked DIVERSITY max_per_creator:2 LIMIT 50` executes correctly, returns personalized results, excludes seen/blocked/hidden items, enforces diversity, includes exploration candidates, and reflects signal events written 100ms ago.
|
|
|
|
## Acceptance Criteria
|
|
|
|
- [ ] `FILTER unseen` excludes items the user has viewed
|
|
- [ ] `FILTER unblocked` excludes items from blocked creators and hidden items
|
|
- [ ] `FILTER saved` returns only items the user has saved
|
|
- [ ] `FILTER liked` returns only items the user has liked
|
|
- [ ] `FILTER in_progress` returns items with partial completion signal
|
|
- [ ] User-state filters compose with metadata filters: `FILTER unseen, category:jazz, format:video`
|
|
- [ ] User-state filters require `FOR USER` clause; used without it returns an error
|
|
- [ ] `FilterExpr` extended with `Unseen`, `Unblocked`, `Saved`, `Liked`, `InProgress` variants
|
|
- [ ] Filter evaluation produces `FilterResult::Predicate` for user-state filters (not bitmap)
|
|
- [ ] The RETRIEVE executor intersects user-state predicates with metadata filter bitmaps
|
|
- [ ] Full M3 UAT integration test passes (see Task 02)
|
|
|
|
## Dependencies
|
|
|
|
- **Requires:** m3p3 (personalized profiles, `FOR USER` query context, cold-start handling), m3p2 (feedback loop: seen bitmaps populated, hard negatives enforced), m3p1 (user state index, relationships), m2p2 (filter engine, `FilterExpr`, `FilterResult`)
|
|
- **Blocks:** Nothing (this is the final M3 phase)
|
|
|
|
## Research References
|
|
|
|
- [VISION.md](../../../../VISION.md) -- User-state filters as first-class query primitives
|
|
- [USE_CASES.md](../../../../USE_CASES.md) -- Filter reference (Appendix A: unseen, unblocked, saved, liked)
|
|
- [API.md](../../../../API.md) -- FILTER clause syntax
|
|
|
|
## Task Index
|
|
|
|
| # | Task | Delivers | Depends On | Complexity |
|
|
|---|------|----------|------------|------------|
|
|
| 01 | User State Filters | `Unseen`, `Unblocked`, `Saved`, `Liked`, `InProgress` filter variants, executor integration | None | M |
|
|
| 02 | M3 UAT Integration Test | End-to-end test proving the full M3 scenario | Task 01 | L |
|
|
|
|
## Task Dependency DAG
|
|
|
|
```
|
|
Task 01: User State Filters
|
|
|
|
|
v
|
|
Task 02: M3 UAT Integration Test
|
|
```
|
|
|
|
Task 02 depends on Task 01 because the UAT test exercises user-state filters as part of the full query.
|
|
|
|
## File Layout
|
|
|
|
```
|
|
tidal/src/
|
|
storage/
|
|
indexes/
|
|
filter.rs -- Extended with Unseen, Unblocked, Saved, Liked, InProgress variants
|
|
user_filter.rs -- User-state filter evaluation logic (new file)
|
|
query/
|
|
mod.rs -- Parser extended to recognize user-state filter keywords
|
|
tidal/tests/
|
|
m3p4_user_filters.rs -- Task 01 integration tests
|
|
m3_uat.rs -- Task 02 full M3 UAT integration test
|
|
```
|
|
|
|
## Open Questions
|
|
|
|
1. **Saved/liked bitmap maintenance**: The `saved` and `liked` filters require per-user bitmaps of saved/liked item IDs. These bitmaps are maintained by signal dispatch (m3p2) when "save" and "like" signals are written. Should a "save" signal also mark the item as "seen"? Recommendation: yes -- saving an item implies the user has seen it. The `mark_seen` call is made alongside the save bitmap update.
|
|
|
|
2. **in_progress threshold**: What constitutes "in progress"? Recommendation: an item is in_progress if it has a completion signal with value > 0.0 and < 0.8 (i.e., started but not finished). The threshold is configurable. Items with completion >= 0.8 are considered "completed" and excluded from in_progress.
|
|
|
|
3. **Filter error for missing FOR USER**: When `FILTER unseen` is used without `FOR USER`, should the query fail with an error or silently ignore the filter? Recommendation: return a `LumenError::Query` error with message "FILTER unseen requires FOR USER clause". Silent ignoring would produce confusing results.
|