158 lines
7.3 KiB
Markdown
158 lines
7.3 KiB
Markdown
# Design: Quality & Diversity Baseline
|
|
|
|
## Overview
|
|
|
|
This is a backend-only change. No UI, no API surface change, no schema migration. The deliverable is a new built-in ranking profile (`brief`) and integration tests proving quality and diversity invariants.
|
|
|
|
## Architecture
|
|
|
|
All changes are confined to three areas:
|
|
|
|
```
|
|
tidal/src/ranking/builtins.rs ← new profile definition + registration
|
|
tidal/src/query/executor/mod.rs ← format enrichment fix + metadata loading trigger
|
|
tidal/tests/p1_quality_diversity.rs ← integration tests
|
|
```
|
|
|
|
No new modules, no new types, no new traits. The existing `Gate`, `DiversitySpec`, `Boost`, and `Sort` types are sufficient.
|
|
|
|
## Profile Definition
|
|
|
|
```
|
|
ranking/builtins.rs
|
|
└── fn brief() -> RankingProfile
|
|
├── sort: Hot { gravity: 1.5 }
|
|
├── gates:
|
|
│ ├── Gate { signal: "view", agg: Value, window: AllTime, min_threshold: 3.0 }
|
|
│ └── Gate { signal: "completion", agg: Value, window: AllTime, min_threshold: 1.0 }
|
|
├── boosts:
|
|
│ ├── Boost { signal: "view", agg: DecayScore, window: AllTime, weight: 1.0 }
|
|
│ ├── Boost { signal: "like", agg: DecayScore, window: AllTime, weight: 2.0 }
|
|
│ └── Boost { signal: "completion", agg: DecayScore, window: AllTime, weight: 1.5 }
|
|
├── diversity:
|
|
│ ├── max_per_creator: 2
|
|
│ └── format_mix_max_fraction: 0.6
|
|
└── exploration: 0.0 (disabled — bypasses quality gates)
|
|
```
|
|
|
|
### Gate Behavior
|
|
|
|
Gates are evaluated in `passes_gates()` (in `ranking/executor/helpers.rs`) which runs **before** scoring. This means:
|
|
|
|
1. `view / Value / AllTime >= 3.0` — reads the AllTime windowed count for "view". Items with fewer than 3 total views are excluded before any scoring happens.
|
|
2. `completion / Value / AllTime >= 1.0` — reads the AllTime windowed count for "completion". Items with zero completions are excluded.
|
|
|
|
Both gates use `DegradationLevel::Full` (never coarsened), ensuring gate evaluation precision is never sacrificed under load.
|
|
|
|
**Schema dependency:** Gates reference signals `view` and `completion`. If either signal is not in the schema, `read_agg` returns 0.0, which fails the gate threshold, meaning all items are excluded. This is the correct behavior — a brief without view/completion signals has no quality signal to rank on and should surface nothing rather than surface garbage.
|
|
|
|
### Exploration Disabled
|
|
|
|
Exploration is set to 0.0 because `inject_exploration()` runs **after** gate filtering and injects random candidates from the unscored universe. These candidates bypass quality gates, defeating the curated-brief guarantee. This was discovered during integration testing.
|
|
|
|
### Diversity Behavior
|
|
|
|
Diversity is enforced by the existing `DiversitySelector` in Stage 5 of the scoring pipeline:
|
|
|
|
1. **max_per_creator: 2** — Greedy selection skips items from any creator that already has 2 items in the result set. Multi-stage relaxation ensures the result count invariant (INV-RANK-5) holds.
|
|
2. **format_mix_max_fraction: 0.6** — No single format can exceed 60% of results. For a 10-item brief, max 6 items of one format.
|
|
|
|
These are profile-level defaults. Callers can override via `RetrieveBuilder::diversity()` for query-level control.
|
|
|
|
### Pipeline Fixes
|
|
|
|
Two fixes in the scoring pipeline were required:
|
|
|
|
1. **Format enrichment** (`query/executor/mod.rs`): The metadata enrichment loop only populated `ScoredCandidate.creator_id`. Extended to also populate `ScoredCandidate.format` from item metadata's "format" field.
|
|
|
|
2. **Metadata loading trigger** (`query/executor/mod.rs`): `needs_metadata_for_creator_grouping` only checked `max_per_creator`. Extended to also check `format_mix_max_fraction`, ensuring metadata is loaded when format diversity is configured.
|
|
|
|
### Scoring Formula
|
|
|
|
The sort mode is `Hot { gravity: 1.5 }`, which uses the Reddit/HN age-decay formula. This balances recency with engagement — appropriate for a daily brief cadence where yesterday's high-engagement content should still appear but not dominate.
|
|
|
|
Three boosts add quality signals:
|
|
- `view * 1.0` — baseline popularity
|
|
- `like * 2.0` — quality signal (2x weight reflects explicit positive engagement)
|
|
- `completion * 1.5` — deeper engagement signal
|
|
|
|
## Registration
|
|
|
|
In `register_builtins()`:
|
|
|
|
```rust
|
|
registry.register(brief())?; // added after date_saved()
|
|
```
|
|
|
|
Total built-in profiles: 27 (includes profiles from parallel features).
|
|
|
|
## Integration Test Design
|
|
|
|
`tidal/tests/p1_quality_diversity.rs`:
|
|
|
|
### Test 1: `quality_gate_excludes_low_view_items`
|
|
- Create schema with `view`, `completion`, `like` signals.
|
|
- Write 10 items: 5 with >= 3 views + >= 1 completion, 5 with 1-2 views.
|
|
- Retrieve using `brief` profile.
|
|
- Assert: only the 5 high-quality items appear.
|
|
|
|
### Test 2: `quality_gate_excludes_zero_completion_items`
|
|
- Write 10 items: all have >= 3 views, but 5 have 0 completions.
|
|
- Retrieve using `brief` profile.
|
|
- Assert: only the 5 items with completions appear.
|
|
|
|
### Test 3: `creator_diversity_enforced`
|
|
- Write 30 items from 5 creators (10 from creator 1, 5 each from creators 2-5).
|
|
- All pass quality gates.
|
|
- Retrieve with limit 10.
|
|
- Assert: no single creator has all 10 items; at least 2 creators represented.
|
|
|
|
### Test 4: `format_diversity_enforced`
|
|
- Write 12 items: 6 video, 6 podcast, all different creators, all pass quality gates.
|
|
- Retrieve with limit 10.
|
|
- Assert: no format exceeds 60%.
|
|
|
|
### Test 5: `brief_profile_registered`
|
|
- Open DB, retrieve with `brief` profile on empty DB.
|
|
- Assert: no error (profile exists), empty results.
|
|
|
|
### Test 6: `combined_quality_and_diversity`
|
|
- 13 items: 10 high-quality from 5 creators, 3 low-quality.
|
|
- Retrieve brief with limit 10.
|
|
- Assert all constraints: quality gates exclude low-quality, creator diversity <= 2 per creator, format diversity <= 60%.
|
|
|
|
## Data Flow
|
|
|
|
```
|
|
RETRIEVE items USING PROFILE brief LIMIT 10
|
|
│
|
|
├─ Stage 1: Candidate Generation (Scan)
|
|
│ └─ All items from universe
|
|
│
|
|
├─ Stage 2: Filter Evaluation
|
|
│ └─ User-level filters (unseen, unblocked, etc.)
|
|
│
|
|
├─ Stage 3: Signal Scoring
|
|
│ ├─ Gate: view/Value/AllTime >= 3 → exclude if below
|
|
│ ├─ Gate: completion/Value/AllTime >= 1 → exclude if below
|
|
│ ├─ Base: Hot(gravity=1.5) scoring
|
|
│ ├─ Boost: view * 1.0 + like * 2.0 + completion * 1.5
|
|
│ └─ Normalize [0, 1]
|
|
│
|
|
├─ Stage 5: Diversity
|
|
│ ├─ max_per_creator: 2
|
|
│ └─ format_mix: 0.6
|
|
│
|
|
└─ Stage 6: Result Assembly
|
|
└─ Top 10, cursor, stats
|
|
```
|
|
|
|
## Risks & Mitigations
|
|
|
|
| Risk | Mitigation |
|
|
|------|-----------|
|
|
| Schema missing `view` or `completion` signal | Gates return 0.0 for unknown signals, excluding all items. This is correct — no quality signal = no brief. Integration tests verify this is the expected behavior. |
|
|
| Small catalog where all items are from same creator | DiversitySelector's multi-stage relaxation fills to target count even when constraints are unsatisfiable. The result set reports `constraints_satisfied: false` but still returns results. |
|
|
| Gate thresholds too aggressive for small alpha catalog | Thresholds are deliberately low (3 views, 1 completion). Can be tuned by registering a custom profile with adjusted gates. |
|
|
| Exploration re-introduces gate-failing candidates | Exploration disabled (0.0) for this profile. Discovery surfaces use separate profiles with exploration enabled. |
|