tidaldb/.sdlc/features/p1-quality-diversity-baseline/spec.md

# Spec: Quality & Diversity Baseline

## Problem

The daily "Today Brief" ranking surface can surface low-quality content (items with near-zero completion rates, spam-level engagement, or negligible scores) and can be dominated by a single prolific creator. Without quality gates and diversity enforcement active by default, the briefing experience degrades for pilot users, undermining the P1 concierge alpha's retention goals.

tidalDB already has the primitives — `Gate` (min-threshold filters on signal aggregations), `DiversityConstraints` (max-per-creator, format-mix), and the `DiversitySelector` with greedy multi-stage relaxation. What is missing is a **curated "brief" ranking profile** that wires these primitives into sensible defaults, and **integration tests** that prove the quality floor and diversity invariants hold under realistic conditions.

## Goals

1. **Quality floor on brief results.** Items in the brief must pass minimum engagement thresholds before they are eligible for ranking. Specifically:
   - A minimum view count gate ensures items have been seen by enough users to have a meaningful signal.
   - A minimum completion rate gate (via `completion` signal) ensures items are not abandoned content.
   - Items failing either gate are excluded before scoring, not after.

2. **Creator diversity in top results.** No single creator should dominate the brief. The brief profile must enforce a max-per-creator constraint tight enough that a brief of N items has content from at least `ceil(N / max_per_creator)` distinct creators (when sufficient creator variety exists).

3. **Format diversity in top results.** No single content format should dominate the brief beyond a configurable fraction (e.g., no more than 60% video in a 10-item brief).

4. **A named "brief" ranking profile** registered as a built-in profile that codifies these quality gates and diversity constraints. The profile is usable via `RETRIEVE items USING PROFILE brief` or `SearchBuilder::profile("brief")`.

5. **Integration tests** proving:
   - Items below the quality floor are excluded from brief results.
   - Creator diversity is enforced: multiple creators represented in results.
   - Format diversity is enforced: no format exceeds the `format_mix_max_fraction`.
   - The brief profile is registered and resolvable by name.

## Non-Goals

- Custom per-user quality thresholds (future personalization work).
- Dynamic threshold tuning based on catalog size (future operational work).
- Changing the `DiversitySelector` algorithm itself (already correct and well-tested).
- Exposing quality gate configuration via an API or schema DSL (profiles are code-defined for now).

## Existing Primitives

### Gates (`ranking::profile::Gate`)
Already implemented and enforced in the scoring pipeline. `passes_gates()` in `ranking/executor/helpers.rs` evaluates each gate before a candidate enters the scoring loop. Gates check `agg(signal, window) >= min_threshold`. This is exactly what we need for the quality floor.

### DiversityConstraints / DiversitySelector
Already implemented with greedy multi-stage relaxation in `ranking/diversity/`. Enforces `max_per_creator` and `format_mix_max_fraction`. Applied as Stage 5 of the scoring pipeline. Both the `Retrieve` builder (query-level override) and the `RankingProfile` (profile-level default) can specify diversity constraints.

### Built-in Profiles
25 built-in profiles registered in `ranking/builtins.rs`. The new "brief" profile follows the same pattern.

## Design

### "brief" Profile Definition

```
Name: brief
Version: 1
Sort: Hot { gravity: 1.5 }
Candidate Strategy: Scan { sort_field: "created_at" }
Gates:
  - view / Value / AllTime >= 3.0   (min 3 total views)
  - completion / Value / AllTime >= 1.0   (at least 1 completion)
Boosts:
  - view / DecayScore / AllTime * 1.0
  - like / DecayScore / AllTime * 2.0
  - completion / DecayScore / AllTime * 1.5
Diversity:
  - max_per_creator: 2
  - format_mix_max_fraction: 0.6
Exploration: 0.0  (disabled — exploration injects candidates that bypass quality gates)
```

Rationale:
- **View gate >= 3**: Items with fewer than 3 views have insufficient signal to rank meaningfully. This is a low bar appropriate for an alpha with a small catalog.
- **Completion gate >= 1**: At least one user must have completed (or substantially engaged with) the item. Filters out abandoned/broken content.
- **max_per_creator: 2**: In a typical 10-item brief, this ensures at least 5 distinct creators. Tight enough to prevent domination, loose enough to allow a standout creator to appear twice.
- **format_mix: 0.6**: No single format exceeds 60% of the brief. In a 10-item brief, max 6 of one format.
- **Exploration: 0.0**: Disabled because `inject_exploration()` runs after gate filtering and can re-introduce candidates that failed quality gates, undermining the curated-brief guarantee. Exploration is appropriate for discovery surfaces (`for_you`) but not for quality-gated surfaces.
- **Hot sort with gravity 1.5**: Balances recency with engagement, appropriate for a daily brief cadence.

### Implementation Notes

Two additional fixes were required in the scoring pipeline to make diversity constraints effective:

1. **Format enrichment**: `ScoredCandidate.format` was never populated from item metadata, making `format_mix_max_fraction` enforcement impossible. Fixed by extending the metadata enrichment loop in `query/executor/mod.rs` to populate both `creator_id` and `format`.

2. **Metadata loading trigger**: The `needs_metadata_for_creator_grouping` flag only checked `max_per_creator`, not `format_mix_max_fraction`. Extended to check both, ensuring metadata is loaded when format diversity is configured.

## Acceptance Criteria

- [x] A `brief` built-in ranking profile exists and is registered at startup.
- [x] Items with fewer than 3 total views are excluded from brief results.
- [x] Items with 0 completions are excluded from brief results.
- [x] Multiple creators represented in brief results (no single creator dominates).
- [x] No single format exceeds 60% of a brief result set.
- [x] All existing tests continue to pass (no regressions in the 25 existing profiles).
- [x] Integration tests demonstrate all scenarios above.
- [x] `cargo clippy -D warnings` and `cargo fmt` pass cleanly.