# Milestone 2, Phase 3: Ranking Profile Engine

## Phase Deliverable

Named ranking profiles declared as runtime data (not compiled code), stored in the schema, parsed, validated, and executed by the database. Profiles reference signal decay scores, windowed aggregates, velocity, and metadata fields. They define quality gates, boosts, penalties, and candidate generation strategies. Profiles are versioned and swappable at query time without recompile. The executor takes a profile and a candidate set and produces a scored, sorted result list in under 10 microseconds for 200 candidates.

This is the phase that turns signals from "primitives the application reads" into "primitives the database scores over." After this phase, a developer can name a profile and get ranked results -- the database does the math, not the application.

## Acceptance Criteria

- [ ] `RankingProfile` struct: name, version, candidate_strategy, scoring_rules (boosts, penalties, gates, excludes), sort mode override, diversity config, exploration budget
- [ ] `ScoringRule` enum: `Boost { signal, window, aggregation, weight }`, `Gate { condition, threshold }`, `Penalize { signal, window, weight }`, `Exclude { condition }`
- [ ] `Sort` enum with formula variants: `Hot { gravity }`, `Controversial`, `HiddenGems`, `New`, `Shuffle { seed }`, `TopWindow { window }`, `MostViewed`, `MostLiked`, `Rising`
- [ ] `CandidateStrategy` enum: `Ann`, `Scan`, `Hybrid`, `Relationship`, `CohortTrending` (M2 implements only `Scan`; others are type stubs used by profiles but not executed until their retrieval strategy is built)
- [ ] `ProfileRegistry` maps profile name to versioned `RankingProfile` instances, supports `get`, `get_versioned`, `register`, `list`
- [ ] Profile validation: duplicate names rejected, unknown signal references rejected (if schema available), gate threshold range [0.0, 1.0], weight normalization warning, version monotonicity (INV-PROF-1)
- [ ] Profiles serializable via serde for schema checkpoint/reload
- [ ] Built-in profiles registered at `SchemaBuilder::build()` time: `trending`, `hot`, `new`, `top_week`, `top_month`, `top_all_time`, `hidden_gems`, `controversial`, `most_viewed`, `most_liked`, `shuffle`
- [ ] Built-in profiles are standard `RankingProfile` instances -- not special-cased in the executor
- [ ] Built-in profiles with unavailable signals degrade gracefully (skip missing signals, not fatal error)
- [ ] `hot` formula: `log10(max(|positive - negative|, 1)) / (age_hours + 2)^gravity` with configurable gravity (default 1.8) -- Spec 09 Section 11.1
- [ ] `controversial` formula: `(positive * negative) / (positive + negative)^2` -- Spec 09 Section 11.4
- [ ] `hidden_gems` formula: `quality_score * (1 / log10(view_count + 10))` -- Spec 09 Section 11.5
- [ ] `ProfileExecutor::score()` takes `&[EntityId]` candidates and `&RankingProfile`, returns `Vec<ScoredCandidate>` sorted by score descending
- [ ] `ScoredCandidate` includes: entity_id, score, signal_snapshot (key signal values used for scoring transparency)
- [ ] Gate failure sets score to 0.0; candidates with score 0.0 are filtered out before returning
- [ ] Shuffle profile uses deterministic seeded RNG (stable per user_id + profile_name + page_cursor)
- [ ] Profile change does not require recompile -- profiles are runtime data
- [ ] 200-candidate scoring pass with decay-only profile < 10 microseconds, with velocity-based profile (trending) < 100 microseconds (both Criterion benchmarked)
- [ ] Deterministic scoring: same candidates + same profile + same signal state = identical results (INV-RANK-1)
- [ ] Normalized scores in [0.0, 1.0] after min-max normalization (INV-RANK-2)

## Dependencies

- **Requires:** m1p1 (types: `EntityId`, `EntityKind`, `SignalTypeDef`, `Window`, `WindowSet`, `DecayModel`, `Score`), m1p4 (SignalLedger: profiles read decay scores and windowed counts via `SignalLedger` API), m1p5 (entity read API: `TidalDb::read_decay_score`, `TidalDb::read_windowed_count`, `TidalDb::read_velocity`)
- **Blocks:** m2p4 (diversity enforcement takes scored lists from profile executor), m2p5 (RETRIEVE executor uses profiles to score candidates)

## Research References

- [docs/research/tidaldb_signal_ledger.md](../../../research/tidaldb_signal_ledger.md) -- Signal read latencies (~15ns decay score, ~200ns windowed count) that establish the per-candidate scoring budget
- [thoughts.md](../../../../thoughts.md) -- Part V.14 (cache-line alignment for hot-path structs), scoring pipeline architecture

## Spec References

- [docs/specs/09-ranking-scoring.md](../../../specs/09-ranking-scoring.md) -- THE authoritative spec for this phase:
  - Section 2 (ProfileDef structure, versioning, inheritance, A/B testing)
  - Section 3 (CandidateStrategy variants: ANN, Scan, Hybrid, Relationship, CohortTrending)
  - Section 4 (Scoring pipeline: 9-stage fixed-order transformation)
  - Section 5 (Boost types: signal, relationship, social proof, recency, cohort, preference match)
  - Section 6 (Penalty types: signal-based negative scoring)
  - Section 7 (Quality gates: minimum signal, ratio, count gates with exploration bypass)
  - Section 8 (Score composition: composite formula, min-max normalization, percentile signal normalization)
  - Section 11 (Built-in sort modes: Hot, Trending, Rising, Controversial, HiddenGems, Shuffle, Top windowed, simple field sorts)
  - Section 13 (Profile presets: for_you, trending, search, following, related, browse, hidden_gems, notification, live, hot, rising, controversial)
  - Section 15 (Performance targets: total scoring pipeline < 500us for 200 candidates, per-candidate scoring ~1.5us)
  - Section 16 (Invariants INV-RANK-1 through INV-RANK-7, INV-PROF-1 through INV-PROF-3, property tests P1-P6)

## Task Index

| # | Task | Delivers | Depends On | Complexity |
|---|------|----------|------------|------------|
| 01 | Ranking Profile Type System | `RankingProfile`, `ScoringRule`, `Sort`, `CandidateStrategy`, `ProfileRegistry`, validation, serde | None | L |
| 02 | Built-in Profiles | All 11 built-in profile definitions as `RankingProfile` instances, signal dependency validation, graceful degradation for missing signals | Task 01 | M |
| 03 | Profile Executor + Benchmarks | `ProfileExecutor`, `ScoredCandidate`, `ShuffleExecutor`, sort formula implementations, min-max normalization, Criterion benchmarks | Task 01, Task 02 | L |

## Task Dependency DAG

```
Task 01: Ranking Profile Type System
    |
    v
Task 02: Built-in Profiles
    |
    +---> Task 03: Profile Executor + Benchmarks
    |        (also depends on Task 01)
```

Task 01 is the foundation -- it defines all types that Task 02 instantiates and Task 03 executes. Task 02 constructs the built-in profiles from Task 01 types. Task 03 requires both the types (Task 01) and the profiles (Task 02) to implement and benchmark execution.

## File Layout

```
tidal/src/
  ranking/
    mod.rs        -- pub use re-exports: RankingProfile, ScoringRule, Sort, CandidateStrategy,
                     ProfileRegistry, ProfileExecutor, ScoredCandidate
    profile.rs    -- RankingProfile struct, ScoringRule, Sort, CandidateStrategy,
                     SignalAgg, Boost, Gate, Penalty, Exclude, validation (Task 01)
    registry.rs   -- ProfileRegistry, built-in profile construction, signal dependency
                     checking (Task 01 registry types, Task 02 built-in definitions)
    executor.rs   -- ProfileExecutor, ScoredCandidate, score() method,
                     sort formula implementations (Task 03)
    shuffle.rs    -- ShuffleExecutor, seeded RNG (Task 03)
  lib.rs          -- (unchanged, already declares pub mod ranking)
tidal/benches/
  ranking.rs      -- Criterion benchmarks (Task 03)
```

## Open Questions

1. **`SmallRng` vs `rand_xoshiro`**: The shuffle profile needs a stable-per-session RNG seeded from `(user_id, profile_name, page_cursor)`. `SmallRng` from the `rand` crate is fast and seedable. `rand_xoshiro::Xoshiro256StarStar` is available via `rand_xoshiro`. Decision: use `SmallRng` for M2 -- it is already in `rand`'s dependency tree, performs well, and is reproducible given the same seed. Add `rand_xoshiro` only if `SmallRng` proves non-deterministic across platforms.

2. **`signal_snapshot` in ScoredCandidate**: The spec says results should include key signal values used in scoring for debugging (Spec 09 Section 4, Stage 10). For M2, include all signals referenced in the profile's scoring rules (typically 2-5 signals). Cap at 10 signal values per candidate. The snapshot is a `Vec<(String, f64)>` (signal name, value) rather than a HashMap to keep allocation small and ordering deterministic.

3. **Gate vs Exclude vs Penalize semantics**: Gate zeros the score (candidate excluded from results). Exclude physically removes the candidate before scoring (it never enters the pipeline). Penalize multiplies by a factor < 1. The executor filters out score <= 0.0 candidates before returning. For M2, `Exclude` variants (`signal("hide")`, `relationship("blocked")`) are type stubs -- the actual exclusion logic requires user state from M3. The executor skips Exclude rules when no user context is available.

4. **Profile versioning**: Version is a `u32` monotonic counter per profile name. `ProfileRegistry` keeps all versions per name (INV-PROF-1 requires monotonic increase). `get()` returns latest version. `get_versioned(name, version)` returns a specific version. For M2, no version pruning. Pruning deferred to M5+ for A/B testing lifecycle management.

5. **Built-in profiles with unavailable signals**: At M2, the schema may not define all signals that a built-in profile references (e.g., `trending` requires `share` velocity, but the UAT schema might only have `view` and `like`). Built-in profiles must be resilient: if a referenced signal type is not in the schema, that boost/penalty is silently skipped (contributes 0.0). A `tracing::warn!` is emitted at registration time listing missing signals. The profile is still registered and usable -- it just scores with fewer signals.

6. **Sort mode vs boost/penalty pipeline**: When a profile has a `sort` override (e.g., `Sort::Hot`), the sort formula replaces stages 4-5 (boost and penalty application) of the scoring pipeline (Spec 09 Section 11.9). Gates, exclusions, normalization, and diversity still apply. The executor must check for a sort override before running the boost/penalty loop.

7. **Candidate strategy as type stub**: `CandidateStrategy` variants are defined as types in Task 01 but not executed in m2p3. The executor receives a pre-generated `&[EntityId]` candidate set. Candidate generation is the responsibility of the RETRIEVE executor (m2p5), which calls the appropriate retrieval strategy (ANN, scan, etc.) and passes the results to the profile executor for scoring. The `CandidateStrategy` on the profile is informational -- it tells the RETRIEVE executor how to generate candidates, but the profile executor itself does not generate candidates.
