tidaldb/docs/planning/milestone-2/phase-3/OVERVIEW.md

Milestone 2, Phase 3: Ranking Profile Engine

Phase Deliverable

Named ranking profiles declared as runtime data (not compiled code), stored in the schema, parsed, validated, and executed by the database. Profiles reference signal decay scores, windowed aggregates, velocity, and metadata fields. They define quality gates, boosts, penalties, and candidate generation strategies. Profiles are versioned and swappable at query time without recompile. The executor takes a profile and a candidate set and produces a scored, sorted result list in under 10 microseconds for 200 candidates.

This is the phase that turns signals from "primitives the application reads" into "primitives the database scores over." After this phase, a developer can name a profile and get ranked results -- the database does the math, not the application.

Acceptance Criteria

• RankingProfile struct: name, version, candidate_strategy, scoring_rules (boosts, penalties, gates, excludes), sort mode override, diversity config, exploration budget
• ScoringRule enum: Boost { signal, window, aggregation, weight }, Gate { condition, threshold }, Penalize { signal, window, weight }, Exclude { condition }
• Sort enum with formula variants: Hot { gravity }, Controversial, HiddenGems, New, Shuffle { seed }, TopWindow { window }, MostViewed, MostLiked, Rising
• CandidateStrategy enum: Ann, Scan, Hybrid, Relationship, CohortTrending (M2 implements only Scan; others are type stubs used by profiles but not executed until their retrieval strategy is built)
• ProfileRegistry maps profile name to versioned RankingProfile instances, supports get, get_versioned, register, list
• Profile validation: duplicate names rejected, unknown signal references rejected (if schema available), gate threshold range [0.0, 1.0], weight normalization warning, version monotonicity (INV-PROF-1)
• Profiles serializable via serde for schema checkpoint/reload
• Built-in profiles registered at SchemaBuilder::build() time: trending, hot, new, top_week, top_month, top_all_time, hidden_gems, controversial, most_viewed, most_liked, shuffle
• Built-in profiles are standard RankingProfile instances -- not special-cased in the executor
• Built-in profiles with unavailable signals degrade gracefully (skip missing signals, not fatal error)
• hot formula: log10(max(|positive - negative|, 1)) / (age_hours + 2)^gravity with configurable gravity (default 1.8) -- Spec 09 Section 11.1
• controversial formula: (positive * negative) / (positive + negative)^2 -- Spec 09 Section 11.4
• hidden_gems formula: quality_score * (1 / log10(view_count + 10)) -- Spec 09 Section 11.5
• ProfileExecutor::score() takes &[EntityId] candidates and &RankingProfile, returns Vec<ScoredCandidate> sorted by score descending
• ScoredCandidate includes: entity_id, score, signal_snapshot (key signal values used for scoring transparency)
• Gate failure sets score to 0.0; candidates with score 0.0 are filtered out before returning
• Shuffle profile uses deterministic seeded RNG (stable per user_id + profile_name + page_cursor)
• Profile change does not require recompile -- profiles are runtime data
• 200-candidate scoring pass with decay-only profile < 10 microseconds, with velocity-based profile (trending) < 100 microseconds (both Criterion benchmarked)
• Deterministic scoring: same candidates + same profile + same signal state = identical results (INV-RANK-1)
• Normalized scores in [0.0, 1.0] after min-max normalization (INV-RANK-2)

Dependencies

• Requires: m1p1 (types: EntityId, EntityKind, SignalTypeDef, Window, WindowSet, DecayModel, Score), m1p4 (SignalLedger: profiles read decay scores and windowed counts via SignalLedger API), m1p5 (entity read API: TidalDb::read_decay_score, TidalDb::read_windowed_count, TidalDb::read_velocity)
• Blocks: m2p4 (diversity enforcement takes scored lists from profile executor), m2p5 (RETRIEVE executor uses profiles to score candidates)

Research References

• docs/research/tidaldb_signal_ledger.md -- Signal read latencies (~15ns decay score, ~200ns windowed count) that establish the per-candidate scoring budget
• thoughts.md -- Part V.14 (cache-line alignment for hot-path structs), scoring pipeline architecture

Spec References

• docs/specs/09-ranking-scoring.md -- THE authoritative spec for this phase:
  • Section 2 (ProfileDef structure, versioning, inheritance, A/B testing)
  • Section 3 (CandidateStrategy variants: ANN, Scan, Hybrid, Relationship, CohortTrending)
  • Section 4 (Scoring pipeline: 9-stage fixed-order transformation)
  • Section 5 (Boost types: signal, relationship, social proof, recency, cohort, preference match)
  • Section 6 (Penalty types: signal-based negative scoring)
  • Section 7 (Quality gates: minimum signal, ratio, count gates with exploration bypass)
  • Section 8 (Score composition: composite formula, min-max normalization, percentile signal normalization)
  • Section 11 (Built-in sort modes: Hot, Trending, Rising, Controversial, HiddenGems, Shuffle, Top windowed, simple field sorts)
  • Section 13 (Profile presets: for_you, trending, search, following, related, browse, hidden_gems, notification, live, hot, rising, controversial)
  • Section 15 (Performance targets: total scoring pipeline < 500us for 200 candidates, per-candidate scoring ~1.5us)
  • Section 16 (Invariants INV-RANK-1 through INV-RANK-7, INV-PROF-1 through INV-PROF-3, property tests P1-P6)

Task Index

#	Task	Delivers	Depends On	Complexity
01	Ranking Profile Type System	RankingProfile, ScoringRule, Sort, CandidateStrategy, ProfileRegistry, validation, serde	None	L
02	Built-in Profiles	All 11 built-in profile definitions as RankingProfile instances, signal dependency validation, graceful degradation for missing signals	Task 01	M
03	Profile Executor + Benchmarks	ProfileExecutor, ScoredCandidate, ShuffleExecutor, sort formula implementations, min-max normalization, Criterion benchmarks	Task 01, Task 02	L

Task Dependency DAG

Task 01: Ranking Profile Type System
    |
    v
Task 02: Built-in Profiles
    |
    +---> Task 03: Profile Executor + Benchmarks
    |        (also depends on Task 01)

Task 01 is the foundation -- it defines all types that Task 02 instantiates and Task 03 executes. Task 02 constructs the built-in profiles from Task 01 types. Task 03 requires both the types (Task 01) and the profiles (Task 02) to implement and benchmark execution.

File Layout

tidal/src/
  ranking/
    mod.rs        -- pub use re-exports: RankingProfile, ScoringRule, Sort, CandidateStrategy,
                     ProfileRegistry, ProfileExecutor, ScoredCandidate
    profile.rs    -- RankingProfile struct, ScoringRule, Sort, CandidateStrategy,
                     SignalAgg, Boost, Gate, Penalty, Exclude, validation (Task 01)
    registry.rs   -- ProfileRegistry, built-in profile construction, signal dependency
                     checking (Task 01 registry types, Task 02 built-in definitions)
    executor.rs   -- ProfileExecutor, ScoredCandidate, score() method,
                     sort formula implementations (Task 03)
    shuffle.rs    -- ShuffleExecutor, seeded RNG (Task 03)
  lib.rs          -- (unchanged, already declares pub mod ranking)
tidal/benches/
  ranking.rs      -- Criterion benchmarks (Task 03)

Open Questions

1. SmallRng vs rand_xoshiro: The shuffle profile needs a stable-per-session RNG seeded from (user_id, profile_name, page_cursor). SmallRng from the rand crate is fast and seedable. rand_xoshiro::Xoshiro256StarStar is available via rand_xoshiro. Decision: use SmallRng for M2 -- it is already in rand's dependency tree, performs well, and is reproducible given the same seed. Add rand_xoshiro only if SmallRng proves non-deterministic across platforms.

2. signal_snapshot in ScoredCandidate: The spec says results should include key signal values used in scoring for debugging (Spec 09 Section 4, Stage 10). For M2, include all signals referenced in the profile's scoring rules (typically 2-5 signals). Cap at 10 signal values per candidate. The snapshot is a Vec<(String, f64)> (signal name, value) rather than a HashMap to keep allocation small and ordering deterministic.

3. Gate vs Exclude vs Penalize semantics: Gate zeros the score (candidate excluded from results). Exclude physically removes the candidate before scoring (it never enters the pipeline). Penalize multiplies by a factor < 1. The executor filters out score <= 0.0 candidates before returning. For M2, Exclude variants (signal("hide"), relationship("blocked")) are type stubs -- the actual exclusion logic requires user state from M3. The executor skips Exclude rules when no user context is available.

4. Profile versioning: Version is a u32 monotonic counter per profile name. ProfileRegistry keeps all versions per name (INV-PROF-1 requires monotonic increase). get() returns latest version. get_versioned(name, version) returns a specific version. For M2, no version pruning. Pruning deferred to M5+ for A/B testing lifecycle management.

5. Built-in profiles with unavailable signals: At M2, the schema may not define all signals that a built-in profile references (e.g., trending requires share velocity, but the UAT schema might only have view and like). Built-in profiles must be resilient: if a referenced signal type is not in the schema, that boost/penalty is silently skipped (contributes 0.0). A tracing::warn! is emitted at registration time listing missing signals. The profile is still registered and usable -- it just scores with fewer signals.

6. Sort mode vs boost/penalty pipeline: When a profile has a sort override (e.g., Sort::Hot), the sort formula replaces stages 4-5 (boost and penalty application) of the scoring pipeline (Spec 09 Section 11.9). Gates, exclusions, normalization, and diversity still apply. The executor must check for a sort override before running the boost/penalty loop.

7. Candidate strategy as type stub: CandidateStrategy variants are defined as types in Task 01 but not executed in m2p3. The executor receives a pre-generated &[EntityId] candidate set. Candidate generation is the responsibility of the RETRIEVE executor (m2p5), which calls the appropriate retrieval strategy (ANN, scan, etc.) and passes the results to the profile executor for scoring. The CandidateStrategy on the profile is informational -- it tells the RETRIEVE executor how to generate candidates, but the profile executor itself does not generate candidates.