tidaldb/tidal/src/ranking/builtins.rs
jx12n bb21e69ae6 feat(m12): vector retrieval G1/G2 — recall harness, ANN in RETRIEVE, index tuning
m12p1 (measurement truth): TidalDb::vector_search_items pure k-NN probe +
POST /vector_search (standalone + region node, merge-by-distance) +
tidal-stress --verify-recall (deterministic id-keyed corpus, in-RAM brute-force
cosine oracle, open-loop ramp → recall@k + true p99 + read-knee + JSON/gate exit).
Repaired fabricated p99 columns (mean-as-p99) in social-scale.md / scale.rs.
Verified real: recall@10=0.9997 at 20k/1536-D vs brute-force.

m12p2 (G1 unblock): ANN candidate-gen wired into RETRIEVE — for_you=preference
vector, related=seed embedding (similar_to), graceful scan-fallback. Cached
per-signal-type top-K (signals/ledger/hot_top_k.rs, decay-order-invariant) so
trending serves O(K). related over HTTP (FeedQuery.similar_to). Harness gains
--feed-profile / --seed-preferences. Verified: trending retrieve p99 3.5-7.7ms.

m12p3 (G2): per-query ef_search now honored (RwLock epoch-guard with_expansion,
shared guard for same-ef concurrency) + dimension-aware brute→HNSW crossover
usearch_min_vectors(dim) + memory_usage() + examples/ann_grid_search.rs.
Measured 1536-D/100k clustered: default M=16/ef_c=400/F16/ef_s=200 clears
G1+G2 (recall 0.997, p99 1.4ms); F16 -0.25% vs F32; Int8 rejected (-28%).
Recall corpus is now clustered (Gaussian mixture) in grid + harness.
2026-06-14 11:07:09 -06:00

592 lines
21 KiB
Rust
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

//! Built-in ranking profiles.
//!
//! `TidalDB` ships 25 default profiles covering the most common content ranking
//! patterns: trending, hot, new, top (by window), hidden gems, controversial,
//! most viewed, most liked, shuffle, four personalized profiles added in M3
//! (`for_you`, `following`, `related`, `notification`), a search profile (M5),
//! a cohort-scoped trending profile (M6, `cohort_trending`), and full sort mode
//! coverage profiles (M6p3: `live`, `alphabetical_asc`, `alphabetical_desc`,
//! `shortest`, `longest`, `most_commented`, `most_shared`, `date_saved`).
//!
//! Population-level profiles use `CandidateStrategy::Scan` with `sort_field =
//! "created_at"`. Personalized profiles use `Relationship` strategy.
//! All are registered at version 1 with `is_builtin = true`.
//!
//! ## Sort modes intentionally without a built-in profile
//!
//! Three [`Sort`] modes are deliberately reachable only via a *custom schema
//! profile* (`SchemaBuilder::ranking_profile(..).sort(..)`), not via a shipped
//! built-in, so there is no orphaned executor path — the scoring logic
//! (`executor::scoring`) and its unit tests (`executor::tests::sort_tests`)
//! cover all three; only the named-convenience-profile shortcut is omitted:
//!
//! - [`Sort::MostFollowed`] and [`Sort::CreatorEngagementRate`] rank
//! **creators**, not items. Every built-in here defaults to
//! [`CandidateStrategy::Scan`] over the item keyspace (`sort_field =
//! "created_at"`); a creator-leaderboard built-in would need a creator-scoped
//! candidate strategy and a `follow` signal that the generic schema does not
//! guarantee. Applications declare these against their own creator schema.
//! - [`Sort::Rising`] (1h/24h view-velocity ratio) overlaps the shipped
//! [`Sort::Trending`] / [`Sort::Hot`] profiles for the population-feed slot;
//! its acceleration-ratio semantics are application-tuning territory (the
//! ratio is sensitive to the exact short/long window pair), so it is left to
//! custom profiles rather than baked into a one-size default.
//!
//! If a future milestone ships creator-leaderboard or rising-feed surfaces as
//! first-class defaults, add the matching built-ins here (and bump the count in
//! `register_builtins` and `builtins::tests`).
use super::{
profile::{Boost, CandidateStrategy, DiversitySpec, RankingProfile, SignalAgg, Sort},
registry::{ProfileError, ProfileRegistry},
};
use crate::schema::Window;
/// Default candidate strategy for built-in profiles.
fn default_strategy() -> CandidateStrategy {
CandidateStrategy::Scan {
sort_field: "created_at".into(),
}
}
/// Build a profile skeleton with common defaults.
fn skeleton(name: &str) -> RankingProfile {
RankingProfile {
name: name.to_owned(),
version: 1,
candidate_strategy: default_strategy(),
boosts: vec![],
decay: None,
gates: vec![],
penalties: vec![],
excludes: vec![],
diversity: DiversitySpec::default(),
exploration: 0.0,
sort: None,
is_builtin: true,
}
}
// ── Profile tuning constants ──────────────────────────────────────────────
/// Age-decay gravity for the hot sort (Reddit-style HN algorithm).
/// Higher values decay older content faster.
const HOT_GRAVITY: f64 = 1.8;
/// Exploration fraction injected into shuffle results.
/// 0.5 = 50% random exploration, 50% signal-ranked.
const SHUFFLE_EXPLORATION: f64 = 0.5;
/// Signal weight multiplier for share events in the trending score.
const TRENDING_SHARE_WEIGHT: f64 = 2.0;
/// Maximum items per creator in trending results (content diversity).
const TRENDING_MAX_PER_CREATOR: usize = 1;
/// Maximum items per creator in hot results.
const HOT_MAX_PER_CREATOR: usize = 2;
/// Register all 25 built-in ranking profiles into the given registry.
///
/// # Errors
///
/// Returns `ProfileError` if any profile fails validation. This should never
/// happen for built-in profiles -- if it does, it is a bug in the profile
/// definitions.
pub fn register_builtins(registry: &mut ProfileRegistry) -> Result<(), ProfileError> {
// Population-level profiles (M2).
registry.register(trending())?;
registry.register(hot())?;
registry.register(new())?;
registry.register(top_week())?;
registry.register(top_month())?;
registry.register(top_all_time())?;
registry.register(hidden_gems())?;
registry.register(controversial())?;
registry.register(most_viewed())?;
registry.register(most_liked())?;
registry.register(shuffle())?;
// Personalized profiles (M3).
registry.register(for_you())?;
registry.register(following())?;
registry.register(related())?;
registry.register(notification())?;
// Search profile (M5).
registry.register(search())?;
// Cohort profile (M6).
registry.register(cohort_trending())?;
// M6p3 profiles: live content + full sort mode coverage.
registry.register(live())?;
registry.register(alphabetical_asc())?;
registry.register(alphabetical_desc())?;
registry.register(shortest())?;
registry.register(longest())?;
registry.register(most_commented())?;
registry.register(most_shared())?;
registry.register(date_saved())?;
Ok(())
}
/// The global trending profile.
///
/// `Sort::Trending` is the ordering authority on the global (non-cohort) path:
/// its formula `view_vel + 2*share_vel` over the 24h window *replaces* stages 4-5
/// (spec §11.9), so the executor skips the boost loop when this sort is active
/// (no double-count — previously the base and these boosts both read the same
/// signals and summed to `2*view_vel + 4*share_vel`).
///
/// The `view`/`share` velocity boosts below are NOT redundant: they are the
/// signal definition the **cohort-scoped rescore** (query Stage 3b,
/// `rescore_with_cohort`) reads to re-score candidates against a cohort's signal
/// ledger — that path consumes `profile.boosts` directly and does not run the
/// `Sort::Trending` formula. Their weights deliberately mirror the formula so the
/// cohort ordering matches the global one. Keep the two in sync.
fn trending() -> RankingProfile {
let mut p = skeleton("trending");
// m12p2: source candidates from the most-viewed items via the cached
// per-signal-type top-K, so trending ranks the actually-engaged corpus at
// scale (O(K)) instead of an arbitrary low-id scan slice. Sort::Trending then
// re-ranks this pool by velocity. Degrades to a scan when no `view` signals
// exist yet (executor fallback), so a fresh corpus still serves.
p.candidate_strategy = CandidateStrategy::SignalRanked {
signal: "view".into(),
window: Window::TwentyFourHours,
};
p.sort = Some(Sort::Trending);
// Cohort-rescore signal definition (see doc above); the global path uses the
// Sort::Trending formula and skips these per spec §11.9.
p.boosts = vec![
Boost {
signal: "share".into(),
agg: SignalAgg::Velocity,
window: Window::TwentyFourHours,
weight: TRENDING_SHARE_WEIGHT,
},
Boost {
signal: "view".into(),
agg: SignalAgg::Velocity,
window: Window::TwentyFourHours,
weight: 1.0,
},
];
// Gate on engagement_ratio deferred to application-level profiles.
// Built-in profiles avoid gates on signals that may not be in the schema.
p.diversity = DiversitySpec {
max_per_creator: Some(TRENDING_MAX_PER_CREATOR),
..DiversitySpec::default()
};
p
}
fn hot() -> RankingProfile {
let mut p = skeleton("hot");
p.sort = Some(Sort::Hot {
gravity: HOT_GRAVITY,
});
p.boosts = vec![Boost {
signal: "view".into(),
agg: SignalAgg::Velocity,
window: Window::OneHour,
weight: 1.0,
}];
p.diversity = DiversitySpec {
max_per_creator: Some(HOT_MAX_PER_CREATOR),
..DiversitySpec::default()
};
p
}
fn new() -> RankingProfile {
let mut p = skeleton("new");
p.sort = Some(Sort::New);
p
}
fn top_week() -> RankingProfile {
let mut p = skeleton("top_week");
p.sort = Some(Sort::TopWindow {
window: Window::SevenDays,
});
p
}
fn top_month() -> RankingProfile {
let mut p = skeleton("top_month");
p.sort = Some(Sort::TopWindow {
window: Window::ThirtyDays,
});
p
}
fn top_all_time() -> RankingProfile {
let mut p = skeleton("top_all_time");
p.sort = Some(Sort::TopWindow {
window: Window::AllTime,
});
p
}
fn hidden_gems() -> RankingProfile {
let mut p = skeleton("hidden_gems");
p.sort = Some(Sort::HiddenGems);
// Gates on completion/view deferred to application-level profiles.
// Built-in profiles avoid gates on signals that may not be in the schema.
p
}
fn controversial() -> RankingProfile {
let mut p = skeleton("controversial");
p.sort = Some(Sort::Controversial);
// Gates on like/dislike deferred to application-level profiles.
// Built-in profiles avoid gates on signals that may not be in the schema.
p
}
fn most_viewed() -> RankingProfile {
let mut p = skeleton("most_viewed");
p.sort = Some(Sort::MostViewed {
window: Window::SevenDays,
});
p
}
fn most_liked() -> RankingProfile {
let mut p = skeleton("most_liked");
p.sort = Some(Sort::MostLiked {
window: Window::SevenDays,
});
p
}
fn shuffle() -> RankingProfile {
let mut p = skeleton("shuffle");
p.sort = Some(Sort::Shuffle);
p.exploration = SHUFFLE_EXPLORATION;
p
}
// ── M3 Personalized Profiles ────────────────────────────────────────────────
/// Maximum items per creator in `for_you` results.
const FOR_YOU_MAX_PER_CREATOR: usize = 2;
/// Exploration fraction for the `for_you` profile.
/// 10% random exploration to prevent filter bubbles.
const FOR_YOU_EXPLORATION: f64 = 0.1;
/// Maximum items per creator in following results.
const FOLLOWING_MAX_PER_CREATOR: usize = 3;
/// `for_you`: personalized home feed ranking.
///
/// Combines interaction-weighted decay scores with exploration injection.
/// Uses `Scan` strategy (M3 user-context filtering in Stage 2.5 handles
/// unseen/unblocked). The `for_user` clause triggers preference-aware
/// scoring in the executor.
fn for_you() -> RankingProfile {
let mut p = skeleton("for_you");
// m12p2: ANN candidate generation over the user's preference vector — the
// nearest content to the user's learned taste, O(ef_search), not an arbitrary
// low-id scan slice. Degrades to a scan for anonymous reads or a user with no
// preference vector yet (executor handles the fallback). `limit` caps the ANN
// candidate pool; the executor over-fetches `query.limit × 10` within it.
p.candidate_strategy = CandidateStrategy::Ann {
slot: "content".into(),
limit: 1000,
};
p.sort = Some(Sort::Hot { gravity: 1.5 });
p.boosts = vec![
Boost {
signal: "view".into(),
agg: SignalAgg::DecayScore,
window: Window::AllTime,
weight: 1.0,
},
Boost {
signal: "like".into(),
agg: SignalAgg::DecayScore,
window: Window::AllTime,
weight: 2.0,
},
Boost {
signal: "share".into(),
agg: SignalAgg::Velocity,
window: Window::TwentyFourHours,
weight: 1.5,
},
];
p.diversity = DiversitySpec {
max_per_creator: Some(FOR_YOU_MAX_PER_CREATOR),
format_mix_max_fraction: Some(0.4),
};
p.exploration = FOR_YOU_EXPLORATION;
p
}
/// following: content from followed creators, ranked by interaction strength.
///
/// Uses `Relationship` candidate strategy -- the executor sources
/// candidates from the user's relationship graph. Ranked by recency
/// (`Sort::New`) with a view decay boost.
fn following() -> RankingProfile {
let mut p = skeleton("following");
p.candidate_strategy = CandidateStrategy::Relationship;
p.sort = Some(Sort::New);
p.boosts = vec![Boost {
signal: "view".into(),
agg: SignalAgg::Velocity,
window: Window::OneHour,
weight: 0.5,
}];
p.diversity = DiversitySpec {
max_per_creator: Some(FOLLOWING_MAX_PER_CREATOR),
..DiversitySpec::default()
};
p
}
/// related: "more like this" ranking for a seed item.
///
/// Uses `Scan` strategy. When `similar_to` is set in the query, the
/// executor will use the seed item's embedding for ANN in Stage 1
/// (when vector indexes are wired). For M3, falls back to scan with
/// content-type boosting.
fn related() -> RankingProfile {
let mut p = skeleton("related");
// m12p2: ANN candidate generation over the seed item's embedding (resolved
// from the query's `similar_to`) — true "more like this", O(ef_search).
// Degrades to a scan when no `similar_to` is supplied (executor fallback).
p.candidate_strategy = CandidateStrategy::Ann {
slot: "content".into(),
limit: 1000,
};
p.sort = Some(Sort::Hot { gravity: 1.2 });
p.boosts = vec![
Boost {
signal: "view".into(),
agg: SignalAgg::DecayScore,
window: Window::AllTime,
weight: 1.0,
},
Boost {
signal: "completion".into(),
agg: SignalAgg::DecayScore,
window: Window::AllTime,
weight: 1.5,
},
];
p.diversity = DiversitySpec {
max_per_creator: Some(2),
..DiversitySpec::default()
};
p
}
/// notification: items a user should be notified about.
///
/// Prioritizes high-velocity content from creators the user follows.
/// Strict diversity ensures no single creator dominates the notification
/// tray.
fn notification() -> RankingProfile {
let mut p = skeleton("notification");
p.candidate_strategy = CandidateStrategy::Relationship;
p.sort = Some(Sort::Trending);
p.boosts = vec![
Boost {
signal: "view".into(),
agg: SignalAgg::Velocity,
window: Window::OneHour,
weight: 2.0,
},
Boost {
signal: "like".into(),
agg: SignalAgg::Velocity,
window: Window::OneHour,
weight: 1.0,
},
];
p.diversity = DiversitySpec {
max_per_creator: Some(1),
..DiversitySpec::default()
};
p
}
// ── M5 Search Profile ───────────────────────────────────────────────────────
/// Weight for view decay score in the search profile.
///
/// Lower than personalized profiles to let text relevance dominate.
const SEARCH_VIEW_WEIGHT: f64 = 0.5;
/// Weight for like decay score in the search profile.
///
/// Captures quality signal: items frequently liked tend to be good results.
const SEARCH_LIKE_WEIGHT: f64 = 0.8;
/// `search`: text and vector relevance plus light signal re-ranking.
///
/// Default profile for the SEARCH query type. The heavy lifting is done by
/// RRF fusion (BM25 + ANN) in Stage 1c of the `SearchExecutor`. This profile
/// provides a lightweight signal overlay: a view-decay and like-decay boost to
/// surface quality content without overriding text relevance signals.
///
/// - No exploration injection (`exploration = 0.0`): search results must be
/// deterministic for a given query.
/// - No diversity enforcement: callers specify diversity explicitly via
/// `SearchBuilder::diversity()`.
/// - `sort = None`: the fused RRF score from Stage 1c is the primary ordering
/// signal; the profile adds a small quality overlay on top.
fn search() -> RankingProfile {
let mut p = skeleton("search");
p.boosts = vec![
Boost {
signal: "view".into(),
agg: SignalAgg::DecayScore,
window: Window::AllTime,
weight: SEARCH_VIEW_WEIGHT,
},
Boost {
signal: "like".into(),
agg: SignalAgg::DecayScore,
window: Window::AllTime,
weight: SEARCH_LIKE_WEIGHT,
},
];
// No diversity: callers control diversity via SearchBuilder.
// No exploration: search results are deterministic.
p.exploration = 0.0;
p.sort = None;
p
}
// ── M6 Cohort Profiles ───────────────────────────────────────────────────────
/// `cohort_trending`: trending content scoped to a named cohort.
///
/// Identical boosts and sort mode to the global `trending` profile, but
/// intended for use with `RetrieveBuilder::cohort("my_cohort")`. With a `cohort`
/// clause, the executor reads signal values from the cohort signal ledger via the
/// Stage 3b rescore (`rescore_with_cohort`), which consumes `profile.boosts`
/// directly. Without a `cohort` clause, this profile behaves identically to
/// `trending`: the `Sort::Trending` formula orders results and the boosts below
/// are skipped (spec §11.9), so there is no double-count.
fn cohort_trending() -> RankingProfile {
let mut p = skeleton("cohort_trending");
p.sort = Some(Sort::Trending);
// Cohort-rescore signal definition (read by Stage 3b when a cohort is set);
// the no-cohort path uses the Sort::Trending formula and skips these.
p.boosts = vec![
Boost {
signal: "share".into(),
agg: SignalAgg::Velocity,
window: Window::TwentyFourHours,
weight: TRENDING_SHARE_WEIGHT,
},
Boost {
signal: "view".into(),
agg: SignalAgg::Velocity,
window: Window::TwentyFourHours,
weight: 1.0,
},
];
p.diversity = DiversitySpec {
max_per_creator: Some(TRENDING_MAX_PER_CREATOR),
..DiversitySpec::default()
};
p
}
// ── M6p3 Live Content Profile ───────────────────────────────────────────────
/// Weight for the relationship-preference boost in the live profile.
///
/// Lower than the `following` profile's 0.5 because the primary sort key
/// (`LiveViewerCount`) already dominates ordering. The boost lifts content
/// from creators the user actively views (social circle), making relationship
/// weight "dominant" without overriding the viewer-count signal entirely.
const LIVE_RELATIONSHIP_BOOST_WEIGHT: f64 = 0.3;
/// `live`: live content ranking by current viewer count.
///
/// Sorts by `LiveViewerCount` (decayed `viewer_count` signal) with strict
/// per-creator diversity (max 1 per creator). A `view` velocity boost gives
/// preference to content from creators the user's social circle actively
/// watches (relationship-weight dominant, per UC-12).
fn live() -> RankingProfile {
let mut p = skeleton("live");
p.sort = Some(Sort::LiveViewerCount);
p.boosts = vec![Boost {
signal: "view".into(),
agg: SignalAgg::Velocity,
window: Window::OneHour,
weight: LIVE_RELATIONSHIP_BOOST_WEIGHT,
}];
p.diversity = DiversitySpec {
max_per_creator: Some(1),
..DiversitySpec::default()
};
p
}
/// `alphabetical_asc`: sort by item title A-Z (case-insensitive).
fn alphabetical_asc() -> RankingProfile {
let mut p = skeleton("alphabetical_asc");
p.sort = Some(Sort::AlphabeticalAsc);
p
}
/// `alphabetical_desc`: sort by item title Z-A (case-insensitive).
fn alphabetical_desc() -> RankingProfile {
let mut p = skeleton("alphabetical_desc");
p.sort = Some(Sort::AlphabeticalDesc);
p
}
/// `shortest`: sort by item duration, shortest first.
fn shortest() -> RankingProfile {
let mut p = skeleton("shortest");
p.sort = Some(Sort::Shortest);
p
}
/// `longest`: sort by item duration, longest first.
fn longest() -> RankingProfile {
let mut p = skeleton("longest");
p.sort = Some(Sort::Longest);
p
}
/// `most_commented`: sort by comment count (`AllTime` window).
fn most_commented() -> RankingProfile {
let mut p = skeleton("most_commented");
p.sort = Some(Sort::MostCommented {
window: Window::AllTime,
});
p
}
/// `most_shared`: sort by share count (`AllTime` window).
fn most_shared() -> RankingProfile {
let mut p = skeleton("most_shared");
p.sort = Some(Sort::MostShared {
window: Window::AllTime,
});
p
}
/// `date_saved`: sort by when the querying user saved the item (latest first).
///
/// Requires `FOR USER` context in the query. Without it, the executor
/// returns `QueryError::InvalidFilter`.
fn date_saved() -> RankingProfile {
let mut p = skeleton("date_saved");
p.sort = Some(Sort::DateSaved);
p
}
// ── Tests ───────────────────────────────────────────────────────────────────
#[cfg(test)]
#[allow(clippy::unwrap_used, clippy::float_cmp)]
mod tests;