413b712c0a
- Schema phase 1 (tasks 01-02): EntityId, EntityKind, Timestamp, Score, SignalTypeDef, DecayModel, Window, WindowSet — all with property tests and benchmarks scaffolding - Stub modules for storage, signals, query, ranking - Full documentation suite: VISION, USE_CASES, SEQUENCE, API, CODING_GUIDELINES, ai-lookup, research docs, specs, roadmap, planning docs - Marketing site (Next.js) with blog infrastructure - .claude/ agents and skills for the tidalDB development workflow - Foundation standards enforced: thiserror + tracing declared as dependencies, clippy::unwrap_used = deny added to lint config - .gitignore hardened: .next/, node_modules/, .env, secrets, logs Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2068 lines
83 KiB
Markdown
2068 lines
83 KiB
Markdown
# Ranking and Scoring Specification
|
|
|
|
**Status:** Draft
|
|
**Authors:** tidalDB Engineering
|
|
**Date:** 2026-02-20
|
|
**Depends on:** Signal System (03), Relationships (04), Cohorts (05), Text Retrieval (06), Vector Retrieval (07)
|
|
**Research:** `docs/research/ann_for_tidaldb.md`, `docs/research/tidaldb_signal_ledger.md`, `docs/research/tantivy.md`
|
|
|
|
---
|
|
|
|
## Table of Contents
|
|
|
|
1. [Overview](#1-overview)
|
|
2. [Ranking Profile Declaration](#2-ranking-profile-declaration)
|
|
3. [Candidate Generation Strategies](#3-candidate-generation-strategies)
|
|
4. [Scoring Pipeline](#4-scoring-pipeline)
|
|
5. [Boost Types](#5-boost-types)
|
|
6. [Penalty Types](#6-penalty-types)
|
|
7. [Quality Gates](#7-quality-gates)
|
|
8. [Score Composition and Normalization](#8-score-composition-and-normalization)
|
|
9. [Diversity Enforcement](#9-diversity-enforcement)
|
|
10. [Exploration Budget](#10-exploration-budget)
|
|
11. [Built-In Sort Modes](#11-built-in-sort-modes)
|
|
12. [Cohort-Aware Ranking](#12-cohort-aware-ranking)
|
|
13. [Profile Presets](#13-profile-presets)
|
|
14. [Pagination and Cursors](#14-pagination-and-cursors)
|
|
15. [Performance Targets](#15-performance-targets)
|
|
16. [Invariants and Correctness Guarantees](#16-invariants-and-correctness-guarantees)
|
|
17. [Integration Points](#17-integration-points)
|
|
|
|
---
|
|
|
|
## 1. Overview
|
|
|
|
The ranking and scoring system is the core value proposition of tidalDB. It replaces the external ranking service that today stitches together signals from Elasticsearch, Redis, a feature store, and a vector database. In tidalDB, ranking is a database primitive: the application names a profile, the database executes the entire pipeline.
|
|
|
|
The ranking system takes as input a set of candidate entities (generated by one of several retrieval strategies), a user context (preference vector, relationship graph, signal history), and a ranking profile (a named, versioned scoring function declared in schema). It produces as output a scored, diversified, paginated result set ready for rendering -- no re-ranking by the application, ever.
|
|
|
|
### Design Principles
|
|
|
|
1. **Profiles are data, not code.** Ranking profiles are schema-level declarations stored in the database. A profile change never requires recompilation or redeployment. The query planner reasons about profile structure to optimize execution.
|
|
|
|
2. **The pipeline is fixed; the weights are configurable.** The nine-stage scoring pipeline (Section 4) executes in the same order for every query. Profiles configure what each stage does -- which signals to boost, which gates to apply, which diversity constraints to enforce -- but cannot alter the stage order.
|
|
|
|
3. **Negative signals are structurally equal to positive signals.** Skips, hides, downvotes are first-class inputs to the scoring function with the same weight, precision, and update immediacy as likes.
|
|
|
|
4. **Diversity is a post-scoring constraint.** Diversity enforcement reorders results after scoring. It never filters candidates out of the result set -- it demotes items that violate constraints and promotes items that satisfy them.
|
|
|
|
5. **Graceful degradation, never failure.** Under load, the system returns less precise rankings rather than errors. Degradation order: reduce candidate set, use coarser signal aggregates, skip diversity, serve from materialized cache.
|
|
|
|
6. **Cold start is a database responsibility.** New items with no signals and new users with no history receive sensible treatment via exploration budgets and population priors. The application does not manage this.
|
|
|
|
---
|
|
|
|
## 2. Ranking Profile Declaration
|
|
|
|
### 2.1 ProfileDef Structure
|
|
|
|
A ranking profile is a named, versioned scoring function that fully specifies how candidates are retrieved, scored, filtered, diversified, and paginated. The application says `USING PROFILE for_you`. The database executes everything.
|
|
|
|
```rust
|
|
pub struct ProfileDef {
|
|
/// Unique profile name. Lowercase alphanumeric plus underscores.
|
|
pub name: &str,
|
|
|
|
/// Monotonically increasing version. Old versions remain queryable
|
|
/// by specifying name + version at query time.
|
|
pub version: u32,
|
|
|
|
/// How candidates are generated (Section 3).
|
|
pub candidate: CandidateStrategy,
|
|
|
|
/// Optional parent profile. This profile inherits all fields from
|
|
/// the parent and overrides only the fields explicitly set.
|
|
pub extends: Option<ProfileRef>,
|
|
|
|
/// Positive signal boosts applied to candidate scores (Section 5).
|
|
pub boosts: Vec<Boost>,
|
|
|
|
/// Content age decay applied to all candidates (Section 5.4).
|
|
pub decay: Option<ProfileDecay>,
|
|
|
|
/// Quality gates -- hard thresholds that exclude candidates (Section 7).
|
|
pub gates: Vec<Gate>,
|
|
|
|
/// Negative signal penalties subtracted from scores (Section 6).
|
|
pub penalties: Vec<Penalty>,
|
|
|
|
/// Hard exclusions -- items matching these are removed before scoring.
|
|
pub excludes: Vec<Exclude>,
|
|
|
|
/// Post-scoring diversity constraints (Section 9).
|
|
pub diversity: Option<DiversitySpec>,
|
|
|
|
/// Fraction of results reserved for exploration (Section 10).
|
|
/// Range: 0.0 to 0.5. Default: 0.0 (no exploration).
|
|
pub exploration: f64,
|
|
|
|
/// Optional explicit sort mode override. When set, bypasses the
|
|
/// boost/penalty scoring pipeline and uses a formula-based sort
|
|
/// (Section 11). Used by sort modes like Hot, Trending, Rising.
|
|
pub sort: Option<Sort>,
|
|
}
|
|
```
|
|
|
|
### 2.2 Version Semantics
|
|
|
|
Profiles are versioned to enable safe iteration and A/B testing.
|
|
|
|
**Versioning rules:**
|
|
|
|
1. Each call to `db.define_profile()` with an existing profile name creates a new version. The version number is monotonically increasing.
|
|
2. The latest version is used by default when a query specifies `USING PROFILE for_you` without a version qualifier.
|
|
3. Previous versions remain queryable by specifying `profile: "for_you@1"` or equivalently `profile_version: Some(1)`.
|
|
4. Versions are immutable once defined. To modify a profile, define a new version.
|
|
5. Maximum 100 versions per profile name. Older versions can be garbage-collected with `db.prune_profile_versions("for_you", keep_latest: 10)`.
|
|
|
|
**Storage:** Profiles are stored in the schema catalog alongside entity definitions and signal definitions. They are loaded into memory at startup and cached for the lifetime of the database instance. Profile definitions are persisted in the WAL for crash recovery.
|
|
|
|
### 2.3 Profile Inheritance
|
|
|
|
Profiles can extend other profiles to reduce duplication. A child profile inherits all fields from its parent and overrides only the fields explicitly set.
|
|
|
|
```rust
|
|
// Base browse profile
|
|
db.define_profile(ProfileDef {
|
|
name: "browse",
|
|
version: 1,
|
|
candidate: Candidate::Scan { entity: EntityKind::Item },
|
|
boosts: vec![
|
|
Boost::signal("completion", Window::all_time(), Value, 0.5),
|
|
Boost::signal("like", Window::all_time(), Ratio, 0.3),
|
|
Boost::signal("view", Window::all_time(), Value, 0.2),
|
|
],
|
|
decay: Some(ProfileDecay {
|
|
field: "created_at",
|
|
half_life: Duration::days(30),
|
|
}),
|
|
diversity: Some(DiversitySpec {
|
|
max_per_creator: Some(2),
|
|
..Default::default()
|
|
}),
|
|
..ProfileDef::default()
|
|
})?;
|
|
|
|
// Personalized browse -- extends browse with user preference boost
|
|
db.define_profile(ProfileDef {
|
|
name: "browse_personalized",
|
|
version: 1,
|
|
extends: Some(ProfileRef::latest("browse")),
|
|
// Inherits candidate, boosts, decay, diversity from browse.
|
|
// Adds preference match boost on top.
|
|
boosts: vec![
|
|
Boost::preference_match(0.3),
|
|
],
|
|
// Inherited boosts from parent are appended, not replaced.
|
|
// To replace, set extends: None and redefine all boosts.
|
|
..ProfileDef::default()
|
|
})?;
|
|
```
|
|
|
|
**Inheritance resolution:**
|
|
|
|
| Field | Behavior |
|
|
|-------|----------|
|
|
| `candidate` | Child overrides parent if set; otherwise inherits. |
|
|
| `boosts` | Child boosts are appended to parent boosts. |
|
|
| `decay` | Child overrides parent if set. |
|
|
| `gates` | Child gates are appended to parent gates. |
|
|
| `penalties` | Child penalties are appended to parent penalties. |
|
|
| `excludes` | Child excludes are appended to parent excludes. |
|
|
| `diversity` | Child overrides parent if set. |
|
|
| `exploration` | Child overrides parent if set. |
|
|
| `sort` | Child overrides parent if set. |
|
|
|
|
**Inheritance depth:** Maximum 3 levels. Deeper inheritance chains are rejected at definition time with `SchemaError::InheritanceDepthExceeded`.
|
|
|
|
### 2.4 A/B Testing
|
|
|
|
A/B testing is performed by defining multiple profile versions or separate profile names and specifying the desired variant at query time.
|
|
|
|
```rust
|
|
// Define two variants
|
|
db.define_profile(ProfileDef {
|
|
name: "for_you",
|
|
version: 2,
|
|
// ... same as v1 but with adjusted weights
|
|
boosts: vec![
|
|
Boost::signal("view", Window::hours(24), Velocity, 0.4), // was 0.3
|
|
Boost::relationship("interaction_weight", 0.15), // was 0.2
|
|
Boost::social_proof(0.20), // was 0.15
|
|
],
|
|
..base_for_you.clone()
|
|
})?;
|
|
|
|
// Control group
|
|
let control = db.retrieve(Retrieve {
|
|
profile: "for_you",
|
|
profile_version: Some(1),
|
|
for_user: Some("user_123"),
|
|
..query.clone()
|
|
})?;
|
|
|
|
// Treatment group
|
|
let variant = db.retrieve(Retrieve {
|
|
profile: "for_you",
|
|
profile_version: Some(2),
|
|
for_user: Some("user_123"),
|
|
..query.clone()
|
|
})?;
|
|
```
|
|
|
|
The database does not manage A/B assignment. The application decides which version each user sees. The database executes whichever version is requested.
|
|
|
|
---
|
|
|
|
## 3. Candidate Generation Strategies
|
|
|
|
Candidate generation is the first stage of the ranking pipeline. It produces a raw set of entities with initial retrieval scores. The strategy determines how candidates are found; subsequent pipeline stages determine how they are scored and ordered.
|
|
|
|
### 3.1 ANN (Approximate Nearest Neighbor)
|
|
|
|
Vector similarity search over embeddings. Used for personalized feeds and related content.
|
|
|
|
```rust
|
|
Candidate::Ann {
|
|
/// Source of the query vector.
|
|
query_vector: VectorSource,
|
|
/// Entity type to search over.
|
|
index: EntityKind,
|
|
/// Number of candidates to retrieve from the ANN index.
|
|
top_k: u32,
|
|
}
|
|
```
|
|
|
|
**VectorSource variants:**
|
|
|
|
| Source | Description |
|
|
|--------|-------------|
|
|
| `VectorSource::UserPreference` | The querying user's preference vector. Used by `for_you`. |
|
|
| `VectorSource::ItemEmbedding(item_id)` | A specific item's embedding. Used by `related`. |
|
|
| `VectorSource::QueryEmbedding` | The query vector passed inline (for SEARCH). |
|
|
| `VectorSource::CreatorEmbedding(creator_id)` | A creator's catalog embedding. Used by creator discovery. |
|
|
|
|
**Initial score:** Cosine similarity in range [0.0, 1.0] (embeddings are normalized at insertion time per Coding Guidelines Section 4).
|
|
|
|
**Filter interaction:** Pre-filters (user state, blocked, unseen) are applied as predicate callbacks during HNSW traversal when selectivity is 2-100%. For selectivity below 2%, a roaring bitmap pre-filter with brute-force L2 scan is used. See Vector Retrieval spec (07) for the adaptive strategy.
|
|
|
|
### 3.2 Scan
|
|
|
|
Full entity scan with signal-based ranking. Used for trending, hot, and sort-mode-dominant queries where no embedding similarity is involved.
|
|
|
|
```rust
|
|
Candidate::Scan {
|
|
/// Entity type to scan.
|
|
entity: EntityKind,
|
|
}
|
|
```
|
|
|
|
**Initial score:** 0.0 for all candidates. Scoring is entirely determined by boosts, penalties, and sort mode formulas.
|
|
|
|
**Optimization:** A full scan of 10M entities is infeasible at query time. The scan strategy uses the following acceleration:
|
|
|
|
1. **Signal-indexed scan.** For velocity-based profiles (trending, rising), only entities with non-zero velocity in the relevant window are candidates. The warm tier's active-entity index provides this set (typically <500K entities out of 10M).
|
|
2. **Metadata-indexed scan.** Filters on keyword fields (category, format, status) are resolved to roaring bitmaps and intersected before any signal reads.
|
|
3. **Top-K early termination.** After the first pass produces rough scores, a heap-based top-K selection eliminates low-scoring candidates before expensive signal reads.
|
|
|
|
**Performance:** For a trending query with one category filter, the effective candidate set is typically 10K-50K entities, not 10M.
|
|
|
|
### 3.3 Hybrid (Text + Vector Fusion)
|
|
|
|
Combines full-text BM25 retrieval with vector similarity search. Used by the `search` profile.
|
|
|
|
```rust
|
|
Candidate::Hybrid {
|
|
/// Weight of text (BM25) relevance in the fused score.
|
|
text_weight: f64,
|
|
/// Weight of vector (ANN) similarity in the fused score.
|
|
vector_weight: f64,
|
|
/// Fusion strategy.
|
|
fusion: Fusion,
|
|
}
|
|
|
|
pub enum Fusion {
|
|
/// Reciprocal Rank Fusion. Rank-based, no score normalization needed.
|
|
/// k controls convergence -- higher k = more weight to lower-ranked items.
|
|
/// Default k=60 (Cormack et al., SIGIR 2009).
|
|
Rrf { k: u32 },
|
|
|
|
/// Weighted linear combination of normalized scores.
|
|
/// Requires min-max normalization of both score distributions.
|
|
/// Use only after relevance labels exist to tune alpha.
|
|
Linear { alpha: f64 },
|
|
}
|
|
```
|
|
|
|
**RRF formula:**
|
|
|
|
```
|
|
RRF_score(d) = text_weight / (k + rank_bm25(d)) + vector_weight / (k + rank_ann(d))
|
|
```
|
|
|
|
**Initial score:** The fused RRF or linear combination score.
|
|
|
|
**Filter interaction:** Text filters (keyword fields) are applied within the Tantivy query. Vector filters use the adaptive strategy from the Vector Retrieval spec.
|
|
|
|
### 3.4 Relationship (Graph Traversal)
|
|
|
|
Candidate generation via graph traversal. Used by the `following` profile and social-graph-scoped queries.
|
|
|
|
```rust
|
|
Candidate::Relationship {
|
|
/// The relationship edge type to traverse.
|
|
edge: &str,
|
|
}
|
|
```
|
|
|
|
**Execution:** Starting from the querying user, traverse outgoing edges of type `edge` (e.g., `"follows"`). Collect all items authored by the target entities (creators). These items form the candidate set.
|
|
|
|
**Initial score:** 0.0 for all candidates. Sort is typically by `created_at DESC` (chronological).
|
|
|
|
**Filter interaction:** Standard metadata filters apply after traversal. The traversal itself acts as a hard filter (only items from related entities are included).
|
|
|
|
**Fan-out control:** Maximum fan-out is bounded by the user's relationship count (e.g., 500 follows). Each creator's recent items are fetched using a bounded scan on the `creator_id` prefix in the entity store, limited to `LIMIT * 2` items per creator to bound total candidate set size.
|
|
|
|
### 3.5 CohortTrending
|
|
|
|
Candidate generation scoped to items trending within a specific cohort. Used by cohort-aware trending profiles.
|
|
|
|
```rust
|
|
Candidate::CohortTrending {
|
|
/// Which cohort to scope to.
|
|
cohort: CohortSource,
|
|
/// Time window for velocity computation.
|
|
window: Window,
|
|
/// Number of top trending candidates to retrieve.
|
|
top_k: u32,
|
|
}
|
|
|
|
pub enum CohortSource {
|
|
/// Derive cohort from the querying user's attributes.
|
|
Auto,
|
|
/// Use a specific named cohort.
|
|
Named(String),
|
|
/// Inline predicate.
|
|
Predicate(Predicate),
|
|
}
|
|
```
|
|
|
|
**Execution:**
|
|
|
|
1. Resolve the cohort to a signal aggregation scope (see Cohorts spec, Section 7).
|
|
2. Scan all items with cohort tracking active (~100K items at the Signal System's threshold).
|
|
3. Read cohort-scoped velocity for the specified window.
|
|
4. Return the top `top_k` items by cohort velocity.
|
|
|
|
**Initial score:** Cohort-scoped velocity (events per unit time within the window).
|
|
|
|
**Filter interaction:** Metadata filters are applied after cohort velocity ranking.
|
|
|
|
### 3.6 Strategy Summary
|
|
|
|
| Strategy | Use Cases | Initial Score | Typical Candidate Count |
|
|
|----------|-----------|---------------|------------------------|
|
|
| ANN | for_you, related, visual search | Cosine similarity [0, 1] | 200-1000 |
|
|
| Scan | trending, hot, rising, browse | 0.0 (scored by boosts/sort) | 10K-50K (after index acceleration) |
|
|
| Hybrid | search | Fused text + vector score | 100-500 |
|
|
| Relationship | following | 0.0 (sorted by created_at) | 500-5000 |
|
|
| CohortTrending | trending_for_you, cohort trending | Cohort velocity | 200-500 |
|
|
|
|
---
|
|
|
|
## 4. Scoring Pipeline
|
|
|
|
The scoring pipeline is a nine-stage transformation that converts raw candidates into a ranked, diversified, paginated result set. The stages execute in fixed order. Every ranking query passes through all nine stages, though some stages may be no-ops depending on the profile configuration.
|
|
|
|
### Pipeline Diagram
|
|
|
|
```
|
|
Raw candidate set from retrieval strategy
|
|
|
|
|
v
|
|
+------------------------------------------+
|
|
| 1. CANDIDATE RETRIEVAL |
|
|
| ANN / Scan / Hybrid / Relationship / |
|
|
| CohortTrending |
|
|
| Output: candidates[] with initial |
|
|
| scores from retrieval strategy |
|
|
+------------------------------------------+
|
|
|
|
|
v
|
|
+------------------------------------------+
|
|
| 2. HARD EXCLUSION |
|
|
| Remove: hidden items, blocked |
|
|
| creators, exclude_ids |
|
|
| Cost: O(1) per candidate (bitmap) |
|
|
+------------------------------------------+
|
|
|
|
|
v
|
|
+------------------------------------------+
|
|
| 3. FILTER EVALUATION |
|
|
| Apply user-specified filters: |
|
|
| metadata, date, engagement threshold, |
|
|
| user state, geographic |
|
|
| Cost: O(1) per filter per candidate |
|
|
+------------------------------------------+
|
|
|
|
|
v
|
|
+------------------------------------------+
|
|
| 4. BOOST APPLICATION |
|
|
| Add weighted signal, relationship, |
|
|
| social proof, recency, cohort boosts |
|
|
| Cost: ~50ns per candidate per boost |
|
|
+------------------------------------------+
|
|
|
|
|
v
|
|
+------------------------------------------+
|
|
| 5. PENALTY APPLICATION |
|
|
| Subtract weighted negative signal |
|
|
| penalties (skip, dislike, downvote) |
|
|
| Cost: ~30ns per candidate per penalty |
|
|
+------------------------------------------+
|
|
|
|
|
v
|
|
+------------------------------------------+
|
|
| 6. GATE EVALUATION |
|
|
| Remove candidates below quality |
|
|
| thresholds (completion, engagement |
|
|
| ratio). Exploration items bypass. |
|
|
| Cost: O(1) per gate per candidate |
|
|
+------------------------------------------+
|
|
|
|
|
v
|
|
+------------------------------------------+
|
|
| 7. SCORE NORMALIZATION |
|
|
| Normalize composite scores to |
|
|
| [0.0, 1.0] range using min-max |
|
|
| within the surviving candidate set |
|
|
| Cost: O(n) for min/max scan, O(n) |
|
|
| for normalization |
|
|
+------------------------------------------+
|
|
|
|
|
v
|
|
+------------------------------------------+
|
|
| 8. DIVERSITY ENFORCEMENT |
|
|
| Greedy MMR reranking to enforce: |
|
|
| max_per_creator, format_mix, |
|
|
| topic_diversity |
|
|
| Cost: O(n * LIMIT) in the worst case |
|
|
+------------------------------------------+
|
|
|
|
|
v
|
|
+------------------------------------------+
|
|
| 9. EXPLORATION INJECTION |
|
|
| Replace exploration_budget % of |
|
|
| results with exploration candidates |
|
|
| (new items, cold-start, hidden gems) |
|
|
| Cost: O(LIMIT) |
|
|
+------------------------------------------+
|
|
|
|
|
v
|
|
+------------------------------------------+
|
|
| 10. PAGINATION |
|
|
| Slice to requested page via cursor |
|
|
| or offset. Assemble response with |
|
|
| signal snapshots. |
|
|
| Cost: O(LIMIT) |
|
|
+------------------------------------------+
|
|
|
|
|
v
|
|
Final ranked result set
|
|
```
|
|
|
|
### Stage Details
|
|
|
|
**Stage 1: Candidate Retrieval.** Executes the profile's `CandidateStrategy` (Section 3). Produces a raw candidate set with initial retrieval scores. For ANN, the initial score is cosine similarity. For Scan, the initial score is 0.0. For Hybrid, the initial score is the fused text + vector score.
|
|
|
|
**Stage 2: Hard Exclusion.** Removes candidates that must never appear for this user, regardless of score. This stage evaluates the profile's `excludes` list:
|
|
|
|
```rust
|
|
pub enum Exclude {
|
|
/// Items where this user has the named signal. e.g., Exclude::signal("hide")
|
|
Signal(&str),
|
|
/// Items by creators with this relationship to the user. e.g., Exclude::relationship("blocked")
|
|
Relationship(&str),
|
|
}
|
|
```
|
|
|
|
Implementation: For `Exclude::signal("hide")`, check the user-to-item relationship for the `hide` flag (O(1) bitmap lookup). For `Exclude::relationship("blocked")`, resolve the user's blocked creator set (cached as a roaring bitmap) and filter. Additionally, any `exclude_ids` from the query are removed here.
|
|
|
|
**Stage 3: Filter Evaluation.** Applies user-specified query filters (metadata, date, engagement threshold, user state, geographic). All filters from the query's `filters: Vec<Filter>` are evaluated. Filters are AND-composed across dimensions; OR-composed within a dimension (e.g., `category IN [jazz, blues]`). Implementation uses pre-computed roaring bitmaps for keyword fields and range scans for numeric fields.
|
|
|
|
**Stage 4: Boost Application.** Adds weighted positive signals to each candidate's score (Section 5). Each boost reads one signal value or relationship weight per candidate and multiplies it by the boost weight. The result is added to the candidate's composite score.
|
|
|
|
**Stage 5: Penalty Application.** Subtracts weighted negative signals from each candidate's score (Section 6). Same mechanics as boosts but with negative contribution.
|
|
|
|
**Stage 6: Gate Evaluation.** Removes candidates below quality thresholds (Section 7). Gates are hard filters, not soft penalties. A candidate below the gate threshold is removed from the result set entirely. Exception: items flagged for exploration bypass gates (they have not accumulated enough signals for gate evaluation to be meaningful).
|
|
|
|
**Stage 7: Score Normalization.** Normalizes composite scores to the [0.0, 1.0] range using min-max normalization within the surviving candidate set (Section 8).
|
|
|
|
**Stage 8: Diversity Enforcement.** Reranks the scored candidates to enforce variety constraints (Section 9). This stage reorders results -- it does not remove them.
|
|
|
|
**Stage 9: Exploration Injection.** Replaces a configurable percentage of results with exploration candidates (Section 10). Exploration items are selected from the cold-start pool, the hidden-gems candidate set, or quality-weighted random sampling.
|
|
|
|
**Stage 10: Pagination.** Slices the final ranked set to the requested page using cursor-based pagination (Section 14). Assembles the response with signal snapshots for each result.
|
|
|
|
---
|
|
|
|
## 5. Boost Types
|
|
|
|
Boosts are the primary scoring mechanism. Each boost reads a signal value, a relationship weight, or a derived metric for a candidate and adds a weighted contribution to the candidate's composite score.
|
|
|
|
### 5.1 Signal Boost
|
|
|
|
Boosts a candidate's score based on a signal's value within a time window.
|
|
|
|
```rust
|
|
Boost::signal(
|
|
signal_name: &str, // "view", "like", "share", etc.
|
|
window: Window, // Window::hours(24), Window::days(7), etc.
|
|
aggregation: SignalAgg, // How to read the signal
|
|
weight: f64, // Contribution weight (typically 0.0 to 1.0)
|
|
)
|
|
```
|
|
|
|
**SignalAgg variants:**
|
|
|
|
| Aggregation | Description | Example |
|
|
|-------------|-------------|---------|
|
|
| `Value` | Raw aggregate value (count or weighted sum) in the window | `view.value(24h)` = 12,450 views in last 24h |
|
|
| `Velocity` | Rate of change within the window (events per hour) | `view.velocity(24h)` = 518.75 views/hour |
|
|
| `Ratio` | Signal value divided by view count (engagement ratio) | `like.ratio(7d)` = likes_7d / views_7d = 0.08 |
|
|
| `UniqueRatio` | Unique users / total count (new-user reach) | `view.unique_ratio(24h)` = unique viewers / total views |
|
|
| `DecayScore` | Running exponential decay score from hot tier | `view.decay_score()` -- no window, uses running score |
|
|
| `RelativeVelocity` | Short-window velocity / long-window velocity | `view.relative_velocity(1h, 24h)` = acceleration |
|
|
|
|
**Score contribution:**
|
|
|
|
```
|
|
candidate.score += normalize(signal_value) * weight
|
|
```
|
|
|
|
Where `normalize` maps the raw signal value to a [0, 1] range using the candidate set's percentile distribution (Section 8.3).
|
|
|
|
### 5.2 Relationship Boost
|
|
|
|
Boosts a candidate's score based on the querying user's relationship with the candidate's creator.
|
|
|
|
```rust
|
|
Boost::relationship(
|
|
edge_kind: &str, // "interaction_weight", "engagement_affinity"
|
|
weight: f64,
|
|
)
|
|
```
|
|
|
|
**Execution:** For each candidate, look up the relationship edge from the querying user to the candidate's creator. The edge weight (0.0 to 1.0) is multiplied by the boost weight and added to the score.
|
|
|
|
```
|
|
candidate.score += user_creator_edge_weight * weight
|
|
```
|
|
|
|
If no relationship edge exists, the contribution is 0.0.
|
|
|
|
### 5.3 Social Proof Boost
|
|
|
|
Boosts candidates that the user's follows have engaged with.
|
|
|
|
```rust
|
|
Boost::social_proof(weight: f64)
|
|
```
|
|
|
|
**Execution:**
|
|
|
|
1. Load the querying user's follow set (cached as a roaring bitmap).
|
|
2. For each candidate, count how many users in the follow set have a positive engagement signal (view, like, share) with this item in the last 24 hours.
|
|
3. Compute social proof score: `social_count / follow_count` (fraction of follows who engaged).
|
|
4. Contribution: `social_proof_score * weight`.
|
|
|
|
**Performance:** Social proof requires a per-candidate set intersection. This is the most expensive boost type. For 200 candidates with a follow set of 500, the cost is ~200 * ~50 ns = ~10 us (roaring bitmap intersection). Acceptable within the scoring budget.
|
|
|
|
**Optimization:** For large follow sets (>1000), pre-compute a "follow-engaged items in last 24h" bitmap during the background materializer cycle. The social proof check becomes a single bitmap test per candidate: O(1).
|
|
|
|
### 5.4 Recency Boost (Content Age Decay)
|
|
|
|
Applies time-based decay to candidate scores based on content age.
|
|
|
|
```rust
|
|
Boost::recency(
|
|
field: &str, // "created_at" typically
|
|
half_life: Duration, // how fast content ages out
|
|
)
|
|
```
|
|
|
|
Equivalently specified via `ProfileDecay`:
|
|
|
|
```rust
|
|
pub struct ProfileDecay {
|
|
pub field: &str,
|
|
pub half_life: Duration,
|
|
}
|
|
```
|
|
|
|
**Formula:**
|
|
|
|
```
|
|
recency_score = exp(-ln(2) / half_life_secs * (now - created_at).as_secs())
|
|
```
|
|
|
|
This produces a score in (0.0, 1.0] where items at age 0 score 1.0 and items at age `half_life` score 0.5.
|
|
|
|
**Application:** The recency score is multiplied into the composite score as a scaling factor, not added:
|
|
|
|
```
|
|
candidate.score *= recency_score
|
|
```
|
|
|
|
This ensures that old content's score decays proportionally, rather than being offset by a fixed amount.
|
|
|
|
| Half-Life | Interpretation |
|
|
|-----------|----------------|
|
|
| 12 hours | Aggressive decay. News, real-time surfaces. Score halves every 12 hours. |
|
|
| 48 hours | Standard feed decay. For You surfaces. |
|
|
| 7 days | Moderate decay. Browse and category pages. |
|
|
| 30 days | Slow decay. Search results, evergreen content. |
|
|
| 90 days | Very slow decay. Search for tutorials, documentation. |
|
|
|
|
### 5.5 Cohort Signal Boost
|
|
|
|
Boosts a candidate based on signal velocity within a specific cohort.
|
|
|
|
```rust
|
|
Boost::cohort_signal(
|
|
signal_name: &str, // "view", "share", etc.
|
|
cohort: CohortSource, // Named, Auto, or Predicate
|
|
window: Window,
|
|
aggregation: SignalAgg,
|
|
weight: f64,
|
|
)
|
|
```
|
|
|
|
**Execution:** Reads the cohort-scoped signal aggregate for the candidate (see Cohorts spec, Section 8). The cohort source determines how the aggregation scope is resolved:
|
|
|
|
- `CohortSource::Auto` -- derives the cohort from the querying user's attributes (region, age_range, top inferred interest).
|
|
- `CohortSource::Named(name)` -- uses a pre-defined named cohort.
|
|
- `CohortSource::Predicate(pred)` -- evaluates an ad-hoc cohort predicate.
|
|
|
|
**Score contribution:**
|
|
|
|
```
|
|
candidate.score += normalize(cohort_signal_value) * weight
|
|
```
|
|
|
|
**Fallback:** If cohort signal data is sparse (fewer than 50 events in the window for this item in this cohort), fall back to the global signal value with a 0.5x dampening factor:
|
|
|
|
```
|
|
if cohort_signal_count < 50 {
|
|
effective_value = global_signal_value * 0.5
|
|
}
|
|
```
|
|
|
|
### 5.6 Cohort-Relative Boost
|
|
|
|
Boosts items that are disproportionately popular within a cohort compared to the general population.
|
|
|
|
```rust
|
|
Boost::cohort_relative(
|
|
cohort: CohortSource,
|
|
window: Window,
|
|
weight: f64,
|
|
)
|
|
```
|
|
|
|
**Formula:**
|
|
|
|
```
|
|
cohort_relative_score = cohort_velocity / max(global_velocity, floor)
|
|
```
|
|
|
|
Where `floor` prevents division by near-zero (default: 10.0 events/hour, configurable via `CohortConfig::relative_score_floor`).
|
|
|
|
**Interpretation:** A score of 5.0 means the item is 5x more popular within this cohort than globally. This surfaces content with specific cohort resonance.
|
|
|
|
### 5.7 Preference Match Boost
|
|
|
|
Boosts candidates whose embedding is similar to the querying user's preference vector.
|
|
|
|
```rust
|
|
Boost::preference_match(weight: f64)
|
|
```
|
|
|
|
**Formula:**
|
|
|
|
```
|
|
preference_score = cosine_sim(user.preference_vector, candidate.embedding)
|
|
```
|
|
|
|
This is distinct from ANN candidate generation. ANN retrieves candidates by similarity; preference match re-scores candidates that may have been retrieved by a different strategy (e.g., CohortTrending candidates re-ranked by preference match).
|
|
|
|
---
|
|
|
|
## 6. Penalty Types
|
|
|
|
Penalties subtract from a candidate's score based on negative signals. They mirror the boost mechanics but with negative contribution.
|
|
|
|
```rust
|
|
pub struct Penalty {
|
|
pub signal: &str,
|
|
pub window: Window,
|
|
pub weight: f64, // Stored as positive; subtracted during scoring.
|
|
}
|
|
|
|
// Construction:
|
|
Penalty::signal(
|
|
signal_name: &str, // "skip", "dislike", "downvote"
|
|
window: Window,
|
|
weight: f64, // Positive value. Applied as -weight.
|
|
)
|
|
```
|
|
|
|
**Score contribution:**
|
|
|
|
```
|
|
candidate.score -= normalize(signal_value) * weight
|
|
```
|
|
|
|
**Per-user vs. per-item penalties:**
|
|
|
|
| Penalty Scope | Description | Example |
|
|
|---------------|-------------|---------|
|
|
| Per-item (global) | The signal count on the item itself from all users | `skip.value(24h)` = 500 skips in 24h (item is low quality) |
|
|
| Per-user-item | The signal from this specific user on this item | User skipped this item 3 seconds ago (personal negative) |
|
|
|
|
When a penalty signal name matches a user-to-item relationship signal (e.g., the user has a `skip` signal on this item), the per-user signal takes precedence and is applied as a stronger penalty multiplier:
|
|
|
|
```
|
|
if user_has_signal(user, item, signal_name) {
|
|
candidate.score -= user_signal_weight * weight * USER_PENALTY_MULTIPLIER
|
|
// USER_PENALTY_MULTIPLIER = 3.0 (per-user skip is 3x stronger than global skip rate)
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 7. Quality Gates
|
|
|
|
Gates are hard thresholds that exclude candidates from the result set. Unlike penalties (which reduce scores), gates produce binary accept/reject decisions.
|
|
|
|
### 7.1 Minimum Signal Gate
|
|
|
|
```rust
|
|
Gate::min(
|
|
signal_name: &str, // "completion"
|
|
window: Window, // Window::all_time()
|
|
threshold: f64, // 0.3
|
|
)
|
|
```
|
|
|
|
Items where `signal.value(window) < threshold` are excluded. The signal value is the weighted aggregate, not the raw count.
|
|
|
|
**Example:** `Gate::min("completion", Window::all_time(), 0.3)` excludes items with an average completion rate below 30%. This filters out content that most people abandon.
|
|
|
|
### 7.2 Ratio Gate
|
|
|
|
```rust
|
|
Gate::min_ratio(
|
|
ratio_name: &str, // "engagement_ratio"
|
|
threshold: f64, // 0.03
|
|
)
|
|
```
|
|
|
|
**Built-in ratios:**
|
|
|
|
| Ratio Name | Formula | Description |
|
|
|-----------|---------|-------------|
|
|
| `engagement_ratio` | `(likes + comments + shares) / views` | Overall engagement quality |
|
|
| `like_ratio` | `likes / views` | Positive sentiment rate |
|
|
| `completion_rate` | `weighted_sum(completion) / count(view)` | Content quality |
|
|
| `skip_ratio` | `skips / impressions` | Negative quality indicator |
|
|
|
|
### 7.3 Minimum Count Gate
|
|
|
|
```rust
|
|
Gate::min_count(
|
|
signal_name: &str, // "view"
|
|
window: Window, // Window::all_time()
|
|
count: u64, // 100
|
|
)
|
|
```
|
|
|
|
Items with fewer than `count` events are excluded. Used to ensure statistical significance before applying ratio-based quality gates.
|
|
|
|
### 7.4 Gate Bypass for Exploration
|
|
|
|
Items in the exploration pool (Section 10) bypass all gates. These are new items that have not accumulated enough signals for gate evaluation to be meaningful. Without this bypass, cold-start items would be permanently excluded by quality gates that require historical engagement data.
|
|
|
|
**Bypass mechanism:** During Stage 6 (Gate Evaluation), candidates flagged with `is_exploration_candidate: true` skip all gate checks. The exploration flag is set during Stage 9 (Exploration Injection) for items selected from the cold-start pool.
|
|
|
|
**Implementation detail:** Gates are evaluated before exploration injection in the pipeline order. To enable bypass, the pipeline performs a two-pass approach:
|
|
|
|
1. First pass: evaluate gates on all non-exploration candidates.
|
|
2. Reserve `exploration_budget * LIMIT` slots for exploration candidates (gate-exempt).
|
|
3. Final pass: inject exploration candidates into reserved slots.
|
|
|
|
---
|
|
|
|
## 8. Score Composition and Normalization
|
|
|
|
### 8.1 Composite Score Formula
|
|
|
|
The composite score for a candidate is computed as:
|
|
|
|
```
|
|
raw_score = initial_retrieval_score
|
|
+ SUM(boost_i.normalize(signal_i) * boost_i.weight)
|
|
- SUM(penalty_j.normalize(signal_j) * penalty_j.weight)
|
|
|
|
final_score = raw_score * recency_decay_factor
|
|
```
|
|
|
|
Where:
|
|
- `initial_retrieval_score` comes from the candidate generation strategy (cosine similarity for ANN, RRF score for Hybrid, 0.0 for Scan).
|
|
- Each boost and penalty contribution is independently normalized before weighting.
|
|
- Recency decay is applied multiplicatively (it scales the entire score, not offsets it).
|
|
|
|
### 8.2 Score Normalization: Min-Max Within Candidate Set
|
|
|
|
After all boosts and penalties are applied, the composite scores are normalized to the [0.0, 1.0] range using min-max normalization within the surviving candidate set:
|
|
|
|
```
|
|
normalized_score = (raw_score - min_score) / (max_score - min_score)
|
|
```
|
|
|
|
If `max_score == min_score` (all candidates scored equally), all normalized scores are set to 0.5.
|
|
|
|
**Why min-max, not z-score:** Min-max normalization is deterministic and produces scores in a bounded range, which is required for the `score` field in the response. Z-score normalization can produce unbounded negative values, which violates the non-negative score invariant. Min-max is also simpler to reason about when combining scores from different profiles.
|
|
|
|
### 8.3 Signal Value Normalization
|
|
|
|
Raw signal values (e.g., 12,450 views) must be normalized before weighting. Without normalization, a signal with large absolute values (views) would dominate a signal with small absolute values (share ratio).
|
|
|
|
**Normalization strategy: percentile rank within the candidate set.**
|
|
|
|
For each signal used in a boost or penalty, compute the percentile rank of each candidate's signal value within the candidate set:
|
|
|
|
```
|
|
percentile_rank(candidate, signal) = rank_of(candidate.signal_value) / candidate_count
|
|
```
|
|
|
|
This produces values in [0.0, 1.0] regardless of the signal's absolute scale. A candidate at the 90th percentile of views within the candidate set receives a normalized value of 0.9.
|
|
|
|
**Why percentile, not min-max on raw values:** Min-max normalization on raw signal values is sensitive to outliers. A single viral item with 10M views would compress all other items to near-zero. Percentile ranking is robust to outliers and ensures that boost weights behave consistently regardless of the signal's absolute scale.
|
|
|
|
**Pre-computed percentile tables:** For the most common signals (view, like, share, completion), the background materializer maintains approximate percentile tables (1000-bucket histograms) updated hourly. Query-time percentile lookup is O(1) via binary search on the histogram.
|
|
|
|
### 8.4 Cross-Signal Comparability
|
|
|
|
The percentile normalization strategy ensures that a 0.3 weight on `view.velocity(24h)` and a 0.2 weight on `like.ratio(7d)` produce comparable contributions regardless of the absolute scales of these signals. The weight directly controls the relative importance of each signal in the final score.
|
|
|
|
**Guideline for weight selection:**
|
|
|
|
| Total Weight | Interpretation |
|
|
|-------------|----------------|
|
|
| Sum of all boost weights = 1.0 | Each weight is the fraction of the score controlled by that signal |
|
|
| Any single weight > 0.5 | That signal dominates the ranking |
|
|
| All weights equal | Uniform blend of signals |
|
|
|
|
Weights are not required to sum to 1.0. The normalization step (Stage 7) rescales the composite score to [0, 1] regardless.
|
|
|
|
---
|
|
|
|
## 9. Diversity Enforcement
|
|
|
|
### 9.1 DiversitySpec
|
|
|
|
```rust
|
|
pub struct DiversitySpec {
|
|
/// Maximum number of items from the same creator in the result set.
|
|
/// None = no creator constraint.
|
|
pub max_per_creator: Option<u32>,
|
|
|
|
/// Ensure variety of content formats (video, short, article, etc.)
|
|
/// across the result set.
|
|
pub format_mix: bool,
|
|
|
|
/// Topic diversity score from 0.0 (no enforcement) to 1.0 (maximize).
|
|
/// Uses embedding-space spread via Maximal Marginal Relevance.
|
|
pub topic_diversity: Option<f64>,
|
|
|
|
/// Minimum representation per category. Ensures at least N items
|
|
/// from each represented category appear in the result set.
|
|
/// Only meaningful when results span multiple categories.
|
|
pub category_min: Option<u32>,
|
|
}
|
|
```
|
|
|
|
### 9.2 Algorithm: Greedy MMR Reranking
|
|
|
|
Diversity enforcement uses a greedy algorithm inspired by Maximal Marginal Relevance (Carbonell & Goldstein, SIGIR 1998). The algorithm iteratively selects the next item that maximizes a combination of relevance score and diversity contribution.
|
|
|
|
**Pseudocode:**
|
|
|
|
```
|
|
function diversity_rerank(scored_candidates, diversity_spec, limit):
|
|
selected = []
|
|
remaining = scored_candidates.sorted_by_score_desc()
|
|
creator_counts = {}
|
|
format_counts = {}
|
|
|
|
while |selected| < limit AND |remaining| > 0:
|
|
best_candidate = None
|
|
best_mmr_score = -inf
|
|
|
|
for candidate in remaining:
|
|
// Check hard diversity constraints
|
|
if max_per_creator is set:
|
|
if creator_counts[candidate.creator] >= max_per_creator:
|
|
continue // skip: creator already at limit
|
|
|
|
// Compute MMR score
|
|
relevance = candidate.normalized_score
|
|
diversity = 0.0
|
|
|
|
if topic_diversity is set:
|
|
// Embedding-space spread: minimum distance to any selected item
|
|
if |selected| > 0:
|
|
min_distance = min(embedding_distance(candidate, s) for s in selected)
|
|
diversity = min_distance // higher = more diverse
|
|
else:
|
|
diversity = 1.0 // first item has maximum diversity
|
|
|
|
// Format mix bonus
|
|
format_bonus = 0.0
|
|
if format_mix:
|
|
if format_counts[candidate.format] == 0:
|
|
format_bonus = 0.1 // bonus for introducing a new format
|
|
|
|
// Category minimum bonus
|
|
category_bonus = 0.0
|
|
if category_min is set:
|
|
if category_counts[candidate.category] < category_min:
|
|
category_bonus = 0.1 // bonus for underrepresented category
|
|
|
|
lambda = topic_diversity.unwrap_or(0.0)
|
|
mmr_score = (1.0 - lambda) * relevance
|
|
+ lambda * diversity
|
|
+ format_bonus
|
|
+ category_bonus
|
|
|
|
if mmr_score > best_mmr_score:
|
|
best_mmr_score = mmr_score
|
|
best_candidate = candidate
|
|
|
|
if best_candidate is None:
|
|
// All remaining candidates violate hard constraints.
|
|
// Relax max_per_creator by 1 and retry.
|
|
max_per_creator += 1
|
|
continue
|
|
|
|
selected.push(best_candidate)
|
|
remaining.remove(best_candidate)
|
|
creator_counts[best_candidate.creator] += 1
|
|
format_counts[best_candidate.format] += 1
|
|
category_counts[best_candidate.category] += 1
|
|
|
|
return selected
|
|
```
|
|
|
|
### 9.3 Diversity Constraint Details
|
|
|
|
**max_per_creator:** No more than N items from the same creator in the result page. This is the most common diversity constraint. When a creator has more than N items in the candidate set, only the top-N by score are eligible for selection; the rest are deferred to subsequent pages.
|
|
|
|
**format_mix:** When enabled, the algorithm introduces a bonus for selecting items of formats not yet represented in the result set. This ensures a feed of all-video does not dominate when articles, shorts, and podcasts are also available. The bonus is small (0.1) -- it does not override relevance, only breaks ties.
|
|
|
|
**topic_diversity:** Controls embedding-space spread of results. At 0.0, no topic diversity is enforced (pure relevance). At 1.0, maximum diversity is enforced (the algorithm strongly prefers items far from already-selected items in embedding space). Values of 0.3-0.7 are typical for feed surfaces.
|
|
|
|
**category_min:** Ensures that if results span multiple categories, each category gets at least N items. This prevents a dominant category from monopolizing the result set.
|
|
|
|
### 9.4 Diversity and Pagination
|
|
|
|
Diversity constraints apply **per page**, not globally across all pages. Each page independently satisfies the diversity spec. This means:
|
|
|
|
- Page 1 may have 2 items from creator X.
|
|
- Page 2 may also have 2 items from creator X (different items).
|
|
- The user never sees more than `max_per_creator` items from any creator in a single rendered batch.
|
|
|
|
**Rationale:** Global diversity across pages would require the database to maintain state across paginated queries, which conflicts with stateless cursor-based pagination. Per-page diversity is simpler, stateless, and matches user expectations (they process one page at a time).
|
|
|
|
### 9.5 Diversity as Reordering, Not Filtering
|
|
|
|
Diversity enforcement never reduces the result count. If `max_per_creator: 2` and a creator has 10 items in the top 50, 2 items appear in positions the algorithm selects, and the remaining 8 are pushed to lower positions or subsequent pages. No items are removed from the result set.
|
|
|
|
**Relaxation under pressure:** If hard diversity constraints make it impossible to fill the requested result count (e.g., only 3 creators exist in the candidate set with `max_per_creator: 1` and `LIMIT 50`), the algorithm relaxes `max_per_creator` incrementally until the result count is met.
|
|
|
|
---
|
|
|
|
## 10. Exploration Budget
|
|
|
|
The exploration budget injects items from outside the scoring pipeline's natural ranking into a percentage of results. This serves two purposes: cold-start item discovery and serendipitous discovery.
|
|
|
|
### 10.1 Configuration
|
|
|
|
```rust
|
|
pub exploration: f64, // Fraction of results reserved for exploration.
|
|
// Range: 0.0 to 0.5. Default: 0.0.
|
|
```
|
|
|
|
An exploration budget of 0.10 means 10% of results (e.g., 5 out of 50) are exploration items.
|
|
|
|
### 10.2 Exploration Candidate Selection
|
|
|
|
Exploration items are selected from three pools, in priority order:
|
|
|
|
**Pool 1: Cold-start items.** Items created within the cold-start window (configurable, default 7 days) that have fewer than the cold-start signal threshold (configurable, default 100 views). These items have not had enough exposure for the scoring pipeline to evaluate them fairly.
|
|
|
|
Selection within Pool 1: Quality-weighted random sampling. The quality weight is derived from the creator's historical performance (average completion rate of their catalog). Items from creators with high historical quality are more likely to be selected.
|
|
|
|
```
|
|
cold_start_weight(item) = creator_avg_completion_rate(item.creator) * recency_factor(item)
|
|
```
|
|
|
|
**Pool 2: Cohort trending.** Items trending within the querying user's auto-detected cohort that are not present in the main result set. These are items the user's demographic peers are engaging with but that the user's personal preference vector has not surfaced.
|
|
|
|
**Pool 3: Hidden gems.** Items with high quality signals (completion rate, like ratio) but low total reach (view count). These are items the algorithm has not surfaced widely but that perform well with their limited audience.
|
|
|
|
### 10.3 Exploration Injection
|
|
|
|
Exploration items are injected after diversity enforcement. They replace items at specific positions within the result set:
|
|
|
|
```
|
|
Injection positions for exploration_budget = 0.10, LIMIT = 50:
|
|
5 exploration items at positions: [4, 12, 23, 35, 45]
|
|
(distributed throughout the result set, not clustered)
|
|
```
|
|
|
|
**Position distribution:** Exploration items are placed at evenly-spaced intervals through the result set. They are never placed at positions 0-2 (the top results must be the highest-confidence recommendations) and never at the last position.
|
|
|
|
### 10.4 Exploration Decay
|
|
|
|
As a user engages more with the platform (accumulates more signals), the effective exploration percentage can decrease:
|
|
|
|
```
|
|
effective_exploration = base_exploration * exploration_decay_factor(user)
|
|
|
|
exploration_decay_factor(user) = max(0.3, 1.0 - log10(user_signal_count + 1) / 5.0)
|
|
```
|
|
|
|
| User Signal Count | Decay Factor | Effective Exploration (base 10%) |
|
|
|-------------------|-------------|----------------------------------|
|
|
| 0 (new user) | 1.0 | 10.0% |
|
|
| 10 | 0.8 | 8.0% |
|
|
| 100 | 0.6 | 6.0% |
|
|
| 1,000 | 0.4 | 4.0% |
|
|
| 10,000+ | 0.3 (floor) | 3.0% |
|
|
|
|
The floor of 30% of the base rate ensures that even heavily-engaged users continue to see some exploration content. This prevents the "filter bubble" effect.
|
|
|
|
### 10.5 Cold-Start User Handling
|
|
|
|
A new user with no signal history has no preference vector, no relationship graph, and no engagement history. The scoring pipeline has no personalization data to work with. The exploration budget is critical here:
|
|
|
|
- New users receive a boosted exploration budget: `min(0.50, exploration * 3.0)` (capped at 50% of results).
|
|
- Cold-start items in the exploration pool are selected using population-level priors: items with the highest global quality signals weighted by the user's declared metadata (region, language).
|
|
- As the user accumulates signals, the boosted exploration rate decays toward the base rate per Section 10.4.
|
|
|
|
---
|
|
|
|
## 11. Built-In Sort Modes
|
|
|
|
Sort modes are formula-based ranking functions that bypass the boost/penalty scoring pipeline. When a query specifies a `sort` mode (either at the query level or within the profile), the sort formula replaces stages 4-5 (boost and penalty application) of the scoring pipeline. Stages 2-3 (exclusion, filter), 6 (gates), 7 (normalization), 8 (diversity), 9 (exploration), and 10 (pagination) still apply.
|
|
|
|
### 11.1 Hot
|
|
|
|
```
|
|
hot_score(item) = log10(max(|positive - negative|, 1))
|
|
/ (age_hours + 2) ^ gravity
|
|
|
|
Where:
|
|
positive = upvotes + likes
|
|
negative = downvotes + dislikes
|
|
age_hours = (now - created_at).as_hours()
|
|
gravity = configurable, default 1.8
|
|
```
|
|
|
|
**Behavior:** Hot rewards early engagement but punishes age. An hour-old post with 500 upvotes scores higher than a day-old post with 2,000 upvotes. The gravity parameter controls how aggressively age suppresses score. Higher gravity = faster decay.
|
|
|
|
| Gravity | Behavior |
|
|
|---------|----------|
|
|
| 1.0 | Very slow decay. Content stays hot for days. |
|
|
| 1.5 | Moderate decay. Content refreshes every ~6 hours. |
|
|
| 1.8 | Standard (Reddit default). Content refreshes every ~3 hours. |
|
|
| 2.5 | Aggressive decay. Content refreshes hourly. |
|
|
|
|
**Use cases:** UC-06 (Browse/Category), UC-14 (Hot Surfaces), any community frontpage.
|
|
|
|
### 11.2 Trending
|
|
|
|
```
|
|
trending_score(item) = share_velocity(6h) * 0.5
|
|
+ view_velocity(6h) * 0.3
|
|
+ new_user_reach(24h) * 0.2
|
|
|
|
Where:
|
|
share_velocity(w) = share.count(w) / w.as_hours()
|
|
view_velocity(w) = view.count(w) / w.as_hours()
|
|
new_user_reach(w) = unique_view.count(w) / view.count(w)
|
|
// fraction of viewers new to this creator
|
|
```
|
|
|
|
**Behavior:** Pure velocity, no personalization, no total-count signals. A video with 500 total views but 400 in the last hour outranks a video with 10M total views and 200 in the last hour.
|
|
|
|
**Gate:** `engagement_ratio >= 0.03` to filter clickbait (high views, zero engagement).
|
|
|
|
**Use cases:** UC-03 (Trending), global/category/social-scoped trending.
|
|
|
|
### 11.3 Rising
|
|
|
|
```
|
|
rising_score(item) = relative_velocity(item) * age_boost(item)
|
|
|
|
Where:
|
|
relative_velocity(item) = view.velocity(1h) / max(creator_baseline_velocity, floor)
|
|
creator_baseline_velocity = creator.avg_view_velocity(7d)
|
|
floor = 1.0 // prevents division by zero for new creators
|
|
age_boost(item) = max(0.1, 1.0 - age_hours / 48.0)
|
|
// linear boost for items under 48 hours old
|
|
```
|
|
|
|
**Behavior:** Surfaces content overperforming relative to its creator's historical baseline. A small creator getting 10x their normal engagement is "rising" even if their absolute numbers are modest.
|
|
|
|
**Use cases:** UC-03 (Rising), UC-13 (Hidden Gems variant), breakout detection.
|
|
|
|
### 11.4 Controversial
|
|
|
|
```
|
|
controversial_score(item) = (positive * negative) / (positive + negative) ^ 2
|
|
|
|
Where:
|
|
positive = likes + upvotes + shares
|
|
negative = dislikes + downvotes + reports
|
|
```
|
|
|
|
**Behavior:** Maximizes the product of positive and negative engagement. A post with 1,000 upvotes and 1,000 downvotes (controversial score = 0.25) scores higher than a post with 1,800 upvotes and 200 downvotes (controversial score = 0.09).
|
|
|
|
**Gate:** `(positive + negative) >= 100` to filter items without enough total engagement to be genuinely controversial (not just unpopular).
|
|
|
|
**Use cases:** UC-14 (Controversial), debate surfaces, "spicy" content sections.
|
|
|
|
### 11.5 Hidden Gems
|
|
|
|
```
|
|
hidden_gems_score(item) = quality_score(item) * inverse_reach(item)
|
|
|
|
Where:
|
|
quality_score(item) = completion_rate(all_time) * 0.6
|
|
+ like_ratio(all_time) * 0.4
|
|
inverse_reach(item) = 1.0 / log10(view.count(all_time) + 10)
|
|
```
|
|
|
|
**Behavior:** Surfaces high-quality content with low total reach. The logarithmic inverse ensures diminishing penalty as reach grows -- an item with 100 views is penalized similarly to one with 1,000 views, but both score much higher than one with 1M views.
|
|
|
|
**Gate:** `completion_rate(all_time) >= 0.5` (quality floor).
|
|
|
|
**Filter:** `created_within(30d)` typically applied (only recent hidden gems, not decade-old obscure content).
|
|
|
|
**Use cases:** UC-13 (Hidden Gems), "You Might Have Missed," editorial discovery.
|
|
|
|
### 11.6 Shuffle
|
|
|
|
```
|
|
shuffle_score(item) = random(seed) * quality_weight(item)
|
|
|
|
Where:
|
|
quality_weight(item) = sqrt(quality_score(item))
|
|
quality_score(item) = completion_rate * 0.5 + like_ratio * 0.3 + log10(views + 1) * 0.2
|
|
seed = hash(user_id, timestamp_minute) // same results for same user within 1 minute
|
|
```
|
|
|
|
**Behavior:** Quality-weighted random sampling. High-quality items are more likely to appear but not guaranteed. The seed ensures deterministic results within short time windows to prevent jarring re-shuffles on page refresh.
|
|
|
|
**Use cases:** Music playlists, "surprise me" buttons, mood-based discovery.
|
|
|
|
### 11.7 Top (Windowed)
|
|
|
|
```
|
|
top_score(item, window) = weighted_signal_sum(item, window)
|
|
|
|
Where:
|
|
weighted_signal_sum(item, w) = view.count(w) * 0.3
|
|
+ like.count(w) * 0.3
|
|
+ share.count(w) * 0.2
|
|
+ comment.count(w) * 0.1
|
|
+ completion_rate(w) * view.count(w) * 0.1
|
|
```
|
|
|
|
**Window variants:**
|
|
|
|
| Sort Mode | Window | Description |
|
|
|-----------|--------|-------------|
|
|
| `Sort::TopHour` | 1 hour | Real-time quality |
|
|
| `Sort::TopToday` | 24 hours | Daily best |
|
|
| `Sort::TopWeek` | 7 days | Weekly digest |
|
|
| `Sort::TopMonth` | 30 days | Monthly recap |
|
|
| `Sort::TopYear` | 365 days | Annual best |
|
|
| `Sort::TopAllTime` | All time | Classic / best-of |
|
|
|
|
**Use cases:** UC-06 (Browse with sort mode), community "best of" surfaces.
|
|
|
|
### 11.8 Simple Field Sorts
|
|
|
|
These sort modes are direct field-value sorts without formula computation.
|
|
|
|
| Sort Mode | Implementation | Notes |
|
|
|-----------|---------------|-------|
|
|
| `Sort::New` | `created_at DESC` | Pure chronological, no scoring |
|
|
| `Sort::Old` | `created_at ASC` | Archives, sequential viewing |
|
|
| `Sort::MostViewed` | `view.count(all_time) DESC` | Raw popularity |
|
|
| `Sort::MostLiked` | `like.count(all_time) DESC` | Positive sentiment |
|
|
| `Sort::MostCommented` | `comment.count(all_time) DESC` | Discussion |
|
|
| `Sort::MostShared` | `share.count(all_time) DESC` | Virality |
|
|
| `Sort::Shortest` | `duration ASC` | Quick content |
|
|
| `Sort::Longest` | `duration DESC` | Deep dives |
|
|
| `Sort::AlphabeticalAsc` | `title ASC` | Structured catalogs |
|
|
| `Sort::AlphabeticalDesc` | `title DESC` | Reverse alphabetical |
|
|
| `Sort::LiveViewerCount` | `live_viewer_count DESC` | Live surfaces |
|
|
| `Sort::DateSaved` | `user.saved_at(item) DESC` | Personal library |
|
|
| `Sort::CreatorEngagementRate` | `creator.engagement_rate DESC` | Creator discovery |
|
|
| `Sort::Relevance` | Text + semantic match score | Search only |
|
|
| `Sort::Personalized` | User preference match score | For You surfaces |
|
|
|
|
### 11.9 Sort Mode and Profile Interaction
|
|
|
|
When a query specifies both a `profile` and a `sort` override, the sort override replaces the profile's scoring logic:
|
|
|
|
| Query | Scoring Behavior |
|
|
|-------|-----------------|
|
|
| `USING PROFILE for_you` | Full boost/penalty pipeline from for_you profile |
|
|
| `USING PROFILE browse SORT hot` | Hot formula replaces boosts/penalties; browse filters, diversity, gates still apply |
|
|
| `SORT new` (no profile) | Pure `created_at DESC`; no boosts, no penalties, no gates |
|
|
|
|
The sort mode controls what determines the ordering. The profile (if specified) still controls candidate generation, filtering, gates, diversity, and exploration.
|
|
|
|
---
|
|
|
|
## 12. Cohort-Aware Ranking
|
|
|
|
### 12.1 Cohort Integration Points
|
|
|
|
Cohorts integrate with the ranking pipeline at three levels:
|
|
|
|
**Candidate generation.** The `CohortTrending` strategy (Section 3.5) generates candidates from items trending within a specific cohort.
|
|
|
|
**Boost signals.** `Boost::cohort_signal` and `Boost::cohort_relative` (Sections 5.5, 5.6) add cohort-scoped signal values as scoring components.
|
|
|
|
**Profile scoping.** The same profile can operate on different signal scopes without modification. A `trending` profile uses global velocity by default. When the query includes `FOR COHORT young_us_jazz`, the same profile reads cohort-scoped velocity instead.
|
|
|
|
### 12.2 Same Profile, Different Signal Scope
|
|
|
|
The key design decision: ranking profiles do not change when cohort scoping is applied. The profile defines *which signals matter* and *how to weight them*. The cohort defines *whose signals are counted*.
|
|
|
|
```
|
|
// Global trending
|
|
RETRIEVE items USING PROFILE trending LIMIT 25
|
|
--> reads view.velocity(24h) from global counters
|
|
|
|
// Cohort trending
|
|
RETRIEVE items USING PROFILE trending FOR COHORT young_us_jazz LIMIT 25
|
|
--> reads view.velocity(24h) from young_us_jazz cohort counters
|
|
--> same profile weights, same gates, same diversity
|
|
--> different signal data
|
|
```
|
|
|
|
**Implementation:** When a `FOR COHORT` clause is present, the signal read path in Stage 4 (Boost Application) routes signal reads to the cohort-scoped counters instead of global counters. This routing is transparent to the profile definition.
|
|
|
|
### 12.3 Cohort Fallback Behavior
|
|
|
|
When cohort signal data is sparse, the system falls back gracefully:
|
|
|
|
| Scenario | Behavior |
|
|
|----------|----------|
|
|
| Item has cohort tracking active and sufficient data | Use exact cohort signal values |
|
|
| Item has cohort tracking active but sparse data (<50 events) | Blend: `0.7 * cohort_value + 0.3 * global_value` |
|
|
| Item does not have cohort tracking active | Use global signal values with a dampening note in response |
|
|
| Cohort population below minimum threshold | Fall back to nearest parent cohort (see Cohorts spec, Section 9.4) |
|
|
|
|
The blending formula for sparse data prevents noisy cohort signals from dominating when the sample size is small. As cohort data accumulates, the blend converges to pure cohort values.
|
|
|
|
### 12.4 Auto-Cohort Detection
|
|
|
|
When `CohortSource::Auto` is specified, the system derives a cohort predicate from the querying user's attributes:
|
|
|
|
```
|
|
auto_cohort(user) = Predicate::and(
|
|
Predicate::eq("region", user.region),
|
|
Predicate::eq("age_range", user.age_range),
|
|
Predicate::contains("inferred_interests", user.top_interest),
|
|
)
|
|
```
|
|
|
|
If the auto-detected cohort has fewer active users than `min_trending_population`, the system progressively drops predicate terms (starting with `inferred_interests`, then `age_range`) until the population threshold is met.
|
|
|
|
---
|
|
|
|
## 13. Profile Presets
|
|
|
|
tidalDB ships with built-in profile presets for every standard surface. Applications can use these directly or override any aspect.
|
|
|
|
### 13.1 for_you
|
|
|
|
```rust
|
|
ProfileDef {
|
|
name: "for_you",
|
|
version: 1,
|
|
candidate: Candidate::Ann {
|
|
query_vector: VectorSource::UserPreference,
|
|
index: EntityKind::Item,
|
|
top_k: 500,
|
|
},
|
|
boosts: vec![
|
|
Boost::signal("view", Window::hours(24), Velocity, 0.3),
|
|
Boost::relationship("interaction_weight", 0.2),
|
|
Boost::social_proof(0.15),
|
|
],
|
|
decay: Some(ProfileDecay {
|
|
field: "created_at",
|
|
half_life: Duration::hours(48),
|
|
}),
|
|
gates: vec![
|
|
Gate::min("completion", Window::all_time(), 0.3),
|
|
],
|
|
penalties: vec![
|
|
Penalty::signal("skip", Window::hours(24), 0.5),
|
|
],
|
|
excludes: vec![
|
|
Exclude::signal("hide"),
|
|
Exclude::relationship("blocked"),
|
|
],
|
|
diversity: Some(DiversitySpec {
|
|
max_per_creator: Some(2),
|
|
format_mix: true,
|
|
topic_diversity: None,
|
|
category_min: None,
|
|
}),
|
|
exploration: 0.10,
|
|
sort: None,
|
|
extends: None,
|
|
}
|
|
```
|
|
|
|
**Surfaces:** UC-01 (For You Feed).
|
|
|
|
### 13.2 trending
|
|
|
|
```rust
|
|
ProfileDef {
|
|
name: "trending",
|
|
version: 1,
|
|
candidate: Candidate::Scan { entity: EntityKind::Item },
|
|
boosts: vec![
|
|
Boost::signal("share", Window::hours(6), Velocity, 0.5),
|
|
Boost::signal("view", Window::hours(6), Velocity, 0.3),
|
|
Boost::signal("view", Window::hours(24), UniqueRatio, 0.2),
|
|
],
|
|
gates: vec![
|
|
Gate::min_ratio("engagement_ratio", 0.03),
|
|
],
|
|
penalties: vec![],
|
|
excludes: vec![],
|
|
diversity: Some(DiversitySpec {
|
|
max_per_creator: Some(1),
|
|
format_mix: false,
|
|
topic_diversity: None,
|
|
category_min: None,
|
|
}),
|
|
exploration: 0.0,
|
|
decay: None,
|
|
sort: None,
|
|
extends: None,
|
|
}
|
|
```
|
|
|
|
**Surfaces:** UC-03 (Trending). Same profile for global, category-scoped, social-scoped, and cohort-scoped trending. The scope is determined by query filters and the `FOR COHORT` clause, not the profile.
|
|
|
|
### 13.3 search
|
|
|
|
```rust
|
|
ProfileDef {
|
|
name: "search",
|
|
version: 1,
|
|
candidate: Candidate::Hybrid {
|
|
text_weight: 0.6,
|
|
vector_weight: 0.4,
|
|
fusion: Fusion::Rrf { k: 60 },
|
|
},
|
|
boosts: vec![
|
|
Boost::signal("completion", Window::all_time(), Value, 0.15),
|
|
Boost::signal("like", Window::all_time(), Ratio, 0.10),
|
|
],
|
|
decay: Some(ProfileDecay {
|
|
field: "created_at",
|
|
half_life: Duration::days(90),
|
|
}),
|
|
gates: vec![],
|
|
penalties: vec![],
|
|
excludes: vec![
|
|
Exclude::signal("hide"),
|
|
Exclude::relationship("blocked"),
|
|
],
|
|
diversity: Some(DiversitySpec {
|
|
max_per_creator: Some(2),
|
|
format_mix: false,
|
|
topic_diversity: None,
|
|
category_min: None,
|
|
}),
|
|
exploration: 0.0,
|
|
sort: None,
|
|
extends: None,
|
|
}
|
|
```
|
|
|
|
**Surfaces:** UC-02 (Search). Text relevance is the floor. Personalization (via user preference match from ANN component) reorders within the relevant set.
|
|
|
|
### 13.4 following
|
|
|
|
```rust
|
|
ProfileDef {
|
|
name: "following",
|
|
version: 1,
|
|
candidate: Candidate::Relationship { edge: "follows" },
|
|
boosts: vec![],
|
|
decay: None,
|
|
gates: vec![],
|
|
penalties: vec![],
|
|
excludes: vec![],
|
|
diversity: None,
|
|
exploration: 0.0,
|
|
sort: Some(Sort::New),
|
|
extends: None,
|
|
}
|
|
```
|
|
|
|
**Surfaces:** UC-04 (Following Feed). Pure reverse chronological from followed creators. Minimal algorithmic intervention.
|
|
|
|
### 13.5 related
|
|
|
|
```rust
|
|
ProfileDef {
|
|
name: "related",
|
|
version: 1,
|
|
candidate: Candidate::Ann {
|
|
query_vector: VectorSource::ItemEmbedding("$anchor_item"),
|
|
index: EntityKind::Item,
|
|
top_k: 200,
|
|
},
|
|
boosts: vec![
|
|
Boost::preference_match(0.3),
|
|
Boost::signal("completion", Window::all_time(), Value, 0.2),
|
|
],
|
|
decay: Some(ProfileDecay {
|
|
field: "created_at",
|
|
half_life: Duration::days(14),
|
|
}),
|
|
gates: vec![
|
|
Gate::min("completion", Window::all_time(), 0.4),
|
|
],
|
|
penalties: vec![
|
|
Penalty::signal("skip", Window::hours(24), 0.3),
|
|
],
|
|
excludes: vec![
|
|
Exclude::signal("hide"),
|
|
Exclude::relationship("blocked"),
|
|
],
|
|
diversity: Some(DiversitySpec {
|
|
max_per_creator: Some(1),
|
|
format_mix: false,
|
|
topic_diversity: Some(0.3),
|
|
category_min: None,
|
|
}),
|
|
exploration: 0.05,
|
|
sort: None,
|
|
extends: None,
|
|
}
|
|
```
|
|
|
|
**Surfaces:** UC-05 (Related/Up Next). Semantic similarity as primary retrieval, personalization as secondary reranking.
|
|
|
|
### 13.6 browse
|
|
|
|
```rust
|
|
ProfileDef {
|
|
name: "browse",
|
|
version: 1,
|
|
candidate: Candidate::Scan { entity: EntityKind::Item },
|
|
boosts: vec![
|
|
Boost::signal("completion", Window::all_time(), Value, 0.5),
|
|
Boost::signal("like", Window::all_time(), Ratio, 0.3),
|
|
Boost::signal("view", Window::all_time(), Value, 0.2),
|
|
],
|
|
decay: Some(ProfileDecay {
|
|
field: "created_at",
|
|
half_life: Duration::days(30),
|
|
}),
|
|
gates: vec![],
|
|
penalties: vec![],
|
|
excludes: vec![],
|
|
diversity: Some(DiversitySpec {
|
|
max_per_creator: Some(2),
|
|
format_mix: false,
|
|
topic_diversity: None,
|
|
category_min: None,
|
|
}),
|
|
exploration: 0.05,
|
|
sort: None,
|
|
extends: None,
|
|
}
|
|
```
|
|
|
|
**Surfaces:** UC-06 (Browse/Category). Quality-dominant with moderate recency bias. Sort mode typically overridden at query time (`SORT hot`, `SORT new`, `SORT top_week`).
|
|
|
|
### 13.7 hidden_gems
|
|
|
|
```rust
|
|
ProfileDef {
|
|
name: "hidden_gems",
|
|
version: 1,
|
|
candidate: Candidate::Scan { entity: EntityKind::Item },
|
|
boosts: vec![],
|
|
gates: vec![
|
|
Gate::min("completion", Window::all_time(), 0.5),
|
|
Gate::min_count("view", Window::all_time(), 50),
|
|
],
|
|
penalties: vec![],
|
|
excludes: vec![],
|
|
diversity: Some(DiversitySpec {
|
|
max_per_creator: Some(1),
|
|
format_mix: true,
|
|
topic_diversity: Some(0.5),
|
|
category_min: None,
|
|
}),
|
|
exploration: 0.0,
|
|
sort: Some(Sort::HiddenGems),
|
|
extends: None,
|
|
decay: None,
|
|
}
|
|
```
|
|
|
|
**Surfaces:** UC-13 (Hidden Gems). High quality, low reach. Sort formula from Section 11.5.
|
|
|
|
### 13.8 notification
|
|
|
|
```rust
|
|
ProfileDef {
|
|
name: "notification",
|
|
version: 1,
|
|
candidate: Candidate::Relationship { edge: "follows" },
|
|
boosts: vec![
|
|
Boost::relationship("interaction_weight", 0.5),
|
|
Boost::signal("view", Window::hours(24), Velocity, 0.3),
|
|
],
|
|
decay: Some(ProfileDecay {
|
|
field: "created_at",
|
|
half_life: Duration::hours(12),
|
|
}),
|
|
gates: vec![],
|
|
penalties: vec![
|
|
Penalty::signal("notification_dismiss", Window::days(7), 0.3),
|
|
],
|
|
excludes: vec![
|
|
Exclude::relationship("muted"),
|
|
Exclude::relationship("blocked"),
|
|
],
|
|
diversity: Some(DiversitySpec {
|
|
max_per_creator: Some(1),
|
|
format_mix: false,
|
|
topic_diversity: None,
|
|
category_min: None,
|
|
}),
|
|
exploration: 0.0,
|
|
sort: None,
|
|
extends: None,
|
|
}
|
|
```
|
|
|
|
**Surfaces:** UC-07 (Notifications). Relationship strength dominant, aggressive recency decay (12h half-life).
|
|
|
|
### 13.9 live
|
|
|
|
```rust
|
|
ProfileDef {
|
|
name: "live",
|
|
version: 1,
|
|
candidate: Candidate::Scan { entity: EntityKind::Item },
|
|
boosts: vec![
|
|
Boost::relationship("interaction_weight", 0.4),
|
|
Boost::signal("live_viewer_count", Window::hours(1), Value, 0.3),
|
|
Boost::preference_match(0.3),
|
|
],
|
|
gates: vec![],
|
|
penalties: vec![],
|
|
excludes: vec![
|
|
Exclude::relationship("blocked"),
|
|
],
|
|
diversity: Some(DiversitySpec {
|
|
max_per_creator: Some(1),
|
|
format_mix: false,
|
|
topic_diversity: None,
|
|
category_min: None,
|
|
}),
|
|
exploration: 0.0,
|
|
decay: None,
|
|
sort: None,
|
|
extends: None,
|
|
}
|
|
```
|
|
|
|
**Surfaces:** UC-12 (Live Content). Requires `Filter::eq("status", "live")` at query time.
|
|
|
|
### 13.10 hot
|
|
|
|
```rust
|
|
ProfileDef {
|
|
name: "hot",
|
|
version: 1,
|
|
candidate: Candidate::Scan { entity: EntityKind::Item },
|
|
boosts: vec![],
|
|
gates: vec![],
|
|
penalties: vec![],
|
|
excludes: vec![],
|
|
diversity: Some(DiversitySpec {
|
|
max_per_creator: Some(2),
|
|
format_mix: false,
|
|
topic_diversity: None,
|
|
category_min: None,
|
|
}),
|
|
exploration: 0.0,
|
|
sort: Some(Sort::Hot { gravity: 1.8 }),
|
|
decay: None,
|
|
extends: None,
|
|
}
|
|
```
|
|
|
|
**Surfaces:** UC-14 (Hot Surfaces), community frontpages.
|
|
|
|
### 13.11 rising
|
|
|
|
```rust
|
|
ProfileDef {
|
|
name: "rising",
|
|
version: 1,
|
|
candidate: Candidate::Scan { entity: EntityKind::Item },
|
|
boosts: vec![],
|
|
gates: vec![
|
|
Gate::min_count("view", Window::hours(1), 10),
|
|
],
|
|
penalties: vec![],
|
|
excludes: vec![],
|
|
diversity: Some(DiversitySpec {
|
|
max_per_creator: Some(1),
|
|
format_mix: false,
|
|
topic_diversity: None,
|
|
category_min: None,
|
|
}),
|
|
exploration: 0.0,
|
|
sort: Some(Sort::Rising),
|
|
decay: None,
|
|
extends: None,
|
|
}
|
|
```
|
|
|
|
**Surfaces:** UC-03 (Rising), breakout detection.
|
|
|
|
### 13.12 controversial
|
|
|
|
```rust
|
|
ProfileDef {
|
|
name: "controversial",
|
|
version: 1,
|
|
candidate: Candidate::Scan { entity: EntityKind::Item },
|
|
boosts: vec![],
|
|
gates: vec![
|
|
Gate::min_count("like", Window::all_time(), 50),
|
|
Gate::min_count("dislike", Window::all_time(), 50),
|
|
],
|
|
penalties: vec![],
|
|
excludes: vec![],
|
|
diversity: Some(DiversitySpec {
|
|
max_per_creator: Some(2),
|
|
format_mix: false,
|
|
topic_diversity: None,
|
|
category_min: None,
|
|
}),
|
|
exploration: 0.0,
|
|
sort: Some(Sort::Controversial),
|
|
decay: None,
|
|
extends: None,
|
|
}
|
|
```
|
|
|
|
**Surfaces:** UC-14 (Controversial), debate surfaces.
|
|
|
|
### 13.13 Profile Preset Override
|
|
|
|
Applications can override any preset by defining a profile with the same name. The application's definition takes precedence. To restore a preset, drop the custom profile.
|
|
|
|
```rust
|
|
// Override the built-in trending profile with custom weights
|
|
db.define_profile(ProfileDef {
|
|
name: "trending",
|
|
version: 1,
|
|
candidate: Candidate::Scan { entity: EntityKind::Item },
|
|
boosts: vec![
|
|
Boost::signal("share", Window::hours(3), Velocity, 0.6), // shorter window, higher weight
|
|
Boost::signal("view", Window::hours(3), Velocity, 0.2),
|
|
Boost::signal("comment", Window::hours(6), Velocity, 0.2), // added comment velocity
|
|
],
|
|
gates: vec![
|
|
Gate::min_ratio("engagement_ratio", 0.05), // stricter gate
|
|
],
|
|
..ProfileDef::default()
|
|
})?;
|
|
```
|
|
|
|
---
|
|
|
|
## 14. Pagination and Cursors
|
|
|
|
### 14.1 Cursor-Based Pagination
|
|
|
|
Pagination uses opaque cursor tokens for stable result sets across pages. The cursor encodes the scoring state needed to resume retrieval without re-scoring previous pages.
|
|
|
|
```rust
|
|
pub struct Cursor {
|
|
/// The score of the last item on the previous page.
|
|
/// Used as the upper bound for the next page's candidates.
|
|
last_score: f64,
|
|
|
|
/// The ID of the last item on the previous page.
|
|
/// Used as a tiebreaker when scores are equal.
|
|
last_id: EntityId,
|
|
|
|
/// The profile version used for the previous page.
|
|
/// Ensures consistent scoring across pages.
|
|
profile_version: u32,
|
|
|
|
/// Timestamp when the cursor was created.
|
|
/// Used for staleness detection.
|
|
created_at: u64,
|
|
|
|
/// HMAC of the above fields to prevent tampering.
|
|
signature: [u8; 16],
|
|
}
|
|
```
|
|
|
|
**Staleness:** Cursors older than 30 minutes are rejected with `QueryError::StaleCursor`. The application must re-query from page 1. This prevents long-lived cursors from producing inconsistent results as signals change.
|
|
|
|
### 14.2 Diversity Across Pages
|
|
|
|
Diversity constraints are applied per page. The cursor does not carry cross-page diversity state. Each page independently satisfies the diversity spec.
|
|
|
|
To prevent the same item from appearing on multiple pages, the cursor's `last_score` and `last_id` act as an exclusion boundary: candidates with score >= `last_score` (and ID < `last_id` at equal score) are excluded from subsequent pages.
|
|
|
|
### 14.3 exclude_ids Alternative
|
|
|
|
For applications that prefer explicit exclusion over cursor-based pagination:
|
|
|
|
```rust
|
|
let page2 = db.retrieve(Retrieve {
|
|
profile: "for_you",
|
|
for_user: Some("user_123"),
|
|
exclude_ids: page1.results.iter().map(|r| r.id.clone()).collect(),
|
|
limit: 50,
|
|
..Default::default()
|
|
})?;
|
|
```
|
|
|
|
This re-executes the full scoring pipeline minus the excluded items. More expensive than cursor-based pagination but guaranteed fresh results on each page.
|
|
|
|
---
|
|
|
|
## 15. Performance Targets
|
|
|
|
These targets define the latency and throughput bounds for the ranking and scoring system. Regressions against these numbers are treated as bugs.
|
|
|
|
### 15.1 End-to-End Query Latency
|
|
|
|
| Query Type | LIMIT | Target (p50) | Target (p99) | Measurement Point |
|
|
|-----------|-------|-------------|-------------|-------------------|
|
|
| RETRIEVE with ANN profile (for_you) | 50 | < 30ms | < 50ms | `db.retrieve()` return |
|
|
| RETRIEVE with Scan profile (trending) | 25 | < 20ms | < 40ms | `db.retrieve()` return |
|
|
| RETRIEVE with Relationship (following) | 50 | < 15ms | < 30ms | `db.retrieve()` return |
|
|
| SEARCH with Hybrid profile | 20 | < 30ms | < 50ms | `db.search()` return |
|
|
| RETRIEVE with CohortTrending | 25 | < 30ms | < 50ms | `db.retrieve()` return |
|
|
| SEARCH WITHIN TRENDING FOR COHORT | 20 | < 35ms | < 50ms | `db.search()` return |
|
|
|
|
### 15.2 Scoring Pipeline Stage Latency
|
|
|
|
| Stage | Target (200 candidates) | Target (500 candidates) |
|
|
|-------|------------------------|------------------------|
|
|
| Hard exclusion (bitmap) | < 50 us | < 100 us |
|
|
| Filter evaluation | < 100 us | < 200 us |
|
|
| Boost application (3 boosts) | < 30 us | < 75 us |
|
|
| Penalty application (1 penalty) | < 10 us | < 25 us |
|
|
| Gate evaluation (2 gates) | < 20 us | < 50 us |
|
|
| Score normalization (min-max) | < 5 us | < 10 us |
|
|
| Diversity enforcement (MMR) | < 200 us | < 500 us |
|
|
| Exploration injection | < 10 us | < 20 us |
|
|
| Total scoring pipeline | < 500 us | < 1.2 ms |
|
|
|
|
### 15.3 Per-Candidate Scoring Cost
|
|
|
|
| Operation | Target per Candidate |
|
|
|-----------|---------------------|
|
|
| Decay score read (1 signal, 1 lambda) | ~15 ns |
|
|
| Windowed count read (1h window) | ~200 ns |
|
|
| Velocity computation | ~500 ns |
|
|
| Relationship edge weight lookup | ~50 ns |
|
|
| Social proof check (bitmap test) | ~50 ns |
|
|
| Cosine similarity (1536D, normalized) | ~500 ns |
|
|
| Total per candidate (typical for_you) | ~1.5 us |
|
|
|
|
### 15.4 Profile Definition Latency
|
|
|
|
| Operation | Target |
|
|
|-----------|--------|
|
|
| `define_profile()` | < 1ms |
|
|
| `get_profile()` | < 100 us |
|
|
| `list_profiles()` | < 500 us |
|
|
| Profile validation (at definition time) | < 5ms |
|
|
|
|
---
|
|
|
|
## 16. Invariants and Correctness Guarantees
|
|
|
|
### Scoring Invariants
|
|
|
|
**INV-RANK-1: Deterministic scoring.** Given the same candidate set, the same profile, and the same signal state, the scoring pipeline produces identical results. No randomness in scoring (shuffle mode uses a deterministic seed).
|
|
|
|
**INV-RANK-2: Score non-negativity.** After normalization, all scores are in the range [0.0, 1.0]. No candidate has a negative normalized score.
|
|
|
|
**INV-RANK-3: Exclusion completeness.** Items matching any `Exclude` predicate in the active profile never appear in results. Blocked creators' items never appear in any query for the blocking user. Hidden items never appear for the hiding user.
|
|
|
|
**INV-RANK-4: Gate strictness.** Non-exploration items below a gate threshold never appear in results. This is a hard invariant, not a soft preference.
|
|
|
|
**INV-RANK-5: Diversity satisfaction.** The diversity spec is satisfied in every result page, unless impossible due to insufficient candidate variety (in which case constraints are relaxed and the response includes a `DiversityWarning`).
|
|
|
|
**INV-RANK-6: Exploration budget bounds.** The number of exploration items in a result set is at most `ceil(exploration * LIMIT)`. The exploration budget is never exceeded.
|
|
|
|
**INV-RANK-7: Pagination consistency.** Items returned on page N do not appear on page N+1 (given a valid cursor). No duplicate items across cursor-paginated pages.
|
|
|
|
### Profile Invariants
|
|
|
|
**INV-PROF-1: Version monotonicity.** Profile versions are monotonically increasing. Defining a profile with a version <= the current latest version is rejected with `SchemaError::VersionConflict`.
|
|
|
|
**INV-PROF-2: Inheritance acyclicity.** Profile inheritance must form a DAG. Circular inheritance chains are rejected at definition time.
|
|
|
|
**INV-PROF-3: Signal reference validity.** Every signal name referenced in a profile boost, penalty, or gate must correspond to a defined signal type. Referencing an undefined signal returns `SchemaError::UnknownSignal`.
|
|
|
|
### Property Tests
|
|
|
|
```rust
|
|
// P1: Exclusion completeness -- blocked/hidden items never appear.
|
|
proptest! {
|
|
fn exclusion_completeness(
|
|
items in arb_items(100),
|
|
user in arb_user(),
|
|
blocked_creators in arb_creator_ids(5),
|
|
hidden_items in arb_item_ids(10),
|
|
) {
|
|
let results = score_pipeline(items, user, profile_with_excludes());
|
|
for result in &results {
|
|
prop_assert!(!blocked_creators.contains(&result.creator_id),
|
|
"blocked creator {} appeared in results", result.creator_id);
|
|
prop_assert!(!hidden_items.contains(&result.id),
|
|
"hidden item {} appeared in results", result.id);
|
|
}
|
|
}
|
|
}
|
|
|
|
// P2: Diversity constraints satisfied.
|
|
proptest! {
|
|
fn diversity_constraints_hold(
|
|
candidates in arb_scored_candidates(200),
|
|
max_per_creator in 1u32..5,
|
|
limit in 10u32..50,
|
|
) {
|
|
let spec = DiversitySpec {
|
|
max_per_creator: Some(max_per_creator),
|
|
..Default::default()
|
|
};
|
|
let results = diversity_rerank(&candidates, &spec, limit as usize);
|
|
|
|
let mut creator_counts: HashMap<CreatorId, u32> = HashMap::new();
|
|
for result in &results {
|
|
*creator_counts.entry(result.creator_id).or_default() += 1;
|
|
}
|
|
for (creator, count) in &creator_counts {
|
|
prop_assert!(*count <= max_per_creator,
|
|
"creator {} has {} items, max is {}",
|
|
creator, count, max_per_creator);
|
|
}
|
|
}
|
|
}
|
|
|
|
// P3: Gate bypass for exploration items.
|
|
proptest! {
|
|
fn exploration_items_bypass_gates(
|
|
candidates in arb_scored_candidates(100),
|
|
exploration_budget in 0.05f64..0.20,
|
|
) {
|
|
let profile = profile_with_gates_and_exploration(exploration_budget);
|
|
let results = full_pipeline(&candidates, &profile, 50);
|
|
|
|
let exploration_items: Vec<_> = results.iter()
|
|
.filter(|r| r.is_exploration)
|
|
.collect();
|
|
|
|
// Exploration items may have signals below gate thresholds
|
|
// (that's the point -- they're new items). This test verifies
|
|
// they are included despite not meeting gate criteria.
|
|
prop_assert!(exploration_items.len() <= (exploration_budget * 50.0).ceil() as usize);
|
|
}
|
|
}
|
|
|
|
// P4: Pagination produces no duplicates.
|
|
proptest! {
|
|
fn pagination_no_duplicates(
|
|
candidates in arb_scored_candidates(200),
|
|
page_size in 10u32..50,
|
|
) {
|
|
let page1 = retrieve_with_cursor(&candidates, None, page_size);
|
|
let page2 = retrieve_with_cursor(&candidates, page1.next_cursor, page_size);
|
|
|
|
let page1_ids: HashSet<_> = page1.results.iter().map(|r| &r.id).collect();
|
|
let page2_ids: HashSet<_> = page2.results.iter().map(|r| &r.id).collect();
|
|
|
|
let overlap: Vec<_> = page1_ids.intersection(&page2_ids).collect();
|
|
prop_assert!(overlap.is_empty(),
|
|
"duplicate items across pages: {:?}", overlap);
|
|
}
|
|
}
|
|
|
|
// P5: Score normalization produces valid range.
|
|
proptest! {
|
|
fn normalized_scores_in_range(
|
|
raw_scores in prop::collection::vec(
|
|
prop::num::f64::NORMAL | prop::num::f64::POSITIVE,
|
|
10..500
|
|
),
|
|
) {
|
|
let normalized = min_max_normalize(&raw_scores);
|
|
for &score in &normalized {
|
|
prop_assert!(score >= 0.0 && score <= 1.0,
|
|
"normalized score {} out of range", score);
|
|
}
|
|
}
|
|
}
|
|
|
|
// P6: Deterministic scoring.
|
|
proptest! {
|
|
fn scoring_deterministic(
|
|
candidates in arb_scored_candidates(100),
|
|
profile in arb_profile(),
|
|
) {
|
|
let results1 = full_pipeline(&candidates, &profile, 50);
|
|
let results2 = full_pipeline(&candidates, &profile, 50);
|
|
|
|
for (r1, r2) in results1.iter().zip(results2.iter()) {
|
|
prop_assert_eq!(r1.id, r2.id);
|
|
prop_assert!((r1.score - r2.score).abs() < f64::EPSILON,
|
|
"non-deterministic scoring: {} vs {}", r1.score, r2.score);
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 17. Integration Points
|
|
|
|
### 17.1 Signal System Integration
|
|
|
|
The ranking pipeline reads signal data from the three-tier signal ledger (Signal System spec, Section 3):
|
|
|
|
| Ranking Need | Signal Tier | Read Latency |
|
|
|-------------|------------|-------------|
|
|
| Decay scores for boost application | Hot tier (atomic reads) | ~15 ns per entity |
|
|
| Windowed counts for velocity | Warm tier (bucket sums) | ~200-500 ns per entity |
|
|
| Cohort-scoped aggregates | Cohort counters (disk-backed) | ~500 ns - 2 us per entity |
|
|
| All-time counts for gates | Warm tier (atomic counter) | ~2 ns per entity |
|
|
| Signal snapshot for response | All tiers | ~5 us per entity |
|
|
|
|
The ranking module never writes to the signal system. It is a pure consumer of signal state.
|
|
|
|
### 17.2 Relationship System Integration
|
|
|
|
The ranking pipeline reads relationship data for:
|
|
|
|
- **Candidate generation** (Relationship strategy): traverses follow edges.
|
|
- **Boost application** (Relationship boost): reads `interaction_weight` edges.
|
|
- **Social proof**: reads follow-set bitmap and per-item engagement flags.
|
|
- **Hard exclusion**: reads blocked/muted edges.
|
|
|
|
Relationship reads use the adjacency list storage format from the Relationships spec (Section 5). Forward adjacency lists (user -> creators they follow) are cached in memory as roaring bitmaps for O(1) membership tests.
|
|
|
|
### 17.3 Query Engine Integration
|
|
|
|
The query engine is the orchestrator. It receives a `Retrieve` or `Search` request, resolves the profile, executes candidate generation (delegating to the ANN index, Tantivy, or relationship store as needed), and invokes the scoring pipeline.
|
|
|
|
```
|
|
Query Engine
|
|
├── Profile Resolution
|
|
│ └── Schema Catalog (profiles, signals, entities)
|
|
├── Candidate Generation
|
|
│ ├── ANN Index (USearch) -- for ANN strategy
|
|
│ ├── Tantivy -- for Hybrid strategy (text component)
|
|
│ ├── Relationship Store -- for Relationship strategy
|
|
│ └── Signal System -- for CohortTrending strategy
|
|
├── Scoring Pipeline (this spec)
|
|
│ ├── Signal reads (hot/warm/cold tier)
|
|
│ ├── Relationship reads (adjacency lists)
|
|
│ └── Cohort reads (dimensional rollups)
|
|
└── Response Assembly
|
|
└── Signal snapshots for rendering
|
|
```
|
|
|
|
### 17.4 Cohort System Integration
|
|
|
|
When a query includes `FOR COHORT`, the query engine:
|
|
|
|
1. Resolves the cohort to a signal aggregation scope (Cohorts spec, Section 5).
|
|
2. Passes the scope to the scoring pipeline.
|
|
3. The scoring pipeline routes signal reads to cohort-scoped counters instead of global counters.
|
|
4. The response includes `cohort_info` with the cohort name, cardinality, and accuracy level.
|
|
|
|
---
|
|
|
|
## Appendix A: Glossary
|
|
|
|
| Term | Definition |
|
|
|------|------------|
|
|
| **Ranking Profile** | A named, versioned scoring function declared in schema that fully specifies how candidates are retrieved, scored, filtered, diversified, and paginated |
|
|
| **Candidate Generation** | The first stage of the ranking pipeline that produces a raw set of entities with initial retrieval scores |
|
|
| **Boost** | A positive scoring component that adds a weighted signal value, relationship weight, or derived metric to a candidate's score |
|
|
| **Penalty** | A negative scoring component that subtracts a weighted signal value from a candidate's score |
|
|
| **Gate** | A hard quality threshold that excludes candidates from the result set (binary accept/reject) |
|
|
| **Diversity Spec** | A set of constraints that control variety in the result set (max per creator, format mix, topic diversity) |
|
|
| **Exploration Budget** | The fraction of results reserved for cold-start items, hidden gems, and serendipitous discovery |
|
|
| **Sort Mode** | A formula-based ranking function that replaces the boost/penalty scoring pipeline |
|
|
| **MMR** | Maximal Marginal Relevance -- a greedy reranking algorithm that balances relevance with diversity |
|
|
| **RRF** | Reciprocal Rank Fusion -- a rank-based score fusion strategy for combining text and vector retrieval results |
|
|
| **Profile Decay** | Multiplicative time-based decay applied to scores based on content age |
|
|
| **Percentile Normalization** | Mapping raw signal values to [0, 1] using rank within the candidate set |
|
|
| **Cold Start** | The state where an item or user has insufficient signal history for the scoring pipeline to evaluate them |
|
|
| **Cohort Scoping** | Routing signal reads to cohort-specific counters instead of global counters, changing the data source without changing the profile |
|
|
|
|
## Appendix B: References
|
|
|
|
1. Carbonell, J., Goldstein, J. "The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries." SIGIR 1998.
|
|
2. Cormack, G.V., Clarke, C.L.A., Buettcher, S. "Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods." SIGIR 2009.
|
|
3. Cormode, G., Shkapenyuk, V., Srivastava, D., Xu, B. "Forward Decay: A Practical Time Decay Model for Streaming Systems." ICDE 2009.
|
|
4. Reddit Hot Ranking Algorithm. "How Reddit Ranking Algorithms Work." Amir Salihefendic, 2015.
|
|
5. Hacker News Ranking Algorithm. Paul Graham, Y Combinator.
|
|
6. Wilson, E.B. "Probable Inference, the Law of Succession, and Statistical Inference." Journal of the American Statistical Association, 1927. (Lower bound of Wilson score interval for rating-based ranking.)
|
|
7. Signal System Specification. `docs/specs/03-signal-system.md`.
|
|
8. Relationships Specification. `docs/specs/04-relationships.md`.
|
|
9. Cohorts Specification. `docs/specs/05-cohorts.md`.
|
|
10. Vector Retrieval Specification. `docs/specs/07-vector-retrieval.md`.
|
|
11. Text Retrieval Specification. `docs/specs/06-text-retrieval.md`.
|