tidaldb/docs/specs/09-ranking-scoring.md

Ranking and Scoring Specification

Status: Draft Authors: tidalDB Engineering Date: 2026-02-20 Depends on: Signal System (03), Relationships (04), Cohorts (05), Text Retrieval (06), Vector Retrieval (07) Research: docs/research/ann_for_tidaldb.md, docs/research/tidaldb_signal_ledger.md, docs/research/tantivy.md

---

Table of Contents

1. Overview
2. Ranking Profile Declaration
3. Candidate Generation Strategies
4. Scoring Pipeline
5. Boost Types
6. Penalty Types
7. Quality Gates
8. Score Composition and Normalization
9. Diversity Enforcement
10. Exploration Budget
11. Built-In Sort Modes
12. Cohort-Aware Ranking
13. Profile Presets
14. Pagination and Cursors
15. Performance Targets
16. Invariants and Correctness Guarantees
17. Integration Points

---

1. Overview

The ranking and scoring system is the core value proposition of tidalDB. It replaces the external ranking service that today stitches together signals from Elasticsearch, Redis, a feature store, and a vector database. In tidalDB, ranking is a database primitive: the application names a profile, the database executes the entire pipeline.

The ranking system takes as input a set of candidate entities (generated by one of several retrieval strategies), a user context (preference vector, relationship graph, signal history), and a ranking profile (a named, versioned scoring function declared in schema). It produces as output a scored, diversified, paginated result set ready for rendering -- no re-ranking by the application, ever.

Design Principles

1. Profiles are data, not code. Ranking profiles are schema-level declarations stored in the database. A profile change never requires recompilation or redeployment. The query planner reasons about profile structure to optimize execution.

2. The pipeline is fixed; the weights are configurable. The nine-stage scoring pipeline (Section 4) executes in the same order for every query. Profiles configure what each stage does -- which signals to boost, which gates to apply, which diversity constraints to enforce -- but cannot alter the stage order.

3. Negative signals are structurally equal to positive signals. Skips, hides, downvotes are first-class inputs to the scoring function with the same weight, precision, and update immediacy as likes.

4. Diversity is a post-scoring constraint. Diversity enforcement reorders results after scoring. It never filters candidates out of the result set -- it demotes items that violate constraints and promotes items that satisfy them.

5. Graceful degradation, never failure. Under load, the system returns less precise rankings rather than errors. Degradation order: reduce candidate set, use coarser signal aggregates, skip diversity, serve from materialized cache.

6. Cold start is a database responsibility. New items with no signals and new users with no history receive sensible treatment via exploration budgets and population priors. The application does not manage this.

---

2. Ranking Profile Declaration

2.1 ProfileDef Structure

A ranking profile is a named, versioned scoring function that fully specifies how candidates are retrieved, scored, filtered, diversified, and paginated. The application says USING PROFILE for_you. The database executes everything.

pub struct ProfileDef {
    /// Unique profile name. Lowercase alphanumeric plus underscores.
    pub name: &str,

    /// Monotonically increasing version. Old versions remain queryable
    /// by specifying name + version at query time.
    pub version: u32,

    /// How candidates are generated (Section 3).
    pub candidate: CandidateStrategy,

    /// Optional parent profile. This profile inherits all fields from
    /// the parent and overrides only the fields explicitly set.
    pub extends: Option<ProfileRef>,

    /// Positive signal boosts applied to candidate scores (Section 5).
    pub boosts: Vec<Boost>,

    /// Content age decay applied to all candidates (Section 5.4).
    pub decay: Option<ProfileDecay>,

    /// Quality gates -- hard thresholds that exclude candidates (Section 7).
    pub gates: Vec<Gate>,

    /// Negative signal penalties subtracted from scores (Section 6).
    pub penalties: Vec<Penalty>,

    /// Hard exclusions -- items matching these are removed before scoring.
    pub excludes: Vec<Exclude>,

    /// Post-scoring diversity constraints (Section 9).
    pub diversity: Option<DiversitySpec>,

    /// Fraction of results reserved for exploration (Section 10).
    /// Range: 0.0 to 0.5. Default: 0.0 (no exploration).
    pub exploration: f64,

    /// Optional explicit sort mode override. When set, bypasses the
    /// boost/penalty scoring pipeline and uses a formula-based sort
    /// (Section 11). Used by sort modes like Hot, Trending, Rising.
    pub sort: Option<Sort>,
}

2.2 Version Semantics

Profiles are versioned to enable safe iteration and A/B testing.

Versioning rules:

1. Each call to db.define_profile() with an existing profile name creates a new version. The version number is monotonically increasing.
2. The latest version is used by default when a query specifies USING PROFILE for_you without a version qualifier.
3. Previous versions remain queryable by specifying profile: "for_you@1" or equivalently profile_version: Some(1).
4. Versions are immutable once defined. To modify a profile, define a new version.
5. Maximum 100 versions per profile name. Older versions can be garbage-collected with db.prune_profile_versions("for_you", keep_latest: 10).

Storage: Profiles are stored in the schema catalog alongside entity definitions and signal definitions. They are loaded into memory at startup and cached for the lifetime of the database instance. Profile definitions are persisted in the WAL for crash recovery.

2.3 Profile Inheritance

Profiles can extend other profiles to reduce duplication. A child profile inherits all fields from its parent and overrides only the fields explicitly set.

// Base browse profile
db.define_profile(ProfileDef {
    name: "browse",
    version: 1,
    candidate: Candidate::Scan { entity: EntityKind::Item },
    boosts: vec![
        Boost::signal("completion", Window::all_time(), Value, 0.5),
        Boost::signal("like", Window::all_time(), Ratio, 0.3),
        Boost::signal("view", Window::all_time(), Value, 0.2),
    ],
    decay: Some(ProfileDecay {
        field: "created_at",
        half_life: Duration::days(30),
    }),
    diversity: Some(DiversitySpec {
        max_per_creator: Some(2),
        ..Default::default()
    }),
    ..ProfileDef::default()
})?;

// Personalized browse -- extends browse with user preference boost
db.define_profile(ProfileDef {
    name: "browse_personalized",
    version: 1,
    extends: Some(ProfileRef::latest("browse")),
    // Inherits candidate, boosts, decay, diversity from browse.
    // Adds preference match boost on top.
    boosts: vec![
        Boost::preference_match(0.3),
    ],
    // Inherited boosts from parent are appended, not replaced.
    // To replace, set extends: None and redefine all boosts.
    ..ProfileDef::default()
})?;

Inheritance resolution:

Field	Behavior
candidate	Child overrides parent if set; otherwise inherits.
boosts	Child boosts are appended to parent boosts.
decay	Child overrides parent if set.
gates	Child gates are appended to parent gates.
penalties	Child penalties are appended to parent penalties.
excludes	Child excludes are appended to parent excludes.
diversity	Child overrides parent if set.
exploration	Child overrides parent if set.
sort	Child overrides parent if set.

Inheritance depth: Maximum 3 levels. Deeper inheritance chains are rejected at definition time with SchemaError::InheritanceDepthExceeded.

2.4 A/B Testing

A/B testing is performed by defining multiple profile versions or separate profile names and specifying the desired variant at query time.

// Define two variants
db.define_profile(ProfileDef {
    name: "for_you",
    version: 2,
    // ... same as v1 but with adjusted weights
    boosts: vec![
        Boost::signal("view", Window::hours(24), Velocity, 0.4),  // was 0.3
        Boost::relationship("interaction_weight", 0.15),           // was 0.2
        Boost::social_proof(0.20),                                 // was 0.15
    ],
    ..base_for_you.clone()
})?;

// Control group
let control = db.retrieve(Retrieve {
    profile: "for_you",
    profile_version: Some(1),
    for_user: Some("user_123"),
    ..query.clone()
})?;

// Treatment group
let variant = db.retrieve(Retrieve {
    profile: "for_you",
    profile_version: Some(2),
    for_user: Some("user_123"),
    ..query.clone()
})?;

The database does not manage A/B assignment. The application decides which version each user sees. The database executes whichever version is requested.

---

3. Candidate Generation Strategies

Candidate generation is the first stage of the ranking pipeline. It produces a raw set of entities with initial retrieval scores. The strategy determines how candidates are found; subsequent pipeline stages determine how they are scored and ordered.

3.1 ANN (Approximate Nearest Neighbor)

Vector similarity search over embeddings. Used for personalized feeds and related content.

Candidate::Ann {
    /// Source of the query vector.
    query_vector: VectorSource,
    /// Entity type to search over.
    index: EntityKind,
    /// Number of candidates to retrieve from the ANN index.
    top_k: u32,
}

VectorSource variants:

Source	Description
VectorSource::UserPreference	The querying user's preference vector. Used by for_you.
VectorSource::ItemEmbedding(item_id)	A specific item's embedding. Used by related.
VectorSource::QueryEmbedding	The query vector passed inline (for SEARCH).
VectorSource::CreatorEmbedding(creator_id)	A creator's catalog embedding. Used by creator discovery.

Initial score: Cosine similarity in range [0.0, 1.0] (embeddings are normalized at insertion time per Coding Guidelines Section 4).

Filter interaction: Pre-filters (user state, blocked, unseen) are applied as predicate callbacks during HNSW traversal when selectivity is 2-100%. For selectivity below 2%, a roaring bitmap pre-filter with brute-force L2 scan is used. See Vector Retrieval spec (07) for the adaptive strategy.

3.2 Scan

Full entity scan with signal-based ranking. Used for trending, hot, and sort-mode-dominant queries where no embedding similarity is involved.

Candidate::Scan {
    /// Entity type to scan.
    entity: EntityKind,
}

Initial score: 0.0 for all candidates. Scoring is entirely determined by boosts, penalties, and sort mode formulas.

Optimization: A full scan of 10M entities is infeasible at query time. The scan strategy uses the following acceleration:

1. Signal-indexed scan. For velocity-based profiles (trending, rising), only entities with non-zero velocity in the relevant window are candidates. The warm tier's active-entity index provides this set (typically <500K entities out of 10M).
2. Metadata-indexed scan. Filters on keyword fields (category, format, status) are resolved to roaring bitmaps and intersected before any signal reads.
3. Top-K early termination. After the first pass produces rough scores, a heap-based top-K selection eliminates low-scoring candidates before expensive signal reads.

Performance: For a trending query with one category filter, the effective candidate set is typically 10K-50K entities, not 10M.

3.3 Hybrid (Text + Vector Fusion)

Combines full-text BM25 retrieval with vector similarity search. Used by the search profile.

Candidate::Hybrid {
    /// Weight of text (BM25) relevance in the fused score.
    text_weight: f64,
    /// Weight of vector (ANN) similarity in the fused score.
    vector_weight: f64,
    /// Fusion strategy.
    fusion: Fusion,
}

pub enum Fusion {
    /// Reciprocal Rank Fusion. Rank-based, no score normalization needed.
    /// k controls convergence -- higher k = more weight to lower-ranked items.
    /// Default k=60 (Cormack et al., SIGIR 2009).
    Rrf { k: u32 },

    /// Weighted linear combination of normalized scores.
    /// Requires min-max normalization of both score distributions.
    /// Use only after relevance labels exist to tune alpha.
    Linear { alpha: f64 },
}

RRF formula:

RRF_score(d) = text_weight / (k + rank_bm25(d)) + vector_weight / (k + rank_ann(d))

Initial score: The fused RRF or linear combination score.

Filter interaction: Text filters (keyword fields) are applied within the Tantivy query. Vector filters use the adaptive strategy from the Vector Retrieval spec.

3.4 Relationship (Graph Traversal)

Candidate generation via graph traversal. Used by the following profile and social-graph-scoped queries.

Candidate::Relationship {
    /// The relationship edge type to traverse.
    edge: &str,
}

Execution: Starting from the querying user, traverse outgoing edges of type edge (e.g., "follows"). Collect all items authored by the target entities (creators). These items form the candidate set.

Initial score: 0.0 for all candidates. Sort is typically by created_at DESC (chronological).

Filter interaction: Standard metadata filters apply after traversal. The traversal itself acts as a hard filter (only items from related entities are included).

Fan-out control: Maximum fan-out is bounded by the user's relationship count (e.g., 500 follows). Each creator's recent items are fetched using a bounded scan on the creator_id prefix in the entity store, limited to LIMIT * 2 items per creator to bound total candidate set size.

3.5 CohortTrending

Candidate generation scoped to items trending within a specific cohort. Used by cohort-aware trending profiles.

Candidate::CohortTrending {
    /// Which cohort to scope to.
    cohort: CohortSource,
    /// Time window for velocity computation.
    window: Window,
    /// Number of top trending candidates to retrieve.
    top_k: u32,
}

pub enum CohortSource {
    /// Derive cohort from the querying user's attributes.
    Auto,
    /// Use a specific named cohort.
    Named(String),
    /// Inline predicate.
    Predicate(Predicate),
}

Execution:

1. Resolve the cohort to a signal aggregation scope (see Cohorts spec, Section 7).
2. Scan all items with cohort tracking active (~100K items at the Signal System's threshold).
3. Read cohort-scoped velocity for the specified window.
4. Return the top top_k items by cohort velocity.

Initial score: Cohort-scoped velocity (events per unit time within the window).

Filter interaction: Metadata filters are applied after cohort velocity ranking.

3.6 Strategy Summary

Strategy	Use Cases	Initial Score	Typical Candidate Count
ANN	for_you, related, visual search	Cosine similarity [0, 1]	200-1000
Scan	trending, hot, rising, browse	0.0 (scored by boosts/sort)	10K-50K (after index acceleration)
Hybrid	search	Fused text + vector score	100-500
Relationship	following	0.0 (sorted by created_at)	500-5000
CohortTrending	trending_for_you, cohort trending	Cohort velocity	200-500

---

4. Scoring Pipeline

The scoring pipeline is a nine-stage transformation that converts raw candidates into a ranked, diversified, paginated result set. The stages execute in fixed order. Every ranking query passes through all nine stages, though some stages may be no-ops depending on the profile configuration.

Pipeline Diagram

                    Raw candidate set from retrieval strategy
                                    |
                                    v
             +------------------------------------------+
             |  1. CANDIDATE RETRIEVAL                   |
             |     ANN / Scan / Hybrid / Relationship /  |
             |     CohortTrending                        |
             |     Output: candidates[] with initial     |
             |     scores from retrieval strategy        |
             +------------------------------------------+
                                    |
                                    v
             +------------------------------------------+
             |  2. HARD EXCLUSION                        |
             |     Remove: hidden items, blocked          |
             |     creators, exclude_ids                  |
             |     Cost: O(1) per candidate (bitmap)     |
             +------------------------------------------+
                                    |
                                    v
             +------------------------------------------+
             |  3. FILTER EVALUATION                     |
             |     Apply user-specified filters:          |
             |     metadata, date, engagement threshold, |
             |     user state, geographic                |
             |     Cost: O(1) per filter per candidate   |
             +------------------------------------------+
                                    |
                                    v
             +------------------------------------------+
             |  4. BOOST APPLICATION                     |
             |     Add weighted signal, relationship,    |
             |     social proof, recency, cohort boosts  |
             |     Cost: ~50ns per candidate per boost   |
             +------------------------------------------+
                                    |
                                    v
             +------------------------------------------+
             |  5. PENALTY APPLICATION                   |
             |     Subtract weighted negative signal     |
             |     penalties (skip, dislike, downvote)   |
             |     Cost: ~30ns per candidate per penalty |
             +------------------------------------------+
                                    |
                                    v
             +------------------------------------------+
             |  6. GATE EVALUATION                       |
             |     Remove candidates below quality       |
             |     thresholds (completion, engagement    |
             |     ratio). Exploration items bypass.     |
             |     Cost: O(1) per gate per candidate     |
             +------------------------------------------+
                                    |
                                    v
             +------------------------------------------+
             |  7. SCORE NORMALIZATION                   |
             |     Normalize composite scores to         |
             |     [0.0, 1.0] range using min-max        |
             |     within the surviving candidate set    |
             |     Cost: O(n) for min/max scan, O(n)    |
             |     for normalization                     |
             +------------------------------------------+
                                    |
                                    v
             +------------------------------------------+
             |  8. DIVERSITY ENFORCEMENT                 |
             |     Greedy MMR reranking to enforce:      |
             |     max_per_creator, format_mix,          |
             |     topic_diversity                       |
             |     Cost: O(n * LIMIT) in the worst case |
             +------------------------------------------+
                                    |
                                    v
             +------------------------------------------+
             |  9. EXPLORATION INJECTION                 |
             |     Replace exploration_budget % of       |
             |     results with exploration candidates   |
             |     (new items, cold-start, hidden gems)  |
             |     Cost: O(LIMIT)                        |
             +------------------------------------------+
                                    |
                                    v
             +------------------------------------------+
             |  10. PAGINATION                           |
             |     Slice to requested page via cursor    |
             |     or offset. Assemble response with    |
             |     signal snapshots.                     |
             |     Cost: O(LIMIT)                        |
             +------------------------------------------+
                                    |
                                    v
                        Final ranked result set

Stage Details

Stage 1: Candidate Retrieval. Executes the profile's CandidateStrategy (Section 3). Produces a raw candidate set with initial retrieval scores. For ANN, the initial score is cosine similarity. For Scan, the initial score is 0.0. For Hybrid, the initial score is the fused text + vector score.

Stage 2: Hard Exclusion. Removes candidates that must never appear for this user, regardless of score. This stage evaluates the profile's excludes list:

pub enum Exclude {
    /// Items where this user has the named signal. e.g., Exclude::signal("hide")
    Signal(&str),
    /// Items by creators with this relationship to the user. e.g., Exclude::relationship("blocked")
    Relationship(&str),
}

Implementation: For Exclude::signal("hide"), check the user-to-item relationship for the hide flag (O(1) bitmap lookup). For Exclude::relationship("blocked"), resolve the user's blocked creator set (cached as a roaring bitmap) and filter. Additionally, any exclude_ids from the query are removed here.

Stage 3: Filter Evaluation. Applies user-specified query filters (metadata, date, engagement threshold, user state, geographic). All filters from the query's filters: Vec<Filter> are evaluated. Filters are AND-composed across dimensions; OR-composed within a dimension (e.g., category IN [jazz, blues]). Implementation uses pre-computed roaring bitmaps for keyword fields and range scans for numeric fields.

Stage 4: Boost Application. Adds weighted positive signals to each candidate's score (Section 5). Each boost reads one signal value or relationship weight per candidate and multiplies it by the boost weight. The result is added to the candidate's composite score.

Stage 5: Penalty Application. Subtracts weighted negative signals from each candidate's score (Section 6). Same mechanics as boosts but with negative contribution.

Stage 6: Gate Evaluation. Removes candidates below quality thresholds (Section 7). Gates are hard filters, not soft penalties. A candidate below the gate threshold is removed from the result set entirely. Exception: items flagged for exploration bypass gates (they have not accumulated enough signals for gate evaluation to be meaningful).

Stage 7: Score Normalization. Normalizes composite scores to the [0.0, 1.0] range using min-max normalization within the surviving candidate set (Section 8).

Stage 8: Diversity Enforcement. Reranks the scored candidates to enforce variety constraints (Section 9). This stage reorders results -- it does not remove them.

Stage 9: Exploration Injection. Replaces a configurable percentage of results with exploration candidates (Section 10). Exploration items are selected from the cold-start pool, the hidden-gems candidate set, or quality-weighted random sampling.

Stage 10: Pagination. Slices the final ranked set to the requested page using cursor-based pagination (Section 14). Assembles the response with signal snapshots for each result.

---

5. Boost Types

Boosts are the primary scoring mechanism. Each boost reads a signal value, a relationship weight, or a derived metric for a candidate and adds a weighted contribution to the candidate's composite score.

5.1 Signal Boost

Boosts a candidate's score based on a signal's value within a time window.

Boost::signal(
    signal_name: &str,      // "view", "like", "share", etc.
    window: Window,          // Window::hours(24), Window::days(7), etc.
    aggregation: SignalAgg,  // How to read the signal
    weight: f64,             // Contribution weight (typically 0.0 to 1.0)
)

SignalAgg variants:

Aggregation	Description	Example
Value	Raw aggregate value (count or weighted sum) in the window	view.value(24h) = 12,450 views in last 24h
Velocity	Rate of change within the window (events per hour)	view.velocity(24h) = 518.75 views/hour
Ratio	Signal value divided by view count (engagement ratio)	like.ratio(7d) = likes_7d / views_7d = 0.08
UniqueRatio	Unique users / total count (new-user reach)	view.unique_ratio(24h) = unique viewers / total views
DecayScore	Running exponential decay score from hot tier	view.decay_score() -- no window, uses running score
RelativeVelocity	Short-window velocity / long-window velocity	view.relative_velocity(1h, 24h) = acceleration

Score contribution:

candidate.score += normalize(signal_value) * weight

Where normalize maps the raw signal value to a [0, 1] range using the candidate set's percentile distribution (Section 8.3).

5.2 Relationship Boost

Boosts a candidate's score based on the querying user's relationship with the candidate's creator.

Boost::relationship(
    edge_kind: &str,  // "interaction_weight", "engagement_affinity"
    weight: f64,
)

Execution: For each candidate, look up the relationship edge from the querying user to the candidate's creator. The edge weight (0.0 to 1.0) is multiplied by the boost weight and added to the score.

candidate.score += user_creator_edge_weight * weight

If no relationship edge exists, the contribution is 0.0.

5.3 Social Proof Boost

Boosts candidates that the user's follows have engaged with.

Boost::social_proof(weight: f64)

Execution:

1. Load the querying user's follow set (cached as a roaring bitmap).
2. For each candidate, count how many users in the follow set have a positive engagement signal (view, like, share) with this item in the last 24 hours.
3. Compute social proof score: social_count / follow_count (fraction of follows who engaged).
4. Contribution: social_proof_score * weight.

Performance: Social proof requires a per-candidate set intersection. This is the most expensive boost type. For 200 candidates with a follow set of 500, the cost is ~200 * ~50 ns = ~10 us (roaring bitmap intersection). Acceptable within the scoring budget.

Optimization: For large follow sets (>1000), pre-compute a "follow-engaged items in last 24h" bitmap during the background materializer cycle. The social proof check becomes a single bitmap test per candidate: O(1).

5.4 Recency Boost (Content Age Decay)

Applies time-based decay to candidate scores based on content age.

Boost::recency(
    field: &str,        // "created_at" typically
    half_life: Duration, // how fast content ages out
)

Equivalently specified via ProfileDecay:

pub struct ProfileDecay {
    pub field: &str,
    pub half_life: Duration,
}

Formula:

recency_score = exp(-ln(2) / half_life_secs * (now - created_at).as_secs())

This produces a score in (0.0, 1.0] where items at age 0 score 1.0 and items at age half_life score 0.5.

Application: The recency score is multiplied into the composite score as a scaling factor, not added:

candidate.score *= recency_score

This ensures that old content's score decays proportionally, rather than being offset by a fixed amount.

Half-Life	Interpretation
12 hours	Aggressive decay. News, real-time surfaces. Score halves every 12 hours.
48 hours	Standard feed decay. For You surfaces.
7 days	Moderate decay. Browse and category pages.
30 days	Slow decay. Search results, evergreen content.
90 days	Very slow decay. Search for tutorials, documentation.

5.5 Cohort Signal Boost

Boosts a candidate based on signal velocity within a specific cohort.

Boost::cohort_signal(
    signal_name: &str,   // "view", "share", etc.
    cohort: CohortSource, // Named, Auto, or Predicate
    window: Window,
    aggregation: SignalAgg,
    weight: f64,
)

Execution: Reads the cohort-scoped signal aggregate for the candidate (see Cohorts spec, Section 8). The cohort source determines how the aggregation scope is resolved:

• CohortSource::Auto -- derives the cohort from the querying user's attributes (region, age_range, top inferred interest).
• CohortSource::Named(name) -- uses a pre-defined named cohort.
• CohortSource::Predicate(pred) -- evaluates an ad-hoc cohort predicate.

Score contribution:

candidate.score += normalize(cohort_signal_value) * weight

Fallback: If cohort signal data is sparse (fewer than 50 events in the window for this item in this cohort), fall back to the global signal value with a 0.5x dampening factor:

if cohort_signal_count < 50 {
    effective_value = global_signal_value * 0.5
}

5.6 Cohort-Relative Boost

Boosts items that are disproportionately popular within a cohort compared to the general population.

Boost::cohort_relative(
    cohort: CohortSource,
    window: Window,
    weight: f64,
)

Formula:

cohort_relative_score = cohort_velocity / max(global_velocity, floor)

Where floor prevents division by near-zero (default: 10.0 events/hour, configurable via CohortConfig::relative_score_floor).

Interpretation: A score of 5.0 means the item is 5x more popular within this cohort than globally. This surfaces content with specific cohort resonance.

5.7 Preference Match Boost

Boosts candidates whose embedding is similar to the querying user's preference vector.

Boost::preference_match(weight: f64)

Formula:

preference_score = cosine_sim(user.preference_vector, candidate.embedding)

This is distinct from ANN candidate generation. ANN retrieves candidates by similarity; preference match re-scores candidates that may have been retrieved by a different strategy (e.g., CohortTrending candidates re-ranked by preference match).

---

6. Penalty Types

Penalties subtract from a candidate's score based on negative signals. They mirror the boost mechanics but with negative contribution.

pub struct Penalty {
    pub signal: &str,
    pub window: Window,
    pub weight: f64,  // Stored as positive; subtracted during scoring.
}

// Construction:
Penalty::signal(
    signal_name: &str,  // "skip", "dislike", "downvote"
    window: Window,
    weight: f64,        // Positive value. Applied as -weight.
)

Score contribution:

candidate.score -= normalize(signal_value) * weight

Per-user vs. per-item penalties:

Penalty Scope	Description	Example
Per-item (global)	The signal count on the item itself from all users	skip.value(24h) = 500 skips in 24h (item is low quality)
Per-user-item	The signal from this specific user on this item	User skipped this item 3 seconds ago (personal negative)

When a penalty signal name matches a user-to-item relationship signal (e.g., the user has a skip signal on this item), the per-user signal takes precedence and is applied as a stronger penalty multiplier:

if user_has_signal(user, item, signal_name) {
    candidate.score -= user_signal_weight * weight * USER_PENALTY_MULTIPLIER
    // USER_PENALTY_MULTIPLIER = 3.0 (per-user skip is 3x stronger than global skip rate)
}

---

7. Quality Gates

Gates are hard thresholds that exclude candidates from the result set. Unlike penalties (which reduce scores), gates produce binary accept/reject decisions.

7.1 Minimum Signal Gate

Gate::min(
    signal_name: &str,   // "completion"
    window: Window,       // Window::all_time()
    threshold: f64,       // 0.3
)

Items where signal.value(window) < threshold are excluded. The signal value is the weighted aggregate, not the raw count.

Example: Gate::min("completion", Window::all_time(), 0.3) excludes items with an average completion rate below 30%. This filters out content that most people abandon.

7.2 Ratio Gate

Gate::min_ratio(
    ratio_name: &str,    // "engagement_ratio"
    threshold: f64,      // 0.03
)

Built-in ratios:

Ratio Name	Formula	Description
engagement_ratio	(likes + comments + shares) / views	Overall engagement quality
like_ratio	likes / views	Positive sentiment rate
completion_rate	weighted_sum(completion) / count(view)	Content quality
skip_ratio	skips / impressions	Negative quality indicator

7.3 Minimum Count Gate

Gate::min_count(
    signal_name: &str,   // "view"
    window: Window,       // Window::all_time()
    count: u64,           // 100
)

Items with fewer than count events are excluded. Used to ensure statistical significance before applying ratio-based quality gates.

7.4 Gate Bypass for Exploration

Items in the exploration pool (Section 10) bypass all gates. These are new items that have not accumulated enough signals for gate evaluation to be meaningful. Without this bypass, cold-start items would be permanently excluded by quality gates that require historical engagement data.

Bypass mechanism: During Stage 6 (Gate Evaluation), candidates flagged with is_exploration_candidate: true skip all gate checks. The exploration flag is set during Stage 9 (Exploration Injection) for items selected from the cold-start pool.

Implementation detail: Gates are evaluated before exploration injection in the pipeline order. To enable bypass, the pipeline performs a two-pass approach:

1. First pass: evaluate gates on all non-exploration candidates.
2. Reserve exploration_budget * LIMIT slots for exploration candidates (gate-exempt).
3. Final pass: inject exploration candidates into reserved slots.

---

8. Score Composition and Normalization

8.1 Composite Score Formula

The composite score for a candidate is computed as:

raw_score = initial_retrieval_score
          + SUM(boost_i.normalize(signal_i) * boost_i.weight)
          - SUM(penalty_j.normalize(signal_j) * penalty_j.weight)

final_score = raw_score * recency_decay_factor

Where:

• initial_retrieval_score comes from the candidate generation strategy (cosine similarity for ANN, RRF score for Hybrid, 0.0 for Scan).
• Each boost and penalty contribution is independently normalized before weighting.
• Recency decay is applied multiplicatively (it scales the entire score, not offsets it).

8.2 Score Normalization: Min-Max Within Candidate Set

After all boosts and penalties are applied, the composite scores are normalized to the [0.0, 1.0] range using min-max normalization within the surviving candidate set:

normalized_score = (raw_score - min_score) / (max_score - min_score)

If max_score == min_score (all candidates scored equally), all normalized scores are set to 0.5.

Why min-max, not z-score: Min-max normalization is deterministic and produces scores in a bounded range, which is required for the score field in the response. Z-score normalization can produce unbounded negative values, which violates the non-negative score invariant. Min-max is also simpler to reason about when combining scores from different profiles.

8.3 Signal Value Normalization

Raw signal values (e.g., 12,450 views) must be normalized before weighting. Without normalization, a signal with large absolute values (views) would dominate a signal with small absolute values (share ratio).

Normalization strategy: percentile rank within the candidate set.

For each signal used in a boost or penalty, compute the percentile rank of each candidate's signal value within the candidate set:

percentile_rank(candidate, signal) = rank_of(candidate.signal_value) / candidate_count

This produces values in [0.0, 1.0] regardless of the signal's absolute scale. A candidate at the 90th percentile of views within the candidate set receives a normalized value of 0.9.

Why percentile, not min-max on raw values: Min-max normalization on raw signal values is sensitive to outliers. A single viral item with 10M views would compress all other items to near-zero. Percentile ranking is robust to outliers and ensures that boost weights behave consistently regardless of the signal's absolute scale.

Pre-computed percentile tables: For the most common signals (view, like, share, completion), the background materializer maintains approximate percentile tables (1000-bucket histograms) updated hourly. Query-time percentile lookup is O(1) via binary search on the histogram.

8.4 Cross-Signal Comparability

The percentile normalization strategy ensures that a 0.3 weight on view.velocity(24h) and a 0.2 weight on like.ratio(7d) produce comparable contributions regardless of the absolute scales of these signals. The weight directly controls the relative importance of each signal in the final score.

Guideline for weight selection:

Total Weight	Interpretation
Sum of all boost weights = 1.0	Each weight is the fraction of the score controlled by that signal
Any single weight > 0.5	That signal dominates the ranking
All weights equal	Uniform blend of signals

Weights are not required to sum to 1.0. The normalization step (Stage 7) rescales the composite score to [0, 1] regardless.

---

9. Diversity Enforcement

9.1 DiversitySpec

pub struct DiversitySpec {
    /// Maximum number of items from the same creator in the result set.
    /// None = no creator constraint.
    pub max_per_creator: Option<u32>,

    /// Ensure variety of content formats (video, short, article, etc.)
    /// across the result set.
    pub format_mix: bool,

    /// Topic diversity score from 0.0 (no enforcement) to 1.0 (maximize).
    /// Uses embedding-space spread via Maximal Marginal Relevance.
    pub topic_diversity: Option<f64>,

    /// Minimum representation per category. Ensures at least N items
    /// from each represented category appear in the result set.
    /// Only meaningful when results span multiple categories.
    pub category_min: Option<u32>,
}

9.2 Algorithm: Greedy MMR Reranking

Diversity enforcement uses a greedy algorithm inspired by Maximal Marginal Relevance (Carbonell & Goldstein, SIGIR 1998). The algorithm iteratively selects the next item that maximizes a combination of relevance score and diversity contribution.

Pseudocode:

function diversity_rerank(scored_candidates, diversity_spec, limit):
    selected = []
    remaining = scored_candidates.sorted_by_score_desc()
    creator_counts = {}
    format_counts = {}

    while |selected| < limit AND |remaining| > 0:
        best_candidate = None
        best_mmr_score = -inf

        for candidate in remaining:
            // Check hard diversity constraints
            if max_per_creator is set:
                if creator_counts[candidate.creator] >= max_per_creator:
                    continue  // skip: creator already at limit

            // Compute MMR score
            relevance = candidate.normalized_score
            diversity = 0.0

            if topic_diversity is set:
                // Embedding-space spread: minimum distance to any selected item
                if |selected| > 0:
                    min_distance = min(embedding_distance(candidate, s) for s in selected)
                    diversity = min_distance  // higher = more diverse
                else:
                    diversity = 1.0  // first item has maximum diversity

            // Format mix bonus
            format_bonus = 0.0
            if format_mix:
                if format_counts[candidate.format] == 0:
                    format_bonus = 0.1  // bonus for introducing a new format

            // Category minimum bonus
            category_bonus = 0.0
            if category_min is set:
                if category_counts[candidate.category] < category_min:
                    category_bonus = 0.1  // bonus for underrepresented category

            lambda = topic_diversity.unwrap_or(0.0)
            mmr_score = (1.0 - lambda) * relevance
                      + lambda * diversity
                      + format_bonus
                      + category_bonus

            if mmr_score > best_mmr_score:
                best_mmr_score = mmr_score
                best_candidate = candidate

        if best_candidate is None:
            // All remaining candidates violate hard constraints.
            // Relax max_per_creator by 1 and retry.
            max_per_creator += 1
            continue

        selected.push(best_candidate)
        remaining.remove(best_candidate)
        creator_counts[best_candidate.creator] += 1
        format_counts[best_candidate.format] += 1
        category_counts[best_candidate.category] += 1

    return selected

9.3 Diversity Constraint Details

max_per_creator: No more than N items from the same creator in the result page. This is the most common diversity constraint. When a creator has more than N items in the candidate set, only the top-N by score are eligible for selection; the rest are deferred to subsequent pages.

format_mix: When enabled, the algorithm introduces a bonus for selecting items of formats not yet represented in the result set. This ensures a feed of all-video does not dominate when articles, shorts, and podcasts are also available. The bonus is small (0.1) -- it does not override relevance, only breaks ties.

topic_diversity: Controls embedding-space spread of results. At 0.0, no topic diversity is enforced (pure relevance). At 1.0, maximum diversity is enforced (the algorithm strongly prefers items far from already-selected items in embedding space). Values of 0.3-0.7 are typical for feed surfaces.

category_min: Ensures that if results span multiple categories, each category gets at least N items. This prevents a dominant category from monopolizing the result set.

9.4 Diversity and Pagination

Diversity constraints apply per page, not globally across all pages. Each page independently satisfies the diversity spec. This means:

• Page 1 may have 2 items from creator X.
• Page 2 may also have 2 items from creator X (different items).
• The user never sees more than max_per_creator items from any creator in a single rendered batch.

Rationale: Global diversity across pages would require the database to maintain state across paginated queries, which conflicts with stateless cursor-based pagination. Per-page diversity is simpler, stateless, and matches user expectations (they process one page at a time).

9.5 Diversity as Reordering, Not Filtering

Diversity enforcement never reduces the result count. If max_per_creator: 2 and a creator has 10 items in the top 50, 2 items appear in positions the algorithm selects, and the remaining 8 are pushed to lower positions or subsequent pages. No items are removed from the result set.

Relaxation under pressure: If hard diversity constraints make it impossible to fill the requested result count (e.g., only 3 creators exist in the candidate set with max_per_creator: 1 and LIMIT 50), the algorithm relaxes max_per_creator incrementally until the result count is met.

---

10. Exploration Budget

The exploration budget injects items from outside the scoring pipeline's natural ranking into a percentage of results. This serves two purposes: cold-start item discovery and serendipitous discovery.

10.1 Configuration

pub exploration: f64,  // Fraction of results reserved for exploration.
                       // Range: 0.0 to 0.5. Default: 0.0.

An exploration budget of 0.10 means 10% of results (e.g., 5 out of 50) are exploration items.

10.2 Exploration Candidate Selection

Exploration items are selected from three pools, in priority order:

Pool 1: Cold-start items. Items created within the cold-start window (configurable, default 7 days) that have fewer than the cold-start signal threshold (configurable, default 100 views). These items have not had enough exposure for the scoring pipeline to evaluate them fairly.

Selection within Pool 1: Quality-weighted random sampling. The quality weight is derived from the creator's historical performance (average completion rate of their catalog). Items from creators with high historical quality are more likely to be selected.

cold_start_weight(item) = creator_avg_completion_rate(item.creator) * recency_factor(item)

Pool 2: Cohort trending. Items trending within the querying user's auto-detected cohort that are not present in the main result set. These are items the user's demographic peers are engaging with but that the user's personal preference vector has not surfaced.

Pool 3: Hidden gems. Items with high quality signals (completion rate, like ratio) but low total reach (view count). These are items the algorithm has not surfaced widely but that perform well with their limited audience.

10.3 Exploration Injection

Exploration items are injected after diversity enforcement. They replace items at specific positions within the result set:

Injection positions for exploration_budget = 0.10, LIMIT = 50:
    5 exploration items at positions: [4, 12, 23, 35, 45]
    (distributed throughout the result set, not clustered)

Position distribution: Exploration items are placed at evenly-spaced intervals through the result set. They are never placed at positions 0-2 (the top results must be the highest-confidence recommendations) and never at the last position.

10.4 Exploration Decay

As a user engages more with the platform (accumulates more signals), the effective exploration percentage can decrease:

effective_exploration = base_exploration * exploration_decay_factor(user)

exploration_decay_factor(user) = max(0.3, 1.0 - log10(user_signal_count + 1) / 5.0)

User Signal Count	Decay Factor	Effective Exploration (base 10%)
0 (new user)	1.0	10.0%
10	0.8	8.0%
100	0.6	6.0%
1,000	0.4	4.0%
10,000+	0.3 (floor)	3.0%

The floor of 30% of the base rate ensures that even heavily-engaged users continue to see some exploration content. This prevents the "filter bubble" effect.

10.5 Cold-Start User Handling

A new user with no signal history has no preference vector, no relationship graph, and no engagement history. The scoring pipeline has no personalization data to work with. The exploration budget is critical here:

• New users receive a boosted exploration budget: min(0.50, exploration * 3.0) (capped at 50% of results).
• Cold-start items in the exploration pool are selected using population-level priors: items with the highest global quality signals weighted by the user's declared metadata (region, language).
• As the user accumulates signals, the boosted exploration rate decays toward the base rate per Section 10.4.

---

11. Built-In Sort Modes

Sort modes are formula-based ranking functions that bypass the boost/penalty scoring pipeline. When a query specifies a sort mode (either at the query level or within the profile), the sort formula replaces stages 4-5 (boost and penalty application) of the scoring pipeline. Stages 2-3 (exclusion, filter), 6 (gates), 7 (normalization), 8 (diversity), 9 (exploration), and 10 (pagination) still apply.

11.1 Hot

hot_score(item) = log10(max(|positive - negative|, 1))
                  / (age_hours + 2) ^ gravity

Where:
    positive = upvotes + likes
    negative = downvotes + dislikes
    age_hours = (now - created_at).as_hours()
    gravity = configurable, default 1.8

Behavior: Hot rewards early engagement but punishes age. An hour-old post with 500 upvotes scores higher than a day-old post with 2,000 upvotes. The gravity parameter controls how aggressively age suppresses score. Higher gravity = faster decay.

Gravity	Behavior
1.0	Very slow decay. Content stays hot for days.
1.5	Moderate decay. Content refreshes every ~6 hours.
1.8	Standard (Reddit default). Content refreshes every ~3 hours.
2.5	Aggressive decay. Content refreshes hourly.

Use cases: UC-06 (Browse/Category), UC-14 (Hot Surfaces), any community frontpage.

11.2 Trending

trending_score(item) = share_velocity(6h) * 0.5
                     + view_velocity(6h) * 0.3
                     + new_user_reach(24h) * 0.2

Where:
    share_velocity(w) = share.count(w) / w.as_hours()
    view_velocity(w) = view.count(w) / w.as_hours()
    new_user_reach(w) = unique_view.count(w) / view.count(w)
                        // fraction of viewers new to this creator

Behavior: Pure velocity, no personalization, no total-count signals. A video with 500 total views but 400 in the last hour outranks a video with 10M total views and 200 in the last hour.

Gate: engagement_ratio >= 0.03 to filter clickbait (high views, zero engagement).

Use cases: UC-03 (Trending), global/category/social-scoped trending.

11.3 Rising

rising_score(item) = relative_velocity(item) * age_boost(item)

Where:
    relative_velocity(item) = view.velocity(1h) / max(creator_baseline_velocity, floor)
    creator_baseline_velocity = creator.avg_view_velocity(7d)
    floor = 1.0  // prevents division by zero for new creators
    age_boost(item) = max(0.1, 1.0 - age_hours / 48.0)
                      // linear boost for items under 48 hours old

Behavior: Surfaces content overperforming relative to its creator's historical baseline. A small creator getting 10x their normal engagement is "rising" even if their absolute numbers are modest.

Use cases: UC-03 (Rising), UC-13 (Hidden Gems variant), breakout detection.

11.4 Controversial

controversial_score(item) = (positive * negative) / (positive + negative) ^ 2

Where:
    positive = likes + upvotes + shares
    negative = dislikes + downvotes + reports

Behavior: Maximizes the product of positive and negative engagement. A post with 1,000 upvotes and 1,000 downvotes (controversial score = 0.25) scores higher than a post with 1,800 upvotes and 200 downvotes (controversial score = 0.09).

Gate: (positive + negative) >= 100 to filter items without enough total engagement to be genuinely controversial (not just unpopular).

Use cases: UC-14 (Controversial), debate surfaces, "spicy" content sections.

11.5 Hidden Gems

hidden_gems_score(item) = quality_score(item) * inverse_reach(item)

Where:
    quality_score(item) = completion_rate(all_time) * 0.6
                        + like_ratio(all_time) * 0.4
    inverse_reach(item) = 1.0 / log10(view.count(all_time) + 10)

Behavior: Surfaces high-quality content with low total reach. The logarithmic inverse ensures diminishing penalty as reach grows -- an item with 100 views is penalized similarly to one with 1,000 views, but both score much higher than one with 1M views.

Gate: completion_rate(all_time) >= 0.5 (quality floor).

Filter: created_within(30d) typically applied (only recent hidden gems, not decade-old obscure content).

Use cases: UC-13 (Hidden Gems), "You Might Have Missed," editorial discovery.

11.6 Shuffle

shuffle_score(item) = random(seed) * quality_weight(item)

Where:
    quality_weight(item) = sqrt(quality_score(item))
    quality_score(item) = completion_rate * 0.5 + like_ratio * 0.3 + log10(views + 1) * 0.2
    seed = hash(user_id, timestamp_minute)  // same results for same user within 1 minute

Behavior: Quality-weighted random sampling. High-quality items are more likely to appear but not guaranteed. The seed ensures deterministic results within short time windows to prevent jarring re-shuffles on page refresh.

Use cases: Music playlists, "surprise me" buttons, mood-based discovery.

11.7 Top (Windowed)

top_score(item, window) = weighted_signal_sum(item, window)

Where:
    weighted_signal_sum(item, w) = view.count(w) * 0.3
                                 + like.count(w) * 0.3
                                 + share.count(w) * 0.2
                                 + comment.count(w) * 0.1
                                 + completion_rate(w) * view.count(w) * 0.1

Window variants:

Sort Mode	Window	Description
Sort::TopHour	1 hour	Real-time quality
Sort::TopToday	24 hours	Daily best
Sort::TopWeek	7 days	Weekly digest
Sort::TopMonth	30 days	Monthly recap
Sort::TopYear	365 days	Annual best
Sort::TopAllTime	All time	Classic / best-of

Use cases: UC-06 (Browse with sort mode), community "best of" surfaces.

11.8 Simple Field Sorts

These sort modes are direct field-value sorts without formula computation.

Sort Mode	Implementation	Notes
Sort::New	created_at DESC	Pure chronological, no scoring
Sort::Old	created_at ASC	Archives, sequential viewing
Sort::MostViewed	view.count(all_time) DESC	Raw popularity
Sort::MostLiked	like.count(all_time) DESC	Positive sentiment
Sort::MostCommented	comment.count(all_time) DESC	Discussion
Sort::MostShared	share.count(all_time) DESC	Virality
Sort::Shortest	duration ASC	Quick content
Sort::Longest	duration DESC	Deep dives
Sort::AlphabeticalAsc	title ASC	Structured catalogs
Sort::AlphabeticalDesc	title DESC	Reverse alphabetical
Sort::LiveViewerCount	live_viewer_count DESC	Live surfaces
Sort::DateSaved	user.saved_at(item) DESC	Personal library
Sort::CreatorEngagementRate	creator.engagement_rate DESC	Creator discovery
Sort::Relevance	Text + semantic match score	Search only
Sort::Personalized	User preference match score	For You surfaces

11.9 Sort Mode and Profile Interaction

When a query specifies both a profile and a sort override, the sort override replaces the profile's scoring logic:

Query	Scoring Behavior
USING PROFILE for_you	Full boost/penalty pipeline from for_you profile
USING PROFILE browse SORT hot	Hot formula replaces boosts/penalties; browse filters, diversity, gates still apply
SORT new (no profile)	Pure created_at DESC; no boosts, no penalties, no gates

The sort mode controls what determines the ordering. The profile (if specified) still controls candidate generation, filtering, gates, diversity, and exploration.

---

12. Cohort-Aware Ranking

12.1 Cohort Integration Points

Cohorts integrate with the ranking pipeline at three levels:

Candidate generation. The CohortTrending strategy (Section 3.5) generates candidates from items trending within a specific cohort.

Boost signals. Boost::cohort_signal and Boost::cohort_relative (Sections 5.5, 5.6) add cohort-scoped signal values as scoring components.

Profile scoping. The same profile can operate on different signal scopes without modification. A trending profile uses global velocity by default. When the query includes FOR COHORT young_us_jazz, the same profile reads cohort-scoped velocity instead.

12.2 Same Profile, Different Signal Scope

The key design decision: ranking profiles do not change when cohort scoping is applied. The profile defines which signals matter and how to weight them. The cohort defines whose signals are counted.

// Global trending
RETRIEVE items USING PROFILE trending LIMIT 25
    --> reads view.velocity(24h) from global counters

// Cohort trending
RETRIEVE items USING PROFILE trending FOR COHORT young_us_jazz LIMIT 25
    --> reads view.velocity(24h) from young_us_jazz cohort counters
    --> same profile weights, same gates, same diversity
    --> different signal data

Implementation: When a FOR COHORT clause is present, the signal read path in Stage 4 (Boost Application) routes signal reads to the cohort-scoped counters instead of global counters. This routing is transparent to the profile definition.

12.3 Cohort Fallback Behavior

When cohort signal data is sparse, the system falls back gracefully:

Scenario	Behavior
Item has cohort tracking active and sufficient data	Use exact cohort signal values
Item has cohort tracking active but sparse data (<50 events)	Blend: 0.7 * cohort_value + 0.3 * global_value
Item does not have cohort tracking active	Use global signal values with a dampening note in response
Cohort population below minimum threshold	Fall back to nearest parent cohort (see Cohorts spec, Section 9.4)

The blending formula for sparse data prevents noisy cohort signals from dominating when the sample size is small. As cohort data accumulates, the blend converges to pure cohort values.

12.4 Auto-Cohort Detection

When CohortSource::Auto is specified, the system derives a cohort predicate from the querying user's attributes:

auto_cohort(user) = Predicate::and(
    Predicate::eq("region", user.region),
    Predicate::eq("age_range", user.age_range),
    Predicate::contains("inferred_interests", user.top_interest),
)

If the auto-detected cohort has fewer active users than min_trending_population, the system progressively drops predicate terms (starting with inferred_interests, then age_range) until the population threshold is met.

---

13. Profile Presets

tidalDB ships with built-in profile presets for every standard surface. Applications can use these directly or override any aspect.

13.1 for_you

ProfileDef {
    name: "for_you",
    version: 1,
    candidate: Candidate::Ann {
        query_vector: VectorSource::UserPreference,
        index: EntityKind::Item,
        top_k: 500,
    },
    boosts: vec![
        Boost::signal("view", Window::hours(24), Velocity, 0.3),
        Boost::relationship("interaction_weight", 0.2),
        Boost::social_proof(0.15),
    ],
    decay: Some(ProfileDecay {
        field: "created_at",
        half_life: Duration::hours(48),
    }),
    gates: vec![
        Gate::min("completion", Window::all_time(), 0.3),
    ],
    penalties: vec![
        Penalty::signal("skip", Window::hours(24), 0.5),
    ],
    excludes: vec![
        Exclude::signal("hide"),
        Exclude::relationship("blocked"),
    ],
    diversity: Some(DiversitySpec {
        max_per_creator: Some(2),
        format_mix: true,
        topic_diversity: None,
        category_min: None,
    }),
    exploration: 0.10,
    sort: None,
    extends: None,
}

Surfaces: UC-01 (For You Feed).

13.2 trending

ProfileDef {
    name: "trending",
    version: 1,
    candidate: Candidate::Scan { entity: EntityKind::Item },
    boosts: vec![
        Boost::signal("share", Window::hours(6), Velocity, 0.5),
        Boost::signal("view", Window::hours(6), Velocity, 0.3),
        Boost::signal("view", Window::hours(24), UniqueRatio, 0.2),
    ],
    gates: vec![
        Gate::min_ratio("engagement_ratio", 0.03),
    ],
    penalties: vec![],
    excludes: vec![],
    diversity: Some(DiversitySpec {
        max_per_creator: Some(1),
        format_mix: false,
        topic_diversity: None,
        category_min: None,
    }),
    exploration: 0.0,
    decay: None,
    sort: None,
    extends: None,
}

Surfaces: UC-03 (Trending). Same profile for global, category-scoped, social-scoped, and cohort-scoped trending. The scope is determined by query filters and the FOR COHORT clause, not the profile.

13.3 search

ProfileDef {
    name: "search",
    version: 1,
    candidate: Candidate::Hybrid {
        text_weight: 0.6,
        vector_weight: 0.4,
        fusion: Fusion::Rrf { k: 60 },
    },
    boosts: vec![
        Boost::signal("completion", Window::all_time(), Value, 0.15),
        Boost::signal("like", Window::all_time(), Ratio, 0.10),
    ],
    decay: Some(ProfileDecay {
        field: "created_at",
        half_life: Duration::days(90),
    }),
    gates: vec![],
    penalties: vec![],
    excludes: vec![
        Exclude::signal("hide"),
        Exclude::relationship("blocked"),
    ],
    diversity: Some(DiversitySpec {
        max_per_creator: Some(2),
        format_mix: false,
        topic_diversity: None,
        category_min: None,
    }),
    exploration: 0.0,
    sort: None,
    extends: None,
}

Surfaces: UC-02 (Search). Text relevance is the floor. Personalization (via user preference match from ANN component) reorders within the relevant set.

13.4 following

ProfileDef {
    name: "following",
    version: 1,
    candidate: Candidate::Relationship { edge: "follows" },
    boosts: vec![],
    decay: None,
    gates: vec![],
    penalties: vec![],
    excludes: vec![],
    diversity: None,
    exploration: 0.0,
    sort: Some(Sort::New),
    extends: None,
}

Surfaces: UC-04 (Following Feed). Pure reverse chronological from followed creators. Minimal algorithmic intervention.

13.5 related

ProfileDef {
    name: "related",
    version: 1,
    candidate: Candidate::Ann {
        query_vector: VectorSource::ItemEmbedding("$anchor_item"),
        index: EntityKind::Item,
        top_k: 200,
    },
    boosts: vec![
        Boost::preference_match(0.3),
        Boost::signal("completion", Window::all_time(), Value, 0.2),
    ],
    decay: Some(ProfileDecay {
        field: "created_at",
        half_life: Duration::days(14),
    }),
    gates: vec![
        Gate::min("completion", Window::all_time(), 0.4),
    ],
    penalties: vec![
        Penalty::signal("skip", Window::hours(24), 0.3),
    ],
    excludes: vec![
        Exclude::signal("hide"),
        Exclude::relationship("blocked"),
    ],
    diversity: Some(DiversitySpec {
        max_per_creator: Some(1),
        format_mix: false,
        topic_diversity: Some(0.3),
        category_min: None,
    }),
    exploration: 0.05,
    sort: None,
    extends: None,
}

Surfaces: UC-05 (Related/Up Next). Semantic similarity as primary retrieval, personalization as secondary reranking.

13.6 browse

ProfileDef {
    name: "browse",
    version: 1,
    candidate: Candidate::Scan { entity: EntityKind::Item },
    boosts: vec![
        Boost::signal("completion", Window::all_time(), Value, 0.5),
        Boost::signal("like", Window::all_time(), Ratio, 0.3),
        Boost::signal("view", Window::all_time(), Value, 0.2),
    ],
    decay: Some(ProfileDecay {
        field: "created_at",
        half_life: Duration::days(30),
    }),
    gates: vec![],
    penalties: vec![],
    excludes: vec![],
    diversity: Some(DiversitySpec {
        max_per_creator: Some(2),
        format_mix: false,
        topic_diversity: None,
        category_min: None,
    }),
    exploration: 0.05,
    sort: None,
    extends: None,
}

Surfaces: UC-06 (Browse/Category). Quality-dominant with moderate recency bias. Sort mode typically overridden at query time (SORT hot, SORT new, SORT top_week).

13.7 hidden_gems

ProfileDef {
    name: "hidden_gems",
    version: 1,
    candidate: Candidate::Scan { entity: EntityKind::Item },
    boosts: vec![],
    gates: vec![
        Gate::min("completion", Window::all_time(), 0.5),
        Gate::min_count("view", Window::all_time(), 50),
    ],
    penalties: vec![],
    excludes: vec![],
    diversity: Some(DiversitySpec {
        max_per_creator: Some(1),
        format_mix: true,
        topic_diversity: Some(0.5),
        category_min: None,
    }),
    exploration: 0.0,
    sort: Some(Sort::HiddenGems),
    extends: None,
    decay: None,
}

Surfaces: UC-13 (Hidden Gems). High quality, low reach. Sort formula from Section 11.5.

13.8 notification

ProfileDef {
    name: "notification",
    version: 1,
    candidate: Candidate::Relationship { edge: "follows" },
    boosts: vec![
        Boost::relationship("interaction_weight", 0.5),
        Boost::signal("view", Window::hours(24), Velocity, 0.3),
    ],
    decay: Some(ProfileDecay {
        field: "created_at",
        half_life: Duration::hours(12),
    }),
    gates: vec![],
    penalties: vec![
        Penalty::signal("notification_dismiss", Window::days(7), 0.3),
    ],
    excludes: vec![
        Exclude::relationship("muted"),
        Exclude::relationship("blocked"),
    ],
    diversity: Some(DiversitySpec {
        max_per_creator: Some(1),
        format_mix: false,
        topic_diversity: None,
        category_min: None,
    }),
    exploration: 0.0,
    sort: None,
    extends: None,
}

Surfaces: UC-07 (Notifications). Relationship strength dominant, aggressive recency decay (12h half-life).

13.9 live

ProfileDef {
    name: "live",
    version: 1,
    candidate: Candidate::Scan { entity: EntityKind::Item },
    boosts: vec![
        Boost::relationship("interaction_weight", 0.4),
        Boost::signal("live_viewer_count", Window::hours(1), Value, 0.3),
        Boost::preference_match(0.3),
    ],
    gates: vec![],
    penalties: vec![],
    excludes: vec![
        Exclude::relationship("blocked"),
    ],
    diversity: Some(DiversitySpec {
        max_per_creator: Some(1),
        format_mix: false,
        topic_diversity: None,
        category_min: None,
    }),
    exploration: 0.0,
    decay: None,
    sort: None,
    extends: None,
}

Surfaces: UC-12 (Live Content). Requires Filter::eq("status", "live") at query time.

13.10 hot

ProfileDef {
    name: "hot",
    version: 1,
    candidate: Candidate::Scan { entity: EntityKind::Item },
    boosts: vec![],
    gates: vec![],
    penalties: vec![],
    excludes: vec![],
    diversity: Some(DiversitySpec {
        max_per_creator: Some(2),
        format_mix: false,
        topic_diversity: None,
        category_min: None,
    }),
    exploration: 0.0,
    sort: Some(Sort::Hot { gravity: 1.8 }),
    decay: None,
    extends: None,
}

Surfaces: UC-14 (Hot Surfaces), community frontpages.

13.11 rising

ProfileDef {
    name: "rising",
    version: 1,
    candidate: Candidate::Scan { entity: EntityKind::Item },
    boosts: vec![],
    gates: vec![
        Gate::min_count("view", Window::hours(1), 10),
    ],
    penalties: vec![],
    excludes: vec![],
    diversity: Some(DiversitySpec {
        max_per_creator: Some(1),
        format_mix: false,
        topic_diversity: None,
        category_min: None,
    }),
    exploration: 0.0,
    sort: Some(Sort::Rising),
    decay: None,
    extends: None,
}

Surfaces: UC-03 (Rising), breakout detection.

13.12 controversial

ProfileDef {
    name: "controversial",
    version: 1,
    candidate: Candidate::Scan { entity: EntityKind::Item },
    boosts: vec![],
    gates: vec![
        Gate::min_count("like", Window::all_time(), 50),
        Gate::min_count("dislike", Window::all_time(), 50),
    ],
    penalties: vec![],
    excludes: vec![],
    diversity: Some(DiversitySpec {
        max_per_creator: Some(2),
        format_mix: false,
        topic_diversity: None,
        category_min: None,
    }),
    exploration: 0.0,
    sort: Some(Sort::Controversial),
    decay: None,
    extends: None,
}

Surfaces: UC-14 (Controversial), debate surfaces.

13.13 Profile Preset Override

Applications can override any preset by defining a profile with the same name. The application's definition takes precedence. To restore a preset, drop the custom profile.

// Override the built-in trending profile with custom weights
db.define_profile(ProfileDef {
    name: "trending",
    version: 1,
    candidate: Candidate::Scan { entity: EntityKind::Item },
    boosts: vec![
        Boost::signal("share", Window::hours(3), Velocity, 0.6),  // shorter window, higher weight
        Boost::signal("view", Window::hours(3), Velocity, 0.2),
        Boost::signal("comment", Window::hours(6), Velocity, 0.2), // added comment velocity
    ],
    gates: vec![
        Gate::min_ratio("engagement_ratio", 0.05),  // stricter gate
    ],
    ..ProfileDef::default()
})?;

---

14. Pagination and Cursors

14.1 Cursor-Based Pagination

Pagination uses opaque cursor tokens for stable result sets across pages. The cursor encodes the scoring state needed to resume retrieval without re-scoring previous pages.

pub struct Cursor {
    /// The score of the last item on the previous page.
    /// Used as the upper bound for the next page's candidates.
    last_score: f64,

    /// The ID of the last item on the previous page.
    /// Used as a tiebreaker when scores are equal.
    last_id: EntityId,

    /// The profile version used for the previous page.
    /// Ensures consistent scoring across pages.
    profile_version: u32,

    /// Timestamp when the cursor was created.
    /// Used for staleness detection.
    created_at: u64,

    /// HMAC of the above fields to prevent tampering.
    signature: [u8; 16],
}

Staleness: Cursors older than 30 minutes are rejected with QueryError::StaleCursor. The application must re-query from page 1. This prevents long-lived cursors from producing inconsistent results as signals change.

14.2 Diversity Across Pages

Diversity constraints are applied per page. The cursor does not carry cross-page diversity state. Each page independently satisfies the diversity spec.

To prevent the same item from appearing on multiple pages, the cursor's last_score and last_id act as an exclusion boundary: candidates with score >= last_score (and ID < last_id at equal score) are excluded from subsequent pages.

14.3 exclude_ids Alternative

For applications that prefer explicit exclusion over cursor-based pagination:

let page2 = db.retrieve(Retrieve {
    profile: "for_you",
    for_user: Some("user_123"),
    exclude_ids: page1.results.iter().map(|r| r.id.clone()).collect(),
    limit: 50,
    ..Default::default()
})?;

This re-executes the full scoring pipeline minus the excluded items. More expensive than cursor-based pagination but guaranteed fresh results on each page.

---

15. Performance Targets

These targets define the latency and throughput bounds for the ranking and scoring system. Regressions against these numbers are treated as bugs.

15.1 End-to-End Query Latency

Query Type	LIMIT	Target (p50)	Target (p99)	Measurement Point
RETRIEVE with ANN profile (for_you)	50	< 30ms	< 50ms	db.retrieve() return
RETRIEVE with Scan profile (trending)	25	< 20ms	< 40ms	db.retrieve() return
RETRIEVE with Relationship (following)	50	< 15ms	< 30ms	db.retrieve() return
SEARCH with Hybrid profile	20	< 30ms	< 50ms	db.search() return
RETRIEVE with CohortTrending	25	< 30ms	< 50ms	db.retrieve() return
SEARCH WITHIN TRENDING FOR COHORT	20	< 35ms	< 50ms	db.search() return

15.2 Scoring Pipeline Stage Latency

Stage	Target (200 candidates)	Target (500 candidates)
Hard exclusion (bitmap)	< 50 us	< 100 us
Filter evaluation	< 100 us	< 200 us
Boost application (3 boosts)	< 30 us	< 75 us
Penalty application (1 penalty)	< 10 us	< 25 us
Gate evaluation (2 gates)	< 20 us	< 50 us
Score normalization (min-max)	< 5 us	< 10 us
Diversity enforcement (MMR)	< 200 us	< 500 us
Exploration injection	< 10 us	< 20 us
Total scoring pipeline	< 500 us	< 1.2 ms

15.3 Per-Candidate Scoring Cost

Operation	Target per Candidate
Decay score read (1 signal, 1 lambda)	~15 ns
Windowed count read (1h window)	~200 ns
Velocity computation	~500 ns
Relationship edge weight lookup	~50 ns
Social proof check (bitmap test)	~50 ns
Cosine similarity (1536D, normalized)	~500 ns
Total per candidate (typical for_you)	~1.5 us

15.4 Profile Definition Latency

Operation	Target
define_profile()	< 1ms
get_profile()	< 100 us
list_profiles()	< 500 us
Profile validation (at definition time)	< 5ms

---

16. Invariants and Correctness Guarantees

Scoring Invariants

INV-RANK-1: Deterministic scoring. Given the same candidate set, the same profile, and the same signal state, the scoring pipeline produces identical results. No randomness in scoring (shuffle mode uses a deterministic seed).

INV-RANK-2: Score non-negativity. After normalization, all scores are in the range [0.0, 1.0]. No candidate has a negative normalized score.

INV-RANK-3: Exclusion completeness. Items matching any Exclude predicate in the active profile never appear in results. Blocked creators' items never appear in any query for the blocking user. Hidden items never appear for the hiding user.

INV-RANK-4: Gate strictness. Non-exploration items below a gate threshold never appear in results. This is a hard invariant, not a soft preference.

INV-RANK-5: Diversity satisfaction. The diversity spec is satisfied in every result page, unless impossible due to insufficient candidate variety (in which case constraints are relaxed and the response includes a DiversityWarning).

INV-RANK-6: Exploration budget bounds. The number of exploration items in a result set is at most ceil(exploration * LIMIT). The exploration budget is never exceeded.

INV-RANK-7: Pagination consistency. Items returned on page N do not appear on page N+1 (given a valid cursor). No duplicate items across cursor-paginated pages.

Profile Invariants

INV-PROF-1: Version monotonicity. Profile versions are monotonically increasing. Defining a profile with a version <= the current latest version is rejected with SchemaError::VersionConflict.

INV-PROF-2: Inheritance acyclicity. Profile inheritance must form a DAG. Circular inheritance chains are rejected at definition time.

INV-PROF-3: Signal reference validity. Every signal name referenced in a profile boost, penalty, or gate must correspond to a defined signal type. Referencing an undefined signal returns SchemaError::UnknownSignal.

Property Tests

// P1: Exclusion completeness -- blocked/hidden items never appear.
proptest! {
    fn exclusion_completeness(
        items in arb_items(100),
        user in arb_user(),
        blocked_creators in arb_creator_ids(5),
        hidden_items in arb_item_ids(10),
    ) {
        let results = score_pipeline(items, user, profile_with_excludes());
        for result in &results {
            prop_assert!(!blocked_creators.contains(&result.creator_id),
                "blocked creator {} appeared in results", result.creator_id);
            prop_assert!(!hidden_items.contains(&result.id),
                "hidden item {} appeared in results", result.id);
        }
    }
}

// P2: Diversity constraints satisfied.
proptest! {
    fn diversity_constraints_hold(
        candidates in arb_scored_candidates(200),
        max_per_creator in 1u32..5,
        limit in 10u32..50,
    ) {
        let spec = DiversitySpec {
            max_per_creator: Some(max_per_creator),
            ..Default::default()
        };
        let results = diversity_rerank(&candidates, &spec, limit as usize);

        let mut creator_counts: HashMap<CreatorId, u32> = HashMap::new();
        for result in &results {
            *creator_counts.entry(result.creator_id).or_default() += 1;
        }
        for (creator, count) in &creator_counts {
            prop_assert!(*count <= max_per_creator,
                "creator {} has {} items, max is {}",
                creator, count, max_per_creator);
        }
    }
}

// P3: Gate bypass for exploration items.
proptest! {
    fn exploration_items_bypass_gates(
        candidates in arb_scored_candidates(100),
        exploration_budget in 0.05f64..0.20,
    ) {
        let profile = profile_with_gates_and_exploration(exploration_budget);
        let results = full_pipeline(&candidates, &profile, 50);

        let exploration_items: Vec<_> = results.iter()
            .filter(|r| r.is_exploration)
            .collect();

        // Exploration items may have signals below gate thresholds
        // (that's the point -- they're new items). This test verifies
        // they are included despite not meeting gate criteria.
        prop_assert!(exploration_items.len() <= (exploration_budget * 50.0).ceil() as usize);
    }
}

// P4: Pagination produces no duplicates.
proptest! {
    fn pagination_no_duplicates(
        candidates in arb_scored_candidates(200),
        page_size in 10u32..50,
    ) {
        let page1 = retrieve_with_cursor(&candidates, None, page_size);
        let page2 = retrieve_with_cursor(&candidates, page1.next_cursor, page_size);

        let page1_ids: HashSet<_> = page1.results.iter().map(|r| &r.id).collect();
        let page2_ids: HashSet<_> = page2.results.iter().map(|r| &r.id).collect();

        let overlap: Vec<_> = page1_ids.intersection(&page2_ids).collect();
        prop_assert!(overlap.is_empty(),
            "duplicate items across pages: {:?}", overlap);
    }
}

// P5: Score normalization produces valid range.
proptest! {
    fn normalized_scores_in_range(
        raw_scores in prop::collection::vec(
            prop::num::f64::NORMAL | prop::num::f64::POSITIVE,
            10..500
        ),
    ) {
        let normalized = min_max_normalize(&raw_scores);
        for &score in &normalized {
            prop_assert!(score >= 0.0 && score <= 1.0,
                "normalized score {} out of range", score);
        }
    }
}

// P6: Deterministic scoring.
proptest! {
    fn scoring_deterministic(
        candidates in arb_scored_candidates(100),
        profile in arb_profile(),
    ) {
        let results1 = full_pipeline(&candidates, &profile, 50);
        let results2 = full_pipeline(&candidates, &profile, 50);

        for (r1, r2) in results1.iter().zip(results2.iter()) {
            prop_assert_eq!(r1.id, r2.id);
            prop_assert!((r1.score - r2.score).abs() < f64::EPSILON,
                "non-deterministic scoring: {} vs {}", r1.score, r2.score);
        }
    }
}

---

17. Integration Points

17.1 Signal System Integration

The ranking pipeline reads signal data from the three-tier signal ledger (Signal System spec, Section 3):

Ranking Need	Signal Tier	Read Latency
Decay scores for boost application	Hot tier (atomic reads)	~15 ns per entity
Windowed counts for velocity	Warm tier (bucket sums)	~200-500 ns per entity
Cohort-scoped aggregates	Cohort counters (disk-backed)	~500 ns - 2 us per entity
All-time counts for gates	Warm tier (atomic counter)	~2 ns per entity
Signal snapshot for response	All tiers	~5 us per entity

The ranking module never writes to the signal system. It is a pure consumer of signal state.

17.2 Relationship System Integration

The ranking pipeline reads relationship data for:

• Candidate generation (Relationship strategy): traverses follow edges.
• Boost application (Relationship boost): reads interaction_weight edges.
• Social proof: reads follow-set bitmap and per-item engagement flags.
• Hard exclusion: reads blocked/muted edges.

Relationship reads use the adjacency list storage format from the Relationships spec (Section 5). Forward adjacency lists (user -> creators they follow) are cached in memory as roaring bitmaps for O(1) membership tests.

17.3 Query Engine Integration

The query engine is the orchestrator. It receives a Retrieve or Search request, resolves the profile, executes candidate generation (delegating to the ANN index, Tantivy, or relationship store as needed), and invokes the scoring pipeline.

Query Engine
    ├── Profile Resolution
    │     └── Schema Catalog (profiles, signals, entities)
    ├── Candidate Generation
    │     ├── ANN Index (USearch) -- for ANN strategy
    │     ├── Tantivy -- for Hybrid strategy (text component)
    │     ├── Relationship Store -- for Relationship strategy
    │     └── Signal System -- for CohortTrending strategy
    ├── Scoring Pipeline (this spec)
    │     ├── Signal reads (hot/warm/cold tier)
    │     ├── Relationship reads (adjacency lists)
    │     └── Cohort reads (dimensional rollups)
    └── Response Assembly
          └── Signal snapshots for rendering

17.4 Cohort System Integration

When a query includes FOR COHORT, the query engine:

1. Resolves the cohort to a signal aggregation scope (Cohorts spec, Section 5).
2. Passes the scope to the scoring pipeline.
3. The scoring pipeline routes signal reads to cohort-scoped counters instead of global counters.
4. The response includes cohort_info with the cohort name, cardinality, and accuracy level.

---

Appendix A: Glossary

Term	Definition
Ranking Profile	A named, versioned scoring function declared in schema that fully specifies how candidates are retrieved, scored, filtered, diversified, and paginated
Candidate Generation	The first stage of the ranking pipeline that produces a raw set of entities with initial retrieval scores
Boost	A positive scoring component that adds a weighted signal value, relationship weight, or derived metric to a candidate's score
Penalty	A negative scoring component that subtracts a weighted signal value from a candidate's score
Gate	A hard quality threshold that excludes candidates from the result set (binary accept/reject)
Diversity Spec	A set of constraints that control variety in the result set (max per creator, format mix, topic diversity)
Exploration Budget	The fraction of results reserved for cold-start items, hidden gems, and serendipitous discovery
Sort Mode	A formula-based ranking function that replaces the boost/penalty scoring pipeline
MMR	Maximal Marginal Relevance -- a greedy reranking algorithm that balances relevance with diversity
RRF	Reciprocal Rank Fusion -- a rank-based score fusion strategy for combining text and vector retrieval results
Profile Decay	Multiplicative time-based decay applied to scores based on content age
Percentile Normalization	Mapping raw signal values to [0, 1] using rank within the candidate set
Cold Start	The state where an item or user has insufficient signal history for the scoring pipeline to evaluate them
Cohort Scoping	Routing signal reads to cohort-specific counters instead of global counters, changing the data source without changing the profile

Appendix B: References

1. Carbonell, J., Goldstein, J. "The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries." SIGIR 1998.
2. Cormack, G.V., Clarke, C.L.A., Buettcher, S. "Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods." SIGIR 2009.
3. Cormode, G., Shkapenyuk, V., Srivastava, D., Xu, B. "Forward Decay: A Practical Time Decay Model for Streaming Systems." ICDE 2009.
4. Reddit Hot Ranking Algorithm. "How Reddit Ranking Algorithms Work." Amir Salihefendic, 2015.
5. Hacker News Ranking Algorithm. Paul Graham, Y Combinator.
6. Wilson, E.B. "Probable Inference, the Law of Succession, and Statistical Inference." Journal of the American Statistical Association, 1927. (Lower bound of Wilson score interval for rating-based ranking.)
7. Signal System Specification. docs/specs/03-signal-system.md.
8. Relationships Specification. docs/specs/04-relationships.md.
9. Cohorts Specification. docs/specs/05-cohorts.md.
10. Vector Retrieval Specification. docs/specs/07-vector-retrieval.md.
11. Text Retrieval Specification. docs/specs/06-text-retrieval.md.