1092d34c39
- Add k8s/ manifests (StatefulSet, kustomize, PDB, ServiceMonitor) + docs/runbooks/kubernetes.md - Add tidal-server/src/openapi.rs (utoipa OpenAPI spec) and wire into router - Add docs/guides/ (build-a-feed-app, embeddings, server-deployment) + foryou_feed example - Consolidate tidal/docker/ into root docker/ (single canonical home) - Update API.md, QUICKSTART.md, README.md, CLAUDE.md, check-docs.sh accordingly
116 lines
12 KiB
Markdown
116 lines
12 KiB
Markdown
# Ranking Profiles
|
||
|
||
**Last Updated:** 2026-06-09
|
||
**Confidence:** High
|
||
|
||
## Summary
|
||
|
||
A ranking profile is a named, versioned bundle that fully specifies how a query turns candidates into an ordered result page: it names the **retrieval strategy** (how candidates are sourced — scan, ANN, signal-ranked, relationship, hybrid, cohort), the **primary sort** (the ordering formula — trending, hot, top-window, etc.), any **boosts / gates / penalties / excludes** (signal-driven score adjustments and filters), the **diversity** rules (per-creator caps and format-mix caps), and an **exploration** fraction (random injection to break filter bubbles). Profiles are selected by name at query time (`.profile("for_you")`), declared in schema, and versioned alongside data — old versions remain queryable. tidalDB ships **25 built-in profiles** registered at `version = 1` with `is_builtin = true`; applications add their own with the schema builder or the server config YAML.
|
||
|
||
**Key Facts:**
|
||
- Profiles are schema-level declarations, not application code. The app names a profile; the database executes the whole pipeline.
|
||
- The same profile operates over different candidate sets (global / category / social graph / cohort) depending on its candidate strategy and the query's filters.
|
||
- Personalization (`for_you`, `following`, `related`, `notification`, `date_saved`) requires `FOR USER` context — pass `.for_user(uid)` on the query. Without it the executor cannot read the user's preference vector / saved-state and either degrades to non-personalized scoring or, for `date_saved`, returns an error.
|
||
- **Preference-vector personalization only updates from signals declared `.positive_engagement(true)`.** `signal_with_context(..)` folds the item embedding into the user's taste vector only for positive-engagement signal types; negative signals (skip/hide/block) update seen-state and hard-negatives but do not pull the taste vector toward the item. Declare engagement signals accordingly or `for_you` will not learn.
|
||
- Built-in defaults below are read straight from `tidal/src/ranking/builtins.rs`. Where a value is intentionally application-tuning territory, it is described qualitatively rather than invented.
|
||
|
||
**File Pointers:** `tidal/src/ranking/builtins.rs` (the 25 built-ins), `tidal/src/ranking/profile.rs` (`RankingProfile`, `DiversitySpec`, `Sort`, `Boost`, `CandidateStrategy`), `VISION.md:43-55`.
|
||
|
||
## The 25 Built-in Profiles
|
||
|
||
`DiversitySpec` defaults to **no diversity** (`max_per_creator = None`, `format_mix_max_fraction = None`) and `exploration` defaults to `0.0`. Profiles below list only the values they override. "Needs FOR USER" marks profiles whose ordering or scoring depends on per-user context.
|
||
|
||
| Profile | Optimizes for | Primary sort | Diversity / exploration (defaults) | Needs FOR USER | Use case |
|
||
|---------|---------------|--------------|------------------------------------|----------------|----------|
|
||
| `trending` | Pure short-window engagement velocity (`view_vel + 2·share_vel`, 24h) | `Trending` | `max_per_creator=1`; no exploration | No | UC-03 |
|
||
| `hot` | Cumulative score decayed by age (Reddit/HN style) | `Hot { gravity = 1.8 }` | `max_per_creator=2`; no exploration | No | UC-14 |
|
||
| `new` | Freshest content first | `New` (`created_at DESC`) | none | No | UC-03, UC-04 |
|
||
| `for_you` | Personalized home feed (decayed view/like + 24h share velocity, on top of preference match) | `Hot { gravity = 1.5 }` | `max_per_creator=2`, `format_mix_max_fraction=0.4`, `exploration=0.1` | Yes | UC-01 |
|
||
| `following` | Recent content from creators the user follows | `New` | `max_per_creator=3`; no exploration | Yes | UC-04 |
|
||
| `related` | "More like this" for a seed item (semantic similarity + completion-weighted quality) | `Hot { gravity = 1.2 }` | `max_per_creator=2`; no exploration | No (use `.similar_to(item_id)`) | UC-05 |
|
||
| `notification` | High-velocity content from followed creators, worth interrupting for | `Trending` | `max_per_creator=1`; no exploration | Yes | UC-07 |
|
||
| `search` | Text + vector relevance (RRF fusion) with a light view/like quality overlay | `None` (fused RRF score from search executor is the primary order) | none (caller sets diversity); no exploration (deterministic) | No (personalizable via `.for_user`) | UC-02, UC-10, UC-11 |
|
||
| `top_week` | Best-of over a 7-day window | `TopWindow { SevenDays }` | none | No | UC-06 |
|
||
| `top_month` | Best-of over a 30-day window | `TopWindow { ThirtyDays }` | none | No | UC-06 |
|
||
| `top_all_time` | Cumulative quality, no decay (classic / best-of) | `TopWindow { AllTime }` | none | No | UC-06 |
|
||
| `hidden_gems` | High quality, low reach (inverse-reach boost) | `HiddenGems` | none | No | UC-13 |
|
||
| `controversial` | Maximum product of positive × negative signals (debate surfaces) | `Controversial` | none | No | UC-14 |
|
||
| `most_viewed` | Raw view count, 7-day window | `MostViewed { SevenDays }` | none | No | UC-06 |
|
||
| `most_liked` | Raw like count, 7-day window | `MostLiked { SevenDays }` | none | No | UC-06 |
|
||
| `shuffle` | Quality-weighted random ("surprise me") | `Shuffle` | `exploration=0.5` (half random, half signal-ranked); no creator cap | No | UC-06 |
|
||
| `cohort_trending` | Trending scoped to a named audience cohort ("trending for people like you") | `Trending` | `max_per_creator=1`; no exploration | No (use `.cohort("name")`) | UC-15 |
|
||
| `live` | Currently-live content by viewer count, lifting creators the user's circle watches | `LiveViewerCount` | `max_per_creator=1`; no exploration | Personalizable (relationship boost uses `.for_user`) | UC-12 |
|
||
| `alphabetical_asc` | Title A→Z (case-insensitive) | `AlphabeticalAsc` | none | No | UC-06 |
|
||
| `alphabetical_desc` | Title Z→A (case-insensitive) | `AlphabeticalDesc` | none | No | UC-06 |
|
||
| `shortest` | Quick content (`duration ASC`) | `Shortest` | none | No | UC-06 |
|
||
| `longest` | Deep dives (`duration DESC`) | `Longest` | none | No | UC-06 |
|
||
| `most_commented` | Discussion volume (comment count, all-time) | `MostCommented { AllTime }` | none | No | UC-06, UC-14 |
|
||
| `most_shared` | Virality (share count, all-time) | `MostShared { AllTime }` | none | No | UC-06 |
|
||
| `date_saved` | The user's personal library, latest save first | `DateSaved` | none | **Yes (required)** — missing `FOR USER` returns `InvalidFilter` | UC-09 |
|
||
|
||
### Notes on specific built-ins
|
||
|
||
- **`trending` / `cohort_trending` and the `view`/`share` boosts.** The `Sort::Trending` formula (`view_vel + 2·share_vel` over 24h) is the ordering authority on the global path and *replaces* the boost loop, so the executor skips boosts there (no double-count). The same `view`/`share` velocity boosts are not dead weight: the cohort rescore path (`rescore_with_cohort`) consumes `profile.boosts` directly to re-score against a cohort's ledger. Keep the boost weights mirroring the formula so cohort ordering matches global ordering.
|
||
- **`for_you` boosts vs. preference match.** Beyond the `Hot { gravity = 1.5 }` recency curve, `for_you` boosts decayed `view` (weight 1.0), decayed `like` (weight 2.0), and 24h `share` velocity (weight 1.5), layered on top of the user-preference-vector match. The 10% exploration injection is what keeps it from collapsing into a filter bubble.
|
||
- **`search` has `sort = None` on purpose.** The fused RRF (BM25 + ANN) score from the search executor is the primary ordering signal; the profile only adds a small `view`/`like` quality overlay (weights 0.5 / 0.8) so relevance stays in charge. Exploration is `0.0` so results are deterministic for a given query. Set diversity yourself via `SearchBuilder::diversity(..)`.
|
||
- **Three sort modes have no built-in profile shortcut.** `MostFollowed` and `CreatorEngagementRate` rank **creators** (not items) and need a creator-scoped candidate strategy plus a `follow` signal the generic schema does not guarantee; `Rising` (1h/24h view-velocity acceleration ratio) overlaps `trending`/`hot` and is sensitive to the exact window pair. All three are reachable via a custom schema profile (`SchemaBuilder::ranking_profile(..).sort(..)`); only the named-convenience built-in is omitted.
|
||
|
||
## Candidate (retrieval) strategies
|
||
|
||
The candidate strategy decides *which* items the profile even considers before sorting:
|
||
|
||
| Strategy | What it does | Built-ins that use it |
|
||
|----------|--------------|-----------------------|
|
||
| `Scan` (`sort_field`) | Population scan over the item keyspace; default for built-ins (`sort_field = "created_at"`) | most population + sort-coverage profiles |
|
||
| `Relationship` | Sources candidates from the user's relationship graph (followed creators) | `following`, `notification` |
|
||
| `Ann { slot, limit }` | k-NN over an embedding slot (set via the query's `.similar_to` / `Search`'s `.vector`) | custom; `related` uses the seed embedding via the query |
|
||
| `SignalRanked { signal, window }` | Pre-orders candidates by a signal's windowed aggregate | custom |
|
||
| `Hybrid` | Fuses lexical + vector retrieval | the `search` query path (RRF fusion in the search executor) |
|
||
| `CohortTrending` | Cohort-scoped signal aggregation | reached via `.cohort("name")` with `cohort_trending` |
|
||
|
||
> Embeddings are app-supplied. tidalDB retrieves and ranks over vectors but does **not** generate them; `RETRIEVE`/`SEARCH` route ANN through the **first declared embedding slot only** (fuse multi-modal vectors offline or use separate entity kinds). See [API.md](../../API.md) and [QUICKSTART.md](../../QUICKSTART.md).
|
||
|
||
## Defining a custom profile
|
||
|
||
Profiles are declared in schema (Rust `SchemaBuilder`) or, for the HTTP server, in the config YAML loaded at startup. The YAML shape mirrors the `RankingProfile` type exactly:
|
||
|
||
```yaml
|
||
profiles:
|
||
- name: my_feed
|
||
version: 1
|
||
# candidate_strategy: scan | { ann: { slot, limit } }
|
||
# | { signal_ranked: { signal, window } }
|
||
# | relationship | hybrid | cohort_trending
|
||
candidate_strategy: scan # default: scan over created_at
|
||
# sort: trending | rising | controversial | hidden_gems | shuffle | new
|
||
# | { hot: { gravity } } | { top_window: { window } }
|
||
# | { most_viewed: { window } } | ...
|
||
sort: { hot: { gravity: 1.8 } }
|
||
boosts:
|
||
# agg: value | velocity | decay_score | ratio | relative_velocity
|
||
- { signal: like, agg: decay_score, window: all_time, weight: 2.0 }
|
||
- { signal: share, agg: velocity, window: twenty_four_hours, weight: 1.5 }
|
||
gates: [] # drop candidates failing a signal threshold
|
||
penalties: [] # subtractive score adjustments
|
||
excludes: [] # hard removals
|
||
diversity:
|
||
max_per_creator: 2 # at most 2 items per creator in the page
|
||
format_mix_max_fraction: 0.4 # no single format > 40% of the page
|
||
exploration: 0.1 # 10% random injection (0.0 = none)
|
||
```
|
||
|
||
Windows are `one_hour`, `twenty_four_hours`, `seven_days`, `thirty_days`, `all_time`. Only signals declared in the same schema may be referenced by boosts/gates — the server validates references at load time and rejects a profile naming an unknown signal. Gates, penalties, and excludes are deliberately omitted from the built-ins because they would assume signals not guaranteed to exist in a generic schema; add them in your own profile once you know your signal set.
|
||
|
||
In Rust, the equivalent lives behind `SchemaBuilder::ranking_profile(..)`; see [API.md](../../API.md) for the builder surface and the runnable `tidal/examples/foryou_feed.rs` / `tidal/examples/quickstart.rs` for end-to-end usage.
|
||
|
||
## Related Topics
|
||
|
||
- [Signals](./signals.md) — the decay/velocity/windowed-aggregation primitives boosts read
|
||
- [Entities](./entities.md) — items/users/creators the profiles rank over
|
||
- [Query Language](../features/query-language.md) — `RETRIEVE` / `SEARCH` / `SIGNAL` surface
|
||
- [Sort Modes](../features/sort-modes.md) — the 25+ native sort formulas profiles select from
|
||
- [Filters](../features/filters.md) — composable filter dimensions layered onto any profile
|
||
- [USE_CASES.md](../../USE_CASES.md) — UC-01…UC-15 and **Appendix B · Sort Mode Reference**
|
||
- [API.md](../../API.md) — builder API + HTTP routes
|
||
- [QUICKSTART.md](../../QUICKSTART.md) — first feed in minutes
|