tidaldb/site/content/blog/one-query-six-systems.mdx

---
title: "One query. Six systems. Under 50 milliseconds."
date: "2026-02-21"
author: "Jordan Washburn"
description: "A single RETRIEVE query retrieves candidates, filters by metadata, scores using live decay signals and velocity, enforces creator diversity, and returns a ranked list. That is what Elasticsearch + Redis + a ranking service produce. It is one query here."
tags: ["query", "ranking", "performance", "rust"]
---

Here is what it takes to answer "what should this user see?" in the 6-system stack:

1. The ranking service calls Elasticsearch with a query and a filter.
2. Elasticsearch returns candidate documents sorted by a `trending_score` field that was last updated by a cron job 4 minutes ago.
3. The ranking service calls Redis for each candidate to read current view counts and velocity.
4. The ranking service calls the feature store for the user's preference vector.
5. The ranking service scores each candidate using the preference vector, the Redis signals, and the Elasticsearch metadata.
6. The ranking service applies diversity rules in application code ("no more than 2 items from the same creator").
7. The ranking service returns the sorted list.

That is 3 network round-trips, 2 consistency boundaries, 1 stale score field, and a diversity pass that nobody wrote tests for because it lives in a microservice that seven people have touched and two people understand.

In tidalDB, this is one function call:

```rust
let query = Retrieve::builder()
    .profile("trending")
    .filter(FilterExpr::CategoryEq("jazz".into()))
    .diversity(DiversityConstraints::new().max_per_creator(1))
    .limit(25)
    .build()?;

let results = db.retrieve(&query)?;
```

The query works. This post explains what it does, how it does it, and what it costs.

## The RETRIEVE query

A `Retrieve` is a declarative struct. It says *what* you want. It never says *how* to get it.

```rust
// Builder pattern — handles defaults for omitted fields
// (for_user, similar_to, context, exclude, cursor).
let query = Retrieve::builder()
    .entity(EntityKind::Item)
    .profile("trending")
    .filter(FilterExpr::CategoryEq("jazz".into()))
    .diversity(DiversityConstraints::new().max_per_creator(1))
    .limit(25)
    .build()?;
```

The caller names a ranking profile and specifies constraints. The database figures out the rest: how to source candidates, which index to use, how to score, how to enforce diversity, how to paginate. The caller does not know whether the candidates came from an ANN search, a full entity scan, or a signal-ranked top-K. That is a decision the profile makes, not the application.

The builder validates at construction time. A limit of 0 or 501 fails before the query reaches the executor. A missing profile name fails before the pipeline starts. The error types are explicit:

```rust
pub enum QueryError {
    ProfileNotFound(String),
    InvalidFilter { field: String, reason: String },
    InvalidLimit { requested: usize, min: usize, max: usize },
    IndexNotAvailable(String),
    StorageError(String),
    InvalidCursor(String),
    UnsupportedStrategy(String),
}
```

No stringly-typed errors. No "something went wrong." The caller knows exactly what failed and why.

## The five-stage pipeline

The executor runs five stages in sequence. Every stage receives the output of the prior stage. No branching. No async fan-out. One path through the data.

```
Stage 1: Candidate Generation
  Profile's CandidateStrategy -> Vec<EntityId>
  (Scan, SignalRanked, or ANN)

Stage 2: Filter Evaluation
  FilterEvaluator + RoaringBitmap intersection
  -> surviving candidates

Stage 3: Signal Scoring
  ProfileExecutor::score() -> Vec<ScoredCandidate>
  sorted descending, normalized to [0.0, 1.0]

Stage 4: Diversity Enforcement
  DiversitySelector::select() -> reordered candidates
  max_per_creator and format_mix applied

Stage 5: Result Assembly
  Slice to [offset..offset+limit], build RetrieveResult
  with rank, score, and signal snapshot
```

### Stage 1: Candidate generation

The ranking profile declares a `CandidateStrategy`. The executor reads it and routes accordingly.

Most built-in profiles use `Scan` as their candidate strategy -- iterate the universe bitmap of all known entity IDs. The differentiation happens at Stage 3, where the profile's `Sort` mode determines the scoring formula. `Sort::MostViewed` reads windowed view counts. `Sort::Trending` reads velocity. `Sort::Hot` applies a gravity function. Same candidates in, different scores out. Two profiles break this pattern: `following` and `notification` use `CandidateStrategy::Relationship`, sourcing candidates from the user's relationship graph rather than scanning the full universe. `SignalRanked` is available as a candidate strategy but no built-in profile uses it yet.

The scan reads a `RoaringBitmap` -- the universe of all item IDs written to the database. At 10K items, this produces candidates in under a millisecond. Overprovisioning is 4x the requested limit or 200, whichever is larger, so there are enough candidates to survive filtering and diversity.

```rust
fn scan_candidates(&self, limit: usize) -> Vec<EntityId> {
    let max_candidates = (limit * 4).max(200);
    let mut candidates = Vec::with_capacity(max_candidates);

    if let Some(universe) = self.universe
        && let Ok(bm) = universe.read()
    {
        for id_u32 in bm.iter() {
            candidates.push(EntityId::new(u64::from(id_u32)));
            if candidates.len() >= max_candidates {
                break;
            }
        }
    }
    candidates
}
```

The `SignalRanked` strategy also exists -- it walks the signal ledger's hot tier, reads the decay score for a named signal, sorts descending, and returns the top-K. No built-in profile uses it yet, but custom profiles can opt into it when the scan-then-score path is too expensive for their candidate universe.

No external index. No precomputed field that went stale 4 minutes ago. The signal values are computed from the running accumulators at the instant the query executes. If a signal was written 100 milliseconds ago, the scoring sees it.

### Stage 2: Filter evaluation

Filters narrow the candidate set using bitmap indexes. When you write an item with `category: "jazz"`, the database inserts the item's ID into a `RoaringBitmap` keyed by `("category", "jazz")`. At query time, the filter evaluator retrieves that bitmap and intersects it with the candidate set.

```rust
if let Some(filter_expr) = query.combined_filter() {
    let evaluator = FilterEvaluator::new(
        cat, fmt, cre, tag, dur, ts, universe_ref
    );
    let filter_result = evaluator.evaluate(&filter_expr);

    match filter_result {
        FilterResult::Bitmap(bitmap) => {
            candidates.retain(|id| bitmap.contains(id.as_u64() as u32));
        }
        FilterResult::Predicate(pred) => {
            candidates.retain(|id| pred(id.as_u64()));
        }
    }
}
```

Bitmap intersection is a bitwise AND. At 10K items, it takes microseconds. The candidate set shrinks. Everything downstream operates on fewer entities.

The acceptance test proves filter correctness: every result from a `hot` query filtered by `CategoryEq("jazz")` has category "jazz." No exceptions.

```rust
let query = Retrieve::builder()
    .profile("hot")
    .filter(FilterExpr::CategoryEq("jazz".into()))
    .limit(20)
    .build()?;

let results = db.retrieve(&query)?;

// Every result has category "jazz" — the filter is enforced before scoring.
// Verified in the acceptance test: zero results with a non-matching category.
```

### Stage 3: Signal scoring

This is where the ranking profile earns its name. The `ProfileExecutor` takes the surviving candidate IDs, reads their signal state from the ledger, and applies the profile's scoring formula.

tidalDB ships 15 built-in profiles. Each is a standard `RankingProfile` struct -- not special-cased in the executor. The sort mode determines the formula:

| Profile | Sort Mode | Formula |
|---------|-----------|---------|
| `trending` | `Trending` | Share velocity (24h, 2x weight) + view velocity (24h) |
| `hot` | `Hot { gravity: 1.8 }` | `log10(max(views, 1)) / (age_hours + 2)^1.8` |
| `new` | `New` | Entity recency (higher ID = newer) |
| `top_week` | `TopWindow { SevenDays }` | `views * 0.3 + likes * 0.3 + shares * 0.2 + completion * views * 0.1` within 7-day window |
| `top_month` | `TopWindow { ThirtyDays }` | Same multi-signal formula within 30-day window |
| `top_all_time` | `TopWindow { AllTime }` | Same multi-signal formula, all-time window |
| `hidden_gems` | `HiddenGems` | `quality_score * (1 / log10(view_count + 10))` |
| `controversial` | `Controversial` | `(positive * negative) / (positive + negative)^2` |
| `most_viewed` | `MostViewed { SevenDays }` | Raw view count within window |
| `most_liked` | `MostLiked { SevenDays }` | Raw like count within window |
| `shuffle` | `Shuffle` | Deterministic seeded RNG |
| `for_you` | `ForYou { exploration: 0.1 }` | Interaction boosts + exploration injection |
| `following` | `Following` | Candidates from followed creators, recency ordered |
| `related` | `Related` | Item similarity (ANN, M5) |
| `notification` | `Notification` | Relationship-sourced candidates, recency ordered |

Every profile reads from the same signal ledger. The `trending` profile reads velocity. The `hot` profile reads view counts and applies a gravity decay by age. The `hidden_gems` profile reads completion rates and penalizes reach. The data is the same. The lens is different.

The profiles are data, not code. Defined as structs. Registered at runtime. Versioned. Swappable by name at query time. Changing which ranking formula your feed uses does not require a deployment. It requires changing a string.

```rust
fn score_by_sort(&self, entity_id: EntityId, sort: Option<&Sort>, now: Timestamp) -> f64 {
    match sort {
        Some(Sort::Hot { gravity }) => self.score_hot(entity_id, *gravity, now),
        Some(Sort::Trending) => self.score_trending(entity_id),
        Some(Sort::Controversial) => self.score_controversial(entity_id),
        Some(Sort::HiddenGems) => self.score_hidden_gems(entity_id),
        Some(Sort::New) => { /* entity recency */ },
        Some(Sort::TopWindow { window }) => self.score_top_window(entity_id, *window),
        Some(Sort::MostViewed { window }) => read_agg(entity_id, "view", ...),
        Some(Sort::MostLiked { window }) => read_agg(entity_id, "like", ...),
        Some(Sort::Shuffle) => shuffle_score(entity_id.as_u64()),
        // ...
    }
}
```

After scoring, results are normalized to [0.0, 1.0] via min-max normalization. Deterministic: same candidates, same signal state, same profile produces identical output. The acceptance test asserts this directly.

### Stage 4: Diversity enforcement

Without diversity, a trending creator dominates the feed. If creator A has 5 of the top 10 items, the user sees 5 items from creator A. This is correct by score. It is wrong by experience.

The `DiversitySelector` is a greedy selection pass over the scored list. It walks candidates in score order and accepts each one if it does not violate the active constraints. If `max_per_creator` is 1, the second item from creator A is skipped. The selector moves to the next candidate. Result count does not decrease -- the selector fills the target count from lower-ranked candidates that satisfy constraints.

When constraints cannot be fully satisfied (too few distinct creators in the candidate set), the selector relaxes in a defined order:

- Stage 0: full constraints
- Stage 1: `max_per_creator` doubled
- Stage 2: `format_mix` ignored
- Stage 3: accept anything to fill target count

The result includes a `constraints_satisfied` boolean and a list of `ConstraintViolation` structs. The caller knows exactly what was relaxed and why. No silent degradation.

```rust
let (final_candidates, constraints_satisfied) = if let Some(ref diversity) = query.diversity {
    let result = DiversitySelector::select(&scored, diversity, scored.len());
    let satisfied = result.constraints_satisfied;
    (result.selected, satisfied)
} else {
    (scored, true)
};
```

The UAT test verifies it end-to-end:

```rust
let query = Retrieve::builder()
    .profile("trending")
    .diversity(DiversityConstraints::new().max_per_creator(1))
    .limit(50)
    .build()?;

let results = db.retrieve(&query)?;

// The acceptance test verifies that no creator appears more than once
// when the candidate set is large enough. When too few distinct creators
// exist, the selector relaxes constraints in a defined order and sets
// results.constraints_satisfied = false.
```

### Stage 5: Result assembly

The final stage slices the candidate list to the requested page, assigns 1-based ranks, attaches signal snapshots for debugging transparency, and computes the pagination cursor.

```rust
RetrieveResult {
    entity_id: c.entity_id,
    score: c.score,
    rank: offset + i + 1,   // 1-based
    signals: c.signal_snapshot
        .iter()
        .map(|(name, value)| Signal {
            name: name.clone(),
            value: *value,
            source: "decay_score".to_string(),
        })
        .collect(),
}
```

The signal snapshot is designed to be the ranking equivalent of `EXPLAIN` in SQL -- each result will carry the key signal values that contributed to its score. The plumbing is in place: the `signal_snapshot` field exists on every `ScoredCandidate` and flows through to the result struct. Population of that field with actual signal breakdowns is coming in a future milestone. Today, the snapshot is empty. The score is accurate; the explanation of that score is not yet attached.

Pagination uses offset-based cursors encoded as base64. The cursor is opaque to the caller -- pass it back on the next request. The acceptance test verifies no overlap between pages and correct rank continuation.

## The UAT

The acceptance test writes 1,000 items with metadata (category, format, creator_id, created_at), writes 1,000 signal events across 5 signal types spanning 7 days, and runs 6 queries:

1. **Trending with diversity** -- 50 results, scores descending, max 1 per creator.
2. **Hot filtered by category** -- only jazz items, scores descending.
3. **New** -- entity recency descending.
4. **Top week** -- multi-signal weighted score within 7-day window.
5. **Hidden gems** -- quality/reach ratio ordering.
6. **Controversial** -- dual-signal ranking.

After the 6 queries, the test writes a burst of 100 share signals for a single entity and verifies the trending rank changes. Then it shuts down the database, reopens it, verifies signal state survived via WAL replay, repopulates the in-memory indexes, and re-runs queries to confirm correctness.

```rust
// Write the burst.
for j in 0..100_u64 {
    let ts = Timestamp::from_nanos(burst_ts.as_nanos() + j * 1_000_000);
    db.signal("share", burst_entity, 1.0, ts)?;
}

// Read back the windowed count — all 100 signals are immediately visible.
let share_count_after = db.read_windowed_count(burst_entity, "share", Window::AllTime)?;
// share_count_after == share_count_before + 100 — no lag, no consumer to wait for.
```

The signal burst test is the thesis in microcosm. Write 100 signals. Re-execute the same query. The ranking reflects the new data. No cache invalidation. No consumer lag. No batch pipeline. The signals and the query share a process, a memory space, and a ledger.

## What this replaces

Here is the dependency graph for a trending feed query in the 6-system stack:

```
Application
  -> Ranking Service
       -> Elasticsearch (candidates + stale trending_score field)
       -> Redis (current view counts, velocity)
       -> Feature Store (user preference vector)
       -> [merge, score, diversity in application code]
  <- sorted results
```

Three network calls. Three consistency models. One stale field. Diversity rules that live in a microservice nobody wants to refactor.

Here is the same query in tidalDB:

```
Application
  -> db.retrieve(&query)
       Stage 1: universe bitmap scan -> candidates
       Stage 2: RoaringBitmap filter -> surviving candidates
       Stage 3: signal ledger read + profile formula -> scored candidates
       Stage 4: greedy diversity selection -> reordered candidates
       Stage 5: pagination + signal snapshot -> Results
  <- Results
```

One function call. One process. One consistency model. The data is never stale because the write path and the read path share the same signal ledger.

## What is not here yet

Personalization shipped in M3 and M4. The `FOR USER` clause is live: pass a user ID and the query applies seen/blocked exclusions, interaction boosts on creators the user has engaged with, and preference vector updates. The `following` and `notification` profiles source candidates from the user's relationship graph. The `for_you` profile blends interaction signals with exploration injection. Session context (M4) adds ephemeral preferences that shape ranking within a single session.

ANN candidate generation (vector similarity search over embeddings via USearch) falls back to a full scan with a warning. The infrastructure is integrated and tested, but wiring it as a first-class retrieval path comes next. The scan-based approach is sufficient for the item counts this version targets.

In-memory indexes (bitmap, range) are not persisted to disk. After a crash and restart, signal state survives via WAL checkpoint and replay, but items must be re-written to repopulate the indexes. The acceptance test verifies this path explicitly. Full index persistence is on the roadmap.

The text query parser (`RETRIEVE items USING PROFILE trending LIMIT 25` as a string) is not yet implemented. Queries today are constructed via the Rust builder API. The semantics are identical -- the parser will produce the same `Retrieve` struct the builder produces.

Signal snapshot population -- attaching the individual signal values that contributed to each result's score -- is plumbed but not yet producing data. The field exists on every result; the producer does not yet write to it.

## What is next

Personalized ranking is operational. User entities carry preference vectors. The relationship graph tracks follows, blocks, and interactions. The `FOR USER` clause shapes candidate generation, scoring, and exclusions in a single query. When a user likes an item, the database updates the item's signal ledger and the user-to-creator relationship weight. One write. The next query reflects it.

Next: the text query parser, full ANN as a first-class candidate strategy, signal snapshot population for per-result explainability, and index persistence across restarts. The ranking pipeline works. What remains is the tooling around it.

---

*The acceptance test is at [tidal/tests/m2_uat.rs](https://github.com/orchard9/tidalDB/blob/main/tidal/tests/m2_uat.rs). The query executor is at [tidal/src/query/executor.rs](https://github.com/orchard9/tidalDB/blob/main/tidal/src/query/executor.rs). The 15 built-in profiles are at [tidal/src/ranking/builtins.rs](https://github.com/orchard9/tidalDB/blob/main/tidal/src/ranking/builtins.rs). Follow the build on [GitHub](https://github.com/orchard9/tidalDB).*