tidaldb/API.md

# API Reference

> **Quick API Reference:** The examples below reflect the current implementation API. Use `cargo doc --manifest-path tidal/Cargo.toml --open` for full documentation.

How developers interact with tidalDB. This document covers initialization, schema definition, write operations, queries, and the feedback loop.

tidalDB has two interfaces:

1. **Rust library** — embed it in your process. No network overhead, no serialization. The API is Rust types and method calls.
2. **HTTP server** (`tidal-server`) — a standalone Axum-based server exposing a REST API for write, query, and health operations. Useful for polyglot stacks or when embedding isn't practical.

---

## Table of Contents

- [Initialization](#initialization)
- [Schema Definition](#schema-definition)
  - [Entity Types](#entity-types)
  - [Signal Definitions](#signal-definitions)
  - [Ranking Profiles](#ranking-profiles)
- [Write Path](#write-path)
  - [Ingesting Entities](#ingesting-entities)
  - [Writing Embeddings](#writing-embeddings)
  - [Writing Relationships](#writing-relationships)
  - [Writing Signals](#writing-signals)
- [Query Language](#query-language)
  - [RETRIEVE — Feeds, Browse, Related](#retrieve--feeds-browse-related)
  - [SEARCH — Text + Semantic Retrieval](#search--text--semantic-retrieval)
  - [SUGGEST — Autocomplete and Suggestions](#suggest--autocomplete-and-suggestions)
- [Filters](#filters)
- [Sort Modes](#sort-modes)
- [Diversity Constraints](#diversity-constraints)
- [Pagination](#pagination)
- [Response Format](#response-format)
- [Lifecycle and Operations](#lifecycle-and-operations)
- [HTTP Server API](#http-server-api)
  - [Authentication](#authentication)
  - [Write Endpoints](#write-endpoints)
  - [Query Endpoints](#query-endpoints)
  - [Health Endpoints](#health-endpoints)
  - [Error Responses](#error-responses)

---

## Initialization

Open a database using the builder pattern. Define the schema first, then pass it to the builder.

```rust
use tidaldb::TidalDb;
use tidaldb::schema::{SchemaBuilder, EntityKind, DecaySpec, Window};
use std::time::Duration;

// 1. Define the schema (signal types, text fields, embedding slots).
let mut schema = SchemaBuilder::new();
let _ = schema.signal("view", EntityKind::Item, DecaySpec::Exponential {
    half_life: Duration::from_secs(7 * 24 * 3600),
})
    .windows(&[Window::OneHour, Window::TwentyFourHours, Window::ThirtyDays, Window::AllTime])
    .velocity(true)
    .add();
let schema = schema.build().expect("valid schema");

// 2a. Ephemeral (in-memory) -- no filesystem access, ideal for testing.
let db = TidalDb::builder()
    .ephemeral()
    .with_schema(schema.clone())
    .open()?;

// 2b. Persistent -- durable storage at the given path.
let db = TidalDb::builder()
    .with_data_dir("/var/lib/tidaldb/my_app")
    .with_schema(schema)
    .open()?;
```

The database is `Send + Sync`. Share it across threads with `Arc<TidalDb>`.

---

## Schema Definition

Schema is defined before opening the database using `SchemaBuilder`. It declares signal types, text fields for full-text search, and embedding slots for vector search.

### Entity Types

Entities are the nodes of the system. Three built-in types: **Item**, **User**, **Creator**. Entity metadata is stored as `HashMap<String, String>` key-value pairs.

### Signal Definitions

Signals are typed, timestamped event streams. Decay, velocity, and windowed aggregation are declared in schema -- not computed in application code.

```rust
use tidaldb::schema::{SchemaBuilder, EntityKind, DecaySpec, Window, TextFieldType};
use std::time::Duration;

let mut schema = SchemaBuilder::new();

// View signal: exponential decay, 7-day half-life, three windows + velocity.
let _ = schema.signal("view", EntityKind::Item, DecaySpec::Exponential {
    half_life: Duration::from_secs(7 * 24 * 3600),
})
    .windows(&[Window::OneHour, Window::TwentyFourHours, Window::SevenDays, Window::ThirtyDays, Window::AllTime])
    .velocity(true)
    .add();

// Like signal: slower decay (14 days).
let _ = schema.signal("like", EntityKind::Item, DecaySpec::Exponential {
    half_life: Duration::from_secs(14 * 24 * 3600),
})
    .windows(&[Window::TwentyFourHours, Window::SevenDays, Window::AllTime])
    .velocity(true)
    .add();

// Skip signal: fast decay (1 day), no velocity.
let _ = schema.signal("skip", EntityKind::Item, DecaySpec::Exponential {
    half_life: Duration::from_secs(24 * 3600),
})
    .windows(&[Window::OneHour, Window::TwentyFourHours])
    .velocity(false)
    .add();

// Hide signal: permanent (never decays), no windows.
let _ = schema.signal("hide", EntityKind::Item, DecaySpec::Permanent).add();

// Share signal: for trending and social features.
let _ = schema.signal("share", EntityKind::Item, DecaySpec::Exponential {
    half_life: Duration::from_secs(3 * 24 * 3600),
})
    .windows(&[Window::OneHour, Window::TwentyFourHours, Window::AllTime])
    .velocity(true)
    .add();

// Completion signal: long-lived quality metric.
let _ = schema.signal("completion", EntityKind::Item, DecaySpec::Exponential {
    half_life: Duration::from_secs(30 * 24 * 3600),
})
    .windows(&[Window::AllTime])
    .velocity(false)
    .add();

// Text fields for BM25 full-text search.
schema.text_field("title", TextFieldType::Text);
schema.text_field("description", TextFieldType::Text);
schema.text_field("category", TextFieldType::Keyword);
schema.text_field("tags", TextFieldType::Keyword);

// Creator text fields for creator search.
schema.creator_text_field("name", TextFieldType::Text);
schema.creator_text_field("handle", TextFieldType::Keyword);

// Embedding slots for vector search (you provide the vectors).
schema.embedding_slot("content", EntityKind::Item, 128);
schema.embedding_slot("content", EntityKind::Creator, 128);

let schema = schema.build()?;
```

**Decay types:**

| Decay | Behavior |
|---|---|
| `Exponential { half_life }` | Signal weight halves every `half_life` duration |
| `Linear { lifetime }` | Signal weight drops linearly to zero over `lifetime` |
| `Permanent` | Never decays -- hides, blocks, follows |

The full signal reference is in [USE_CASES.md Appendix C](USE_CASES.md#appendix-c--signal-reference).

### Ranking Profiles

tidalDB ships 25 built-in ranking profiles. The application says `profile("trending")`. The database executes the entire pipeline.

Built-in profiles include: `trending`, `hot`, `new`, `for_you`, `following`, `related`, `notification`, `search`, `top_week`, `top_month`, `top_all_time`, `hidden_gems`, `controversial`, `most_viewed`, `most_liked`, `shuffle`, `cohort_trending`, `live`, `alphabetical_asc`, `alphabetical_desc`, `shortest`, `longest`, `most_commented`, `most_shared`, `date_saved`.

See [ai-lookup/services/ranking-profiles.md](ai-lookup/services/ranking-profiles.md) for the full list of built-in profiles.

### Cohort Definitions

Cohorts are named predicates over user attributes. They define audience segments for scoped signal aggregation and trending.

```rust
use tidaldb::schema::EntityId;

// Define a cohort via db.define_cohort() after opening.
// Cohort signal aggregation happens at signal write time.
// Use RetrieveBuilder::cohort("us_young_music") to scope queries.
```

---

## Write Path

### Ingesting Entities

Items enter the system with metadata as `HashMap<String, String>` key-value pairs. The application provides the embedding -- tidalDB does not generate vectors.

```rust
use std::collections::HashMap;
use tidaldb::schema::EntityId;

let mut metadata = HashMap::new();
metadata.insert("title".to_string(), "Introduction to Jazz Piano".to_string());
metadata.insert("description".to_string(), "A beginner's guide...".to_string());
metadata.insert("category".to_string(), "music".to_string());
metadata.insert("tags".to_string(), "jazz,piano,tutorial,beginner".to_string());
metadata.insert("format".to_string(), "video".to_string());
metadata.insert("language".to_string(), "en".to_string());
metadata.insert("duration".to_string(), "1320".to_string()); // seconds
metadata.insert("creator_id".to_string(), "100".to_string());

db.write_item_with_metadata(EntityId::new(1), &metadata)?;
```

On commit, the item is:
1. Stored in the entity store
2. Text fields indexed in the inverted index (BM25)
3. Inserted into bitmap and range indexes for filtering
4. Added to the universe bitmap for RETRIEVE queries
5. **Immediately queryable**

### Writing Embeddings

Embeddings are written separately from metadata. tidalDB L2-normalizes and indexes them into the HNSW vector index.

```rust
use tidaldb::schema::EntityId;

// Item embedding (you compute this externally).
let embedding: Vec<f32> = compute_embedding("Introduction to Jazz Piano");
db.write_item_embedding(EntityId::new(1), &embedding)?;

// Creator embedding.
let creator_embedding: Vec<f32> = compute_creator_embedding("Jazz Academy");
db.write_creator_embedding(EntityId::new(100), &creator_embedding)?;
```

### Writing Relationships

Relationships are directional edges between entities (follows, blocks). Used for the `following` profile and blocked-creator filtering.

```rust
use tidaldb::schema::EntityId;
use tidaldb::schema::Timestamp;
use tidaldb::entities::RelationshipType;

// User follows a creator.
db.write_relationship(
    EntityId::new(123),            // from (user)
    RelationshipType::Follows,     // relationship type
    EntityId::new(100),            // to (creator)
    1.0,                           // weight
    Timestamp::now(),
)?;
```

**Relationship types:**

| Variant | Meaning |
|---|---|
| `Follows` | User follows a creator |
| `Blocks` | User blocks a creator |
| `InteractionWeight` | Weighted interaction edge |
| `Hide` | User hides a creator |
| `Mute` | User mutes a creator |

### Writing Signals

Signals are how the feedback loop closes. A single signal write atomically updates:
1. The item's signal ledger (windowed aggregates, velocity, decay score)
2. The WAL (write-ahead log) for durability

```rust
use tidaldb::schema::{EntityId, Timestamp};

// User viewed an item.
db.signal("view", EntityId::new(1), 1.0, Timestamp::now())?;

// User completed 94% of the video.
db.signal("completion", EntityId::new(1), 0.94, Timestamp::now())?;

// User liked an item.
db.signal("like", EntityId::new(1), 1.0, Timestamp::now())?;

// User skipped after 3 seconds (strong negative).
db.signal("skip", EntityId::new(2), 1.0, Timestamp::now())?;

// User tapped "Not interested" (permanent negative on this item).
db.signal("hide", EntityId::new(2), 1.0, Timestamp::now())?;
```

For signals with user context (updates preference vectors, seen state, interaction weights):

```rust
use tidaldb::schema::{EntityId, Timestamp};

db.signal_with_context(
    "view",
    EntityId::new(1),       // item
    1.0,                     // weight
    Timestamp::now(),
    Some(123),               // for_user
    Some(100),               // creator_id
)?;
```

The next ranking query -- even 100ms later -- reflects the updated state.

---

## Query Language

Three operations: **RETRIEVE** (feed generation, browse, related), **SEARCH** (text + semantic retrieval), **SUGGEST** (autocomplete).

All queries return ranked results with scores. The application renders -- it never re-ranks.

### RETRIEVE — Feeds, Browse, Related

RETRIEVE generates ranked content lists. It handles personalized feeds, category browse, trending, following, related content, and every other surface described in [USE_CASES.md](USE_CASES.md).

```rust
use tidaldb::query::retrieve::Retrieve;
use tidaldb::schema::EntityId;

// Personalized For You feed.
let query = Retrieve::builder()
    .for_user(123)
    .profile("for_you")
    .limit(50)
    .build()?;
let results = db.retrieve(&query)?;
```

```rust
// Trending globally.
let query = Retrieve::builder()
    .profile("trending")
    .limit(25)
    .build()?;
let results = db.retrieve(&query)?;
```

```rust
use tidaldb::storage::indexes::filter::FilterExpr;

// Trending in a category.
let query = Retrieve::builder()
    .profile("trending")
    .filter(FilterExpr::eq("category", "jazz"))
    .limit(25)
    .build()?;
let results = db.retrieve(&query)?;
```

```rust
// Trending within a cohort -- what's hot among US young music fans.
let query = Retrieve::builder()
    .profile("cohort_trending")
    .cohort("us_young_music")
    .limit(25)
    .build()?;
let results = db.retrieve(&query)?;
```

```rust
// Following feed -- content from followed creators.
let query = Retrieve::builder()
    .for_user(123)
    .profile("following")
    .limit(50)
    .build()?;
let results = db.retrieve(&query)?;
```

```rust
use tidaldb::ranking::diversity::DiversityConstraints;

// Related content / Up Next -- anchored to a specific item.
let query = Retrieve::builder()
    .for_user(123)
    .profile("related")
    .similar_to(EntityId::new(1))
    .diversity(DiversityConstraints::new().max_per_creator(1))
    .limit(10)
    .build()?;
let results = db.retrieve(&query)?;
```

```rust
// Browse category with explicit sort mode.
let query = Retrieve::builder()
    .profile("top_week")
    .filter(FilterExpr::eq("category", "jazz"))
    .limit(20)
    .build()?;
let results = db.retrieve(&query)?;
```

```rust
// Hidden gems -- high quality, low reach.
let query = Retrieve::builder()
    .profile("hidden_gems")
    .limit(20)
    .build()?;
let results = db.retrieve(&query)?;
```

```rust
// Exclude previously seen items.
let query = Retrieve::builder()
    .for_user(123)
    .profile("for_you")
    .exclude(vec![EntityId::new(1), EntityId::new(2)])
    .limit(50)
    .build()?;
let results = db.retrieve(&query)?;
```

```rust
// Creator profile -- items from a specific creator.
let query = Retrieve::builder()
    .profile("new")
    .for_creator(EntityId::new(100))
    .limit(20)
    .build()?;
let results = db.retrieve(&query)?;
```

```rust
// Notification prioritization.
let query = Retrieve::builder()
    .for_user(123)
    .profile("notification")
    .limit(20)
    .build()?;
let results = db.retrieve(&query)?;
```

### SEARCH — Text + Semantic Retrieval

Search combines full-text BM25 relevance with semantic similarity via RRF (Reciprocal Rank Fusion). Text relevance is the floor -- an irrelevant result never surfaces just because the user likes the creator.

```rust
use tidaldb::query::search::Search;

// Basic keyword search, personalized for this user.
let query = Search::builder()
    .query("rust tutorial beginner")
    .for_user(123)
    .limit(20)
    .build()?;
let results = db.search(&query)?;
```

```rust
// Hybrid search: text + vector.
let query_embedding: Vec<f32> = embed("rust tutorial beginner");
let query = Search::builder()
    .query("rust tutorial beginner")
    .vector(query_embedding)
    .for_user(123)
    .limit(20)
    .build()?;
let results = db.search(&query)?;
```

```rust
// Creator search.
use tidaldb::schema::EntityKind;

let query = Search::builder()
    .query("jazz piano")
    .entity_kind(EntityKind::Creator)
    .limit(10)
    .build()?;
let results = db.search(&query)?;
```

### Query Composition — SEARCH within Scoped Results

SEARCH can be composed with scope constraints. This enables searching within trending, within a cohort, or within any candidate set.

```rust
use tidaldb::query::search::{Search, WithinScope};

// Search within globally trending items.
let query = Search::builder()
    .query("jazz piano")
    .within(WithinScope::Trending { window_hours: 24 })
    .limit(20)
    .build()?;
let results = db.search(&query)?;
```

```rust
// Search within cohort-scoped trending.
let query = Search::builder()
    .query("jazz piano")
    .within(WithinScope::CohortTrending {
        cohort: "us_young_music".into(),
        window_hours: 24,
    })
    .limit(20)
    .build()?;
let results = db.search(&query)?;
```

```rust
// Search within a user's following feed.
let query = Search::builder()
    .query("jazz piano")
    .for_user(123)
    .within(WithinScope::Following)
    .limit(20)
    .build()?;
let results = db.search(&query)?;
```

**`WithinScope`:**

| Scope | Candidate Set |
|---|---|
| `Trending { window_hours }` | Items with high global velocity in window |
| `CohortTrending { cohort, window_hours }` | Items with high velocity among cohort members |
| `Following` | Items from followed creators (requires `for_user`) |
| `Category { name }` | Items in a category |
| `Collection { id }` | Items in a collection |

### SUGGEST — Autocomplete and Suggestions

```rust
use tidaldb::query::suggest::Suggest;
use tidaldb::schema::EntityId;

// Autocomplete on partial query.
let req = Suggest { prefix: "jazz pia".into(), for_user: None, limit: 5 };
let suggestions = db.suggest(&req)?;
// Returns Vec<Suggestion> with text and frequency.

// Personalized autocomplete.
let req = Suggest { prefix: "jazz pia".into(), for_user: Some(EntityId::new(123)), limit: 5 };
let suggestions = db.suggest(&req)?;

// Trending searches (empty prefix).
let req = Suggest { prefix: "".into(), for_user: None, limit: 10 };
let trending = db.suggest(&req)?;
```

Note: `for_user` is `Option<EntityId>`, not `Option<u64>`.

---

## Filters

All filters are composable. Any combination of filters produces a valid, efficiently-executed query. Filters use the `FilterExpr` type from `tidaldb::storage::indexes::filter::FilterExpr`.

### Content Attribute Filters

```rust
use tidaldb::storage::indexes::filter::FilterExpr;

FilterExpr::eq("category", "jazz")           // exact match on category
FilterExpr::eq("format", "video")            // exact match on format
FilterExpr::Tag("tutorial".to_string())      // tag match
FilterExpr::CreatorEq(100)                   // exact match on creator ID
FilterExpr::DurationMin(60)                  // minimum duration (seconds)
FilterExpr::DurationMax(600)                 // maximum duration (seconds)
FilterExpr::CreatedAfter(ts_nanos)           // created after timestamp (nanoseconds)
FilterExpr::CreatedBefore(ts_nanos)          // created before timestamp (nanoseconds)
```

Note: `FilterExpr::eq()` only routes `"category"` and `"format"` to typed variants. For tags, use `FilterExpr::Tag(...)` directly.

### Engagement Threshold Filters

```rust
FilterExpr::MinSignal { signal: "view".into(), threshold: 10000.0 }
FilterExpr::MaxSignal { signal: "view".into(), threshold: 5000.0 }
```

### Geographic Filters

```rust
FilterExpr::NearLocation { lat: 40.7128, lng: -74.0060, radius_km: 50.0 }
```

### Collection Filters

```rust
use tidaldb::entities::CollectionId;

FilterExpr::InCollection(CollectionId::new(42))
```

See [USE_CASES.md Appendix A](USE_CASES.md#appendix-a--filter-reference) for the complete filter reference.

---

## Sort Modes

Sort modes are embedded in ranking profiles. The application names a profile. The database executes the ranking pipeline. 25 built-in profiles cover the most common sort needs.

| Profile | Sort Mode |
|---|---|
| `new` | `created_at` DESC |
| `trending` | Engagement velocity |
| `hot` | Score / (age + 2)^gravity |
| `top_week` / `top_month` / `top_all_time` | Cumulative quality by window |
| `most_viewed` / `most_liked` | Signal count by window |
| `most_commented` / `most_shared` | Signal count (AllTime) |
| `hidden_gems` | High quality, low reach |
| `controversial` | max(positive * negative signals) |
| `shuffle` | Random, quality-weighted |
| `live` | Live viewer count DESC |
| `date_saved` | When user bookmarked DESC |
| `alphabetical_asc` / `alphabetical_desc` | Title A-Z / Z-A |
| `shortest` / `longest` | Duration ASC / DESC |

See [USE_CASES.md Appendix B](USE_CASES.md#appendix-b--sort-mode-reference) for the complete sort mode reference.

---

## Diversity Constraints

Diversity is a post-scoring pass. After candidates are scored, diversity constraints reorder the result set to enforce variety -- without reducing the result count.

```rust
use tidaldb::ranking::diversity::DiversityConstraints;

let diversity = DiversityConstraints::new()
    .max_per_creator(2)   // No more than 2 items per creator
    .format_mix(0.4);     // No format > 40% of results

let query = Retrieve::builder()
    .profile("for_you")
    .for_user(123)
    .diversity(diversity)
    .limit(50)
    .build()?;
```

Diversity is specified per query or per ranking profile. Query-level diversity overrides the profile default.

---

## Pagination

Cursor-based pagination for stable result sets across pages.

```rust
use tidaldb::query::retrieve::Retrieve;

// First page.
let query = Retrieve::builder()
    .for_user(123)
    .profile("for_you")
    .limit(50)
    .build()?;
let page1 = db.retrieve(&query)?;

// Next page -- pass the cursor from the previous response.
if let Some(cursor) = page1.next_cursor {
    let query = Retrieve::builder()
        .for_user(123)
        .profile("for_you")
        .cursor(cursor)
        .limit(50)
        .build()?;
    let page2 = db.retrieve(&query)?;
}
```

Alternatively, use `exclude` to exclude previously returned items:

```rust
let seen_ids: Vec<_> = page1.items.iter().map(|r| r.entity_id).collect();
let query = Retrieve::builder()
    .for_user(123)
    .profile("for_you")
    .exclude(seen_ids)
    .limit(50)
    .build()?;
let page2 = db.retrieve(&query)?;
```

---

## Response Format

### RETRIEVE Response

```rust
pub struct Results {
    /// Ranked items with scores.
    pub items: Vec<RetrieveResult>,
    /// Cursor for fetching the next page.
    pub next_cursor: Option<Cursor>,
    /// Total candidate count before diversity/limit.
    pub total_candidates: usize,
    /// Whether all diversity constraints were satisfied.
    pub constraints_satisfied: bool,
    /// Warnings generated during query execution.
    pub warnings: Vec<String>,
    /// Session snapshot at query time (populated when `for_session` is set).
    pub session_snapshot: Option<SessionSnapshot>,
    /// The degradation level under which this query was executed.
    pub degradation_level: DegradationLevel,
    /// Per-query execution statistics (timing, candidate counts, profile name).
    pub stats: QueryStats,
}

pub struct RetrieveResult {
    /// Entity ID.
    pub entity_id: EntityId,
    /// Normalized score in [0.0, 1.0].
    pub score: f64,
    /// 1-based rank.
    pub rank: usize,
    /// Signal values that contributed to this score.
    pub signals: Vec<Signal>,
}
```

### SEARCH Response

```rust
pub struct SearchResults {
    pub items: Vec<SearchResultItem>,
    pub next_cursor: Option<Cursor>,
    pub total_candidates: usize,
    pub constraints_satisfied: bool,
    pub warnings: Vec<String>,
    pub session_snapshot: Option<SessionSnapshot>,
    pub degradation_level: DegradationLevel,
    pub stats: QueryStats,
}

pub struct SearchResultItem {
    pub entity_id: EntityId,
    pub score: f64,
    pub rank: usize,
    pub bm25_score: Option<f32>,
    pub semantic_score: Option<f32>,
    pub signals: Vec<Signal>,
    pub metadata: Option<HashMap<String, String>>,
}
```

The application uses `items` to render the UI. It uses `signals` to display engagement counts (views, likes, etc.). It never re-ranks -- the order from tidalDB is the final order.

---

## Lifecycle and Operations

### Shutdown

```rust
// Graceful shutdown -- flushes WAL, checkpoints signal state, persists indexes.
db.close()?;
// Or equivalently:
db.shutdown()?;
```

### Health Check

```rust
db.health_check()?; // Returns Ok(()) if operational.
```

### Item Count

```rust
let count: u64 = db.item_count(); // Number of items in the universe bitmap.
```

### Reading Signal State

```rust
use tidaldb::schema::{EntityId, Window};

// Read decay score (applies lazy decay to current time).
let score: Option<f64> = db.read_decay_score(EntityId::new(1), "view", 0)?;

// Read windowed event count.
let count: u64 = db.read_windowed_count(EntityId::new(1), "view", Window::OneHour)?;

// Read velocity (events per second).
let velocity: f64 = db.read_velocity(EntityId::new(1), "view", Window::OneHour)?;
```

### Saved Searches

```rust
use tidaldb::schema::{EntityId, Timestamp};

// Save a search as a persistent feed.
db.save_search(EntityId::new(123), "Jazz tutorials", "jazz tutorial", None)?;

// Query a saved search for new results since a timestamp.
let results = db.retrieve_saved_search(EntityId::new(123), "Jazz tutorials", Some(since))?;

// List all saved searches for a user.
let searches = db.list_saved_searches(EntityId::new(123))?;

// Delete a saved search.
db.delete_saved_search(EntityId::new(123), "Jazz tutorials")?;
```

### Collections

```rust
use tidaldb::schema::EntityId;
use tidaldb::entities::collection::Visibility;

// Create a user collection (playlist, board, etc.)
let collection_id = db.create_collection(EntityId::new(123), "Jazz Favorites", Visibility::Private)?;

// Add an item to a collection.
db.add_to_collection(collection_id, EntityId::new(1))?;

// Remove an item from a collection.
db.remove_from_collection(collection_id, EntityId::new(1))?;

// List collections for a user.
let collections = db.list_collections(EntityId::new(123))?;
```

### Text Index Management

```rust
// Force a synchronous commit and reload of the text index.
// Useful in tests after writing items to make them immediately searchable.
db.flush_text_index()?;
db.flush_creator_text_index()?;

// Manual reload (for ephemeral mode).
db.reload_text_index()?;
```

---

## Summary

| Operation | What the Application Does | What tidalDB Does |
|---|---|---|
| **Ingest content** | Compute embedding, call `write_item_with_metadata` + `write_item_embedding` | Index text, insert vector, initialize signals, apply cold start |
| **Record engagement** | Call `signal` with event type | Update signal ledger, WAL-backed durability |
| **Record engagement with context** | Call `signal_with_context` with user/creator IDs | Update ledger + user preferences + interaction weights + cohort attribution |
| **Serve a feed** | Call `retrieve` with a profile name | Candidate retrieval, scoring, diversity enforcement, pagination |
| **Search** | Embed query, call `search` | BM25 + ANN + RRF fusion + personalization + diversity |
| **Handle cold start** | Nothing | Exploration budget, population priors -- automatic |
| **Handle negative signals** | Call `signal` with skip/hide | Preference decay, exclusion in future queries |
| **Scope trending by cohort** | Specify cohort name in retrieve query | Cohort-scoped signal aggregation, same ranking profile |
| **Search within scope** | Specify `within` on search query | Intersects text/vector retrieval with scoped candidate set |
| **HTTP write** | `POST /items`, `/embeddings`, `/signals` | Same as library write path, via JSON |
| **HTTP query** | `GET /feed`, `/search` | Same as library query, via query params |

One process. One query interface. One operational model.

---

## HTTP Server API

`tidal-server` is a standalone HTTP server wrapping the Rust library API. It exposes a REST interface for write, query, and health operations.

### Running the Server

```bash
# Ephemeral (in-memory) mode on default port 9400.
tidal-server standalone

# Persistent mode with custom schema.
tidal-server standalone --data-dir /var/lib/tidaldb --schema config/schema.yaml

# Custom port and metrics endpoint.
PORT=8080 tidal-server standalone --metrics 127.0.0.1:9091
```

| Flag / Env | Default | Description |
|---|---|---|
| `--data-dir <PATH>` | ephemeral | Persistent data directory |
| `--schema <PATH>` | bundled default | YAML schema file |
| `--metrics <ADDR>` | disabled | Bind Prometheus metrics server |
| `PORT` | `9400` | Listen port |
| `TIDAL_API_KEY` | unset (no auth) | Bearer token for protected routes |

### Authentication

When `TIDAL_API_KEY` is set, all write and query endpoints require a Bearer token:

```
Authorization: Bearer <your-api-key>
```

Health endpoints are always public. If the key is missing or invalid, the server returns:

```json
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer

{"error": "missing or invalid api key"}
```

### Middleware

Protected routes have these limits applied:

| Layer | Behavior |
|---|---|
| Body limit | 2 MB max request body (413 if exceeded) |
| Timeout | 30 second wall-clock limit (408 if exceeded) |
| Concurrency | 100 max in-flight requests (queued beyond that) |

Health endpoints are exempt from timeout and concurrency limits so probes are never dropped under load.

### Write Endpoints

#### `POST /items`

Create an item with metadata.

```json
{
  "entity_id": 1,
  "metadata": {
    "title": "Introduction to Jazz Piano",
    "category": "music",
    "tags": "jazz,piano,tutorial"
  }
}
```

**Response:** `201 Created` (no body)

#### `POST /embeddings`

Write an embedding vector for an item.

```json
{
  "entity_id": 1,
  "values": [0.1, 0.2, 0.3, ...]
}
```

**Response:** `204 No Content`

#### `POST /signals`

Record an engagement signal. `user_id` and `creator_id` are optional — when provided, the signal updates user preferences and interaction weights (equivalent to `signal_with_context` in the library API).

```json
{
  "entity_id": 1,
  "signal": "view",
  "weight": 1.0,
  "user_id": 123,
  "creator_id": 100
}
```

**Response:** `204 No Content`

### Query Endpoints

#### `GET /feed`

Retrieve a ranked feed.

| Parameter | Type | Default | Description |
|---|---|---|---|
| `user_id` | `u64` | — | User for personalization (optional) |
| `profile` | `string` | `"for_you"` | Ranking profile name |
| `limit` | `u32` | `20` | Max results (clamped to 1000) |
| `similar_to` | `u64` | — | Seed item for "more like this": with `profile=related`, sources candidates by ANN nearest-neighbor over this item's embedding |
| `region` | `string` | — | Target region (cluster mode only; rejected with `400` in standalone) |

```
GET /feed?user_id=123&profile=trending&limit=25
GET /feed?profile=related&similar_to=42&limit=20
```

**Response:**

```json
{
  "items": [
    {
      "entity_id": 42,
      "score": 0.95,
      "rank": 1,
      "signals": [
        {"name": "view", "value": 1523.0},
        {"name": "like", "value": 89.0}
      ]
    }
  ],
  "total_candidates": 1000,
  "region": null,
  "unavailable_shards": ["group-2"]
}
```

The `signals` field is omitted when empty. `region` is the region the feed was
served from in cluster mode (`null` standalone). `unavailable_shards` is present
**only** when a cross-shard read was degraded — it lists the shard groups that
could not be reached, so a partial page (fewer items, lower `total_candidates`)
is never silently indistinguishable from a complete one. It is omitted on a
complete read.

#### `GET /search`

Text search with optional personalization.

| Parameter | Type | Default | Description |
|---|---|---|---|
| `query` | `string` | — | Search text (**required**) |
| `user_id` | `u64` | — | User for personalization (optional) |
| `limit` | `u32` | `20` | Max results (clamped to 1000) |
| `region` | `string` | — | Target region (cluster mode only; rejected with `400` in standalone) |

```
GET /search?query=jazz+piano&user_id=123&limit=10
```

**Response:**

```json
{
  "items": [
    {
      "entity_id": 42,
      "score": 0.88,
      "rank": 1,
      "bm25_score": 12.5,
      "semantic_score": 0.92
    }
  ],
  "total_candidates": 50,
  "region": null,
  "unavailable_shards": ["group-2"]
}
```

`bm25_score` and `semantic_score` are omitted when not applicable. `region` and
`unavailable_shards` behave exactly as on [`GET /feed`](#get-feed): `region` is
the serving region in cluster mode (`null` standalone), and `unavailable_shards`
is present only when a cross-shard read was degraded (partial page) and omitted
on a complete read.

#### `POST /vector_search`

Pure k-NN ANN probe over the item content vector slot — **no** profile scoring,
fusion, or diversity. This is the recall-measurement / raw nearest-neighbor
surface (used by the `tidal-stress --verify-recall` harness).

**Request body:**

```json
{
  "vector": [0.013, -0.41, 0.22],
  "k": 10,
  "ef_search": 200
}
```

| Field | Type | Default | Description |
|---|---|---|---|
| `vector` | `f32[]` | — | Dense query vector; must match the item content slot's dimensionality (**required**) |
| `k` | `u32` | `10` | Number of nearest neighbors to return (clamped to 1000) |
| `ef_search` | `u32` | — | Optional per-request HNSW beam-width override (the recall/latency knob); omitted = the slot's configured default |

**Response:**

```json
{
  "items": [
    {"entity_id": 42, "distance": 0.018},
    {"entity_id": 17, "distance": 0.041}
  ],
  "region": null,
  "unavailable_shards": ["group-2"]
}
```

Each match carries `entity_id` and `distance` (L2-squared distance from the query
vector, lower = more similar; for the L2-normalized vectors tidalDB stores this
lies in `[0.0, 4.0]` and is monotonic with cosine distance), ordered closest-first.
`region` is always `null` here (the probe serves locally — standalone, or merged
across a node's hosted shard groups in cluster mode). `unavailable_shards` is
present only when a cross-shard probe was degraded (partial nearest-set).

### Health Endpoints

| Endpoint | Auth | Description |
|---|---|---|
| `GET /health` | No | Readiness probe — 200 when ready, 503 when shutting down |
| `GET /health/startup` | No | Startup probe — always 200 |
| `GET /health/live` | No | Liveness probe — always 200 |
| `GET /openapi.json` | No | Machine-readable OpenAPI 3.1 spec for this server (data + cluster routes) |

These map directly to Kubernetes startup/liveness/readiness probes — see
[docs/runbooks/kubernetes.md](docs/runbooks/kubernetes.md).

**`GET /openapi.json`** returns the canonical, served HTTP API contract. It is
generated from the handlers (so it never drifts from the code) and is exempt
from auth — clients need it to learn how to authenticate. Load it into any
OpenAPI viewer or generate a client:

```bash
curl -s http://localhost:9400/openapi.json | jq '.info, (.paths | keys)'
```

The cluster server serves a superset document that also includes the
`/cluster/*` and `/sharded/*` routes. See
[docs/guides/server-deployment.md](docs/guides/server-deployment.md).

### Cluster Endpoints

These routes exist **only on cluster (region) nodes** — they are absent from the
standalone server. Status routes are read-only; the rebalance verbs require an
admin API key. Full operational detail is in
[docs/guides/server-deployment.md](docs/guides/server-deployment.md) and
[docs/runbooks/kubernetes.md](docs/runbooks/kubernetes.md).

| Method | Path | Purpose |
|---|---|---|
| `GET` | `/cluster/status` | Aggregated cluster view — leader, relay high-water-mark, per-region lag, and a per-shard-group `shards` array |
| `GET` | `/cluster/status/local` | This node's local view — its hosted shard groups and which leaderships it holds |
| `POST` | `/cluster/shards/{id}/transfer` | Transfer shard group `{id}`'s leadership to a named replica (fenced promote; body `{"region": "<name>"}`) |
| `POST` | `/cluster/shards/{id}/replicas` | Add or remove a replica of shard group `{id}` (body `{"action": "add"\|"remove", "name": ..., "grpc_addr": ..., "http_addr": ...}`; addrs required for `add`) |

The cluster node also exposes the membership/recovery verbs `POST /cluster/join`,
`/cluster/promote`, `/cluster/members{,/remove}`, `/cluster/partition`, `/cluster/heal`,
`/cluster/catchup`, `/cluster/reseed`, and `/cluster/reconcile`, plus the sharded
data surface `POST /sharded/{items,embeddings,signals}` and `GET /sharded/{feed,search}`.

### Cluster-Node-Only Data Endpoints

| Method | Path | Purpose |
|---|---|---|
| `POST` | `/hardnegs` | Record a hide hard-negative for `(user, item)` (body `{"user_id": ..., "item_id": ...}`); `204 No Content`. Cluster-node-only — converges via the `/cluster/reconcile` LWW snapshot |

> `POST /vector_search` (documented above) is served by **both** the standalone
> and cluster servers.

**`GET /health` response:**

```json
{"ok": true, "service": "tidaldb", "mode": "standalone", "items": 1234}
```

During shutdown:

```json
HTTP/1.1 503 Service Unavailable

{"ok": false, "service": "tidaldb", "cause": "shutting down"}
```

### Error Responses

All errors return JSON with an `error` field:

```json
{"error": "description of what went wrong"}
```

| Condition | HTTP Status |
|---|---|
| Invalid input, bad schema | `400 Bad Request` |
| Missing/invalid API key | `401 Unauthorized` |
| Policy violation, expired session | `403 Forbidden` |
| Entity not found | `404 Not Found` |
| Request timeout | `408 Request Timeout` |
| Body too large | `413 Payload Too Large` |
| Backpressure, rate limited | `429 Too Many Requests` |
| Internal error | `500 Internal Server Error` |