tidaldb/docs/guides/embeddings.md

# Wiring a Real Embedding Model into tidalDB

tidalDB does **not** generate embeddings. It stores them, indexes them into HNSW,
and ranks over them. You bring the model; tidalDB owns retrieval and ranking. This
guide shows how to wire a production embedding model — OpenAI, Cohere, or a local
sentence-transformers / CLIP model — into the embedded engine and the HTTP server.

> If you only need a runnable end-to-end demo with random vectors first, run
> `cargo run -p tidaldb --example foryou_feed` and read [QUICKSTART.md](../../QUICKSTART.md).
> This guide is the next step: replacing those random vectors with a real model.

**See also:** [QUICKSTART.md](../../QUICKSTART.md) · [API.md — Writing Embeddings](../../API.md#writing-embeddings) · [API.md — `POST /embeddings`](../../API.md#post-embeddings) · [docs/guides/build-a-feed-app.md](build-a-feed-app.md) (companion: the full app around this).

---

## 1. The contract

tidalDB enforces a small, strict contract on every vector you write. Honoring it is
not optional — violations are rejected at the write boundary, not silently coerced.

| Rule | What tidalDB does | Failure mode |
|------|-------------------|--------------|
| **You generate, tidalDB stores** | `write_item_embedding` / `POST /embeddings` accept a `&[f32]` you computed. No model runs inside the database. | — |
| **L2 normalization is automatic** | The vector is L2-normalized on write so that L2 distance on the HNSW index equals cosine distance. You may pass an un-normalized vector; tidalDB normalizes it. | — |
| **Dimensions are strict: `[2, 4096]`** | A slot must declare dimensions in `[2, 4096]`. Out-of-range slot declarations fail at `schema.build()`. | `InvalidEmbeddingDimensions { slot, dimensions }` — "must be in [2, 4096]" |
| **Vector length must equal the slot** | Every inserted vector's `.len()` must equal the slot's declared dimensions exactly. | `DimensionMismatch { expected, got }` at insert |
| **Zero-norm vectors are rejected** | A vector whose squared sum is below `f32::EPSILON` (all zeros / underflow) has no direction and cannot do cosine similarity. | `ZeroNormVector` |

Three more facts that shape your integration:

- **Normalization is automatic, but you should still hand tidalDB a clean vector.**
  Most embedding APIs already return unit-norm vectors (OpenAI and Cohere do).
  If yours does not, tidalDB will normalize it — but a zero vector will be rejected
  outright, so guard against degenerate output (empty input text, blank frames).
- **Multi-slot routing caveat.** A schema may declare several embedding slots, but
  `RETRIEVE` and `SEARCH` route ANN through the **first declared Item slot only**
  (resolved by `item_embedding_slot()`, defaulting to `"content"`). Multi-modal apps
  must fuse modalities offline into one vector per item, or model each modality as a
  separate entity kind. See [§7 Failure modes](#7-failure-modes).
- **Personalization needs `positive_engagement(true)`.** The per-user preference
  vector that folds item embeddings into a user's taste is updated **only** for
  signals declared `.positive_engagement(true)` and written via `signal_with_context`.
  A `view` or `skip` that is not positive-engagement does not move the taste vector.
  See [§4 The query side](#4-the-query-side).

---

## 2. Declare an embedding slot sized to your model

The slot's dimensions must equal your model's output dimensionality. Pick the slot
size **before** opening the database — it is baked into the on-disk schema fingerprint,
and changing it later is a new vector space (see [§6 Versioning](#6-model-versioning)).

### Embedded (Rust)

```rust
use tidaldb::schema::{SchemaBuilder, EntityKind};

let mut s = SchemaBuilder::new();

// ... declare your signals first (see QUICKSTART.md) ...

// One Item content slot, sized to text-embedding-3-small (1536D).
let _ = s.embedding_slot("content", EntityKind::Item, 1536);

let schema = s.build()?; // fails here if dimensions are outside [2, 4096]
```

Common model dimensions:

| Model | Native dims | Slot declaration |
|-------|-------------|------------------|
| OpenAI `text-embedding-3-small` | 1536 | `embedding_slot("content", EntityKind::Item, 1536)` |
| OpenAI `text-embedding-3-large` | 3072 (shortenable) | `embedding_slot("content", EntityKind::Item, 3072)` — or pass `dimensions: 1024` to the API and declare `1024` (see §3) |
| Cohere `embed-english-v3.0` / `embed-multilingual-v3.0` | 1024 | `embedding_slot("content", EntityKind::Item, 1024)` |
| `sentence-transformers/all-MiniLM-L6-v2` | 384 | `embedding_slot("content", EntityKind::Item, 384)` |
| `sentence-transformers/all-mpnet-base-v2` | 768 | `embedding_slot("content", EntityKind::Item, 768)` |
| OpenAI CLIP `ViT-B/32` (image/frame) | 512 | `embedding_slot("content", EntityKind::Item, 512)` |

> The 3072D of `text-embedding-3-large` is within the `[2, 4096]` ceiling, so it works
> directly. Larger models do not fit — if you need them, project down to ≤4096 (most
> have a documented `dimensions`/Matryoshka truncation parameter; re-normalize after).

### HTTP server (config YAML)

The server loads the same schema from YAML via `--schema`:

```yaml
embedding_slots:
  - name: content
    entity: item
    dimensions: 1536   # must match your model and be in [2, 4096]
```

Start the server pointed at it:

```bash
tidal-server standalone \
  --listen 0.0.0.0:9400 \
  --schema ./schema.yaml \
  --data-dir /var/lib/myapp/tidaldb \
  --metrics 127.0.0.1:9091
```

> Set `TIDAL_API_KEY` to require `Authorization: Bearer <key>` on the data routes.
> If it is unset the server runs **unauthenticated** and logs a WARN. The `--metrics`
> endpoint is always unauthenticated — bind it to loopback / cluster-internal only.

---

## 3. Worked wiring per model

The shape is identical for every model: **your app calls its model, then hands the
vector to tidalDB.** Below, `db` is an `Arc<TidalDb>` opened against a schema whose
`content` slot matches the model's dimensions.

### OpenAI `text-embedding-3-small` (1536D)

```rust
use std::collections::HashMap;
use tidaldb::schema::{EntityId, Timestamp};

// Your model client. tidalDB never sees your API key or HTTP client.
async fn embed_openai(text: &str) -> anyhow::Result<Vec<f32>> {
    let resp: serde_json::Value = reqwest::Client::new()
        .post("https://api.openai.com/v1/embeddings")
        .bearer_auth(std::env::var("OPENAI_API_KEY")?)
        .json(&serde_json::json!({
            "model": "text-embedding-3-small",
            "input": text,
        }))
        .send().await?
        .json().await?;
    let v: Vec<f32> = resp["data"][0]["embedding"]
        .as_array().ok_or_else(|| anyhow::anyhow!("no embedding"))?
        .iter().map(|x| x.as_f64().unwrap_or(0.0) as f32).collect();
    anyhow::ensure!(v.len() == 1536, "expected 1536D, got {}", v.len());
    Ok(v)
}

// Ingest one item: metadata first, then its embedding.
async fn ingest(db: &tidaldb::TidalDb, id: u64, title: &str) -> anyhow::Result<()> {
    let mut meta = HashMap::new();
    meta.insert("title".into(), title.to_string());
    meta.insert("created_at".into(), Timestamp::now().as_nanos().to_string());
    db.write_item_with_metadata(EntityId::new(id), &meta)?;

    let vec = embed_openai(title).await?;     // app computes the vector
    db.write_item_embedding(EntityId::new(id), &vec)?; // tidalDB stores + indexes it
    Ok(())
}
```

curl, against the HTTP server:

```bash
# 1. Compute the vector with your model (here, OpenAI).
VEC=$(curl -s https://api.openai.com/v1/embeddings \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"text-embedding-3-small","input":"Introduction to Jazz Piano"}' \
  | jq -c '.data[0].embedding')

# 2. Write metadata, then the embedding, to tidalDB.
curl -s -X POST http://localhost:9400/items \
  -H "Authorization: Bearer $TIDAL_API_KEY" -H "Content-Type: application/json" \
  -d '{"entity_id":1,"metadata":{"title":"Introduction to Jazz Piano"}}'

curl -s -X POST http://localhost:9400/embeddings \
  -H "Authorization: Bearer $TIDAL_API_KEY" -H "Content-Type: application/json" \
  -d "{\"entity_id\":1,\"values\":$VEC}"   # 204 No Content on success
```

### OpenAI `text-embedding-3-large` (3072D, shortenable)

`text-embedding-3-large` is 3072D natively — within the `[2, 4096]` ceiling, so it
works directly with a `dimensions: 3072` slot. It is also **Matryoshka-shortenable**:
pass the `dimensions` parameter to trade a little accuracy for a smaller, faster index.
**The slot must match whatever you choose** — and once chosen, it is fixed for that
slot (see [§6](#6-model-versioning)).

```rust
async fn embed_openai_large(text: &str, dims: u32) -> anyhow::Result<Vec<f32>> {
    let resp: serde_json::Value = reqwest::Client::new()
        .post("https://api.openai.com/v1/embeddings")
        .bearer_auth(std::env::var("OPENAI_API_KEY")?)
        .json(&serde_json::json!({
            "model": "text-embedding-3-large",
            "input": text,
            "dimensions": dims,   // e.g. 1024 to shorten; omit for native 3072
        }))
        .send().await?.json().await?;
    let v: Vec<f32> = resp["data"][0]["embedding"].as_array()
        .ok_or_else(|| anyhow::anyhow!("no embedding"))?
        .iter().map(|x| x.as_f64().unwrap_or(0.0) as f32).collect();
    anyhow::ensure!(v.len() == dims as usize, "asked {dims}D, got {}", v.len());
    Ok(v)
}
// Slot must be declared with the SAME dims you pass here:
//   s.embedding_slot("content", EntityKind::Item, 1024);  // if dims = 1024
//   s.embedding_slot("content", EntityKind::Item, 3072);  // if native
```

### Cohere `embed-english-v3.0` (1024D)

Cohere distinguishes `input_type` between indexing documents and embedding queries —
use `search_document` when ingesting items and `search_query` at query time. **Both
must be the same model.**

```rust
async fn embed_cohere(text: &str, input_type: &str) -> anyhow::Result<Vec<f32>> {
    let resp: serde_json::Value = reqwest::Client::new()
        .post("https://api.cohere.com/v1/embed")
        .bearer_auth(std::env::var("COHERE_API_KEY")?)
        .json(&serde_json::json!({
            "model": "embed-english-v3.0",
            "texts": [text],
            "input_type": input_type,        // "search_document" for items
            "embedding_types": ["float"],
        }))
        .send().await?.json().await?;
    let v: Vec<f32> = resp["embeddings"]["float"][0].as_array()
        .ok_or_else(|| anyhow::anyhow!("no embedding"))?
        .iter().map(|x| x.as_f64().unwrap_or(0.0) as f32).collect();
    anyhow::ensure!(v.len() == 1024, "expected 1024D, got {}", v.len());
    Ok(v)
}

// At ingest:
let vec = embed_cohere(title, "search_document").await?;
db.write_item_embedding(EntityId::new(id), &vec)?;
```

### Local model — sentence-transformers (text) or CLIP (video frames)

A local model is the same contract: produce a `Vec<f32>`, write it. Run the model in
your own service (Python, ONNX Runtime, Candle, etc.) and pass tidalDB the result.

**sentence-transformers (384D MiniLM):** your embedding service exposes, say,
`POST /embed -> {"vector": [...]}`. The tidalDB integration is just:

```rust
let vec: Vec<f32> = my_embed_service.embed(title).await?; // 384 floats
db.write_item_embedding(EntityId::new(id), &vec)?;        // slot declared 384D
```

**CLIP for video frames (512D):** a short-video item has no single text to embed —
embed the content directly. tidalDB stores one vector per item, so fuse frames into
one vector before writing (sample N frames, CLIP each, mean-pool, then write):

```rust
// In your encoder service (pseudocode of the offline step):
//   frames  = sample_keyframes(video, n = 8)
//   vecs    = frames.map(clip_image_encode)   // each 512D
//   fused   = mean_pool(vecs)                  // one 512D vector
// Hand the fused vector to tidalDB:
let fused: Vec<f32> = encoder.embed_video(&video_id).await?; // 512 floats, non-zero
db.write_item_embedding(EntityId::new(id), &fused)?;          // slot declared 512D
```

> Guard against degenerate output: a blank frame or empty caption can yield an
> all-zero vector, which tidalDB rejects with `ZeroNormVector`. Skip or backfill those
> items rather than writing a zero.

curl is identical regardless of which model produced the vector — the server only sees
floats:

```bash
curl -s -X POST http://localhost:9400/embeddings \
  -H "Authorization: Bearer $TIDAL_API_KEY" -H "Content-Type: application/json" \
  -d '{"entity_id":7,"values":[0.013,-0.044, ... 512 floats ...]}'
```

---

## 4. The query side

**The query vector MUST come from the same model and version as the item vectors.**
A vector from a different model lives in a different geometric space; cosine similarity
across spaces is meaningless. There is no cross-space conversion — re-embed the query
text/image with the exact model you indexed with.

### Hybrid search — `Search::builder().vector(...)`

`SEARCH` fuses BM25 text relevance with ANN semantic similarity (Reciprocal Rank
Fusion). Provide both the query text and a query vector you computed:

```rust
use tidaldb::query::search::Search;

// SAME model as the items. For Cohere, use input_type = "search_query" here.
let qvec: Vec<f32> = embed_openai("relaxing jazz piano").await?; // 1536D, matches slot

let q = Search::builder()
    .query("relaxing jazz piano")   // BM25 side
    .vector(qvec)                   // ANN side — same model/version as items
    .for_user(42)                   // personalization (optional)
    .limit(20)
    .build()?;
let results = db.search(&q)?;
```

curl (hybrid): compute the query vector with your model, then pass it on the search.
The HTTP `GET /search` route takes `query`, `user_id`, and `limit`; for a query vector,
use the library API or the `POST`-bodied search if your deployment exposes one. Text-only
search is always available:

```bash
curl -s "http://localhost:9400/search?query=relaxing%20jazz%20piano&user_id=42&limit=20" \
  -H "Authorization: Bearer $TIDAL_API_KEY"
```

### More like this — `.similar_to(item_id)`

To find items near an existing item, you do not need to embed anything — tidalDB uses
the stored vector of `item_id` as the query, through the first content slot:

```rust
let q = Search::builder()
    .similar_to(EntityId::new(1))   // "more like item 1"
    .for_user(42)
    .exclude(vec![EntityId::new(1)]) // don't return the seed item
    .limit(20)
    .build()?;
let results = db.search(&q)?;
```

`Search::builder()` also takes `.diversity(DiversityConstraints { max_per_creator,
format_mix_max_fraction })` and `.exclude([ids])`. See
[API.md — SEARCH](../../API.md#search--text--semantic-retrieval).

> Pure feeds use `Retrieve` (`profile("for_you")`, etc.), which leans on signals and
> the per-user preference vector rather than an ad-hoc query vector. The preference
> vector is built from item embeddings folded in by `positive_engagement(true)` signals
> — so the query side of personalization is implicitly "the same model as your items."

---

## 5. Batching 1M items

The write path is one vector per call, and **re-writes are idempotent** —
`write_item_embedding` overwrites the stored vector and re-inserts into the index, so
re-running a batch is safe (use it for resumable backfills). Batch the **model** calls
(that is where the latency and cost are), then write each result individually.

```rust
use tidaldb::schema::EntityId;

// Embedding APIs accept many inputs per request — batch the model, not the DB write.
for chunk in items.chunks(256) {
    // One model call for up to 256 inputs (OpenAI/Cohere accept arrays).
    let vectors: Vec<Vec<f32>> =
        embed_batch_openai(chunk.iter().map(|i| i.title.as_str())).await?;

    for (item, vec) in chunk.iter().zip(vectors) {
        // Length is checked here against the slot — a bad chunk fails fast.
        db.write_item_embedding(EntityId::new(item.id), &vec)?; // idempotent
    }
}
db.flush_text_index()?; // make freshly written items searchable (ephemeral mode)
```

Operational notes for a 1M backfill:

- **Resumability.** Track the last successfully written id. On restart, re-run from
  there — already-written vectors overwrite harmlessly. No dedup bookkeeping needed.
- **Index promotion is automatic.** A cold slot uses an exact brute-force index; once a
  slot's durable vector count crosses the internal threshold, the next process restart
  promotes it to HNSW (`UsearchIndex`). No config flag — just let the backfill complete
  and restart.
- **Cost.** The model calls dominate cost and wall-clock. tidalDB's per-vector write is
  cheap; size your batch to the model API's limits, not the database's.
- **HTTP path.** Over the server, the same shape: batch your model calls, then fire one
  `POST /embeddings` per item (`{entity_id, values}` → 204). Mind the server's 2MB body
  limit (413), 100 max in-flight (429), and 30s timeout (408) — a single 3072D vector is
  well under 2MB, but do not pack many vectors into one body.

---

## 6. Model versioning

**A new embedding model is a new vector space. You cannot mix vectors from two models
in one slot** — their geometry is incomparable, and ANN results would be silently wrong
(no error, just nonsense neighbors). The slot's dimensions are part of the on-disk
schema fingerprint, so even a dimension change is a hard break: opening an existing
data directory with a different slot dimension fails schema validation.

When you upgrade or switch models, choose one of:

1. **Re-embed everything (clean cut).** Re-run the full backfill (§5) with the new
   model, overwriting every item's vector in the same slot, then switch query-side
   embedding to the new model atomically. Simplest; requires one full re-embed pass and
   a brief window where new and old must not be queried interleaved. If dimensions
   changed, you also need a fresh data directory (or a slot of the new size).

2. **Separate slot / entity kind (parallel spaces).** Declare a second embedding slot
   (or model items as a distinct entity kind) for the new model, backfill it alongside
   the old, and cut queries over once it is fully populated. This lets old and new
   coexist during migration. **Caveat:** `RETRIEVE`/`SEARCH` ANN routes through the
   **first** declared Item slot only — so the new slot only takes effect for default
   queries once it is the first slot (i.e. on a schema that declares it first). Until
   then, query the new space explicitly via your own search path. Plan the cutover
   schema so the slot you want serving default queries is declared first.

Never write a v2 vector into a v1 slot "to save a re-embed." There is no compatibility
between spaces, and tidalDB cannot detect the mistake — the dimensions may even match.

---

## 7. Failure modes

| Symptom | Cause | Fix |
|---------|-------|-----|
| Insert error `DimensionMismatch { expected, got }` | Vector length ≠ slot dimensions. | Make the model output match the slot, or re-declare the slot. A truncated/padded vector is wrong even if it inserts — fix the model call. |
| `schema.build()` fails: "invalid dimensions … must be in [2, 4096]" | Slot declared outside `[2, 4096]`. | Pick a model ≤4096D, or project the vector down (Matryoshka `dimensions` param) and re-normalize before writing. |
| Insert error `ZeroNormVector` | All-zero / underflowing vector (blank input text, empty frame, model error returning zeros). | Guard the encoder; skip or backfill degenerate items. Never write a zero placeholder. |
| ANN returns nonsense neighbors, no error | Query vector from a different model/version than the items, **or** v2 vectors written into a v1 slot. | Re-embed the query with the exact item model (§4); never mix models in a slot (§6). |
| Multi-modal app: image search ignores the text slot (or vice-versa) | `RETRIEVE`/`SEARCH` route ANN through the **first** declared Item slot only. | Fuse modalities offline into one vector per item, or model each modality as a separate entity kind and query it explicitly. Plan slot order for the cutover. |
| `for_you` doesn't reflect a user's taste despite engagement | The engaged signal is not `positive_engagement(true)`, or you wrote `signal` instead of `signal_with_context`. | Declare taste-bearing signals (completion, like, replay) `.positive_engagement(true)` and write them via `signal_with_context(.., Some(user_id), ..)`. |

---

## What tidalDB owns vs. what you own

tidalDB owns **retrieval and ranking** over the vectors you give it: the HNSW index,
L2 normalization, signal-driven scoring, per-user preference vectors, diversity, and
the feed ordering — one process, one query.

You own **everything upstream**: embedding generation (running the model), the model's
API keys and clients, video/blob storage, CDN/transcoding, auth, moderation, and the
player UI. tidalDB never generates embeddings and never sees your model.

To build the full application around this — schema design, the signal loop, feeds, and
the HTTP surface — see [docs/guides/build-a-feed-app.md](build-a-feed-app.md), the
runnable [`foryou_feed` example](../../tidal/examples/foryou_feed.rs), and
[QUICKSTART.md](../../QUICKSTART.md).