tidaldb/docs/guides/embeddings.md

Wiring a Real Embedding Model into tidalDB

tidalDB does not generate embeddings. It stores them, indexes them into HNSW, and ranks over them. You bring the model; tidalDB owns retrieval and ranking. This guide shows how to wire a production embedding model — OpenAI, Cohere, or a local sentence-transformers / CLIP model — into the embedded engine and the HTTP server.

If you only need a runnable end-to-end demo with random vectors first, run cargo run -p tidaldb --example foryou_feed and read QUICKSTART.md. This guide is the next step: replacing those random vectors with a real model.

See also: QUICKSTART.md · API.md — Writing Embeddings · API.md — POST /embeddings · docs/guides/build-a-feed-app.md (companion: the full app around this).

---

1. The contract

tidalDB enforces a small, strict contract on every vector you write. Honoring it is not optional — violations are rejected at the write boundary, not silently coerced.

Rule	What tidalDB does	Failure mode
You generate, tidalDB stores	write_item_embedding / POST /embeddings accept a &[f32] you computed. No model runs inside the database.	—
L2 normalization is automatic	The vector is L2-normalized on write so that L2 distance on the HNSW index equals cosine distance. You may pass an un-normalized vector; tidalDB normalizes it.	—
Dimensions are strict: [2, 4096]	A slot must declare dimensions in [2, 4096]. Out-of-range slot declarations fail at schema.build().	InvalidEmbeddingDimensions { slot, dimensions } — "must be in [2, 4096]"
Vector length must equal the slot	Every inserted vector's .len() must equal the slot's declared dimensions exactly.	DimensionMismatch { expected, got } at insert
Zero-norm vectors are rejected	A vector whose squared sum is below f32::EPSILON (all zeros / underflow) has no direction and cannot do cosine similarity.	ZeroNormVector

Three more facts that shape your integration:

• Normalization is automatic, but you should still hand tidalDB a clean vector. Most embedding APIs already return unit-norm vectors (OpenAI and Cohere do). If yours does not, tidalDB will normalize it — but a zero vector will be rejected outright, so guard against degenerate output (empty input text, blank frames).
• Multi-slot routing caveat. A schema may declare several embedding slots, but RETRIEVE and SEARCH route ANN through the first declared Item slot only (resolved by item_embedding_slot(), defaulting to "content"). Multi-modal apps must fuse modalities offline into one vector per item, or model each modality as a separate entity kind. See §7 Failure modes.
• Personalization needs positive_engagement(true). The per-user preference vector that folds item embeddings into a user's taste is updated only for signals declared .positive_engagement(true) and written via signal_with_context. A view or skip that is not positive-engagement does not move the taste vector. See §4 The query side.

---

2. Declare an embedding slot sized to your model

The slot's dimensions must equal your model's output dimensionality. Pick the slot size before opening the database — it is baked into the on-disk schema fingerprint, and changing it later is a new vector space (see §6 Versioning).

Embedded (Rust)

use tidaldb::schema::{SchemaBuilder, EntityKind};

let mut s = SchemaBuilder::new();

// ... declare your signals first (see QUICKSTART.md) ...

// One Item content slot, sized to text-embedding-3-small (1536D).
let _ = s.embedding_slot("content", EntityKind::Item, 1536);

let schema = s.build()?; // fails here if dimensions are outside [2, 4096]

Common model dimensions:

Model	Native dims	Slot declaration
OpenAI text-embedding-3-small	1536	embedding_slot("content", EntityKind::Item, 1536)
OpenAI text-embedding-3-large	3072 (shortenable)	embedding_slot("content", EntityKind::Item, 3072) — or pass dimensions: 1024 to the API and declare 1024 (see §3)
Cohere embed-english-v3.0 / embed-multilingual-v3.0	1024	embedding_slot("content", EntityKind::Item, 1024)
sentence-transformers/all-MiniLM-L6-v2	384	embedding_slot("content", EntityKind::Item, 384)
sentence-transformers/all-mpnet-base-v2	768	embedding_slot("content", EntityKind::Item, 768)
OpenAI CLIP ViT-B/32 (image/frame)	512	embedding_slot("content", EntityKind::Item, 512)

The 3072D of text-embedding-3-large is within the [2, 4096] ceiling, so it works directly. Larger models do not fit — if you need them, project down to ≤4096 (most have a documented dimensions/Matryoshka truncation parameter; re-normalize after).

HTTP server (config YAML)

The server loads the same schema from YAML via --schema:

embedding_slots:
  - name: content
    entity: item
    dimensions: 1536   # must match your model and be in [2, 4096]

Start the server pointed at it:

tidal-server standalone \
  --listen 0.0.0.0:9400 \
  --schema ./schema.yaml \
  --data-dir /var/lib/myapp/tidaldb \
  --metrics 127.0.0.1:9091

Set TIDAL_API_KEY to require Authorization: Bearer <key> on the data routes. If it is unset the server runs unauthenticated and logs a WARN. The --metrics endpoint is always unauthenticated — bind it to loopback / cluster-internal only.

---

3. Worked wiring per model

The shape is identical for every model: your app calls its model, then hands the vector to tidalDB. Below, db is an Arc<TidalDb> opened against a schema whose content slot matches the model's dimensions.

OpenAI text-embedding-3-small (1536D)

use std::collections::HashMap;
use tidaldb::schema::{EntityId, Timestamp};

// Your model client. tidalDB never sees your API key or HTTP client.
async fn embed_openai(text: &str) -> anyhow::Result<Vec<f32>> {
    let resp: serde_json::Value = reqwest::Client::new()
        .post("https://api.openai.com/v1/embeddings")
        .bearer_auth(std::env::var("OPENAI_API_KEY")?)
        .json(&serde_json::json!({
            "model": "text-embedding-3-small",
            "input": text,
        }))
        .send().await?
        .json().await?;
    let v: Vec<f32> = resp["data"][0]["embedding"]
        .as_array().ok_or_else(|| anyhow::anyhow!("no embedding"))?
        .iter().map(|x| x.as_f64().unwrap_or(0.0) as f32).collect();
    anyhow::ensure!(v.len() == 1536, "expected 1536D, got {}", v.len());
    Ok(v)
}

// Ingest one item: metadata first, then its embedding.
async fn ingest(db: &tidaldb::TidalDb, id: u64, title: &str) -> anyhow::Result<()> {
    let mut meta = HashMap::new();
    meta.insert("title".into(), title.to_string());
    meta.insert("created_at".into(), Timestamp::now().as_nanos().to_string());
    db.write_item_with_metadata(EntityId::new(id), &meta)?;

    let vec = embed_openai(title).await?;     // app computes the vector
    db.write_item_embedding(EntityId::new(id), &vec)?; // tidalDB stores + indexes it
    Ok(())
}

curl, against the HTTP server:

# 1. Compute the vector with your model (here, OpenAI).
VEC=$(curl -s https://api.openai.com/v1/embeddings \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"text-embedding-3-small","input":"Introduction to Jazz Piano"}' \
  | jq -c '.data[0].embedding')

# 2. Write metadata, then the embedding, to tidalDB.
curl -s -X POST http://localhost:9400/items \
  -H "Authorization: Bearer $TIDAL_API_KEY" -H "Content-Type: application/json" \
  -d '{"entity_id":1,"metadata":{"title":"Introduction to Jazz Piano"}}'

curl -s -X POST http://localhost:9400/embeddings \
  -H "Authorization: Bearer $TIDAL_API_KEY" -H "Content-Type: application/json" \
  -d "{\"entity_id\":1,\"values\":$VEC}"   # 204 No Content on success

OpenAI text-embedding-3-large (3072D, shortenable)

text-embedding-3-large is 3072D natively — within the [2, 4096] ceiling, so it works directly with a dimensions: 3072 slot. It is also Matryoshka-shortenable: pass the dimensions parameter to trade a little accuracy for a smaller, faster index. The slot must match whatever you choose — and once chosen, it is fixed for that slot (see §6).

async fn embed_openai_large(text: &str, dims: u32) -> anyhow::Result<Vec<f32>> {
    let resp: serde_json::Value = reqwest::Client::new()
        .post("https://api.openai.com/v1/embeddings")
        .bearer_auth(std::env::var("OPENAI_API_KEY")?)
        .json(&serde_json::json!({
            "model": "text-embedding-3-large",
            "input": text,
            "dimensions": dims,   // e.g. 1024 to shorten; omit for native 3072
        }))
        .send().await?.json().await?;
    let v: Vec<f32> = resp["data"][0]["embedding"].as_array()
        .ok_or_else(|| anyhow::anyhow!("no embedding"))?
        .iter().map(|x| x.as_f64().unwrap_or(0.0) as f32).collect();
    anyhow::ensure!(v.len() == dims as usize, "asked {dims}D, got {}", v.len());
    Ok(v)
}
// Slot must be declared with the SAME dims you pass here:
//   s.embedding_slot("content", EntityKind::Item, 1024);  // if dims = 1024
//   s.embedding_slot("content", EntityKind::Item, 3072);  // if native

Cohere embed-english-v3.0 (1024D)

Cohere distinguishes input_type between indexing documents and embedding queries — use search_document when ingesting items and search_query at query time. Both must be the same model.

async fn embed_cohere(text: &str, input_type: &str) -> anyhow::Result<Vec<f32>> {
    let resp: serde_json::Value = reqwest::Client::new()
        .post("https://api.cohere.com/v1/embed")
        .bearer_auth(std::env::var("COHERE_API_KEY")?)
        .json(&serde_json::json!({
            "model": "embed-english-v3.0",
            "texts": [text],
            "input_type": input_type,        // "search_document" for items
            "embedding_types": ["float"],
        }))
        .send().await?.json().await?;
    let v: Vec<f32> = resp["embeddings"]["float"][0].as_array()
        .ok_or_else(|| anyhow::anyhow!("no embedding"))?
        .iter().map(|x| x.as_f64().unwrap_or(0.0) as f32).collect();
    anyhow::ensure!(v.len() == 1024, "expected 1024D, got {}", v.len());
    Ok(v)
}

// At ingest:
let vec = embed_cohere(title, "search_document").await?;
db.write_item_embedding(EntityId::new(id), &vec)?;

Local model — sentence-transformers (text) or CLIP (video frames)

A local model is the same contract: produce a Vec<f32>, write it. Run the model in your own service (Python, ONNX Runtime, Candle, etc.) and pass tidalDB the result.

sentence-transformers (384D MiniLM): your embedding service exposes, say, POST /embed -> {"vector": [...]}. The tidalDB integration is just:

let vec: Vec<f32> = my_embed_service.embed(title).await?; // 384 floats
db.write_item_embedding(EntityId::new(id), &vec)?;        // slot declared 384D

CLIP for video frames (512D): a short-video item has no single text to embed — embed the content directly. tidalDB stores one vector per item, so fuse frames into one vector before writing (sample N frames, CLIP each, mean-pool, then write):

// In your encoder service (pseudocode of the offline step):
//   frames  = sample_keyframes(video, n = 8)
//   vecs    = frames.map(clip_image_encode)   // each 512D
//   fused   = mean_pool(vecs)                  // one 512D vector
// Hand the fused vector to tidalDB:
let fused: Vec<f32> = encoder.embed_video(&video_id).await?; // 512 floats, non-zero
db.write_item_embedding(EntityId::new(id), &fused)?;          // slot declared 512D

Guard against degenerate output: a blank frame or empty caption can yield an all-zero vector, which tidalDB rejects with ZeroNormVector. Skip or backfill those items rather than writing a zero.

curl is identical regardless of which model produced the vector — the server only sees floats:

curl -s -X POST http://localhost:9400/embeddings \
  -H "Authorization: Bearer $TIDAL_API_KEY" -H "Content-Type: application/json" \
  -d '{"entity_id":7,"values":[0.013,-0.044, ... 512 floats ...]}'

---

4. The query side

The query vector MUST come from the same model and version as the item vectors. A vector from a different model lives in a different geometric space; cosine similarity across spaces is meaningless. There is no cross-space conversion — re-embed the query text/image with the exact model you indexed with.

Hybrid search — Search::builder().vector(...)

SEARCH fuses BM25 text relevance with ANN semantic similarity (Reciprocal Rank Fusion). Provide both the query text and a query vector you computed:

use tidaldb::query::search::Search;

// SAME model as the items. For Cohere, use input_type = "search_query" here.
let qvec: Vec<f32> = embed_openai("relaxing jazz piano").await?; // 1536D, matches slot

let q = Search::builder()
    .query("relaxing jazz piano")   // BM25 side
    .vector(qvec)                   // ANN side — same model/version as items
    .for_user(42)                   // personalization (optional)
    .limit(20)
    .build()?;
let results = db.search(&q)?;

curl (hybrid): compute the query vector with your model, then pass it on the search. The HTTP GET /search route takes query, user_id, and limit; for a query vector, use the library API or the POST-bodied search if your deployment exposes one. Text-only search is always available:

curl -s "http://localhost:9400/search?query=relaxing%20jazz%20piano&user_id=42&limit=20" \
  -H "Authorization: Bearer $TIDAL_API_KEY"

More like this — .similar_to(item_id)

To find items near an existing item, you do not need to embed anything — tidalDB uses the stored vector of item_id as the query, through the first content slot:

let q = Search::builder()
    .similar_to(EntityId::new(1))   // "more like item 1"
    .for_user(42)
    .exclude(vec![EntityId::new(1)]) // don't return the seed item
    .limit(20)
    .build()?;
let results = db.search(&q)?;

Search::builder() also takes .diversity(DiversityConstraints { max_per_creator, format_mix_max_fraction }) and .exclude([ids]). See API.md — SEARCH.

Pure feeds use Retrieve (profile("for_you"), etc.), which leans on signals and the per-user preference vector rather than an ad-hoc query vector. The preference vector is built from item embeddings folded in by positive_engagement(true) signals — so the query side of personalization is implicitly "the same model as your items."

---

5. Batching 1M items

The write path is one vector per call, and re-writes are idempotent — write_item_embedding overwrites the stored vector and re-inserts into the index, so re-running a batch is safe (use it for resumable backfills). Batch the model calls (that is where the latency and cost are), then write each result individually.

use tidaldb::schema::EntityId;

// Embedding APIs accept many inputs per request — batch the model, not the DB write.
for chunk in items.chunks(256) {
    // One model call for up to 256 inputs (OpenAI/Cohere accept arrays).
    let vectors: Vec<Vec<f32>> =
        embed_batch_openai(chunk.iter().map(|i| i.title.as_str())).await?;

    for (item, vec) in chunk.iter().zip(vectors) {
        // Length is checked here against the slot — a bad chunk fails fast.
        db.write_item_embedding(EntityId::new(item.id), &vec)?; // idempotent
    }
}
db.flush_text_index()?; // make freshly written items searchable (ephemeral mode)

Operational notes for a 1M backfill:

• Resumability. Track the last successfully written id. On restart, re-run from there — already-written vectors overwrite harmlessly. No dedup bookkeeping needed.
• Index promotion is automatic. A cold slot uses an exact brute-force index; once a slot's durable vector count crosses the internal threshold, the next process restart promotes it to HNSW (UsearchIndex). No config flag — just let the backfill complete and restart.
• Cost. The model calls dominate cost and wall-clock. tidalDB's per-vector write is cheap; size your batch to the model API's limits, not the database's.
• HTTP path. Over the server, the same shape: batch your model calls, then fire one POST /embeddings per item ({entity_id, values} → 204). Mind the server's 2MB body limit (413), 100 max in-flight (429), and 30s timeout (408) — a single 3072D vector is well under 2MB, but do not pack many vectors into one body.

---

6. Model versioning

A new embedding model is a new vector space. You cannot mix vectors from two models in one slot — their geometry is incomparable, and ANN results would be silently wrong (no error, just nonsense neighbors). The slot's dimensions are part of the on-disk schema fingerprint, so even a dimension change is a hard break: opening an existing data directory with a different slot dimension fails schema validation.

When you upgrade or switch models, choose one of:

1. Re-embed everything (clean cut). Re-run the full backfill (§5) with the new model, overwriting every item's vector in the same slot, then switch query-side embedding to the new model atomically. Simplest; requires one full re-embed pass and a brief window where new and old must not be queried interleaved. If dimensions changed, you also need a fresh data directory (or a slot of the new size).

2. Separate slot / entity kind (parallel spaces). Declare a second embedding slot (or model items as a distinct entity kind) for the new model, backfill it alongside the old, and cut queries over once it is fully populated. This lets old and new coexist during migration. Caveat: RETRIEVE/SEARCH ANN routes through the first declared Item slot only — so the new slot only takes effect for default queries once it is the first slot (i.e. on a schema that declares it first). Until then, query the new space explicitly via your own search path. Plan the cutover schema so the slot you want serving default queries is declared first.

Never write a v2 vector into a v1 slot "to save a re-embed." There is no compatibility between spaces, and tidalDB cannot detect the mistake — the dimensions may even match.

---

7. Failure modes

Symptom	Cause	Fix
Insert error DimensionMismatch { expected, got }	Vector length ≠ slot dimensions.	Make the model output match the slot, or re-declare the slot. A truncated/padded vector is wrong even if it inserts — fix the model call.
schema.build() fails: "invalid dimensions … must be in [2, 4096]"	Slot declared outside [2, 4096].	Pick a model ≤4096D, or project the vector down (Matryoshka dimensions param) and re-normalize before writing.
Insert error ZeroNormVector	All-zero / underflowing vector (blank input text, empty frame, model error returning zeros).	Guard the encoder; skip or backfill degenerate items. Never write a zero placeholder.
ANN returns nonsense neighbors, no error	Query vector from a different model/version than the items, or v2 vectors written into a v1 slot.	Re-embed the query with the exact item model (§4); never mix models in a slot (§6).
Multi-modal app: image search ignores the text slot (or vice-versa)	RETRIEVE/SEARCH route ANN through the first declared Item slot only.	Fuse modalities offline into one vector per item, or model each modality as a separate entity kind and query it explicitly. Plan slot order for the cutover.
for_you doesn't reflect a user's taste despite engagement	The engaged signal is not positive_engagement(true), or you wrote signal instead of signal_with_context.	Declare taste-bearing signals (completion, like, replay) .positive_engagement(true) and write them via signal_with_context(.., Some(user_id), ..).

---

What tidalDB owns vs. what you own

tidalDB owns retrieval and ranking over the vectors you give it: the HNSW index, L2 normalization, signal-driven scoring, per-user preference vectors, diversity, and the feed ordering — one process, one query.

You own everything upstream: embedding generation (running the model), the model's API keys and clients, video/blob storage, CDN/transcoding, auth, moderation, and the player UI. tidalDB never generates embeddings and never sees your model.

To build the full application around this — schema design, the signal loop, feeds, and the HTTP surface — see docs/guides/build-a-feed-app.md, the runnable foryou_feed example, and QUICKSTART.md.