- Add k8s/ manifests (StatefulSet, kustomize, PDB, ServiceMonitor) + docs/runbooks/kubernetes.md - Add tidal-server/src/openapi.rs (utoipa OpenAPI spec) and wire into router - Add docs/guides/ (build-a-feed-app, embeddings, server-deployment) + foryou_feed example - Consolidate tidal/docker/ into root docker/ (single canonical home) - Update API.md, QUICKSTART.md, README.md, CLAUDE.md, check-docs.sh accordingly
621 lines
28 KiB
Markdown
621 lines
28 KiB
Markdown
# Build a Short-Video "For You" Feed on tidalDB
|
|
|
|
This is an end-to-end tutorial for building a TikTok/Reels-style short-video feed —
|
|
a swipeable home feed ("For You"), a Following feed, and a Discover/Trending tab —
|
|
on top of tidalDB. You will model the signals a short-video app actually emits,
|
|
ingest videos, wire the swipe feedback loop so a user's taste updates *immediately*,
|
|
and serve all three feed surfaces. You will do it twice: once embedded in a Rust
|
|
process, and once over HTTP with `curl`.
|
|
|
|
There is a verified, runnable version of the embedded path in this repo:
|
|
|
|
```bash
|
|
cargo run -p tidaldb --example foryou_feed
|
|
```
|
|
|
|
That example ([`tidal/examples/foryou_feed.rs`](../../tidal/examples/foryou_feed.rs))
|
|
is the source of truth for every Rust call shown below — this guide reproduces and
|
|
explains its key calls. Skim it after reading section 6.
|
|
|
|
**New to tidalDB?** Read [QUICKSTART.md](../../QUICKSTART.md) first for the 5-minute
|
|
schema → ingest → rank loop. This guide assumes you have run the quickstart and
|
|
understand schemas, signals, and the `RETRIEVE` query. The product framing for this
|
|
surface is [USE_CASES.md → UC-01 Personalized Feed (For You)](../../USE_CASES.md#uc-01--personalized-feed--for-you).
|
|
|
|
---
|
|
|
|
## 1. What tidalDB owns vs. what you build
|
|
|
|
tidalDB answers exactly one question: **given a user and a context, what content
|
|
should they see, in what order?** It owns *retrieval and ranking* — and the
|
|
feedback loop that learns taste from engagement. It owns nothing else.
|
|
|
|
Everything that produces, stores, or serves the actual video bytes is yours.
|
|
|
|
| Concern | Who owns it |
|
|
|---|---|
|
|
| Video upload, blob storage, object store | **You** |
|
|
| CDN, transcoding, HLS/DASH packaging, thumbnails | **You** |
|
|
| Embedding generation (run the video through a multimodal model) | **You** — see [docs/guides/embeddings.md](./embeddings.md) |
|
|
| Authentication, identity, sessions | **You** |
|
|
| Moderation, trust & safety, takedowns | **You** |
|
|
| Payments, creator payouts | **You** |
|
|
| Player UI, swipe gestures, autoplay | **You** |
|
|
| **Candidate retrieval (ANN + filters)** | **tidalDB** |
|
|
| **Signal ledgers: decay, velocity, windowed counts** | **tidalDB** |
|
|
| **Per-user preference (taste) vectors** | **tidalDB** |
|
|
| **Ranking profiles (`for_you`, `following`, `trending`, …)** | **tidalDB** |
|
|
| **Diversity enforcement + exploration** | **tidalDB** |
|
|
| **The swipe feedback loop** (engagement → updated taste, no Kafka lag) | **tidalDB** |
|
|
|
|
### Architecture sketch
|
|
|
|
```
|
|
┌──────────────┐ upload ┌─────────────────────────────────────┐
|
|
│ Creator app │────────────▶│ YOUR services │
|
|
└──────────────┘ │ • blob store + CDN + transcode │
|
|
│ • multimodal model → embedding │
|
|
│ • auth / identity / moderation │
|
|
└───────────────┬─────────────────────┘
|
|
│
|
|
write_item_with_metadata(id, meta)
|
|
write_item_embedding(id, vector) ← you bring the vector
|
|
│
|
|
▼
|
|
┌──────────────┐ ┌───────────────────────────────┐
|
|
│ Viewer app │ GET feed │ tidalDB │
|
|
│ (player UI) │◀──── retrieve ────│ HNSW + signal ledgers + │
|
|
│ │ │ preference vectors + │
|
|
│ swipe ─────┼──── signals ─────▶│ ranking profiles + diversity │
|
|
│ (complete, │ signal_with_ │ │
|
|
│ like,skip) │ context(…) │ one process, one query API │
|
|
└──────────────┘ └───────────────────────────────┘
|
|
```
|
|
|
|
The viewer app does two things against tidalDB: it **reads** a ranked feed, and as
|
|
the user swipes it **writes** signals back. The write updates that user's taste in
|
|
the same process that serves the next read — there is no Kafka topic, no feature
|
|
store, and no batch job in between (section 4).
|
|
|
|
> tidalDB does **not** generate embeddings. Your model turns the video
|
|
> (frames + audio + caption) into a vector; tidalDB L2-normalizes it, inserts it
|
|
> into the HNSW index, and ranks over it. See [docs/guides/embeddings.md](./embeddings.md).
|
|
|
|
---
|
|
|
|
## 2. Model the signals for short video
|
|
|
|
A signal is a typed, timestamped event stream with native decay, velocity, and
|
|
windowed aggregation. You declare each signal once in the schema; the engine then
|
|
maintains its running decay score, per-window counts, and (optionally) velocity for
|
|
free. For short video, six signals capture the surface:
|
|
|
|
| Signal | Decay half-life | `positive_engagement` | Why |
|
|
|---|---|---|---|
|
|
| `view` | 7 days | no | The raw impression. Low-intent — a glance, not a preference. Drives trending velocity, not taste. |
|
|
| `completion` | 1 year (barely decays) | **yes** | Finishing a short video is the strongest, most durable taste signal there is. Weight = fraction watched ∈ [0,1]. |
|
|
| `like` | 30 days | **yes** | An explicit, durable positive. |
|
|
| `share` | 3 days (bursty) | no | Tracks `velocity` — shares spike, then fade; great for trending, not a private taste signal. |
|
|
| `skip` | 1 day | **no** | The swipe-away negative. Must **never** pull taste toward the skipped video, and marks it as a hard negative for that user. |
|
|
| `replay` | 14 days | **yes** | Re-watching is a meaningful, deliberate positive. |
|
|
|
|
### The `positive_engagement(true)` rule — read this twice
|
|
|
|
This is the single most important correctness rule for a personalized feed:
|
|
|
|
> **Preference-vector personalization fires only for signals declared
|
|
> `.positive_engagement(true)`.** When you record an engagement through
|
|
> `signal_with_context`, tidalDB folds the *item's content embedding* into the
|
|
> *acting user's* preference (taste) vector — but only if that signal type is a
|
|
> declared positive-engagement signal.
|
|
|
|
Why each classification matters:
|
|
|
|
- **`completion`, `like`, `replay` are `positive_engagement(true)`** → engaging
|
|
with a video nudges the user's taste vector toward that video's content. This is
|
|
the learning step. Get it right and the feed converges on what the user loves.
|
|
- **`view` is NOT positive-engagement** → a view is just "the video was shown."
|
|
If views folded into taste, a user's vector would be dragged toward everything
|
|
the feed happened to surface — a feedback bubble that learns nothing.
|
|
- **`skip` is NOT positive-engagement** → this is the load-bearing negative. A
|
|
swipe-away must never pull taste *toward* the rejected content. Routed through
|
|
`signal_with_context`, `skip` instead marks the video as a **hard negative** and
|
|
**seen** for that user, so the feed stops surfacing it.
|
|
|
|
If you accidentally mark `skip` or `view` as positive-engagement, the feed slowly
|
|
poisons every user's taste vector. Declare them honestly.
|
|
|
|
> **Subtle but important — the rule is all-or-nothing per database.** If *any*
|
|
> signal in your schema declares `positive_engagement(true)`, then **only** the
|
|
> signals you explicitly flagged fold the preference vector; nothing else does.
|
|
> If *no* signal declares it, tidalDB falls back to a built-in name allowlist
|
|
> (`like`, `share`, `completion`, `search_click`). The embedded schema below
|
|
> declares the flags explicitly, so it is fully in control. This matters for the
|
|
> HTTP path — see the caveat in section 9.
|
|
|
|
### Schema (embedded Rust)
|
|
|
|
This is lifted directly from
|
|
[`tidal/examples/foryou_feed.rs`](../../tidal/examples/foryou_feed.rs):
|
|
|
|
```rust
|
|
use std::time::Duration;
|
|
use tidaldb::schema::{DecaySpec, EntityKind, SchemaBuilder, Window};
|
|
|
|
let mut schema = SchemaBuilder::new();
|
|
|
|
// view: 7-day half-life, velocity + windows. Low-intent — NOT positive engagement.
|
|
let _ = schema
|
|
.signal("view", EntityKind::Item,
|
|
DecaySpec::Exponential { half_life: Duration::from_secs(7 * 24 * 3600) })
|
|
.windows(&[Window::OneHour, Window::TwentyFourHours, Window::AllTime])
|
|
.velocity(true)
|
|
.add();
|
|
|
|
// completion: ~1-year half-life. Weight = fraction watched. POSITIVE engagement.
|
|
let _ = schema
|
|
.signal("completion", EntityKind::Item,
|
|
DecaySpec::Exponential { half_life: Duration::from_secs(365 * 24 * 3600) })
|
|
.windows(&[Window::AllTime])
|
|
.positive_engagement(true)
|
|
.add();
|
|
|
|
// like: 30-day half-life. POSITIVE engagement.
|
|
let _ = schema
|
|
.signal("like", EntityKind::Item,
|
|
DecaySpec::Exponential { half_life: Duration::from_secs(30 * 24 * 3600) })
|
|
.windows(&[Window::AllTime])
|
|
.positive_engagement(true)
|
|
.add();
|
|
|
|
// share: 3-day half-life, bursty → velocity. NOT positive engagement.
|
|
let _ = schema
|
|
.signal("share", EntityKind::Item,
|
|
DecaySpec::Exponential { half_life: Duration::from_secs(3 * 24 * 3600) })
|
|
.windows(&[Window::TwentyFourHours, Window::AllTime])
|
|
.velocity(true)
|
|
.add();
|
|
|
|
// skip: fast 1-day decay — the swipe-away negative. NOT positive engagement.
|
|
let _ = schema
|
|
.signal("skip", EntityKind::Item,
|
|
DecaySpec::Exponential { half_life: Duration::from_secs(24 * 3600) })
|
|
.windows(&[Window::OneHour, Window::TwentyFourHours])
|
|
.add();
|
|
|
|
// replay: 14-day half-life. POSITIVE engagement.
|
|
let _ = schema
|
|
.signal("replay", EntityKind::Item,
|
|
DecaySpec::Exponential { half_life: Duration::from_secs(14 * 24 * 3600) })
|
|
.windows(&[Window::AllTime])
|
|
.positive_engagement(true)
|
|
.add();
|
|
|
|
// Content embedding slot for items: 128 dimensions (your model's output size).
|
|
let _ = schema.embedding_slot("content", EntityKind::Item, 128);
|
|
|
|
let schema = schema.build()?;
|
|
```
|
|
|
|
`DecaySpec` has three variants: `Exponential { half_life }`, `Linear { lifetime }`,
|
|
and `Permanent`. Available windows: `OneHour`, `TwentyFourHours`, `SevenDays`,
|
|
`ThirtyDays`, `AllTime`. Only declare the windows a profile actually reads —
|
|
each one is real bookkeeping.
|
|
|
|
---
|
|
|
|
## 3. Ingest videos (metadata + embeddings)
|
|
|
|
After your services have uploaded, transcoded, and embedded a video, write two
|
|
records into tidalDB: the metadata and the embedding vector.
|
|
|
|
```rust
|
|
use std::collections::HashMap;
|
|
use tidaldb::{TidalDb, schema::{EntityId, Timestamp}};
|
|
|
|
let db = TidalDb::builder().ephemeral().with_schema(schema).open()?;
|
|
// For durability: .with_data_dir("/var/lib/tidaldb") instead of .ephemeral()
|
|
|
|
let now = Timestamp::now();
|
|
|
|
let mut meta = HashMap::new();
|
|
meta.insert("title".to_string(), "Lo-fi piano loop".to_string());
|
|
meta.insert("category".to_string(), "music".to_string());
|
|
meta.insert("format".to_string(), "short".to_string());
|
|
meta.insert("creator_id".to_string(), "42".to_string());
|
|
meta.insert("duration".to_string(), "27".to_string()); // seconds
|
|
meta.insert("created_at".to_string(), now.as_nanos().to_string());
|
|
|
|
db.write_item_with_metadata(EntityId::new(1001), &meta)?;
|
|
|
|
// You bring the vector. This MUST be your model's output for THIS video.
|
|
let embedding: Vec<f32> = your_model.embed(&video); // length == slot dimensions (128)
|
|
db.write_item_embedding(EntityId::new(1001), &embedding)?;
|
|
```
|
|
|
|
Indexed metadata keys the engine understands: `title`, `category`, `format`,
|
|
`creator_id`, `tags`, `duration`, `created_at`. `creator_id` is what the
|
|
`max_per_creator` diversity cap (section 5) keys on, so always set it.
|
|
|
|
**Embedding rules (strict — read [docs/guides/embeddings.md](./embeddings.md) for
|
|
real vectors):**
|
|
|
|
- `write_item_embedding` L2-normalizes the vector and inserts it into the HNSW index.
|
|
- Dimensions are strict: **min 2, max 4096**, and the length **must equal the slot's
|
|
declared dimensions** (128 here) or the insert fails.
|
|
- **Zero-norm vectors are rejected.**
|
|
- **Multi-slot caveat:** `RETRIEVE`/`SEARCH` route through the **first declared slot
|
|
only**. A multi-modal app (separate text and image vectors) must fuse them offline
|
|
into one vector per slot, or model the modalities as separate entity kinds.
|
|
|
|
The example seeds 128-D *random* vectors purely so it runs standalone. A real app
|
|
must use genuine embeddings — random vectors give you random nearest-neighbors.
|
|
|
|
---
|
|
|
|
## 4. The swipe feedback loop (no Kafka, no feature-store lag)
|
|
|
|
This is what makes tidalDB a *feed* engine and not just a vector store. As the user
|
|
swipes, you record each engagement through `signal_with_context`, passing the user
|
|
id and the video's creator id. tidalDB writes the signal, updates windowed counts
|
|
and velocity, marks seen/hard-negative state, and — for positive-engagement signals —
|
|
folds the video's embedding into *that user's* preference vector. The next
|
|
`retrieve` call in the same process sees the updated taste. No queue, no batch job,
|
|
no feature store to keep in sync.
|
|
|
|
```rust
|
|
let user_id: u64 = 1001;
|
|
let creator_id: u64 = 42;
|
|
let video = EntityId::new(2001);
|
|
let now = Timestamp::now();
|
|
|
|
// User watched the whole thing → completion weight is the fraction watched (1.0).
|
|
// completion is positive-engagement → folds the video's embedding into the user's taste.
|
|
db.signal_with_context("completion", video, 1.0, now, Some(user_id), Some(creator_id))?;
|
|
|
|
// User tapped like → another positive fold, plus strengthens the (user, creator) edge.
|
|
db.signal_with_context("like", video, 1.0, now, Some(user_id), Some(creator_id))?;
|
|
|
|
// Different video — user swiped away after a glance. Partial view, then skip.
|
|
let rejected = EntityId::new(2002);
|
|
db.signal_with_context("view", rejected, 0.1, now, Some(user_id), Some(creator_id))?;
|
|
// skip is NOT positive-engagement: it does NOT pull taste toward `rejected`.
|
|
// It marks `rejected` as a hard negative + seen for this user, so the feed drops it.
|
|
db.signal_with_context("skip", rejected, 1.0, now, Some(user_id), Some(creator_id))?;
|
|
```
|
|
|
|
Contrast the two write APIs:
|
|
|
|
- `db.signal(type, id, weight, ts)` — **global** signal, no user context. Use it for
|
|
aggregate counters that feed `trending` (e.g. a server-side view counter). It does
|
|
**not** touch any preference vector or seen-state.
|
|
- `db.signal_with_context(type, id, weight, ts, Some(user_id), Some(creator_id))` —
|
|
**personalized**. Updates the user's preference vector (positive-engagement only),
|
|
seen-state, and hard negatives (skip/hide/block). This is the swipe-loop API.
|
|
|
|
A `completion` weight is the **fraction watched** in `[0.0, 1.0]` — 0.3 for a
|
|
three-second glance at a ten-second clip, 1.0 for a full watch (or a watch past the
|
|
end, if you clamp). The decay score blends these naturally: many partial completions
|
|
sum to less than a few full ones.
|
|
|
|
You can verify the loop landed by reading the live decay score:
|
|
|
|
```rust
|
|
let score = db.read_decay_score(EntityId::new(2001), "completion", 0)?; // window index 0
|
|
```
|
|
|
|
---
|
|
|
|
## 5. Serve the three feed surfaces
|
|
|
|
tidalDB ships **25 built-in ranking profiles**, registered automatically on
|
|
`open()`. A short-video app needs three of them. None require any extra schema — you
|
|
just name the profile in the query.
|
|
|
|
### For You — `profile("for_you")`
|
|
|
|
The personalized home feed. It blends interaction-weighted decay scores
|
|
(`view·1 + like·2 + share-velocity·1.5`) with the user's preference vector, sorts
|
|
with a Hot decay (`gravity = 1.5`), then enforces diversity and exploration. The
|
|
built-in defaults are:
|
|
|
|
- `max_per_creator = 2` — no single creator can dominate the feed.
|
|
- `format_mix_max_fraction = 0.4` — no one format swamps the mix.
|
|
- `exploration = 0.1` — **10% of slots are exploration**: unseen/random items
|
|
injected to break the filter bubble and let *new creators surface*. Without this,
|
|
a feed converges to a narrow loop and new content never gets a chance.
|
|
|
|
Seen items and hard negatives (the videos this user already watched or skipped) are
|
|
filtered out automatically — a feed must surface fresh content.
|
|
|
|
```rust
|
|
use tidaldb::query::retrieve::Retrieve;
|
|
|
|
let for_you = Retrieve::builder()
|
|
.for_user(user_id) // personalization + seen/hard-negative filtering
|
|
.profile("for_you")
|
|
.limit(10)
|
|
.build()?;
|
|
|
|
let results = db.retrieve(&for_you)?;
|
|
for item in &results.items {
|
|
println!("#{} video={} score={:.4}", item.rank, item.entity_id.as_u64(), item.score);
|
|
}
|
|
```
|
|
|
|
Want to override the defaults for one query? Use `.diversity(...)`:
|
|
|
|
```rust
|
|
use tidaldb::ranking::diversity::DiversityConstraints;
|
|
|
|
let q = Retrieve::builder()
|
|
.for_user(user_id)
|
|
.profile("for_you")
|
|
.diversity(DiversityConstraints {
|
|
max_per_creator: Some(1), // stricter: one video per creator
|
|
format_mix_max_fraction: Some(0.5),
|
|
..DiversityConstraints::new()
|
|
})
|
|
.limit(20)
|
|
.build()?;
|
|
```
|
|
|
|
### Following — `profile("following")`
|
|
|
|
Content from creators the user follows, sourced from the relationship graph and
|
|
ranked by recency with a view-velocity boost (`max_per_creator = 3`). This requires
|
|
that you have recorded follow relationships (a follows edge) for the user — see
|
|
[USE_CASES.md](../../USE_CASES.md) for the relationship model.
|
|
|
|
```rust
|
|
let following = Retrieve::builder()
|
|
.for_user(user_id)
|
|
.profile("following")
|
|
.limit(20)
|
|
.build()?;
|
|
let results = db.retrieve(&following)?;
|
|
```
|
|
|
|
### Discover / Trending — `profile("trending")`
|
|
|
|
The global, *non-personalized* tab — what a brand-new visitor with no taste history
|
|
sees. It ranks every item by velocity (`view-velocity + 2·share-velocity` over the
|
|
24h window), capped at `max_per_creator = 1` for maximum variety. Note: velocity
|
|
needs signals arriving over real elapsed time to populate hour-level buckets, so a
|
|
freshly-seeded demo will show sparse trending scores — that is expected.
|
|
|
|
```rust
|
|
let trending = Retrieve::builder() // no .for_user — trending is global
|
|
.profile("trending")
|
|
.limit(20)
|
|
.build()?;
|
|
let results = db.retrieve(&trending)?;
|
|
```
|
|
|
|
> The payoff to look for in [`foryou_feed.rs`](../../tidal/examples/foryou_feed.rs):
|
|
> after a user completes + likes music videos and skips everything else, the top of
|
|
> their `for_you` feed is the *fresh* music videos — surfaced because the swipe
|
|
> session built a strong taste signal, **not** because of the (random) embeddings.
|
|
> The ordering is driven by signals, which is why it is stable across runs.
|
|
|
|
---
|
|
|
|
## 6. The whole thing, embedded (Rust)
|
|
|
|
The complete, verified program is at
|
|
[`tidal/examples/foryou_feed.rs`](../../tidal/examples/foryou_feed.rs). Run it:
|
|
|
|
```bash
|
|
cargo run -p tidaldb --example foryou_feed
|
|
```
|
|
|
|
It walks the full arc end to end:
|
|
|
|
1. Builds the six-signal short-video schema from section 2.
|
|
2. Opens an ephemeral DB and ingests ~24 videos across 5 creators.
|
|
3. Simulates one user's swipe session — completions + likes + a share + a replay on
|
|
music videos, skips on the rest — all via `signal_with_context`.
|
|
4. Retrieves `for_you` (diversity-capped, 10% exploration) and prints the ranked feed.
|
|
5. Retrieves global `trending` for contrast.
|
|
|
|
`Arc<TidalDb>` is `Send + Sync`, so the same handle backs every request thread in a
|
|
real server: clone the `Arc` into your axum/actix handlers, write signals on the
|
|
swipe endpoint, and read feeds on the feed endpoint. See the embedding-integration
|
|
examples (`axum_embedding`, `actix_embedding`, `cli_embedding`) in `tidal/examples/`
|
|
for the web-handler wiring.
|
|
|
|
A `Results` value contains `items: Vec<FeedItem>`; each `FeedItem` exposes `.rank`,
|
|
`.entity_id.as_u64()`, `.score`, and `.signals` (per-item signal snapshots for
|
|
explainability), plus `results.total_candidates` for the pre-diversity pool size.
|
|
|
|
For text + vector hybrid search (a search box rather than a feed), use the parallel
|
|
`Search::builder().query("...").vector(vec).for_user(uid).limit(n).build()?` then
|
|
`db.search(&q)?` — covered in [QUICKSTART.md](../../QUICKSTART.md) and
|
|
[API.md](../../API.md).
|
|
|
|
---
|
|
|
|
## 7. The whole thing, over HTTP
|
|
|
|
The same database behind a JSON HTTP API is `tidal-server`. Start a standalone
|
|
node with your schema:
|
|
|
|
```bash
|
|
cargo run -p tidal-server -- standalone \
|
|
--listen 0.0.0.0:9400 \
|
|
--schema ./short-video-schema.yaml \
|
|
--data-dir /var/lib/tidaldb \
|
|
--metrics 127.0.0.1:9091
|
|
```
|
|
|
|
Set `TIDAL_API_KEY` to require Bearer auth on the data routes. **If `TIDAL_API_KEY`
|
|
is unset, the server runs fully unauthenticated and logs a WARN** — never do that
|
|
outside a private dev box.
|
|
|
|
```bash
|
|
export TIDAL_API_KEY="$(openssl rand -hex 32)"
|
|
```
|
|
|
|
### Schema YAML
|
|
|
|
The HTTP server loads its schema from YAML. The short-video schema looks like:
|
|
|
|
```yaml
|
|
signals:
|
|
- name: view
|
|
entity: item
|
|
decay: { exponential: { half_life_seconds: 604800 } } # 7 days
|
|
windows: [one_hour, twenty_four_hours, all_time]
|
|
velocity: true
|
|
- name: completion
|
|
entity: item
|
|
decay: { exponential: { half_life_seconds: 31536000 } } # 1 year
|
|
windows: [all_time]
|
|
positive_engagement: true
|
|
- name: like
|
|
entity: item
|
|
decay: { exponential: { half_life_seconds: 2592000 } } # 30 days
|
|
windows: [all_time]
|
|
positive_engagement: true
|
|
- name: share
|
|
entity: item
|
|
decay: { exponential: { half_life_seconds: 259200 } } # 3 days
|
|
windows: [twenty_four_hours, all_time]
|
|
velocity: true
|
|
- name: skip
|
|
entity: item
|
|
decay: { exponential: { half_life_seconds: 86400 } } # 1 day
|
|
windows: [one_hour, twenty_four_hours]
|
|
- name: replay
|
|
entity: item
|
|
decay: { exponential: { half_life_seconds: 1209600 } } # 14 days
|
|
windows: [all_time]
|
|
positive_engagement: true
|
|
text_fields:
|
|
- { name: title, kind: text }
|
|
- { name: category, kind: keyword }
|
|
embedding_slots:
|
|
- { name: content, entity: item, dimensions: 128 }
|
|
```
|
|
|
|
> **`positive_engagement` works in YAML too.** Declaring it per signal (as
|
|
> above) gives the HTTP server the same explicit control as the embedded
|
|
> `SchemaBuilder` — `completion`, `like`, and `replay` fold into the user's
|
|
> preference vector; `view` and `skip` do not. If you omit the field on *every*
|
|
> signal, tidalDB falls back to a built-in name allowlist (`like`, `share`,
|
|
> `completion`, `search_click`); declaring it explicitly (recommended) removes
|
|
> any ambiguity about which signals shape taste.
|
|
|
|
The 25 built-in profiles (`for_you`, `following`, `trending`, …) are available on
|
|
the HTTP server with no `profiles:` block — they are registered automatically. Add a
|
|
`profiles:` block only to override a built-in or define your own.
|
|
|
|
### Routes
|
|
|
|
All routes are JSON. Data routes require `Authorization: Bearer $TIDAL_API_KEY` when
|
|
the key is set; `GET /health*` and `GET /openapi.json` are always unauthenticated.
|
|
|
|
```bash
|
|
# Ingest a video's metadata → 201 Created
|
|
curl -X POST http://localhost:9400/items \
|
|
-H "Authorization: Bearer $TIDAL_API_KEY" \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{ "entity_id": 1001,
|
|
"metadata": { "title": "Lo-fi piano loop", "category": "music",
|
|
"format": "short", "creator_id": "42", "duration": "27" } }'
|
|
|
|
# Ingest its embedding (length MUST equal slot dimensions) → 204 No Content
|
|
curl -X POST http://localhost:9400/embeddings \
|
|
-H "Authorization: Bearer $TIDAL_API_KEY" \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{ "entity_id": 1001, "values": [0.013, -0.041, /* … 128 floats … */ 0.092] }'
|
|
|
|
# The swipe loop: record engagement with user + creator context → 204 No Content
|
|
curl -X POST http://localhost:9400/signals \
|
|
-H "Authorization: Bearer $TIDAL_API_KEY" \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{ "entity_id": 1001, "signal": "completion", "weight": 1.0,
|
|
"user_id": 1001, "creator_id": 42 }'
|
|
|
|
curl -X POST http://localhost:9400/signals \
|
|
-H "Authorization: Bearer $TIDAL_API_KEY" \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{ "entity_id": 1001, "signal": "like", "weight": 1.0,
|
|
"user_id": 1001, "creator_id": 42 }'
|
|
|
|
# The swipe-away negative on a different video
|
|
curl -X POST http://localhost:9400/signals \
|
|
-H "Authorization: Bearer $TIDAL_API_KEY" \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{ "entity_id": 1002, "signal": "skip", "weight": 1.0,
|
|
"user_id": 1001, "creator_id": 7 }'
|
|
|
|
# Serve the For You feed
|
|
curl -H "Authorization: Bearer $TIDAL_API_KEY" \
|
|
"http://localhost:9400/feed?user_id=1001&profile=for_you&limit=20"
|
|
|
|
# Following feed and the Discover/Trending tab
|
|
curl -H "Authorization: Bearer $TIDAL_API_KEY" \
|
|
"http://localhost:9400/feed?user_id=1001&profile=following&limit=20"
|
|
curl -H "Authorization: Bearer $TIDAL_API_KEY" \
|
|
"http://localhost:9400/feed?profile=trending&limit=20" # no user_id — global
|
|
```
|
|
|
|
When `?user_id=` and `?profile=` are omitted, `/feed` defaults to `profile=for_you`.
|
|
Passing `user_id` and `creator_id` on `/signals` is what activates the personalized
|
|
loop — omit them and the signal is recorded globally (the `signal` vs.
|
|
`signal_with_context` distinction from section 4).
|
|
|
|
Middleware on the data routes: 30s timeout (`408`), 100 max in-flight (`429`), 2 MB
|
|
body limit (`413`), and an `x-request-id` echoed on every response. The canonical,
|
|
always-current HTTP reference is the served spec at `GET /openapi.json` (unauthenticated).
|
|
|
|
---
|
|
|
|
## 8. Going to production — checklist
|
|
|
|
- [ ] **Durable storage.** Open with `.with_data_dir(...)` (embedded) or
|
|
`--data-dir` (server), not ephemeral. The WAL persists signals across restarts.
|
|
- [ ] **Real embeddings.** Replace the random vectors with your multimodal model's
|
|
output. Same model for ingest and query. See
|
|
[docs/guides/embeddings.md](./embeddings.md).
|
|
- [ ] **Auth on.** Set `TIDAL_API_KEY`. An unset key means an unauthenticated server
|
|
(it logs a WARN). Terminate TLS at your ingress.
|
|
- [ ] **Lock down metrics.** The `--metrics` Prometheus endpoint is **unauthenticated**
|
|
— bind it to loopback or a cluster-internal address only, never the public
|
|
interface.
|
|
- [ ] **Graceful shutdown.** On `SIGTERM` the server flips readiness to `503`,
|
|
drains in-flight requests, then checkpoints and fsyncs the WAL. Wire your
|
|
orchestrator's `preStop`/termination grace period to allow the drain.
|
|
- [ ] **Health probes.** Use `GET /health` for readiness (`200` ok / `503` draining),
|
|
`GET /health/startup` and `GET /health/live` for startup/liveness.
|
|
- [ ] **Observability.** Scrape the metrics endpoint; import the dashboard and alerts
|
|
from [docs/ops/](../ops/) (`grafana-dashboard.json`, `prometheus-alerts.yaml`,
|
|
`monitoring.md`). Plan headroom with [docs/ops/capacity-planning.md](../ops/capacity-planning.md).
|
|
- [ ] **Backup & recovery.** Rehearse restore with [docs/ops/recovery.md](../ops/recovery.md).
|
|
- [ ] **Deployment.** Follow [docs/guides/server-deployment.md](./server-deployment.md)
|
|
for packaging and rollout, and the Kubernetes runbook in
|
|
[docs/runbooks/](../runbooks/) for cluster orchestration. The container images
|
|
live at `docker/` (`standalone` / `cluster` / `deploy`) — build from the repo
|
|
root so `COPY . .` sees the whole workspace.
|
|
|
|
> **On cluster mode — be honest with yourself.** tidalDB's cluster mode is
|
|
> **experimental** and currently runs **all regions in a single process** over
|
|
> loopback gRPC. It is *not* production multi-process HA (true multi-process HA is
|
|
> tracked as m8p10). It also replicates **global signals only** — `user_id` /
|
|
> `creator_id` contexts are rejected in cluster mode, so the personalized swipe loop
|
|
> (section 4) is a **single-node** feature today. Ship the For You feed on a
|
|
> vertically-scaled standalone node; reach for the experimental cluster only for
|
|
> read-scale experiments, per the [cluster runbook](../runbooks/cluster.md).
|
|
|
|
---
|
|
|
|
## See also
|
|
|
|
- [QUICKSTART.md](../../QUICKSTART.md) — the 5-minute ingest → rank loop.
|
|
- [USE_CASES.md → UC-01](../../USE_CASES.md#uc-01--personalized-feed--for-you) — product framing for the For You surface.
|
|
- [docs/guides/embeddings.md](./embeddings.md) — generating and writing real vectors.
|
|
- [docs/guides/server-deployment.md](./server-deployment.md) — packaging and rollout.
|
|
- [API.md](../../API.md) — full embedded + HTTP API reference.
|
|
- [`tidal/examples/foryou_feed.rs`](../../tidal/examples/foryou_feed.rs) — the verified, runnable version of this guide.
|