tidaldb/docs/guides/build-a-feed-app.md

Build a Short-Video "For You" Feed on tidalDB

This is an end-to-end tutorial for building a TikTok/Reels-style short-video feed — a swipeable home feed ("For You"), a Following feed, and a Discover/Trending tab — on top of tidalDB. You will model the signals a short-video app actually emits, ingest videos, wire the swipe feedback loop so a user's taste updates immediately, and serve all three feed surfaces. You will do it twice: once embedded in a Rust process, and once over HTTP with curl.

There is a verified, runnable version of the embedded path in this repo:

cargo run -p tidaldb --example foryou_feed

That example (tidal/examples/foryou_feed.rs) is the source of truth for every Rust call shown below — this guide reproduces and explains its key calls. Skim it after reading section 6.

New to tidalDB? Read QUICKSTART.md first for the 5-minute schema → ingest → rank loop. This guide assumes you have run the quickstart and understand schemas, signals, and the RETRIEVE query. The product framing for this surface is USE_CASES.md → UC-01 Personalized Feed (For You).

---

1. What tidalDB owns vs. what you build

tidalDB answers exactly one question: given a user and a context, what content should they see, in what order? It owns retrieval and ranking — and the feedback loop that learns taste from engagement. It owns nothing else.

Everything that produces, stores, or serves the actual video bytes is yours.

Concern	Who owns it
Video upload, blob storage, object store	You
CDN, transcoding, HLS/DASH packaging, thumbnails	You
Embedding generation (run the video through a multimodal model)	You — see docs/guides/embeddings.md
Authentication, identity, sessions	You
Moderation, trust & safety, takedowns	You
Payments, creator payouts	You
Player UI, swipe gestures, autoplay	You
Candidate retrieval (ANN + filters)	tidalDB
Signal ledgers: decay, velocity, windowed counts	tidalDB
Per-user preference (taste) vectors	tidalDB
Ranking profiles (for_you, following, trending, …)	tidalDB
Diversity enforcement + exploration	tidalDB
The swipe feedback loop (engagement → updated taste, no Kafka lag)	tidalDB

Architecture sketch

   ┌──────────────┐   upload    ┌─────────────────────────────────────┐
   │  Creator app │────────────▶│  YOUR services                      │
   └──────────────┘             │   • blob store + CDN + transcode    │
                                │   • multimodal model → embedding    │
                                │   • auth / identity / moderation    │
                                └───────────────┬─────────────────────┘
                                                │
                          write_item_with_metadata(id, meta)
                          write_item_embedding(id, vector)   ← you bring the vector
                                                │
                                                ▼
   ┌──────────────┐                   ┌───────────────────────────────┐
   │  Viewer app  │   GET feed        │            tidalDB            │
   │  (player UI) │◀──── retrieve ────│  HNSW + signal ledgers +      │
   │              │                   │  preference vectors +         │
   │   swipe ─────┼──── signals ─────▶│  ranking profiles + diversity │
   │  (complete,  │  signal_with_     │                               │
   │   like,skip) │     context(…)    │  one process, one query API   │
   └──────────────┘                   └───────────────────────────────┘

The viewer app does two things against tidalDB: it reads a ranked feed, and as the user swipes it writes signals back. The write updates that user's taste in the same process that serves the next read — there is no Kafka topic, no feature store, and no batch job in between (section 4).

tidalDB does not generate embeddings. Your model turns the video (frames + audio + caption) into a vector; tidalDB L2-normalizes it, inserts it into the HNSW index, and ranks over it. See docs/guides/embeddings.md.

---

2. Model the signals for short video

A signal is a typed, timestamped event stream with native decay, velocity, and windowed aggregation. You declare each signal once in the schema; the engine then maintains its running decay score, per-window counts, and (optionally) velocity for free. For short video, six signals capture the surface:

Signal	Decay half-life	positive_engagement	Why
view	7 days	no	The raw impression. Low-intent — a glance, not a preference. Drives trending velocity, not taste.
completion	1 year (barely decays)	yes	Finishing a short video is the strongest, most durable taste signal there is. Weight = fraction watched ∈ [0,1].
like	30 days	yes	An explicit, durable positive.
share	3 days (bursty)	no	Tracks velocity — shares spike, then fade; great for trending, not a private taste signal.
skip	1 day	no	The swipe-away negative. Must never pull taste toward the skipped video, and marks it as a hard negative for that user.
replay	14 days	yes	Re-watching is a meaningful, deliberate positive.

The positive_engagement(true) rule — read this twice

This is the single most important correctness rule for a personalized feed:

Preference-vector personalization fires only for signals declared .positive_engagement(true). When you record an engagement through signal_with_context, tidalDB folds the item's content embedding into the acting user's preference (taste) vector — but only if that signal type is a declared positive-engagement signal.

Why each classification matters:

• completion, like, replay are positive_engagement(true) → engaging with a video nudges the user's taste vector toward that video's content. This is the learning step. Get it right and the feed converges on what the user loves.
• view is NOT positive-engagement → a view is just "the video was shown." If views folded into taste, a user's vector would be dragged toward everything the feed happened to surface — a feedback bubble that learns nothing.
• skip is NOT positive-engagement → this is the load-bearing negative. A swipe-away must never pull taste toward the rejected content. Routed through signal_with_context, skip instead marks the video as a hard negative and seen for that user, so the feed stops surfacing it.

If you accidentally mark skip or view as positive-engagement, the feed slowly poisons every user's taste vector. Declare them honestly.

Subtle but important — the rule is all-or-nothing per database. If any signal in your schema declares positive_engagement(true), then only the signals you explicitly flagged fold the preference vector; nothing else does. If no signal declares it, tidalDB falls back to a built-in name allowlist (like, share, completion, search_click). The embedded schema below declares the flags explicitly, so it is fully in control. This matters for the HTTP path — see the caveat in section 9.

Schema (embedded Rust)

This is lifted directly from tidal/examples/foryou_feed.rs:

use std::time::Duration;
use tidaldb::schema::{DecaySpec, EntityKind, SchemaBuilder, Window};

let mut schema = SchemaBuilder::new();

// view: 7-day half-life, velocity + windows. Low-intent — NOT positive engagement.
let _ = schema
    .signal("view", EntityKind::Item,
            DecaySpec::Exponential { half_life: Duration::from_secs(7 * 24 * 3600) })
    .windows(&[Window::OneHour, Window::TwentyFourHours, Window::AllTime])
    .velocity(true)
    .add();

// completion: ~1-year half-life. Weight = fraction watched. POSITIVE engagement.
let _ = schema
    .signal("completion", EntityKind::Item,
            DecaySpec::Exponential { half_life: Duration::from_secs(365 * 24 * 3600) })
    .windows(&[Window::AllTime])
    .positive_engagement(true)
    .add();

// like: 30-day half-life. POSITIVE engagement.
let _ = schema
    .signal("like", EntityKind::Item,
            DecaySpec::Exponential { half_life: Duration::from_secs(30 * 24 * 3600) })
    .windows(&[Window::AllTime])
    .positive_engagement(true)
    .add();

// share: 3-day half-life, bursty → velocity. NOT positive engagement.
let _ = schema
    .signal("share", EntityKind::Item,
            DecaySpec::Exponential { half_life: Duration::from_secs(3 * 24 * 3600) })
    .windows(&[Window::TwentyFourHours, Window::AllTime])
    .velocity(true)
    .add();

// skip: fast 1-day decay — the swipe-away negative. NOT positive engagement.
let _ = schema
    .signal("skip", EntityKind::Item,
            DecaySpec::Exponential { half_life: Duration::from_secs(24 * 3600) })
    .windows(&[Window::OneHour, Window::TwentyFourHours])
    .add();

// replay: 14-day half-life. POSITIVE engagement.
let _ = schema
    .signal("replay", EntityKind::Item,
            DecaySpec::Exponential { half_life: Duration::from_secs(14 * 24 * 3600) })
    .windows(&[Window::AllTime])
    .positive_engagement(true)
    .add();

// Content embedding slot for items: 128 dimensions (your model's output size).
let _ = schema.embedding_slot("content", EntityKind::Item, 128);

let schema = schema.build()?;

DecaySpec has three variants: Exponential { half_life }, Linear { lifetime }, and Permanent. Available windows: OneHour, TwentyFourHours, SevenDays, ThirtyDays, AllTime. Only declare the windows a profile actually reads — each one is real bookkeeping.

---

3. Ingest videos (metadata + embeddings)

After your services have uploaded, transcoded, and embedded a video, write two records into tidalDB: the metadata and the embedding vector.

use std::collections::HashMap;
use tidaldb::{TidalDb, schema::{EntityId, Timestamp}};

let db = TidalDb::builder().ephemeral().with_schema(schema).open()?;
// For durability: .with_data_dir("/var/lib/tidaldb") instead of .ephemeral()

let now = Timestamp::now();

let mut meta = HashMap::new();
meta.insert("title".to_string(),      "Lo-fi piano loop".to_string());
meta.insert("category".to_string(),   "music".to_string());
meta.insert("format".to_string(),     "short".to_string());
meta.insert("creator_id".to_string(), "42".to_string());
meta.insert("duration".to_string(),   "27".to_string());            // seconds
meta.insert("created_at".to_string(), now.as_nanos().to_string());

db.write_item_with_metadata(EntityId::new(1001), &meta)?;

// You bring the vector. This MUST be your model's output for THIS video.
let embedding: Vec<f32> = your_model.embed(&video);   // length == slot dimensions (128)
db.write_item_embedding(EntityId::new(1001), &embedding)?;

Indexed metadata keys the engine understands: title, category, format, creator_id, tags, duration, created_at. creator_id is what the max_per_creator diversity cap (section 5) keys on, so always set it.

Embedding rules (strict — read docs/guides/embeddings.md for real vectors):

• write_item_embedding L2-normalizes the vector and inserts it into the HNSW index.
• Dimensions are strict: min 2, max 4096, and the length must equal the slot's declared dimensions (128 here) or the insert fails.
• Zero-norm vectors are rejected.
• Multi-slot caveat: RETRIEVE/SEARCH route through the first declared slot only. A multi-modal app (separate text and image vectors) must fuse them offline into one vector per slot, or model the modalities as separate entity kinds.

The example seeds 128-D random vectors purely so it runs standalone. A real app must use genuine embeddings — random vectors give you random nearest-neighbors.

---

4. The swipe feedback loop (no Kafka, no feature-store lag)

This is what makes tidalDB a feed engine and not just a vector store. As the user swipes, you record each engagement through signal_with_context, passing the user id and the video's creator id. tidalDB writes the signal, updates windowed counts and velocity, marks seen/hard-negative state, and — for positive-engagement signals — folds the video's embedding into that user's preference vector. The next retrieve call in the same process sees the updated taste. No queue, no batch job, no feature store to keep in sync.

let user_id: u64    = 1001;
let creator_id: u64 = 42;
let video           = EntityId::new(2001);
let now             = Timestamp::now();

// User watched the whole thing → completion weight is the fraction watched (1.0).
// completion is positive-engagement → folds the video's embedding into the user's taste.
db.signal_with_context("completion", video, 1.0, now, Some(user_id), Some(creator_id))?;

// User tapped like → another positive fold, plus strengthens the (user, creator) edge.
db.signal_with_context("like", video, 1.0, now, Some(user_id), Some(creator_id))?;

// Different video — user swiped away after a glance. Partial view, then skip.
let rejected = EntityId::new(2002);
db.signal_with_context("view", rejected, 0.1, now, Some(user_id), Some(creator_id))?;
// skip is NOT positive-engagement: it does NOT pull taste toward `rejected`.
// It marks `rejected` as a hard negative + seen for this user, so the feed drops it.
db.signal_with_context("skip", rejected, 1.0, now, Some(user_id), Some(creator_id))?;

Contrast the two write APIs:

• db.signal(type, id, weight, ts) — global signal, no user context. Use it for aggregate counters that feed trending (e.g. a server-side view counter). It does not touch any preference vector or seen-state.
• db.signal_with_context(type, id, weight, ts, Some(user_id), Some(creator_id)) — personalized. Updates the user's preference vector (positive-engagement only), seen-state, and hard negatives (skip/hide/block). This is the swipe-loop API.

A completion weight is the fraction watched in [0.0, 1.0] — 0.3 for a three-second glance at a ten-second clip, 1.0 for a full watch (or a watch past the end, if you clamp). The decay score blends these naturally: many partial completions sum to less than a few full ones.

You can verify the loop landed by reading the live decay score:

let score = db.read_decay_score(EntityId::new(2001), "completion", 0)?; // window index 0

---

5. Serve the three feed surfaces

tidalDB ships 25 built-in ranking profiles, registered automatically on open(). A short-video app needs three of them. None require any extra schema — you just name the profile in the query.

For You — profile("for_you")

The personalized home feed. It blends interaction-weighted decay scores (view·1 + like·2 + share-velocity·1.5) with the user's preference vector, sorts with a Hot decay (gravity = 1.5), then enforces diversity and exploration. The built-in defaults are:

• max_per_creator = 2 — no single creator can dominate the feed.
• format_mix_max_fraction = 0.4 — no one format swamps the mix.
• exploration = 0.1 — 10% of slots are exploration: unseen/random items injected to break the filter bubble and let new creators surface. Without this, a feed converges to a narrow loop and new content never gets a chance.

Seen items and hard negatives (the videos this user already watched or skipped) are filtered out automatically — a feed must surface fresh content.

use tidaldb::query::retrieve::Retrieve;

let for_you = Retrieve::builder()
    .for_user(user_id)        // personalization + seen/hard-negative filtering
    .profile("for_you")
    .limit(10)
    .build()?;

let results = db.retrieve(&for_you)?;
for item in &results.items {
    println!("#{} video={} score={:.4}", item.rank, item.entity_id.as_u64(), item.score);
}

Want to override the defaults for one query? Use .diversity(...):

use tidaldb::ranking::diversity::DiversityConstraints;

let q = Retrieve::builder()
    .for_user(user_id)
    .profile("for_you")
    .diversity(DiversityConstraints {
        max_per_creator: Some(1),          // stricter: one video per creator
        format_mix_max_fraction: Some(0.5),
        ..DiversityConstraints::new()
    })
    .limit(20)
    .build()?;

Following — profile("following")

Content from creators the user follows, sourced from the relationship graph and ranked by recency with a view-velocity boost (max_per_creator = 3). This requires that you have recorded follow relationships (a follows edge) for the user — see USE_CASES.md for the relationship model.

let following = Retrieve::builder()
    .for_user(user_id)
    .profile("following")
    .limit(20)
    .build()?;
let results = db.retrieve(&following)?;

Discover / Trending — profile("trending")

The global, non-personalized tab — what a brand-new visitor with no taste history sees. It ranks every item by velocity (view-velocity + 2·share-velocity over the 24h window), capped at max_per_creator = 1 for maximum variety. Note: velocity needs signals arriving over real elapsed time to populate hour-level buckets, so a freshly-seeded demo will show sparse trending scores — that is expected.

let trending = Retrieve::builder()   // no .for_user — trending is global
    .profile("trending")
    .limit(20)
    .build()?;
let results = db.retrieve(&trending)?;

The payoff to look for in foryou_feed.rs: after a user completes + likes music videos and skips everything else, the top of their for_you feed is the fresh music videos — surfaced because the swipe session built a strong taste signal, not because of the (random) embeddings. The ordering is driven by signals, which is why it is stable across runs.

---

6. The whole thing, embedded (Rust)

The complete, verified program is at tidal/examples/foryou_feed.rs. Run it:

cargo run -p tidaldb --example foryou_feed

It walks the full arc end to end:

1. Builds the six-signal short-video schema from section 2.
2. Opens an ephemeral DB and ingests ~24 videos across 5 creators.
3. Simulates one user's swipe session — completions + likes + a share + a replay on music videos, skips on the rest — all via signal_with_context.
4. Retrieves for_you (diversity-capped, 10% exploration) and prints the ranked feed.
5. Retrieves global trending for contrast.

Arc<TidalDb> is Send + Sync, so the same handle backs every request thread in a real server: clone the Arc into your axum/actix handlers, write signals on the swipe endpoint, and read feeds on the feed endpoint. See the embedding-integration examples (axum_embedding, actix_embedding, cli_embedding) in tidal/examples/ for the web-handler wiring.

A Results value contains items: Vec<FeedItem>; each FeedItem exposes .rank, .entity_id.as_u64(), .score, and .signals (per-item signal snapshots for explainability), plus results.total_candidates for the pre-diversity pool size.

For text + vector hybrid search (a search box rather than a feed), use the parallel Search::builder().query("...").vector(vec).for_user(uid).limit(n).build()? then db.search(&q)? — covered in QUICKSTART.md and API.md.

---

7. The whole thing, over HTTP

The same database behind a JSON HTTP API is tidal-server. Start a standalone node with your schema:

cargo run -p tidal-server -- standalone \
  --listen 0.0.0.0:9400 \
  --schema ./short-video-schema.yaml \
  --data-dir /var/lib/tidaldb \
  --metrics 127.0.0.1:9091

Set TIDAL_API_KEY to require Bearer auth on the data routes. If TIDAL_API_KEY is unset, the server runs fully unauthenticated and logs a WARN — never do that outside a private dev box.

export TIDAL_API_KEY="$(openssl rand -hex 32)"

Schema YAML

The HTTP server loads its schema from YAML. The short-video schema looks like:

signals:
  - name: view
    entity: item
    decay: { exponential: { half_life_seconds: 604800 } }   # 7 days
    windows: [one_hour, twenty_four_hours, all_time]
    velocity: true
  - name: completion
    entity: item
    decay: { exponential: { half_life_seconds: 31536000 } } # 1 year
    windows: [all_time]
    positive_engagement: true
  - name: like
    entity: item
    decay: { exponential: { half_life_seconds: 2592000 } }  # 30 days
    windows: [all_time]
    positive_engagement: true
  - name: share
    entity: item
    decay: { exponential: { half_life_seconds: 259200 } }   # 3 days
    windows: [twenty_four_hours, all_time]
    velocity: true
  - name: skip
    entity: item
    decay: { exponential: { half_life_seconds: 86400 } }    # 1 day
    windows: [one_hour, twenty_four_hours]
  - name: replay
    entity: item
    decay: { exponential: { half_life_seconds: 1209600 } }  # 14 days
    windows: [all_time]
    positive_engagement: true
text_fields:
  - { name: title, kind: text }
  - { name: category, kind: keyword }
embedding_slots:
  - { name: content, entity: item, dimensions: 128 }

positive_engagement works in YAML too. Declaring it per signal (as above) gives the HTTP server the same explicit control as the embedded SchemaBuilder — completion, like, and replay fold into the user's preference vector; view and skip do not. If you omit the field on every signal, tidalDB falls back to a built-in name allowlist (like, share, completion, search_click); declaring it explicitly (recommended) removes any ambiguity about which signals shape taste.

The 25 built-in profiles (for_you, following, trending, …) are available on the HTTP server with no profiles: block — they are registered automatically. Add a profiles: block only to override a built-in or define your own.

Routes

All routes are JSON. Data routes require Authorization: Bearer $TIDAL_API_KEY when the key is set; GET /health* and GET /openapi.json are always unauthenticated.

# Ingest a video's metadata  → 201 Created
curl -X POST http://localhost:9400/items \
  -H "Authorization: Bearer $TIDAL_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1001,
        "metadata": { "title": "Lo-fi piano loop", "category": "music",
                      "format": "short", "creator_id": "42", "duration": "27" } }'

# Ingest its embedding (length MUST equal slot dimensions) → 204 No Content
curl -X POST http://localhost:9400/embeddings \
  -H "Authorization: Bearer $TIDAL_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1001, "values": [0.013, -0.041, /* … 128 floats … */ 0.092] }'

# The swipe loop: record engagement with user + creator context → 204 No Content
curl -X POST http://localhost:9400/signals \
  -H "Authorization: Bearer $TIDAL_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1001, "signal": "completion", "weight": 1.0,
        "user_id": 1001, "creator_id": 42 }'

curl -X POST http://localhost:9400/signals \
  -H "Authorization: Bearer $TIDAL_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1001, "signal": "like", "weight": 1.0,
        "user_id": 1001, "creator_id": 42 }'

# The swipe-away negative on a different video
curl -X POST http://localhost:9400/signals \
  -H "Authorization: Bearer $TIDAL_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{ "entity_id": 1002, "signal": "skip", "weight": 1.0,
        "user_id": 1001, "creator_id": 7 }'

# Serve the For You feed
curl -H "Authorization: Bearer $TIDAL_API_KEY" \
  "http://localhost:9400/feed?user_id=1001&profile=for_you&limit=20"

# Following feed and the Discover/Trending tab
curl -H "Authorization: Bearer $TIDAL_API_KEY" \
  "http://localhost:9400/feed?user_id=1001&profile=following&limit=20"
curl -H "Authorization: Bearer $TIDAL_API_KEY" \
  "http://localhost:9400/feed?profile=trending&limit=20"     # no user_id — global

When ?user_id= and ?profile= are omitted, /feed defaults to profile=for_you. Passing user_id and creator_id on /signals is what activates the personalized loop — omit them and the signal is recorded globally (the signal vs. signal_with_context distinction from section 4).

Middleware on the data routes: 30s timeout (408), 100 max in-flight (429), 2 MB body limit (413), and an x-request-id echoed on every response. The canonical, always-current HTTP reference is the served spec at GET /openapi.json (unauthenticated).

---

8. Going to production — checklist

• Durable storage. Open with .with_data_dir(...) (embedded) or --data-dir (server), not ephemeral. The WAL persists signals across restarts.
• Real embeddings. Replace the random vectors with your multimodal model's output. Same model for ingest and query. See docs/guides/embeddings.md.
• Auth on. Set TIDAL_API_KEY. An unset key means an unauthenticated server (it logs a WARN). Terminate TLS at your ingress.
• Lock down metrics. The --metrics Prometheus endpoint is unauthenticated — bind it to loopback or a cluster-internal address only, never the public interface.
• Graceful shutdown. On SIGTERM the server flips readiness to 503, drains in-flight requests, then checkpoints and fsyncs the WAL. Wire your orchestrator's preStop/termination grace period to allow the drain.
• Health probes. Use GET /health for readiness (200 ok / 503 draining), GET /health/startup and GET /health/live for startup/liveness.
• Observability. Scrape the metrics endpoint; import the dashboard and alerts from docs/ops/ (grafana-dashboard.json, prometheus-alerts.yaml, monitoring.md). Plan headroom with docs/ops/capacity-planning.md.
• Backup & recovery. Rehearse restore with docs/ops/recovery.md.
• Deployment. Follow docs/guides/server-deployment.md for packaging and rollout, and the Kubernetes runbook in docs/runbooks/ for cluster orchestration. The container images live at docker/ (standalone / cluster / deploy) — build from the repo root so COPY . . sees the whole workspace.

On cluster mode — be honest with yourself. tidalDB's cluster mode is experimental and currently runs all regions in a single process over loopback gRPC. It is not production multi-process HA (true multi-process HA is tracked as m8p10). It also replicates global signals only — user_id / creator_id contexts are rejected in cluster mode, so the personalized swipe loop (section 4) is a single-node feature today. Ship the For You feed on a vertically-scaled standalone node; reach for the experimental cluster only for read-scale experiments, per the cluster runbook.

---

See also

• QUICKSTART.md — the 5-minute ingest → rank loop.
• USE_CASES.md → UC-01 — product framing for the For You surface.
• docs/guides/embeddings.md — generating and writing real vectors.
• docs/guides/server-deployment.md — packaging and rollout.
• API.md — full embedded + HTTP API reference.
• tidal/examples/foryou_feed.rs — the verified, runnable version of this guide.