39ada28c6e
M2: RETRIEVE query pipeline with 5-stage execution (candidate → filter → score → diversify → limit),
usearch HNSW vector index, bitmap/range/universe filters, ranking profiles with signal scoring,
MMR diversity enforcement, and m2_uat integration tests.
M3: Entity system with typed metadata, relationship graph (follows/blocks/interactions),
creator entities, session tracking, and m3_uat integration tests.
M4: Advanced ranking with builtin functions (freshness, trending, controversy, wilson),
ranking executor with explain mode, query executor integration, benchmarks for
query/ranking/vector/filters/diversity, and m4_uat integration tests.
Includes: 9 new blog posts, marketing site updates, updated roadmap, and updated vision doc.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
114 lines
6.9 KiB
Plaintext
114 lines
6.9 KiB
Plaintext
---
|
|
title: "Why we're building tidalDB"
|
|
date: "2026-02-20"
|
|
author: "Jordan Washburn"
|
|
description: "tidalDB is a single-process Rust database for personalized content ranking. Here is what it does and how it works."
|
|
tags: ["vision", "architecture"]
|
|
---
|
|
|
|
tidalDB is a database that answers one question: given this user, right now, what should they see?
|
|
|
|
Agents now sit between the user and many surfaces, so session memory still matters. But the core focus is personalized content ranking. tidalDB is not trying to out-feature every search platform. It is a database where signals, decay, negative feedback, and diversity are schema-level primitives — and ranking updates immediately after user actions.
|
|
|
|
I wrote separately about [why every content platform ends up operating six systems](/blog/every-platform-builds-the-same-6-systems) to answer that question. This post is about what we are building instead.
|
|
|
|
## The primitives
|
|
|
|
tidalDB has five core concepts. Everything else follows from them.
|
|
|
|
**Entities** are Items, Users, and Creators. Each carries metadata, an embedding slot, and a signal ledger. You define them in schema with typed fields — text fields are full-text indexed, keyword fields are filterable, embeddings are ANN-indexed. The database owns the indexes.
|
|
|
|
**Signals** are typed, timestamped event streams with decay and velocity built in. You declare a signal type once:
|
|
|
|
```rust
|
|
db.define_signal(SignalDef {
|
|
name: "view",
|
|
target: EntityKind::Item,
|
|
decay: Decay::Exponential { half_life: Duration::days(7) },
|
|
windows: vec![
|
|
Window::hours(1),
|
|
Window::hours(24),
|
|
Window::days(7),
|
|
Window::all_time(),
|
|
],
|
|
velocity: true,
|
|
})?;
|
|
```
|
|
|
|
That declaration tells the database everything it needs. When a view event arrives, the database maintains windowed counts, computes velocity, and applies exponential decay — all at write time, all O(1). You never compute `trending_score = views / (age_hours + 2)^1.8` in application code. You never update a stale float field on a cron schedule. The database does this natively, and it does it correctly.
|
|
|
|
Negative signals — skips, hides, blocks — are the same type. A skip is not the absence of a like. It is data with its own decay rate and its own weight in the scoring function.
|
|
|
|
**Ranking Profiles** are named, versioned scoring functions declared in schema. They reference signals, relationship weights, recency curves, and diversity rules. You swap profiles at query time by name — no redeploy, no recompile. This is how you A/B test ranking: two profiles, one query parameter.
|
|
|
|
**Sessions** capture agent context. A session binds a user, an agent identity, and a short-lived memory lane. Agents append structured signals (preference hints, reward scores, tool metadata) with aggressive decay while policies live in schema: what an agent can read, how often it may write, how long data persists.
|
|
|
|
**The query** brings it together. Candidate retrieval, filtering, personalized ranking, and diversity enforcement in a single operation:
|
|
|
|
```
|
|
RETRIEVE items
|
|
FOR USER @user_id
|
|
FOR SESSION @session_id
|
|
CONTEXT feed
|
|
USING PROFILE for_you
|
|
FILTER unseen, unblocked, format:video, duration:short
|
|
DIVERSITY max_per_creator:2, format_mix:true
|
|
LIMIT 50
|
|
```
|
|
|
|
One call. No network hops between subsystems. No merging results from five data sources. The database handles retrieval strategy (ANN, BM25, graph walk, full scan), applies hard filters, scores candidates against live signal state, enforces diversity constraints, and returns a ranked list. The agent gets the list along with a session snapshot (top signals, reward velocity, last tool it used) so it can explain its answer.
|
|
|
|
## The feedback loop
|
|
|
|
This is the part that makes the architecture honest.
|
|
|
|
When a user likes an item, the database atomically updates the item's signal ledger, the user's preference vector, and the user-to-creator relationship weight. All in the same write transaction. The next ranking query — even 100ms later — reflects the updated state.
|
|
|
|
```rust
|
|
db.signal(Signal {
|
|
kind: "like",
|
|
item: "item_abc",
|
|
user: "user_123",
|
|
session: Some("session_xyz"),
|
|
timestamp: Utc::now(),
|
|
weight: 1.0,
|
|
metadata: Some(json!({ "agent": "assistant", "tool": "planner" })),
|
|
})?;
|
|
```
|
|
|
|
There is no event bus between the engagement and the ranking update. No consumer lag. No cache to invalidate. The write path and the read path are one system. A user who skips three items in a row sees the fourth query adjust — not after a batch pipeline runs, not after a feature store syncs. Now.
|
|
|
|
## Where we are deliberately narrow
|
|
|
|
If your primary problem is operating a large, general search serving platform, systems like Vespa are excellent and mature.
|
|
|
|
Our wedge is narrower and opinionated:
|
|
|
|
- Optimize for the personalization loop, not broad search platform parity.
|
|
- Make negative feedback intent explicit and immediate:
|
|
`skip_for_now` (soft), `not_for_me` (preference), `low_quality` (quality), `hide/mute/block` (hard excludes).
|
|
- Treat "next refresh reflects feedback" as a hard product promise, not a best effort.
|
|
- Keep the first deployment embeddable and in-process for low-latency iteration.
|
|
|
|
## Where the build stands
|
|
|
|
tidalDB is early. I want to be direct about what exists today and what does not.
|
|
|
|
**Built:** Schema system with entity, signal, and profile definitions. Write-ahead log with segment rotation, checksummed records, BLAKE3 deduplication, and crash recovery. Storage engine backed by fjall with trait abstraction, key encoding, and batch writes. Signal ledger with forward-decay scoring, hot-path state, and warm-path persistence.
|
|
|
|
**Next:** Query engine — the RETRIEVE/SEARCH/SUGGEST operations with the execution pipeline described above. Then session-aware APIs, agent policies, vector search (USearch), text search (Tantivy), and hybrid fusion. Then the full query surface with all sort modes and diversity enforcement.
|
|
|
|
The foundation is Rust, single-node, embeddable. The storage layer is designed for horizontal scaling later — key encoding and storage isolation are partition-ready — but single-node correctness comes first. This is how we differentiate from Vespa, Milvus, or any search-first system: tidalDB embeds inside your agent runtime, exposes a declarative query+session API, and guarantees every signal the agent writes is visible on the next read without a distributed hop.
|
|
|
|
The code is on [GitHub](https://github.com/orchard9/tidalDB). Every architectural decision gets documented.
|
|
|
|
## Why open source
|
|
|
|
The personalized content ranking problem is universal. Every content platform needs it. The solution should be a tool you embed in your process and point at your data — not a vendor you depend on for a query you could run locally.
|
|
|
|
MIT licensed. No asterisks.
|
|
|
|
---
|
|
|
|
*If you want the full diagnosis of why the 6-system stack exists and where correctness fails between the seams, read [Every content platform builds the same 6 systems from scratch](/blog/every-platform-builds-the-same-6-systems).*
|