# Thoughts: What Three Databases Teach Us About Building TidalDB A synthesis from studying Engram (cognitive memory), Citadel (defensive logging), and StemeDB (knowledge graph) — three purpose-built databases in this codebase — and what their best practices, patterns, and gaps reveal about how TidalDB should be made. --- ## Part I: What Each Database Does That No One Else Does ### Engram — The Database That Forgets Engram's core insight is that **memory is not storage**. It rejects the 50-year ACID paradigm and models data the way human cognition actually works: memories decay, activate associatively, consolidate over time, and carry confidence rather than certainty. **What it does uniquely well:** - **Confidence as a first-class type.** Not a nullable float column — a zero-cost abstraction with logical operations (`.and()`, `.or()`, `.not()`, `.decayed()`). Every operation propagates uncertainty. This is the most elegant primitive in the entire codebase. When you query Engram, you don't get "the answer" — you get an answer with a confidence interval and an activation path showing *how* the system arrived at it. - **Spreading activation as a database primitive.** Recall is not a lookup — it's a parallel graph traversal where activation propagates through semantic relationships with exponential attenuation per hop. The `ParallelSpreadingEngine` (98KB of lock-free work-stealing code) is effectively a custom graph compute engine embedded in the storage layer. This is 62x faster than Neo4j because it isn't a general graph engine — it's a spreading activation engine that happens to operate on graph structure. - **Biological decay functions.** Hippocampal fast decay (hours), neocortical slow decay (months/years), the SM-18 two-component model, individual cognitive differences — all validated against 40+ years of psychology research (Ebbinghaus 1885, Bahrick 1984, McGaugh 2004). Decay is applied at *query time*, not storage time. The data doesn't rot — its accessibility does. - **Graceful degradation instead of failure.** Operations never return errors in the traditional sense. They return an `Activation` score (0.0-1.0) that indicates quality. Memory pressure doesn't cause failures — it reduces activation. This is a fundamentally different contract than ACID. - **Cache-line aligned hot nodes.** `CacheOptimizedNode` is `#[repr(C, align(64))]` — exactly one L1 cache line. Atomic fields for activation, confidence, visits, source count. No false sharing. This is hardware-aware database design. **The lesson for TidalDB:** The most important thing Engram teaches is that **temporal behavior belongs in the type system, not in application code**. Engram doesn't have a "decay function you call" — decay is intrinsic to how data *is*. TidalDB's signals should work the same way. A signal's velocity, decay rate, and windowed aggregation semantics should be declared in schema and computed at query time. The application should never implement `trending_score = views_24h / (age_hours + 2)^1.8`. That formula should be a named sort mode the database executes. --- ### Citadel — The Database That Never Drops a Log Citadel's core insight is that **durability is not negotiable, but everything else is**. It inverts the normal database priority stack: instead of optimizing for query speed first and hoping durability works, it guarantees durability first (quarantine-fsync-before-ACK) and then optimizes query speed within that constraint. **What it does uniquely well:** - **Quarantine-first architecture.** Every log is written to a binary journal with BLAKE3 checksums and fsynced before the client gets an ACK. The quarantine journal is the *durability boundary* — everything downstream (tiered storage, query engine, episode detection) is optimization, not correctness. If the entire tiered storage system explodes, the quarantine journal still has every log. - **Group commit for amortized fsync.** Individual fsync per write is O(n) in syscall overhead. Citadel's `GroupCommitQueue` batches writes and amortizes fsync cost to O(1) per batch — default 100 writes or 10ms, whichever comes first. This is the same technique used by PostgreSQL's commit delay, but Citadel makes it explicit and configurable per durability level (`Immediate`, `Batched`, `Eventual`). - **Per-tenant filesystem isolation.** Each tenant gets separate directories, separate journal files, separate quotas with atomic counters (`AtomicU64`). The philosophy: "Every tenant is an island. Noisy neighbors die alone." This is simpler and more reliable than any logical isolation scheme — the OS enforces the boundary. - **Append-only design eliminates transaction complexity.** No updates, no deletes, no conflicts, no MVCC. Timestamp ordering provides causality. Crash recovery is trivial: replay the journal from the last checkpoint. This is the same insight that makes Kafka durable — append-only logs are inherently crash-safe. - **Parquet + ZSTD for analytical storage.** Column-oriented storage with dictionary encoding for high-cardinality fields and ZSTD compression achieves 3-10x compression ratios. This is the right format for log data — wide, sparse, read-heavy after initial ingest. **The lesson for TidalDB:** Citadel teaches that **the write path and the durability boundary must be separated from the query optimization path**. TidalDB ingests engagement signals at high velocity — views, likes, skips, completions — and these *cannot be dropped*. A quarantine-first approach where signals are durably recorded before any processing begins would let TidalDB guarantee that every engagement event is captured, even if the signal aggregation system is behind. The `DurabilityLevel` enum (`Immediate`/`Batched`/`Eventual`) is also directly applicable: a "like" signal needs `Batched` durability; an "impression" signal can tolerate `Eventual`. --- ### StemeDB — The Database That Holds Contradictions StemeDB's core insight is that **truth is not singular**. Instead of requiring consensus at write time (the ACID model: only one value can exist for a key), it appends immutable assertions and resolves conflicts at *read time* using pluggable lenses. Contradictions coexist in storage. Resolution is a query-time decision. **What it does uniquely well:** - **Lenses as pluggable resolution strategies.** `Consensus`, `Recency`, `HlcRecency`, `VoteAwareConsensus`, `TrustAwareAuthority`, `EigenTrustAuthority`, `Skeptic` — each lens answers a different question about "what is true?" The same data returns different answers depending on which lens you apply. This is not a bug — it's the core design. The `Skeptic` lens returns *all* candidates with a conflict score, surfacing uncertainty rather than hiding it. - **Content-addressed storage with BLAKE3.** Every assertion's ID is a BLAKE3 hash of its content. Deduplication is automatic. Immutability is guaranteed. Provenance is traceable. This is the Git model applied to knowledge — every claim has a hash, a parent hash, a source hash, and cryptographic signatures. - **Hybrid storage backend routing.** Fjall (LSM-tree) for write-heavy append data (assertions, votes). Redb (B-tree) for read-heavy random-access data (indexes, materialized views, trust ranks). A `route()` function dispatches by key prefix. This is the most pragmatic storage decision in the codebase — rather than forcing one storage engine to be good at everything, pick two and route intelligently. - **Materialized views with changelog.** The `Materializer` background worker continuously recomputes lens-resolved winners and writes them to `MV:{subject}:{predicate}` for O(1) reads. But it also maintains `MVC:{subject}:{predicate}:{timestamp}` — a changelog showing *when the winner changed*. This means you can answer "what did we think was true last Tuesday?" — a question most databases can't touch. - **Subject-prefix key encoding for sharding.** All keys follow `{subject}\x00{TAG}:{suffix}`. This means a prefix scan on `{subject}\x00` returns every assertion, vote, index, and materialized view for that subject. Natural shard boundaries. Co-located data. No joins. - **CRDT replication.** Assertion stores use G-Set semantics (union merge). Vote counts use G-Counter semantics (max merge). This means replication is conflict-free by construction — no distributed consensus required for writes. Combined with HLC timestamps for causal ordering, StemeDB achieves eventual consistency without Raft for the data plane. **The lesson for TidalDB:** StemeDB teaches that **materialization is the bridge between write flexibility and read performance**. TidalDB will have signals arriving continuously, and ranking queries that need sub-10ms response. The answer is the same: materialize aggressively in the background, serve from materialized state on the fast path, and fall back to on-demand computation when materialized state is stale. StemeDB's `MV:` pattern with changelog is directly applicable to TidalDB's signal windowed aggregates — pre-compute `view_velocity_1h`, `view_velocity_24h`, `view_velocity_7d` in a background materializer, serve them as O(1) lookups at query time. The hybrid backend routing is also critical. TidalDB has the same dual personality: high-velocity append writes (signal events) and low-latency random reads (ranking queries). An LSM-tree for the signal event log and a B-tree for entity metadata + materialized aggregates is the right architecture. --- ## Part II: Patterns That Recur Across All Three These are not coincidences. When three independent databases converge on the same solution, it's a signal about the fundamental requirements of purpose-built data systems. ### 1. WAL + fsync Is the Durability Primitive All three databases use a Write-Ahead Log with explicit fsync control: | Database | WAL Implementation | Checksum | fsync Strategy | |---|---|---|---| | Engram | Custom ring buffer, 64-byte cache-aligned headers | CRC32C (hardware-accelerated) | Configurable: PerWrite, PerBatch, Timer, None | | Citadel | Quarantine journal, length-prefixed records | BLAKE3 | Configurable: Immediate, Batched, Eventual | | StemeDB | Segment-based WAL, v2 format | CRC32C + BLAKE3 | Immediate (default) | **The convergence:** Durability is not the storage engine's job — it's the WAL's job. The storage engine is an optimization layer over a durable log. Crash recovery means "replay the WAL from the last checkpoint." This is PostgreSQL's insight, Kafka's insight, and now it's ours. **For TidalDB:** Signal events should be durably logged *before* any signal aggregation occurs. The aggregation system can crash, restart, and replay from the WAL. The WAL *is* the source of truth; the signal ledger is a materialized view over it. ### 2. Tiered Storage Is Universal | Database | Hot | Warm | Cold | |---|---|---|---| | Engram | DashMap (in-memory, ~1ms) | Memory-mapped append log (~10ms) | Columnar + SIMD (~100ms) | | Citadel | Parquet/NVMe (7 days) | HDD/SSD (30 days) | S3/GCS (1+ year) | | StemeDB | Redb B-tree (indexes) | Fjall LSM-tree (assertions) | Not implemented | **The convergence:** Data has a lifecycle. Recent data needs speed. Old data needs economy. Migrating between tiers based on access patterns (not just age) is how you serve both needs. Engram's approach is the most sophisticated — migration is based on *cognitive access patterns*, not timestamps. **For TidalDB:** Signals have natural tiers. The 1-hour view velocity is hot (computed continuously, served from memory). The 7-day aggregate is warm (updated periodically, served from fast storage). The all-time total is cold (append-only counter, served from durable storage). TidalDB should tier signal aggregates by window, not by age. ### 3. Lock-Free Concurrency on the Hot Path All three avoid mutexes in their performance-critical paths: - **Engram:** `AtomicF32` for activation levels, CAS loops for concurrent spreading activation, `DashMap` for hot tier - **Citadel:** `AtomicU64` for per-tenant quota tracking (Relaxed ordering), semaphore-based query gating - **StemeDB:** `fetch_and_add_u64` for vote counts, `compare_and_swap_f32` for aggregate weights, `AtomicBool` for shutdown **The convergence:** When you need millions of updates per second (activation propagation, signal counting, log ingestion), mutex contention is the bottleneck. Atomic operations with careful memory ordering (Relaxed for counters, Acquire/Release for synchronization) eliminate contention at the cost of code complexity. **For TidalDB:** Signal counters (view_count, like_count) and windowed velocity computations must be lock-free. A `like` event should increment an atomic counter, not acquire a write lock. The ranking query should read these atomics without blocking writers. ### 4. Domain-Specific Query Languages, Not SQL None of these databases use SQL. Each invented a query interface that matches its domain: - **Engram:** `RECALL`, `SPREAD`, `CONSOLIDATE`, `COMPLETE`, `IMAGINE` — cognitive operations - **Citadel:** CPL (Citadel Pipe Language) — `| filter level="error" | stats count by service` - **StemeDB:** Subject/predicate/lens queries — `query(subject="X", predicate="Y", lens=Consensus)` **The convergence:** SQL is a general-purpose query language for general-purpose databases. When your database has domain-specific semantics (decay, velocity, lenses, spreading activation), SQL becomes a hindrance — it can express the query but it cannot *optimize* for the domain. A custom query interface lets the database understand *intent*, not just structure. **For TidalDB:** The query language in VISION.md is already right: ``` RETRIEVE items FOR USER @user_id CONTEXT feed USING PROFILE for_you FILTER unseen, unblocked, format:video, duration:short DIVERSITY max_per_creator:2, format_mix:true LIMIT 50 ``` This cannot be expressed in SQL without losing the semantics. `FOR USER` means "load this user's preference vector and relationship graph." `USING PROFILE` means "apply this named scoring function." `DIVERSITY` means "enforce these post-ranking constraints." These are not WHERE clauses — they are database primitives. ### 5. Append-Only / Immutable Core With Mutable Views | Database | Core Storage | Mutable Layer | |---|---|---| | Engram | Append-only WAL | Activation levels (mutable atomics), tier placement | | Citadel | Append-only quarantine journal | Parquet segments (immutable once written), compaction | | StemeDB | Append-only assertions | Materialized views (recomputed), vote counters (CAS) | **The convergence:** The source of truth is immutable. Mutability is confined to derived state that can be recomputed from the immutable core. This makes crash recovery trivial, replication simple, and auditing possible. **For TidalDB:** Signal events (a user liked an item at timestamp T) are immutable facts. Signal aggregates (this item has 1,247 likes in the last 24h) are mutable derived state. Ranking profiles are immutable declarations. Ranking results are ephemeral computations. Keep these layers distinct. ### 6. Content Addressing / Hash-Based Identity | Database | Hash Algorithm | What Gets Hashed | |---|---|---| | Engram | Content hashing for deduplication | Memory content for semantic dedup | | Citadel | BLAKE3 checksums | Every log record | | StemeDB | BLAKE3 for assertion ID | Assertion content, source evidence, vote identity | **The convergence:** When identity is derived from content rather than assigned by sequence, deduplication is automatic, immutability is verifiable, and distributed systems don't need coordination to generate IDs. **For TidalDB:** Signal events should be content-addressed. A "user U liked item I at time T" event has a deterministic hash. If the same event arrives twice (duplicate webhook, retry), it's automatically deduplicated. Item and user entity IDs can remain application-assigned, but signal event IDs should be content-derived. --- ## Part III: The Gaps — What None of Them Solve These are the problems TidalDB must solve that Engram, Citadel, and StemeDB do not address. These gaps define TidalDB's unique contribution. ### Gap 1: Temporal Signals as Schema-Level Primitives Engram has decay functions validated against psychology research. Citadel stores timestamped events. StemeDB has HLC timestamps and epoch-based paradigm tracking. But **none of them model signal velocity, windowed aggregation, or temporal decay as declarable schema primitives**. In all three, temporal behavior is *implemented* — but not *declared*. You can't say "this signal has a 7-day half-life and I want its 24-hour velocity" in schema. You write code that computes it. **TidalDB's answer:** Signals are a schema type: ``` DEFINE SIGNAL view ON item DECAY exponential HALF_LIFE 7d WINDOWS 1h, 24h, 7d, 30d, all_time VELOCITY enabled ``` The database maintains windowed aggregates and computes velocity automatically. The application writes `SIGNAL view item:@id user:@uid`. The ranking profile references `view.velocity(24h)`. No application code touches temporal math. ### Gap 2: Ranking as a Database Operation Engram ranks by spreading activation. StemeDB ranks by lens resolution. Citadel doesn't rank at all. But **none of them model multi-signal, multi-factor ranking with user context as a database primitive**. Ranking in content platforms requires combining: - Text relevance (BM25) - Semantic similarity (vector distance) - Temporal signals (velocity, decay) - User preferences (embedding similarity) - Social graph signals (who the user follows) - Negative signals (skips, hides, blocks) - Quality gates (completion rate floor) - Diversity constraints (max per creator) No existing database — general-purpose or purpose-built — combines all of these in a single query operation. **TidalDB's answer:** Named ranking profiles declared in schema: ``` DEFINE PROFILE for_you CANDIDATE ANN(user.preference_vector, item.embedding, top_k=500) BOOST view.velocity(24h) WEIGHT 0.3 BOOST user_creator.interaction_weight WEIGHT 0.2 BOOST social_proof(user, item) WEIGHT 0.15 DECAY item.age HALF_LIFE 48h GATE completion_rate > 0.3 PENALIZE skip.count(user, item) WEIGHT -0.5 EXCLUDE user_item.hidden = true EXCLUDE user_creator.blocked = true DIVERSITY max_per_creator: 2, format_mix: true EXPLORATION 10% ``` The application says `USING PROFILE for_you`. The database executes the entire pipeline. ### Gap 3: Closed-Loop Feedback Without ETL Engram updates activation on access, but doesn't model the feedback loop between ranking and engagement. StemeDB accepts votes that affect future lens resolution, which is the closest — but it's designed for knowledge claims, not engagement signals. Citadel is purely ingestion — no feedback path. **None of them close the feedback loop in a single write transaction.** In all three, "user engaged with content" and "content's ranking changes" are separate operations in separate systems or at separate times. **TidalDB's answer:** The engagement write path is the ranking update path: ``` SIGNAL like item:@item_id user:@user_id ``` This single write atomically: 1. Appends to the item's signal ledger 2. Updates windowed aggregates (like_count_1h, _24h, _7d) 3. Recomputes like_velocity 4. Updates user-to-item relationship weight 5. Increments user-to-creator interaction weight 6. Shifts user preference vector toward item's topic embedding The next ranking query — even 100ms later — reflects all of this. ### Gap 4: Diversity as a Query Constraint None of the three databases have any concept of result diversity. Engram returns by activation level. StemeDB returns the lens-resolved winner. Citadel returns matching logs. If you want "no more than 2 items per creator in the top 10 results," you implement that in application code. **TidalDB's answer:** Diversity is a first-class query parameter: ``` DIVERSITY max_per_creator:2, format_mix:true, topic_diversity:0.7 ``` The database enforces this after scoring but before returning results. The application never filters or reorders. ### Gap 5: Cold Start as a Database Responsibility None of the three databases handle cold start. Engram has no concept of "new content with no activation history." StemeDB treats new assertions the same as established ones. Citadel doesn't rank at all. **TidalDB's answer:** New items get an exploration budget declared in the ranking profile. New users get a sensible default experience based on population-level signals. The application doesn't manage either of these — the database does. ### Gap 6: Negative Signals as Equal Citizens Engram has decay (forgetting) but not explicit negative feedback. StemeDB has votes but treats them as weighted agreement, not negative engagement signals. Citadel has no concept of sentiment. **TidalDB's answer:** Skips, hides, blocks, "not interested," downvotes, mutes — these are not the absence of positive engagement. They are data. They carry the same weight and precision as likes. They update the system with the same immediacy. A "hide" creates a permanent hard-negative on the user-item relationship. A skip within 3 seconds is a strong quality signal. The ranking function treats these as first-class inputs. --- ## Part IV: Progressive Thinking About How Databases Should Be Made The three databases in this codebase, combined with the history of database systems, reveal a progression in how we think about what a database *is*. TidalDB sits at the leading edge of this progression. ### Stage 1: Storage (1970s-1990s) **Question:** How do I durably store and retrieve rows? **Answer:** B-trees, ACID transactions, SQL, relational model. **Assumption:** The application knows what it wants. The database stores and retrieves. Computation happens elsewhere. Every database from this era (Oracle, MySQL, PostgreSQL) treats data as inert. You put it in, you get it out, unchanged. The database's job is correctness and durability. ### Stage 2: Indexing (2000s-2010s) **Question:** How do I find data quickly across many dimensions? **Answer:** Inverted indexes (Elasticsearch), column stores (Vertica), distributed key-value (DynamoDB), graph traversal (Neo4j), vector indexes (Pinecone). **Assumption:** The application still knows what it wants, but the *shape* of the question has changed. Full-text search, graph queries, nearest-neighbor lookup — these can't be expressed as B-tree lookups. New index structures enable new query shapes. This era fragmented the database landscape. Each new query shape got a new database. The result: every application runs 3-7 databases, and the seams between them are where correctness dies. ### Stage 3: Domain-Specific Semantics (2020s — where we are) **Question:** What if the database understood the *meaning* of my data, not just its shape? **Answer:** Engram understands memory and forgetting. Citadel understands logs and durability. StemeDB understands truth and conflict. TidalDB will understand content, users, signals, and ranking. **Assumption:** The database and the application share a domain model. The database doesn't just store "a float" — it stores "a confidence score that decays over time and propagates through a graph." It doesn't store "a row" — it stores "a signal event that updates windowed aggregates and shifts a user preference vector." This is what makes purpose-built databases valuable: **the query optimizer can reason about domain semantics, not just data structure.** When TidalDB sees `USING PROFILE trending`, it doesn't plan a generic sort — it knows to use velocity signals, apply windowed aggregation, skip total-count indexes, and enforce per-creator diversity. A general-purpose database would execute the same computation as an opaque UDF with no optimization possible. ### Stage 4: Closed-Loop Systems (where TidalDB should aim) **Question:** What if the database didn't just respond to queries — but *learned* from them? **Answer:** The feedback loop between "what the user sees" and "what the system knows about the user" is not ETL. It is a database primitive. In Stage 3 databases, queries read and writes write. They're separate paths that the application stitches together. In Stage 4, the distinction blurs: - A ranking query is also an implicit write (the user was *shown* these items — that's an exposure event) - An engagement signal is also an implicit read (the system must *load* the user's preference vector to update it correctly) - A diversity constraint is also a *state query* (the system must know what the user has already seen) TidalDB should be a Stage 4 database. The query `RETRIEVE items FOR USER @u123` is simultaneously: 1. A read (fetch candidates, score them) 2. A write (record which items were shown, update exposure counts) 3. A state transition (the user's "seen" set now includes these items) The application doesn't manage this. The database does. ### What This Means Architecturally A Stage 4 database has properties none of the three current databases fully exhibit: 1. **The write path and read path share state.** Not "eventually consistent" sharing via replication — *same process, same memory, same transaction*. When a signal event arrives, the ranking query that runs 10ms later sees it. 2. **Schema encodes behavior, not just shape.** A signal declaration includes decay rate, velocity computation, and windowed aggregation. A ranking profile includes scoring weights, quality gates, and diversity rules. These are not configuration files — they are part of the schema, versioned with the data. 3. **The database manages derived state.** Windowed aggregates, velocity computations, user preference vectors, relationship weights, materialized rankings — these are not application caches. They are database-managed derived state with defined consistency guarantees. 4. **Cold start is a first-class concern.** The database knows when an entity is new and has no signals. It applies exploration budgets, default profiles, and population-level priors without application intervention. 5. **Negative signals are structurally equal to positive signals.** Not an afterthought column, not a filter — a full signal type with decay, velocity, and ranking weight. --- ## Part V: Concrete Recommendations for TidalDB Drawing from all three databases, these are the architectural decisions TidalDB should make: ### Steal From Engram 1. **Confidence/quality as a zero-cost type.** Engram's `Confidence` type with `.and()`, `.or()`, `.decayed()` is the right pattern. TidalDB's signal values should have similar ergonomics — `signal.velocity(24h)`, `signal.windowed(7d)`, `signal.decayed(half_life)`. 2. **Cache-line aligned hot data.** Any data touched on every ranking query (entity metadata, signal aggregates, user preference vectors) should be `#[repr(C, align(64))]`. Engram proved this matters for concurrent access patterns. 3. **Graceful degradation.** Under load, TidalDB should return slightly less precise rankings, not errors. Reduce candidate set size, use coarser signal aggregates, skip diversity enforcement — but always return results. 4. **Three-tier storage with cognitive migration.** Not time-based tier migration — *access-pattern-based*. A video that's 6 months old but still getting steady views stays in the hot tier. A video from yesterday that no one watched moves to warm. ### Steal From Citadel 5. **Quarantine-first signal ingestion.** Every engagement event is durably logged (fsync'd) before any processing. The signal aggregation system consumes from this durable log. If aggregation crashes, replay from the log. Never lose a signal. 6. **Group commit for signal events.** Individual fsync per signal event is too expensive at scale. Batch signals and fsync per batch (100 events or 10ms). Configurable per signal type — a purchase event gets `Immediate`; an impression gets `Eventual`. 7. **Per-entity isolation in storage.** Citadel isolates per-tenant. TidalDB should isolate per-entity-type. Item signal ledgers, user preference vectors, and creator profiles should be in separate storage namespaces. A burst of signal events for a viral item should not slow down user profile reads. 8. **BLAKE3 checksums everywhere.** Not just for durability — for deduplication. Duplicate signal events (webhook retries, client double-submissions) should be silently deduplicated by content hash. ### Steal From StemeDB 9. **Hybrid storage backend.** LSM-tree (Fjall or similar) for the signal event log (write-heavy, append-only). B-tree (Redb or similar) for entity metadata, relationship graph, and materialized aggregates (read-heavy, random access). Route by key prefix. 10. **Background materializer for aggregates.** StemeDB's `Materializer` pattern — background worker that continuously recomputes materialized views — is the right architecture for signal windowed aggregates. The materializer computes `view_velocity_1h`, `like_count_7d`, `completion_rate_all_time` and writes them to O(1) lookup keys. Ranking queries read from materialized state. 11. **Changelog on materialized state.** When a signal aggregate changes significantly (a video's view velocity doubles), record that change with a timestamp. This enables "what was trending yesterday?" queries and debugging ranking behavior over time. 12. **Subject-prefix key encoding.** StemeDB's `{subject}\x00{TAG}:{suffix}` pattern gives TidalDB a template: `{item_id}\x00SIG:{signal_type}:{window}` for signal aggregates, `{user_id}\x00PREF:{dimension}` for user preferences. All data for one entity is co-located. Natural shard boundary. ### Invent for TidalDB 13. **Ranking profiles as versioned schema objects.** Not code. Not config files. Schema-level declarations that the query optimizer can reason about. Versioned alongside data. Swappable at query time by name. A/B testable by specifying `USING PROFILE for_you_v2`. 14. **Diversity enforcement as a post-scoring pass.** After candidates are scored, apply diversity constraints as a separate pass. This is not filtering (which reduces candidates) — it's reordering (which maintains the target result count while respecting constraints). Maximal marginal relevance (MMR) or similar. 15. **Exploration budget as a schema-level declaration.** New items get injected into a percentage of ranking results regardless of their signal state. The percentage decays as signals accumulate. This is declared per ranking profile, not managed by the application. 16. **User preference vector as a database-managed embedding.** When a user engages with content, the database shifts their preference vector toward the content's embedding (positive signal) or away from it (negative signal). The application doesn't compute this. The database does, with configurable learning rate and momentum. --- ## Part VI: What "Correctly Built" Looks Like If TidalDB follows these patterns, a content platform should be able to replace Elasticsearch + Redis + Kafka + a feature store + a vector database + a ranking service with a single process, a single query interface, and a single operational model. The test is simple. This query: ``` RETRIEVE items FOR USER @user_id CONTEXT feed USING PROFILE for_you FILTER unseen, unblocked, format:video, duration:short DIVERSITY max_per_creator:2, format_mix:true LIMIT 50 ``` Should execute in under 50ms, incorporate signals that were written 100ms ago, enforce diversity without application logic, handle cold-start items without application intervention, and return results that a user would describe as "it knows what I want." That is what six systems currently produce. It should be one query here.