tidaldb/docs/planning/architecture-review.md

# Architecture Review: The "Materialized Views Over Event Stream" Reframing

**Date:** 2026-02-20
**Author:** @tidal-visionary (Spencer Kimball)
**Status:** Assessment

---

## Context

After the product owner introduced five new requirements -- cohorts as a first-class primitive, three-layer trending, dynamic cohorts, rich user model, and query composition -- the engineering team produced 14 detailed specifications (01-14) and a revised roadmap (M1-M7). A subsequent architectural review proposed reframing the entire system around a single insight:

> The signal ledger IS a materialized view. Cohort-scoped signals are just more materialized views over the same event stream.

This reframing proposes collapsing 14 specs into 10 subsystems centered on a generalized `Materializer<Scope>` abstraction, where a single event stream feeds multiple materializers keyed by different scopes (Global, Cohort, User, Relationship), and a columnar event store retains events with full user context for GROUP BY operations.

This document assesses whether this reframing holds, how it changes the product, and what I would do differently.

---

## Question 1: Does the "Materialized Views Over Event Stream" Reframing Hold?

**Yes. But with an important qualification: the existing specs already embody this pattern. The reframing names something that is already true, not something that needs to change.**

Let me be precise about what the existing architecture says:

- **Spec 01 (Storage Engine):** "The WAL is the source of truth. Everything else is derived state." This is the event stream.
- **Spec 03 (Signal System), Section 3:** "Immutable events, mutable aggregates." This is the materialized view pattern.
- **Spec 03, Section 7:** The hierarchical dimensional rollup system already materializes the same event stream into Level 0 (global), Level 1 (region/language/age), and Level 2 (behavioral segments) views. Each level is a materialized view keyed by a different scope.
- **Spec 03, Section 9:** The background materializer already performs bucket rotation, rollup generation, checkpointing, and cohort segment recomputation -- exactly the responsibilities a generalized materializer framework would have.
- **Spec 10 (Feedback Loop), Section 2:** The seven-step signal ingestion pipeline shows a single event atomically updating the signal ledger (global view), user preference vector (user view), relationship weight (relationship view), cohort counters (cohort view), and user state (user-item view).

The proposed reframing says "these are all materialized views." The existing specs say "the WAL is truth, everything else is derived, and here is exactly how each derived state is updated." These are the same statement expressed in different vocabularies.

**Where the reframing adds genuine value:**

1. **It names the abstraction.** The existing specs describe five separate update paths (signal ledger, user preference, relationship, cohort, user state) without calling them instances of the same pattern. A `Materializer<Scope>` trait would make the shared structure explicit in code.

2. **It clarifies the extension model.** If someone asks "how do I add a new kind of derived state?" the answer with the reframing is clear: implement `Materializer<YourScope>` and register it with the event stream. Without the reframing, the answer is "find all the places in the seven-step pipeline where state is updated and add another step."

3. **It justifies the columnar event store.** The current specs store signal events with minimal context (item_id, user_id, signal_type, weight, timestamp, context blob). The reframing argues for storing events with full user attributes at write time, enabling retrospective cohort analysis without joining against user state. This is a genuine architectural addition.

**Where the reframing overstates its case:**

The proposal implies the existing architecture is organized wrong ("14 specs vs. 10 subsystems"). In fact, the existing architecture already has one event stream (WAL) feeding multiple derived state stores with different keys and different update logic. The specs are organized by domain concern (signals, entities, relationships, cohorts, text, vectors, queries, ranking, feedback, cold start, concurrency, schema, scale), not by storage topology. This is deliberate and correct -- domain organization is what engineers need when implementing and debugging. Storage topology is an implementation detail that emerges from the domain model.

**Verdict: The reframing is correct as an implementation insight. It should influence the trait design in Rust code. It should not drive a reorganization of the specification documents.**

---

## Question 2: Does the 10-Subsystem Decomposition Make More Sense Than 14 Specs?

**No. The 14 specs are the right organization. The 10-subsystem proposal conflates two different questions.**

The 14 specs answer: "What does each domain concept need to do, and what are its invariants?" The 10-subsystem proposal answers: "How should the code be organized at the module level?" These are different questions with different correct answers.

Here is what the 10-subsystem proposal merges:

| Proposed Subsystem | What It Combines | What Is Lost |
|---|---|---|
| Event Store | WAL (from spec 01) + signal events (from spec 03) | The WAL handles ALL mutations (entities, relationships, schema, signals, checkpoints). Calling it the "event store" and scoping it to signals misses that entity writes and relationship writes also go through the WAL. |
| Materializer Framework | Signal ledger (spec 03) + feedback loop (spec 10) + cohort attribution (spec 05) + background materializer (spec 03) | The feedback loop (spec 10) is not just materialization -- it defines the semantic mapping from signal types to preference vector directions, to relationship weight deltas, to user state transitions. These are domain rules, not materialization mechanics. |
| Entity Store | Entity model (spec 02) | Fine. |
| Cohort Engine | Cohorts (spec 05) | Fine. |
| Text Index | Text retrieval (spec 06) | Fine. |
| Vector Index | Vector retrieval (spec 07) | Fine. |
| Relationship Graph | Relationships (spec 04) | Fine. |
| Ranking Engine | Ranking (spec 09) + cold start (spec 12) | Reasonable merge. |
| Query Engine | Query engine (spec 08) | Fine. |
| Schema System | Schema (spec 11) | Fine. |

**What is lost entirely:**

- **Concurrency spec (13):** The lock-free hot-path design, atomic CAS patterns, memory ordering rationale, and the DashMap sharding strategy for concurrent entity state access. This is not part of any of the 10 proposed subsystems. It is a cross-cutting concern that the 14-spec approach correctly isolates.

- **Scale architecture spec (14):** The four-tier scaling model (Seed/Growth/Scale/Hyperscale), resource estimates, and the single-node ceiling analysis. This is a product strategy document, not a subsystem. It belongs in its own spec.

- **Cold start spec (12):** The exploration budget, the cold-start item injection strategy, and the new-user fallback to population-level signals. The proposal absorbs this into the ranking engine, which is defensible but loses the explicit treatment of a critical product behavior.

**The real issue with the proposal:** It is optimizing for code module count. CockroachDB has far more than 14 packages in its codebase. The question is not "can we reduce the number of specs?" but "does each spec have a coherent responsibility and clear invariants?" The answer for the existing 14 is yes.

**What I would actually do:** Keep the 14 specs as domain-level documentation. In the Rust codebase, organize modules around the materializer insight where it improves code structure. These are compatible. The spec is not the code. The spec is the contract. The code is an implementation that satisfies the contract.

**Specific code structure recommendation:**

```
tidal/src/
  wal/              # Spec 01: WAL, segments, crash recovery
  storage/          # Spec 01: fjall/redb backend, key encoding
  entities/         # Spec 02: Item, User, Creator, embedding slots
  signals/          # Spec 03: Signal types, decay, velocity, windowed agg
    materializer.rs # The Materializer<Scope> trait lives here
    global.rs       # GlobalMaterializer (Level 0)
    cohort.rs       # CohortMaterializer (Levels 1-2)
    user.rs         # UserPreferenceMaterializer
    relationship.rs # RelationshipWeightMaterializer
  relationships/    # Spec 04: edges, weights, graph traversal
  cohorts/          # Spec 05: predicates, bitmaps, resolution
  text/             # Spec 06: Tantivy integration
  vectors/          # Spec 07: USearch integration
  query/            # Spec 08: parser, planner, executor
  ranking/          # Spec 09 + 12: profiles, scoring, diversity, cold start
  feedback/         # Spec 10: signal ingestion pipeline orchestration
  schema/           # Spec 11: validation, migrations
```

This gives you the materializer abstraction where it matters (inside `signals/`) without reorganizing the domain model.

---

## Question 3: How Does This Change the Roadmap?

**It does not change the milestone order. It adds one phase to Milestone 1 and refines one phase in Milestone 4.**

The existing roadmap (from `docs/planning/ROADMAP.md` as amended by `roadmap-cohort-analysis.md`) already has the right milestone sequence:

- M1: Signal Engine
- M2: Ranked Retrieval
- M3: Personalized Ranking (expanded with rich user model)
- M4: Cohort-Scoped Ranking (new)
- M5: Hybrid Search (expanded with query composition)
- M6: Full Surface Coverage
- M7: Production Hardening

**What changes:**

### M1: Add m1p3a -- Materializer Trait

Insert a small phase between m1p3 (Storage Engine) and m1p4 (Signal Ledger):

**m1p3a: Materializer Trait**
- Defines `Materializer<Scope>` with `on_event(&self, event: &WalEvent) -> Result<()>` and `checkpoint(&self) -> Result<()>` and `restore(&self, checkpoint: &[u8]) -> Result<()>`
- Defines `Scope` enum: `Global`, `User`, `Cohort`, `Relationship`
- `GlobalSignalMaterializer` is the first implementation (used by m1p4)
- The materializer registry is created (initially holding one materializer)
- Complexity: S

This is the "design for distribution from the start" principle applied to the materializer pattern. Building the trait now costs almost nothing. Retrofitting it into m1p4's signal ledger later costs a refactor of every call site.

### M3: m3p2 Becomes a Materializer Implementation

m3p2 (Feedback Loop -- Signal Writes Update User State) is currently specified as a monolithic change to the signal write path. With the materializer insight, this phase implements two new materializers:

- `UserPreferenceMaterializer` (updates preference vector on positive/negative signals)
- `RelationshipWeightMaterializer` (updates user-creator interaction weights)

Both register with the materializer registry. The signal write path does not change -- it calls `registry.on_event()` and all registered materializers are invoked. This is cleaner than the current spec's seven-step pipeline, which hardcodes each update step.

### M4: m4p2 Becomes a Materializer Implementation

m4p2 (Cohort-Scoped Signal Aggregation) -- already identified as XL complexity and the highest-risk phase -- implements `CohortMaterializer`. This materializer receives signal events, resolves the user's cohort memberships, and increments the appropriate dimensional rollup counters.

The materializer trait boundary means m4p2 can be developed and tested in isolation: give it a stream of events with user context, verify it produces correct cohort-scoped counters. It does not need to understand the signal ledger internals or the WAL format -- it receives typed events and produces typed state.

### What Does NOT Change

- M1 UAT is identical. The materializer trait is invisible to the UAT scenario.
- M2 UAT is identical. The materializer trait does not affect query execution.
- M5-M7 are unchanged.
- The milestone order is unchanged.
- The complexity estimates are unchanged (the Materializer trait is S; the cohort materializer remains XL).

**The columnar event store question:**

The reframing proposes retaining signal events with full user context in a columnar format for GROUP BY operations. This is the most substantive architectural addition. Here is my assessment:

- **Defer to M4.** The columnar event store is only needed for retrospective cohort analysis ("recalculate trending for a cohort that was defined after the events occurred"). During M1-M3, signal events are stored in the WAL (which is the event stream) and the cold-tier signal_events CF (which has item_id, timestamp, signal_type, user_id, weight, context). This is sufficient.
- **In M4, add user attributes to the cold-tier event format.** When cohort tracking is activated for an item, the signal write path already looks up `UserCohortMemberships`. Storing these 22 bytes alongside the event in the cold tier enables retrospective analysis without a full columnar store.
- **A full columnar event store (Arrow/Parquet-style) is a post-M7 concern.** The use case is offline analytics, not real-time ranking. The real-time path uses pre-computed dimensional rollups. Adding a columnar engine before M7 violates the "does the UAT require it?" test.

---

## Question 4: What Does This Do to the Product Story?

**The reframing strengthens the product story, but not in the way the proposal suggests.**

The proposal suggests the story becomes "one event stream, multiple materialized views." This is an architecture story. Users do not care about materialized views. They care about what the database does for them.

The real product story is unchanged and already excellent:

> Write a signal event. The database instantly updates the item's trending score, the user's preference vector, the relationship weight, and the cohort-scoped trending metrics. The next query, issued 100ms later, reflects all of these updates. No ETL. No Kafka. No feature store sync. No stale data. One write, six updates, zero application logic.

The materialized view insight strengthens this story by making it extensible:

> And when you define a new cohort, the database starts materializing that cohort's trending signals from the existing event stream. No backfill job. No pipeline reconfiguration. Define the cohort. Query it. Done.

This is the "define a cohort and it works immediately" story, which is genuinely new and powerful. It comes from the materializer framework, but the story is about the user experience, not the implementation pattern.

**The competitive positioning:**

The three-layer trending model (global, cohort, search-within-cohort) is a capability that Algolia, Typesense, Meilisearch, and Elasticsearch cannot offer at all. They have no concept of cohort-scoped signal aggregation. This is the strongest differentiator tidalDB has, and the reframing does not change it -- the existing specs already define it in detail (spec 03 Section 7, spec 05 Section 6).

The product story on the website should emphasize: "Define a cohort. See what is trending for that audience. Search within those trends." The implementation -- materialized views, dimensional rollups, independence estimation -- stays behind the curtain.

---

## Question 5: What Is the Biggest Risk?

**The biggest risk is not architectural. It is scope.**

The 14 specs total approximately 40,000 words of detailed specification. They describe a system that, when fully implemented, handles 15 use cases across 25+ sort modes with 40 signal types, cohort-scoped trending, query composition, cold start, graceful degradation, and crash recovery at 1M items.

This is an enormous amount of functionality for a product that has zero lines of implementation code.

CockroachDB's first release (beta, 2015) was a KV store with Raft consensus and basic SQL parsing. It did not have window functions, JSON support, change data capture, or geographic partitioning. Those came over years of iteration informed by real usage.

**The risk with tidalDB's current trajectory is that the specifications are so detailed and so comprehensive that the team feels obligated to implement all of them before shipping anything.** The specs describe the end state. The roadmap describes the journey. But the specs' level of detail creates pressure to get everything right before writing code.

**Specific risk items, ranked:**

1. **m4p2 (Cohort-Scoped Signal Aggregation) at XL complexity.** This is the longest pole in the roadmap and blocks the most downstream work. The dimensional rollup system with threshold-gated activation, hierarchical Level 0/1/2/3 aggregation, independence estimation for composites, and write amplification management is genuinely hard. The spec (03, Section 7) runs to 3000+ words of detailed design. The risk is that implementation reveals edge cases the spec did not anticipate, and the cohort system ships 2-3 months later than planned.

2. **The warm tier memory model.** Spec 03 Section 3 calculates that the warm tier at full population (10M entities, 6 signal types, 1.8KB per entity per signal) would require 108 GB. The solution is sparse allocation (only active entities). But the active/inactive boundary, eviction policy, and promotion-on-demand strategy are complex to implement correctly under concurrent read/write load. Getting this wrong means either excessive memory consumption or cold-read latency spikes.

3. **The preference vector update.** Spec 10 Section 3 describes shifting a 1536-dimension preference vector on every positive/negative signal. The learning rate, normalization, and convergence properties of this approach are not well-studied for the tidalDB use case. If the preference vector drifts too fast, the For You feed becomes unstable. If it drifts too slowly, it does not reflect recent interests. This is a machine learning tuning problem disguised as a database implementation problem.

4. **Query composition performance.** The three-layer query (`SEARCH items QUERY "piano" WITHIN TRENDING FOR COHORT young_us_jazz WINDOW 24h`) has a 50ms latency budget. Spec 05 Section 6.3 breaks this into: cohort resolution (2ms) + candidate generation (20ms) + text search within candidates (10ms) + ranking (5ms) + diversity (1ms) = 38ms. This is tight. If any step exceeds its budget, the entire query misses the target.

5. **Spec-to-implementation drift.** With 14 specs and 7 milestones, the probability that implementation reveals a design flaw in one spec that forces changes in 2-3 others is high. The specs cross-reference each other extensively (spec 03 Section 7 references spec 05, spec 08 references specs 01-07, spec 10 references specs 01-04). A change in one spec's invariants can cascade.

**The materialized view reframing does not change these risks.** The risks are in the domain complexity, not the code organization.

---

## Question 6: What Would I Change?

### 1. Implement M1 Now. Stop Specifying.

The specs are good enough. They are detailed enough to build from. The marginal value of further specification is negative -- it delays the feedback loop between design and implementation. m1p1 (Core Type System) is S complexity. m1p2 (WAL) is L complexity. m1p3 (Storage Engine) is M complexity. Start writing Rust.

The most valuable thing that can happen right now is discovering, in the first 1000 lines of Rust code, which assumptions in the specs are wrong. This always happens. CockroachDB's first key-value store invalidated several assumptions in the design document. The sooner you find these, the cheaper the corrections.

### 2. Add the Materializer Trait in M1, Not M3

As described in Question 3. This is an S-complexity addition that prevents an M-complexity refactor later. The trait is:

```rust
pub trait Materializer: Send + Sync {
    fn on_event(&self, event: &WalEvent) -> Result<()>;
    fn checkpoint(&self, writer: &mut dyn Write) -> Result<()>;
    fn restore(&self, reader: &mut dyn Read) -> Result<()>;
}
```

Three methods. Implement once for `GlobalSignalMaterializer` in M1. Add `UserPreferenceMaterializer` and `RelationshipWeightMaterializer` in M3. Add `CohortMaterializer` in M4. The trait boundary keeps each materializer testable in isolation.

### 3. Simplify the Warm Tier in M1

The warm tier spec (03, Section 3) describes per-minute and per-hour bucketed counters with SWAG stacks, EWMA smoothing, and Scotty stream-slicing. This is the correct end-state design, but it is too much for M1.

For M1, implement:
- Hot tier: running decay scores (cache-line aligned, atomic CAS). This is the core.
- Simple windowed counters: fixed-size circular buffer of per-minute counts. No SWAG stacks. No EWMA. No Scotty slicing. Just count events per minute, sum the last N minutes for an N-minute window.

This passes the M1 UAT. Windowed count will be exact. Velocity will be count/duration. The M1 integration test does not require EWMA or sub-minute granularity.

Upgrade to the full warm tier design in M2 or M3 when ranking profiles need it. The spec describes the end state. The implementation builds toward it incrementally.

### 4. Defer the Columnar Event Store Indefinitely

The reframing's most substantive proposal -- a columnar event store with full user context for GROUP BY -- is premature. Here is why:

- The real-time ranking path uses pre-computed dimensional rollups, not event scanning.
- Retrospective cohort analysis ("recalculate trending for a newly-defined cohort") can be served by scanning the cold-tier signal_events CF with a user_id join against current user attributes. This is slow (minutes, not milliseconds) but correct, and it is an admin/analytics operation, not a user-facing query.
- A columnar engine (Arrow, DataFusion, Polars) is a significant dependency with its own complexity. Adding it before there is a production workload that demands it is premature optimization of the analytics path at the expense of shipping the ranking path.

Store user cohort memberships (22 bytes) alongside signal events in the cold tier. This enables efficient retrospective filtering. A full columnar store can be added post-M7 when analytics requirements are concrete.

### 5. Reduce the Signal Type Count for M1-M2

Spec 03 Section 11 defines 40 signal types across 5 categories. For M1 and M2, implement 6: view, like, skip, share, completion, dwell_time. These cover the UAT scenarios for both milestones. The remaining 34 signal types add configuration complexity but no new code paths -- they use the same decay/velocity/windowed infrastructure.

Define the signal type registry so that adding new types is a schema operation, not a code change. Then add signal types as use cases demand them.

### 6. Keep 14 Specs, Add an Architecture Overview

The 14 specs are well-organized by domain concern. What is missing is a single document that shows how they connect -- the data flow from signal write to ranking query, touching all 14 specs in sequence. Add a document called `docs/specs/00-architecture-overview.md` that:

- Shows the single event stream (WAL) feeding multiple derived state stores
- Names the materialized view pattern explicitly
- Maps each spec to its role in the overall data flow
- Shows the dependency graph between specs

This gives the reader the forest before the trees. The 14 specs are the trees. Both are needed.

---

## Summary

| Question | Answer |
|----------|--------|
| Does the materialized view reframing hold? | Yes, but the existing specs already embody it. Name the pattern in code (Materializer trait), not in spec reorganization. |
| Is 10 subsystems better than 14 specs? | No. Keep 14 specs for domain documentation. Use the materializer insight for code structure within the signals module. |
| How does this change the roadmap? | Adds one S-complexity phase to M1 (Materializer trait). Refines M3 and M4 phases to use the trait. No milestone order changes. |
| What does this do to the product story? | Strengthens "define a cohort, see trends immediately." Does not change the core "replace 6 systems" narrative. |
| What is the biggest risk? | Scope. 14 detailed specs with zero implementation. The risk is specification paralysis, not architectural incorrectness. |
| What would I change? | Start implementing M1 immediately. Add Materializer trait in M1. Simplify the warm tier for M1. Defer columnar event store. Reduce initial signal types to 6. Add an architecture overview document. |

---

## The CockroachDB Parallel

When we built CockroachDB, the design document described Raft consensus, distributed SQL, range replication, and transaction isolation. The first thing we shipped was a monolithic key-value store that ran on one machine. It did not have Raft. It did not have distributed transactions. It had a RocksDB backend, a key-value API, and a test suite that proved the basics worked.

Every subsequent release added one layer from the design document. But -- and this is the critical point -- every layer was informed by what we learned building the previous one. The Raft implementation looked different from the design document because we discovered things about the key-value layer that the document did not anticipate. The SQL layer looked different because we discovered things about the Raft layer.

tidalDB is in the same position. The specs are the design document. They are good. They are detailed. They describe the right system. Now ship M1 and discover what the specs got wrong. The materialized view reframing is an insight that should live in the code, not in a specification reorganization. The 14 specs are the right documentation structure. The 7-milestone roadmap is the right delivery sequence.

The next step is not another architecture review. It is `cargo init`.