stemedb/ai-lookup/services/materializer.md

Materializer

Last Updated: 2026-01-31 Confidence: High Status: Implemented

Summary

The Materializer is a background worker that pre-computes winning assertions for each subject+predicate pair, storing them at MV:{subject}:{predicate} for O(1) reads. It bridges the gap between O(N) lens resolution and sub-millisecond query latency.

Key Facts:

• Scans all SP: compound indexes to discover pairs
• Resolves each pair through an AsyncLens (default: VoteAwareConsensus)
• Stores MaterializedView with winner + provenance metadata
• step() for one-shot, run() for polling loop, run_notified() for event-driven mode
• Event-driven: IngestWorker signals tokio::sync::Notify on new data; Materializer reacts immediately
• QueryEngine uses fast-path: checks MV: key before falling back to SP: index
• Error-resilient: individual pair failures are logged and skipped

File Pointers:

• crates/stemedb-query/src/materializer.rs - Materializer worker
• crates/stemedb-core/src/types.rs - MaterializedView type

Storage Layout

Key Pattern	Value	Purpose
MV:{subject}:{predicate}	Serialized MaterializedView	Pre-computed winner + metadata

MaterializedView Type

pub struct MaterializedView {
    pub winner: Assertion,              // The resolved winner
    pub lens_name: String,              // Which lens produced this (e.g., "VoteAwareConsensus")
    pub resolution_confidence: f32,     // Confidence in the resolution [0.0, 1.0]
    pub candidates_count: usize,        // How many candidates were considered
    pub materialized_at: u64,           // When this view was last computed
}

Materializer Interface

pub struct Materializer<S> {
    store: Arc<S>,
    index_store: GenericIndexStore<Arc<S>>,
    lens: Box<dyn AsyncLens>,
}

impl<S: KVStore + 'static> Materializer<S> {
    /// Create with any AsyncLens implementation
    pub fn new(store: Arc<S>, lens: Box<dyn AsyncLens>) -> Self;

    /// One full materialization pass over all SP: pairs
    pub async fn step(&self) -> Result<MaterializeReport>;

    /// Materialize a single subject+predicate pair
    pub async fn materialize_pair(&self, subject: &str, predicate: &str)
        -> Result<Option<MaterializedView>>;

    /// Read a pre-computed view (O(1))
    pub async fn get_materialized_view(&self, subject: &str, predicate: &str)
        -> Result<Option<MaterializedView>>;

    /// Run continuously with configurable interval (polling mode)
    pub async fn run(&self, interval: Duration);

    /// Run in event-driven mode, triggered by IngestWorker notifications
    pub async fn run_notified(&self, notify: Arc<Notify>, max_interval: Duration);
}

Read Path Integration

The QueryEngine automatically uses the fast path when both subject and predicate are specified.

Fast Path (O(1)):

QueryEngine::execute() -> MV:{subject}:{predicate} -> MaterializedView.winner -> QueryResult

Slow Path (O(N)):

QueryEngine::execute() -> SP:{subject}:{predicate} -> [H:{hash}...] -> candidates -> filter -> QueryResult

The fast path is used when a materialized view exists and the winner matches query filters (lifecycle, epoch). The slow path is the fallback when no MV exists, the winner doesn't match filters, or only a subject is specified.

File: crates/stemedb-query/src/engine.rs — try_fast_path() method

Event-Driven Mode

The Materializer supports two operating modes:

Polling mode (run(interval)): Fixed-interval passes. Simple but wastes cycles when idle and adds latency after writes.

Event-driven mode (run_notified(notify, max_interval)): The IngestWorker signals a tokio::sync::Notify after each successful record ingestion. The Materializer awaits this signal, running a pass immediately when new data arrives. A max_interval timeout acts as a safety net for missed notifications.

IngestWorker::step() -> notify.notify_one() -> Materializer::run_notified() wakes -> step()

Wiring:

let notify = Arc::new(tokio::sync::Notify::new());
let worker = IngestWorker::new(journal, store.clone()).await?.with_notify(Arc::clone(&notify));
let materializer = Materializer::new(store, Box::new(lens));
// In separate tasks:
tokio::spawn(async move { worker.run().await });
tokio::spawn(async move { materializer.run_notified(notify, Duration::from_secs(30)).await });

File: crates/stemedb-ingest/src/worker.rs — with_notify() method

Design Rationale

Why a Background Worker?

Inline materialization (on every write) would:

1. Add latency to the write path
2. Create contention when many agents write simultaneously
3. Couple write and read concerns

The background worker approach:

1. Keeps the write path fast (append-only)
2. Batches resolution work efficiently
3. Tolerates temporary staleness (eventual consistency)

Why Store Metadata?

The MaterializedView includes lens_name, confidence, candidates_count, and materialized_at because:

1. Provenance: Agents can verify how truth was determined
2. Debugging: "Why does the system think Tesla's revenue is X?"
3. Staleness detection: Readers can check materialized_at to decide if a slow-path re-resolution is needed

Related Topics

• Ballot Box - Vote data consumed by the Materializer
• Storage - KV layout and key patterns
• Architecture - Section 3 (Write Path) and Section 4 (Read Path)