stemedb/vision.md

Episteme: The Probabilistic Knowledge Lattice

Internal Codename: StemeDB Category: Infrastructure / Database Role: The Cortex (Reasoning & Truth)

1. The Manifesto: "A Marketplace of Truth"

We are building the shared, long-term memory for autonomous research agents.

Current databases (Postgres, Neo4j, Vector DBs) suffer from The Tower of Babel problem: they store Data, not Evidence. They are deterministic, stateless, and brittle.

Episteme rejects the idea of a single, static "database state." Instead, it models knowledge as a Probabilistic Marketplace.

• Democracy: Truth is established via high-velocity consensus (Voting), not just overwrite privileges.
• Federation: "Waze for Deep Research." Free users contribute to a Global Lattice; Paid users get private silos.
• Economics: Reasoning has a cost. The system enforces efficiency via "The Meter."
• Curation: We are not the Ministry of Truth. We are the App Store for Trust. Users subscribe to "Trust Packs" (e.g., "Mayo Clinic", "Rust Experts") to filter reality.

2. The Core Data Model: The Hyper-Edge

The atomic unit of Episteme is not a Row, Document, or Embedding. It is the Signed Assertion.

struct Assertion {
    // The Proposition (The "What")
    subject: EntityId,       // e.g., "Tesla_Inc"
    predicate: RelationId,   // e.g., "has_annual_revenue"
    object: Value,           // e.g., "$96.7B"

    // The Meta-Cognition (The "Why")
    confidence: f32,         // 0.0 to 1.0 (Agent's subjective certainty)
    source_hash: Hash,       // Content-addressed link to source (PDF, URL, Log)
    visual_hash: Option<Hash>, // pHash for visual anchoring against web drift
    agent_id: PublicKey,     // Who made this claim? (Cryptographic multi-sig)
    timestamp: u64,          // When?
    
    // The Semantic Vector (The "Meaning")
    vector: Vec<f32>,        // Embedding for semantic navigation

    // The Paradigm (The "Context")
    epoch: Option<EpochId>,  // "covid-guidelines-2020", "gaap-2024"
}

3. The Query Engine: "Truth Lenses"

Reading is a compute-heavy operation. You must apply a Lens to collapse the probabilistic field into a concrete answer. To ensure sub-millisecond latency, Episteme uses Materialized Views to pre-calculate the results of standard lenses.

Standard Lenses

1. Lens::Consensus: Returns the value with the highest cluster density (Weighted by Trust Pack).
2. Lens::Authority: Filters by subscribed Trust Packs (e.g., "Show me reality according to The Financial Times").
3. Lens::Recency: Returns the latest assertion, ignoring history.
4. Lens::EpochAware: Validates assertions against the current paradigm, filtering superseded epochs.
5. Lens::Skeptic: Returns the variance between claims (identifies high-conflict/unstable truth).

4. Features for the Agentive Team

4.1. "Forking Reality" (Branching)

Agents need to simulate futures without polluting the main branch. Episteme supports Copy-on-Write Branching via Sparse Merkle Trees.

4.2. The Ballot Box: High-Velocity Consensus

To avoid write contention, Episteme separates the "Candidate" (Assertion) from the "Votes" (Signatures).

• Agents write Votes to a high-speed append-only log ("The Ballot Box").
• A background process aggregates these votes to update the Materialized View.

4.3. The Hive: Learning & Trust

• Trust Packs (The Curator Economy): Users can publish and subscribe to Lists of Trusted Agents.
  • Example: "The Skeptical Cardio Pack" filters out low-quality studies.
  • Mechanism: A BitSet overlay that filters the Consensus Lens efficiently.
• The Simulator (Mid-Training): A pipeline that converts high-confidence failure logs into Synthetic Trajectories.
• The Super Curator (Judicial Branch): A specialized swarm of "Reviewer Agents" that audits high-variance facts.

4.4. The Meter: Economics of Reasoning

Deep Research is computationally expensive. Episteme enforces Temporal Advantage Normalization (TAN).

• Budgeting: Every Job carries a budget (tokens/dollars).
• Throttling: The system rejects "Fork Reality" requests if the projected cost exceeds the Value of Information.

5. Architecture: The Rust Stack

Episteme follows the "Defensive by Default" best practices.

Tier 1: The Spine (Durability)

• Component: episteme-wal (Quarantine Pattern)
• Role: Raw, serialized append-only log. Ensures we never lose a claim.

Tier 2: The Lattice (Graph/Index)

• Component: episteme-core (Hot/Warm memory)
• Warm Tier: sled (LSM Tree) for the Merkle DAG + hnsw for vector search.
• Ballot Box: High-velocity stream for vote ingestion.

Tier 3: The Cortex (Compute)

• Component: episteme-lens
• Role: The WASM runtime for executing Lenses and resolving probabilistic state.
• Materializer: Background worker maintaining O(1) read views.

6. The Ecosystem Triad

System	Biological Analogy	Function	Question Answered
LogDB	The Spine	Immutable Event Log	"What happened?"
AssociativeDB	The Hippocampus	Associative Memory	"What is this like?"
Episteme	The Cortex	Structured Reasoning	"Is this true?"