stemedb/architecture.md

Episteme (StemeDB) Architecture

Design Philosophy: Immutable History, Probabilistic Resolution, Materialized Speed. Status: Implementation v1.0

1. System Overview

Episteme is a Log-Structured, Content-Addressed Knowledge Graph. Unlike traditional databases that mutate state in place, Episteme appends Assertions to an immutable ledger (Merkle DAG). State resolution happens via Lenses.

Caveat: Aphoria's scan observations flow through this append-only path today. Aphoria's authored claims (AuthoredClaim) do not -- they are stored in a mutable TOML file (.aphoria/claims.toml) and bypass the WAL/Merkle DAG entirely. Routing claims through StemeDB as proper Assertions is a planned gap closure.

To solve the O(N) read latency of conflict resolution, Episteme employs a Materialized View layer that pre-calculates the "Current Truth" for standard lenses.

High-Level Data Flow

[Writer Agent]      [Reader Agent]
      │                   ▲
      │ (1) Sign &        │ (6) Sub-millisecond Answer
      │     Propose       │     (Pre-computed)
      ▼                   │
┌────────────┐      ┌────────────┐
│  Ingestion │      │ Resolution │
│  Gateway   │      │ Engine     │
└─────┬──────┘      └─────┬──────┘
      │ (2) Append        │ (5) Apply Lens + Trust Pack
      │     to Ballot     │     (BitSet Filter)
      ▼                   │
┌────────────┐      ┌────────────┐
│ Quarantine │      │ Indexing   │
│ Journal    │──────► Service    │
└─────┬──────┘ (3)  └─────┬──────┘
      │                   │ (4) Compaction & Materialization
      ▼                   ▼
┌────────────┐      ┌────────────┐
│ Job Manager│      │ Materialized│
└────────────┘      │ Views      │
 (TAN Meter)        └────────────┘

---

2. Core Data Structures

2.1. The Atomic Unit: Assertion (The Candidate)

Assertions are proposals of truth. They are immutable.

struct Assertion {
    pub subject: EntityId,
    pub predicate: RelationId,
    pub object: Value,
    pub epoch: Option<EpochId>,
    pub agent_id: PublicKey,     // The Proposer
    pub timestamp: u64,
    // ... lineage and vector fields ...
}

2.2. The Ballot Box: Vote (The High-Velocity Stream)

To prevent lock contention on Assertions, Agents write Votes to a separate high-velocity log.

struct Vote {
    pub assertion_hash: Hash,    // What are we voting on?
    pub agent_id: PublicKey,     // Who is voting?
    pub weight: f32,             // 0.0 - 1.0 (Confidence)
    pub signature: Signature,    // Cryptographic proof
    pub timestamp: u64,
}

2.3. The Trust Pack (The Overlay)

A curated list of trusted agents, used to filter consensus efficiently.

struct TrustPack {
    pub id: PackId,
    pub name: String,
    pub maintainer: PublicKey,
    pub agents: BitSet, // BloomFilter or RoaringBitmap for fast intersection
}

2.4. The Storage Layout (Hybrid Store)

Episteme uses a Hybrid Storage architecture to balance write throughput and read latency:

• Fjall (LSM-Tree): Used for write-heavy, append-only data (Assertions, Votes, WAL).
• Redb (B-Tree): Used for read-heavy, random-access data (Indexes, Materialized Views).

Key	Value	Purpose	Backend
H:{Hash}	Assertion	Immutable Content Store	Fjall
V:{Hash}	List<Vote>	The Ballot Box (Append-only)	Fjall
MV:{Subject}:{Predicate}	Assertion	Materialized View (The "Winner")	Redb
TP:{PackID}	TrustPack	Curation Lists	Redb
S:{Subject}	List<Hash>	Adjacency Index	Redb

---

3. The Write Path (The Ballot Box)

1. Ingest: Agents submit Assertions or Votes.
2. Journal: Written to episteme-wal (Quarantine Pattern).
3. Ballot Box: Votes are appended to the V:{Hash} stream.
4. Compactor (Async): A background worker aggregates Votes + TrustRank to update the MV key.

---

4. The Read Path (The Cortex)

Fast Path (Standard Lenses):

• Query: GET /query?lens=Consensus
• Action: GET MV:{Subject}:{Predicate}
• Cost: O(1). Low latency.

Trusted Path (Trust Packs):

• Query: GET /query?lens=Authority&trust_pack=Science_Pack
• Action:
  1. Fetch Candidate Assertions.
  2. Fetch Votes.
  3. Filter: Intersect Votes with TrustPack.agents (BitSet operation).
  4. Sum weights of remaining votes.
• Cost: O(1) (if Materialized per Pack) or O(M) (Fast calculation).

Standard Lenses (Implemented)

• Consensus: Highest cluster density (Vote-aware).
• Authority: Filter by Trust Pack and TrustRank.
• Recency: Last Writer Wins (Hybrid Logical Clock).
• EpochAware: Validates against current paradigm.
• Skeptic: Surfaces conflicts and divergence.

---

5. The Meter (Economic Safety)

To prevent infinite loops, the Job Manager enforces Temporal Advantage Normalization (TAN).

• Budgeting: Every Job must declare a max_cost.
• Throttling: Forking Reality or Deep Recursion is rejected if current_cost + projected_cost > max_cost.

---

6. The Simulator (Mid-Training Pipeline)

The system continuously exports data to train the next generation of Agents.

• Negative Samples: High-confidence assertions that were later superseded (Failures).
• Golden Paths: Branches that successfully merged to Main (Successes).
• Format: Exported as HuggingFace-compatible datasets for LoRA fine-tuning.

---

7. Implementation Roadmap

Phase 1: The Spine (Foundation)

• Reuse quarantine-journal pattern for WAL (stemedb-wal).
• Implement Assertion, Epoch, and Vote structs (stemedb-core).
• Hybrid Storage backend (stemedb-storage).

Phase 2: The Lattice (Connectivity)

• The Ballot Box: Separate Vote storage stream.
• Materializer: Background worker to maintain MV keys.
• Trust Packs: Agent sets for filtering.
• The Meter: Implement Budget/TAN middleware in Job Manager.
• Agent Wallet: Sidecar for key management/signing.

Phase 3: The Cortex (Reasoning)

• Lenses: Recency, Consensus, Authority, Skeptic implemented (stemedb-lens).
• SMT Backend & Branching.
• Vector Search.
• Lens: Constraints: Implement the pre-flight check logic.

Phase 4: The Hive (Learning)

• The Simulator: Log exporter pipeline.
• Trust Marketplace: API for publishing/subscribing to Trust Packs.
• The Super Curator: Implement "Judge" agent with Visual Anchoring.