1ce4004807
This commit adds the read path (Cortex) to complement the write path (Spine): ## Crates - stemedb-api: HTTP API with axum + utoipa OpenAPI - /v1/assert, /v1/query, /v1/epoch, /v1/skeptic, /v1/trace, /v1/audit - Metered endpoints with quota enforcement - Ed25519 signature verification - stemedb-lens: Truth resolution lenses - RecencyLens, ConsensusLens, ConfidenceLens - VoteAwareConsensusLens (Ballot Box pattern) - TrustAwareAuthorityLens (The Hive pattern) - SkepticLens (conflict analysis) - EpochAwareLens (paradigm-safe queries) - stemedb-query: Query engine with materialized views ## Storage Extensions - VoteStore: Vote aggregation with cached counts - TrustRankStore: Agent reputation with decay - AuditStore: Query audit trail - IndexStore: SP/P/S index structures - SupersessionStore: Epoch supersession chains ## SDKs - sdk/go/steme: Go HTTP client with Ed25519 signing - sdk/go/adk: ADK-Go tools for AI agents ## Documentation - Updated CLAUDE.md, architecture.md, roadmap.md - New ai-lookup entries for all services - Use case docs for consumer health intelligence - Arena roadmap for simulation advancement Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
96 lines
3.0 KiB
Markdown
96 lines
3.0 KiB
Markdown
# Storage
|
|
|
|
**Last Updated:** 2026-01-31
|
|
**Confidence:** High
|
|
|
|
## Summary
|
|
|
|
Episteme uses a Log-Structured, Content-Addressed storage model. Writes append to WAL, then index asynchronously. Reads query indexes and apply Lenses.
|
|
|
|
**Key Facts:**
|
|
- Append-only (never mutate)
|
|
- WAL for durability (fsync on write)
|
|
- KV store for indexes (sled MVP, trait-abstracted)
|
|
- Content-addressed by BLAKE3 hash
|
|
|
|
**File Pointers:**
|
|
- `crates/stemedb-storage/src/traits.rs` - KVStore trait
|
|
- `crates/stemedb-storage/src/sled_backend.rs` - Sled implementation
|
|
- `crates/stemedb-storage/src/serde_helpers.rs` - Storage-layer serialize/deserialize helpers
|
|
- `crates/stemedb-storage/src/vote_store.rs` - VoteStore (Ballot Box)
|
|
- `crates/stemedb-storage/src/index_store.rs` - IndexStore (S: and SP: indexes)
|
|
- `crates/stemedb-storage/src/trust_rank_store.rs` - TrustRankStore (TR:)
|
|
|
|
## KV Layout
|
|
|
|
| Key Pattern | Value | Purpose |
|
|
|-------------|-------|---------|
|
|
| `H:{Hash}` | `Assertion` (serialized) | Main content store |
|
|
| `V:{assertion_hash}:{vote_hash}` | `Vote` (serialized) | Ballot Box votes |
|
|
| `VC:{assertion_hash}` | `u64` (LE bytes) | Vote count cache |
|
|
| `VW:{assertion_hash}` | `f32` (LE bytes) | Aggregate weight cache |
|
|
| `E:{epoch_id}` | `Epoch` (serialized) | Paradigm definitions |
|
|
| `S:{Subject}` | `Vec<Hash>` (rkyv) | Subject index (IndexStore) |
|
|
| `SP:{Subject}:{Predicate}` | `Vec<Hash>` (rkyv) | Compound index (IndexStore) |
|
|
| `TR:{AgentId}` | `TrustRank` (rkyv) | Agent reputation (TrustRankStore) |
|
|
| `MV:{Subject}:{Predicate}` | `MaterializedView` (rkyv) | Pre-computed winner (Materializer) |
|
|
| `__CURSOR__:ingest` | `u64` (LE bytes) | Ingestion WAL offset checkpoint |
|
|
|
|
## Serialization
|
|
|
|
### stemedb-core (shared types)
|
|
|
|
For core types, use the canonical module:
|
|
|
|
```rust
|
|
use stemedb_core::serde::{serialize, deserialize};
|
|
|
|
let bytes = serialize(&my_value)?;
|
|
let value: MyType = deserialize(&bytes)?;
|
|
```
|
|
|
|
**File:** `crates/stemedb-core/src/serde.rs`
|
|
|
|
Raw `AllocSerializer` usage is prohibited in production code (enforced via CLAUDE.md).
|
|
|
|
### stemedb-storage (store implementations)
|
|
|
|
In storage modules, use the storage-layer helpers that map to `StorageError`:
|
|
|
|
```rust
|
|
use crate::serde_helpers::{serialize, deserialize};
|
|
|
|
let bytes = serialize(&my_value)?; // Returns Result<Vec<u8>, StorageError>
|
|
let value: MyType = deserialize(&bytes)?;
|
|
```
|
|
|
|
**File:** `crates/stemedb-storage/src/serde_helpers.rs`
|
|
|
|
This provides unified error handling across all store implementations (VoteStore, IndexStore, TrustRankStore, AuditStore, TrustPackStore, QuotaStore).
|
|
|
|
## Write Path
|
|
|
|
```
|
|
1. Agent submits signed Assertion
|
|
2. Validate signature
|
|
3. Append to WAL (fsync)
|
|
4. Return 202 Accepted with Hash
|
|
5. Background: tail WAL -> update indexes
|
|
```
|
|
|
|
## Read Path
|
|
|
|
```
|
|
1. Query: GET(Subject, Predicate, Lens)
|
|
2. Lookup: SP:{Subject}:{Predicate} -> [Hash...]
|
|
3. Hydrate: Load assertions from H:{Hash}
|
|
4. Resolve: Apply Lens
|
|
5. Return: Deterministic answer
|
|
```
|
|
|
|
## Related Topics
|
|
|
|
- [Assertion](./assertion.md)
|
|
- [Ballot Box](./ballot-box.md) - High-velocity vote storage
|
|
- [Architecture](../../../architecture.md)
|