3cfaa1e1d3
Phase 1 delivers the complete durability and storage layer:
- WAL with crash recovery: Append-only journal with BLAKE3 checksums,
fsync guarantees, and proper seek-to-EOF on reopen
- Storage engine: sled-backed KVStore with scan_prefix for range queries
- Content-addressed storage: H:{hash}, V:{hash}, E:{hash} key patterns
- Ingestor: Background worker tailing WAL, writing to KV with 8-byte
aligned record headers for rkyv zero-copy deserialization
- Comprehensive tests: 31 tests covering crash recovery, round-trips,
and multi-cycle durability
New crates: stemedb-wal, stemedb-storage, stemedb-ingest
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
83 lines
3.7 KiB
Markdown
83 lines
3.7 KiB
Markdown
# Episteme (StemeDB)
|
|
|
|
A probabilistic knowledge graph database that stores Claims, not Facts. Append-only Merkle DAG with read-time resolution via Lenses.
|
|
|
|
**Core Concept:** "Git for Truth" - conflicting assertions coexist, resolved at query time through Consensus, Recency, Authority, or custom Lenses.
|
|
|
|
## Find Your Guide
|
|
|
|
| If you need to... | Read this |
|
|
|-------------------|-----------|
|
|
| **Understand the vision** | [vision.md](./vision.md) |
|
|
| **See use cases** | [use-cases/README.md](./use-cases/README.md) |
|
|
| **Understand architecture** | [architecture.md](./architecture.md) |
|
|
| **See the roadmap** | [roadmap.md](./roadmap.md) |
|
|
| **Write Rust code** | [.claude/guides/backend/rust-guidelines.md](.claude/guides/backend/rust-guidelines.md) |
|
|
| **Set up local dev** | [.claude/guides/local/setup.md](.claude/guides/local/setup.md) |
|
|
| **Run tests** | [.claude/guides/local/testing.md](.claude/guides/local/testing.md) |
|
|
| **Understand quality checks** | [.claude/guides/local/quality-checks.md](.claude/guides/local/quality-checks.md) |
|
|
| **Learn about simulation** | [ai-lookup/features/simulation.md](ai-lookup/features/simulation.md) |
|
|
| **Work on storage/DAG** | Load skill: `stemedb-core` |
|
|
| **Implement a Lens** | Load skill: `stemedb-lens` |
|
|
| **Plan a milestone** | `/plan-milestone` command |
|
|
| **Integrate with AI tools** | [.claude/guides/integrations/ai-coding-assistant-integration.md](.claude/guides/integrations/ai-coding-assistant-integration.md) |
|
|
|
|
## Critical Rules
|
|
|
|
- **Append-Only:** NEVER mutate existing Assertions. Create new ones.
|
|
- **Content-Addressed:** Assertion ID = BLAKE3 hash of content.
|
|
- **No Unwrap:** NEVER use `unwrap()` or `expect()` in production code. CI enforces via `clippy::unwrap_used` and `clippy::expect_used` at deny level.
|
|
- **Defensive Writes:** All writes go through WAL with fsync.
|
|
- **Zero-Copy:** Use `rkyv` for serialization.
|
|
- **Instrument Critical Paths:** Use `#[instrument]` on public methods in WAL, storage, and ingestion code. Include meaningful fields (key_len, payload_len, offset).
|
|
- **Document Changes:** Update `ai-lookup/` when adding new types/concepts. Keep skills in sync with code.
|
|
|
|
## Quick Reference
|
|
|
|
```bash
|
|
# Build
|
|
cargo build --workspace
|
|
|
|
# Test
|
|
cargo test --workspace
|
|
|
|
# Lint (must pass before commit)
|
|
cargo clippy --workspace -- -D warnings
|
|
cargo fmt --check
|
|
```
|
|
|
|
## Specialized Agents
|
|
|
|
| Domain | Agent | When to use |
|
|
|--------|-------|-------------|
|
|
| **Product Vision** | `episteme-product-visionary` | Use cases, "why not Postgres?", product-market fit |
|
|
| General Rust | `primary-developer` | Feature implementation, refactoring |
|
|
| Code Quality | `rust-quality-engineer` | Reviews, test coverage, clippy |
|
|
| Storage | `storage-engine-architect` | WAL, LSM, crash recovery |
|
|
| Graph Engine | `rust-graph-engine-architect` | Lock-free structures, cache optimization |
|
|
| Defensive | `defensive-systems-architect` | Rate limiting, circuit breakers, hostile input |
|
|
| Lenses | `stemedb-lens-architect` | Query resolution, ranking algorithms |
|
|
| Planning | `stemedb-planner` | Milestone planning, roadmap |
|
|
|
|
## Architecture Overview
|
|
|
|
```
|
|
Write Path (Spine): Read Path (Cortex):
|
|
[Agent] -> [Ingestion] [Agent] <- [Lens Engine]
|
|
| |
|
|
v |
|
|
[WAL/Fsync] [Index Lookup]
|
|
| |
|
|
v |
|
|
[KV Store] <--------------------+
|
|
```
|
|
|
|
## Crates
|
|
|
|
| Crate | Purpose |
|
|
|-------|---------|
|
|
| `stemedb-core` | Assertion struct, types, storage traits |
|
|
| `stemedb-wal` | Write-ahead log (planned) |
|
|
| `stemedb-index` | Subject/Predicate indexing (planned) |
|
|
| `stemedb-lens` | Lens trait and implementations (planned) |
|