jordan 3cfaa1e1d3 feat: Complete Phase 1 (The Spine) - storage foundation

Phase 1 delivers the complete durability and storage layer:

- WAL with crash recovery: Append-only journal with BLAKE3 checksums,
  fsync guarantees, and proper seek-to-EOF on reopen
- Storage engine: sled-backed KVStore with scan_prefix for range queries
- Content-addressed storage: H:{hash}, V:{hash}, E:{hash} key patterns
- Ingestor: Background worker tailing WAL, writing to KV with 8-byte
  aligned record headers for rkyv zero-copy deserialization
- Comprehensive tests: 31 tests covering crash recovery, round-trips,
  and multi-cycle durability

New crates: stemedb-wal, stemedb-storage, stemedb-ingest

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

2026-01-31 14:15:34 -07:00

3.7 KiB

Raw Blame History

Episteme (StemeDB)

A probabilistic knowledge graph database that stores Claims, not Facts. Append-only Merkle DAG with read-time resolution via Lenses.

Core Concept: "Git for Truth" - conflicting assertions coexist, resolved at query time through Consensus, Recency, Authority, or custom Lenses.

Find Your Guide

If you need to...	Read this
Understand the vision	vision.md
See use cases	use-cases/README.md
Understand architecture	architecture.md
See the roadmap	roadmap.md
Write Rust code	.claude/guides/backend/rust-guidelines.md
Set up local dev	.claude/guides/local/setup.md
Run tests	.claude/guides/local/testing.md
Understand quality checks	.claude/guides/local/quality-checks.md
Learn about simulation	ai-lookup/features/simulation.md
Work on storage/DAG	Load skill: `stemedb-core`
Implement a Lens	Load skill: `stemedb-lens`
Plan a milestone	`/plan-milestone` command
Integrate with AI tools	.claude/guides/integrations/ai-coding-assistant-integration.md

Critical Rules

Append-Only: NEVER mutate existing Assertions. Create new ones.
Content-Addressed: Assertion ID = BLAKE3 hash of content.
No Unwrap: NEVER use unwrap() or expect() in production code. CI enforces via clippy::unwrap_used and clippy::expect_used at deny level.
Defensive Writes: All writes go through WAL with fsync.
Zero-Copy: Use rkyv for serialization.
Instrument Critical Paths: Use #[instrument] on public methods in WAL, storage, and ingestion code. Include meaningful fields (key_len, payload_len, offset).
Document Changes: Update ai-lookup/ when adding new types/concepts. Keep skills in sync with code.

Quick Reference

# Build
cargo build --workspace

# Test
cargo test --workspace

# Lint (must pass before commit)
cargo clippy --workspace -- -D warnings
cargo fmt --check

Specialized Agents

Domain	Agent	When to use
Product Vision	`episteme-product-visionary`	Use cases, "why not Postgres?", product-market fit
General Rust	`primary-developer`	Feature implementation, refactoring
Code Quality	`rust-quality-engineer`	Reviews, test coverage, clippy
Storage	`storage-engine-architect`	WAL, LSM, crash recovery
Graph Engine	`rust-graph-engine-architect`	Lock-free structures, cache optimization
Defensive	`defensive-systems-architect`	Rate limiting, circuit breakers, hostile input
Lenses	`stemedb-lens-architect`	Query resolution, ranking algorithms
Planning	`stemedb-planner`	Milestone planning, roadmap

Architecture Overview

Write Path (Spine):           Read Path (Cortex):
[Agent] -> [Ingestion]        [Agent] <- [Lens Engine]
              |                              |
              v                              |
         [WAL/Fsync]                  [Index Lookup]
              |                              |
              v                              |
         [KV Store] <--------------------+

Crates

Crate	Purpose
`stemedb-core`	Assertion struct, types, storage traits
`stemedb-wal`	Write-ahead log (planned)
`stemedb-index`	Subject/Predicate indexing (planned)
`stemedb-lens`	Lens trait and implementations (planned)

3.7 KiB Raw Blame History