stemedb/roadmap.md

Episteme (StemeDB) Roadmap

Goal: Build the "Git for Truth" substrate for autonomous AI research. Current Phase: Phase 7-8 (The Shield + The Swarm) — Phase 6 complete ✅ Target Vertical: BioTech/Pharma ("The Living Review") Endgame: Distributed multi-writer cluster for millions of concurrent agents

---

High-Level Timeline

Phase	Codename	Focus	Key Deliverable
1	The Spine	Storage & Safety	Append-only WAL + KV Store
2	The Lattice	Indexing & Async	Materialized Views + Ballot Box
2.5	Hardening	Camp 2 Fixes	MV staleness, epoch behavior, lens cleanup
3	The Pilot	Vertical Integration	Pharma Ingestion + Living Review Agent
4	The Hive	Trust & Learning	TrustRank, metadata indexing, change tracking
5	The Forge	Foundation Hardening	Replace sled, fix WAL, persist indices, concept hierarchy
6	The Mesh	Distributed Writes	CRDT replication, Raft coordination, cluster membership
7	The Shield	Trust at Scale	EigenTrust, PoW admission, anti-spam, quarantine
8	The Swarm	Production Cluster	Chaos testing, observability, geo-distribution
9	The Bunker	Disaster Planning	Backup/restore, corruption recovery, GDPR compliance

---

Detailed Milestones

Phase 1: The Spine (Foundation)

Goal: Securely ingest assertions and persist them without data loss.

• Project Scaffold: Initialize Rust workspace, set up linting/CI (clippy, fmt).
• Assertion Schema: Define the Assertion struct with rkyv serialization.
  • Add dependencies: rkyv, blake3, ed25519-dalek, image_hasher.
  • Define Assertion struct (Subject, Predicate, Object, Confidence, SourceHash).
  • Multi-Sig Expansion: Implement SignatureEntry struct and signatures: Vec<SignatureEntry> field.
  • Visual Expansion: Add visual_hash: Option<pHash> field for image provenance.
  • Test serialization round-trips.
• Ballot Schema: Define the Vote struct for multi-agent consensus.
  • Add Vote struct: assertion_hash, agent_id, weight, signature.
  • Test serialization round-trips.
• Paradigm Schema (Epochs): Define the Epoch and SupersessionType structs.
  • Add epoch: Option<EpochId> to Assertion.
  • Implement Epoch struct with supersedes and SupersessionType.
  • Test serialization round-trips.
• WAL Integration: Implement the Quarantine Pattern for write-ahead logging.
  • Create stemedb-wal crate.
  • Port FsyncGuard and Record logic from established durability patterns.
  • Implement Record format with BLAKE3 checksums and Headers.
  • Verify fsync behavior with tests.
• Storage Engine: Implement the Store trait using sled (embedded KV).
  • Add sled dependency.
  • Define KVStore trait (put, get, delete, scan_prefix, flush).
  • Implement SledStore wrapper.
• Basic Ingestor: Background worker that tails WAL and writes to KV.
  • Implement async loop reading from WAL.
  • Write deserialized assertions, votes, and epochs to sled.
  • Ed25519 signature verification during ingestion.
  • Maintains S: and SP: indexes on ingest.
  • Persistent cursor/checkpoint (resumes from __CURSOR__:ingest in KV store).
• Verification: Crash recovery tests (write -> crash -> restart -> read).
  • Single and multi-record crash recovery.
  • Multiple crash cycles tested.

Phase 2: The Lattice (Connectivity)

Goal: Query data with sub-millisecond latency using Materialized Views.

• Lifecycle Schema: Add LifecycleStage to Assertion.
  • Define enum: Proposed, UnderReview, Approved, Deprecated, Rejected.
  • Update Assertion struct and serialization tests.
• The Ballot Box: Implement high-velocity vote ingestion.
  • VoteStore trait and implementation.
  • VoteAwareConsensusLens for real vote-based resolution.
• Index Infrastructure: Compound indexes for O(1) queries.
  • IndexStore trait with S: and SP: indexes.
  • QueryEngine smart routing (SP -> S -> scan).
• Materializer: Background worker for O(1) Read Performance.
  • MaterializedView type in stemedb-core.
  • Materializer worker in stemedb-query with step() and run().
  • Aggregates Votes via VoteAwareConsensusLens (or any AsyncLens).
  • Updates MV:{Subject}:{Predicate} with the winning Assertion + metadata.
  • Event-driven mode via run_notified() with tokio::sync::Notify.
  • Fast-path MV lookup in QueryEngine::try_fast_path().
• The Meter: Implement Economic Throttling (TAN).
  • QuotaStore trait and GenericQuotaStore implementation.
  • Token Bucket algorithm with per-agent per-hour quotas.
  • MeterLayer tower middleware for request cost tracking.
  • Cost model: Assert=10, Vote=1, Query=5+lens, +1/KB payload.
  • GET /v1/meter/quota endpoint to check remaining quota.
  • POST /v1/meter/quota/limit admin endpoint to set custom limits.
• API Surface: axum HTTP server with OpenAPI (utoipa).
  • POST /v1/assert -> Accepts JSON, writes to WAL.
  • POST /v1/vote -> High-throughput vote endpoint.
  • POST /v1/epoch -> Create epoch with optional supersession.
  • GET /v1/query -> Subject/Predicate/Lens/Lifecycle/Epoch filtering.
  • GET /v1/health -> Health check with assertion count.
  • GET /swagger-ui -> Interactive API docs.
  • 5 lens types available: Recency, Consensus, Authority, VoteAwareConsensus, TrustAwareAuthority.
• Query Audit: Log every read with provenance.
  • Define QueryAudit struct: query_id, agent_id, timestamp, params, result_hash, contributing_assertions.
  • Storage at AUD:{query_id} with agent index at AUDA:{agent_id}:{timestamp}:{query_id}.
  • GET /v1/audit/queries -> Returns history of agent decisions.
  • GET /v1/audit/query/{id} -> Full reasoning trace for a single query.
  • Auto-logging on every query via X-Agent-Id header.

Phase 2.5: Hardening (Camp 2 Fixes)

Goal: Close the gaps between "built" and "works right." Every item here addresses a feature that exists but doesn't fully deliver on its promise.

• 2.1 MV Staleness Detection: Make the fast-path aware of stale materialized views.

  • Status: ✅ COMPLETE
  • Implementation:
    • Added max_stale: Option<u64> to Query struct in crates/stemedb-query/src/query.rs.
    • Added .max_stale(secs) builder method to QueryBuilder.
    • In try_fast_path(): if query.max_stale is set and MV age exceeds threshold, falls through to slow path with debug! log.
    • Added max_stale to API QueryParams DTO in crates/stemedb-api/src/dto.rs.
    • Wired through query handler in crates/stemedb-api/src/handlers/query.rs.
  • Tests:
    • test_fast_path_stale_view_falls_back: MV 1000 seconds old, max_stale = 60 → slow path used.
    • test_fast_path_fresh_view_used: Fresh MV, max_stale = 300 → fast path used.
    • test_fast_path_no_max_stale_always_uses_mv: No max_stale → any MV age accepted (backward compatible).
    • test_fast_path_max_stale_zero_rejects_old_mv: max_stale = 0, MV 1 second old → slow path.
    • test_fast_path_max_stale_zero_accepts_brand_new_mv: max_stale = 0, brand new MV → fast path.

• 2.2 AuthorityLens -> ConfidenceLens Rename: Eliminate the misleading name.

  • Problem: AuthorityLens selects by confidence field, not by agent reputation. TrustAwareAuthorityLens is the real authority lens. The name creates confusion about what "Authority" means.
  • Solution implemented:
    • Renamed authority.rs → confidence.rs, AuthorityLens → ConfidenceLens
    • Added LensDto::Confidence for the confidence-field selector
    • Changed LensDto::Authority to route to TrustAwareAuthorityLens (the real authority lens)
    • Updated query handler routing
    • Updated ai-lookup/services/lens.md and skill documentation

• 2.3 EpochAwareLens: Give epoch supersession runtime behavior.

  • Status: ✅ COMPLETE
  • Implementation:
    • EpochAwareLens in crates/stemedb-lens/src/epoch_aware.rs
    • Decorator pattern wrapping any inner lens (default: RecencyLens)
    • Walks supersession chain from E:{epoch_id} keys
    • Cycle detection + max depth guard (100)
    • Fail-open on missing epochs
    • LensDto::EpochAware added to API
    • 11 tests: excludes_superseded, chain_supersession, no_epochs_passes_all, missing_epoch_includes, cycle_detection, consensus_lens_inner, mixed_epochs, etc.
    • Documentation updated in ai-lookup/services/lens.md
  • Known Limitation: Filtering only occurs when assertions from the superseding epoch are present in candidates. If all candidates are from old epoch (no new epoch assertions), they pass through (fail-open behavior).

• 2.4 Visual Hash Query Support: Make the stored visual_hash queryable.

  • Status: ✅ COMPLETE
  • Implementation:
    • hamming_distance(a: &PHash, b: &PHash) -> u32 in crates/stemedb-query/src/query.rs (lines 26-28)
    • visual_near: Option<String> and visual_threshold: Option<u32> in Query struct (lines 84-90)
    • .visual_near(hash, threshold) builder method
    • Query::matches() computes hamming distance when visual_near is set
    • API QueryParams DTO has visual_near and visual_threshold
    • 10+ tests: exact_match, within_threshold, exceeds_threshold, skips_without_hash, invalid_hex, wrong_length, combines_with_subject, default_threshold, max_threshold, threshold_63_rejects
  • Note: Brute-force O(N) scan. VP-tree/BK-tree index is Phase 3+.

• 2.5 Vector Field: No changes needed. Already roadmapped for Phase 3.

  • Status: ✅ N/A (No Phase 2 work required)
  • Current state: vector: Option<Vec<f32>> on Assertion. Stored and returned by API. No index, no search.
  • Phase 3 plan: Integrate hnsw-rs or lance for k-NN search.

• 2.6 E2E Integration Test (Write -> Materialize -> Read): Prove the full pipeline works end-to-end.

  • Status: ✅ COMPLETE
  • Implementation:
    • crates/stemedb-query/tests/e2e_pipeline.rs with 5 comprehensive tests:
      • test_e2e_write_materialize_read - Basic happy path
      • test_e2e_vote_consensus - Vote-weighted resolution
      • test_e2e_update_winner - Winner changes on re-materialize
      • test_e2e_cursor_persistence - Cursor survives worker restart
      • test_e2e_notify_integration - Event-driven notification channel
    • stemedb-wal and stemedb-ingest added as dev-dependencies
    • Helper functions: create_signed_assertion(), compute_assertion_hash(), create_vote()
    • Uses Ed25519 signing for authentic signature verification
  • Also: crates/stemedb-api/tests/e2e_flow_test.rs tests the HTTP API layer end-to-end.

Phase 3: The Pilot (BioTech/Pharma)

Goal: Prove value in the "High-Liability" beachhead. Close every Camp 4 gap that blocks a credible demo.

3A. Schema Expansion (Prerequisite for everything below)

• 3A.1 Source-Class Field: Add source_class: SourceClass to Assertion.

  • Status: ✅ COMPLETE
  • Implementation:
    • SourceClass enum in crates/stemedb-core/src/types.rs (lines 68-88).
    • 6-tier system: Regulatory (0), Clinical (1), Observational (2), Expert (3), Community (4), Anecdotal (5).
    • tier() method returns tier number for ordering.
    • default_decay_days() method for tier-specific confidence decay.
    • authority_weight() method for conflict resolution weighting.
    • Field on Assertion struct at line 152.
    • Full serialization and indexing support.

• 3A.2 Conflict Score on Resolution: Add conflict_score: f32 to Resolution.

  • Status: ✅ COMPLETE
  • Implementation:
    • Added conflict_score: f32 field to Resolution in crates/stemedb-lens/src/traits.rs.
    • Updated Resolution::empty() to set conflict_score: 0.0.
    • Updated Resolution::with_winner() to accept conflict_score parameter.
    • Added compute_conflict_score(candidates: &[Assertion]) -> f32 utility function:
      • Uses normalized variance of confidence values.
      • 0 or 1 candidates → 0.0 (no conflict possible).
      • All same confidence → 0.0 (unanimous).
      • Max variance (0.0 vs 1.0) → 1.0 (maximum conflict).
      • Defensive NaN handling (returns 0.0 for malformed data).
    • Updated all lens implementations to compute and pass conflict score:
      • crates/stemedb-lens/src/recency.rs
      • crates/stemedb-lens/src/consensus.rs
      • crates/stemedb-lens/src/confidence.rs
      • crates/stemedb-lens/src/vote_aware_consensus.rs
      • crates/stemedb-lens/src/trust_aware_authority.rs
    • Added conflict_score: f32 to MaterializedView in crates/stemedb-core/src/types.rs.
    • Updated Materializer::materialize_pair() to write conflict_score from resolution.
    • Added conflict_score: Option<f32> and resolution_confidence: Option<f32> to QueryResponse DTO in crates/stemedb-api/src/dto.rs (only present when lens is applied).
    • Wired through query handler in crates/stemedb-api/src/handlers/query.rs.
  • Tests:
    • test_conflict_score_zero_for_empty: Empty candidates → 0.0.
    • test_conflict_score_zero_for_single: 1 candidate → 0.0.
    • test_conflict_score_zero_for_agreement: All same confidence → near 0.0.
    • test_conflict_score_high_for_disagreement: Candidates at 0.1, 0.5, 0.9 → score > 0.3.
    • test_conflict_score_max_for_extremes: 0.0 vs 1.0 → score ≈ 1.0.
    • test_conflict_score_handles_nan_defensively: NaN confidences → 0.0 (fail-safe).

• 3A.3 Rich Source Metadata: Add structured provenance beyond source_hash.

  • Status: ✅ COMPLETE
  • Implementation:
    • Added source_metadata: Option<Vec<u8>> field to Assertion in crates/stemedb-core/src/types.rs (after epoch, before lifecycle).
    • Uses Vec<u8> (not String) for rkyv zero-copy compatibility. Callers encode/decode JSON on their side.
    • Added source_metadata: Option<Vec<u8>> to AssertionBuilder in crates/stemedb-core/src/testing.rs.
    • Added .source_metadata_json(json: &str) and .source_metadata(bytes) builder methods.
    • Added source_metadata: Option<String> to CreateAssertionRequest DTO (JSON string in API, converted to bytes internally).
    • Added source_metadata: Option<String> to AssertionResponse DTO (bytes converted to JSON string with defensive UTF-8 handling).
    • Wired through create handler (dto_to_assertion()) and query handler (assertion_to_dto()).
  • Tests:
    • test_serialize_deserialize_assertion_with_metadata: Serialization roundtrip with metadata present.
    • test_serialize_deserialize_assertion_without_metadata: Serialization roundtrip with metadata absent.
  • Note: Metadata is stored but NOT indexed in Phase 3. Indexing individual metadata fields is Phase 4+.

3B. Time & Decay (Core Query Features)

• 3B.1 Time-Travel Engine: as_of parameter for historical queries.

  • Status: ✅ COMPLETE
  • Implementation:
    • Added as_of: Option<u64> to Query struct in crates/stemedb-query/src/query.rs:92-99.
    • Added .as_of(timestamp: u64) to QueryBuilder.
    • In Query::matches(): if as_of is Some(ts), check assertion.timestamp <= ts. Assertions created after as_of are excluded.
    • In QueryEngine::execute(): if query.as_of is set, skip the fast path entirely (MVs reflect current state, not historical).
    • Added as_of: Option<u64> to QueryParams DTO in crates/stemedb-api/src/dto.rs.
    • Wired through query handler.
  • Tests:
    • test_as_of_excludes_future_assertions: Assertions filtered by timestamp.
    • test_as_of_bypasses_fast_path: MV exists, but as_of is set. Slow path used.
    • test_as_of_none_uses_fast_path: Normal query still uses fast path (backwards-compatible).
    • test_as_of_with_lens_resolves_among_historical_candidates: Time-travel + lens = resolve only among pre-as_of candidates.
    • test_as_of_returns_empty_when_all_assertions_are_future: All assertions are future, returns empty.
    • test_as_of_with_exact_timestamp_match: Edge case where assertion.timestamp == as_of.

• 3B.2 Semantic Decay: Confidence Half-Life at query time.

  • Status: ✅ COMPLETE
  • Implementation:
    • Added decay_halflife: Option<u64> to Query struct in crates/stemedb-query/src/query.rs.
    • Added .decay_halflife(seconds: u64) and .source_class_decay(enabled: bool) to QueryBuilder.
    • Added decay_halflife and source_class_decay to QueryParams DTO.
    • Created new crates/stemedb-query/src/decay.rs module with:
      • apply_decay(): Uniform decay using formula confidence * 2^(-(age / halflife)).
      • apply_source_class_decay(): Tier-specific decay (Regulatory=none, Clinical=2yr, Anecdotal=30d).
      • compute_decayed_confidence(): Core decay calculation with clamping.
    • Integrated in QueryEngine::execute(): decay applied after filtering, before lens resolution.
    • Time-travel compatible: uses as_of timestamp if set, otherwise current time.
    • Source-class-aware decay fully implemented using SourceClass::default_decay_days().
  • Tests: (11 unit tests in decay.rs + 1 E2E test)
    • test_decay_reduces_old_assertion_confidence: 1yr old, 1yr halflife → ~50% confidence.
    • test_decay_preserves_fresh_assertions: 1hr old, 1yr halflife → ~100% confidence.
    • test_decay_interacts_with_lens: Older high-confidence loses to newer low-confidence after decay.
    • test_source_aware_decay_tier0_no_decay: Regulatory never decays.
    • test_source_aware_decay_tier5_rapid_decay: Anecdotal decays rapidly (30-day halflife).
    • test_source_aware_decay_mixed_tiers: Clinical vs Anecdotal tier comparison.
    • test_decay_zero_halflife_no_change: Zero halflife skips decay (avoids div-by-zero).
    • test_decay_future_assertion_no_change: Future assertions don't decay.
    • test_decay_empty_assertions: Empty input returns empty output.
    • test_decay_confidence_clamps_to_valid_range: Very old assertions clamp to [0.0, 1.0].
    • test_decay_preserves_other_fields: Only confidence changes; other fields preserved.
    • test_e2e_decay_reduces_old_confidence: Full pipeline E2E test in e2e_pipeline.rs.
  • Note: When decay is enabled, materialized views (fast path) are bypassed because MVs store pre-computed winners without decay applied.

3C. New Lenses

• 3C.1 Skeptic Lens: Surface disagreement, not winners. ✅ COMPLETED

  • Status: ✅ COMPLETE
  • Implementation:
    • crates/stemedb-lens/src/skeptic.rs - Full implementation.
    • AnalysisLens trait for lenses that analyze conflict instead of resolving it.
    • SkepticLens uses normalized Shannon entropy for conflict scoring.
    • Returns ConflictAnalysis with:
      • conflict_score: f32 (0.0 = unanimous, 1.0 = chaos)
      • status: ResolutionStatus (Unanimous, Agreed, Contested)
      • claims: Vec<ClaimSummary> - all claims ranked by weight
    • SkepticResolver + SkepticView in stemedb-query/src/skeptic.rs.
    • GET /v1/skeptic?subject=X&predicate=Y API endpoint.
    • Core types in stemedb-core/src/types.rs:
      • ResolutionStatus enum
      • ConflictAnalysis struct
      • ClaimSummary, SourceSummary, AgentSummary
    • Comprehensive test coverage (21 test cases).

• 3C.2 Layered Consensus Lens: Per-source-class consensus.

  • Status: ✅ COMPLETE
  • Implementation:
    • crates/stemedb-lens/src/layered_consensus.rs - Full implementation.
    • TierResolution struct: per-tier result with tier, source_class, winner, candidates_count, conflict_score, resolution_confidence.
    • LayeredResolution struct: multi-tier result with tiers vec, overall_winner, overall_conflict_score, total_candidates.
    • LayeredLens trait: resolve_layered(&[Assertion]) -> LayeredResolution, name() -> &'static str.
    • LayeredConsensusLens implements both LayeredLens and Lens traits.
    • Cross-tier conflict score uses normalized Shannon entropy of tier winner object values.
    • LensDto::LayeredConsensus variant (redirects to /v1/layered endpoint).
    • GET /v1/layered?subject=X&predicate=Y API endpoint with LayeredQueryResponse.
    • Exported from crates/stemedb-lens/src/lib.rs.
  • Tests:
    • test_layered_empty_candidates: Empty input returns empty resolution.
    • test_layered_single_tier: All same source_class, returns one tier result.
    • test_layered_multi_tier_agreement: Tier 0 and Tier 5 agree, low cross-tier conflict.
    • test_layered_multi_tier_disagreement: Tier 1 vs Tier 5 disagree, high conflict, Tier 1 wins.
    • test_layered_overall_winner_from_highest_authority: Tier 0 wins despite fewer assertions.
    • test_layered_lens_trait_compatibility: Standard Lens trait works.
    • test_layered_within_tier_conflict: High internal conflict within a tier.
    • test_layered_all_tiers_present: One assertion from each tier.
    • test_layered_lens_name: Both trait names work.
    • test_layered_numeric_values: Works with numeric object values.

• 3C.3 Constraints Lens: Pre-flight check for must_use/forbidden.

  • Status: ✅ COMPLETE
  • Implementation:
    • crates/stemedb-lens/src/constraints.rs - Full implementation.
    • ConstraintSet struct: holds categorized assertions (must_use, forbidden, prefer) with conflict_score.
    • ConstraintsLens struct with resolve_constraints(&[Assertion]) -> ConstraintSet method.
    • Categorizes by predicate pattern: must_use:*, forbidden:*, prefer:*.
    • Implements Lens trait for compatibility (priority: must_use > forbidden > prefer).
    • Sorted by confidence (highest first), with timestamp as tiebreaker.
    • LensDto::Constraints added (redirects to /v1/constraints endpoint).
    • GET /v1/constraints?subject=X API endpoint with ConstraintsResponse.
    • DTOs: ConstraintsQueryParams, ConstraintEntryDto, ConstraintsResponse.
    • Exported from crates/stemedb-lens/src/lib.rs.
  • Tests: (16 test cases)
    • test_constraints_categorizes_by_predicate: Mixed predicates sorted into must_use/forbidden/prefer.
    • test_constraints_empty_categories: Only prefer, no must_use/forbidden.
    • test_constraints_non_constraint_predicates_ignored: Regular predicates filtered out.
    • test_constraints_sorted_by_confidence: Within-category confidence ordering.
    • test_constraints_empty_candidates: Empty input returns empty set.
    • test_constraints_has_constraints_true: Helper method works.
    • test_constraints_all_regular_predicates: All non-constraint predicates returns no constraints.
    • test_lens_trait_picks_must_use_winner: Standard Lens trait picks must_use first.
    • test_lens_trait_falls_back_to_forbidden: Falls back to forbidden when no must_use.
    • test_lens_trait_falls_back_to_prefer: Falls back to prefer when no must_use/forbidden.
    • test_lens_trait_empty_for_no_constraints: Returns empty when no constraint predicates.
    • test_lens_name: Name returns "Constraints".
    • test_lens_empty_candidates: Empty input to Lens trait returns empty resolution.
    • test_multiple_must_use_picks_highest_confidence: Multiple must_use picks highest confidence.
    • test_confidence_tiebreaker_uses_timestamp: Same confidence uses newer timestamp.
    • test_predicate_pattern_exact_prefix: must_use_something not matched (only must_use:*).

3D. Epoch Enhancement

• 3D.1 Epoch Cascade Logic (enhancement of Phase 2.5 EpochAwareLens):
  • Status: ✅ COMPLETE
  • Implementation:
    • write_supersession_cascade() in crates/stemedb-ingest/src/worker.rs:
      • Writes SUPERSEDED:{old_epoch_id} markers for full transitive closure.
      • All markers point to the LATEST superseding epoch.
      • Max depth guard (100 levels) and cycle detection via visited set.
    • is_epoch_superseded() in crates/stemedb-lens/src/epoch_aware.rs:
      • O(1) marker lookup instead of O(chain_length) chain walks.
      • Fail-open semantics: missing marker = not superseded.
    • compute_superseded_epochs() uses marker lookups for filtering.
  • Tests:
    • test_cascade_writes_superseded_marker: Epoch B supersedes A → SUPERSEDED:A exists.
    • test_cascade_transitive: C→B→A chain → both SUPERSEDED:A and SUPERSEDED:B point to C.
    • test_cascade_cycle_detection: Mutual supersession handled gracefully.
    • test_epoch_aware_uses_marker: EpochAwareLens uses O(1) marker lookup.
    • test_superseded_epoch_filtered_even_without_new_assertions: Marker-based filtering works.

3E. Similarity Search

• 3E.1 Vector Search: Semantic k-NN queries via embeddings.

  • Status: ✅ COMPLETE
  • Implementation:
    • Added hnsw_rs = "0.3" and parking_lot = "0.12" to stemedb-storage/Cargo.toml.
    • New module: crates/stemedb-storage/src/vector_index.rs.
    • VectorIndex trait: insert(hash: &Hash, vector: &[f32]), search(query: &[f32], k: usize) -> Vec<(Hash, f32)>, dimension(), len(), is_empty().
    • HnswVectorIndex implementation with HNSW graph, RwLock protection, hash↔ID mappings.
    • Input validation: dimension mismatch, NaN, Infinite values rejected.
    • Idempotent insert (same hash twice = no-op).
    • Arc<dyn VectorIndex> trait object support for sharing.
    • IngestWorker::with_vector_index() builder method for index attachment.
    • IngestWorker: if assertion has vector, inserts into vector index after KV write.
    • Added vector_near: Option<Vec<f32>> and k: Option<usize> to Query struct in crates/stemedb-query/src/query.rs.
    • Added .vector_near(vector, k) builder method to QueryBuilder.
    • Added vector_near and k to API QueryParams DTO.
    • QueryEngine::with_vector_index() builder method for index attachment.
    • QueryEngine: if vector_near is set and index configured, uses O(log N) HNSW lookup for candidates.
    • Falls back to standard path if no index configured (with debug log).
  • Tests: (12 unit tests for VectorIndex + 4 integration tests in engine.rs)
    • test_create_index, test_insert_and_search, test_dimension_mismatch.
    • test_idempotent_insert, test_search_empty_index, test_search_k_zero.
    • test_nan_rejection, test_infinite_rejection, test_contains.
    • test_larger_scale (100 vectors, exact match first), test_custom_params, test_zero_dimension_panics.
    • test_vector_search_returns_nearest_neighbors, test_vector_search_with_subject_filter.
    • test_vector_search_without_index_falls_back, test_vector_search_with_as_of_filter.
  • Note: Index is in-memory only. Persistence is Phase 4+.

• 3E.2 Visual Hash Index: BK-tree for O(log N) visual similarity.

  • Status: ✅ COMPLETE
  • Implementation:
    • New module: crates/stemedb-storage/src/visual_index.rs.
    • VisualIndex trait: insert(hash: &Hash, phash: &PHash), search(query: &PHash, threshold: u32) -> Vec<(Hash, u32)>, len(), is_empty().
    • BkTreeVisualIndex implementation using BK-tree over hamming distance.
    • hamming_distance(a: &PHash, b: &PHash) -> u32 utility function.
    • Threshold clamped to 0-64 range (max 64 bits).
    • Results sorted by distance ascending.
    • Idempotent insert (same hash twice = no-op).
    • Arc<dyn VisualIndex> trait object support for sharing.
    • IngestWorker::with_visual_index() builder method for index attachment.
    • IngestWorker: if assertion has visual_hash, inserts into BK-tree after KV write.
    • QueryEngine::with_visual_index() builder method for index attachment.
    • QueryEngine: if visual_near is set and index configured, uses O(log N) BK-tree lookup.
    • Falls back to brute-force scan (via query.matches()) if no index configured.
    • Invalid hex input returns QueryError::InvalidInput with clear message.
  • Tests: (14 unit tests for VisualIndex + 6 integration tests in engine.rs)
    • test_hamming_distance_zero, test_hamming_distance_max, test_hamming_distance_partial.
    • test_create_index, test_insert_and_search_exact, test_search_within_threshold.
    • test_search_no_matches, test_search_empty_index, test_idempotent_insert.
    • test_contains, test_results_sorted_by_distance, test_threshold_clamped_to_64.
    • test_larger_scale (1000 hashes), test_default_impl.
    • test_visual_search_returns_similar_images, test_visual_search_with_lifecycle_filter.
    • test_visual_search_invalid_hex_returns_error, test_visual_search_without_index_uses_brute_force.
    • test_visual_search_with_limit, test_vector_search_empty_index.
  • Note: Index is in-memory only. Persistence is Phase 4+.

3F. Provenance

• 3F.1 Source Document Storage & Provenance Lookup: Enable 100% citation recall.
  • Status: ✅ COMPLETE
  • Implementation:
    • POST /v1/source endpoint to store source documents by BLAKE3 content hash.
    • GET /v1/provenance/{hash} endpoint to retrieve source documents by hash.
    • Source storage at SRC:{hash} keys with format: [content_type_len:4][content_type][content].
    • Base64 encoding for binary-safe JSON transport.
    • 10MB size limit per document.
    • Content-addressed storage: same content → same hash (idempotent uploads).
    • DTOs: StoreSourceRequest, StoreSourceResponse, ProvenanceResponse.
    • OpenAPI documentation under "provenance" tag.
  • Tests: (5 test cases)
    • test_store_and_retrieve_source: Happy path store + retrieve.
    • test_store_source_invalid_base64: Bad base64 → 400.
    • test_get_provenance_not_found: Unknown hash → 404.
    • test_get_provenance_invalid_hash: Bad hash format → 400.
    • test_store_source_idempotent: Same content twice → same hash.
  • Note: Benchmark utility for verifying all assertions have retrievable sources is future work.

3G. API Cleanup

• 3G.1 Document epoch supersession via existing endpoint: No new /epoch/supersede endpoint needed.
  • Status: ✅ COMPLETE
  • Implementation:
    • Updated use case docs (consumer-health-intelligence.md, glp1-living-review.md) to use POST /v1/epoch with supersedes field.
    • Added OpenAPI examples showing both new epoch and supersession flows in handlers/epoch.rs.
    • Documented all 5 supersession types: Invalidate, Temporal, Refinement, RequiresReview, Additive.
  • No code change. Documentation fix only.

Phase 4: The Hive (Trust & Scale)

Goal: Change tracking, metadata indexing, and the database primitives for training pipelines.

• TrustRank Engine: Foundation for trust-based resolution.

  • TrustRankStore for per-agent reputation storage.
  • TrustAwareAuthorityLens for reputation-weighted resolution.
  • Confidence Half-Life: Implement decay calculation engine.
  • Learning loop: record_outcome() for accuracy tracking.

• 4.1 "Since" Parameter: Change tracking for returning consumers.

  • Status: ✅ COMPLETE
  • Problem: Consumer Health shows GET /query?since=2023-10-01 returning changes_since with dated change entries. The "returning consumer" story: "What changed since I last looked?"
  • Depends on: Time-Travel (3B.1) and Materializer.
  • Implementation:
    • Added since: Option<u64> to Query struct and QueryBuilder in crates/stemedb-query/src/query.rs.
    • Added since to QueryParams DTO in crates/stemedb-api/src/dto.rs.
    • MV Changelog: Track when materialized views change.
      • New key pattern: MVC:{subject}:{predicate}:{timestamp_nanos} using nanosecond precision to prevent collisions.
      • ChangeEntry struct in crates/stemedb-core/src/types.rs with: timestamp, previous_winner_hash, new_winner_hash, subject, predicate, lens_name.
      • In Materializer::materialize_pair(): before overwriting MV, read existing MV. If winner changed (different hash), write changelog entry.
      • get_previous_winner_hash() and write_changelog_entry() helper methods.
    • In QueryEngine: if since is set, skip fast path (MVs reflect current state). fetch_changes_since() scans MVC: keys and returns entries >= since timestamp.
    • ChangeEntryDto in crates/stemedb-api/src/dto.rs with hex-encoded hashes.
    • changes_since: Option<Vec<ChangeEntryDto>> added to QueryResponse DTO.
    • fetch_changelog_if_needed() helper in query handler.
    • Re-exported ChangeEntry from stemedb-core crate root.
  • Tests:
    • test_changelog_written_on_first_materialization: First MV creates changelog with previous_winner_hash: None.
    • test_changelog_written_on_winner_change: Winner change creates changelog with previous hash.
    • test_no_changelog_when_winner_unchanged: Re-materialize same winner = no new changelog.
    • test_fetch_changes_since_returns_entries: Fetch filtered by timestamp, sorted ascending.
    • test_fetch_changes_since_empty_on_no_entries: No entries returns empty vec.
    • test_since_bypasses_fast_path: Query with since uses slow path, returns all assertions.

• 4.2 Source Metadata Indexing (extension of 3A.3): Index key metadata fields.

  • Status: ✅ COMPLETE
  • Problem: Phase 3 stores source_metadata as an opaque blob. Phase 4 makes key fields queryable.
  • Depends on: Rich Source Metadata (3A.3).
  • Implementation:
    • Defined indexed metadata keys: journal, doi, platform, study_design in INDEXED_METADATA_FIELDS.
    • New module: crates/stemedb-storage/src/source_metadata_index.rs.
    • Key pattern: SMV:{field}:{value} storing Vec<Hash> (rkyv-serialized) for efficient batch lookup.
    • SourceMetadataIndexStore trait with add_to_metadata_indexes() and get_by_metadata_field().
    • GenericSourceMetadataIndexStore implementation with case-insensitive value normalization.
    • IngestWorker: on ingestion, if source_metadata is present, parse JSON, extract indexed fields, write index entries.
    • Added source_journal, source_doi, source_platform, source_study_design to Query struct.
    • Added .source_journal(), .source_doi(), .source_platform(), .source_study_design() builder methods.
    • Added matches_metadata_filters() helper with AND semantics and case-insensitive matching.
    • Added metadata field filters to QueryParams DTO (e.g., ?source_journal=NEJM).
    • Wired through query handler in crates/stemedb-api/src/handlers/query.rs.
  • Tests: (9 unit tests in source_metadata_index.rs + 9 unit tests in query.rs)
    • test_add_and_get_by_metadata_field: Basic round-trip.
    • test_case_insensitive_indexing: "NEJM" stored, "nejm" query matches.
    • test_multiple_assertions_same_field: Vec accumulates correctly.
    • test_malformed_json_gracefully_skipped: Invalid JSON logs warning, doesn't error.
    • test_missing_field_not_indexed: Only indexed fields processed.
    • test_idempotent_insert: Same assertion twice = no duplicates.
    • test_all_indexed_fields: All 4 fields indexed correctly.
    • test_empty_index_returns_empty_vec: Non-existent field returns empty.
    • test_non_string_values_ignored: Non-string JSON values skipped.

• 4.3 Batch TrustRank Decay API: Expose scheduled decay for external orchestration.

  • Status: ✅ COMPLETE
  • Implementation:
    • New module: crates/stemedb-api/src/handlers/admin.rs
    • POST /v1/admin/decay-trust-ranks endpoint.
    • Request accepts now: Option<u64> (defaults to current time) and half_life_seconds: Option<u64> (defaults to 30 days).
    • Response includes decayed_count, timestamp_used, half_life_used, status.
    • OpenAPI docs with "admin" tag.
    • Thin handler delegates to TrustRankStore::decay_trust_ranks().
  • Tests: (3 integration tests in http_integration.rs)
    • test_decay_trust_ranks_empty_store: Empty store returns 0 decayed.
    • test_decay_trust_ranks_with_custom_params: Custom timestamp and half-life honored.
    • test_decay_trust_ranks_response_structure: All 4 fields present in response.
  • Note: The Gardener (Camp 5.2, app layer) calls this endpoint on a schedule. The database just exposes the primitive.

• 4.4 Vote Provenance Witness: Transform votes from opinions into cryptographic witnesses.

  • Status: ✅ COMPLETE
  • Problem: Votes record "I agree" but the browser extension product needs "I saw this exact text at this URL at this time." Without provenance, votes are Tier 5 noise.
  • Implementation:
    • Added source_url: Option<String> to Vote struct — URL where claim was observed.
    • Added observed_context: Option<Vec<u8>> to Vote struct — page context bytes (rkyv zero-copy pattern).
    • API DTOs: source_url and observed_context on CreateVoteRequest/VoteResponse.
    • Input validation: source_url max 2048 chars (non-empty if provided), observed_context max 64KB.
    • Backward compatible: existing votes without provenance remain valid.
  • Tests: Serialization roundtrip (with/without provenance, mixed fields), storage roundtrip, API integration (validation).

• 4.5 Conflict Score Filtering: Query-time filtering by conflict score.

  • Status: ✅ COMPLETE
  • Problem: The browser extension needs "only show me claims where conflict > 0.7" to implement contradiction-only overlays. Conflict score was computed but not filterable.
  • Implementation:
    • Added min_conflict_score: Option<f32> and max_conflict_score: Option<f32> to Query struct.
    • Builder methods: .min_conflict_score(), .max_conflict_score().
    • Fast-path filtering: checks MV conflict_score against thresholds.
    • API validation: scores must be 0.0-1.0, finite (rejects NaN/Inf).
    • QueryParams DTO wired through query handler.
  • Tests: 13 engine tests (min/max thresholds, exact boundary, range, combination with lifecycle), API validation tests.
  • Note: Filtering works on fast path (MV reads) only. Slow path returns raw assertions without conflict scores.

• 4.6 Escalation Triggers: Active safety system for high-conflict assertions.

  • Status: ✅ COMPLETE
  • Problem: High conflict was invisible. Episteme should be an active safety system that fires escalations when disagreement exceeds thresholds.
  • Implementation:
    • EscalationLevel enum: Low, Medium, High, Critical.
    • EscalationEvent struct: content-addressed (BLAKE3 ID), subject, predicate, conflict_score, level, reason, timestamp, resolved.
    • EscalationPolicy struct: configurable threshold + level + optional predicate pattern.
    • EscalationStore trait + GenericEscalationStore: write, query since, resolve, get pending.
    • Key pattern: ESC:{timestamp_nanos}:{id_hex}.
    • Materializer integration: after computing conflict_score, checks policies and writes events.
    • with_escalation(store, policies) builder on Materializer.
    • GET /v1/admin/escalations and POST /v1/admin/escalations/{id}/resolve endpoints.
  • Tests: Core type tests, storage roundtrip, materializer integration (high/low conflict, predicate matching, no store configured).

• 4.7 Gold Standard Verification: Sybil defense via proof of knowledge.

  • Status: ✅ COMPLETE
  • Problem: New agents need a way to earn TrustRank. Gold standards are admin-verified assertions that agents can be tested against.
  • Implementation:
    • GoldStandard struct with rkyv serialization: assertion_hash, subject, predicate, expected_object, created_at, created_by.
    • GoldStandardStore trait + GenericGoldStandardStore: set, get, list, remove.
    • Key pattern: GS:{subject}:{predicate}.
    • TrustAdjustment enum: Rewarded(+0.05), Penalized(-0.1), AlreadyVerified.
    • verify_agent_against_gold_standard() on TrustRankStore with deduplication (agents can only verify each gold standard once).
    • Verification markers at GS_VERIFIED:{agent_id}:{subject}:{predicate} prevent gaming.
    • API: POST/GET/DELETE /v1/admin/gold-standards, POST /v1/admin/verify-agent.
  • Tests: Core (creation, matching, case-sensitivity), storage CRUD, trust adjustment (reward, penalty, clamping, dedup), API integration (5 tests).

Note: The following items were reclassified as Application Layer responsibilities (see tmp/ambition-vs-reality.md, Camp 5). They are not Episteme database features. They consume the Episteme API and are built by integrators or vertical-specific teams.

• The Simulator (Training Data Pipeline) -> Camp 5.3
• The Super Curator (Reviewer Agent swarm) -> Camp 5.4
• Background Gardener (Cluster detection, signal processing) -> Camp 5.2
• Agent Wallet (Key management sidecar) -> Camp 5.1

Phase 5: The Forge (Foundation Hardening)

Goal: Replace abandoned dependencies, fix WAL gaps, persist indices. Prerequisite for distribution.

Research: docs/research/wal-crash-recovery-research.md

5A. Storage Engine Replacement

• 5A.1 Replace sled with redb + fjall: sled is abandoned (author recommends alternatives).

  • Problem: sled is alpha-stage with known performance regressions and no active development. Our entire storage layer depends on it.
  • Solution: HybridStore routes keys by prefix — fjall (LSM) for write-heavy paths (H:, V:, VC:, VW:, E:, SUPERSEDED:, __CURSOR__:) and redb (B-tree) for read-heavy paths (S:, SP:, MV:, TR:, QA:, QT:, TP:, GS:, ESC:).
  • Tasks:
    • Generalize StorageError::Sled to StorageError::Backend(String).
    • Implement FjallStore backend with DashMap per-key locks for atomics.
    • Implement RedbStore backend with ACID transactions.
    • Implement HybridStore routing layer with prefix-based dispatch.
    • Migrate all ~500 tests from SledStore to HybridStore.
    • Remove sled dependency entirely.
    • Add criterion benchmarks (sequential put, random get, prefix scan, atomic increment, mixed workload).
  • Crates: redb = "2", fjall = "2", dashmap = "6"

• 5A.2 Key Layout Redesign: Prepare keys for subject-prefix range sharding.

  • Problem: Current keys (H:{hash}, S:{subject}, MV:{subject}:{predicate}) scatter related data across the keyspace. Distributed sharding needs co-location.
  • Solution: Subject-prefix key layout with \x00 separator for subject-scoped keys, \x00 prefix for global keys (sort-first):

    Subject-prefixed (co-located):
    {subject}\x00H:{hash}       → Assertion data
    {subject}\x00S:{hash_list}  → Subject index
    {subject}\x00SP:{predicate} → Compound index
    {subject}\x00MV:{predicate} → Materialized view
    {subject}\x00V:{hash}:{vh}  → Votes
    {subject}\x00VC:{hash}      → Vote count cache
    {subject}\x00VW:{hash}      → Vote weight cache
    {subject}\x00GS:{predicate} → Gold standards
    
    Global (sort first via \x00 prefix):
    \x00TRUST:{agent_id}        → Trust ranks
    \x00QUOTA:{agent_id}:{win}  → Quota records
    \x00QLIMIT:{agent_id}       → Quota limits
    \x00E:{epoch_id}            → Epochs
    \x00SUPERSEDED:{epoch_id}   → Supersession markers
    \x00SUP:{hash}              → Supersession records
    \x00AUD:{query_id}          → Audit records
    \x00ESC:{ts}:{id}           → Escalation events
    \x00TP:{pack_id}            → Trust packs
    \x00META:{key}              → System metadata
    \x00HASH_SUBJECT:{hash}     → Reverse lookup index
    \x00SUBJECTS:{subject}      → Known subjects index
    \x00GS_LIST:{subj}:{pred}   → Gold standard listing
    

  • Implementation:
    • key_codec.rs (573 lines): 40+ key builder functions, subject validation, extraction utilities, 30+ unit tests.
    • All stores migrated to key_codec:: functions (91 call sites across 10 store files, zero hardcoded key patterns).
    • Ingestion pipeline uses key_codec (11 usages across 3 files).
    • Query engine uses key_codec (34 usages across 7 files).
    • Subject co-location verified by test_subject_colocation test.
    • Global key sort-first verified by test_global_keys_sort_first test.

5B. WAL Hardening

• 5B.1 CRC32C Checksums: Add hardware-accelerated torn write detection.

  • Status: ✅ COMPLETE
  • Implementation:
    • Added crc32c = "0.6" crate dependency.
    • Updated WAL record format with CRC32C in format.rs.
    • CRC32C verified on read before deserializing.
    • Hardware-accelerated via SSE 4.2 on supported CPUs.

• 5B.2 Crash Recovery Implementation: Replace recovery stub with production recovery.

  • Status: ✅ COMPLETE
  • Implementation:
    • recovery/mod.rs (236 lines): Full sequential record scan with CRC32C validation.
    • Truncates file at first corrupted/incomplete record.
    • RecoveryMetrics struct tracks: valid_records, invalid_records, bytes_truncated, recovery_duration.
    • Comprehensive test suite in recovery/tests.rs.

• 5B.3 Group Commit: Batch fsync for throughput.

  • Status: ✅ COMPLETE
  • Implementation:
    • group_commit.rs (342 lines): GroupCommitBuffer with configurable max_writes and max_duration.
    • Writers append to buffer and wait on Notify.
    • Background flusher calls fsync and notifies all waiters.
    • GroupCommitConfig for tuning batch size and flush interval.

• 5B.4 Log Rotation: Bounded WAL disk usage.

  • Status: ✅ COMPLETE
  • Implementation:
    • segment.rs (368 lines): Full segment management.
    • Segment naming: {seq:08x}.wal.
    • Rotation when segment exceeds configurable threshold.
    • SegmentManager tracks active and archived segments.
    • Safe deletion of segments after cursor passes them.

5C. Index Persistence

• 5C.1 Persistent Vector Index: Move HNSW from in-memory to disk-backed.

  • Status: ✅ COMPLETE
  • Implementation:
    • PersistentVectorIndex in crates/stemedb-storage/src/vector_index/persistent.rs.
    • Hybrid hot/cold architecture:
      • Hot: In-memory HNSW for recent vectors.
      • Cold: Disk-backed HNSW loaded from checkpoint files.
    • persistence/format.rs: SnapshotMetadata, IdMappingTable with rkyv serialization.
    • hot_cold.rs: merge_search_results() for combining hot and cold query results.
    • Background checkpoint task with atomic write pattern.
    • CRC32C integrity verification on load.
    • MAX_PAYLOAD_SIZE (1GB) validation to prevent memory exhaustion.
  • Crates: memmap2 = "0.9", crc32c = "0.6", byteorder = "1.5"
  • Known Limitation: Cold index currently stores ID mappings only; full vector persistence with mmap'd HNSW graphs planned for future phase.

• 5C.2 Persistent Visual Index: Persist BK-tree to disk.

  • Status: ✅ COMPLETE
  • Implementation:
    • PersistentVisualIndex in crates/stemedb-storage/src/visual_index.rs.
    • BkTreeSnapshot with rkyv serialization for BK-tree state.
    • Checkpoint file format: [MAGIC:4][VERSION:1][RESERVED:3][PAYLOAD_LEN:u64][CRC32C:u32][PAYLOAD].
    • Atomic write pattern: temp file → fsync → rename → fsync parent.
    • Background checkpoint task with configurable interval.
    • CRC32C integrity verification on load.
    • Shared checkpoint_format.rs module for common read/write utilities.

5D. Concept Hierarchy ✅ COMPLETE

Spec: docs/specs/concept-hierarchy.md Purpose: Hierarchical, scheme-qualified subject identifiers with cross-scheme alias resolution. Enables applications like Aphoria that need to connect code:// paths to rfc:// paths.

• 5D.1 ConceptPath Type: Structured subject identifiers. ✅

  • Tasks:
    • Add ConceptPath struct to stemedb-core/src/types/concept.rs.
    • Wire format: {scheme}://{segment_0}/{segment_1}/.../{leaf}.
    • parse(), to_wire_format(), leaf(), parent(), is_prefix_of().
    • Backward-compatible: bare strings parse as custom://{string}.
    • Unit tests for parsing, round-trip, prefix matching (Battery 8).
  • Crate: stemedb-core

• 5D.2 Source Scheme Registry: Map schemes to default source tiers. ✅

  • Tasks:
    • Add SourceScheme enum to stemedb-core.
    • Scheme → default SourceClass mapping (e.g., rfc:// → Tier 0, code:// → Tier 3).
    • ConceptPath::default_source_class() method.
  • Crate: stemedb-core

• 5D.3 Alias Store: Cross-scheme entity resolution. ✅

  • Tasks:
    • Add ConceptAlias struct to stemedb-core.
    • Add AliasStore trait to stemedb-storage.
    • Key prefixes: CA:{alias_path} → canonical, CAR:{canonical} → all aliases.
    • Transitive alias resolution with cycle detection.
    • GenericAliasStore implementation over KVStore.
  • Crates: stemedb-core, stemedb-storage

• 5D.4 Hierarchical Query: Prefix-based subject queries. ✅

  • Tasks:
    • fetch_by_subject_prefix() using scan_prefix in query engine (already implemented in Battery 5).
    • Trailing / handling to prevent auth matching authentication.
  • Crate: stemedb-query
  • Note: Hierarchical prefix scanning was already working; Battery 5 validates it.

• 5D.5 Alias Resolution in Queries: Expand queries to aliased paths. ✅

  • Tasks:
    • AliasStore::resolve_all() for transitive alias expansion.
    • API endpoint GET /v1/concepts/resolve?path=...&transitive=true.
  • Crate: stemedb-query, stemedb-api
  • Note: Resolution available via API; QueryEngine integration is future work.

• 5D.6 Source Class Inference: Infer tier from scheme. ✅

  • Tasks:
    • ConceptPath::default_source_class() returns tier based on scheme.
    • SourceScheme::parse() maps scheme strings to enum variants.
  • Crate: stemedb-core
  • Note: Inference at ingestion time would break content-addressing (signature verification). Inference is available at query time or before signing.

• 5D.7 Concept API Endpoints: CRUD for aliases and hierarchy browsing. ✅

  • Tasks:
    • POST /v1/concepts/alias — Create alias.
    • GET /v1/concepts/aliases — List all aliases (with optional canonical filter).
    • DELETE /v1/concepts/alias — Remove alias.
    • GET /v1/concepts/resolve — Resolve path to canonical/transitive aliases.
    • GET /v1/concepts/suggest — Suggested aliases (shared leaf detection).
    • GET /v1/concepts/parse — Parse path and return ConceptPath info.
  • Crate: stemedb-api

• 5D.8 Battery Tests: Validate concept hierarchy end-to-end. ✅

  • Tests:
    • Battery 8 (7 tests): ConceptPath parsing, round-trip, prefix matching, source class inference.
    • Battery 9 (8 tests): Alias resolution, transitive resolution, cycle detection, bidirectional lookup, delete, suggestions.
  • Crate: stemedb-query/tests/battery_pre_sentinel.rs

Phase 6: The Mesh (Distributed Writes)

Goal: Multi-node cluster with CRDT replication and Raft coordination. The endgame.

Research: docs/research/distributed-write-path.md Agent: distributed-systems-engineer Key Insight: Episteme's append-only model eliminates ~75% of CockroachDB complexity. Assertions are a G-Set CRDT. Votes are G-Counters. No distributed transactions needed.

6A. CRDT Foundation (Single-Node Validation) ✅ COMPLETE

• 6A.1 Integrate CRDT Crate: Wrap assertion storage in G-Set semantics.

  • Status: ✅ COMPLETE
  • Implementation:
    • CrdtAssertionStore in crates/stemedb-storage/src/crdt/assertion_store.rs — G-Set semantics for assertions.
    • CrdtVoteStore in crates/stemedb-storage/src/crdt/vote_store.rs — G-Counter semantics for votes.
    • CrdtMerge trait in crates/stemedb-storage/src/crdt/traits.rs for generic merge operations.
    • Property tests: commutativity, associativity, idempotence (proptest-based).
    • AssertionTransfer type for efficient cross-node data transfer.
  • Tests: 9 unit tests + 3 property tests (assertion_store), 6 unit tests (vote_store).
  • Note: Did not use external crdts crate — implemented native CRDT semantics over existing storage.

• 6A.2 Hybrid Logical Clocks: Add causal ordering to supersessions.

  • Status: ✅ COMPLETE
  • Implementation:
    • HlcTimestamp in crates/stemedb-core/src/types/hlc.rs — serializable HLC with uhlc integration.
    • Added uhlc = "0.8" dependency to stemedb-core.
    • HlcTimestamp::from_uhlc(), to_uhlc(), now() for clock management.
    • Total ordering via NTP64 time + node_id tiebreaker.
    • detect_clock_skew() utility for monitoring clock drift between nodes.
    • millis(), is_before(), is_concurrent_with() helper methods.
  • Tests: 10 unit tests covering ordering, equality, concurrency, serialization, clock skew detection.
  • Crate: uhlc = "0.8"

• 6A.3 Merkle Tree Over Assertions: Efficient diff detection.

  • Status: ✅ COMPLETE
  • Implementation:
    • New stemedb-merkle crate with BLAKE3-based Merkle tree.
    • MerkleTree struct: O(log N) insert, O(1) root, O(log N) diff.
    • DiffResult::diff() for computing missing hashes between trees.
    • roots_equal() for O(1) identity check.
    • Zero-copy serialization via rkyv for network transfer.
    • MerkleTreeManager in stemedb-sync for persistence and coordination.
  • Crate: crates/stemedb-merkle/

6B. Two-Node Replication (Proof of Concept) ✅ COMPLETE

Why "Proof of Concept": All primitives are implemented and unit/integration tested. The PoC validates that CRDT merge, HLC ordering, Merkle diff, gossip broadcast, and anti-entropy sync work correctly in isolation. Full network tests (two running gRPC servers, partition tolerance, concurrent writes) are deferred to 6C where cluster infrastructure provides a natural testing environment.

• 6B.1 RPC Layer: Node-to-node communication.

  • Status: ✅ COMPLETE
  • Implementation:
    • New stemedb-rpc crate with tonic gRPC.
    • proto/sync.proto defines: GossipRequest/Response, RootExchangeRequest/Response, FetchRequest/Response, PingRequest/Response, GetLeavesRequest/Response.
    • SyncClient in src/client.rs with RetryConfig for exponential backoff.
    • SyncServiceHandler in src/server.rs implementing SyncService trait.
    • SyncStorage trait for pluggable storage backends.
  • Crates: tonic = "0.12", prost = "0.13"
  • Crate: crates/stemedb-rpc/

• 6B.2 Gossip Broadcast: Push new assertions to peers.

  • Status: ✅ COMPLETE
  • Implementation:
    • GossipBroadcaster in crates/stemedb-sync/src/gossip.rs.
    • Configurable fanout (default: 3 peers).
    • Token bucket rate limiting via with_rate_limit().
    • Enable/disable support for maintenance windows.
    • Metrics: messages_sent, send_failures, rate_limited.
    • Best-effort delivery: failures logged but don't block ingestion.
    • GossipBroadcast trait in stemedb-ingest for dependency injection.
  • Tests: 3 unit tests (noop, no peers, enable/disable).

• 6B.3 Merkle Anti-Entropy Sync: Background convergence.

  • Status: ✅ COMPLETE
  • Implementation:
    • AntiEntropyWorker in crates/stemedb-sync/src/anti_entropy.rs.
    • Periodic root exchange via RootExchangeRequest.
    • compute_missing_hashes() compares local and remote leaf sets.
    • FetchRequest retrieves missing assertion data by hash.
    • Merge via CrdtAssertionStore::merge_with_data().
    • Merkle tree update after merge.
    • Configurable interval via SyncConfig.
    • Metrics: sync_cycles, sync_failures, assertions_synced.
    • Graceful shutdown support.
  • Tests: 1 unit test (SyncResult variants).

• 6B.4 Integration Test: Two-Node Convergence:

  • Status: ✅ COMPLETE (component-level validation)
  • Implementation:
    • battery11_replication.rs with 8 tests validating replication primitives:
      • test_identical_trees_same_root — Merkle root equality.
      • test_different_trees_different_roots — Merkle root divergence.
      • test_merkle_diff_finds_missing — Diff algorithm correctness.
      • test_gossip_enable_disable — Gossip control.
      • test_merkle_checkpoint_restore — Persistence roundtrip.
      • test_content_addressed_idempotent — Idempotent storage.
      • test_crdt_merge_with_data — CRDT merge semantics.
      • test_sync_config_builder — Configuration validation.
  • Note: Tests validate primitives in isolation. Live network tests (real gRPC servers, partition healing, concurrent writes) deferred to 6C cluster testing.
  • Crate: crates/stemedb-query/tests/battery/battery11_replication.rs

6C. Multi-Node Cluster ✅

• 6C.1 Cluster Membership (SWIM Gossip): Node discovery and failure detection.

  • Tasks:
    • Implement SwimMembership with SWIM-like protocol in stemedb-cluster.
    • NodeId (UUID-based), NodeInfo, NodeState, MembershipEvent types.
    • Seed-node based discovery (bootstrap nodes in config).
    • Failure detection: ping, indirect probe, suspicion with timeouts.
    • Membership change events via tokio::broadcast channel.
    • Gossip queue for piggybacked membership propagation.
    • ClusterConfig with SwimConfig (tunable intervals, timeouts).
  • Crate: stemedb-cluster

• 6C.2 Subject-Prefix Range Sharding: Distribute data across nodes.

  • Tasks:
    • Implement RangeRouter: map subject → shard via BLAKE3 + jump hash.
    • RangeDescriptor: start key, end key, replicas, size, generation.
    • MetaRange: collection of descriptors with version and merge logic.
    • Automatic range split when size exceeds threshold (configurable, default 64MB).
    • Range merge when adjacent ranges shrink below threshold (configurable, default 20MB).
    • Meta-range gossip merge for cluster-wide propagation.
    • ShardingConfig with tunable shard count, replication factor, thresholds.
  • Crate: stemedb-cluster

• 6C.3 Raft for MV Coordination (Optional): DEFERRED.

  • Decision: Skipped for this delivery. MVs are eventually consistent (converge once assertions sync via anti-entropy). Lenses are deterministic: same inputs produce same output. Can add Raft later if strong MV consistency becomes a requirement.

• 6C.4 Gateway: Stateless request routing.

  • Tasks:
    • Implement Gateway HTTP service (axum) with full routing.
    • Route writes by subject hash → shard → leader node.
    • Route reads to nearest replica (prefer local).
    • Health check endpoint (/v1/health).
    • Cluster status endpoint (/v1/cluster/status).
    • Shard info and route test endpoints.
    • CORS and tracing middleware.
  • Crate: stemedb-cluster

• 6C.5 Integration Tests: 82 tests covering membership, sharding, and gateway.

  • Membership: 3-node discovery, failure detection, rejoin, gossip propagation.
  • Sharding: routing consistency, distribution, split/merge, meta-range gossip.
  • Gateway: HTTP endpoint testing via axum oneshot for all routes.

6D. Consistency Guarantees

Property	Guarantee	Mechanism
Convergence	Eventually consistent	G-Set merge (CRDT)
Causality	Supersessions ordered	HLC timestamps
Partition Tolerance	Writes never blocked	Any node accepts via CRDT
Availability	Reads/writes always succeed	Every node is master for CRDTs
Durability	WAL + fsync per node	Existing WAL infra
Conflict Resolution	Deterministic	Lens algorithms (same inputs → same output)

Phase 7: The Shield (Trust at Scale)

Goal: Defend against spam, Sybil attacks, and knowledge poisoning when open to millions of agents.

Key Insight: Open agent access requires layered defense. PoW for admission, EigenTrust for reputation, circuit breakers for misbehavior, quarantine for suspicious content.

7A. Admission Control ✅

• 7A.1 Proof-of-Work Admission: BLAKE3 hashcash for new agents.

  • New agents (trust < 0.3) must solve PoW puzzle before first N assertions accepted.
  • Graduated difficulty: first 10 assertions = 16 bits (~16 sec), 11-50 = 1 bit (trivial), 50+ or trust > 0.6 = exempt.
  • Puzzle: BLAKE3(nonce || agent_id || timestamp) must have difficulty leading zero bits.
  • Implemented: PowProof struct, AdmissionLayer middleware, HTTP 428 responses.

• 7A.2 Graduated Trust Tiers: Privilege escalation based on reputation.

  • Untrusted (0.0-0.3): Quarantine mode, assertions hidden by default, 0.1x quota.
  • Limited (0.3-0.5): Low quota, PoW required, 0.5x quota.
  • Verified (0.5-0.7): Standard quota, normal privileges.
  • Trusted (0.7-0.9): 2x quota, skip PoW.
  • Authority (0.9-1.0): 10x quota, no limits.
  • Implemented: TrustTier enum, AdmissionStore trait, /v1/admission/status endpoint.

7B. EigenTrust ✅

• 7B.1 Trust Graph Store: Store direct trust relationships for propagation.

  • Key pattern: TG:{from}:{to} → TrustEdge, TGR:{to}:{from} → reverse index.
  • Methods: add_trust_edge(), get_trusts(), get_trusted_by(), compute_eigentrust().
  • Seed trust: ET:seed:{agent} for pre-trusted agents (P vector).

• 7B.2 EigenTrust Computation: Global trust via power iteration (daily batch).

  • Formula: T = (1-α)C^T*T + αP where C = normalized trust matrix, P = seed trust, α = 0.1.
  • Convergence: 10-100 iterations, ε = 1e-4 threshold.
  • Sybil resistance: isolated rings get near-zero trust (not connected to seeds).
  • Dangling node handling: redistribute to seed vector.

• 7B.3 Domain-Specific Trust: Per-predicate-namespace reputation.

  • DomainTrust tracks accuracy by domain (medicine, finance, technology, etc.).
  • extract_domain() maps predicates to domains.
  • domain_factor = 0.5 + (score × 0.5) scales weight by expertise.
  • EigenTrustAuthorityLens: weight = confidence × eigentrust × domain_factor.

7C. Content Defense ✅ COMPLETE

• 7C.1 MinHash Deduplication: Near-duplicate detection with LSH bucketing.

  • SimilarityIndex trait with GenericSimilarityIndex<S: KVStore> implementation.
  • MinHash signatures (k=128) for {subject}:{predicate} pairs.
  • LSH buckets (16 bands × 8 rows) for O(1) average-case lookup.
  • Bloom filter pre-check for fast "definitely not duplicate" path.
  • Threshold: 0.9 Jaccard similarity = duplicate.
  • Implemented: similarity_index/ module in stemedb-storage.

• 7C.2 Content Quality Scoring: Heuristic-based spam detection.

  • ContentQualityScorer with configurable thresholds.
  • Shannon entropy check (low entropy = likely random noise/repetitive).
  • Minimum subject/predicate length (3 chars default).
  • Structured data bonus (JSON objects, numbers, URLs).
  • Untrusted agent + high confidence (>0.8) = suspicious.
  • Implemented: content_defense/quality.rs in stemedb-storage.

• 7C.3 Quarantine Store: Suspicious assertions held for review.

  • QuarantineStore trait with GenericQuarantineStore<S: KVStore> implementation.
  • Key pattern: QUAR:{timestamp}:{hash} → assertion data (time-ordered).
  • Secondary index: QUAR_IDX:{hash} → timestamp for O(1) hash lookups.
  • Quarantined assertions NOT indexed (invisible to queries).
  • Triggers: quality < 0.4, duplicate, untrusted high-confidence.
  • Admin API: GET /v1/admin/quarantine, POST .../approve, POST .../reject.
  • ContentDefenseLayer integration in stemedb-ingest.

7D. Circuit Breakers

• 7D.1 Per-Agent Circuit Breakers: Ban misbehaving agents temporarily.
  • States: Closed (normal), Open (banned), HalfOpen (testing recovery).
  • 5 failures → Open for 30 seconds → HalfOpen → 1 success → Closed.
  • Metrics: circuit_breaker_state, circuit_breaker_trips_total.
  • Crate: governor for GCRA rate limiting.

Phase 8: The Swarm (Production Cluster)

Goal: Production-ready distributed deployment with chaos testing and observability.

8A. Chaos Testing

• 8A.1 Partition Testing: Verify convergence after network partitions.

  • 5-node cluster, kill 2 nodes, verify remaining nodes converge.
  • Network partition between groups, verify both sides accept writes, verify convergence after healing.
  • Message reordering and duplication tests.

• 8A.2 Jepsen-Style Consistency Testing: Formal verification.

  • Linearizability checks for Raft-coordinated operations.
  • Eventual consistency checks for CRDT-replicated data.
  • Clock skew injection (verify HLC handles drift).

8B. Observability

• 8B.1 Distributed Metrics: Per-node, per-range, per-agent metrics.

  • sync_lag_seconds{peer}, merkle_diff_size{peer}, convergence_latency_p99.
  • raft_leader_changes{range}, raft_commit_latency{range}.
  • assertions_total{node}, writes_per_second{node}.
  • Crate: metrics + metrics-exporter-prometheus.

• 8B.2 Admin Dashboard: Cluster health visibility.

  • GET /v1/admin/cluster → node list, range assignments, leader locations.
  • GET /v1/admin/ranges → range sizes, split/merge history.
  • POST /v1/admin/sync → force anti-entropy sync.

8C. Production Hardening

• 8C.1 Snapshot/Restore: Fast replica bootstrap.

  • Serialize full node state as snapshot.
  • New nodes join by restoring snapshot + replaying recent WAL.
  • Faster than full Merkle sync for new nodes.

• 8C.2 Backpressure: Don't overwhelm slow nodes.

  • Track per-peer sync queue depth.
  • Throttle gossip to slow peers.
  • Alert when peer is consistently behind.

• 8C.3 Geo-Distribution: Multi-region deployment.

  • Regional clusters with CRDT federation between regions.
  • Lazy cross-region sync (background, low priority).
  • Locality-aware reads (query nearest replica).
  • Regional compliance (GDPR data residency).

Phase 9: The Bunker (Disaster Planning)

Goal: Survive the worst. Backup, restore, recover from corruption, comply with regulations, and plan for unbounded growth.

Key Insight: Append-only CRDTs are a double-edged sword. They provide partition tolerance and conflict-free merge, but once bad data is merged, it's everywhere forever. Phase 9 addresses the failure modes that Phases 6-8 introduce.

9A. Backup & Cold Storage

• 9A.1 Full Cluster Backup: Point-in-time snapshot to cold storage.

  • Problem: 8C.1 snapshots are for node bootstrap, not disaster recovery. Need immutable backups to S3/GCS.
  • Tasks:
    • BackupCoordinator: elect leader, pause writes, snapshot all nodes, upload to object storage.
    • Incremental backups: WAL segments since last full backup.
    • Backup manifest: cluster topology, Merkle roots, HLC high-water mark.
    • Retention policy: 7 daily, 4 weekly, 12 monthly.
    • POST /v1/admin/backup/trigger, GET /v1/admin/backup/status.

• 9A.2 Point-in-Time Recovery (PITR): Restore to any timestamp.

  • Problem: "Restore yesterday's backup" isn't enough. Need "restore to 3:47pm yesterday."
  • Tasks:
    • WAL archiving to object storage (continuous).
    • Restore = snapshot + replay WAL until target HLC timestamp.
    • POST /v1/admin/restore?target_hlc=<timestamp>.
    • Validation: Merkle root matches expected state after restore.

• 9A.3 Backup Verification: Prove backups actually work.

  • Problem: Backups that can't restore are useless. Verify automatically.
  • Tasks:
    • Weekly "fire drill": restore backup to ephemeral cluster, run integrity checks.
    • Merkle root comparison: restored cluster root == source cluster root at backup time.
    • Alert on verification failure.
    • GET /v1/admin/backup/verification-history.

9B. Data Corruption & Rollback

• 9B.1 Corruption Detection: Catch bad data before it spreads.

  • Problem: Malformed assertions, invalid signatures, or logical corruption can poison the cluster via CRDT merge.
  • Tasks:
    • IngestionValidator: deep validation before accepting gossip (beyond signature check).
    • Schema validation: required fields, type constraints, value ranges.
    • Semantic validation: subject/predicate format, confidence bounds, timestamp sanity.
    • QuarantineStore: ✅ Implemented in Phase 7C. Extend with new QuarantineReason variants.
    • Metrics: assertions_quarantined, assertions_rejected.

• 9B.2 Assertion Tombstones: "Delete" in an append-only world.

  • Problem: Can't actually delete from a G-Set. Need a way to mark assertions as invalid.
  • Tasks:
    • TombstoneAssertion: special assertion type that marks another assertion as dead.
    • Tombstones propagate via CRDT like regular assertions.
    • Lenses skip tombstoned assertions during resolution.
    • POST /v1/admin/tombstone/{assertion_hash} (admin only).
    • Tombstone reasons: Corrupted, Malicious, Legal, Retracted.

• 9B.3 Cluster Rollback: "Undo" a time range across all nodes.

  • Problem: If bad data got merged cluster-wide, need to roll back the entire cluster.
  • Tasks:
    • RollbackCoordinator: elect leader, compute affected assertions, generate tombstones.
    • Input: time range (HLC from/to) or list of assertion hashes.
    • Output: batch of TombstoneAssertion propagated cluster-wide.
    • Audit log: who triggered rollback, why, what was affected.
    • POST /v1/admin/rollback?from_hlc=X&to_hlc=Y&reason=....

• 9B.4 Fork Recovery: Heal split-brain after extended partition.

  • Problem: Two clusters evolve independently during partition. After healing, they have divergent state that technically "merges" but may have semantic conflicts.
  • Tasks:
    • ForkDetector: identify assertions created during partition on each side.
    • ConflictReport: list all subject/predicate pairs with divergent winners.
    • Manual resolution: admin reviews conflicts, chooses winners, tombstones losers.
    • GET /v1/admin/fork-analysis, POST /v1/admin/fork-resolve.

9C. Compliance & Legal

• 9C.1 GDPR Right to Erasure: Handle deletion requests in append-only system.

  • Problem: GDPR requires "right to be forgotten." Append-only means data exists forever. Legal conflict.
  • Strategy: Cryptographic erasure — encrypt agent data with per-agent key, delete key to "erase."
  • Tasks:
    • Agent data encrypted with per-agent key (AES-256-GCM).
    • Key stored in AgentKeyStore (separate from assertion data).
    • "Erasure" = delete agent's key → their data becomes unreadable garbage.
    • Tombstones for their assertions (semantically dead).
    • DELETE /v1/agents/{agent_id} triggers erasure workflow.
    • Audit log: erasure requests, completion timestamp, affected assertion count.

• 9C.2 Data Retention Policies: Don't keep data forever.

  • Problem: Append-only doesn't mean keep-forever. Old data has storage cost and legal liability.
  • Tasks:
    • RetentionPolicy: per-subject or per-predicate retention rules.
    • Default: 7 years (financial), configurable per use case.
    • RetentionWorker: background job generates tombstones for expired assertions.
    • "Archive tier": cold storage for expired-but-not-deleted assertions.
    • GET/PUT /v1/admin/retention-policies.

• 9C.3 Audit Trail for Compliance: Prove what happened when.

  • Problem: Regulators ask "who changed what when." Need immutable audit log.
  • Tasks:
    • AuditStore: immutable log of admin actions (separate from assertions).
    • Events: backup, restore, rollback, tombstone, erasure, policy change.
    • Tamper-evident: Merkle chain over audit entries.
    • GET /v1/admin/audit?from=X&to=Y.
    • Export to external SIEM (Splunk, DataDog, etc.).

9D. Storage Management

• 9D.1 Compaction: Reclaim space from tombstoned data.

  • Problem: Tombstones don't free storage. Need compaction to actually reclaim space.
  • Tasks:
    • CompactionWorker: background job removes tombstoned assertions from storage.
    • Compaction delay: wait N days after tombstone before physical deletion.
    • Update Merkle tree after compaction (tree shrinks).
    • Compaction manifest: what was removed, when.
    • Metrics: storage_reclaimed_bytes, assertions_compacted.

• 9D.2 Tiered Storage: Hot/warm/cold based on access patterns.

  • Problem: Most queries hit recent data. Old assertions waste fast storage.
  • Tasks:
    • Hot tier: NVMe (< 30 days old, frequently accessed).
    • Warm tier: SSD (30-365 days, occasionally accessed).
    • Cold tier: Object storage (> 365 days, rarely accessed).
    • Transparent access: queries fetch from appropriate tier.
    • Migration worker: move data between tiers based on age/access.
    • Metrics: tier_hot_bytes, tier_warm_bytes, tier_cold_bytes.

• 9D.3 Storage Quotas: Prevent runaway growth.

  • Problem: Open agent access + append-only = potential unbounded growth.
  • Tasks:
    • Per-agent storage quota (in bytes or assertion count).
    • Per-subject storage quota (prevent subject stuffing).
    • Cluster-wide storage limit with alerting.
    • Rejection when quota exceeded: HTTP 429 with Retry-After.
    • GET /v1/admin/storage/usage, PUT /v1/admin/storage/quotas.

9E. Incident Response

• 9E.1 Alerting & Escalation: Know when things break.

  • Tasks:
    • Alert definitions: sync lag > 5min, Merkle divergence, node unreachable, storage > 80%.
    • Escalation tiers: P1 (page immediately), P2 (Slack + 15min), P3 (email).
    • Integration: PagerDuty, OpsGenie, Slack, email.
    • Runbook links in alerts (what to do when this fires).

• 9E.2 Operational Runbooks: Documented procedures for common failures.

  • Runbooks to write:
    • Node won't start (WAL corruption, disk full, config error).
    • Node behind on sync (network, slow disk, backpressure).
    • Cluster split-brain (partition detection, resolution).
    • Restore from backup (step-by-step with validation).
    • Emergency rollback (bad data merged, need to undo).
    • Capacity expansion (add nodes, rebalance ranges).
    • Security incident (compromised node, leaked keys).

• 9E.3 Chaos Engineering: Break things on purpose.

  • Problem: Can't trust disaster recovery you've never tested.
  • Tasks:
    • Scheduled chaos: monthly "game days" with controlled failures.
    • Scenarios: node death, network partition, disk corruption, clock skew.
    • Automated chaos: chaos-monkey style random failures in staging.
    • Post-mortem template and review process.

9F. Security Hardening

• 9F.1 TLS Everywhere: Encrypt all node-to-node traffic.

  • Tasks:
    • mTLS for gRPC (SyncService, gossip, anti-entropy).
    • Certificate rotation without downtime.
    • CA management: internal CA or external (Vault, ACME).
    • Reject unencrypted connections.

• 9F.2 Encryption at Rest: Protect stored data.

  • Tasks:
    • WAL encryption (AES-256-GCM).
    • KV store encryption (fjall supports this).
    • Key management: external KMS (AWS KMS, Vault) or local.
    • Key rotation without full re-encryption.

• 9F.3 Node Authentication: Verify cluster membership.

  • Tasks:
    • Node identity via Ed25519 keypair.
    • Cluster join requires signed invitation from existing member.
    • Revocation: remove compromised node's key, propagate via gossip.
    • Audit: log all join/leave/revoke events.

---

Tracking

Active Tasks

• Phase 3 The Pilot: Consumer Health vertical integration. ✅ COMPLETE
• Phase 4 The Hive: Trust & Scale + Extension Primitives. ✅ COMPLETE
• Phase 5 The Forge: Foundation hardening — replace sled, fix WAL, persist indices. ✅ COMPLETE
  • 5A: Replace sled with redb/fjall (HybridStore), key layout redesign. ✅ COMPLETE
  • 5B: WAL hardening — CRC32C, crash recovery, group commit, log rotation. ✅ COMPLETE
  • 5C: Index persistence — vector hot/cold, visual checkpoint. ✅ COMPLETE
  • 5D: Concept hierarchy — ConceptPath, AliasStore, scheme-based inference. ✅ COMPLETE

Phase 6 Progress

• 6A: CRDT Foundation — G-Set/G-Counter stores, HLC timestamps, Merkle tree. ✅ COMPLETE
• 6B: Two-Node Replication (PoC) — RPC layer, gossip, anti-entropy. ✅ COMPLETE
• 6C: Multi-Node Cluster — SWIM membership, range sharding, gateway. ✅ COMPLETE

Phase 7 Progress

• 7A: Admission Control — TrustTier, PowProof, AdmissionLayer, /v1/admission/status. ✅ COMPLETE
• 7B: EigenTrust — TrustGraphStore, DomainTrustStore, EigenTrustAuthorityLens. ✅ COMPLETE
• 7C: Content Defense — SimilarityIndex, ContentQualityScorer, QuarantineStore, Admin API. ✅ COMPLETE
• 7D: Circuit Breakers — Agent banning, automatic recovery.

Next Up

• Phase 7D: Circuit breakers (per-agent banning, automatic recovery).
• Phase 8: Chaos testing, observability, geo-distribution (The Swarm).

App Layer (External)

• Browser Extension Phase 1 (Read-Only Overlay) -> All DB dependencies complete. Extension is app layer.
• Browser Extension Phase 2 (Active Layer / Vote-to-See) -> ✅ All blockers resolved. 7A PoW + 7B EigenTrust complete.
• The Simulator (Training Data Pipeline) -> App layer, consumes Episteme API.
• The Super Curator (Reviewer Agent swarm) -> App layer.
• Background Gardener (Cluster detection, signal processing) -> App layer.
• Agent Wallet (Key management sidecar) -> App layer.

Recently Completed

• Phase 7C Content Defense (The Shield): Spam and duplicate detection with quarantine workflow.
  • SimilarityIndex trait with MinHash (k=128) + LSH (16 bands × 8 rows) for near-duplicate detection.
  • Bloom filter pre-check for O(1) "definitely not duplicate" fast path.
  • ContentQualityScorer with Shannon entropy, length checks, structured data detection.
  • QuarantineStore with time-ordered keys + O(1) hash index for admin lookups.
  • ContentDefenseLayer in stemedb-ingest orchestrating all checks.
  • Admin API: GET /v1/admin/quarantine, POST .../approve, POST .../reject.
  • Triggers: quality < 0.4, 0.9+ Jaccard similarity, untrusted + confidence > 0.8.
• Phase 6C Multi-Node Cluster (The Mesh): Distributed cluster infrastructure.
  • SwimMembership with SWIM gossip protocol for node discovery and failure detection.
  • RangeRouter with BLAKE3 + jump hash for subject-prefix range sharding.
  • Gateway HTTP service with routing, health checks, and read-your-writes.
  • 82 integration tests covering membership, sharding, availability, partition tolerance.
• Phase 7B EigenTrust (The Shield): Sybil-resistant global trust propagation.
  • TrustGraphStore trait with edge CRUD, seed trust management, EigenTrust computation.
  • Power iteration: T = (1-α)C^T*T + αP with dangling node handling.
  • DomainTrustStore for per-domain expertise tracking (medicine, finance, etc.).
  • EigenTrustAuthorityLens: weight = confidence × eigentrust × domain_factor.
  • Sybil resistance: isolated rings get near-zero trust (not connected to seeds).
• Phase 7A Admission Control (The Shield): PoW-based spam protection for new agents.
  • TrustTier enum with 5 tiers, quota multipliers, PoW requirements.
  • PowProof struct with BLAKE3 verification, graduated difficulty (16→1→0 bits).
  • AdmissionStore trait + AdmissionLayer middleware + /v1/admission/status endpoint.
  • Fail-open design for availability, milestone tracking for client UX.
• Phase 5D Concept Hierarchy: Hierarchical subjects with cross-scheme alias resolution.
  • ConceptPath struct with scheme://segments/leaf format, backward-compatible parsing.
  • SourceScheme enum mapping schemes to source tiers (rfc→Regulatory, code→Expert, etc.).
  • AliasStore trait with transitive resolution and cycle detection.
  • API: POST/DELETE /v1/concepts/alias, GET /v1/concepts/resolve|aliases|suggest|parse.
  • Battery 8 (7 tests) + Battery 9 (8 tests).
• Phase 5C Index Persistence: Vector hot/cold tiering, visual checkpoint.
• Phase 5B WAL Hardening: CRC32C checksums, crash recovery, group commit, log rotation.
• Gold Standard Verification (4.7): Sybil defense via proof of knowledge.
  • GoldStandard struct with rkyv serialization, GoldStandardStore trait + implementation.
  • TrustAdjustment enum: Rewarded(+0.05), Penalized(-0.1), AlreadyVerified.
  • Anti-gaming: agents can only verify each gold standard once (dedup markers).
  • API: POST/GET/DELETE /v1/admin/gold-standards, POST /v1/admin/verify-agent.
  • 21 trust rank tests including 4 new verification tests.
• Escalation Triggers (4.6): Active safety system for high-conflict assertions.
  • EscalationPolicy with configurable conflict_score thresholds and predicate matching.
  • EscalationEvent (content-addressed BLAKE3 ID) written by Materializer when policies fire.
  • EscalationStore at ESC:{timestamp_nanos}:{id_hex} with resolve workflow.
  • GET /v1/admin/escalations, POST /v1/admin/escalations/{id}/resolve.
• Conflict Score Filtering (4.5): Query-time filtering by conflict score.
  • min_conflict_score and max_conflict_score on Query + QueryParams.
  • Fast-path MV filtering with 0.0-1.0 validation.
  • 13 engine tests covering thresholds, boundaries, ranges, combinations.
• Vote Provenance Witness (4.4): Transform votes into cryptographic witnesses.
  • source_url: Option<String> and observed_context: Option<Vec<u8>> on Vote.
  • Input validation: URL max 2048 chars, context max 64KB.
  • Backward compatible with existing votes.
• "Since" Parameter (4.1): Change tracking for returning consumers.
  • since: Option<u64> on Query for incremental sync.
  • MV Changelog at MVC:{subject}:{predicate}:{timestamp_nanos} with nanosecond precision.
  • ChangeEntry struct with previous_winner_hash, new_winner_hash, lens_name.
  • fetch_changes_since() in QueryEngine with timestamp filtering.
  • changes_since: Option<Vec<ChangeEntryDto>> on QueryResponse.
  • 6 tests covering changelog creation, fetching, and fast path bypass.
• Batch TrustRank Decay API (4.3): Admin endpoint for scheduled trust decay.
  • POST /v1/admin/decay-trust-ranks with optional now and half_life_seconds.
  • Returns decayed_count, timestamp_used, half_life_used, status.
  • 3 integration tests.

Recently Completed

• Source Metadata Indexing (4.2): Index key source metadata fields for filtered queries.
  • New module: crates/stemedb-storage/src/source_metadata_index.rs.
  • Indexed fields: journal, doi, platform, study_design.
  • Key pattern: SMV:{field}:{value} -> Vec<Hash> (case-insensitive).
  • Query filters: ?source_journal=NEJM&source_platform=PubMed.
  • AND semantics for multiple metadata filters.
  • 18 unit tests covering indexing and query filtering.
• Source Document Storage (3F.1): Provenance lookup for 100% citation recall.
  • POST /v1/source stores source documents by BLAKE3 content hash.
  • GET /v1/provenance/{hash} retrieves source documents.
  • Content-addressed storage at SRC:{hash} keys.
  • Base64 encoding, 10MB limit, idempotent uploads.
  • 5 unit tests covering happy path and error cases.
• Epoch Cascade Logic (3D.1): O(1) supersession lookup via pre-computed markers.
  • write_supersession_cascade() writes SUPERSEDED: markers for full transitive closure at ingest time.
  • is_epoch_superseded() uses O(1) marker lookup instead of chain walking.
  • Cycle detection and max depth guard (100 levels).
  • 5 tests covering markers, transitive closure, cycles, and marker-based filtering.
• Semantic Decay (3B.2): Confidence half-life at query time.
  • decay_halflife: Option<u64> and source_class_decay: bool on Query.
  • New decay.rs module with apply_decay() and apply_source_class_decay().
  • Formula: effective_confidence = confidence * 2^(-(age / halflife)).
  • Tier-specific decay: Regulatory=none, Clinical=2yr, Anecdotal=30d.
  • 11 unit tests + 1 E2E integration test.
• Layered Consensus Lens (3C.2): Per-source-class consensus with tier-by-tier visibility.
  • LayeredConsensusLens with LayeredLens trait.
  • TierResolution and LayeredResolution types.
  • GET /v1/layered?subject=X&predicate=Y endpoint.
  • Cross-tier conflict score using Shannon entropy.
  • 10 comprehensive tests.
• Time-Travel Engine (3B.1): as_of parameter for historical queries.
  • as_of: Option<u64> field on Query for querying historical state.
  • Bypasses fast path (MVs reflect current state).
  • Query::matches() filters by assertion.timestamp <= as_of.
  • 6 tests covering edge cases.
• Rich Source Metadata (3A.3): Structured provenance beyond source_hash.
  • source_metadata: Option<Vec<u8>> field on Assertion.
  • Vec<u8> for rkyv zero-copy compatibility, callers handle JSON encoding.
  • Builder methods: .source_metadata_json() and .source_metadata().
  • API exposes as Option<String> with defensive UTF-8 handling.
  • 2 serialization tests.
• Conflict Score on Resolution (3A.2): Numeric disagreement metric across all lenses.
  • conflict_score: f32 field on Resolution (0.0 = unanimous, 1.0 = max conflict).
  • compute_conflict_score() utility using normalized variance.
  • Updated all 5 lens implementations to compute and propagate conflict score.
  • MaterializedView now stores conflict score.
  • API QueryResponse exposes conflict_score and resolution_confidence when lens is applied.
  • 7 unit tests including NaN handling.
• SkepticLens + SkepticView (3C.1): "Trust but Verify" conflict analysis that surfaces all claims with conflict scores.
  • AnalysisLens trait for lenses that map conflict instead of resolving it.
  • SkepticLens using normalized Shannon entropy for conflict scoring.
  • SkepticResolver + SkepticView in stemedb-query.
  • GET /v1/skeptic?subject=X&predicate=Y API endpoint.
  • Types: ResolutionStatus, ConflictAnalysis, ClaimSummary, SourceSummary, AgentSummary.
• Source-Class Field (3A.1): 6-tier SourceClass enum with authority weighting and decay rates.
  • SourceClass enum: Regulatory, Clinical, Observational, Expert, Community, Anecdotal.
  • tier(), default_decay_days(), authority_weight() methods.
  • Field on Assertion struct with full serialization support.
• The Meter: Token bucket quota system with MeterLayer middleware (10K tokens/agent/hour).
• Query Audit Trail: Every query logged with provenance at AUD:{query_id}. X-Agent-Id header for attribution.
• Event-Driven Materialization: run_notified() + IngestWorker Notify integration.
• Fast-Path MV Lookup: QueryEngine::try_fast_path() for O(1) reads.
• Materializer: Background worker for O(1) MV reads via AsyncLens.
• VoteAwareConsensusLens: Real vote-based consensus resolution.
• Compound SP Index: O(1) subject+predicate lookups.
• TrustRank System: Agent reputation with decay and learning loop.
• API Surface: axum HTTP server with 7 endpoints + OpenAPI docs.

Research Documents

• docs/research/wal-crash-recovery-research.md — WAL patterns from CockroachDB, TiKV, FoundationDB, SQLite.
• docs/research/distributed-write-path.md — Spanner/CockroachDB-style distributed writes adapted for append-only model.

Key Architectural Decisions

• sled → redb/fjall: sled is abandoned. HybridStore routes by key prefix: redb for reads, fjall for writes. ✅ COMPLETE
• Raft log = WAL: TiKV eliminated duplicate WAL in v5.4. We should too.
• CRDT for data, Raft for coordination: Assertions are a G-Set CRDT (merge = set union). Only cluster metadata needs Raft.
• Subject-prefix ranges: Co-locate all data for a subject on one shard. Split hot subjects via range split.
• HLC over TrueTime: Hybrid Logical Clocks work on commodity hardware. No GPS/atomic clocks needed.
• AP model: Writes never blocked during partitions. Eventual consistency via CRDT convergence.

Blockers

• Phase 5: ✅ COMPLETE — All foundation hardening done.
• Phase 6: ✅ COMPLETE — CRDT foundation, two-node replication, multi-node cluster.
• Phase 7A-7C: ✅ COMPLETE — Admission control + EigenTrust + Content Defense.
• Phase 7D: Unblocked. Can proceed with circuit breakers.
• Phase 8: Unblocked. Can proceed with chaos testing, observability, geo-distribution.
• Phase 9: Partially blocked. 9A-9B need Phase 8 (can't backup what doesn't exist). 9C-9F can start earlier (compliance planning, security design).

---

Dependency Graph

Phase 2.5 (Hardening)          Phase 3 (The Pilot)                    Phase 4 (The Hive)
========================       ========================               ==================

[2.1 MV Staleness]  ---------> [3B.1 Time-Travel] ✅ --+
                                       |                |
[2.2 Confidence Rename] -----> (API clarity for all)    +----------> [4.1 "Since" Param] ✅
                                                        |
[2.3 EpochAwareLens] --------> [3D.1 Epoch Cascade] ✅ |----------> Invalidation Cascades pillar
                                                        |
[2.4 Visual Hash Query] -----> [3E.2 Visual Hash Index] ✅
                                                        |
[2.6 E2E Integration] -------> (pipeline confidence)    |
                                                        |
                               [3A.1 Source-Class] ✅ --+----------> [3C.2 Layered Consensus] ✅
                                       |                             [3B.2 Semantic Decay] ✅
                                       +-----------------------------[4.2 Metadata Indexing] ✅
                                                        |
                               [3A.2 Conflict Score] ✅ --> (enhance Resolution)
                                                        |
                               [3A.3 Source Metadata] ✅ --> [4.2 Metadata Indexing] ✅
                                                        |
                               [3C.1 Skeptic Lens] ✅    (standalone, COMPLETE)
                               [3C.3 Constraints Lens] ✅ (standalone, COMPLETE)
                               [3E.1 Vector Search] ✅    (standalone, COMPLETE)
                               [3E.2 Visual Hash Index] ✅ (standalone, COMPLETE)
                               [3F.1 Provenance] ✅       (standalone, COMPLETE)

Critical Path for Consumer Health Demo

[3A.1 Source-Class] ✅ --> [3A.2 Conflict Score] ✅ --> [3C.2 Layered Consensus] ✅
                                                              |
                                                              +----> CONSUMER HEALTH MVP ✅
                                                              |
[3B.1 Time-Travel] ✅ ---------------------------------------+
                                                              |
[3A.3 Source Metadata] ✅ -----------------------------------+
                                                              |
[3C.1 Skeptic Lens] ✅ --------------------------------------+

Critical Path for Financial DD Demo

[3A.2 Conflict Score] ✅ --> [3C.1 Skeptic Lens] ✅ ---+
                                                         |
[3B.1 Time-Travel] ✅ ----------------------------------+----> FINANCIAL DD MVP
                                                         |
[2.3 EpochAwareLens] --> [3D.1 Epoch Cascade] ✅ -------+
                                                         |
[3B.2 Semantic Decay] ✅ -------------------------------+

Critical Path for Agile Agent Team Demo

[3C.3 Constraints Lens] (standalone) ------+
                                            |
[3B.1 Time-Travel] ✅ --------------------+----> AGENT TEAM MVP
                                            |
[2.3 EpochAwareLens] ✅ ------------------+
                                            |
[Query Audit (Phase 2)] ✅ ----------------+

Critical Path for Browser Extension

Phase 3 (Data Foundation)          Phase 4 (Extension Primitives)       Extension Product
=========================          ==============================       =================

[3A.1 Source-Class] ✅ ──────────> [4.5 Conflict Filtering] ✅ ──┐
                                                                   |
[3A.2 Conflict Score] ✅ ─────────────────────────────────────────┤
                                                                   |
[3C.1 Skeptic Lens] ✅ ──────────────────────────────────────────-┤
                                                                   ├──> PHASE 1: Read-Only Overlay
[3B.2 Semantic Decay] ✅ ────────────────────────────────────────-┤     (Benign Layer)
                                                                   |
[3C.2 Layered Consensus] ✅ ─────────────────────────────────────-┘

[Phase 2 Vote System] ✅ ────────> [4.4 Vote Provenance] ✅ ─────┐
                                                                   |
[TrustRank Engine] ✅ ───────────> [4.7 Gold Standards] ✅ ───────┤
                                                                   ├──> PHASE 2: Active Layer
[3A.2 Conflict Score] ✅ ────────> [4.6 Escalation Triggers] ✅ ──┤     (Vote to See)
                                                                   |
                                   [7A PoW Admission] ✅ ───────────┤
                                   [7B EigenTrust] ✅ ──────────────┘

Phase 1 (Read-Only) requires: Source tiers, conflict scores, conflict filtering, skeptic lens, decay, layered consensus. All complete.

Phase 2 (Active) requires: Vote provenance, gold standards, escalation triggers, PLUS Phase 7 Sybil defense. ✅ All complete. Ready to build.

Critical Path to Distributed Cluster

Phase 5 (The Forge) ✅              Phase 6 (The Mesh) ✅                 Phase 7+8
=======================             =======================              ==================

[5A.1 Replace sled ✅] ───────────> [6A.1 CRDT Foundation ✅] ──┐
                                            |                    |
[5A.2 Key Layout ✅] ────────────> [6C.2 Range Sharding ✅] ───> |
                                                                 |
[5B.1 CRC32C Checksums ✅] ──┐                                  |
[5B.2 Crash Recovery ✅] ────┼───> [6B.1 RPC Layer ✅] ─────────┤
[5B.3 Group Commit ✅] ──────┘            |                     |
                                          v                     |
[5C.1 Persistent Vector ✅] ─── (independent, no blocker)       |
[5C.2 Persistent Visual ✅] ─── (independent, no blocker)       |
                                                                 |
                                   [6A.2 HLC Timestamps ✅] ────┤
                                   [6A.3 Merkle Tree ✅] ───────┤
                                           |                     |
                                           v                     v
                                   [6B.2 Gossip ✅] ──> [6B.3 Anti-Entropy ✅] ──> [6B.4 PoC Tests ✅]
                                                                 |
                                                                 v
                                   [6C.1 SWIM Membership ✅] ───> [6C.3 Raft MV Coord] (DEFERRED)
                                   [6C.4 Gateway ✅] ───────────> │
                                                                 v
                                                    DISTRIBUTED CLUSTER ✅
                                                                 |
                                                    [7A PoW Admission ✅] ┐
                                                    [7B EigenTrust ✅] ──┤──> THE SHIELD
                                                    [7C Content Defense ✅]┤
                                                    [7D Circuit Breakers]┘
                                                              |
                                                    [8A Chaos Testing] ──┐
                                                    [8B Observability] ──┤──> THE SWARM
                                                    [8C Geo-Distribution]┘
                                                              |
                                                    [9A Backup/PITR] ─────┐
                                                    [9B Corruption/Rollback]┤
                                                    [9C GDPR/Retention] ──┤──> THE BUNKER
                                                    [9D Storage Mgmt] ────┤
                                                    [9E Incident Response]┤
                                                    [9F Security Hardening]┘

New Crates (Phases 5-9)

stemedb-merkle   (Phase 6A)  ── BLAKE3 Merkle tree for diff detection ✅ IMPLEMENTED
stemedb-rpc      (Phase 6B)  ── gRPC services for node-to-node communication ✅ IMPLEMENTED
stemedb-sync     (Phase 6B)  ── Merkle sync, gossip broadcast, anti-entropy ✅ IMPLEMENTED
stemedb-cluster  (Phase 6C)  ── Cluster membership, range routing, gateway ✅ IMPLEMENTED
stemedb-backup   (Phase 9A)  ── Backup coordination, PITR, verification (PLANNED)
stemedb-admin    (Phase 9B)  ── Tombstones, rollback, fork recovery, compliance (PLANNED)