jordan bbe6aedc40 feat: Aphoria security extractors + LLM evaluation architecture + ontology docs

New security extractors:
- insecure_deserialization, orm_injection, path_traversal, security_headers
- ssrf, unvalidated_redirects, weak_password, xxe
- Enhanced tls_version extractor with comprehensive cipher/protocol checks

Architecture docs:
- Scout-judge extraction pattern for LLM-based code analysis
- LLM prompt evaluation framework
- LLM eval implementation guide

Core improvements:
- stemedb-ontology README and client enhancements
- WAL journal/segment instrumentation
- Signing and ingestion refinements
- Consumer health demo script

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

2026-02-05 15:22:55 -07:00

112 KiB

Raw Blame History

Episteme (StemeDB) Roadmap

Goal: Build the "Git for Truth" substrate for autonomous AI research. Current Focus: Consumer Health MVP — proving the value proposition with real data Target Vertical: BioTech/Pharma ("The Living Review") Endgame: Distributed multi-writer cluster for millions of concurrent agents

Infrastructure Status: Phases 1-7 complete ✅ | Phase 8A (Chaos) complete ✅ Pilot Status: Consumer Health MVP in progress 🚧

High-Level Timeline

Phase	Codename	Focus	Key Deliverable
1	The Spine	Storage & Safety	Append-only WAL + KV Store
2	The Lattice	Indexing & Async	Materialized Views + Ballot Box
2.5	Hardening	Camp 2 Fixes	MV staleness, epoch behavior, lens cleanup
3	The Pilot	Vertical Integration	Pharma Ingestion + Living Review Agent
4	The Hive	Trust & Learning	TrustRank, metadata indexing, change tracking
5	The Forge	Foundation Hardening	Replace sled, fix WAL, persist indices, concept hierarchy
6	The Mesh	Distributed Writes	CRDT replication, Raft coordination, cluster membership
7	The Shield	Trust at Scale	EigenTrust, PoW admission, anti-spam, quarantine
8	The Swarm	Production Cluster	Chaos testing, observability, geo-distribution
9	The Bunker	Disaster Planning	Backup/restore, corruption recovery, GDPR compliance
MVP	Consumer Health	Prove Value	Real FDA data → conflicts detected → demo convinces

🎯 Consumer Health MVP (Current Focus)

The Question We Must Answer: Can Episteme demonstrate value that's impossible with Postgres?

Success Criteria: A stakeholder watches a 5-minute demo and says "I couldn't do that with a regular database."

MVP Definition of Done

Checkpoint	Description	Status
Real Data Flows	FDA drug labels for 3+ GLP-1 drugs ingested as signed assertions	🚧 Week 3
Conflicts Detected	SkepticLens shows `conflict_score > 0.5` when sources disagree	🚧 Week 3
Source Hierarchy Works	Tier 0 (FDA) outweighs 100x Tier 5 (anecdotal) volume	🚧 Week 3
Time Travel Works	`as_of=2024-01-01` returns historical snapshot	✅ Infrastructure ready
Decay Works	6-month-old Reddit claim has lower effective confidence than fresh FDA	✅ Infrastructure ready
UAT Passes	Consumer Health scenarios documented and verified	✅ Week 4
Self-Serve Demo	CLI tool lets anyone explore without code	✅ Week 5

The Demo Script

1. INGEST: "Here's semaglutide data from FDA, PubMed abstract, and a Reddit thread"
   → Show 3 assertions with different source_class values

2. CONFLICT: "FDA says nausea rate is 44%. Reddit anecdotes claim 80%."
   → Query with SkepticLens, show conflict_score = 0.72

3. HIERARCHY: "But FDA is Tier 0, Reddit is Tier 5."
   → Query with LayeredConsensusLens, show FDA wins despite volume

4. TIME TRAVEL: "What did we know 6 months ago?"
   → Query with as_of, show different winner before Reddit data existed

5. DECAY: "That Reddit data is 8 months old now."
   → Query with source_class_decay=true, show Reddit's effective confidence dropped

6. PARADIGM SHIFT: "FDA just updated the label."
   → Create new epoch, show old assertions superseded

MVP Workstream

Week	Deliverable	Owner	Depends On
Week 1 ✅	Domain definitions, SubjectBuilder, pharma schema	Ontology	—
Week 2 ✅	FDA extractor, claim-to-assertion signing	Ontology	Week 1
Week 3 ✅	Ingest FDA claims, mock conflicts, SkepticLens demo	Ontology	Week 2
Week 4 ✅	UAT scenarios documented and verified	Ontology	Week 3
Week 5 ✅	`steme-pharma` CLI for self-serve exploration	Ontology	Week 3
Week 6	Polish, factor out reusable patterns, document	Ontology	Week 4-5

What's NOT in MVP

These are valuable but not required to prove the core value proposition:

Phase 8B-C (Observability, geo-distribution) — production concerns
Phase 9 (Backup, GDPR, disaster recovery) — operational concerns
PubMed PDF extraction — complex, mock data sufficient for demo
Browser extension — app layer, consumes MVP

Detailed Milestones

Phase 1: The Spine (Foundation)

Goal: Securely ingest assertions and persist them without data loss.

Project Scaffold: Initialize Rust workspace, set up linting/CI (clippy, fmt).
Assertion Schema: Define the Assertion struct with rkyv serialization.
- Add dependencies: rkyv, blake3, ed25519-dalek, image_hasher.
- Define Assertion struct (Subject, Predicate, Object, Confidence, SourceHash).
- Multi-Sig Expansion: Implement SignatureEntry struct and signatures: Vec<SignatureEntry> field.
- Visual Expansion: Add visual_hash: Option<pHash> field for image provenance.
- Test serialization round-trips.
Ballot Schema: Define the Vote struct for multi-agent consensus.
- Add Vote struct: assertion_hash, agent_id, weight, signature.
- Test serialization round-trips.
Paradigm Schema (Epochs): Define the Epoch and SupersessionType structs.
- Add epoch: Option<EpochId> to Assertion.
- Implement Epoch struct with supersedes and SupersessionType.
- Test serialization round-trips.
WAL Integration: Implement the Quarantine Pattern for write-ahead logging.
- Create stemedb-wal crate.
- Port FsyncGuard and Record logic from established durability patterns.
- Implement Record format with BLAKE3 checksums and Headers.
- Verify fsync behavior with tests.
Storage Engine: Implement the Store trait using sled (embedded KV).
- Add sled dependency.
- Define KVStore trait (put, get, delete, scan_prefix, flush).
- Implement SledStore wrapper.
Basic Ingestor: Background worker that tails WAL and writes to KV.
- Implement async loop reading from WAL.
- Write deserialized assertions, votes, and epochs to sled.
- Ed25519 signature verification during ingestion.
- Maintains S: and SP: indexes on ingest.
- Persistent cursor/checkpoint (resumes from __CURSOR__:ingest in KV store).
Verification: Crash recovery tests (write -> crash -> restart -> read).
- Single and multi-record crash recovery.
- Multiple crash cycles tested.

Phase 2: The Lattice (Connectivity)

Goal: Query data with sub-millisecond latency using Materialized Views.

Lifecycle Schema: Add LifecycleStage to Assertion.
- Define enum: Proposed, UnderReview, Approved, Deprecated, Rejected.
- Update Assertion struct and serialization tests.
The Ballot Box: Implement high-velocity vote ingestion.
- VoteStore trait and implementation.
- VoteAwareConsensusLens for real vote-based resolution.
Index Infrastructure: Compound indexes for O(1) queries.
- IndexStore trait with S: and SP: indexes.
- QueryEngine smart routing (SP -> S -> scan).
Materializer: Background worker for O(1) Read Performance.
- MaterializedView type in stemedb-core.
- Materializer worker in stemedb-query with step() and run().
- Aggregates Votes via VoteAwareConsensusLens (or any AsyncLens).
- Updates MV:{Subject}:{Predicate} with the winning Assertion + metadata.
- Event-driven mode via run_notified() with tokio::sync::Notify.
- Fast-path MV lookup in QueryEngine::try_fast_path().
The Meter: Implement Economic Throttling (TAN).
- QuotaStore trait and GenericQuotaStore implementation.
- Token Bucket algorithm with per-agent per-hour quotas.
- MeterLayer tower middleware for request cost tracking.
- Cost model: Assert=10, Vote=1, Query=5+lens, +1/KB payload.
- GET /v1/meter/quota endpoint to check remaining quota.
- POST /v1/meter/quota/limit admin endpoint to set custom limits.
API Surface: axum HTTP server with OpenAPI (utoipa).
- POST /v1/assert -> Accepts JSON, writes to WAL.
- POST /v1/vote -> High-throughput vote endpoint.
- POST /v1/epoch -> Create epoch with optional supersession.
- GET /v1/query -> Subject/Predicate/Lens/Lifecycle/Epoch filtering.
- GET /v1/health -> Health check with assertion count.
- GET /swagger-ui -> Interactive API docs.
- 5 lens types available: Recency, Consensus, Authority, VoteAwareConsensus, TrustAwareAuthority.
Query Audit: Log every read with provenance.
- Define QueryAudit struct: query_id, agent_id, timestamp, params, result_hash, contributing_assertions.
- Storage at AUD:{query_id} with agent index at AUDA:{agent_id}:{timestamp}:{query_id}.
- GET /v1/audit/queries -> Returns history of agent decisions.
- GET /v1/audit/query/{id} -> Full reasoning trace for a single query.
- Auto-logging on every query via X-Agent-Id header.

Phase 2.5: Hardening (Camp 2 Fixes)

Goal: Close the gaps between "built" and "works right." Every item here addresses a feature that exists but doesn't fully deliver on its promise.

2.1 MV Staleness Detection: Make the fast-path aware of stale materialized views.
- Status: ✅ COMPLETE
- Implementation:
  - Added max_stale: Option<u64> to Query struct in crates/stemedb-query/src/query.rs.
  - Added .max_stale(secs) builder method to QueryBuilder.
  - In try_fast_path(): if query.max_stale is set and MV age exceeds threshold, falls through to slow path with debug! log.
  - Added max_stale to API QueryParams DTO in crates/stemedb-api/src/dto.rs.
  - Wired through query handler in crates/stemedb-api/src/handlers/query.rs.
- Tests:
  - test_fast_path_stale_view_falls_back: MV 1000 seconds old, max_stale = 60 → slow path used.
  - test_fast_path_fresh_view_used: Fresh MV, max_stale = 300 → fast path used.
  - test_fast_path_no_max_stale_always_uses_mv: No max_stale → any MV age accepted (backward compatible).
  - test_fast_path_max_stale_zero_rejects_old_mv: max_stale = 0, MV 1 second old → slow path.
  - test_fast_path_max_stale_zero_accepts_brand_new_mv: max_stale = 0, brand new MV → fast path.
2.2 AuthorityLens -> ConfidenceLens Rename: Eliminate the misleading name.
- Problem: AuthorityLens selects by confidence field, not by agent reputation. TrustAwareAuthorityLens is the real authority lens. The name creates confusion about what "Authority" means.
- Solution implemented:
  - Renamed authority.rs → confidence.rs, AuthorityLens → ConfidenceLens
  - Added LensDto::Confidence for the confidence-field selector
  - Changed LensDto::Authority to route to TrustAwareAuthorityLens (the real authority lens)
  - Updated query handler routing
  - Updated ai-lookup/services/lens.md and skill documentation
2.3 EpochAwareLens: Give epoch supersession runtime behavior.
- Status: ✅ COMPLETE
- Implementation:
  - EpochAwareLens in crates/stemedb-lens/src/epoch_aware.rs
  - Decorator pattern wrapping any inner lens (default: RecencyLens)
  - Walks supersession chain from E:{epoch_id} keys
  - Cycle detection + max depth guard (100)
  - Fail-open on missing epochs
  - LensDto::EpochAware added to API
  - 11 tests: excludes_superseded, chain_supersession, no_epochs_passes_all, missing_epoch_includes, cycle_detection, consensus_lens_inner, mixed_epochs, etc.
  - Documentation updated in ai-lookup/services/lens.md
- Known Limitation: Filtering only occurs when assertions from the superseding epoch are present in candidates. If all candidates are from old epoch (no new epoch assertions), they pass through (fail-open behavior).
2.4 Visual Hash Query Support: Make the stored visual_hash queryable.
- Status: ✅ COMPLETE
- Implementation:
  - hamming_distance(a: &PHash, b: &PHash) -> u32 in crates/stemedb-query/src/query.rs (lines 26-28)
  - visual_near: Option<String> and visual_threshold: Option<u32> in Query struct (lines 84-90)
  - .visual_near(hash, threshold) builder method
  - Query::matches() computes hamming distance when visual_near is set
  - API QueryParams DTO has visual_near and visual_threshold
  - 10+ tests: exact_match, within_threshold, exceeds_threshold, skips_without_hash, invalid_hex, wrong_length, combines_with_subject, default_threshold, max_threshold, threshold_63_rejects
- Note: Brute-force O(N) scan. VP-tree/BK-tree index is Phase 3+.
2.5 Vector Field: No changes needed. Already roadmapped for Phase 3.
- Status: ✅ N/A (No Phase 2 work required)
- Current state: vector: Option<Vec<f32>> on Assertion. Stored and returned by API. No index, no search.
- Phase 3 plan: Integrate hnsw-rs or lance for k-NN search.
2.6 E2E Integration Test (Write -> Materialize -> Read): Prove the full pipeline works end-to-end.
- Status: ✅ COMPLETE
- Implementation:
  - crates/stemedb-query/tests/e2e_pipeline.rs with 5 comprehensive tests:
    - test_e2e_write_materialize_read - Basic happy path
    - test_e2e_vote_consensus - Vote-weighted resolution
    - test_e2e_update_winner - Winner changes on re-materialize
    - test_e2e_cursor_persistence - Cursor survives worker restart
    - test_e2e_notify_integration - Event-driven notification channel
  - stemedb-wal and stemedb-ingest added as dev-dependencies
  - Helper functions: create_signed_assertion(), compute_assertion_hash(), create_vote()
  - Uses Ed25519 signing for authentic signature verification
- Also: crates/stemedb-api/tests/e2e_flow_test.rs tests the HTTP API layer end-to-end.

Phase 3: The Pilot (BioTech/Pharma)

Goal: Prove value in the "High-Liability" beachhead. Close every Camp 4 gap that blocks a credible demo.

3A. Schema Expansion (Prerequisite for everything below)

3A.1 Source-Class Field: Add source_class: SourceClass to Assertion.
- Status: ✅ COMPLETE
- Implementation:
  - SourceClass enum in crates/stemedb-core/src/types.rs (lines 68-88).
  - 6-tier system: Regulatory (0), Clinical (1), Observational (2), Expert (3), Community (4), Anecdotal (5).
  - tier() method returns tier number for ordering.
  - default_decay_days() method for tier-specific confidence decay.
  - authority_weight() method for conflict resolution weighting.
  - Field on Assertion struct at line 152.
  - Full serialization and indexing support.
3A.2 Conflict Score on Resolution: Add conflict_score: f32 to Resolution.
- Status: ✅ COMPLETE
- Implementation:
  - Added conflict_score: f32 field to Resolution in crates/stemedb-lens/src/traits.rs.
  - Updated Resolution::empty() to set conflict_score: 0.0.
  - Updated Resolution::with_winner() to accept conflict_score parameter.
  - Added compute_conflict_score(candidates: &[Assertion]) -> f32 utility function:
    - Uses normalized variance of confidence values.
    - 0 or 1 candidates → 0.0 (no conflict possible).
    - All same confidence → 0.0 (unanimous).
    - Max variance (0.0 vs 1.0) → 1.0 (maximum conflict).
    - Defensive NaN handling (returns 0.0 for malformed data).
  - Updated all lens implementations to compute and pass conflict score:
    - crates/stemedb-lens/src/recency.rs
    - crates/stemedb-lens/src/consensus.rs
    - crates/stemedb-lens/src/confidence.rs
    - crates/stemedb-lens/src/vote_aware_consensus.rs
    - crates/stemedb-lens/src/trust_aware_authority.rs
  - Added conflict_score: f32 to MaterializedView in crates/stemedb-core/src/types.rs.
  - Updated Materializer::materialize_pair() to write conflict_score from resolution.
  - Added conflict_score: Option<f32> and resolution_confidence: Option<f32> to QueryResponse DTO in crates/stemedb-api/src/dto.rs (only present when lens is applied).
  - Wired through query handler in crates/stemedb-api/src/handlers/query.rs.
- Tests:
  - test_conflict_score_zero_for_empty: Empty candidates → 0.0.
  - test_conflict_score_zero_for_single: 1 candidate → 0.0.
  - test_conflict_score_zero_for_agreement: All same confidence → near 0.0.
  - test_conflict_score_high_for_disagreement: Candidates at 0.1, 0.5, 0.9 → score > 0.3.
  - test_conflict_score_max_for_extremes: 0.0 vs 1.0 → score ≈ 1.0.
  - test_conflict_score_handles_nan_defensively: NaN confidences → 0.0 (fail-safe).
3A.3 Rich Source Metadata: Add structured provenance beyond source_hash.
- Status: ✅ COMPLETE
- Implementation:
  - Added source_metadata: Option<Vec<u8>> field to Assertion in crates/stemedb-core/src/types.rs (after epoch, before lifecycle).
  - Uses Vec<u8> (not String) for rkyv zero-copy compatibility. Callers encode/decode JSON on their side.
  - Added source_metadata: Option<Vec<u8>> to AssertionBuilder in crates/stemedb-core/src/testing.rs.
  - Added .source_metadata_json(json: &str) and .source_metadata(bytes) builder methods.
  - Added source_metadata: Option<String> to CreateAssertionRequest DTO (JSON string in API, converted to bytes internally).
  - Added source_metadata: Option<String> to AssertionResponse DTO (bytes converted to JSON string with defensive UTF-8 handling).
  - Wired through create handler (dto_to_assertion()) and query handler (assertion_to_dto()).
- Tests:
  - test_serialize_deserialize_assertion_with_metadata: Serialization roundtrip with metadata present.
  - test_serialize_deserialize_assertion_without_metadata: Serialization roundtrip with metadata absent.
- Note: Metadata is stored but NOT indexed in Phase 3. Indexing individual metadata fields is Phase 4+.

3B. Time & Decay (Core Query Features)

3B.1 Time-Travel Engine: as_of parameter for historical queries.
- Status: ✅ COMPLETE
- Implementation:
  - Added as_of: Option<u64> to Query struct in crates/stemedb-query/src/query.rs:92-99.
  - Added .as_of(timestamp: u64) to QueryBuilder.
  - In Query::matches(): if as_of is Some(ts), check assertion.timestamp <= ts. Assertions created after as_of are excluded.
  - In QueryEngine::execute(): if query.as_of is set, skip the fast path entirely (MVs reflect current state, not historical).
  - Added as_of: Option<u64> to QueryParams DTO in crates/stemedb-api/src/dto.rs.
  - Wired through query handler.
- Tests:
  - test_as_of_excludes_future_assertions: Assertions filtered by timestamp.
  - test_as_of_bypasses_fast_path: MV exists, but as_of is set. Slow path used.
  - test_as_of_none_uses_fast_path: Normal query still uses fast path (backwards-compatible).
  - test_as_of_with_lens_resolves_among_historical_candidates: Time-travel + lens = resolve only among pre-as_of candidates.
  - test_as_of_returns_empty_when_all_assertions_are_future: All assertions are future, returns empty.
  - test_as_of_with_exact_timestamp_match: Edge case where assertion.timestamp == as_of.
3B.2 Semantic Decay: Confidence Half-Life at query time.
- Status: ✅ COMPLETE
- Implementation:
  - Added decay_halflife: Option<u64> to Query struct in crates/stemedb-query/src/query.rs.
  - Added .decay_halflife(seconds: u64) and .source_class_decay(enabled: bool) to QueryBuilder.
  - Added decay_halflife and source_class_decay to QueryParams DTO.
  - Created new crates/stemedb-query/src/decay.rs module with:
    - apply_decay(): Uniform decay using formula confidence * 2^(-(age / halflife)).
    - apply_source_class_decay(): Tier-specific decay (Regulatory=none, Clinical=2yr, Anecdotal=30d).
    - compute_decayed_confidence(): Core decay calculation with clamping.
  - Integrated in QueryEngine::execute(): decay applied after filtering, before lens resolution.
  - Time-travel compatible: uses as_of timestamp if set, otherwise current time.
  - Source-class-aware decay fully implemented using SourceClass::default_decay_days().
- Tests: (11 unit tests in decay.rs + 1 E2E test)
  - test_decay_reduces_old_assertion_confidence: 1yr old, 1yr halflife → ~50% confidence.
  - test_decay_preserves_fresh_assertions: 1hr old, 1yr halflife → ~100% confidence.
  - test_decay_interacts_with_lens: Older high-confidence loses to newer low-confidence after decay.
  - test_source_aware_decay_tier0_no_decay: Regulatory never decays.
  - test_source_aware_decay_tier5_rapid_decay: Anecdotal decays rapidly (30-day halflife).
  - test_source_aware_decay_mixed_tiers: Clinical vs Anecdotal tier comparison.
  - test_decay_zero_halflife_no_change: Zero halflife skips decay (avoids div-by-zero).
  - test_decay_future_assertion_no_change: Future assertions don't decay.
  - test_decay_empty_assertions: Empty input returns empty output.
  - test_decay_confidence_clamps_to_valid_range: Very old assertions clamp to [0.0, 1.0].
  - test_decay_preserves_other_fields: Only confidence changes; other fields preserved.
  - test_e2e_decay_reduces_old_confidence: Full pipeline E2E test in e2e_pipeline.rs.
- Note: When decay is enabled, materialized views (fast path) are bypassed because MVs store pre-computed winners without decay applied.

3C. New Lenses

3C.1 Skeptic Lens: Surface disagreement, not winners. ✅ COMPLETED
- Status: ✅ COMPLETE
- Implementation:
  - crates/stemedb-lens/src/skeptic.rs - Full implementation.
  - AnalysisLens trait for lenses that analyze conflict instead of resolving it.
  - SkepticLens uses normalized Shannon entropy for conflict scoring.
  - Returns ConflictAnalysis with:
    - conflict_score: f32 (0.0 = unanimous, 1.0 = chaos)
    - status: ResolutionStatus (Unanimous, Agreed, Contested)
    - claims: Vec<ClaimSummary> - all claims ranked by weight
  - SkepticResolver + SkepticView in stemedb-query/src/skeptic.rs.
  - GET /v1/skeptic?subject=X&predicate=Y API endpoint.
  - Core types in stemedb-core/src/types.rs:
    - ResolutionStatus enum
    - ConflictAnalysis struct
    - ClaimSummary, SourceSummary, AgentSummary
  - Comprehensive test coverage (21 test cases).
3C.2 Layered Consensus Lens: Per-source-class consensus.
- Status: ✅ COMPLETE
- Implementation:
  - crates/stemedb-lens/src/layered_consensus.rs - Full implementation.
  - TierResolution struct: per-tier result with tier, source_class, winner, candidates_count, conflict_score, resolution_confidence.
  - LayeredResolution struct: multi-tier result with tiers vec, overall_winner, overall_conflict_score, total_candidates.
  - LayeredLens trait: resolve_layered(&[Assertion]) -> LayeredResolution, name() -> &'static str.
  - LayeredConsensusLens implements both LayeredLens and Lens traits.
  - Cross-tier conflict score uses normalized Shannon entropy of tier winner object values.
  - LensDto::LayeredConsensus variant (redirects to /v1/layered endpoint).
  - GET /v1/layered?subject=X&predicate=Y API endpoint with LayeredQueryResponse.
  - Exported from crates/stemedb-lens/src/lib.rs.
- Tests:
  - test_layered_empty_candidates: Empty input returns empty resolution.
  - test_layered_single_tier: All same source_class, returns one tier result.
  - test_layered_multi_tier_agreement: Tier 0 and Tier 5 agree, low cross-tier conflict.
  - test_layered_multi_tier_disagreement: Tier 1 vs Tier 5 disagree, high conflict, Tier 1 wins.
  - test_layered_overall_winner_from_highest_authority: Tier 0 wins despite fewer assertions.
  - test_layered_lens_trait_compatibility: Standard Lens trait works.
  - test_layered_within_tier_conflict: High internal conflict within a tier.
  - test_layered_all_tiers_present: One assertion from each tier.
  - test_layered_lens_name: Both trait names work.
  - test_layered_numeric_values: Works with numeric object values.
3C.3 Constraints Lens: Pre-flight check for must_use/forbidden.
- Status: ✅ COMPLETE
- Implementation:
  - crates/stemedb-lens/src/constraints.rs - Full implementation.
  - ConstraintSet struct: holds categorized assertions (must_use, forbidden, prefer) with conflict_score.
  - ConstraintsLens struct with resolve_constraints(&[Assertion]) -> ConstraintSet method.
  - Categorizes by predicate pattern: must_use:*, forbidden:*, prefer:*.
  - Implements Lens trait for compatibility (priority: must_use > forbidden > prefer).
  - Sorted by confidence (highest first), with timestamp as tiebreaker.
  - LensDto::Constraints added (redirects to /v1/constraints endpoint).
  - GET /v1/constraints?subject=X API endpoint with ConstraintsResponse.
  - DTOs: ConstraintsQueryParams, ConstraintEntryDto, ConstraintsResponse.
  - Exported from crates/stemedb-lens/src/lib.rs.
- Tests: (16 test cases)
  - test_constraints_categorizes_by_predicate: Mixed predicates sorted into must_use/forbidden/prefer.
  - test_constraints_empty_categories: Only prefer, no must_use/forbidden.
  - test_constraints_non_constraint_predicates_ignored: Regular predicates filtered out.
  - test_constraints_sorted_by_confidence: Within-category confidence ordering.
  - test_constraints_empty_candidates: Empty input returns empty set.
  - test_constraints_has_constraints_true: Helper method works.
  - test_constraints_all_regular_predicates: All non-constraint predicates returns no constraints.
  - test_lens_trait_picks_must_use_winner: Standard Lens trait picks must_use first.
  - test_lens_trait_falls_back_to_forbidden: Falls back to forbidden when no must_use.
  - test_lens_trait_falls_back_to_prefer: Falls back to prefer when no must_use/forbidden.
  - test_lens_trait_empty_for_no_constraints: Returns empty when no constraint predicates.
  - test_lens_name: Name returns "Constraints".
  - test_lens_empty_candidates: Empty input to Lens trait returns empty resolution.
  - test_multiple_must_use_picks_highest_confidence: Multiple must_use picks highest confidence.
  - test_confidence_tiebreaker_uses_timestamp: Same confidence uses newer timestamp.
  - test_predicate_pattern_exact_prefix: must_use_something not matched (only must_use:*).

3D. Epoch Enhancement

3D.1 Epoch Cascade Logic (enhancement of Phase 2.5 EpochAwareLens):
- Status: ✅ COMPLETE
- Implementation:
  - write_supersession_cascade() in crates/stemedb-ingest/src/worker.rs:
    - Writes SUPERSEDED:{old_epoch_id} markers for full transitive closure.
    - All markers point to the LATEST superseding epoch.
    - Max depth guard (100 levels) and cycle detection via visited set.
  - is_epoch_superseded() in crates/stemedb-lens/src/epoch_aware.rs:
    - O(1) marker lookup instead of O(chain_length) chain walks.
    - Fail-open semantics: missing marker = not superseded.
  - compute_superseded_epochs() uses marker lookups for filtering.
- Tests:
  - test_cascade_writes_superseded_marker: Epoch B supersedes A → SUPERSEDED:A exists.
  - test_cascade_transitive: C→B→A chain → both SUPERSEDED:A and SUPERSEDED:B point to C.
  - test_cascade_cycle_detection: Mutual supersession handled gracefully.
  - test_epoch_aware_uses_marker: EpochAwareLens uses O(1) marker lookup.
  - test_superseded_epoch_filtered_even_without_new_assertions: Marker-based filtering works.

3E. Similarity Search

3E.1 Vector Search: Semantic k-NN queries via embeddings.
- Status: ✅ COMPLETE
- Implementation:
  - Added hnsw_rs = "0.3" and parking_lot = "0.12" to stemedb-storage/Cargo.toml.
  - New module: crates/stemedb-storage/src/vector_index.rs.
  - VectorIndex trait: insert(hash: &Hash, vector: &[f32]), search(query: &[f32], k: usize) -> Vec<(Hash, f32)>, dimension(), len(), is_empty().
  - HnswVectorIndex implementation with HNSW graph, RwLock protection, hash↔ID mappings.
  - Input validation: dimension mismatch, NaN, Infinite values rejected.
  - Idempotent insert (same hash twice = no-op).
  - Arc<dyn VectorIndex> trait object support for sharing.
  - IngestWorker::with_vector_index() builder method for index attachment.
  - IngestWorker: if assertion has vector, inserts into vector index after KV write.
  - Added vector_near: Option<Vec<f32>> and k: Option<usize> to Query struct in crates/stemedb-query/src/query.rs.
  - Added .vector_near(vector, k) builder method to QueryBuilder.
  - Added vector_near and k to API QueryParams DTO.
  - QueryEngine::with_vector_index() builder method for index attachment.
  - QueryEngine: if vector_near is set and index configured, uses O(log N) HNSW lookup for candidates.
  - Falls back to standard path if no index configured (with debug log).
- Tests: (12 unit tests for VectorIndex + 4 integration tests in engine.rs)
  - test_create_index, test_insert_and_search, test_dimension_mismatch.
  - test_idempotent_insert, test_search_empty_index, test_search_k_zero.
  - test_nan_rejection, test_infinite_rejection, test_contains.
  - test_larger_scale (100 vectors, exact match first), test_custom_params, test_zero_dimension_panics.
  - test_vector_search_returns_nearest_neighbors, test_vector_search_with_subject_filter.
  - test_vector_search_without_index_falls_back, test_vector_search_with_as_of_filter.
- Note: Index is in-memory only. Persistence is Phase 4+.
3E.2 Visual Hash Index: BK-tree for O(log N) visual similarity.
- Status: ✅ COMPLETE
- Implementation:
  - New module: crates/stemedb-storage/src/visual_index.rs.
  - VisualIndex trait: insert(hash: &Hash, phash: &PHash), search(query: &PHash, threshold: u32) -> Vec<(Hash, u32)>, len(), is_empty().
  - BkTreeVisualIndex implementation using BK-tree over hamming distance.
  - hamming_distance(a: &PHash, b: &PHash) -> u32 utility function.
  - Threshold clamped to 0-64 range (max 64 bits).
  - Results sorted by distance ascending.
  - Idempotent insert (same hash twice = no-op).
  - Arc<dyn VisualIndex> trait object support for sharing.
  - IngestWorker::with_visual_index() builder method for index attachment.
  - IngestWorker: if assertion has visual_hash, inserts into BK-tree after KV write.
  - QueryEngine::with_visual_index() builder method for index attachment.
  - QueryEngine: if visual_near is set and index configured, uses O(log N) BK-tree lookup.
  - Falls back to brute-force scan (via query.matches()) if no index configured.
  - Invalid hex input returns QueryError::InvalidInput with clear message.
- Tests: (14 unit tests for VisualIndex + 6 integration tests in engine.rs)
  - test_hamming_distance_zero, test_hamming_distance_max, test_hamming_distance_partial.
  - test_create_index, test_insert_and_search_exact, test_search_within_threshold.
  - test_search_no_matches, test_search_empty_index, test_idempotent_insert.
  - test_contains, test_results_sorted_by_distance, test_threshold_clamped_to_64.
  - test_larger_scale (1000 hashes), test_default_impl.
  - test_visual_search_returns_similar_images, test_visual_search_with_lifecycle_filter.
  - test_visual_search_invalid_hex_returns_error, test_visual_search_without_index_uses_brute_force.
  - test_visual_search_with_limit, test_vector_search_empty_index.
- Note: Index is in-memory only. Persistence is Phase 4+.

3F. Provenance

3F.1 Source Document Storage & Provenance Lookup: Enable 100% citation recall.
- Status: ✅ COMPLETE
- Implementation:
  - POST /v1/source endpoint to store source documents by BLAKE3 content hash.
  - GET /v1/provenance/{hash} endpoint to retrieve source documents by hash.
  - Source storage at SRC:{hash} keys with format: [content_type_len:4][content_type][content].
  - Base64 encoding for binary-safe JSON transport.
  - 10MB size limit per document.
  - Content-addressed storage: same content → same hash (idempotent uploads).
  - DTOs: StoreSourceRequest, StoreSourceResponse, ProvenanceResponse.
  - OpenAPI documentation under "provenance" tag.
- Tests: (5 test cases)
  - test_store_and_retrieve_source: Happy path store + retrieve.
  - test_store_source_invalid_base64: Bad base64 → 400.
  - test_get_provenance_not_found: Unknown hash → 404.
  - test_get_provenance_invalid_hash: Bad hash format → 400.
  - test_store_source_idempotent: Same content twice → same hash.
- Note: Benchmark utility for verifying all assertions have retrievable sources is future work.

3G. API Cleanup

3G.1 Document epoch supersession via existing endpoint: No new /epoch/supersede endpoint needed.
- Status: ✅ COMPLETE
- Implementation:
  - Updated use case docs (consumer-health-intelligence.md, glp1-living-review.md) to use POST /v1/epoch with supersedes field.
  - Added OpenAPI examples showing both new epoch and supersession flows in handlers/epoch.rs.
  - Documented all 5 supersession types: Invalidate, Temporal, Refinement, RequiresReview, Additive.
- No code change. Documentation fix only.

Phase 4: The Hive (Trust & Scale)

Goal: Change tracking, metadata indexing, and the database primitives for training pipelines.

TrustRank Engine: Foundation for trust-based resolution.
- TrustRankStore for per-agent reputation storage.
- TrustAwareAuthorityLens for reputation-weighted resolution.
- Confidence Half-Life: Implement decay calculation engine.
- Learning loop: record_outcome() for accuracy tracking.
4.1 "Since" Parameter: Change tracking for returning consumers.
- Status: ✅ COMPLETE
- Problem: Consumer Health shows GET /query?since=2023-10-01 returning changes_since with dated change entries. The "returning consumer" story: "What changed since I last looked?"
- Depends on: Time-Travel (3B.1) and Materializer.
- Implementation:
  - Added since: Option<u64> to Query struct and QueryBuilder in crates/stemedb-query/src/query.rs.
  - Added since to QueryParams DTO in crates/stemedb-api/src/dto.rs.
  - MV Changelog: Track when materialized views change.
    - New key pattern: MVC:{subject}:{predicate}:{timestamp_nanos} using nanosecond precision to prevent collisions.
    - ChangeEntry struct in crates/stemedb-core/src/types.rs with: timestamp, previous_winner_hash, new_winner_hash, subject, predicate, lens_name.
    - In Materializer::materialize_pair(): before overwriting MV, read existing MV. If winner changed (different hash), write changelog entry.
    - get_previous_winner_hash() and write_changelog_entry() helper methods.
  - In QueryEngine: if since is set, skip fast path (MVs reflect current state). fetch_changes_since() scans MVC: keys and returns entries >= since timestamp.
  - ChangeEntryDto in crates/stemedb-api/src/dto.rs with hex-encoded hashes.
  - changes_since: Option<Vec<ChangeEntryDto>> added to QueryResponse DTO.
  - fetch_changelog_if_needed() helper in query handler.
  - Re-exported ChangeEntry from stemedb-core crate root.
- Tests:
  - test_changelog_written_on_first_materialization: First MV creates changelog with previous_winner_hash: None.
  - test_changelog_written_on_winner_change: Winner change creates changelog with previous hash.
  - test_no_changelog_when_winner_unchanged: Re-materialize same winner = no new changelog.
  - test_fetch_changes_since_returns_entries: Fetch filtered by timestamp, sorted ascending.
  - test_fetch_changes_since_empty_on_no_entries: No entries returns empty vec.
  - test_since_bypasses_fast_path: Query with since uses slow path, returns all assertions.
4.2 Source Metadata Indexing (extension of 3A.3): Index key metadata fields.
- Status: ✅ COMPLETE
- Problem: Phase 3 stores source_metadata as an opaque blob. Phase 4 makes key fields queryable.
- Depends on: Rich Source Metadata (3A.3).
- Implementation:
  - Defined indexed metadata keys: journal, doi, platform, study_design in INDEXED_METADATA_FIELDS.
  - New module: crates/stemedb-storage/src/source_metadata_index.rs.
  - Key pattern: SMV:{field}:{value} storing Vec<Hash> (rkyv-serialized) for efficient batch lookup.
  - SourceMetadataIndexStore trait with add_to_metadata_indexes() and get_by_metadata_field().
  - GenericSourceMetadataIndexStore implementation with case-insensitive value normalization.
  - IngestWorker: on ingestion, if source_metadata is present, parse JSON, extract indexed fields, write index entries.
  - Added source_journal, source_doi, source_platform, source_study_design to Query struct.
  - Added .source_journal(), .source_doi(), .source_platform(), .source_study_design() builder methods.
  - Added matches_metadata_filters() helper with AND semantics and case-insensitive matching.
  - Added metadata field filters to QueryParams DTO (e.g., ?source_journal=NEJM).
  - Wired through query handler in crates/stemedb-api/src/handlers/query.rs.
- Tests: (9 unit tests in source_metadata_index.rs + 9 unit tests in query.rs)
  - test_add_and_get_by_metadata_field: Basic round-trip.
  - test_case_insensitive_indexing: "NEJM" stored, "nejm" query matches.
  - test_multiple_assertions_same_field: Vec accumulates correctly.
  - test_malformed_json_gracefully_skipped: Invalid JSON logs warning, doesn't error.
  - test_missing_field_not_indexed: Only indexed fields processed.
  - test_idempotent_insert: Same assertion twice = no duplicates.
  - test_all_indexed_fields: All 4 fields indexed correctly.
  - test_empty_index_returns_empty_vec: Non-existent field returns empty.
  - test_non_string_values_ignored: Non-string JSON values skipped.
4.3 Batch TrustRank Decay API: Expose scheduled decay for external orchestration.
- Status: ✅ COMPLETE
- Implementation:
  - New module: crates/stemedb-api/src/handlers/admin.rs
  - POST /v1/admin/decay-trust-ranks endpoint.
  - Request accepts now: Option<u64> (defaults to current time) and half_life_seconds: Option<u64> (defaults to 30 days).
  - Response includes decayed_count, timestamp_used, half_life_used, status.
  - OpenAPI docs with "admin" tag.
  - Thin handler delegates to TrustRankStore::decay_trust_ranks().
- Tests: (3 integration tests in http_integration.rs)
  - test_decay_trust_ranks_empty_store: Empty store returns 0 decayed.
  - test_decay_trust_ranks_with_custom_params: Custom timestamp and half-life honored.
  - test_decay_trust_ranks_response_structure: All 4 fields present in response.
- Note: The Gardener (Camp 5.2, app layer) calls this endpoint on a schedule. The database just exposes the primitive.
4.4 Vote Provenance Witness: Transform votes from opinions into cryptographic witnesses.
- Status: ✅ COMPLETE
- Problem: Votes record "I agree" but the browser extension product needs "I saw this exact text at this URL at this time." Without provenance, votes are Tier 5 noise.
- Implementation:
  - Added source_url: Option<String> to Vote struct — URL where claim was observed.
  - Added observed_context: Option<Vec<u8>> to Vote struct — page context bytes (rkyv zero-copy pattern).
  - API DTOs: source_url and observed_context on CreateVoteRequest/VoteResponse.
  - Input validation: source_url max 2048 chars (non-empty if provided), observed_context max 64KB.
  - Backward compatible: existing votes without provenance remain valid.
- Tests: Serialization roundtrip (with/without provenance, mixed fields), storage roundtrip, API integration (validation).
4.5 Conflict Score Filtering: Query-time filtering by conflict score.
- Status: ✅ COMPLETE
- Problem: The browser extension needs "only show me claims where conflict > 0.7" to implement contradiction-only overlays. Conflict score was computed but not filterable.
- Implementation:
  - Added min_conflict_score: Option<f32> and max_conflict_score: Option<f32> to Query struct.
  - Builder methods: .min_conflict_score(), .max_conflict_score().
  - Fast-path filtering: checks MV conflict_score against thresholds.
  - API validation: scores must be 0.0-1.0, finite (rejects NaN/Inf).
  - QueryParams DTO wired through query handler.
- Tests: 13 engine tests (min/max thresholds, exact boundary, range, combination with lifecycle), API validation tests.
- Note: Filtering works on fast path (MV reads) only. Slow path returns raw assertions without conflict scores.
4.6 Escalation Triggers: Active safety system for high-conflict assertions.
- Status: ✅ COMPLETE
- Problem: High conflict was invisible. Episteme should be an active safety system that fires escalations when disagreement exceeds thresholds.
- Implementation:
  - EscalationLevel enum: Low, Medium, High, Critical.
  - EscalationEvent struct: content-addressed (BLAKE3 ID), subject, predicate, conflict_score, level, reason, timestamp, resolved.
  - EscalationPolicy struct: configurable threshold + level + optional predicate pattern.
  - EscalationStore trait + GenericEscalationStore: write, query since, resolve, get pending.
  - Key pattern: ESC:{timestamp_nanos}:{id_hex}.
  - Materializer integration: after computing conflict_score, checks policies and writes events.
  - with_escalation(store, policies) builder on Materializer.
  - GET /v1/admin/escalations and POST /v1/admin/escalations/{id}/resolve endpoints.
- Tests: Core type tests, storage roundtrip, materializer integration (high/low conflict, predicate matching, no store configured).
4.7 Gold Standard Verification: Sybil defense via proof of knowledge.
- Status: ✅ COMPLETE
- Problem: New agents need a way to earn TrustRank. Gold standards are admin-verified assertions that agents can be tested against.
- Implementation:
  - GoldStandard struct with rkyv serialization: assertion_hash, subject, predicate, expected_object, created_at, created_by.
  - GoldStandardStore trait + GenericGoldStandardStore: set, get, list, remove.
  - Key pattern: GS:{subject}:{predicate}.
  - TrustAdjustment enum: Rewarded(+0.05), Penalized(-0.1), AlreadyVerified.
  - verify_agent_against_gold_standard() on TrustRankStore with deduplication (agents can only verify each gold standard once).
  - Verification markers at GS_VERIFIED:{agent_id}:{subject}:{predicate} prevent gaming.
  - API: POST/GET/DELETE /v1/admin/gold-standards, POST /v1/admin/verify-agent.
- Tests: Core (creation, matching, case-sensitivity), storage CRUD, trust adjustment (reward, penalty, clamping, dedup), API integration (5 tests).

Note: The following items were reclassified as Application Layer responsibilities (see tmp/ambition-vs-reality.md, Camp 5). They are not Episteme database features. They consume the Episteme API and are built by integrators or vertical-specific teams.

The Simulator (Training Data Pipeline) -> Camp 5.3

The Super Curator (Reviewer Agent swarm) -> Camp 5.4

Background Gardener (Cluster detection, signal processing) -> Camp 5.2

Agent Wallet (Key management sidecar) -> Camp 5.1

Phase 5: The Forge (Foundation Hardening)

Goal: Replace abandoned dependencies, fix WAL gaps, persist indices. Prerequisite for distribution.

Research: docs/research/wal-crash-recovery-research.md

5A. Storage Engine Replacement

5A.1 Replace sled with redb + fjall: sled is abandoned (author recommends alternatives).
- Problem: sled is alpha-stage with known performance regressions and no active development. Our entire storage layer depends on it.
- Solution: HybridStore routes keys by prefix — fjall (LSM) for write-heavy paths (H:, V:, VC:, VW:, E:, SUPERSEDED:, __CURSOR__:) and redb (B-tree) for read-heavy paths (S:, SP:, MV:, TR:, QA:, QT:, TP:, GS:, ESC:).
- Tasks:
  - Generalize StorageError::Sled to StorageError::Backend(String).
  - Implement FjallStore backend with DashMap per-key locks for atomics.
  - Implement RedbStore backend with ACID transactions.
  - Implement HybridStore routing layer with prefix-based dispatch.
  - Migrate all ~500 tests from SledStore to HybridStore.
  - Remove sled dependency entirely.
  - Add criterion benchmarks (sequential put, random get, prefix scan, atomic increment, mixed workload).
- Crates: redb = "2", fjall = "2", dashmap = "6"

5A.2 Key Layout Redesign: Prepare keys for subject-prefix range sharding.

Problem: Current keys (H:{hash}, S:{subject}, MV:{subject}:{predicate}) scatter related data across the keyspace. Distributed sharding needs co-location.

Solution: Subject-prefix key layout with \x00 separator for subject-scoped keys, \x00 prefix for global keys (sort-first):

Subject-prefixed (co-located):
{subject}\x00H:{hash}       → Assertion data
{subject}\x00S:{hash_list}  → Subject index
{subject}\x00SP:{predicate} → Compound index
{subject}\x00MV:{predicate} → Materialized view
{subject}\x00V:{hash}:{vh}  → Votes
{subject}\x00VC:{hash}      → Vote count cache
{subject}\x00VW:{hash}      → Vote weight cache
{subject}\x00GS:{predicate} → Gold standards

Global (sort first via \x00 prefix):
\x00TRUST:{agent_id}        → Trust ranks
\x00QUOTA:{agent_id}:{win}  → Quota records
\x00QLIMIT:{agent_id}       → Quota limits
\x00E:{epoch_id}            → Epochs
\x00SUPERSEDED:{epoch_id}   → Supersession markers
\x00SUP:{hash}              → Supersession records
\x00AUD:{query_id}          → Audit records
\x00ESC:{ts}:{id}           → Escalation events
\x00TP:{pack_id}            → Trust packs
\x00META:{key}              → System metadata
\x00HASH_SUBJECT:{hash}     → Reverse lookup index
\x00SUBJECTS:{subject}      → Known subjects index
\x00GS_LIST:{subj}:{pred}   → Gold standard listing

Implementation:
- key_codec.rs (573 lines): 40+ key builder functions, subject validation, extraction utilities, 30+ unit tests.
- All stores migrated to key_codec:: functions (91 call sites across 10 store files, zero hardcoded key patterns).
- Ingestion pipeline uses key_codec (11 usages across 3 files).
- Query engine uses key_codec (34 usages across 7 files).
- Subject co-location verified by test_subject_colocation test.
- Global key sort-first verified by test_global_keys_sort_first test.

5B. WAL Hardening

5B.1 CRC32C Checksums: Add hardware-accelerated torn write detection.
- Status: ✅ COMPLETE
- Implementation:
  - Added crc32c = "0.6" crate dependency.
  - Updated WAL record format with CRC32C in format.rs.
  - CRC32C verified on read before deserializing.
  - Hardware-accelerated via SSE 4.2 on supported CPUs.
5B.2 Crash Recovery Implementation: Replace recovery stub with production recovery.
- Status: ✅ COMPLETE
- Implementation:
  - recovery/mod.rs (236 lines): Full sequential record scan with CRC32C validation.
  - Truncates file at first corrupted/incomplete record.
  - RecoveryMetrics struct tracks: valid_records, invalid_records, bytes_truncated, recovery_duration.
  - Comprehensive test suite in recovery/tests.rs.
5B.3 Group Commit: Batch fsync for throughput.
- Status: ✅ COMPLETE
- Implementation:
  - group_commit.rs (342 lines): GroupCommitBuffer with configurable max_writes and max_duration.
  - Writers append to buffer and wait on Notify.
  - Background flusher calls fsync and notifies all waiters.
  - GroupCommitConfig for tuning batch size and flush interval.
5B.4 Log Rotation: Bounded WAL disk usage.
- Status: ✅ COMPLETE
- Implementation:
  - segment.rs (368 lines): Full segment management.
  - Segment naming: {seq:08x}.wal.
  - Rotation when segment exceeds configurable threshold.
  - SegmentManager tracks active and archived segments.
  - Safe deletion of segments after cursor passes them.

5C. Index Persistence

5C.1 Persistent Vector Index: Move HNSW from in-memory to disk-backed.
- Status: ✅ COMPLETE
- Implementation:
  - PersistentVectorIndex in crates/stemedb-storage/src/vector_index/persistent.rs.
  - Hybrid hot/cold architecture:
    - Hot: In-memory HNSW for recent vectors.
    - Cold: Disk-backed HNSW loaded from checkpoint files.
  - persistence/format.rs: SnapshotMetadata, IdMappingTable with rkyv serialization.
  - hot_cold.rs: merge_search_results() for combining hot and cold query results.
  - Background checkpoint task with atomic write pattern.
  - CRC32C integrity verification on load.
  - MAX_PAYLOAD_SIZE (1GB) validation to prevent memory exhaustion.
- Crates: memmap2 = "0.9", crc32c = "0.6", byteorder = "1.5"
- Known Limitation: Cold index currently stores ID mappings only; full vector persistence with mmap'd HNSW graphs planned for future phase.
5C.2 Persistent Visual Index: Persist BK-tree to disk.
- Status: ✅ COMPLETE
- Implementation:
  - PersistentVisualIndex in crates/stemedb-storage/src/visual_index.rs.
  - BkTreeSnapshot with rkyv serialization for BK-tree state.
  - Checkpoint file format: [MAGIC:4][VERSION:1][RESERVED:3][PAYLOAD_LEN:u64][CRC32C:u32][PAYLOAD].
  - Atomic write pattern: temp file → fsync → rename → fsync parent.
  - Background checkpoint task with configurable interval.
  - CRC32C integrity verification on load.
  - Shared checkpoint_format.rs module for common read/write utilities.

5D. Concept Hierarchy ✅ COMPLETE

Spec: docs/specs/concept-hierarchy.md Purpose: Hierarchical, scheme-qualified subject identifiers with cross-scheme alias resolution. Enables applications like Aphoria that need to connect code:// paths to rfc:// paths.

5D.1 ConceptPath Type: Structured subject identifiers. ✅
- Tasks:
  - Add ConceptPath struct to stemedb-core/src/types/concept.rs.
  - Wire format: {scheme}://{segment_0}/{segment_1}/.../{leaf}.
  - parse(), to_wire_format(), leaf(), parent(), is_prefix_of().
  - Backward-compatible: bare strings parse as custom://{string}.
  - Unit tests for parsing, round-trip, prefix matching (Battery 8).
- Crate: stemedb-core
5D.2 Source Scheme Registry: Map schemes to default source tiers. ✅
- Tasks:
  - Add SourceScheme enum to stemedb-core.
  - Scheme → default SourceClass mapping (e.g., rfc:// → Tier 0, code:// → Tier 3).
  - ConceptPath::default_source_class() method.
- Crate: stemedb-core
5D.3 Alias Store: Cross-scheme entity resolution. ✅
- Tasks:
  - Add ConceptAlias struct to stemedb-core.
  - Add AliasStore trait to stemedb-storage.
  - Key prefixes: CA:{alias_path} → canonical, CAR:{canonical} → all aliases.
  - Transitive alias resolution with cycle detection.
  - GenericAliasStore implementation over KVStore.
- Crates: stemedb-core, stemedb-storage
5D.4 Hierarchical Query: Prefix-based subject queries. ✅
- Tasks:
  - fetch_by_subject_prefix() using scan_prefix in query engine (already implemented in Battery 5).
  - Trailing / handling to prevent auth matching authentication.
- Crate: stemedb-query
- Note: Hierarchical prefix scanning was already working; Battery 5 validates it.
5D.5 Alias Resolution in Queries: Expand queries to aliased paths. ✅
- Tasks:
  - AliasStore::resolve_all() for transitive alias expansion.
  - API endpoint GET /v1/concepts/resolve?path=...&transitive=true.
- Crate: stemedb-query, stemedb-api
- Note: Resolution available via API; QueryEngine integration is future work.
5D.6 Source Class Inference: Infer tier from scheme. ✅
- Tasks:
  - ConceptPath::default_source_class() returns tier based on scheme.
  - SourceScheme::parse() maps scheme strings to enum variants.
- Crate: stemedb-core
- Note: Inference at ingestion time would break content-addressing (signature verification). Inference is available at query time or before signing.
5D.7 Concept API Endpoints: CRUD for aliases and hierarchy browsing. ✅
- Tasks:
  - POST /v1/concepts/alias — Create alias.
  - GET /v1/concepts/aliases — List all aliases (with optional canonical filter).
  - DELETE /v1/concepts/alias — Remove alias.
  - GET /v1/concepts/resolve — Resolve path to canonical/transitive aliases.
  - GET /v1/concepts/suggest — Suggested aliases (shared leaf detection).
  - GET /v1/concepts/parse — Parse path and return ConceptPath info.
- Crate: stemedb-api
5D.8 Battery Tests: Validate concept hierarchy end-to-end. ✅
- Tests:
  - Battery 8 (7 tests): ConceptPath parsing, round-trip, prefix matching, source class inference.
  - Battery 9 (8 tests): Alias resolution, transitive resolution, cycle detection, bidirectional lookup, delete, suggestions.
- Crate: stemedb-query/tests/battery_pre_sentinel.rs

Phase 6: The Mesh (Distributed Writes)

Goal: Multi-node cluster with CRDT replication and Raft coordination. The endgame.

Research: docs/research/distributed-write-path.md Agent: distributed-systems-engineer Key Insight: Episteme's append-only model eliminates ~75% of CockroachDB complexity. Assertions are a G-Set CRDT. Votes are G-Counters. No distributed transactions needed.

6A. CRDT Foundation (Single-Node Validation) ✅ COMPLETE

6A.1 Integrate CRDT Crate: Wrap assertion storage in G-Set semantics.
- Status: ✅ COMPLETE
- Implementation:
  - CrdtAssertionStore in crates/stemedb-storage/src/crdt/assertion_store.rs — G-Set semantics for assertions.
  - CrdtVoteStore in crates/stemedb-storage/src/crdt/vote_store.rs — G-Counter semantics for votes.
  - CrdtMerge trait in crates/stemedb-storage/src/crdt/traits.rs for generic merge operations.
  - Property tests: commutativity, associativity, idempotence (proptest-based).
  - AssertionTransfer type for efficient cross-node data transfer.
- Tests: 9 unit tests + 3 property tests (assertion_store), 6 unit tests (vote_store).
- Note: Did not use external crdts crate — implemented native CRDT semantics over existing storage.
6A.2 Hybrid Logical Clocks: Add causal ordering to supersessions.
- Status: ✅ COMPLETE
- Implementation:
  - HlcTimestamp in crates/stemedb-core/src/types/hlc.rs — serializable HLC with uhlc integration.
  - Added uhlc = "0.8" dependency to stemedb-core.
  - HlcTimestamp::from_uhlc(), to_uhlc(), now() for clock management.
  - Total ordering via NTP64 time + node_id tiebreaker.
  - detect_clock_skew() utility for monitoring clock drift between nodes.
  - millis(), is_before(), is_concurrent_with() helper methods.
- Tests: 10 unit tests covering ordering, equality, concurrency, serialization, clock skew detection.
- Crate: uhlc = "0.8"
6A.3 Merkle Tree Over Assertions: Efficient diff detection.
- Status: ✅ COMPLETE
- Implementation:
  - New stemedb-merkle crate with BLAKE3-based Merkle tree.
  - MerkleTree struct: O(log N) insert, O(1) root, O(log N) diff.
  - DiffResult::diff() for computing missing hashes between trees.
  - roots_equal() for O(1) identity check.
  - Zero-copy serialization via rkyv for network transfer.
  - MerkleTreeManager in stemedb-sync for persistence and coordination.
- Crate: crates/stemedb-merkle/

6B. Two-Node Replication (Proof of Concept) ✅ COMPLETE

Why "Proof of Concept": All primitives are implemented and unit/integration tested. The PoC validates that CRDT merge, HLC ordering, Merkle diff, gossip broadcast, and anti-entropy sync work correctly in isolation. Full network tests (two running gRPC servers, partition tolerance, concurrent writes) are deferred to 6C where cluster infrastructure provides a natural testing environment.

6B.1 RPC Layer: Node-to-node communication.
- Status: ✅ COMPLETE
- Implementation:
  - New stemedb-rpc crate with tonic gRPC.
  - proto/sync.proto defines: GossipRequest/Response, RootExchangeRequest/Response, FetchRequest/Response, PingRequest/Response, GetLeavesRequest/Response.
  - SyncClient in src/client.rs with RetryConfig for exponential backoff.
  - SyncServiceHandler in src/server.rs implementing SyncService trait.
  - SyncStorage trait for pluggable storage backends.
- Crates: tonic = "0.12", prost = "0.13"
- Crate: crates/stemedb-rpc/
6B.2 Gossip Broadcast: Push new assertions to peers.
- Status: ✅ COMPLETE
- Implementation:
  - GossipBroadcaster in crates/stemedb-sync/src/gossip.rs.
  - Configurable fanout (default: 3 peers).
  - Token bucket rate limiting via with_rate_limit().
  - Enable/disable support for maintenance windows.
  - Metrics: messages_sent, send_failures, rate_limited.
  - Best-effort delivery: failures logged but don't block ingestion.
  - GossipBroadcast trait in stemedb-ingest for dependency injection.
- Tests: 3 unit tests (noop, no peers, enable/disable).
6B.3 Merkle Anti-Entropy Sync: Background convergence.
- Status: ✅ COMPLETE
- Implementation:
  - AntiEntropyWorker in crates/stemedb-sync/src/anti_entropy.rs.
  - Periodic root exchange via RootExchangeRequest.
  - compute_missing_hashes() compares local and remote leaf sets.
  - FetchRequest retrieves missing assertion data by hash.
  - Merge via CrdtAssertionStore::merge_with_data().
  - Merkle tree update after merge.
  - Configurable interval via SyncConfig.
  - Metrics: sync_cycles, sync_failures, assertions_synced.
  - Graceful shutdown support.
- Tests: 1 unit test (SyncResult variants).
6B.4 Integration Test: Two-Node Convergence:
- Status: ✅ COMPLETE (component-level validation)
- Implementation:
  - battery11_replication.rs with 8 tests validating replication primitives:
    - test_identical_trees_same_root — Merkle root equality.
    - test_different_trees_different_roots — Merkle root divergence.
    - test_merkle_diff_finds_missing — Diff algorithm correctness.
    - test_gossip_enable_disable — Gossip control.
    - test_merkle_checkpoint_restore — Persistence roundtrip.
    - test_content_addressed_idempotent — Idempotent storage.
    - test_crdt_merge_with_data — CRDT merge semantics.
    - test_sync_config_builder — Configuration validation.
- Note: Tests validate primitives in isolation. Live network tests (real gRPC servers, partition healing, concurrent writes) deferred to 6C cluster testing.
- Crate: crates/stemedb-query/tests/battery/battery11_replication.rs

6C. Multi-Node Cluster ✅

6C.1 Cluster Membership (SWIM Gossip): Node discovery and failure detection.
- Tasks:
  - Implement SwimMembership with SWIM-like protocol in stemedb-cluster.
  - NodeId (UUID-based), NodeInfo, NodeState, MembershipEvent types.
  - Seed-node based discovery (bootstrap nodes in config).
  - Failure detection: ping, indirect probe, suspicion with timeouts.
  - Membership change events via tokio::broadcast channel.
  - Gossip queue for piggybacked membership propagation.
  - ClusterConfig with SwimConfig (tunable intervals, timeouts).
- Crate: stemedb-cluster
6C.2 Subject-Prefix Range Sharding: Distribute data across nodes.
- Tasks:
  - Implement RangeRouter: map subject → shard via BLAKE3 + jump hash.
  - RangeDescriptor: start key, end key, replicas, size, generation.
  - MetaRange: collection of descriptors with version and merge logic.
  - Automatic range split when size exceeds threshold (configurable, default 64MB).
  - Range merge when adjacent ranges shrink below threshold (configurable, default 20MB).
  - Meta-range gossip merge for cluster-wide propagation.
  - ShardingConfig with tunable shard count, replication factor, thresholds.
- Crate: stemedb-cluster
6C.3 Raft for MV Coordination (Optional): DEFERRED.
- Decision: Skipped for this delivery. MVs are eventually consistent (converge once assertions sync via anti-entropy). Lenses are deterministic: same inputs produce same output. Can add Raft later if strong MV consistency becomes a requirement.
6C.4 Gateway: Stateless request routing.
- Tasks:
  - Implement Gateway HTTP service (axum) with full routing.
  - Route writes by subject hash → shard → leader node.
  - Route reads to nearest replica (prefer local).
  - Health check endpoint (/v1/health).
  - Cluster status endpoint (/v1/cluster/status).
  - Shard info and route test endpoints.
  - CORS and tracing middleware.
- Crate: stemedb-cluster
6C.5 Integration Tests: 82 tests covering membership, sharding, and gateway.
- Membership: 3-node discovery, failure detection, rejoin, gossip propagation.
- Sharding: routing consistency, distribution, split/merge, meta-range gossip.
- Gateway: HTTP endpoint testing via axum oneshot for all routes.

6D. Consistency Guarantees

Property	Guarantee	Mechanism
Convergence	Eventually consistent	G-Set merge (CRDT)
Causality	Supersessions ordered	HLC timestamps
Partition Tolerance	Writes never blocked	Any node accepts via CRDT
Availability	Reads/writes always succeed	Every node is master for CRDTs
Durability	WAL + fsync per node	Existing WAL infra
Conflict Resolution	Deterministic	Lens algorithms (same inputs → same output)

Phase 7: The Shield (Trust at Scale)

Goal: Defend against spam, Sybil attacks, and knowledge poisoning when open to millions of agents.

Key Insight: Open agent access requires layered defense. PoW for admission, EigenTrust for reputation, circuit breakers for misbehavior, quarantine for suspicious content.

7A. Admission Control ✅

7A.1 Proof-of-Work Admission: BLAKE3 hashcash for new agents.
- New agents (trust < 0.3) must solve PoW puzzle before first N assertions accepted.
- Graduated difficulty: first 10 assertions = 16 bits (~16 sec), 11-50 = 1 bit (trivial), 50+ or trust > 0.6 = exempt.
- Puzzle: BLAKE3(nonce || agent_id || timestamp) must have difficulty leading zero bits.
- Implemented: PowProof struct, AdmissionLayer middleware, HTTP 428 responses.
7A.2 Graduated Trust Tiers: Privilege escalation based on reputation.
- Untrusted (0.0-0.3): Quarantine mode, assertions hidden by default, 0.1x quota.
- Limited (0.3-0.5): Low quota, PoW required, 0.5x quota.
- Verified (0.5-0.7): Standard quota, normal privileges.
- Trusted (0.7-0.9): 2x quota, skip PoW.
- Authority (0.9-1.0): 10x quota, no limits.
- Implemented: TrustTier enum, AdmissionStore trait, /v1/admission/status endpoint.

7B. EigenTrust ✅

7B.1 Trust Graph Store: Store direct trust relationships for propagation.
- Key pattern: TG:{from}:{to} → TrustEdge, TGR:{to}:{from} → reverse index.
- Methods: add_trust_edge(), get_trusts(), get_trusted_by(), compute_eigentrust().
- Seed trust: ET:seed:{agent} for pre-trusted agents (P vector).
7B.2 EigenTrust Computation: Global trust via power iteration (daily batch).
- Formula: T = (1-α)C^T*T + αP where C = normalized trust matrix, P = seed trust, α = 0.1.
- Convergence: 10-100 iterations, ε = 1e-4 threshold.
- Sybil resistance: isolated rings get near-zero trust (not connected to seeds).
- Dangling node handling: redistribute to seed vector.
7B.3 Domain-Specific Trust: Per-predicate-namespace reputation.
- DomainTrust tracks accuracy by domain (medicine, finance, technology, etc.).
- extract_domain() maps predicates to domains.
- domain_factor = 0.5 + (score × 0.5) scales weight by expertise.
- EigenTrustAuthorityLens: weight = confidence × eigentrust × domain_factor.

7C. Content Defense ✅ COMPLETE

7C.1 MinHash Deduplication: Near-duplicate detection with LSH bucketing.
- SimilarityIndex trait with GenericSimilarityIndex<S: KVStore> implementation.
- MinHash signatures (k=128) for {subject}:{predicate} pairs.
- LSH buckets (16 bands × 8 rows) for O(1) average-case lookup.
- Bloom filter pre-check for fast "definitely not duplicate" path.
- Threshold: 0.9 Jaccard similarity = duplicate.
- Implemented: similarity_index/ module in stemedb-storage.
7C.2 Content Quality Scoring: Heuristic-based spam detection.
- ContentQualityScorer with configurable thresholds.
- Shannon entropy check (low entropy = likely random noise/repetitive).
- Minimum subject/predicate length (3 chars default).
- Structured data bonus (JSON objects, numbers, URLs).
- Untrusted agent + high confidence (>0.8) = suspicious.
- Implemented: content_defense/quality.rs in stemedb-storage.
7C.3 Quarantine Store: Suspicious assertions held for review.
- QuarantineStore trait with GenericQuarantineStore<S: KVStore> implementation.
- Key pattern: QUAR:{timestamp}:{hash} → assertion data (time-ordered).
- Secondary index: QUAR_IDX:{hash} → timestamp for O(1) hash lookups.
- Quarantined assertions NOT indexed (invisible to queries).
- Triggers: quality < 0.4, duplicate, untrusted high-confidence.
- Admin API: GET /v1/admin/quarantine, POST .../approve, POST .../reject.
- ContentDefenseLayer integration in stemedb-ingest.

7D. Circuit Breakers ✅

7D.1 Per-Agent Circuit Breakers: Ban misbehaving agents temporarily. ✅
- States: Closed (normal), Open (banned), HalfOpen (testing recovery).
- 5 failures → Open for 30 seconds → HalfOpen → 1 success → Closed.
- Failure types: InvalidSignature, InputValidation, PowError, QuotaExceeded, ApplicationError.
- CircuitBreakerStore trait + GenericCircuitBreakerStore<S: KVStore>.
- CircuitBreakerLayer middleware (outermost in stack, runs first).
- Admin API: GET /v1/admin/circuit-breaker/{agent_id}, POST .../reset, GET .../tripped.
- 25 unit tests covering full state machine lifecycle.

Phase 8: The Swarm (Production Cluster)

Goal: Production-ready distributed deployment with chaos testing and observability.

8A. Chaos Testing ✅

8A.1 Partition Testing: Verify convergence after network partitions.
- ✅ 5-node cluster, kill 2 nodes, verify remaining nodes converge.
- ✅ Network partition between groups, verify both sides accept writes, verify convergence after healing.
- ✅ Message reordering and duplication tests.
- ✅ Cascading failure recovery.
- ✅ SWIM suspicion handling (false positive prevention).
- ✅ Asymmetric partition testing.
- ✅ Write availability during partition.
- Crate: stemedb-chaos with TestCluster, ChaosNode, NetworkController.
8A.2 Jepsen-Style Consistency Testing: Formal verification.
- ✅ CRDT eventual consistency (1000 concurrent writes across 5 nodes).
- ✅ CRDT commutativity, associativity, idempotence verification.
- ✅ Clock skew injection (+5s/-5s), verify HLC handles drift.
- ✅ HLC monotonicity under partition.
- ✅ Supersession ordering with clock skew.
- ✅ Concurrent writes to same subject under partition (both survive).
- ✅ Large Merkle diff convergence (1500 vs 500 assertions).
- Crate: stemedb-chaos::crdt_properties with property verification functions.

8B. Observability

8B.1 Distributed Metrics: Per-node, per-range, per-agent metrics.
- sync_lag_seconds{peer}, merkle_diff_size{peer}, convergence_latency_p99.
- raft_leader_changes{range}, raft_commit_latency{range}.
- assertions_total{node}, writes_per_second{node}.
- Crate: metrics + metrics-exporter-prometheus.
8B.2 Admin Dashboard: Cluster health visibility.
- GET /v1/admin/cluster → node list, range assignments, leader locations.
- GET /v1/admin/ranges → range sizes, split/merge history.
- POST /v1/admin/sync → force anti-entropy sync.

8C. Production Hardening

8C.1 Snapshot/Restore: Fast replica bootstrap.
- Serialize full node state as snapshot.
- New nodes join by restoring snapshot + replaying recent WAL.
- Faster than full Merkle sync for new nodes.
8C.2 Backpressure: Don't overwhelm slow nodes.
- Track per-peer sync queue depth.
- Throttle gossip to slow peers.
- Alert when peer is consistently behind.
8C.3 Geo-Distribution: Multi-region deployment.
- Regional clusters with CRDT federation between regions.
- Lazy cross-region sync (background, low priority).
- Locality-aware reads (query nearest replica).
- Regional compliance (GDPR data residency).

Phase 9: The Bunker (Disaster Planning)

Goal: Survive the worst. Backup, restore, recover from corruption, comply with regulations, and plan for unbounded growth.

Key Insight: Append-only CRDTs are a double-edged sword. They provide partition tolerance and conflict-free merge, but once bad data is merged, it's everywhere forever. Phase 9 addresses the failure modes that Phases 6-8 introduce.

9A. Backup & Cold Storage

9A.1 Full Cluster Backup: Point-in-time snapshot to cold storage.
- Problem: 8C.1 snapshots are for node bootstrap, not disaster recovery. Need immutable backups to S3/GCS.
- Tasks:
  - BackupCoordinator: elect leader, pause writes, snapshot all nodes, upload to object storage.
  - Incremental backups: WAL segments since last full backup.
  - Backup manifest: cluster topology, Merkle roots, HLC high-water mark.
  - Retention policy: 7 daily, 4 weekly, 12 monthly.
  - POST /v1/admin/backup/trigger, GET /v1/admin/backup/status.
9A.2 Point-in-Time Recovery (PITR): Restore to any timestamp.
- Problem: "Restore yesterday's backup" isn't enough. Need "restore to 3:47pm yesterday."
- Tasks:
  - WAL archiving to object storage (continuous).
  - Restore = snapshot + replay WAL until target HLC timestamp.
  - POST /v1/admin/restore?target_hlc=<timestamp>.
  - Validation: Merkle root matches expected state after restore.
9A.3 Backup Verification: Prove backups actually work.
- Problem: Backups that can't restore are useless. Verify automatically.
- Tasks:
  - Weekly "fire drill": restore backup to ephemeral cluster, run integrity checks.
  - Merkle root comparison: restored cluster root == source cluster root at backup time.
  - Alert on verification failure.
  - GET /v1/admin/backup/verification-history.

9B. Data Corruption & Rollback

9B.1 Corruption Detection: Catch bad data before it spreads.
- Problem: Malformed assertions, invalid signatures, or logical corruption can poison the cluster via CRDT merge.
- Tasks:
  - IngestionValidator: deep validation before accepting gossip (beyond signature check).
  - Schema validation: required fields, type constraints, value ranges.
  - Semantic validation: subject/predicate format, confidence bounds, timestamp sanity.
  - QuarantineStore: ✅ Implemented in Phase 7C. Extend with new QuarantineReason variants.
  - Metrics: assertions_quarantined, assertions_rejected.
9B.2 Assertion Tombstones: "Delete" in an append-only world.
- Problem: Can't actually delete from a G-Set. Need a way to mark assertions as invalid.
- Tasks:
  - TombstoneAssertion: special assertion type that marks another assertion as dead.
  - Tombstones propagate via CRDT like regular assertions.
  - Lenses skip tombstoned assertions during resolution.
  - POST /v1/admin/tombstone/{assertion_hash} (admin only).
  - Tombstone reasons: Corrupted, Malicious, Legal, Retracted.
9B.3 Cluster Rollback: "Undo" a time range across all nodes.
- Problem: If bad data got merged cluster-wide, need to roll back the entire cluster.
- Tasks:
  - RollbackCoordinator: elect leader, compute affected assertions, generate tombstones.
  - Input: time range (HLC from/to) or list of assertion hashes.
  - Output: batch of TombstoneAssertion propagated cluster-wide.
  - Audit log: who triggered rollback, why, what was affected.
  - POST /v1/admin/rollback?from_hlc=X&to_hlc=Y&reason=....
9B.4 Fork Recovery: Heal split-brain after extended partition.
- Problem: Two clusters evolve independently during partition. After healing, they have divergent state that technically "merges" but may have semantic conflicts.
- Tasks:
  - ForkDetector: identify assertions created during partition on each side.
  - ConflictReport: list all subject/predicate pairs with divergent winners.
  - Manual resolution: admin reviews conflicts, chooses winners, tombstones losers.
  - GET /v1/admin/fork-analysis, POST /v1/admin/fork-resolve.

9C. Compliance & Legal

9C.1 GDPR Right to Erasure: Handle deletion requests in append-only system.
- Problem: GDPR requires "right to be forgotten." Append-only means data exists forever. Legal conflict.
- Strategy: Cryptographic erasure — encrypt agent data with per-agent key, delete key to "erase."
- Tasks:
  - Agent data encrypted with per-agent key (AES-256-GCM).
  - Key stored in AgentKeyStore (separate from assertion data).
  - "Erasure" = delete agent's key → their data becomes unreadable garbage.
  - Tombstones for their assertions (semantically dead).
  - DELETE /v1/agents/{agent_id} triggers erasure workflow.
  - Audit log: erasure requests, completion timestamp, affected assertion count.
9C.2 Data Retention Policies: Don't keep data forever.
- Problem: Append-only doesn't mean keep-forever. Old data has storage cost and legal liability.
- Tasks:
  - RetentionPolicy: per-subject or per-predicate retention rules.
  - Default: 7 years (financial), configurable per use case.
  - RetentionWorker: background job generates tombstones for expired assertions.
  - "Archive tier": cold storage for expired-but-not-deleted assertions.
  - GET/PUT /v1/admin/retention-policies.
9C.3 Audit Trail for Compliance: Prove what happened when.
- Problem: Regulators ask "who changed what when." Need immutable audit log.
- Tasks:
  - AuditStore: immutable log of admin actions (separate from assertions).
  - Events: backup, restore, rollback, tombstone, erasure, policy change.
  - Tamper-evident: Merkle chain over audit entries.
  - GET /v1/admin/audit?from=X&to=Y.
  - Export to external SIEM (Splunk, DataDog, etc.).

9D. Storage Management

9D.1 Compaction: Reclaim space from tombstoned data.
- Problem: Tombstones don't free storage. Need compaction to actually reclaim space.
- Tasks:
  - CompactionWorker: background job removes tombstoned assertions from storage.
  - Compaction delay: wait N days after tombstone before physical deletion.
  - Update Merkle tree after compaction (tree shrinks).
  - Compaction manifest: what was removed, when.
  - Metrics: storage_reclaimed_bytes, assertions_compacted.
9D.2 Tiered Storage: Hot/warm/cold based on access patterns.
- Problem: Most queries hit recent data. Old assertions waste fast storage.
- Tasks:
  - Hot tier: NVMe (< 30 days old, frequently accessed).
  - Warm tier: SSD (30-365 days, occasionally accessed).
  - Cold tier: Object storage (> 365 days, rarely accessed).
  - Transparent access: queries fetch from appropriate tier.
  - Migration worker: move data between tiers based on age/access.
  - Metrics: tier_hot_bytes, tier_warm_bytes, tier_cold_bytes.
9D.3 Storage Quotas: Prevent runaway growth.
- Problem: Open agent access + append-only = potential unbounded growth.
- Tasks:
  - Per-agent storage quota (in bytes or assertion count).
  - Per-subject storage quota (prevent subject stuffing).
  - Cluster-wide storage limit with alerting.
  - Rejection when quota exceeded: HTTP 429 with Retry-After.
  - GET /v1/admin/storage/usage, PUT /v1/admin/storage/quotas.

9E. Incident Response

9E.1 Alerting & Escalation: Know when things break.
- Tasks:
  - Alert definitions: sync lag > 5min, Merkle divergence, node unreachable, storage > 80%.
  - Escalation tiers: P1 (page immediately), P2 (Slack + 15min), P3 (email).
  - Integration: PagerDuty, OpsGenie, Slack, email.
  - Runbook links in alerts (what to do when this fires).
9E.2 Operational Runbooks: Documented procedures for common failures.
- Runbooks to write:
  - Node won't start (WAL corruption, disk full, config error).
  - Node behind on sync (network, slow disk, backpressure).
  - Cluster split-brain (partition detection, resolution).
  - Restore from backup (step-by-step with validation).
  - Emergency rollback (bad data merged, need to undo).
  - Capacity expansion (add nodes, rebalance ranges).
  - Security incident (compromised node, leaked keys).
9E.3 Chaos Engineering: Break things on purpose.
- Problem: Can't trust disaster recovery you've never tested.
- Tasks:
  - Scheduled chaos: monthly "game days" with controlled failures.
  - Scenarios: node death, network partition, disk corruption, clock skew.
  - Automated chaos: chaos-monkey style random failures in staging.
  - Post-mortem template and review process.

9F. Security Hardening

9F.1 TLS Everywhere: Encrypt all node-to-node traffic.
- Tasks:
  - mTLS for gRPC (SyncService, gossip, anti-entropy).
  - Certificate rotation without downtime.
  - CA management: internal CA or external (Vault, ACME).
  - Reject unencrypted connections.
9F.2 Encryption at Rest: Protect stored data.
- Tasks:
  - WAL encryption (AES-256-GCM).
  - KV store encryption (fjall supports this).
  - Key management: external KMS (AWS KMS, Vault) or local.
  - Key rotation without full re-encryption.
9F.3 Node Authentication: Verify cluster membership.
- Tasks:
  - Node identity via Ed25519 keypair.
  - Cluster join requires signed invitation from existing member.
  - Revocation: remove compromised node's key, propagate via gossip.
  - Audit: log all join/leave/revoke events.

Tracking

Active Tasks

Phase 3 The Pilot: Consumer Health vertical integration. ✅ COMPLETE
Phase 4 The Hive: Trust & Scale + Extension Primitives. ✅ COMPLETE
Phase 5 The Forge: Foundation hardening — replace sled, fix WAL, persist indices. ✅ COMPLETE
- 5A: Replace sled with redb/fjall (HybridStore), key layout redesign. ✅ COMPLETE
- 5B: WAL hardening — CRC32C, crash recovery, group commit, log rotation. ✅ COMPLETE
- 5C: Index persistence — vector hot/cold, visual checkpoint. ✅ COMPLETE
- 5D: Concept hierarchy — ConceptPath, AliasStore, scheme-based inference. ✅ COMPLETE

Phase 6 Progress

6A: CRDT Foundation — G-Set/G-Counter stores, HLC timestamps, Merkle tree. ✅ COMPLETE
6B: Two-Node Replication (PoC) — RPC layer, gossip, anti-entropy. ✅ COMPLETE
6C: Multi-Node Cluster — SWIM membership, range sharding, gateway. ✅ COMPLETE

Phase 7 Progress

7A: Admission Control — TrustTier, PowProof, AdmissionLayer, /v1/admission/status. ✅ COMPLETE
7B: EigenTrust — TrustGraphStore, DomainTrustStore, EigenTrustAuthorityLens. ✅ COMPLETE
7C: Content Defense — SimilarityIndex, ContentQualityScorer, QuarantineStore, Admin API. ✅ COMPLETE
7D: Circuit Breakers — CircuitBreakerStore, CircuitBreakerLayer middleware, Admin API. ✅ COMPLETE

🎯 Consumer Health MVP Progress (Current Focus)

Week 1: Domain Definition Core — Domain struct, SubjectBuilder, pharma definition. ✅ COMPLETE
Week 2: FDA Extractor + Signing — FdaLabelExtractor, MedicalClaim::to_assertion(), exponential backoff. ✅ COMPLETE
Week 3: StemeDB Integration — StemeClient, pharma-ingest CLI, mock conflict demo. ✅ COMPLETE
Week 4: UAT Scenarios — Document acceptance criteria, validation tests. ✅ COMPLETE
Week 5: CLI Tool — steme-pharma CLI for ingest/query/compare. ✅ COMPLETE
Week 6: Generalization — Factor out reusable patterns, document "Adding a Domain".

Next Up

Week 6 MVP: Factor out reusable patterns, document "Adding a Domain" guide.
Phase 8B-C (deferred): Observability, geo-distribution — production concerns, not MVP blockers.

App Layer (External)

Browser Extension Phase 1 (Read-Only Overlay) -> All DB dependencies complete. Extension is app layer.
Browser Extension Phase 2 (Active Layer / Vote-to-See) -> ✅ All blockers resolved. 7A PoW + 7B EigenTrust complete.
The Simulator (Training Data Pipeline) -> App layer, consumes Episteme API.
The Super Curator (Reviewer Agent swarm) -> App layer.
Background Gardener (Cluster detection, signal processing) -> App layer.
Agent Wallet (Key management sidecar) -> App layer.

Recently Completed

🎯 MVP Week 5: steme-pharma CLI for self-serve exploration.
- Full CLI binary with 5 subcommands: ingest, query, compare, explore, validate.
- Query modes: skeptic (default), layered (per-tier), and lens-based (recency, consensus, etc.).
- Table and JSON output formats via comfy-table.
- Client extensions: layered(), query(), list_predicates() methods.
- Response DTOs: LayeredResponse, QueryResponse, AssertionDto, TierResolutionDto.
- Domain validation for known pharma predicates and subject patterns.
- Modular design: cli.rs, commands.rs, helpers.rs, output.rs.
🎯 MVP Week 4: UAT scenarios documented and verified.
- Integration test suite: crates/stemedb-ontology/tests/consumer_health_uat.rs
- 4 automated UAT scenarios with real Ed25519 signing
- API readiness validator: uat/consumer-health/validate_api_readiness.sh
- Test isolation with unique subject prefixes per run
🎯 MVP Week 3: StemeDB integration with conflict demo.
- StemeClient HTTP client for assertion submission and skeptic queries.
- pharma-ingest CLI binary with --with-conflicts for mock data.
- DTO module mirroring stemedb-api types for client-side use.
- Mock trial conflicts for HbA1c, nausea rate, and weight loss.
- Health check, batch ingestion, and instrumented logging.
- 55 unit tests passing, clippy clean.
🎯 MVP Week 2: FDA extractor with claim-to-assertion conversion.
- FdaLabelExtractor fetches from api.fda.gov with exponential backoff retry.
- MedicalClaim::to_assertion() with Ed25519 v2 enterprise signing.
- Shared stemedb_core::signing::compute_content_hash_v2() utility.
- Lifecycle rules: Regulatory sources → Approved, others → Proposed.
- Drug name normalization handles brand names and formulation variants.
- Integration tests with #[ignore] for live FDA API validation.
- 10 unit tests, 5 integration tests.
🎯 MVP Week 1: Domain definition core for pharma vertical.
- Domain struct with entity types, predicate schemas, source hierarchy.
- SubjectBuilder for schema-driven subject construction.
- Pharma domain definition with GLP-1 drug mappings.
- MedicalExtractor trait and ExtractError enum.
Phase 7D Circuit Breakers (The Shield): Per-agent misbehavior isolation.
- State machine: Closed → Open (5 failures) → HalfOpen (30 sec timeout) → Closed (1 success).
- CircuitBreakerStore trait with GenericCircuitBreakerStore<S: KVStore> implementation.
- Failure types: InvalidSignature, InputValidation, PowError, QuotaExceeded, ApplicationError.
- CircuitBreakerLayer Tower middleware (outermost layer, runs first).
- Admin API: GET /v1/admin/circuit-breaker/{agent_id}, POST .../reset, GET .../tripped.
- 503 Service Unavailable with X-Circuit-Breaker-State, Retry-After headers.
- 25 unit tests covering full state machine lifecycle and edge cases.
Phase 7C Content Defense (The Shield): Spam and duplicate detection with quarantine workflow.
- SimilarityIndex trait with MinHash (k=128) + LSH (16 bands × 8 rows) for near-duplicate detection.
- Bloom filter pre-check for O(1) "definitely not duplicate" fast path.
- ContentQualityScorer with Shannon entropy, length checks, structured data detection.
- QuarantineStore with time-ordered keys + O(1) hash index for admin lookups.
- ContentDefenseLayer in stemedb-ingest orchestrating all checks.
- Admin API: GET /v1/admin/quarantine, POST .../approve, POST .../reject.
- Triggers: quality < 0.4, 0.9+ Jaccard similarity, untrusted + confidence > 0.8.
Phase 6C Multi-Node Cluster (The Mesh): Distributed cluster infrastructure.
- SwimMembership with SWIM gossip protocol for node discovery and failure detection.
- RangeRouter with BLAKE3 + jump hash for subject-prefix range sharding.
- Gateway HTTP service with routing, health checks, and read-your-writes.
- 82 integration tests covering membership, sharding, availability, partition tolerance.
Phase 7B EigenTrust (The Shield): Sybil-resistant global trust propagation.
- TrustGraphStore trait with edge CRUD, seed trust management, EigenTrust computation.
- Power iteration: T = (1-α)C^T*T + αP with dangling node handling.
- DomainTrustStore for per-domain expertise tracking (medicine, finance, etc.).
- EigenTrustAuthorityLens: weight = confidence × eigentrust × domain_factor.
- Sybil resistance: isolated rings get near-zero trust (not connected to seeds).
Phase 7A Admission Control (The Shield): PoW-based spam protection for new agents.
- TrustTier enum with 5 tiers, quota multipliers, PoW requirements.
- PowProof struct with BLAKE3 verification, graduated difficulty (16→1→0 bits).
- AdmissionStore trait + AdmissionLayer middleware + /v1/admission/status endpoint.
- Fail-open design for availability, milestone tracking for client UX.
Phase 5D Concept Hierarchy: Hierarchical subjects with cross-scheme alias resolution.
- ConceptPath struct with scheme://segments/leaf format, backward-compatible parsing.
- SourceScheme enum mapping schemes to source tiers (rfc→Regulatory, code→Expert, etc.).
- AliasStore trait with transitive resolution and cycle detection.
- API: POST/DELETE /v1/concepts/alias, GET /v1/concepts/resolve|aliases|suggest|parse.
- Battery 8 (7 tests) + Battery 9 (8 tests).
Phase 5C Index Persistence: Vector hot/cold tiering, visual checkpoint.
Phase 5B WAL Hardening: CRC32C checksums, crash recovery, group commit, log rotation.
Gold Standard Verification (4.7): Sybil defense via proof of knowledge.
- GoldStandard struct with rkyv serialization, GoldStandardStore trait + implementation.
- TrustAdjustment enum: Rewarded(+0.05), Penalized(-0.1), AlreadyVerified.
- Anti-gaming: agents can only verify each gold standard once (dedup markers).
- API: POST/GET/DELETE /v1/admin/gold-standards, POST /v1/admin/verify-agent.
- 21 trust rank tests including 4 new verification tests.
Escalation Triggers (4.6): Active safety system for high-conflict assertions.
- EscalationPolicy with configurable conflict_score thresholds and predicate matching.
- EscalationEvent (content-addressed BLAKE3 ID) written by Materializer when policies fire.
- EscalationStore at ESC:{timestamp_nanos}:{id_hex} with resolve workflow.
- GET /v1/admin/escalations, POST /v1/admin/escalations/{id}/resolve.
Conflict Score Filtering (4.5): Query-time filtering by conflict score.
- min_conflict_score and max_conflict_score on Query + QueryParams.
- Fast-path MV filtering with 0.0-1.0 validation.
- 13 engine tests covering thresholds, boundaries, ranges, combinations.
Vote Provenance Witness (4.4): Transform votes into cryptographic witnesses.
- source_url: Option<String> and observed_context: Option<Vec<u8>> on Vote.
- Input validation: URL max 2048 chars, context max 64KB.
- Backward compatible with existing votes.
"Since" Parameter (4.1): Change tracking for returning consumers.
- since: Option<u64> on Query for incremental sync.
- MV Changelog at MVC:{subject}:{predicate}:{timestamp_nanos} with nanosecond precision.
- ChangeEntry struct with previous_winner_hash, new_winner_hash, lens_name.
- fetch_changes_since() in QueryEngine with timestamp filtering.
- changes_since: Option<Vec<ChangeEntryDto>> on QueryResponse.
- 6 tests covering changelog creation, fetching, and fast path bypass.
Batch TrustRank Decay API (4.3): Admin endpoint for scheduled trust decay.
- POST /v1/admin/decay-trust-ranks with optional now and half_life_seconds.
- Returns decayed_count, timestamp_used, half_life_used, status.
- 3 integration tests.

Recently Completed

Source Metadata Indexing (4.2): Index key source metadata fields for filtered queries.
- New module: crates/stemedb-storage/src/source_metadata_index.rs.
- Indexed fields: journal, doi, platform, study_design.
- Key pattern: SMV:{field}:{value} -> Vec<Hash> (case-insensitive).
- Query filters: ?source_journal=NEJM&source_platform=PubMed.
- AND semantics for multiple metadata filters.
- 18 unit tests covering indexing and query filtering.
Source Document Storage (3F.1): Provenance lookup for 100% citation recall.
- POST /v1/source stores source documents by BLAKE3 content hash.
- GET /v1/provenance/{hash} retrieves source documents.
- Content-addressed storage at SRC:{hash} keys.
- Base64 encoding, 10MB limit, idempotent uploads.
- 5 unit tests covering happy path and error cases.
Epoch Cascade Logic (3D.1): O(1) supersession lookup via pre-computed markers.
- write_supersession_cascade() writes SUPERSEDED: markers for full transitive closure at ingest time.
- is_epoch_superseded() uses O(1) marker lookup instead of chain walking.
- Cycle detection and max depth guard (100 levels).
- 5 tests covering markers, transitive closure, cycles, and marker-based filtering.
Semantic Decay (3B.2): Confidence half-life at query time.
- decay_halflife: Option<u64> and source_class_decay: bool on Query.
- New decay.rs module with apply_decay() and apply_source_class_decay().
- Formula: effective_confidence = confidence * 2^(-(age / halflife)).
- Tier-specific decay: Regulatory=none, Clinical=2yr, Anecdotal=30d.
- 11 unit tests + 1 E2E integration test.
Layered Consensus Lens (3C.2): Per-source-class consensus with tier-by-tier visibility.
- LayeredConsensusLens with LayeredLens trait.
- TierResolution and LayeredResolution types.
- GET /v1/layered?subject=X&predicate=Y endpoint.
- Cross-tier conflict score using Shannon entropy.
- 10 comprehensive tests.
Time-Travel Engine (3B.1): as_of parameter for historical queries.
- as_of: Option<u64> field on Query for querying historical state.
- Bypasses fast path (MVs reflect current state).
- Query::matches() filters by assertion.timestamp <= as_of.
- 6 tests covering edge cases.
Rich Source Metadata (3A.3): Structured provenance beyond source_hash.
- source_metadata: Option<Vec<u8>> field on Assertion.
- Vec<u8> for rkyv zero-copy compatibility, callers handle JSON encoding.
- Builder methods: .source_metadata_json() and .source_metadata().
- API exposes as Option<String> with defensive UTF-8 handling.
- 2 serialization tests.
Conflict Score on Resolution (3A.2): Numeric disagreement metric across all lenses.
- conflict_score: f32 field on Resolution (0.0 = unanimous, 1.0 = max conflict).
- compute_conflict_score() utility using normalized variance.
- Updated all 5 lens implementations to compute and propagate conflict score.
- MaterializedView now stores conflict score.
- API QueryResponse exposes conflict_score and resolution_confidence when lens is applied.
- 7 unit tests including NaN handling.
SkepticLens + SkepticView (3C.1): "Trust but Verify" conflict analysis that surfaces all claims with conflict scores.
- AnalysisLens trait for lenses that map conflict instead of resolving it.
- SkepticLens using normalized Shannon entropy for conflict scoring.
- SkepticResolver + SkepticView in stemedb-query.
- GET /v1/skeptic?subject=X&predicate=Y API endpoint.
- Types: ResolutionStatus, ConflictAnalysis, ClaimSummary, SourceSummary, AgentSummary.
Source-Class Field (3A.1): 6-tier SourceClass enum with authority weighting and decay rates.
- SourceClass enum: Regulatory, Clinical, Observational, Expert, Community, Anecdotal.
- tier(), default_decay_days(), authority_weight() methods.
- Field on Assertion struct with full serialization support.
The Meter: Token bucket quota system with MeterLayer middleware (10K tokens/agent/hour).
Query Audit Trail: Every query logged with provenance at AUD:{query_id}. X-Agent-Id header for attribution.
Event-Driven Materialization: run_notified() + IngestWorker Notify integration.
Fast-Path MV Lookup: QueryEngine::try_fast_path() for O(1) reads.
Materializer: Background worker for O(1) MV reads via AsyncLens.
VoteAwareConsensusLens: Real vote-based consensus resolution.
Compound SP Index: O(1) subject+predicate lookups.
TrustRank System: Agent reputation with decay and learning loop.
API Surface: axum HTTP server with 7 endpoints + OpenAPI docs.

Research Documents

docs/research/wal-crash-recovery-research.md — WAL patterns from CockroachDB, TiKV, FoundationDB, SQLite.
docs/research/distributed-write-path.md — Spanner/CockroachDB-style distributed writes adapted for append-only model.

Key Architectural Decisions

sled → redb/fjall: sled is abandoned. HybridStore routes by key prefix: redb for reads, fjall for writes. ✅ COMPLETE
Raft log = WAL: TiKV eliminated duplicate WAL in v5.4. We should too.
CRDT for data, Raft for coordination: Assertions are a G-Set CRDT (merge = set union). Only cluster metadata needs Raft.
Subject-prefix ranges: Co-locate all data for a subject on one shard. Split hot subjects via range split.
HLC over TrueTime: Hybrid Logical Clocks work on commodity hardware. No GPS/atomic clocks needed.
AP model: Writes never blocked during partitions. Eventual consistency via CRDT convergence.

Blockers

Phase 5: ✅ COMPLETE — All foundation hardening done.
Phase 6: ✅ COMPLETE — CRDT foundation, two-node replication, multi-node cluster.
Phase 7: ✅ COMPLETE — The Shield (Admission control, EigenTrust, Content Defense, Circuit Breakers).
Phase 8: Unblocked. Can proceed with chaos testing, observability, geo-distribution.
Phase 9: Partially blocked. 9A-9B need Phase 8 (can't backup what doesn't exist). 9C-9F can start earlier (compliance planning, security design).

Dependency Graph

Phase 2.5 (Hardening)          Phase 3 (The Pilot)                    Phase 4 (The Hive)
========================       ========================               ==================

[2.1 MV Staleness]  ---------> [3B.1 Time-Travel] ✅ --+
                                       |                |
[2.2 Confidence Rename] -----> (API clarity for all)    +----------> [4.1 "Since" Param] ✅
                                                        |
[2.3 EpochAwareLens] --------> [3D.1 Epoch Cascade] ✅ |----------> Invalidation Cascades pillar
                                                        |
[2.4 Visual Hash Query] -----> [3E.2 Visual Hash Index] ✅
                                                        |
[2.6 E2E Integration] -------> (pipeline confidence)    |
                                                        |
                               [3A.1 Source-Class] ✅ --+----------> [3C.2 Layered Consensus] ✅
                                       |                             [3B.2 Semantic Decay] ✅
                                       +-----------------------------[4.2 Metadata Indexing] ✅
                                                        |
                               [3A.2 Conflict Score] ✅ --> (enhance Resolution)
                                                        |
                               [3A.3 Source Metadata] ✅ --> [4.2 Metadata Indexing] ✅
                                                        |
                               [3C.1 Skeptic Lens] ✅    (standalone, COMPLETE)
                               [3C.3 Constraints Lens] ✅ (standalone, COMPLETE)
                               [3E.1 Vector Search] ✅    (standalone, COMPLETE)
                               [3E.2 Visual Hash Index] ✅ (standalone, COMPLETE)
                               [3F.1 Provenance] ✅       (standalone, COMPLETE)

🎯 Critical Path for Consumer Health MVP (Current Focus)

INFRASTRUCTURE (Complete)                    VERTICAL INTEGRATION (In Progress)
=========================                    ===================================

[3A.1 Source-Class] ✅ ──────────────────┐
[3A.2 Conflict Score] ✅ ────────────────┤
[3C.2 Layered Consensus] ✅ ─────────────┤
[3B.1 Time-Travel] ✅ ───────────────────┼───> [MVP Week 3: Ingest + Conflicts] ✅
[3A.3 Source Metadata] ✅ ───────────────┤              |
[3C.1 Skeptic Lens] ✅ ──────────────────┤              v
[3B.2 Semantic Decay] ✅ ────────────────┘     [MVP Week 4: UAT Scenarios] ✅
                                                        |
[stemedb-ontology Weeks 1-3] ✅ ───────────────────────┘
    (Domain defs, FDA extractor, StemeClient)               |
                                                        v
                                               [MVP Week 5: CLI Tool] ✅
                                                        |
                                                        v
                                               [MVP Week 6: Polish & Docs] [ ]
                                                        |
                                                        v
                                               🎯 CONSUMER HEALTH MVP
                                               "5-minute demo that convinces"

Critical Path for Financial DD Demo

[3A.2 Conflict Score] ✅ --> [3C.1 Skeptic Lens] ✅ ---+
                                                         |
[3B.1 Time-Travel] ✅ ----------------------------------+----> FINANCIAL DD MVP
                                                         |
[2.3 EpochAwareLens] --> [3D.1 Epoch Cascade] ✅ -------+
                                                         |
[3B.2 Semantic Decay] ✅ -------------------------------+

Critical Path for Agile Agent Team Demo

[3C.3 Constraints Lens] (standalone) ------+
                                            |
[3B.1 Time-Travel] ✅ --------------------+----> AGENT TEAM MVP
                                            |
[2.3 EpochAwareLens] ✅ ------------------+
                                            |
[Query Audit (Phase 2)] ✅ ----------------+

Critical Path for Browser Extension

Phase 3 (Data Foundation)          Phase 4 (Extension Primitives)       Extension Product
=========================          ==============================       =================

[3A.1 Source-Class] ✅ ──────────> [4.5 Conflict Filtering] ✅ ──┐
                                                                   |
[3A.2 Conflict Score] ✅ ─────────────────────────────────────────┤
                                                                   |
[3C.1 Skeptic Lens] ✅ ──────────────────────────────────────────-┤
                                                                   ├──> PHASE 1: Read-Only Overlay
[3B.2 Semantic Decay] ✅ ────────────────────────────────────────-┤     (Benign Layer)
                                                                   |
[3C.2 Layered Consensus] ✅ ─────────────────────────────────────-┘

[Phase 2 Vote System] ✅ ────────> [4.4 Vote Provenance] ✅ ─────┐
                                                                   |
[TrustRank Engine] ✅ ───────────> [4.7 Gold Standards] ✅ ───────┤
                                                                   ├──> PHASE 2: Active Layer
[3A.2 Conflict Score] ✅ ────────> [4.6 Escalation Triggers] ✅ ──┤     (Vote to See)
                                                                   |
                                   [7A PoW Admission] ✅ ───────────┤
                                   [7B EigenTrust] ✅ ──────────────┘

Phase 1 (Read-Only) requires: Source tiers, conflict scores, conflict filtering, skeptic lens, decay, layered consensus. All complete.

Phase 2 (Active) requires: Vote provenance, gold standards, escalation triggers, PLUS Phase 7 Sybil defense. ✅ All complete. Ready to build.

Critical Path to Distributed Cluster

Phase 5 (The Forge) ✅              Phase 6 (The Mesh) ✅                 Phase 7+8
=======================             =======================              ==================

[5A.1 Replace sled ✅] ───────────> [6A.1 CRDT Foundation ✅] ──┐
                                            |                    |
[5A.2 Key Layout ✅] ────────────> [6C.2 Range Sharding ✅] ───> |
                                                                 |
[5B.1 CRC32C Checksums ✅] ──┐                                  |
[5B.2 Crash Recovery ✅] ────┼───> [6B.1 RPC Layer ✅] ─────────┤
[5B.3 Group Commit ✅] ──────┘            |                     |
                                          v                     |
[5C.1 Persistent Vector ✅] ─── (independent, no blocker)       |
[5C.2 Persistent Visual ✅] ─── (independent, no blocker)       |
                                                                 |
                                   [6A.2 HLC Timestamps ✅] ────┤
                                   [6A.3 Merkle Tree ✅] ───────┤
                                           |                     |
                                           v                     v
                                   [6B.2 Gossip ✅] ──> [6B.3 Anti-Entropy ✅] ──> [6B.4 PoC Tests ✅]
                                                                 |
                                                                 v
                                   [6C.1 SWIM Membership ✅] ───> [6C.3 Raft MV Coord] (DEFERRED)
                                   [6C.4 Gateway ✅] ───────────> │
                                                                 v
                                                    DISTRIBUTED CLUSTER ✅
                                                                 |
                                                    [7A PoW Admission ✅] ┐
                                                    [7B EigenTrust ✅] ──┤──> THE SHIELD ✅
                                                    [7C Content Defense ✅]┤
                                                    [7D Circuit Breakers ✅]┘
                                                              |
                                                    [8A Chaos Testing] ──┐
                                                    [8B Observability] ──┤──> THE SWARM
                                                    [8C Geo-Distribution]┘
                                                              |
                                                    [9A Backup/PITR] ─────┐
                                                    [9B Corruption/Rollback]┤
                                                    [9C GDPR/Retention] ──┤──> THE BUNKER
                                                    [9D Storage Mgmt] ────┤
                                                    [9E Incident Response]┤
                                                    [9F Security Hardening]┘

New Crates (Phases 5-9)

stemedb-merkle   (Phase 6A)  ── BLAKE3 Merkle tree for diff detection ✅ IMPLEMENTED
stemedb-rpc      (Phase 6B)  ── gRPC services for node-to-node communication ✅ IMPLEMENTED
stemedb-sync     (Phase 6B)  ── Merkle sync, gossip broadcast, anti-entropy ✅ IMPLEMENTED
stemedb-cluster  (Phase 6C)  ── Cluster membership, range routing, gateway ✅ IMPLEMENTED
stemedb-ontology (Parallel)  ── Domain definitions, subject builders, medical extractors ✅ IMPLEMENTED
stemedb-backup   (Phase 9A)  ── Backup coordination, PITR, verification (PLANNED)
stemedb-admin    (Phase 9B)  ── Tombstones, rollback, fork recovery, compliance (PLANNED)

112 KiB Raw Blame History Unescape Escape

Episteme (StemeDB) Roadmap

High-Level Timeline

🎯 Consumer Health MVP (Current Focus)

MVP Definition of Done

The Demo Script

MVP Workstream

What's NOT in MVP

Detailed Milestones

Phase 1: The Spine (Foundation)

Phase 2: The Lattice (Connectivity)

Phase 2.5: Hardening (Camp 2 Fixes)

Phase 3: The Pilot (BioTech/Pharma)

3A. Schema Expansion (Prerequisite for everything below)

3B. Time & Decay (Core Query Features)

3C. New Lenses

3D. Epoch Enhancement

3E. Similarity Search

3F. Provenance

3G. API Cleanup

Phase 4: The Hive (Trust & Scale)

Phase 5: The Forge (Foundation Hardening)

5A. Storage Engine Replacement

5B. WAL Hardening

5C. Index Persistence

5D. Concept Hierarchy ✅ COMPLETE

Phase 6: The Mesh (Distributed Writes)

6A. CRDT Foundation (Single-Node Validation) ✅ COMPLETE

6B. Two-Node Replication (Proof of Concept) ✅ COMPLETE

6C. Multi-Node Cluster ✅

6D. Consistency Guarantees

Phase 7: The Shield (Trust at Scale)

7A. Admission Control ✅

7B. EigenTrust ✅

7C. Content Defense ✅ COMPLETE

7D. Circuit Breakers ✅

Phase 8: The Swarm (Production Cluster)

8A. Chaos Testing ✅

8B. Observability

8C. Production Hardening

Phase 9: The Bunker (Disaster Planning)

9A. Backup & Cold Storage

9B. Data Corruption & Rollback

9C. Compliance & Legal

9D. Storage Management

9E. Incident Response

9F. Security Hardening

Tracking

Active Tasks

Phase 6 Progress

Phase 7 Progress

🎯 Consumer Health MVP Progress (Current Focus)

Next Up

App Layer (External)

Recently Completed

Recently Completed

Research Documents

Key Architectural Decisions

Blockers

Dependency Graph

🎯 Critical Path for Consumer Health MVP (Current Focus)

Critical Path for Financial DD Demo

Critical Path for Agile Agent Team Demo

Critical Path for Browser Extension

Critical Path to Distributed Cluster

New Crates (Phases 5-9)

112 KiB

Raw Blame History