# Episteme (StemeDB) Roadmap > **Goal:** Build the "Git for Truth" substrate for autonomous AI research. > **Current Phase:** Phase 6 (The Mesh — Distributed Writes) — Phase 5 complete ✅ > **Target Vertical:** BioTech/Pharma ("The Living Review") > **Endgame:** Distributed multi-writer cluster for millions of concurrent agents --- ## High-Level Timeline | Phase | Codename | Focus | Key Deliverable | | :--- | :--- | :--- | :--- | | **1** | **The Spine** | Storage & Safety | Append-only WAL + KV Store | | **2** | **The Lattice** | Indexing & Async | Materialized Views + Ballot Box | | **2.5** | **Hardening** | Camp 2 Fixes | MV staleness, epoch behavior, lens cleanup | | **3** | **The Pilot** | Vertical Integration | Pharma Ingestion + Living Review Agent | | **4** | **The Hive** | Trust & Learning | TrustRank, metadata indexing, change tracking | | **5** | **The Forge** | Foundation Hardening | Replace sled, fix WAL, persist indices, concept hierarchy | | **6** | **The Mesh** | Distributed Writes | CRDT replication, Raft coordination, cluster membership | | **7** | **The Shield** | Trust at Scale | EigenTrust, PoW admission, anti-spam, quarantine | | **8** | **The Swarm** | Production Cluster | Chaos testing, observability, geo-distribution | --- ## Detailed Milestones ### Phase 1: The Spine (Foundation) *Goal: Securely ingest assertions and persist them without data loss.* - [x] **Project Scaffold**: Initialize Rust workspace, set up linting/CI (clippy, fmt). - [x] **Assertion Schema**: Define the `Assertion` struct with `rkyv` serialization. - [x] Add dependencies: `rkyv`, `blake3`, `ed25519-dalek`, `image_hasher`. - [x] Define `Assertion` struct (Subject, Predicate, Object, Confidence, SourceHash). - [x] **Multi-Sig Expansion**: Implement `SignatureEntry` struct and `signatures: Vec` field. - [x] **Visual Expansion**: Add `visual_hash: Option` field for image provenance. - [x] Test serialization round-trips. - [x] **Ballot Schema**: Define the `Vote` struct for multi-agent consensus. - [x] Add `Vote` struct: `assertion_hash`, `agent_id`, `weight`, `signature`. - [x] Test serialization round-trips. - [x] **Paradigm Schema (Epochs)**: Define the `Epoch` and `SupersessionType` structs. - [x] Add `epoch: Option` to `Assertion`. - [x] Implement `Epoch` struct with `supersedes` and `SupersessionType`. - [x] Test serialization round-trips. - [x] **WAL Integration**: Implement the Quarantine Pattern for write-ahead logging. - [x] Create `stemedb-wal` crate. - [x] Port `FsyncGuard` and `Record` logic from established durability patterns. - [x] Implement Record format with BLAKE3 checksums and Headers. - [x] Verify `fsync` behavior with tests. - [x] **Storage Engine**: Implement the `Store` trait using `sled` (embedded KV). - [x] Add `sled` dependency. - [x] Define `KVStore` trait (put, get, delete, scan_prefix, flush). - [x] Implement `SledStore` wrapper. - [x] **Basic Ingestor**: Background worker that tails WAL and writes to KV. - [x] Implement async loop reading from WAL. - [x] Write deserialized assertions, votes, and epochs to `sled`. - [x] Ed25519 signature verification during ingestion. - [x] Maintains S: and SP: indexes on ingest. - [x] Persistent cursor/checkpoint (resumes from `__CURSOR__:ingest` in KV store). - [x] **Verification**: Crash recovery tests (write -> crash -> restart -> read). - [x] Single and multi-record crash recovery. - [x] Multiple crash cycles tested. ### Phase 2: The Lattice (Connectivity) *Goal: Query data with sub-millisecond latency using Materialized Views.* - [x] **Lifecycle Schema**: Add `LifecycleStage` to Assertion. - [x] Define enum: `Proposed`, `UnderReview`, `Approved`, `Deprecated`, `Rejected`. - [x] Update `Assertion` struct and serialization tests. - [x] **The Ballot Box**: Implement high-velocity vote ingestion. - [x] `VoteStore` trait and implementation. - [x] `VoteAwareConsensusLens` for real vote-based resolution. - [x] **Index Infrastructure**: Compound indexes for O(1) queries. - [x] `IndexStore` trait with S: and SP: indexes. - [x] `QueryEngine` smart routing (SP -> S -> scan). - [x] **Materializer**: Background worker for O(1) Read Performance. - [x] `MaterializedView` type in `stemedb-core`. - [x] `Materializer` worker in `stemedb-query` with `step()` and `run()`. - [x] Aggregates Votes via `VoteAwareConsensusLens` (or any `AsyncLens`). - [x] Updates `MV:{Subject}:{Predicate}` with the winning Assertion + metadata. - [x] Event-driven mode via `run_notified()` with `tokio::sync::Notify`. - [x] Fast-path MV lookup in `QueryEngine::try_fast_path()`. - [x] **The Meter**: Implement Economic Throttling (TAN). - [x] `QuotaStore` trait and `GenericQuotaStore` implementation. - [x] Token Bucket algorithm with per-agent per-hour quotas. - [x] `MeterLayer` tower middleware for request cost tracking. - [x] Cost model: Assert=10, Vote=1, Query=5+lens, +1/KB payload. - [x] `GET /v1/meter/quota` endpoint to check remaining quota. - [x] `POST /v1/meter/quota/limit` admin endpoint to set custom limits. - [x] **API Surface**: `axum` HTTP server with OpenAPI (utoipa). - [x] `POST /v1/assert` -> Accepts JSON, writes to WAL. - [x] `POST /v1/vote` -> High-throughput vote endpoint. - [x] `POST /v1/epoch` -> Create epoch with optional supersession. - [x] `GET /v1/query` -> Subject/Predicate/Lens/Lifecycle/Epoch filtering. - [x] `GET /v1/health` -> Health check with assertion count. - [x] `GET /swagger-ui` -> Interactive API docs. - [x] 5 lens types available: Recency, Consensus, Authority, VoteAwareConsensus, TrustAwareAuthority. - [x] **Query Audit**: Log every read with provenance. - [x] Define `QueryAudit` struct: query_id, agent_id, timestamp, params, result_hash, contributing_assertions. - [x] Storage at `AUD:{query_id}` with agent index at `AUDA:{agent_id}:{timestamp}:{query_id}`. - [x] `GET /v1/audit/queries` -> Returns history of agent decisions. - [x] `GET /v1/audit/query/{id}` -> Full reasoning trace for a single query. - [x] Auto-logging on every query via `X-Agent-Id` header. ### Phase 2.5: Hardening (Camp 2 Fixes) *Goal: Close the gaps between "built" and "works right." Every item here addresses a feature that exists but doesn't fully deliver on its promise.* - [x] **2.1 MV Staleness Detection**: Make the fast-path aware of stale materialized views. - **Status:** ✅ COMPLETE - **Implementation:** - [x] Added `max_stale: Option` to `Query` struct in `crates/stemedb-query/src/query.rs`. - [x] Added `.max_stale(secs)` builder method to `QueryBuilder`. - [x] In `try_fast_path()`: if `query.max_stale` is set and MV age exceeds threshold, falls through to slow path with `debug!` log. - [x] Added `max_stale` to API `QueryParams` DTO in `crates/stemedb-api/src/dto.rs`. - [x] Wired through query handler in `crates/stemedb-api/src/handlers/query.rs`. - **Tests:** - [x] `test_fast_path_stale_view_falls_back`: MV 1000 seconds old, `max_stale = 60` → slow path used. - [x] `test_fast_path_fresh_view_used`: Fresh MV, `max_stale = 300` → fast path used. - [x] `test_fast_path_no_max_stale_always_uses_mv`: No `max_stale` → any MV age accepted (backward compatible). - [x] `test_fast_path_max_stale_zero_rejects_old_mv`: `max_stale = 0`, MV 1 second old → slow path. - [x] `test_fast_path_max_stale_zero_accepts_brand_new_mv`: `max_stale = 0`, brand new MV → fast path. - [x] **2.2 AuthorityLens -> ConfidenceLens Rename**: Eliminate the misleading name. - **Problem:** `AuthorityLens` selects by `confidence` field, not by agent reputation. `TrustAwareAuthorityLens` is the real authority lens. The name creates confusion about what "Authority" means. - **Solution implemented:** - [x] Renamed `authority.rs` → `confidence.rs`, `AuthorityLens` → `ConfidenceLens` - [x] Added `LensDto::Confidence` for the confidence-field selector - [x] Changed `LensDto::Authority` to route to `TrustAwareAuthorityLens` (the real authority lens) - [x] Updated query handler routing - [x] Updated ai-lookup/services/lens.md and skill documentation - [x] **2.3 EpochAwareLens**: Give epoch supersession runtime behavior. - **Status:** ✅ COMPLETE - **Implementation:** - [x] `EpochAwareLens` in `crates/stemedb-lens/src/epoch_aware.rs` - [x] Decorator pattern wrapping any inner lens (default: RecencyLens) - [x] Walks supersession chain from `E:{epoch_id}` keys - [x] Cycle detection + max depth guard (100) - [x] Fail-open on missing epochs - [x] `LensDto::EpochAware` added to API - [x] 11 tests: excludes_superseded, chain_supersession, no_epochs_passes_all, missing_epoch_includes, cycle_detection, consensus_lens_inner, mixed_epochs, etc. - [x] Documentation updated in `ai-lookup/services/lens.md` - **Known Limitation:** Filtering only occurs when assertions from the superseding epoch are present in candidates. If all candidates are from old epoch (no new epoch assertions), they pass through (fail-open behavior). - [x] **2.4 Visual Hash Query Support**: Make the stored `visual_hash` queryable. - **Status:** ✅ COMPLETE - **Implementation:** - [x] `hamming_distance(a: &PHash, b: &PHash) -> u32` in `crates/stemedb-query/src/query.rs` (lines 26-28) - [x] `visual_near: Option` and `visual_threshold: Option` in `Query` struct (lines 84-90) - [x] `.visual_near(hash, threshold)` builder method - [x] `Query::matches()` computes hamming distance when `visual_near` is set - [x] API `QueryParams` DTO has `visual_near` and `visual_threshold` - [x] 10+ tests: exact_match, within_threshold, exceeds_threshold, skips_without_hash, invalid_hex, wrong_length, combines_with_subject, default_threshold, max_threshold, threshold_63_rejects - **Note:** Brute-force O(N) scan. VP-tree/BK-tree index is Phase 3+. - [x] **2.5 Vector Field**: No changes needed. Already roadmapped for Phase 3. - **Status:** ✅ N/A (No Phase 2 work required) - **Current state:** `vector: Option>` on `Assertion`. Stored and returned by API. No index, no search. - **Phase 3 plan:** Integrate `hnsw-rs` or `lance` for k-NN search. - [x] **2.6 E2E Integration Test (Write -> Materialize -> Read)**: Prove the full pipeline works end-to-end. - **Status:** ✅ COMPLETE - **Implementation:** - [x] `crates/stemedb-query/tests/e2e_pipeline.rs` with 5 comprehensive tests: - `test_e2e_write_materialize_read` - Basic happy path - `test_e2e_vote_consensus` - Vote-weighted resolution - `test_e2e_update_winner` - Winner changes on re-materialize - `test_e2e_cursor_persistence` - Cursor survives worker restart - `test_e2e_notify_integration` - Event-driven notification channel - [x] `stemedb-wal` and `stemedb-ingest` added as dev-dependencies - [x] Helper functions: `create_signed_assertion()`, `compute_assertion_hash()`, `create_vote()` - [x] Uses Ed25519 signing for authentic signature verification - [x] Also: `crates/stemedb-api/tests/e2e_flow_test.rs` tests the HTTP API layer end-to-end. ### Phase 3: The Pilot (BioTech/Pharma) *Goal: Prove value in the "High-Liability" beachhead. Close every Camp 4 gap that blocks a credible demo.* #### 3A. Schema Expansion (Prerequisite for everything below) - [x] **3A.1 Source-Class Field**: Add `source_class: SourceClass` to Assertion. - **Status:** ✅ COMPLETE - **Implementation:** - [x] `SourceClass` enum in `crates/stemedb-core/src/types.rs` (lines 68-88). - [x] 6-tier system: `Regulatory` (0), `Clinical` (1), `Observational` (2), `Expert` (3), `Community` (4), `Anecdotal` (5). - [x] `tier()` method returns tier number for ordering. - [x] `default_decay_days()` method for tier-specific confidence decay. - [x] `authority_weight()` method for conflict resolution weighting. - [x] Field on `Assertion` struct at line 152. - [x] Full serialization and indexing support. - [x] **3A.2 Conflict Score on Resolution**: Add `conflict_score: f32` to Resolution. - **Status:** ✅ COMPLETE - **Implementation:** - [x] Added `conflict_score: f32` field to `Resolution` in `crates/stemedb-lens/src/traits.rs`. - [x] Updated `Resolution::empty()` to set `conflict_score: 0.0`. - [x] Updated `Resolution::with_winner()` to accept `conflict_score` parameter. - [x] Added `compute_conflict_score(candidates: &[Assertion]) -> f32` utility function: - Uses normalized variance of confidence values. - 0 or 1 candidates → 0.0 (no conflict possible). - All same confidence → 0.0 (unanimous). - Max variance (0.0 vs 1.0) → 1.0 (maximum conflict). - Defensive NaN handling (returns 0.0 for malformed data). - [x] Updated all lens implementations to compute and pass conflict score: - `crates/stemedb-lens/src/recency.rs` - `crates/stemedb-lens/src/consensus.rs` - `crates/stemedb-lens/src/confidence.rs` - `crates/stemedb-lens/src/vote_aware_consensus.rs` - `crates/stemedb-lens/src/trust_aware_authority.rs` - [x] Added `conflict_score: f32` to `MaterializedView` in `crates/stemedb-core/src/types.rs`. - [x] Updated `Materializer::materialize_pair()` to write `conflict_score` from resolution. - [x] Added `conflict_score: Option` and `resolution_confidence: Option` to `QueryResponse` DTO in `crates/stemedb-api/src/dto.rs` (only present when lens is applied). - [x] Wired through query handler in `crates/stemedb-api/src/handlers/query.rs`. - **Tests:** - [x] `test_conflict_score_zero_for_empty`: Empty candidates → 0.0. - [x] `test_conflict_score_zero_for_single`: 1 candidate → 0.0. - [x] `test_conflict_score_zero_for_agreement`: All same confidence → near 0.0. - [x] `test_conflict_score_high_for_disagreement`: Candidates at 0.1, 0.5, 0.9 → score > 0.3. - [x] `test_conflict_score_max_for_extremes`: 0.0 vs 1.0 → score ≈ 1.0. - [x] `test_conflict_score_handles_nan_defensively`: NaN confidences → 0.0 (fail-safe). - [x] **3A.3 Rich Source Metadata**: Add structured provenance beyond `source_hash`. - **Status:** ✅ COMPLETE - **Implementation:** - [x] Added `source_metadata: Option>` field to `Assertion` in `crates/stemedb-core/src/types.rs` (after `epoch`, before `lifecycle`). - [x] Uses `Vec` (not `String`) for rkyv zero-copy compatibility. Callers encode/decode JSON on their side. - [x] Added `source_metadata: Option>` to `AssertionBuilder` in `crates/stemedb-core/src/testing.rs`. - [x] Added `.source_metadata_json(json: &str)` and `.source_metadata(bytes)` builder methods. - [x] Added `source_metadata: Option` to `CreateAssertionRequest` DTO (JSON string in API, converted to bytes internally). - [x] Added `source_metadata: Option` to `AssertionResponse` DTO (bytes converted to JSON string with defensive UTF-8 handling). - [x] Wired through create handler (`dto_to_assertion()`) and query handler (`assertion_to_dto()`). - **Tests:** - [x] `test_serialize_deserialize_assertion_with_metadata`: Serialization roundtrip with metadata present. - [x] `test_serialize_deserialize_assertion_without_metadata`: Serialization roundtrip with metadata absent. - **Note:** Metadata is stored but NOT indexed in Phase 3. Indexing individual metadata fields is Phase 4+. #### 3B. Time & Decay (Core Query Features) - [x] **3B.1 Time-Travel Engine**: `as_of` parameter for historical queries. - **Status:** ✅ COMPLETE - **Implementation:** - [x] Added `as_of: Option` to `Query` struct in `crates/stemedb-query/src/query.rs:92-99`. - [x] Added `.as_of(timestamp: u64)` to `QueryBuilder`. - [x] In `Query::matches()`: if `as_of` is `Some(ts)`, check `assertion.timestamp <= ts`. Assertions created after `as_of` are excluded. - [x] In `QueryEngine::execute()`: if `query.as_of` is set, **skip the fast path entirely** (MVs reflect current state, not historical). - [x] Added `as_of: Option` to `QueryParams` DTO in `crates/stemedb-api/src/dto.rs`. - [x] Wired through query handler. - **Tests:** - [x] `test_as_of_excludes_future_assertions`: Assertions filtered by timestamp. - [x] `test_as_of_bypasses_fast_path`: MV exists, but `as_of` is set. Slow path used. - [x] `test_as_of_none_uses_fast_path`: Normal query still uses fast path (backwards-compatible). - [x] `test_as_of_with_lens_resolves_among_historical_candidates`: Time-travel + lens = resolve only among pre-as_of candidates. - [x] `test_as_of_returns_empty_when_all_assertions_are_future`: All assertions are future, returns empty. - [x] `test_as_of_with_exact_timestamp_match`: Edge case where assertion.timestamp == as_of. - [x] **3B.2 Semantic Decay**: Confidence Half-Life at query time. - **Status:** ✅ COMPLETE - **Implementation:** - [x] Added `decay_halflife: Option` to `Query` struct in `crates/stemedb-query/src/query.rs`. - [x] Added `.decay_halflife(seconds: u64)` and `.source_class_decay(enabled: bool)` to `QueryBuilder`. - [x] Added `decay_halflife` and `source_class_decay` to `QueryParams` DTO. - [x] Created new `crates/stemedb-query/src/decay.rs` module with: - `apply_decay()`: Uniform decay using formula `confidence * 2^(-(age / halflife))`. - `apply_source_class_decay()`: Tier-specific decay (Regulatory=none, Clinical=2yr, Anecdotal=30d). - `compute_decayed_confidence()`: Core decay calculation with clamping. - [x] Integrated in `QueryEngine::execute()`: decay applied after filtering, before lens resolution. - [x] Time-travel compatible: uses `as_of` timestamp if set, otherwise current time. - [x] Source-class-aware decay fully implemented using `SourceClass::default_decay_days()`. - **Tests:** (11 unit tests in decay.rs + 1 E2E test) - [x] `test_decay_reduces_old_assertion_confidence`: 1yr old, 1yr halflife → ~50% confidence. - [x] `test_decay_preserves_fresh_assertions`: 1hr old, 1yr halflife → ~100% confidence. - [x] `test_decay_interacts_with_lens`: Older high-confidence loses to newer low-confidence after decay. - [x] `test_source_aware_decay_tier0_no_decay`: Regulatory never decays. - [x] `test_source_aware_decay_tier5_rapid_decay`: Anecdotal decays rapidly (30-day halflife). - [x] `test_source_aware_decay_mixed_tiers`: Clinical vs Anecdotal tier comparison. - [x] `test_decay_zero_halflife_no_change`: Zero halflife skips decay (avoids div-by-zero). - [x] `test_decay_future_assertion_no_change`: Future assertions don't decay. - [x] `test_decay_empty_assertions`: Empty input returns empty output. - [x] `test_decay_confidence_clamps_to_valid_range`: Very old assertions clamp to [0.0, 1.0]. - [x] `test_decay_preserves_other_fields`: Only confidence changes; other fields preserved. - [x] `test_e2e_decay_reduces_old_confidence`: Full pipeline E2E test in e2e_pipeline.rs. - **Note:** When decay is enabled, materialized views (fast path) are bypassed because MVs store pre-computed winners without decay applied. #### 3C. New Lenses - [x] **3C.1 Skeptic Lens**: Surface disagreement, not winners. ✅ **COMPLETED** - **Status:** ✅ COMPLETE - **Implementation:** - [x] `crates/stemedb-lens/src/skeptic.rs` - Full implementation. - [x] `AnalysisLens` trait for lenses that analyze conflict instead of resolving it. - [x] `SkepticLens` uses normalized Shannon entropy for conflict scoring. - [x] Returns `ConflictAnalysis` with: - `conflict_score: f32` (0.0 = unanimous, 1.0 = chaos) - `status: ResolutionStatus` (Unanimous, Agreed, Contested) - `claims: Vec` - all claims ranked by weight - [x] `SkepticResolver` + `SkepticView` in `stemedb-query/src/skeptic.rs`. - [x] `GET /v1/skeptic?subject=X&predicate=Y` API endpoint. - [x] Core types in `stemedb-core/src/types.rs`: - `ResolutionStatus` enum - `ConflictAnalysis` struct - `ClaimSummary`, `SourceSummary`, `AgentSummary` - [x] Comprehensive test coverage (21 test cases). - [x] **3C.2 Layered Consensus Lens**: Per-source-class consensus. - **Status:** ✅ COMPLETE - **Implementation:** - [x] `crates/stemedb-lens/src/layered_consensus.rs` - Full implementation. - [x] `TierResolution` struct: per-tier result with tier, source_class, winner, candidates_count, conflict_score, resolution_confidence. - [x] `LayeredResolution` struct: multi-tier result with tiers vec, overall_winner, overall_conflict_score, total_candidates. - [x] `LayeredLens` trait: `resolve_layered(&[Assertion]) -> LayeredResolution`, `name() -> &'static str`. - [x] `LayeredConsensusLens` implements both `LayeredLens` and `Lens` traits. - [x] Cross-tier conflict score uses normalized Shannon entropy of tier winner object values. - [x] `LensDto::LayeredConsensus` variant (redirects to `/v1/layered` endpoint). - [x] `GET /v1/layered?subject=X&predicate=Y` API endpoint with `LayeredQueryResponse`. - [x] Exported from `crates/stemedb-lens/src/lib.rs`. - **Tests:** - [x] `test_layered_empty_candidates`: Empty input returns empty resolution. - [x] `test_layered_single_tier`: All same source_class, returns one tier result. - [x] `test_layered_multi_tier_agreement`: Tier 0 and Tier 5 agree, low cross-tier conflict. - [x] `test_layered_multi_tier_disagreement`: Tier 1 vs Tier 5 disagree, high conflict, Tier 1 wins. - [x] `test_layered_overall_winner_from_highest_authority`: Tier 0 wins despite fewer assertions. - [x] `test_layered_lens_trait_compatibility`: Standard Lens trait works. - [x] `test_layered_within_tier_conflict`: High internal conflict within a tier. - [x] `test_layered_all_tiers_present`: One assertion from each tier. - [x] `test_layered_lens_name`: Both trait names work. - [x] `test_layered_numeric_values`: Works with numeric object values. - [x] **3C.3 Constraints Lens**: Pre-flight check for must_use/forbidden. - **Status:** ✅ COMPLETE - **Implementation:** - [x] `crates/stemedb-lens/src/constraints.rs` - Full implementation. - [x] `ConstraintSet` struct: holds categorized assertions (must_use, forbidden, prefer) with conflict_score. - [x] `ConstraintsLens` struct with `resolve_constraints(&[Assertion]) -> ConstraintSet` method. - [x] Categorizes by predicate pattern: `must_use:*`, `forbidden:*`, `prefer:*`. - [x] Implements `Lens` trait for compatibility (priority: must_use > forbidden > prefer). - [x] Sorted by confidence (highest first), with timestamp as tiebreaker. - [x] `LensDto::Constraints` added (redirects to `/v1/constraints` endpoint). - [x] `GET /v1/constraints?subject=X` API endpoint with `ConstraintsResponse`. - [x] DTOs: `ConstraintsQueryParams`, `ConstraintEntryDto`, `ConstraintsResponse`. - [x] Exported from `crates/stemedb-lens/src/lib.rs`. - **Tests:** (16 test cases) - [x] `test_constraints_categorizes_by_predicate`: Mixed predicates sorted into must_use/forbidden/prefer. - [x] `test_constraints_empty_categories`: Only prefer, no must_use/forbidden. - [x] `test_constraints_non_constraint_predicates_ignored`: Regular predicates filtered out. - [x] `test_constraints_sorted_by_confidence`: Within-category confidence ordering. - [x] `test_constraints_empty_candidates`: Empty input returns empty set. - [x] `test_constraints_has_constraints_true`: Helper method works. - [x] `test_constraints_all_regular_predicates`: All non-constraint predicates returns no constraints. - [x] `test_lens_trait_picks_must_use_winner`: Standard Lens trait picks must_use first. - [x] `test_lens_trait_falls_back_to_forbidden`: Falls back to forbidden when no must_use. - [x] `test_lens_trait_falls_back_to_prefer`: Falls back to prefer when no must_use/forbidden. - [x] `test_lens_trait_empty_for_no_constraints`: Returns empty when no constraint predicates. - [x] `test_lens_name`: Name returns "Constraints". - [x] `test_lens_empty_candidates`: Empty input to Lens trait returns empty resolution. - [x] `test_multiple_must_use_picks_highest_confidence`: Multiple must_use picks highest confidence. - [x] `test_confidence_tiebreaker_uses_timestamp`: Same confidence uses newer timestamp. - [x] `test_predicate_pattern_exact_prefix`: `must_use_something` not matched (only `must_use:*`). #### 3D. Epoch Enhancement - [x] **3D.1 Epoch Cascade Logic** (enhancement of Phase 2.5 EpochAwareLens): - **Status:** ✅ COMPLETE - **Implementation:** - [x] `write_supersession_cascade()` in `crates/stemedb-ingest/src/worker.rs`: - Writes `SUPERSEDED:{old_epoch_id}` markers for full transitive closure. - All markers point to the LATEST superseding epoch. - Max depth guard (100 levels) and cycle detection via visited set. - [x] `is_epoch_superseded()` in `crates/stemedb-lens/src/epoch_aware.rs`: - O(1) marker lookup instead of O(chain_length) chain walks. - Fail-open semantics: missing marker = not superseded. - [x] `compute_superseded_epochs()` uses marker lookups for filtering. - **Tests:** - [x] `test_cascade_writes_superseded_marker`: Epoch B supersedes A → `SUPERSEDED:A` exists. - [x] `test_cascade_transitive`: C→B→A chain → both `SUPERSEDED:A` and `SUPERSEDED:B` point to C. - [x] `test_cascade_cycle_detection`: Mutual supersession handled gracefully. - [x] `test_epoch_aware_uses_marker`: EpochAwareLens uses O(1) marker lookup. - [x] `test_superseded_epoch_filtered_even_without_new_assertions`: Marker-based filtering works. #### 3E. Similarity Search - [x] **3E.1 Vector Search**: Semantic k-NN queries via embeddings. - **Status:** ✅ COMPLETE - **Implementation:** - [x] Added `hnsw_rs = "0.3"` and `parking_lot = "0.12"` to `stemedb-storage/Cargo.toml`. - [x] New module: `crates/stemedb-storage/src/vector_index.rs`. - [x] `VectorIndex` trait: `insert(hash: &Hash, vector: &[f32])`, `search(query: &[f32], k: usize) -> Vec<(Hash, f32)>`, `dimension()`, `len()`, `is_empty()`. - [x] `HnswVectorIndex` implementation with HNSW graph, RwLock protection, hash↔ID mappings. - [x] Input validation: dimension mismatch, NaN, Infinite values rejected. - [x] Idempotent insert (same hash twice = no-op). - [x] `Arc` trait object support for sharing. - [x] `IngestWorker::with_vector_index()` builder method for index attachment. - [x] IngestWorker: if assertion has `vector`, inserts into vector index after KV write. - [x] Added `vector_near: Option>` and `k: Option` to `Query` struct in `crates/stemedb-query/src/query.rs`. - [x] Added `.vector_near(vector, k)` builder method to `QueryBuilder`. - [x] Added `vector_near` and `k` to API `QueryParams` DTO. - [x] `QueryEngine::with_vector_index()` builder method for index attachment. - [x] QueryEngine: if `vector_near` is set and index configured, uses O(log N) HNSW lookup for candidates. - [x] Falls back to standard path if no index configured (with debug log). - **Tests:** (12 unit tests for VectorIndex + 4 integration tests in engine.rs) - [x] `test_create_index`, `test_insert_and_search`, `test_dimension_mismatch`. - [x] `test_idempotent_insert`, `test_search_empty_index`, `test_search_k_zero`. - [x] `test_nan_rejection`, `test_infinite_rejection`, `test_contains`. - [x] `test_larger_scale` (100 vectors, exact match first), `test_custom_params`, `test_zero_dimension_panics`. - [x] `test_vector_search_returns_nearest_neighbors`, `test_vector_search_with_subject_filter`. - [x] `test_vector_search_without_index_falls_back`, `test_vector_search_with_as_of_filter`. - **Note:** Index is in-memory only. Persistence is Phase 4+. - [x] **3E.2 Visual Hash Index**: BK-tree for O(log N) visual similarity. - **Status:** ✅ COMPLETE - **Implementation:** - [x] New module: `crates/stemedb-storage/src/visual_index.rs`. - [x] `VisualIndex` trait: `insert(hash: &Hash, phash: &PHash)`, `search(query: &PHash, threshold: u32) -> Vec<(Hash, u32)>`, `len()`, `is_empty()`. - [x] `BkTreeVisualIndex` implementation using BK-tree over hamming distance. - [x] `hamming_distance(a: &PHash, b: &PHash) -> u32` utility function. - [x] Threshold clamped to 0-64 range (max 64 bits). - [x] Results sorted by distance ascending. - [x] Idempotent insert (same hash twice = no-op). - [x] `Arc` trait object support for sharing. - [x] `IngestWorker::with_visual_index()` builder method for index attachment. - [x] IngestWorker: if assertion has `visual_hash`, inserts into BK-tree after KV write. - [x] `QueryEngine::with_visual_index()` builder method for index attachment. - [x] QueryEngine: if `visual_near` is set and index configured, uses O(log N) BK-tree lookup. - [x] Falls back to brute-force scan (via `query.matches()`) if no index configured. - [x] Invalid hex input returns `QueryError::InvalidInput` with clear message. - **Tests:** (14 unit tests for VisualIndex + 6 integration tests in engine.rs) - [x] `test_hamming_distance_zero`, `test_hamming_distance_max`, `test_hamming_distance_partial`. - [x] `test_create_index`, `test_insert_and_search_exact`, `test_search_within_threshold`. - [x] `test_search_no_matches`, `test_search_empty_index`, `test_idempotent_insert`. - [x] `test_contains`, `test_results_sorted_by_distance`, `test_threshold_clamped_to_64`. - [x] `test_larger_scale` (1000 hashes), `test_default_impl`. - [x] `test_visual_search_returns_similar_images`, `test_visual_search_with_lifecycle_filter`. - [x] `test_visual_search_invalid_hex_returns_error`, `test_visual_search_without_index_uses_brute_force`. - [x] `test_visual_search_with_limit`, `test_vector_search_empty_index`. - **Note:** Index is in-memory only. Persistence is Phase 4+. #### 3F. Provenance - [x] **3F.1 Source Document Storage & Provenance Lookup**: Enable 100% citation recall. - **Status:** ✅ COMPLETE - **Implementation:** - [x] `POST /v1/source` endpoint to store source documents by BLAKE3 content hash. - [x] `GET /v1/provenance/{hash}` endpoint to retrieve source documents by hash. - [x] Source storage at `SRC:{hash}` keys with format: `[content_type_len:4][content_type][content]`. - [x] Base64 encoding for binary-safe JSON transport. - [x] 10MB size limit per document. - [x] Content-addressed storage: same content → same hash (idempotent uploads). - [x] DTOs: `StoreSourceRequest`, `StoreSourceResponse`, `ProvenanceResponse`. - [x] OpenAPI documentation under "provenance" tag. - **Tests:** (5 test cases) - [x] `test_store_and_retrieve_source`: Happy path store + retrieve. - [x] `test_store_source_invalid_base64`: Bad base64 → 400. - [x] `test_get_provenance_not_found`: Unknown hash → 404. - [x] `test_get_provenance_invalid_hash`: Bad hash format → 400. - [x] `test_store_source_idempotent`: Same content twice → same hash. - **Note:** Benchmark utility for verifying all assertions have retrievable sources is future work. #### 3G. API Cleanup - [x] **3G.1 Document epoch supersession via existing endpoint**: No new `/epoch/supersede` endpoint needed. - **Status:** ✅ COMPLETE - **Implementation:** - [x] Updated use case docs (consumer-health-intelligence.md, glp1-living-review.md) to use `POST /v1/epoch` with `supersedes` field. - [x] Added OpenAPI examples showing both new epoch and supersession flows in handlers/epoch.rs. - [x] Documented all 5 supersession types: Invalidate, Temporal, Refinement, RequiresReview, Additive. - **No code change.** Documentation fix only. ### Phase 4: The Hive (Trust & Scale) *Goal: Change tracking, metadata indexing, and the database primitives for training pipelines.* - [x] **TrustRank Engine**: Foundation for trust-based resolution. - [x] `TrustRankStore` for per-agent reputation storage. - [x] `TrustAwareAuthorityLens` for reputation-weighted resolution. - [x] **Confidence Half-Life**: Implement decay calculation engine. - [x] Learning loop: `record_outcome()` for accuracy tracking. - [x] **4.1 "Since" Parameter**: Change tracking for returning consumers. - **Status:** ✅ COMPLETE - **Problem:** Consumer Health shows `GET /query?since=2023-10-01` returning `changes_since` with dated change entries. The "returning consumer" story: "What changed since I last looked?" - **Depends on:** Time-Travel (3B.1) and Materializer. - **Implementation:** - [x] Added `since: Option` to `Query` struct and `QueryBuilder` in `crates/stemedb-query/src/query.rs`. - [x] Added `since` to `QueryParams` DTO in `crates/stemedb-api/src/dto.rs`. - [x] **MV Changelog**: Track when materialized views change. - [x] New key pattern: `MVC:{subject}:{predicate}:{timestamp_nanos}` using nanosecond precision to prevent collisions. - [x] `ChangeEntry` struct in `crates/stemedb-core/src/types.rs` with: `timestamp`, `previous_winner_hash`, `new_winner_hash`, `subject`, `predicate`, `lens_name`. - [x] In `Materializer::materialize_pair()`: before overwriting MV, read existing MV. If winner changed (different hash), write changelog entry. - [x] `get_previous_winner_hash()` and `write_changelog_entry()` helper methods. - [x] In QueryEngine: if `since` is set, skip fast path (MVs reflect current state). `fetch_changes_since()` scans `MVC:` keys and returns entries >= since timestamp. - [x] `ChangeEntryDto` in `crates/stemedb-api/src/dto.rs` with hex-encoded hashes. - [x] `changes_since: Option>` added to `QueryResponse` DTO. - [x] `fetch_changelog_if_needed()` helper in query handler. - [x] Re-exported `ChangeEntry` from `stemedb-core` crate root. - **Tests:** - [x] `test_changelog_written_on_first_materialization`: First MV creates changelog with `previous_winner_hash: None`. - [x] `test_changelog_written_on_winner_change`: Winner change creates changelog with previous hash. - [x] `test_no_changelog_when_winner_unchanged`: Re-materialize same winner = no new changelog. - [x] `test_fetch_changes_since_returns_entries`: Fetch filtered by timestamp, sorted ascending. - [x] `test_fetch_changes_since_empty_on_no_entries`: No entries returns empty vec. - [x] `test_since_bypasses_fast_path`: Query with `since` uses slow path, returns all assertions. - [x] **4.2 Source Metadata Indexing** (extension of 3A.3): Index key metadata fields. - **Status:** ✅ COMPLETE - **Problem:** Phase 3 stores `source_metadata` as an opaque blob. Phase 4 makes key fields queryable. - **Depends on:** Rich Source Metadata (3A.3). - **Implementation:** - [x] Defined indexed metadata keys: `journal`, `doi`, `platform`, `study_design` in `INDEXED_METADATA_FIELDS`. - [x] New module: `crates/stemedb-storage/src/source_metadata_index.rs`. - [x] Key pattern: `SMV:{field}:{value}` storing `Vec` (rkyv-serialized) for efficient batch lookup. - [x] `SourceMetadataIndexStore` trait with `add_to_metadata_indexes()` and `get_by_metadata_field()`. - [x] `GenericSourceMetadataIndexStore` implementation with case-insensitive value normalization. - [x] IngestWorker: on ingestion, if `source_metadata` is present, parse JSON, extract indexed fields, write index entries. - [x] Added `source_journal`, `source_doi`, `source_platform`, `source_study_design` to `Query` struct. - [x] Added `.source_journal()`, `.source_doi()`, `.source_platform()`, `.source_study_design()` builder methods. - [x] Added `matches_metadata_filters()` helper with AND semantics and case-insensitive matching. - [x] Added metadata field filters to `QueryParams` DTO (e.g., `?source_journal=NEJM`). - [x] Wired through query handler in `crates/stemedb-api/src/handlers/query.rs`. - **Tests:** (9 unit tests in source_metadata_index.rs + 9 unit tests in query.rs) - [x] `test_add_and_get_by_metadata_field`: Basic round-trip. - [x] `test_case_insensitive_indexing`: "NEJM" stored, "nejm" query matches. - [x] `test_multiple_assertions_same_field`: Vec accumulates correctly. - [x] `test_malformed_json_gracefully_skipped`: Invalid JSON logs warning, doesn't error. - [x] `test_missing_field_not_indexed`: Only indexed fields processed. - [x] `test_idempotent_insert`: Same assertion twice = no duplicates. - [x] `test_all_indexed_fields`: All 4 fields indexed correctly. - [x] `test_empty_index_returns_empty_vec`: Non-existent field returns empty. - [x] `test_non_string_values_ignored`: Non-string JSON values skipped. - [x] **4.3 Batch TrustRank Decay API**: Expose scheduled decay for external orchestration. - **Status:** ✅ COMPLETE - **Implementation:** - [x] New module: `crates/stemedb-api/src/handlers/admin.rs` - [x] `POST /v1/admin/decay-trust-ranks` endpoint. - [x] Request accepts `now: Option` (defaults to current time) and `half_life_seconds: Option` (defaults to 30 days). - [x] Response includes `decayed_count`, `timestamp_used`, `half_life_used`, `status`. - [x] OpenAPI docs with "admin" tag. - [x] Thin handler delegates to `TrustRankStore::decay_trust_ranks()`. - **Tests:** (3 integration tests in http_integration.rs) - [x] `test_decay_trust_ranks_empty_store`: Empty store returns 0 decayed. - [x] `test_decay_trust_ranks_with_custom_params`: Custom timestamp and half-life honored. - [x] `test_decay_trust_ranks_response_structure`: All 4 fields present in response. - **Note:** The Gardener (Camp 5.2, app layer) calls this endpoint on a schedule. The database just exposes the primitive. - [x] **4.4 Vote Provenance Witness**: Transform votes from opinions into cryptographic witnesses. - **Status:** ✅ COMPLETE - **Problem:** Votes record "I agree" but the browser extension product needs "I saw this exact text at this URL at this time." Without provenance, votes are Tier 5 noise. - **Implementation:** - [x] Added `source_url: Option` to Vote struct — URL where claim was observed. - [x] Added `observed_context: Option>` to Vote struct — page context bytes (rkyv zero-copy pattern). - [x] API DTOs: `source_url` and `observed_context` on CreateVoteRequest/VoteResponse. - [x] Input validation: source_url max 2048 chars (non-empty if provided), observed_context max 64KB. - [x] Backward compatible: existing votes without provenance remain valid. - **Tests:** Serialization roundtrip (with/without provenance, mixed fields), storage roundtrip, API integration (validation). - [x] **4.5 Conflict Score Filtering**: Query-time filtering by conflict score. - **Status:** ✅ COMPLETE - **Problem:** The browser extension needs "only show me claims where conflict > 0.7" to implement contradiction-only overlays. Conflict score was computed but not filterable. - **Implementation:** - [x] Added `min_conflict_score: Option` and `max_conflict_score: Option` to Query struct. - [x] Builder methods: `.min_conflict_score()`, `.max_conflict_score()`. - [x] Fast-path filtering: checks MV conflict_score against thresholds. - [x] API validation: scores must be 0.0-1.0, finite (rejects NaN/Inf). - [x] QueryParams DTO wired through query handler. - **Tests:** 13 engine tests (min/max thresholds, exact boundary, range, combination with lifecycle), API validation tests. - **Note:** Filtering works on fast path (MV reads) only. Slow path returns raw assertions without conflict scores. - [x] **4.6 Escalation Triggers**: Active safety system for high-conflict assertions. - **Status:** ✅ COMPLETE - **Problem:** High conflict was invisible. Episteme should be an active safety system that fires escalations when disagreement exceeds thresholds. - **Implementation:** - [x] `EscalationLevel` enum: Low, Medium, High, Critical. - [x] `EscalationEvent` struct: content-addressed (BLAKE3 ID), subject, predicate, conflict_score, level, reason, timestamp, resolved. - [x] `EscalationPolicy` struct: configurable threshold + level + optional predicate pattern. - [x] `EscalationStore` trait + `GenericEscalationStore`: write, query since, resolve, get pending. - [x] Key pattern: `ESC:{timestamp_nanos}:{id_hex}`. - [x] Materializer integration: after computing conflict_score, checks policies and writes events. - [x] `with_escalation(store, policies)` builder on Materializer. - [x] `GET /v1/admin/escalations` and `POST /v1/admin/escalations/{id}/resolve` endpoints. - **Tests:** Core type tests, storage roundtrip, materializer integration (high/low conflict, predicate matching, no store configured). - [x] **4.7 Gold Standard Verification**: Sybil defense via proof of knowledge. - **Status:** ✅ COMPLETE - **Problem:** New agents need a way to earn TrustRank. Gold standards are admin-verified assertions that agents can be tested against. - **Implementation:** - [x] `GoldStandard` struct with rkyv serialization: assertion_hash, subject, predicate, expected_object, created_at, created_by. - [x] `GoldStandardStore` trait + `GenericGoldStandardStore`: set, get, list, remove. - [x] Key pattern: `GS:{subject}:{predicate}`. - [x] `TrustAdjustment` enum: Rewarded(+0.05), Penalized(-0.1), AlreadyVerified. - [x] `verify_agent_against_gold_standard()` on TrustRankStore with deduplication (agents can only verify each gold standard once). - [x] Verification markers at `GS_VERIFIED:{agent_id}:{subject}:{predicate}` prevent gaming. - [x] API: `POST/GET/DELETE /v1/admin/gold-standards`, `POST /v1/admin/verify-agent`. - **Tests:** Core (creation, matching, case-sensitivity), storage CRUD, trust adjustment (reward, penalty, clamping, dedup), API integration (5 tests). > **Note:** The following items were reclassified as **Application Layer** responsibilities (see `tmp/ambition-vs-reality.md`, Camp 5). They are not Episteme database features. They consume the Episteme API and are built by integrators or vertical-specific teams. > > - **The Simulator** (Training Data Pipeline) -> Camp 5.3 > - **The Super Curator** (Reviewer Agent swarm) -> Camp 5.4 > - **Background Gardener** (Cluster detection, signal processing) -> Camp 5.2 > - **Agent Wallet** (Key management sidecar) -> Camp 5.1 ### Phase 5: The Forge (Foundation Hardening) *Goal: Replace abandoned dependencies, fix WAL gaps, persist indices. Prerequisite for distribution.* > **Research:** [docs/research/wal-crash-recovery-research.md](docs/research/wal-crash-recovery-research.md) #### 5A. Storage Engine Replacement - [x] **5A.1 Replace sled with redb + fjall**: sled is abandoned (author recommends alternatives). - **Problem:** sled is alpha-stage with known performance regressions and no active development. Our entire storage layer depends on it. - **Solution:** HybridStore routes keys by prefix — **fjall** (LSM) for write-heavy paths (`H:`, `V:`, `VC:`, `VW:`, `E:`, `SUPERSEDED:`, `__CURSOR__:`) and **redb** (B-tree) for read-heavy paths (`S:`, `SP:`, `MV:`, `TR:`, `QA:`, `QT:`, `TP:`, `GS:`, `ESC:`). - **Tasks:** - [x] Generalize `StorageError::Sled` to `StorageError::Backend(String)`. - [x] Implement `FjallStore` backend with DashMap per-key locks for atomics. - [x] Implement `RedbStore` backend with ACID transactions. - [x] Implement `HybridStore` routing layer with prefix-based dispatch. - [x] Migrate all ~500 tests from `SledStore` to `HybridStore`. - [x] Remove sled dependency entirely. - [x] Add criterion benchmarks (sequential put, random get, prefix scan, atomic increment, mixed workload). - **Crates:** `redb = "2"`, `fjall = "2"`, `dashmap = "6"` - [x] **5A.2 Key Layout Redesign**: Prepare keys for subject-prefix range sharding. - **Problem:** Current keys (`H:{hash}`, `S:{subject}`, `MV:{subject}:{predicate}`) scatter related data across the keyspace. Distributed sharding needs co-location. - **Solution:** Subject-prefix key layout with `\x00` separator for subject-scoped keys, `\x00` prefix for global keys (sort-first): ``` Subject-prefixed (co-located): {subject}\x00H:{hash} → Assertion data {subject}\x00S:{hash_list} → Subject index {subject}\x00SP:{predicate} → Compound index {subject}\x00MV:{predicate} → Materialized view {subject}\x00V:{hash}:{vh} → Votes {subject}\x00VC:{hash} → Vote count cache {subject}\x00VW:{hash} → Vote weight cache {subject}\x00GS:{predicate} → Gold standards Global (sort first via \x00 prefix): \x00TRUST:{agent_id} → Trust ranks \x00QUOTA:{agent_id}:{win} → Quota records \x00QLIMIT:{agent_id} → Quota limits \x00E:{epoch_id} → Epochs \x00SUPERSEDED:{epoch_id} → Supersession markers \x00SUP:{hash} → Supersession records \x00AUD:{query_id} → Audit records \x00ESC:{ts}:{id} → Escalation events \x00TP:{pack_id} → Trust packs \x00META:{key} → System metadata \x00HASH_SUBJECT:{hash} → Reverse lookup index \x00SUBJECTS:{subject} → Known subjects index \x00GS_LIST:{subj}:{pred} → Gold standard listing ``` - **Implementation:** - [x] `key_codec.rs` (573 lines): 40+ key builder functions, subject validation, extraction utilities, 30+ unit tests. - [x] All stores migrated to `key_codec::` functions (91 call sites across 10 store files, zero hardcoded key patterns). - [x] Ingestion pipeline uses `key_codec` (11 usages across 3 files). - [x] Query engine uses `key_codec` (34 usages across 7 files). - [x] Subject co-location verified by `test_subject_colocation` test. - [x] Global key sort-first verified by `test_global_keys_sort_first` test. #### 5B. WAL Hardening - [x] **5B.1 CRC32C Checksums**: Add hardware-accelerated torn write detection. - **Status:** ✅ COMPLETE - **Implementation:** - [x] Added `crc32c = "0.6"` crate dependency. - [x] Updated WAL record format with CRC32C in `format.rs`. - [x] CRC32C verified on read before deserializing. - [x] Hardware-accelerated via SSE 4.2 on supported CPUs. - [x] **5B.2 Crash Recovery Implementation**: Replace recovery stub with production recovery. - **Status:** ✅ COMPLETE - **Implementation:** - [x] `recovery/mod.rs` (236 lines): Full sequential record scan with CRC32C validation. - [x] Truncates file at first corrupted/incomplete record. - [x] `RecoveryMetrics` struct tracks: valid_records, invalid_records, bytes_truncated, recovery_duration. - [x] Comprehensive test suite in `recovery/tests.rs`. - [x] **5B.3 Group Commit**: Batch fsync for throughput. - **Status:** ✅ COMPLETE - **Implementation:** - [x] `group_commit.rs` (342 lines): `GroupCommitBuffer` with configurable `max_writes` and `max_duration`. - [x] Writers append to buffer and wait on `Notify`. - [x] Background flusher calls fsync and notifies all waiters. - [x] `GroupCommitConfig` for tuning batch size and flush interval. - [x] **5B.4 Log Rotation**: Bounded WAL disk usage. - **Status:** ✅ COMPLETE - **Implementation:** - [x] `segment.rs` (368 lines): Full segment management. - [x] Segment naming: `{seq:08x}.wal`. - [x] Rotation when segment exceeds configurable threshold. - [x] `SegmentManager` tracks active and archived segments. - [x] Safe deletion of segments after cursor passes them. #### 5C. Index Persistence - [x] **5C.1 Persistent Vector Index**: Move HNSW from in-memory to disk-backed. - **Status:** ✅ COMPLETE - **Implementation:** - [x] `PersistentVectorIndex` in `crates/stemedb-storage/src/vector_index/persistent.rs`. - [x] Hybrid hot/cold architecture: - **Hot:** In-memory HNSW for recent vectors. - **Cold:** Disk-backed HNSW loaded from checkpoint files. - [x] `persistence/format.rs`: `SnapshotMetadata`, `IdMappingTable` with rkyv serialization. - [x] `hot_cold.rs`: `merge_search_results()` for combining hot and cold query results. - [x] Background checkpoint task with atomic write pattern. - [x] CRC32C integrity verification on load. - [x] MAX_PAYLOAD_SIZE (1GB) validation to prevent memory exhaustion. - **Crates:** `memmap2 = "0.9"`, `crc32c = "0.6"`, `byteorder = "1.5"` - **Known Limitation:** Cold index currently stores ID mappings only; full vector persistence with mmap'd HNSW graphs planned for future phase. - [x] **5C.2 Persistent Visual Index**: Persist BK-tree to disk. - **Status:** ✅ COMPLETE - **Implementation:** - [x] `PersistentVisualIndex` in `crates/stemedb-storage/src/visual_index.rs`. - [x] `BkTreeSnapshot` with rkyv serialization for BK-tree state. - [x] Checkpoint file format: `[MAGIC:4][VERSION:1][RESERVED:3][PAYLOAD_LEN:u64][CRC32C:u32][PAYLOAD]`. - [x] Atomic write pattern: temp file → fsync → rename → fsync parent. - [x] Background checkpoint task with configurable interval. - [x] CRC32C integrity verification on load. - [x] Shared `checkpoint_format.rs` module for common read/write utilities. #### 5D. Concept Hierarchy ✅ COMPLETE > **Spec:** [docs/specs/concept-hierarchy.md](docs/specs/concept-hierarchy.md) > **Purpose:** Hierarchical, scheme-qualified subject identifiers with cross-scheme alias resolution. Enables applications like Aphoria that need to connect `code://` paths to `rfc://` paths. - [x] **5D.1 ConceptPath Type**: Structured subject identifiers. ✅ - **Tasks:** - [x] Add `ConceptPath` struct to `stemedb-core/src/types/concept.rs`. - [x] Wire format: `{scheme}://{segment_0}/{segment_1}/.../{leaf}`. - [x] `parse()`, `to_wire_format()`, `leaf()`, `parent()`, `is_prefix_of()`. - [x] Backward-compatible: bare strings parse as `custom://{string}`. - [x] Unit tests for parsing, round-trip, prefix matching (Battery 8). - **Crate:** `stemedb-core` - [x] **5D.2 Source Scheme Registry**: Map schemes to default source tiers. ✅ - **Tasks:** - [x] Add `SourceScheme` enum to `stemedb-core`. - [x] Scheme → default `SourceClass` mapping (e.g., `rfc://` → Tier 0, `code://` → Tier 3). - [x] `ConceptPath::default_source_class()` method. - **Crate:** `stemedb-core` - [x] **5D.3 Alias Store**: Cross-scheme entity resolution. ✅ - **Tasks:** - [x] Add `ConceptAlias` struct to `stemedb-core`. - [x] Add `AliasStore` trait to `stemedb-storage`. - [x] Key prefixes: `CA:{alias_path}` → canonical, `CAR:{canonical}` → all aliases. - [x] Transitive alias resolution with cycle detection. - [x] `GenericAliasStore` implementation over `KVStore`. - **Crates:** `stemedb-core`, `stemedb-storage` - [x] **5D.4 Hierarchical Query**: Prefix-based subject queries. ✅ - **Tasks:** - [x] `fetch_by_subject_prefix()` using `scan_prefix` in query engine (already implemented in Battery 5). - [x] Trailing `/` handling to prevent `auth` matching `authentication`. - **Crate:** `stemedb-query` - **Note:** Hierarchical prefix scanning was already working; Battery 5 validates it. - [x] **5D.5 Alias Resolution in Queries**: Expand queries to aliased paths. ✅ - **Tasks:** - [x] `AliasStore::resolve_all()` for transitive alias expansion. - [x] API endpoint `GET /v1/concepts/resolve?path=...&transitive=true`. - **Crate:** `stemedb-query`, `stemedb-api` - **Note:** Resolution available via API; QueryEngine integration is future work. - [x] **5D.6 Source Class Inference**: Infer tier from scheme. ✅ - **Tasks:** - [x] `ConceptPath::default_source_class()` returns tier based on scheme. - [x] `SourceScheme::parse()` maps scheme strings to enum variants. - **Crate:** `stemedb-core` - **Note:** Inference at ingestion time would break content-addressing (signature verification). Inference is available at query time or before signing. - [x] **5D.7 Concept API Endpoints**: CRUD for aliases and hierarchy browsing. ✅ - **Tasks:** - [x] `POST /v1/concepts/alias` — Create alias. - [x] `GET /v1/concepts/aliases` — List all aliases (with optional canonical filter). - [x] `DELETE /v1/concepts/alias` — Remove alias. - [x] `GET /v1/concepts/resolve` — Resolve path to canonical/transitive aliases. - [x] `GET /v1/concepts/suggest` — Suggested aliases (shared leaf detection). - [x] `GET /v1/concepts/parse` — Parse path and return ConceptPath info. - **Crate:** `stemedb-api` - [x] **5D.8 Battery Tests**: Validate concept hierarchy end-to-end. ✅ - **Tests:** - [x] Battery 8 (7 tests): ConceptPath parsing, round-trip, prefix matching, source class inference. - [x] Battery 9 (8 tests): Alias resolution, transitive resolution, cycle detection, bidirectional lookup, delete, suggestions. - **Crate:** `stemedb-query/tests/battery_pre_sentinel.rs` ### Phase 6: The Mesh (Distributed Writes) *Goal: Multi-node cluster with CRDT replication and Raft coordination. The endgame.* > **Research:** [docs/research/distributed-write-path.md](docs/research/distributed-write-path.md) > **Agent:** `distributed-systems-engineer` > **Key Insight:** Episteme's append-only model eliminates ~75% of CockroachDB complexity. Assertions are a G-Set CRDT. Votes are G-Counters. No distributed transactions needed. #### 6A. CRDT Foundation (Single-Node Validation) - [ ] **6A.1 Integrate CRDT Crate**: Wrap assertion storage in G-Set semantics. - **Tasks:** - [ ] Add `crdts = "7.4"` dependency to `stemedb-storage`. - [ ] Implement `CrdtAssertionStore` wrapping assertions as `GSet`. - [ ] Implement `CrdtVoteStore` wrapping votes as `GCounter<(Hash, [u8; 32])>`. - [ ] Property tests: commutativity (`merge(A,B) == merge(B,A)`), associativity, idempotence. - [ ] Verify existing tests still pass with CRDT wrapper. - [ ] **6A.2 Hybrid Logical Clocks**: Add causal ordering to supersessions. - **Tasks:** - [ ] Add `uhlc = "0.7"` dependency to `stemedb-core`. - [ ] Replace `timestamp: u64` in `Supersession` with `hlc_timestamp: uhlc::Timestamp`. - [ ] Update `IngestWorker` to generate HLC timestamps. - [ ] Update `EpochAwareLens` to use HLC comparison for ordering. - [ ] Test: concurrent supersessions from different nodes converge to same order. - [ ] **6A.3 Merkle Tree Over Assertions**: Efficient diff detection. - **Tasks:** - [ ] Implement `MerkleTree` over assertion hashes using BLAKE3. - [ ] Incremental update: insert new hash, recompute affected path. - [ ] Root comparison: O(1) check if two nodes have same assertions. - [ ] Recursive diff: O(log N) to find divergent subtrees. - [ ] Serialize tree state for exchange over network. #### 6B. Two-Node Replication (Proof of Concept) - [ ] **6B.1 RPC Layer**: Node-to-node communication. - **Tasks:** - [ ] Create `stemedb-rpc` crate. - [ ] Define protobuf messages: `SyncRequest`, `SyncResponse`, `FetchAssertions`, `GossipBroadcast`. - [ ] Implement gRPC services with `tonic`. - [ ] Connection pooling and retry with exponential backoff. - **Crates:** `tonic = "0.12"`, `prost = "0.13"` - [ ] **6B.2 Gossip Broadcast**: Push new assertions to peers. - **Tasks:** - [ ] On write: gossip new assertion hash + data to N peers (fanout = 3-5). - [ ] Peers merge into local G-Set. - [ ] Deduplicate: content-addressed hashes mean receiving same assertion twice is a no-op. - [ ] Track gossip metrics: `gossip_messages_sent`, `gossip_duplicates_received`. - [ ] **6B.3 Merkle Anti-Entropy Sync**: Background convergence. - **Tasks:** - [ ] Every 60 seconds per peer: exchange Merkle roots. - [ ] If roots differ: recursive diff to find missing hashes. - [ ] Fetch missing assertions from peer. - [ ] Merge into local store + trigger MV recompute. - [ ] Track: `sync_lag_seconds`, `merkle_diff_size`, `convergence_latency_p99`. - [ ] **6B.4 Integration Test: Two-Node Convergence**: - [ ] Write assertion to Node A → appears on Node B within 5 seconds. - [ ] Write to Node A during partition → Node B converges after healing. - [ ] Concurrent writes to both nodes → both converge to same state. #### 6C. Multi-Node Cluster - [ ] **6C.1 Cluster Membership (SWIM Gossip)**: Node discovery and failure detection. - **Tasks:** - [ ] Add `memberlist = "0.4"` dependency. - [ ] Implement `ClusterMembership` with SWIM protocol. - [ ] Seed-node based discovery (bootstrap nodes in config). - [ ] Failure detection: ping, indirect probe, suspicion. - [ ] Membership change events trigger anti-entropy with new peers. - **Crate:** `memberlist = "0.4"` - [ ] **6C.2 Subject-Prefix Range Sharding**: Distribute data across nodes. - **Tasks:** - [ ] Implement `RangeRouter`: map subject → range → node. - [ ] Range descriptor: start key, end key, replica nodes. - [ ] Automatic range split when size exceeds 64MB threshold. - [ ] Range merge when adjacent ranges shrink below 20MB. - [ ] Meta-range: store range descriptors, gossip to all nodes. - [ ] **6C.3 Raft for MV Coordination (Optional)**: Deterministic MV computation. - **Problem:** Without ordering, different nodes may compute different MV winners during convergence. - **Solution:** Lightweight Raft group per subject-range for MV coordinator election. - **Tasks:** - [ ] Add `openraft = "0.10"` dependency. - [ ] Implement `RaftLogStorage` backed by fjall. - [ ] Implement `RaftStateMachine` delegating to `Materializer`. - [ ] Leader coordinates MV recomputation order. - [ ] Followers serve reads from local MVs. - **Note:** This is optional. Without Raft, MVs are eventually consistent (converge once assertions sync). With Raft, MVs are strongly consistent per range. - **Crate:** `openraft = "0.10"` - [ ] **6C.4 Gateway**: Stateless request routing. - **Tasks:** - [ ] Implement `Gateway` HTTP service (axum). - [ ] Route writes by subject → range → node. - [ ] Route reads to nearest replica. - [ ] Health checking and failover. - [ ] Load balancing across replicas. #### 6D. Consistency Guarantees | Property | Guarantee | Mechanism | |----------|-----------|-----------| | **Convergence** | Eventually consistent | G-Set merge (CRDT) | | **Causality** | Supersessions ordered | HLC timestamps | | **Partition Tolerance** | Writes never blocked | Any node accepts via CRDT | | **Availability** | Reads/writes always succeed | Every node is master for CRDTs | | **Durability** | WAL + fsync per node | Existing WAL infra | | **Conflict Resolution** | Deterministic | Lens algorithms (same inputs → same output) | ### Phase 7: The Shield (Trust at Scale) *Goal: Defend against spam, Sybil attacks, and knowledge poisoning when open to millions of agents.* > **Key Insight:** Open agent access requires layered defense. PoW for admission, EigenTrust for reputation, circuit breakers for misbehavior, quarantine for suspicious content. #### 7A. Admission Control - [ ] **7A.1 Proof-of-Work Admission**: BLAKE3 hashcash for new agents. - New agents (trust < 0.3) must solve PoW puzzle before first N assertions accepted. - Graduated difficulty: first 10 assertions = 16 sec, 11-50 = 1 sec, 50+ or trust > 0.6 = none. - Puzzle: `BLAKE3(nonce || agent_id || timestamp)` must have `difficulty` leading zero bits. - [ ] **7A.2 Graduated Trust Tiers**: Privilege escalation based on reputation. - Untrusted (0.0-0.3): Quarantine mode, assertions hidden by default, 0.1x quota. - Limited (0.3-0.5): Low quota, PoW required, 0.5x quota. - Verified (0.5-0.7): Standard quota, normal privileges. - Trusted (0.7-0.9): 2x quota, skip PoW. - Authority (0.9-1.0): 10x quota, no limits. #### 7B. EigenTrust - [ ] **7B.1 Trust Graph Store**: Store direct trust relationships for propagation. - Key pattern: `TG:{agent_a}:{agent_b}` → trust weight (0.0-1.0). - Methods: `add_trust_edge()`, `get_trusted_by()`, `compute_eigentrust()`. - [ ] **7B.2 EigenTrust Computation**: Global trust via power iteration (daily batch). - Formula: `T = (1-α)CT + αP` where C = normalized trust matrix, P = seed trust, α = 0.1. - Convergence: 10-20 iterations for 10K agents. - Sybil resistance: isolated rings have low trust unless connected to real agents. - Crates: `petgraph` (graph structures), `nalgebra` (linear algebra). - [ ] **7B.3 Domain-Specific Trust**: Per-predicate-namespace reputation. - Agent may be expert in medicine but novice in astronomy. - Track accuracy by predicate namespace. - Lens can weight by domain trust. #### 7C. Content Defense - [ ] **7C.1 MinHash Deduplication**: Near-duplicate detection with LSH bucketing. - Compute MinHash signature for `{subject}:{predicate}` pairs. - LSH buckets for O(1) average-case lookup. - Bloom filter pre-check for fast "definitely not duplicate" path. - Threshold: 0.9 Jaccard similarity = duplicate. - [ ] **7C.2 Content Quality Scoring**: Heuristic-based spam detection. - Shannon entropy check (high entropy = likely random noise). - Minimum subject/predicate length. - Structured data bonus (JSON objects, numbers, URLs). - Untrusted agent + high confidence = suspicious. - [ ] **7C.3 Quarantine Store**: Suspicious assertions held for review. - Key pattern: `QUAR:{hash}` → assertion data. - Quarantined assertions NOT indexed (invisible to queries). - Triggers: quality < 0.4, duplicate, untrusted high-confidence. - Admin API: `GET /v1/admin/quarantine`, `POST /v1/admin/quarantine/{hash}/approve`. #### 7D. Circuit Breakers - [ ] **7D.1 Per-Agent Circuit Breakers**: Ban misbehaving agents temporarily. - States: Closed (normal), Open (banned), HalfOpen (testing recovery). - 5 failures → Open for 30 seconds → HalfOpen → 1 success → Closed. - Metrics: `circuit_breaker_state`, `circuit_breaker_trips_total`. - Crate: `governor` for GCRA rate limiting. ### Phase 8: The Swarm (Production Cluster) *Goal: Production-ready distributed deployment with chaos testing and observability.* #### 8A. Chaos Testing - [ ] **8A.1 Partition Testing**: Verify convergence after network partitions. - 5-node cluster, kill 2 nodes, verify remaining nodes converge. - Network partition between groups, verify both sides accept writes, verify convergence after healing. - Message reordering and duplication tests. - [ ] **8A.2 Jepsen-Style Consistency Testing**: Formal verification. - Linearizability checks for Raft-coordinated operations. - Eventual consistency checks for CRDT-replicated data. - Clock skew injection (verify HLC handles drift). #### 8B. Observability - [ ] **8B.1 Distributed Metrics**: Per-node, per-range, per-agent metrics. - `sync_lag_seconds{peer}`, `merkle_diff_size{peer}`, `convergence_latency_p99`. - `raft_leader_changes{range}`, `raft_commit_latency{range}`. - `assertions_total{node}`, `writes_per_second{node}`. - Crate: `metrics` + `metrics-exporter-prometheus`. - [ ] **8B.2 Admin Dashboard**: Cluster health visibility. - `GET /v1/admin/cluster` → node list, range assignments, leader locations. - `GET /v1/admin/ranges` → range sizes, split/merge history. - `POST /v1/admin/sync` → force anti-entropy sync. #### 8C. Production Hardening - [ ] **8C.1 Snapshot/Restore**: Fast replica bootstrap. - Serialize full node state as snapshot. - New nodes join by restoring snapshot + replaying recent WAL. - Faster than full Merkle sync for new nodes. - [ ] **8C.2 Backpressure**: Don't overwhelm slow nodes. - Track per-peer sync queue depth. - Throttle gossip to slow peers. - Alert when peer is consistently behind. - [ ] **8C.3 Geo-Distribution**: Multi-region deployment. - Regional clusters with CRDT federation between regions. - Lazy cross-region sync (background, low priority). - Locality-aware reads (query nearest replica). - Regional compliance (GDPR data residency). --- ## Tracking ### Active Tasks * [x] **Phase 3 The Pilot**: Consumer Health vertical integration. ✅ COMPLETE * [x] **Phase 4 The Hive**: Trust & Scale + Extension Primitives. ✅ COMPLETE * [x] **Phase 5 The Forge**: Foundation hardening — replace sled, fix WAL, persist indices. ✅ COMPLETE * [x] **5A**: Replace sled with redb/fjall (HybridStore), key layout redesign. ✅ COMPLETE * [x] **5B**: WAL hardening — CRC32C, crash recovery, group commit, log rotation. ✅ COMPLETE * [x] **5C**: Index persistence — vector hot/cold, visual checkpoint. ✅ COMPLETE * [x] **5D**: Concept hierarchy — ConceptPath, AliasStore, scheme-based inference. ✅ COMPLETE ### Next Up * **Phase 6**: Distributed writes via CRDT replication + Raft coordination. * **Phase 7A-7B** (Extension blocker): PoW admission + EigenTrust for Phase 2 extension launch. ### App Layer (External) * **Browser Extension Phase 1** (Read-Only Overlay) -> All DB dependencies complete. Extension is app layer. * **Browser Extension Phase 2** (Active Layer / Vote-to-See) -> Blocked by Phase 7A-7B (Sybil defense). * **The Simulator** (Training Data Pipeline) -> App layer, consumes Episteme API. * **The Super Curator** (Reviewer Agent swarm) -> App layer. * **Background Gardener** (Cluster detection, signal processing) -> App layer. * **Agent Wallet** (Key management sidecar) -> App layer. ### Recently Completed * [x] **Phase 5D Concept Hierarchy**: Hierarchical subjects with cross-scheme alias resolution. * `ConceptPath` struct with scheme://segments/leaf format, backward-compatible parsing. * `SourceScheme` enum mapping schemes to source tiers (rfc→Regulatory, code→Expert, etc.). * `AliasStore` trait with transitive resolution and cycle detection. * API: `POST/DELETE /v1/concepts/alias`, `GET /v1/concepts/resolve|aliases|suggest|parse`. * Battery 8 (7 tests) + Battery 9 (8 tests). * [x] **Phase 5C Index Persistence**: Vector hot/cold tiering, visual checkpoint. * [x] **Phase 5B WAL Hardening**: CRC32C checksums, crash recovery, group commit, log rotation. * [x] **Gold Standard Verification** (4.7): Sybil defense via proof of knowledge. * `GoldStandard` struct with rkyv serialization, `GoldStandardStore` trait + implementation. * `TrustAdjustment` enum: Rewarded(+0.05), Penalized(-0.1), AlreadyVerified. * Anti-gaming: agents can only verify each gold standard once (dedup markers). * API: `POST/GET/DELETE /v1/admin/gold-standards`, `POST /v1/admin/verify-agent`. * 21 trust rank tests including 4 new verification tests. * [x] **Escalation Triggers** (4.6): Active safety system for high-conflict assertions. * `EscalationPolicy` with configurable conflict_score thresholds and predicate matching. * `EscalationEvent` (content-addressed BLAKE3 ID) written by Materializer when policies fire. * `EscalationStore` at `ESC:{timestamp_nanos}:{id_hex}` with resolve workflow. * `GET /v1/admin/escalations`, `POST /v1/admin/escalations/{id}/resolve`. * [x] **Conflict Score Filtering** (4.5): Query-time filtering by conflict score. * `min_conflict_score` and `max_conflict_score` on Query + QueryParams. * Fast-path MV filtering with 0.0-1.0 validation. * 13 engine tests covering thresholds, boundaries, ranges, combinations. * [x] **Vote Provenance Witness** (4.4): Transform votes into cryptographic witnesses. * `source_url: Option` and `observed_context: Option>` on Vote. * Input validation: URL max 2048 chars, context max 64KB. * Backward compatible with existing votes. * [x] **"Since" Parameter** (4.1): Change tracking for returning consumers. * `since: Option` on Query for incremental sync. * MV Changelog at `MVC:{subject}:{predicate}:{timestamp_nanos}` with nanosecond precision. * `ChangeEntry` struct with `previous_winner_hash`, `new_winner_hash`, `lens_name`. * `fetch_changes_since()` in QueryEngine with timestamp filtering. * `changes_since: Option>` on QueryResponse. * 6 tests covering changelog creation, fetching, and fast path bypass. * [x] **Batch TrustRank Decay API** (4.3): Admin endpoint for scheduled trust decay. * `POST /v1/admin/decay-trust-ranks` with optional `now` and `half_life_seconds`. * Returns `decayed_count`, `timestamp_used`, `half_life_used`, `status`. * 3 integration tests. ### Recently Completed * [x] **Source Metadata Indexing** (4.2): Index key source metadata fields for filtered queries. * New module: `crates/stemedb-storage/src/source_metadata_index.rs`. * Indexed fields: `journal`, `doi`, `platform`, `study_design`. * Key pattern: `SMV:{field}:{value}` -> `Vec` (case-insensitive). * Query filters: `?source_journal=NEJM&source_platform=PubMed`. * AND semantics for multiple metadata filters. * 18 unit tests covering indexing and query filtering. * [x] **Source Document Storage** (3F.1): Provenance lookup for 100% citation recall. * `POST /v1/source` stores source documents by BLAKE3 content hash. * `GET /v1/provenance/{hash}` retrieves source documents. * Content-addressed storage at `SRC:{hash}` keys. * Base64 encoding, 10MB limit, idempotent uploads. * 5 unit tests covering happy path and error cases. * [x] **Epoch Cascade Logic** (3D.1): O(1) supersession lookup via pre-computed markers. * `write_supersession_cascade()` writes `SUPERSEDED:` markers for full transitive closure at ingest time. * `is_epoch_superseded()` uses O(1) marker lookup instead of chain walking. * Cycle detection and max depth guard (100 levels). * 5 tests covering markers, transitive closure, cycles, and marker-based filtering. * [x] **Semantic Decay** (3B.2): Confidence half-life at query time. * `decay_halflife: Option` and `source_class_decay: bool` on Query. * New `decay.rs` module with `apply_decay()` and `apply_source_class_decay()`. * Formula: `effective_confidence = confidence * 2^(-(age / halflife))`. * Tier-specific decay: Regulatory=none, Clinical=2yr, Anecdotal=30d. * 11 unit tests + 1 E2E integration test. * [x] **Layered Consensus Lens** (3C.2): Per-source-class consensus with tier-by-tier visibility. * `LayeredConsensusLens` with `LayeredLens` trait. * `TierResolution` and `LayeredResolution` types. * `GET /v1/layered?subject=X&predicate=Y` endpoint. * Cross-tier conflict score using Shannon entropy. * 10 comprehensive tests. * [x] **Time-Travel Engine** (3B.1): `as_of` parameter for historical queries. * `as_of: Option` field on `Query` for querying historical state. * Bypasses fast path (MVs reflect current state). * `Query::matches()` filters by `assertion.timestamp <= as_of`. * 6 tests covering edge cases. * [x] **Rich Source Metadata** (3A.3): Structured provenance beyond `source_hash`. * `source_metadata: Option>` field on `Assertion`. * `Vec` for rkyv zero-copy compatibility, callers handle JSON encoding. * Builder methods: `.source_metadata_json()` and `.source_metadata()`. * API exposes as `Option` with defensive UTF-8 handling. * 2 serialization tests. * [x] **Conflict Score on Resolution** (3A.2): Numeric disagreement metric across all lenses. * `conflict_score: f32` field on `Resolution` (0.0 = unanimous, 1.0 = max conflict). * `compute_conflict_score()` utility using normalized variance. * Updated all 5 lens implementations to compute and propagate conflict score. * `MaterializedView` now stores conflict score. * API `QueryResponse` exposes `conflict_score` and `resolution_confidence` when lens is applied. * 7 unit tests including NaN handling. * [x] **SkepticLens + SkepticView** (3C.1): "Trust but Verify" conflict analysis that surfaces all claims with conflict scores. * `AnalysisLens` trait for lenses that map conflict instead of resolving it. * `SkepticLens` using normalized Shannon entropy for conflict scoring. * `SkepticResolver` + `SkepticView` in stemedb-query. * `GET /v1/skeptic?subject=X&predicate=Y` API endpoint. * Types: `ResolutionStatus`, `ConflictAnalysis`, `ClaimSummary`, `SourceSummary`, `AgentSummary`. * [x] **Source-Class Field** (3A.1): 6-tier `SourceClass` enum with authority weighting and decay rates. * `SourceClass` enum: Regulatory, Clinical, Observational, Expert, Community, Anecdotal. * `tier()`, `default_decay_days()`, `authority_weight()` methods. * Field on Assertion struct with full serialization support. * [x] **The Meter**: Token bucket quota system with MeterLayer middleware (10K tokens/agent/hour). * [x] **Query Audit Trail**: Every query logged with provenance at `AUD:{query_id}`. `X-Agent-Id` header for attribution. * [x] **Event-Driven Materialization**: `run_notified()` + IngestWorker Notify integration. * [x] **Fast-Path MV Lookup**: `QueryEngine::try_fast_path()` for O(1) reads. * [x] **Materializer**: Background worker for O(1) MV reads via `AsyncLens`. * [x] **VoteAwareConsensusLens**: Real vote-based consensus resolution. * [x] **Compound SP Index**: O(1) subject+predicate lookups. * [x] **TrustRank System**: Agent reputation with decay and learning loop. * [x] **API Surface**: axum HTTP server with 7 endpoints + OpenAPI docs. ### Research Documents * [docs/research/wal-crash-recovery-research.md](docs/research/wal-crash-recovery-research.md) — WAL patterns from CockroachDB, TiKV, FoundationDB, SQLite. * [docs/research/distributed-write-path.md](docs/research/distributed-write-path.md) — Spanner/CockroachDB-style distributed writes adapted for append-only model. ### Key Architectural Decisions * **sled → redb/fjall**: sled is abandoned. HybridStore routes by key prefix: redb for reads, fjall for writes. ✅ COMPLETE * **Raft log = WAL**: TiKV eliminated duplicate WAL in v5.4. We should too. * **CRDT for data, Raft for coordination**: Assertions are a G-Set CRDT (merge = set union). Only cluster metadata needs Raft. * **Subject-prefix ranges**: Co-locate all data for a subject on one shard. Split hot subjects via range split. * **HLC over TrueTime**: Hybrid Logical Clocks work on commodity hardware. No GPS/atomic clocks needed. * **AP model**: Writes never blocked during partitions. Eventual consistency via CRDT convergence. ### Blockers * **Phase 5**: ✅ COMPLETE — All foundation hardening done. * **Phase 6**: Unblocked. Can start distributed writes. * **Phase 7**: Blocked by Phase 6 (trust at scale requires distributed infra). * **Phase 8**: Blocked by Phase 6 + 7 (chaos testing requires working cluster). --- ## Dependency Graph ``` Phase 2.5 (Hardening) Phase 3 (The Pilot) Phase 4 (The Hive) ======================== ======================== ================== [2.1 MV Staleness] ---------> [3B.1 Time-Travel] ✅ --+ | | [2.2 Confidence Rename] -----> (API clarity for all) +----------> [4.1 "Since" Param] ✅ | [2.3 EpochAwareLens] --------> [3D.1 Epoch Cascade] ✅ |----------> Invalidation Cascades pillar | [2.4 Visual Hash Query] -----> [3E.2 Visual Hash Index] ✅ | [2.6 E2E Integration] -------> (pipeline confidence) | | [3A.1 Source-Class] ✅ --+----------> [3C.2 Layered Consensus] ✅ | [3B.2 Semantic Decay] ✅ +-----------------------------[4.2 Metadata Indexing] ✅ | [3A.2 Conflict Score] ✅ --> (enhance Resolution) | [3A.3 Source Metadata] ✅ --> [4.2 Metadata Indexing] ✅ | [3C.1 Skeptic Lens] ✅ (standalone, COMPLETE) [3C.3 Constraints Lens] ✅ (standalone, COMPLETE) [3E.1 Vector Search] ✅ (standalone, COMPLETE) [3E.2 Visual Hash Index] ✅ (standalone, COMPLETE) [3F.1 Provenance] ✅ (standalone, COMPLETE) ``` ### Critical Path for Consumer Health Demo ``` [3A.1 Source-Class] ✅ --> [3A.2 Conflict Score] ✅ --> [3C.2 Layered Consensus] ✅ | +----> CONSUMER HEALTH MVP ✅ | [3B.1 Time-Travel] ✅ ---------------------------------------+ | [3A.3 Source Metadata] ✅ -----------------------------------+ | [3C.1 Skeptic Lens] ✅ --------------------------------------+ ``` ### Critical Path for Financial DD Demo ``` [3A.2 Conflict Score] ✅ --> [3C.1 Skeptic Lens] ✅ ---+ | [3B.1 Time-Travel] ✅ ----------------------------------+----> FINANCIAL DD MVP | [2.3 EpochAwareLens] --> [3D.1 Epoch Cascade] ✅ -------+ | [3B.2 Semantic Decay] ✅ -------------------------------+ ``` ### Critical Path for Agile Agent Team Demo ``` [3C.3 Constraints Lens] (standalone) ------+ | [3B.1 Time-Travel] ✅ --------------------+----> AGENT TEAM MVP | [2.3 EpochAwareLens] ✅ ------------------+ | [Query Audit (Phase 2)] ✅ ----------------+ ``` ### Critical Path for Browser Extension ``` Phase 3 (Data Foundation) Phase 4 (Extension Primitives) Extension Product ========================= ============================== ================= [3A.1 Source-Class] ✅ ──────────> [4.5 Conflict Filtering] ✅ ──┐ | [3A.2 Conflict Score] ✅ ─────────────────────────────────────────┤ | [3C.1 Skeptic Lens] ✅ ──────────────────────────────────────────-┤ ├──> PHASE 1: Read-Only Overlay [3B.2 Semantic Decay] ✅ ────────────────────────────────────────-┤ (Benign Layer) | [3C.2 Layered Consensus] ✅ ─────────────────────────────────────-┘ [Phase 2 Vote System] ✅ ────────> [4.4 Vote Provenance] ✅ ─────┐ | [TrustRank Engine] ✅ ───────────> [4.7 Gold Standards] ✅ ───────┤ ├──> PHASE 2: Active Layer [3A.2 Conflict Score] ✅ ────────> [4.6 Escalation Triggers] ✅ ──┤ (Vote to See) | [7A PoW Admission] ─────────────┤ [7B EigenTrust] ────────────────┘ ``` **Phase 1 (Read-Only)** requires: Source tiers, conflict scores, conflict filtering, skeptic lens, decay, layered consensus. **All complete.** **Phase 2 (Active)** requires: Vote provenance, gold standards, escalation triggers (all complete), PLUS Phase 7 Sybil defense (PoW, EigenTrust — not started). ### Critical Path to Distributed Cluster ``` Phase 5 (The Forge) Phase 6 (The Mesh) Phase 7+8 ======================= ======================= ================== [5A.1 Replace sled ✅] ───────────> [6A.1 CRDT Foundation] ──┐ | | [5A.2 Key Layout] ───────────────> [6C.2 Range Sharding] ──> | | [5B.1 CRC32C Checksums] ──┐ | [5B.2 Crash Recovery] ────┼──────> [6B.1 RPC Layer] ─────────┤ [5B.3 Group Commit] ──────┘ | | v | [5C.1 Persistent Vector] ─────── (independent, no blocker) | [5C.2 Persistent Visual] ─────── (independent, no blocker) | | [6A.2 HLC Timestamps] ────┤ [6A.3 Merkle Tree] ───────┤ | | v v [6B.2 Gossip] ──> [6B.3 Anti-Entropy] ──> [6B.4 Two-Node Test] | v [6C.1 SWIM Membership] ──> [6C.3 Raft MV Coord] [6C.4 Gateway] ──────────> │ v DISTRIBUTED CLUSTER | [7A PoW Admission] ──┐ [7B EigenTrust] ─────┤──> THE SHIELD [7C Content Defense] ┤ [7D Circuit Breakers]┘ | [8A Chaos Testing] ──┐ [8B Observability] ──┤──> THE SWARM [8C Geo-Distribution]┘ ``` ### New Crates (Phases 5-8) ``` stemedb-rpc (Phase 6B) ── gRPC services for node-to-node communication stemedb-cluster (Phase 6C) ── Cluster membership, range routing, gateway stemedb-sync (Phase 6B) ── Merkle sync, gossip broadcast, anti-entropy ```