stemedb/roadmap.md

Episteme (StemeDB) Roadmap

Goal: Build the "Git for Truth" substrate for autonomous AI research. Current Phase: Phase 2.5 (Hardening) Target Vertical: BioTech/Pharma ("The Living Review")

---

High-Level Timeline

Phase	Codename	Focus	Key Deliverable
1	The Spine	Storage & Safety	Append-only WAL + KV Store
2	The Lattice	Indexing & Async	Materialized Views + Ballot Box
2.5	Hardening	Camp 2 Fixes	MV staleness, epoch behavior, lens cleanup
3	The Pilot	Vertical Integration	Pharma Ingestion + Living Review Agent
4	The Hive	Trust & Learning	The Simulator + Super Curator

---

Detailed Milestones

Phase 1: The Spine (Foundation)

Goal: Securely ingest assertions and persist them without data loss.

• Project Scaffold: Initialize Rust workspace, set up linting/CI (clippy, fmt).
• Assertion Schema: Define the Assertion struct with rkyv serialization.
  • Add dependencies: rkyv, blake3, ed25519-dalek, image_hasher.
  • Define Assertion struct (Subject, Predicate, Object, Confidence, SourceHash).
  • Multi-Sig Expansion: Implement SignatureEntry struct and signatures: Vec<SignatureEntry> field.
  • Visual Expansion: Add visual_hash: Option<pHash> field for image provenance.
  • Test serialization round-trips.
• Ballot Schema: Define the Vote struct for multi-agent consensus.
  • Add Vote struct: assertion_hash, agent_id, weight, signature.
  • Test serialization round-trips.
• Paradigm Schema (Epochs): Define the Epoch and SupersessionType structs.
  • Add epoch: Option<EpochId> to Assertion.
  • Implement Epoch struct with supersedes and SupersessionType.
  • Test serialization round-trips.
• WAL Integration: Implement the Quarantine Pattern for write-ahead logging.
  • Create stemedb-wal crate.
  • Port FsyncGuard and Record logic from established durability patterns.
  • Implement Record format with BLAKE3 checksums and Headers.
  • Verify fsync behavior with tests.
• Storage Engine: Implement the Store trait using sled (embedded KV).
  • Add sled dependency.
  • Define KVStore trait (put, get, delete, scan_prefix, flush).
  • Implement SledStore wrapper.
• Basic Ingestor: Background worker that tails WAL and writes to KV.
  • Implement async loop reading from WAL.
  • Write deserialized assertions, votes, and epochs to sled.
  • Ed25519 signature verification during ingestion.
  • Maintains S: and SP: indexes on ingest.
  • Persistent cursor/checkpoint (resumes from __CURSOR__:ingest in KV store).
• Verification: Crash recovery tests (write -> crash -> restart -> read).
  • Single and multi-record crash recovery.
  • Multiple crash cycles tested.

Phase 2: The Lattice (Connectivity)

Goal: Query data with sub-millisecond latency using Materialized Views.

• Lifecycle Schema: Add LifecycleStage to Assertion.
  • Define enum: Proposed, UnderReview, Approved, Deprecated, Rejected.
  • Update Assertion struct and serialization tests.
• The Ballot Box: Implement high-velocity vote ingestion.
  • VoteStore trait and implementation.
  • VoteAwareConsensusLens for real vote-based resolution.
• Index Infrastructure: Compound indexes for O(1) queries.
  • IndexStore trait with S: and SP: indexes.
  • QueryEngine smart routing (SP -> S -> scan).
• Materializer: Background worker for O(1) Read Performance.
  • MaterializedView type in stemedb-core.
  • Materializer worker in stemedb-query with step() and run().
  • Aggregates Votes via VoteAwareConsensusLens (or any AsyncLens).
  • Updates MV:{Subject}:{Predicate} with the winning Assertion + metadata.
  • Event-driven mode via run_notified() with tokio::sync::Notify.
  • Fast-path MV lookup in QueryEngine::try_fast_path().
• The Meter: Implement Economic Throttling (TAN).
  • QuotaStore trait and GenericQuotaStore implementation.
  • Token Bucket algorithm with per-agent per-hour quotas.
  • MeterLayer tower middleware for request cost tracking.
  • Cost model: Assert=10, Vote=1, Query=5+lens, +1/KB payload.
  • GET /v1/meter/quota endpoint to check remaining quota.
  • POST /v1/meter/quota/limit admin endpoint to set custom limits.
• API Surface: axum HTTP server with OpenAPI (utoipa).
  • POST /v1/assert -> Accepts JSON, writes to WAL.
  • POST /v1/vote -> High-throughput vote endpoint.
  • POST /v1/epoch -> Create epoch with optional supersession.
  • GET /v1/query -> Subject/Predicate/Lens/Lifecycle/Epoch filtering.
  • GET /v1/health -> Health check with assertion count.
  • GET /swagger-ui -> Interactive API docs.
  • 5 lens types available: Recency, Consensus, Authority, VoteAwareConsensus, TrustAwareAuthority.
• Query Audit: Log every read with provenance.
  • Define QueryAudit struct: query_id, agent_id, timestamp, params, result_hash, contributing_assertions.
  • Storage at AUD:{query_id} with agent index at AUDA:{agent_id}:{timestamp}:{query_id}.
  • GET /v1/audit/queries -> Returns history of agent decisions.
  • GET /v1/audit/query/{id} -> Full reasoning trace for a single query.
  • Auto-logging on every query via X-Agent-Id header.

Phase 2.5: Hardening (Camp 2 Fixes)

Goal: Close the gaps between "built" and "works right." Every item here addresses a feature that exists but doesn't fully deliver on its promise.

• 2.1 MV Staleness Detection: Make the fast-path aware of stale materialized views.

  • Status: ✅ COMPLETE
  • Implementation:
    • Added max_stale: Option<u64> to Query struct in crates/stemedb-query/src/query.rs.
    • Added .max_stale(secs) builder method to QueryBuilder.
    • In try_fast_path(): if query.max_stale is set and MV age exceeds threshold, falls through to slow path with debug! log.
    • Added max_stale to API QueryParams DTO in crates/stemedb-api/src/dto.rs.
    • Wired through query handler in crates/stemedb-api/src/handlers/query.rs.
  • Tests:
    • test_fast_path_stale_view_falls_back: MV 1000 seconds old, max_stale = 60 → slow path used.
    • test_fast_path_fresh_view_used: Fresh MV, max_stale = 300 → fast path used.
    • test_fast_path_no_max_stale_always_uses_mv: No max_stale → any MV age accepted (backward compatible).
    • test_fast_path_max_stale_zero_rejects_old_mv: max_stale = 0, MV 1 second old → slow path.
    • test_fast_path_max_stale_zero_accepts_brand_new_mv: max_stale = 0, brand new MV → fast path.

• 2.2 AuthorityLens -> ConfidenceLens Rename: Eliminate the misleading name.

  • Problem: AuthorityLens selects by confidence field, not by agent reputation. TrustAwareAuthorityLens is the real authority lens. The name creates confusion about what "Authority" means.
  • Solution implemented:
    • Renamed authority.rs → confidence.rs, AuthorityLens → ConfidenceLens
    • Added LensDto::Confidence for the confidence-field selector
    • Changed LensDto::Authority to route to TrustAwareAuthorityLens (the real authority lens)
    • Updated query handler routing
    • Updated ai-lookup/services/lens.md and skill documentation

• 2.3 EpochAwareLens: Give epoch supersession runtime behavior.

  • Status: ✅ COMPLETE
  • Implementation:
    • EpochAwareLens in crates/stemedb-lens/src/epoch_aware.rs
    • Decorator pattern wrapping any inner lens (default: RecencyLens)
    • Walks supersession chain from E:{epoch_id} keys
    • Cycle detection + max depth guard (100)
    • Fail-open on missing epochs
    • LensDto::EpochAware added to API
    • 11 tests: excludes_superseded, chain_supersession, no_epochs_passes_all, missing_epoch_includes, cycle_detection, consensus_lens_inner, mixed_epochs, etc.
    • Documentation updated in ai-lookup/services/lens.md
  • Known Limitation: Filtering only occurs when assertions from the superseding epoch are present in candidates. If all candidates are from old epoch (no new epoch assertions), they pass through (fail-open behavior).

• 2.4 Visual Hash Query Support: Make the stored visual_hash queryable.

  • Problem: visual_hash: Option<PHash> exists on Assertion and is stored/returned by the API, but there is no way to query by visual similarity. The field is write-only from a query perspective.
  • Current state: PHash is [u8; 8] (perceptual hash). Stored on assertions. API accepts/returns it. No query parameter. No similarity computation.
  • Add visual_near: Option<String> (hex-encoded pHash) and visual_threshold: Option<u32> (max hamming distance, default: 8) to Query struct.
  • Add .visual_near(hash, threshold) to QueryBuilder.
  • In Query::matches(): if visual_near is set and assertion has visual_hash, compute hamming distance (XOR + popcount on the 8 bytes). If distance <= threshold, match. If assertion has no visual_hash, don't match.
  • Add visual_near and visual_threshold to API QueryParams DTO.
  • Implement hamming_distance(a: &[u8; 8], b: &[u8; 8]) -> u32 utility function.
  • Tests:
    • test_visual_near_exact_match: Same hash, threshold 0. Matches.
    • test_visual_near_within_threshold: Hashes differ by 3 bits, threshold 5. Matches.
    • test_visual_near_exceeds_threshold: Hashes differ by 10 bits, threshold 5. No match.
    • test_visual_near_skips_assertions_without_hash: Assertion has no visual_hash. Not matched.
  • Note: This is a brute-force scan approach (O(N) with hamming distance check). A proper VP-tree or BK-tree index is Phase 3. This gives immediate queryability.

• 2.5 Vector Field: No changes needed. Already roadmapped for Phase 3.

  • Current state: vector: Option<Vec<f32>> on Assertion. Stored and returned by API. No index, no search.
  • Phase 3 plan: Integrate hnsw-rs or lance for k-NN search.
  • No Phase 2.5 work required. The field scaffolding is correct.

• 2.6 E2E Integration Test (Write -> Materialize -> Read): Prove the full pipeline works end-to-end.

  • Problem: The IngestWorker, Materializer, and QueryEngine have been tested in isolation. The Notify integration between IngestWorker and Materializer is tested with a single notification. No test wires all three components together to verify the full write-materialize-read loop.
  • Current state: IngestWorker has with_notify(Arc<Notify>) (worker.rs). Materializer has run_notified(Arc<Notify>, Duration) (materializer.rs). QueryEngine has try_fast_path() (engine.rs). Never tested together.
  • Create integration test in crates/stemedb-query/tests/e2e_pipeline.rs:
    • Setup: Create temp WAL + SledStore + VoteStore + TrustRankStore.
    • Wire: IngestWorker with with_notify(notify), Materializer with run_notified(notify), QueryEngine with same store.
    • Test steps:
      1. Write a signed assertion to WAL.
      2. Run IngestWorker.step() -> verifies assertion ingested to KV.
      3. Verify IngestWorker triggered Notify.
      4. Run Materializer.step() -> verifies MV written.
      5. Execute QueryEngine.execute() with subject+predicate -> verifies fast-path returns MV winner.
    • Variant: Write 2 competing assertions, add votes favoring one, run pipeline, verify correct winner in MV.
    • Variant: Write assertion, materialize, write new assertion with higher timestamp, re-materialize, verify MV updated, verify query returns new winner.
  • Add stemedb-wal and stemedb-ingest as dev-dependencies in stemedb-query/Cargo.toml.

Phase 3: The Pilot (BioTech/Pharma)

Goal: Prove value in the "High-Liability" beachhead. Close every Camp 4 gap that blocks a credible demo.

3A. Schema Expansion (Prerequisite for everything below)

• 3A.1 Source-Class Field: Add source_class: SourceClass to Assertion.

  • Status: ✅ COMPLETE
  • Implementation:
    • SourceClass enum in crates/stemedb-core/src/types.rs (lines 68-88).
    • 6-tier system: Regulatory (0), Clinical (1), Observational (2), Expert (3), Community (4), Anecdotal (5).
    • tier() method returns tier number for ordering.
    • default_decay_days() method for tier-specific confidence decay.
    • authority_weight() method for conflict resolution weighting.
    • Field on Assertion struct at line 152.
    • Full serialization and indexing support.

• 3A.2 Conflict Score on Resolution: Add conflict_score: f32 to Resolution.

  • Problem: All 4 use cases reference conflict_score in query responses. Resolution has resolution_confidence but no conflict metric. Without a conflict score, the "disagreement is the information" thesis has no numeric representation.
  • Current state: Resolution at traits.rs:10-20 has winner, candidates_count, resolution_confidence. No variance computation.
  • Add field to Resolution in crates/stemedb-lens/src/traits.rs:10:

    /// Degree of disagreement among candidates (0.0 = full agreement, 1.0 = max conflict).
    /// Computed as normalized variance of candidate confidence values.
    pub conflict_score: f32,
    

  • Update Resolution::empty() at traits.rs:24: set conflict_score: 0.0.
  • Update Resolution::with_winner() at traits.rs:29: accept conflict_score parameter.
  • Add utility function compute_conflict_score(candidates: &[Assertion]) -> f32:
    • Compute mean confidence: mean = sum(c.confidence) / N.
    • Compute variance: var = sum((c.confidence - mean)^2) / N.
    • Normalize to [0.0, 1.0]: conflict = (4.0 * var).min(1.0) (variance of [0,1] range has max 0.25, so 4x normalizes).
    • Edge cases: 0 or 1 candidates -> 0.0.
  • Update every Lens::resolve() and AsyncLens::resolve_async() implementation to call compute_conflict_score() and pass it to Resolution::with_winner():
    • crates/stemedb-lens/src/recency.rs
    • crates/stemedb-lens/src/consensus.rs
    • crates/stemedb-lens/src/authority.rs (or confidence.rs after rename)
    • crates/stemedb-lens/src/vote_aware_consensus.rs
    • crates/stemedb-lens/src/trust_aware_authority.rs
  • Add conflict_score: f32 to MaterializedView in crates/stemedb-core/src/types.rs:143.
  • Update Materializer::materialize_pair() at materializer.rs:164 to write conflict_score from resolution.
  • Add conflict_score to QueryResponse DTO (or a new ResolutionMeta sub-object) in crates/stemedb-api/src/dto.rs.
  • Wire conflict_score through in the query handler's apply_lens() at handlers/query.rs:111.
  • Tests:
    • test_conflict_score_zero_for_agreement: 3 candidates all confidence 0.9. Score near 0.0.
    • test_conflict_score_high_for_disagreement: Candidates at 0.1, 0.5, 0.9. Score > 0.5.
    • test_conflict_score_zero_for_single: 1 candidate. Score = 0.0.
    • test_conflict_score_on_materialized_view: MV stores conflict_score after materialization.

• 3A.3 Rich Source Metadata: Add structured provenance beyond source_hash.

  • Problem: Assertion has source_hash: Hash (a 32-byte hash) but no structured metadata. Consumer Health use case shows POST bodies with journal, DOI, sample_size, subreddit, upvotes. Without this, assertions are opaque about their provenance.
  • Current state: source_hash: Hash at types.rs:64. No metadata field. API accepts/returns only the hash.
  • Add field to Assertion in crates/stemedb-core/src/types.rs:65 (after source_hash):

    /// Structured source metadata as a JSON-encoded byte string.
    /// Schema is domain-specific (journal info, social metrics, etc.).
    pub source_metadata: Option<Vec<u8>>,
    

  • Use Vec<u8> (not String) for rkyv zero-copy compatibility. Callers encode/decode JSON on their side.
  • Add source_metadata: Option<Vec<u8>> to AssertionBuilder. Add .source_metadata_json(json: &str) builder method that stores json.as_bytes().to_vec().
  • Add source_metadata: Option<String> to CreateAssertionRequest DTO (JSON string in API, converted to bytes internally).
  • Add source_metadata: Option<String> to AssertionResponse DTO (bytes converted to JSON string).
  • Wire through create handler and query handler.
  • Tests:
    • Serialization roundtrip with metadata present and absent.
    • API roundtrip: POST with metadata JSON, GET returns same JSON.
  • Note: Metadata is stored but NOT indexed in Phase 3. Indexing individual metadata fields is Phase 4+.

3B. Time & Decay (Core Query Features)

• 3B.1 Time-Travel Engine: as_of parameter for historical queries.

  • Problem: All 4 use cases reference as_of for historical state. "What was the consensus when I made my decision?" The append-only model stores all history, but there's no way to query a past state.
  • Current state: Query at query.rs:14-29 has no as_of field. try_fast_path() returns current MV regardless.
  • Add as_of: Option<u64> to Query struct in crates/stemedb-query/src/query.rs:14.
  • Add .as_of(timestamp: u64) to QueryBuilder.
  • In Query::matches() at query.rs:43: if as_of is Some(ts), check assertion.timestamp <= ts. Assertions created after as_of are excluded.
  • In QueryEngine::execute() at engine.rs:47: if query.as_of is set, skip the fast path entirely (MVs reflect current state, not historical). Add early check before the try_fast_path call:

    if query.as_of.is_none() {
        if let (Some(subject), Some(predicate)) = (&query.subject, &query.predicate) {
            if let Some(result) = self.try_fast_path(subject, predicate, query).await? {
                return Ok(result);
            }
        }
    }
    

  • Add as_of: Option<u64> to QueryParams DTO in crates/stemedb-api/src/dto.rs:102.
  • Wire in query handler.
  • Tests:
    • test_as_of_excludes_future_assertions: 3 assertions at t=1000, t=2000, t=3000. Query as_of=2500. Returns only first 2.
    • test_as_of_bypasses_fast_path: MV exists, but as_of is set. Slow path used, MV ignored.
    • test_as_of_none_uses_fast_path: Normal query still uses fast path (backwards-compatible).
    • test_as_of_with_lens: Time-travel + lens = resolve only among pre-as_of candidates.

• 3B.2 Semantic Decay: Confidence Half-Life at query time.

  • Problem: Medical knowledge decays at different rates. A Reddit post from 2022 shouldn't compete equally with a 2024 RCT. No assertion-level decay exists. TrustRank has agent-level decay but not assertion-level.
  • Current state: TrustRank.apply_decay() at trust_rank_store.rs does agent decay. No assertion decay. No decay query parameter.
  • Add decay_halflife: Option<u64> to Query struct (seconds). Represents how quickly old assertions lose effective confidence.
  • Add .decay_halflife(seconds: u64) to QueryBuilder.
  • Add decay_halflife to QueryParams DTO.
  • Implementation strategy: Apply decay in QueryEngine before passing to lens. In execute(), after fetching and filtering candidates, if decay_halflife is set:
    1. Get current time (or as_of if set).
    2. For each candidate, compute age = now - assertion.timestamp.
    3. Compute effective_confidence = confidence * 2_f32.powf(-(age as f32) / (halflife as f32)).
    4. Clone the assertion with the decayed confidence.
    5. Pass decayed candidates to the lens.
  • Add helper apply_decay(assertions: &[Assertion], halflife: u64, now: u64) -> Vec<Assertion> in a new crates/stemedb-query/src/decay.rs module.
  • Source-class-aware decay (Phase 3 stretch): If assertion has source_class, use per-tier half-lives from SourceClass::default_decay_days(). This is already implemented on the enum.
  • Tests:
    • test_decay_reduces_old_assertion_confidence: Assertion 1yr old, halflife 1yr. Effective confidence ~= original * 0.5.
    • test_decay_preserves_fresh_assertions: Assertion 1hr old, halflife 1yr. Effective confidence ~= original.
    • test_decay_interacts_with_lens: Two assertions, older has higher base confidence but after decay, newer wins via RecencyLens.
    • test_source_aware_decay: Tier 0 doesn't decay. Tier 5 decays rapidly.

3C. New Lenses

• 3C.1 Skeptic Lens: Surface disagreement, not winners. ✅ COMPLETED

  • Status: ✅ COMPLETE
  • Implementation:
    • crates/stemedb-lens/src/skeptic.rs - Full implementation.
    • AnalysisLens trait for lenses that analyze conflict instead of resolving it.
    • SkepticLens uses normalized Shannon entropy for conflict scoring.
    • Returns ConflictAnalysis with:
      • conflict_score: f32 (0.0 = unanimous, 1.0 = chaos)
      • status: ResolutionStatus (Unanimous, Agreed, Contested)
      • claims: Vec<ClaimSummary> - all claims ranked by weight
    • SkepticResolver + SkepticView in stemedb-query/src/skeptic.rs.
    • GET /v1/skeptic?subject=X&predicate=Y API endpoint.
    • Core types in stemedb-core/src/types.rs:
      • ResolutionStatus enum
      • ConflictAnalysis struct
      • ClaimSummary, SourceSummary, AgentSummary
    • Comprehensive test coverage (21 test cases).

• 3C.2 Layered Consensus Lens: Per-source-class consensus.

  • Problem: Current lenses return a single winner. The Consumer Health use case needs per-tier consensus: "What does Tier 0 say? What does Tier 1 say? What does Tier 5 say?" This is the core differentiator.
  • Depends on: Source-Class Field (3A.1) ✅ COMPLETE.
  • New file: crates/stemedb-lens/src/layered_consensus.rs.
  • New type in crates/stemedb-lens/src/traits.rs:

    /// Per-tier resolution result.
    pub struct TierResolution {
        pub tier: u8,
        pub winner: Option<Assertion>,
        pub candidates_count: usize,
        pub conflict_score: f32,
    }
    
    /// Multi-tier resolution result.
    pub struct LayeredResolution {
        /// Per-tier consensus results, ordered by tier (0 = highest authority).
        pub tiers: Vec<TierResolution>,
        /// Overall winner (highest-tier with a winner).
        pub overall_winner: Option<Assertion>,
        /// Overall conflict score (cross-tier disagreement).
        pub overall_conflict_score: f32,
    }
    

  • LayeredConsensusLens implements a new LayeredLens trait:

    pub trait LayeredLens: Send + Sync {
        fn resolve_layered(&self, candidates: &[Assertion]) -> LayeredResolution;
        fn name(&self) -> &'static str;
    }
    

  • resolve_layered() logic:
    1. Group candidates by source_class (use SourceClass::tier() method).
    2. For each tier group, run ConsensusLens::resolve() to get within-tier winner.
    3. Compute per-tier conflict_score.
    4. Overall winner = winner from the highest-authority tier that has candidates (lowest tier number).
    5. Overall conflict_score = cross-tier disagreement (do tier winners agree on the same object value?).
  • Also implement standard Lens trait on LayeredConsensusLens to maintain compatibility: resolve() returns overall_winner as a regular Resolution. The richer LayeredResolution is accessible via resolve_layered().
  • Add LayeredConsensus to LensDto enum.
  • New API response type: LayeredQueryResponse with per-tier results. Wire as a variant in the query handler when lens=LayeredConsensus.
  • Export from crates/stemedb-lens/src/lib.rs.
  • Tests:
    • test_layered_single_tier: All candidates same source_class. Returns one tier result.
    • test_layered_multi_tier_agreement: Tier 0 and Tier 5 agree on same object. Low cross-tier conflict.
    • test_layered_multi_tier_disagreement: Tier 1 says "safe", Tier 5 says "dangerous". High cross-tier conflict. Overall winner from Tier 1.
    • test_layered_overall_winner_from_highest_authority: Tier 0 present -> its winner is overall winner even if Tier 5 has 1000x more assertions.

• 3C.3 Constraints Lens: Pre-flight check for must_use/forbidden.

  • Problem: Agile Agent Team use case needs lens=constraints returning { must_use: "axios", forbidden: "requests" }. Central to "persistent learning" — agents query constraints before acting.
  • New file: crates/stemedb-lens/src/constraints.rs.
  • Design: Not a traditional lens (doesn't pick one winner from candidates). Instead, it categorizes candidates by predicate pattern:
    • Assertions with predicate matching must_use:* -> must_use list.
    • Assertions with predicate matching forbidden:* -> forbidden list.
    • Assertions with predicate matching prefer:* -> prefer list.
  • Implements Lens trait for compatibility: resolve() returns the highest-confidence must_use assertion as the "winner" (or forbidden if no must_use exists). The richer result is accessible via a dedicated method.
  • Add dedicated resolve_constraints() method:

    pub struct ConstraintSet {
        pub must_use: Vec<Assertion>,
        pub forbidden: Vec<Assertion>,
        pub prefer: Vec<Assertion>,
    }
    
    impl ConstraintsLens {
        pub fn resolve_constraints(&self, candidates: &[Assertion]) -> ConstraintSet { ... }
    }
    

  • Add Constraints to LensDto enum.
  • New API response type: ConstraintResponse with categorized assertions.
  • Export from crates/stemedb-lens/src/lib.rs.
  • Tests:
    • test_constraints_categorizes_by_predicate: Mixed predicates sorted into must_use/forbidden/prefer.
    • test_constraints_empty_categories: No must_use predicates -> empty must_use list.
    • test_constraints_lens_trait_picks_must_use_winner: Standard resolve() returns highest-confidence must_use.
    • test_constraints_non_constraint_predicates_ignored: Regular predicates not categorized.

3D. Epoch Enhancement

• 3D.1 Epoch Cascade Logic (enhancement of Phase 2.5 EpochAwareLens):
  • Problem: Phase 2.5 EpochAwareLens walks the supersedes chain at query time by reading E: keys. This is O(chain_length) per query. For long supersession chains, this is expensive.
  • Depends on: Phase 2.5 EpochAwareLens (basic version).
  • In IngestWorker::process_epoch() at crates/stemedb-ingest/src/worker.rs:
    1. When an epoch with supersedes = Some(old_epoch_id) is ingested, write a marker key SUPERSEDED:{old_epoch_id} with the new epoch ID as value.
    2. Walk the chain: if old_epoch itself superseded another epoch, write SUPERSEDED:{grandparent_id} too (transitive closure).
  • Update EpochAwareLens to check SUPERSEDED:{epoch_id} keys (O(1) lookup per candidate) instead of walking the chain at query time.
  • Tests:
    • test_cascade_writes_superseded_marker: Ingest epoch B superseding A. Verify SUPERSEDED:A key exists.
    • test_cascade_transitive: A supersedes B, B supersedes C. Verify both SUPERSEDED:B and SUPERSEDED:C exist.
    • test_epoch_aware_uses_marker: Query with EpochAwareLens uses marker key, not chain walk.

3E. Similarity Search

• 3E.1 Vector Search: Semantic k-NN queries via embeddings.

  • Current state: vector: Option<Vec<f32>> on Assertion. Stored, returned by API. No index, no search.
  • Add hnsw-rs (or lance) as a dependency in stemedb-storage/Cargo.toml.
  • New module: crates/stemedb-storage/src/vector_index.rs.
  • VectorIndex trait: insert(hash: Hash, vector: &[f32]), search(query: &[f32], k: usize) -> Vec<(Hash, f32)>.
  • Implementation backed by HNSW graph stored alongside KV data.
  • IngestWorker: if assertion has vector, insert into vector index after KV write.
  • Add vector_near: Option<Vec<f32>> and k: Option<usize> to Query struct and API QueryParams.
  • QueryEngine: if vector_near is set, use vector index for candidate retrieval instead of SP/S index.
  • Tests: insert 100 vectors, query nearest 5, verify correct neighbors.

• 3E.2 Visual Hash Index: VP-tree or BK-tree for O(log N) visual similarity.

  • Current state: Phase 2.5 adds brute-force hamming scan. This replaces it with an indexed approach.
  • New module: crates/stemedb-storage/src/visual_index.rs.
  • BK-tree implementation over hamming distance on PHash ([u8; 8]).
  • IngestWorker: if assertion has visual_hash, insert into BK-tree.
  • QueryEngine: if visual_near is set, use BK-tree for O(log N) candidate retrieval instead of brute-force scan.
  • Tests: insert 1000 assertions with visual hashes, query similar, verify correct matches within threshold.

3F. Provenance

• 3F.1 Citation Recall Benchmarking: Verify 100% provenance tracking.
  • Benchmark: for every assertion in the store, verify source_hash resolves to a retrievable source document (or at minimum, the hash is non-zero and unique).
  • Add GET /v1/provenance/{hash} endpoint that looks up source by hash.
  • Add POST /v1/source endpoint to store source documents by content hash.
  • Source storage at SRC:{blake3_hash} keys.

3G. API Cleanup

• 3G.1 Document epoch supersession via existing endpoint: No new /epoch/supersede endpoint needed.
  • Current state: POST /v1/epoch already accepts supersedes field. Use cases show POST /epoch/supersede as if it's a separate endpoint.
  • Update use case docs (consumer-health-intelligence.md, glp1-living-review.md) to use POST /v1/epoch with supersedes field.
  • Add OpenAPI examples showing the supersession flow.
  • No code change. Documentation fix only.

Phase 4: The Hive (Trust & Scale)

Goal: Change tracking, metadata indexing, and the database primitives for training pipelines.

• TrustRank Engine: Foundation for trust-based resolution.

  • TrustRankStore for per-agent reputation storage.
  • TrustAwareAuthorityLens for reputation-weighted resolution.
  • Confidence Half-Life: Implement decay calculation engine.
  • Learning loop: record_outcome() for accuracy tracking.

• 4.1 "Since" Parameter: Change tracking for returning consumers.

  • Problem: Consumer Health shows GET /query?since=2023-10-01 returning changes_since_query with dated change entries. The "returning consumer" story: "What changed since I last looked?"
  • Depends on: Time-Travel (3B.1) and Materializer.
  • Add since: Option<u64> to Query struct and QueryBuilder in crates/stemedb-query/src/query.rs.
  • Add since to QueryParams DTO.
  • MV Changelog: Track when materialized views change.
    • New key pattern: MVC:{subject}:{predicate}:{timestamp} storing the previous winner hash and new winner hash.
    • In Materializer::materialize_pair() at materializer.rs:164: before overwriting MV, read the existing MV. If the winner changed (different assertion hash), write a changelog entry.
  • In QueryEngine: if since is set, scan MVC:{subject}:{predicate}:* for entries with timestamp > since. Return these as a changes_since list alongside the normal query result.
  • New response field on QueryResponse:

    pub struct ChangeEntry {
        pub timestamp: u64,
        pub previous_winner_hash: Option<String>,
        pub new_winner_hash: String,
    }
    

    Add changes_since: Option<Vec<ChangeEntry>> to QueryResponse DTO.
  • Tests:
    • test_since_returns_changes: Materialize, change winner, re-materialize. Query with since returns the change.
    • test_since_no_changes: No MV changes since timestamp. Empty changes list.
    • test_since_multiple_changes: 3 winner changes over time. All returned in order.

• 4.2 Source Metadata Indexing (extension of 3A.3): Index key metadata fields.

  • Problem: Phase 3 stores source_metadata as an opaque blob. Phase 4 makes key fields queryable.
  • Depends on: Rich Source Metadata (3A.3).
  • Define a set of indexed metadata keys: journal, doi, platform, study_design.
  • New key pattern: SM:{field}:{value}:{assertion_hash} for metadata field indexes.
  • IngestWorker: on ingestion, if source_metadata is present, parse JSON, extract indexed fields, write index entries.
  • Add metadata field filters to QueryParams (e.g., ?source_journal=NEJM).
  • Tests: store assertions with metadata, query by journal name, verify correct filtering.

• 4.3 Batch TrustRank Decay API: Expose scheduled decay for external orchestration.

  • Current state: decay_all_trust_ranks() exists on TrustRankStore. No API endpoint.
  • Add POST /v1/admin/decay-trust-ranks endpoint.
  • Accepts now: u64 parameter (or uses current time).
  • Returns count of decayed agents and summary stats.
  • Note: The Gardener (Camp 5.2, app layer) calls this endpoint on a schedule. The database just exposes the primitive.

Note: The following items were reclassified as Application Layer responsibilities (see tmp/ambition-vs-reality.md, Camp 5). They are not Episteme database features. They consume the Episteme API and are built by integrators or vertical-specific teams.

• The Simulator (Training Data Pipeline) -> Camp 5.3
• The Super Curator (Reviewer Agent swarm) -> Camp 5.4
• Background Gardener (Cluster detection, signal processing) -> Camp 5.2
• Agent Wallet (Key management sidecar) -> Camp 5.1

---

Tracking

Active Tasks

• Phase 2.5 Hardening: Camp 2 fixes (staleness, epoch behavior, lens rename, visual query, E2E test).

Next Up (Phase 3 sequencing)

• 3A.2 Conflict Score: Add to Resolution. Propagate through all lenses.
• 3A.3 Rich Source Metadata: Structured provenance field.
• 3B.1 Time-Travel: as_of parameter. Unblocks all 4 use cases.
• 3C.2 Layered Consensus Lens: Per-tier resolution (now unblocked by 3A.1).

Recently Completed

• SkepticLens + SkepticView (3C.1): "Trust but Verify" conflict analysis that surfaces all claims with conflict scores.
  • AnalysisLens trait for lenses that map conflict instead of resolving it.
  • SkepticLens using normalized Shannon entropy for conflict scoring.
  • SkepticResolver + SkepticView in stemedb-query.
  • GET /v1/skeptic?subject=X&predicate=Y API endpoint.
  • Types: ResolutionStatus, ConflictAnalysis, ClaimSummary, SourceSummary, AgentSummary.
• Source-Class Field (3A.1): 6-tier SourceClass enum with authority weighting and decay rates.
  • SourceClass enum: Regulatory, Clinical, Observational, Expert, Community, Anecdotal.
  • tier(), default_decay_days(), authority_weight() methods.
  • Field on Assertion struct with full serialization support.
• The Meter: Token bucket quota system with MeterLayer middleware (10K tokens/agent/hour).
• Query Audit Trail: Every query logged with provenance at AUD:{query_id}. X-Agent-Id header for attribution.
• Event-Driven Materialization: run_notified() + IngestWorker Notify integration.
• Fast-Path MV Lookup: QueryEngine::try_fast_path() for O(1) reads.
• Materializer: Background worker for O(1) MV reads via AsyncLens.
• VoteAwareConsensusLens: Real vote-based consensus resolution.
• Compound SP Index: O(1) subject+predicate lookups.
• TrustRank System: Agent reputation with decay and learning loop.
• API Surface: axum HTTP server with 7 endpoints + OpenAPI docs.

Blockers

• None.

---

Dependency Graph

Phase 2.5 (Hardening)          Phase 3 (The Pilot)                    Phase 4 (The Hive)
========================       ========================               ==================

[2.1 MV Staleness]  ---------> [3B.1 Time-Travel] -----+
                                       |                |
[2.2 Confidence Rename] -----> (API clarity for all)    +----------> [4.1 "Since" Param]
                                                        |
[2.3 EpochAwareLens] --------> [3D.1 Epoch Cascade] ---|----------> Invalidation Cascades pillar
                                                        |
[2.4 Visual Hash Query] -----> [3E.2 Visual Hash Index] |
                                                        |
[2.6 E2E Integration] -------> (pipeline confidence)    |
                                                        |
                               [3A.1 Source-Class] ✅ --+----------> [3C.2 Layered Consensus]
                                       |                             [3B.2 Source-Aware Decay]
                                       +-----------------------------[4.2 Metadata Indexing]
                                                        |
                               [3A.2 Conflict Score] ---> (enhance Resolution)
                                                        |
                               [3A.3 Source Metadata] ---> [4.2 Metadata Indexing]
                                                        |
                               [3C.1 Skeptic Lens] ✅    (standalone, COMPLETE)
                               [3C.3 Constraints Lens]   (standalone)
                               [3E.1 Vector Search]      (standalone)
                               [3F.1 Citation Recall]     (standalone)

Critical Path for Consumer Health Demo

[3A.1 Source-Class] ✅ --> [3A.2 Conflict Score] --> [3C.2 Layered Consensus]
                                                              |
                                                              +----> CONSUMER HEALTH MVP
                                                              |
[3B.1 Time-Travel] ------------------------------------------+
                                                              |
[3A.3 Source Metadata] --------------------------------------+
                                                              |
[3C.1 Skeptic Lens] ✅ --------------------------------------+

Critical Path for Financial DD Demo

[3A.2 Conflict Score] --> [3C.1 Skeptic Lens] ✅ -------+
                                                         |
[3B.1 Time-Travel] -------------------------------------+----> FINANCIAL DD MVP
                                                         |
[2.3 EpochAwareLens] --> [3D.1 Epoch Cascade] ----------+
                                                         |
[3B.2 Semantic Decay] ----------------------------------+

Critical Path for Agile Agent Team Demo

[3C.3 Constraints Lens] (standalone) ------+
                                            |
[3B.1 Time-Travel] -----------------------+----> AGENT TEAM MVP
                                            |
[2.3 EpochAwareLens] ---------------------+
                                            |
[Query Audit (Phase 2)] ✅ ----------------+