jordan/stemedb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
152df4b0b4
stemedb/roadmap.md
jordan 152df4b0b4 docs: Mark Phase 2.4, 2.5, 2.6 as complete in roadmap 
- 2.4 Visual Hash Query: hamming_distance, visual_near/threshold implemented
- 2.5 Vector Field: N/A (Phase 3 work, scaffolding correct)
- 2.6 E2E Integration Test: e2e_pipeline.rs with 5 comprehensive tests

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-01 13:33:03 -07:00

39 KiB
Raw Blame History

Episteme (StemeDB) Roadmap

Goal: Build the "Git for Truth" substrate for autonomous AI research. Current Phase: Phase 2.5 (Hardening) Target Vertical: BioTech/Pharma ("The Living Review")

High-Level Timeline

Phase Codename Focus Key Deliverable
1 The Spine Storage & Safety Append-only WAL + KV Store
2 The Lattice Indexing & Async Materialized Views + Ballot Box
2.5 Hardening Camp 2 Fixes MV staleness, epoch behavior, lens cleanup
3 The Pilot Vertical Integration Pharma Ingestion + Living Review Agent
4 The Hive Trust & Learning The Simulator + Super Curator

Detailed Milestones

Phase 1: The Spine (Foundation)

Goal: Securely ingest assertions and persist them without data loss.

  • Project Scaffold: Initialize Rust workspace, set up linting/CI (clippy, fmt).
  • Assertion Schema: Define the Assertion struct with rkyv serialization.
    • Add dependencies: rkyv, blake3, ed25519-dalek, image_hasher.
    • Define Assertion struct (Subject, Predicate, Object, Confidence, SourceHash).
    • Multi-Sig Expansion: Implement SignatureEntry struct and signatures: Vec<SignatureEntry> field.
    • Visual Expansion: Add visual_hash: Option<pHash> field for image provenance.
    • Test serialization round-trips.
  • Ballot Schema: Define the Vote struct for multi-agent consensus.
    • Add Vote struct: assertion_hash, agent_id, weight, signature.
    • Test serialization round-trips.
  • Paradigm Schema (Epochs): Define the Epoch and SupersessionType structs.
    • Add epoch: Option<EpochId> to Assertion.
    • Implement Epoch struct with supersedes and SupersessionType.
    • Test serialization round-trips.
  • WAL Integration: Implement the Quarantine Pattern for write-ahead logging.
    • Create stemedb-wal crate.
    • Port FsyncGuard and Record logic from established durability patterns.
    • Implement Record format with BLAKE3 checksums and Headers.
    • Verify fsync behavior with tests.
  • Storage Engine: Implement the Store trait using sled (embedded KV).
    • Add sled dependency.
    • Define KVStore trait (put, get, delete, scan_prefix, flush).
    • Implement SledStore wrapper.
  • Basic Ingestor: Background worker that tails WAL and writes to KV.
    • Implement async loop reading from WAL.
    • Write deserialized assertions, votes, and epochs to sled.
    • Ed25519 signature verification during ingestion.
    • Maintains S: and SP: indexes on ingest.
    • Persistent cursor/checkpoint (resumes from __CURSOR__:ingest in KV store).
  • Verification: Crash recovery tests (write -> crash -> restart -> read).
    • Single and multi-record crash recovery.
    • Multiple crash cycles tested.

Phase 2: The Lattice (Connectivity)

Goal: Query data with sub-millisecond latency using Materialized Views.

  • Lifecycle Schema: Add LifecycleStage to Assertion.
    • Define enum: Proposed, UnderReview, Approved, Deprecated, Rejected.
    • Update Assertion struct and serialization tests.
  • The Ballot Box: Implement high-velocity vote ingestion.
    • VoteStore trait and implementation.
    • VoteAwareConsensusLens for real vote-based resolution.
  • Index Infrastructure: Compound indexes for O(1) queries.
    • IndexStore trait with S: and SP: indexes.
    • QueryEngine smart routing (SP -> S -> scan).
  • Materializer: Background worker for O(1) Read Performance.
    • MaterializedView type in stemedb-core.
    • Materializer worker in stemedb-query with step() and run().
    • Aggregates Votes via VoteAwareConsensusLens (or any AsyncLens).
    • Updates MV:{Subject}:{Predicate} with the winning Assertion + metadata.
    • Event-driven mode via run_notified() with tokio::sync::Notify.
    • Fast-path MV lookup in QueryEngine::try_fast_path().
  • The Meter: Implement Economic Throttling (TAN).
    • QuotaStore trait and GenericQuotaStore implementation.
    • Token Bucket algorithm with per-agent per-hour quotas.
    • MeterLayer tower middleware for request cost tracking.
    • Cost model: Assert=10, Vote=1, Query=5+lens, +1/KB payload.
    • GET /v1/meter/quota endpoint to check remaining quota.
    • POST /v1/meter/quota/limit admin endpoint to set custom limits.
  • API Surface: axum HTTP server with OpenAPI (utoipa).
    • POST /v1/assert -> Accepts JSON, writes to WAL.
    • POST /v1/vote -> High-throughput vote endpoint.
    • POST /v1/epoch -> Create epoch with optional supersession.
    • GET /v1/query -> Subject/Predicate/Lens/Lifecycle/Epoch filtering.
    • GET /v1/health -> Health check with assertion count.
    • GET /swagger-ui -> Interactive API docs.
    • 5 lens types available: Recency, Consensus, Authority, VoteAwareConsensus, TrustAwareAuthority.
  • Query Audit: Log every read with provenance.
    • Define QueryAudit struct: query_id, agent_id, timestamp, params, result_hash, contributing_assertions.
    • Storage at AUD:{query_id} with agent index at AUDA:{agent_id}:{timestamp}:{query_id}.
    • GET /v1/audit/queries -> Returns history of agent decisions.
    • GET /v1/audit/query/{id} -> Full reasoning trace for a single query.
    • Auto-logging on every query via X-Agent-Id header.

Phase 2.5: Hardening (Camp 2 Fixes)

Goal: Close the gaps between "built" and "works right." Every item here addresses a feature that exists but doesn't fully deliver on its promise.

  • 2.1 MV Staleness Detection: Make the fast-path aware of stale materialized views.

    • Status: ✅ COMPLETE
    • Implementation:
      • Added max_stale: Option<u64> to Query struct in crates/stemedb-query/src/query.rs.
      • Added .max_stale(secs) builder method to QueryBuilder.
      • In try_fast_path(): if query.max_stale is set and MV age exceeds threshold, falls through to slow path with debug! log.
      • Added max_stale to API QueryParams DTO in crates/stemedb-api/src/dto.rs.
      • Wired through query handler in crates/stemedb-api/src/handlers/query.rs.
    • Tests:
      • test_fast_path_stale_view_falls_back: MV 1000 seconds old, max_stale = 60 → slow path used.
      • test_fast_path_fresh_view_used: Fresh MV, max_stale = 300 → fast path used.
      • test_fast_path_no_max_stale_always_uses_mv: No max_stale → any MV age accepted (backward compatible).
      • test_fast_path_max_stale_zero_rejects_old_mv: max_stale = 0, MV 1 second old → slow path.
      • test_fast_path_max_stale_zero_accepts_brand_new_mv: max_stale = 0, brand new MV → fast path.

  • 2.2 AuthorityLens -> ConfidenceLens Rename: Eliminate the misleading name.

    • Problem: AuthorityLens selects by confidence field, not by agent reputation. TrustAwareAuthorityLens is the real authority lens. The name creates confusion about what "Authority" means.
    • Solution implemented:
      • Renamed authority.rs → confidence.rs, AuthorityLens → ConfidenceLens
      • Added LensDto::Confidence for the confidence-field selector
      • Changed LensDto::Authority to route to TrustAwareAuthorityLens (the real authority lens)
      • Updated query handler routing
      • Updated ai-lookup/services/lens.md and skill documentation

  • 2.3 EpochAwareLens: Give epoch supersession runtime behavior.

    • Status: ✅ COMPLETE
    • Implementation:
      • EpochAwareLens in crates/stemedb-lens/src/epoch_aware.rs
      • Decorator pattern wrapping any inner lens (default: RecencyLens)
      • Walks supersession chain from E:{epoch_id} keys
      • Cycle detection + max depth guard (100)
      • Fail-open on missing epochs
      • LensDto::EpochAware added to API
      • 11 tests: excludes_superseded, chain_supersession, no_epochs_passes_all, missing_epoch_includes, cycle_detection, consensus_lens_inner, mixed_epochs, etc.
      • Documentation updated in ai-lookup/services/lens.md
    • Known Limitation: Filtering only occurs when assertions from the superseding epoch are present in candidates. If all candidates are from old epoch (no new epoch assertions), they pass through (fail-open behavior).

  • 2.4 Visual Hash Query Support: Make the stored visual_hash queryable.

    • Status: ✅ COMPLETE
    • Implementation:
      • hamming_distance(a: &PHash, b: &PHash) -> u32 in crates/stemedb-query/src/query.rs (lines 26-28)
      • visual_near: Option<String> and visual_threshold: Option<u32> in Query struct (lines 84-90)
      • .visual_near(hash, threshold) builder method
      • Query::matches() computes hamming distance when visual_near is set
      • API QueryParams DTO has visual_near and visual_threshold
      • 10+ tests: exact_match, within_threshold, exceeds_threshold, skips_without_hash, invalid_hex, wrong_length, combines_with_subject, default_threshold, max_threshold, threshold_63_rejects
    • Note: Brute-force O(N) scan. VP-tree/BK-tree index is Phase 3+.

  • 2.5 Vector Field: No changes needed. Already roadmapped for Phase 3.

    • Status: ✅ N/A (No Phase 2 work required)
    • Current state: vector: Option<Vec<f32>> on Assertion. Stored and returned by API. No index, no search.
    • Phase 3 plan: Integrate hnsw-rs or lance for k-NN search.

  • 2.6 E2E Integration Test (Write -> Materialize -> Read): Prove the full pipeline works end-to-end.

    • Status: ✅ COMPLETE
    • Implementation:
      • crates/stemedb-query/tests/e2e_pipeline.rs with 5 comprehensive tests:
        • test_e2e_write_materialize_read - Basic happy path
        • test_e2e_vote_consensus - Vote-weighted resolution
        • test_e2e_update_winner - Winner changes on re-materialize
        • test_e2e_cursor_persistence - Cursor survives worker restart
        • test_e2e_notify_integration - Event-driven notification channel
      • stemedb-wal and stemedb-ingest added as dev-dependencies
      • Helper functions: create_signed_assertion(), compute_assertion_hash(), create_vote()
      • Uses Ed25519 signing for authentic signature verification
    • Also: crates/stemedb-api/tests/e2e_flow_test.rs tests the HTTP API layer end-to-end.

Phase 3: The Pilot (BioTech/Pharma)

Goal: Prove value in the "High-Liability" beachhead. Close every Camp 4 gap that blocks a credible demo.

3A. Schema Expansion (Prerequisite for everything below)

  • 3A.1 Source-Class Field: Add source_class: SourceClass to Assertion.

    • Status: ✅ COMPLETE
    • Implementation:
      • SourceClass enum in crates/stemedb-core/src/types.rs (lines 68-88).
      • 6-tier system: Regulatory (0), Clinical (1), Observational (2), Expert (3), Community (4), Anecdotal (5).
      • tier() method returns tier number for ordering.
      • default_decay_days() method for tier-specific confidence decay.
      • authority_weight() method for conflict resolution weighting.
      • Field on Assertion struct at line 152.
      • Full serialization and indexing support.

  • 3A.2 Conflict Score on Resolution: Add conflict_score: f32 to Resolution.

    • Problem: All 4 use cases reference conflict_score in query responses. Resolution has resolution_confidence but no conflict metric. Without a conflict score, the "disagreement is the information" thesis has no numeric representation.
    • Current state: Resolution at traits.rs:10-20 has winner, candidates_count, resolution_confidence. No variance computation.
    • Add field to Resolution in crates/stemedb-lens/src/traits.rs:10: 
      /// Degree of disagreement among candidates (0.0 = full agreement, 1.0 = max conflict).
/// Computed as normalized variance of candidate confidence values.
pub conflict_score: f32,
    • Update Resolution::empty() at traits.rs:24: set conflict_score: 0.0.
    • Update Resolution::with_winner() at traits.rs:29: accept conflict_score parameter.
    • Add utility function compute_conflict_score(candidates: &[Assertion]) -> f32:
      • Compute mean confidence: mean = sum(c.confidence) / N.
      • Compute variance: var = sum((c.confidence - mean)^2) / N.
      • Normalize to [0.0, 1.0]: conflict = (4.0 * var).min(1.0) (variance of [0,1] range has max 0.25, so 4x normalizes).
      • Edge cases: 0 or 1 candidates -> 0.0.
    • Update every Lens::resolve() and AsyncLens::resolve_async() implementation to call compute_conflict_score() and pass it to Resolution::with_winner():
      • crates/stemedb-lens/src/recency.rs
      • crates/stemedb-lens/src/consensus.rs
      • crates/stemedb-lens/src/authority.rs (or confidence.rs after rename)
      • crates/stemedb-lens/src/vote_aware_consensus.rs
      • crates/stemedb-lens/src/trust_aware_authority.rs
    • Add conflict_score: f32 to MaterializedView in crates/stemedb-core/src/types.rs:143.
    • Update Materializer::materialize_pair() at materializer.rs:164 to write conflict_score from resolution.
    • Add conflict_score to QueryResponse DTO (or a new ResolutionMeta sub-object) in crates/stemedb-api/src/dto.rs.
    • Wire conflict_score through in the query handler's apply_lens() at handlers/query.rs:111.
    • Tests:
      • test_conflict_score_zero_for_agreement: 3 candidates all confidence 0.9. Score near 0.0.
      • test_conflict_score_high_for_disagreement: Candidates at 0.1, 0.5, 0.9. Score > 0.5.
      • test_conflict_score_zero_for_single: 1 candidate. Score = 0.0.
      • test_conflict_score_on_materialized_view: MV stores conflict_score after materialization.

  • 3A.3 Rich Source Metadata: Add structured provenance beyond source_hash.

    • Problem: Assertion has source_hash: Hash (a 32-byte hash) but no structured metadata. Consumer Health use case shows POST bodies with journal, DOI, sample_size, subreddit, upvotes. Without this, assertions are opaque about their provenance.
    • Current state: source_hash: Hash at types.rs:64. No metadata field. API accepts/returns only the hash.
    • Add field to Assertion in crates/stemedb-core/src/types.rs:65 (after source_hash): 
      /// Structured source metadata as a JSON-encoded byte string.
/// Schema is domain-specific (journal info, social metrics, etc.).
pub source_metadata: Option<Vec<u8>>,
    • Use Vec<u8> (not String) for rkyv zero-copy compatibility. Callers encode/decode JSON on their side.
    • Add source_metadata: Option<Vec<u8>> to AssertionBuilder. Add .source_metadata_json(json: &str) builder method that stores json.as_bytes().to_vec().
    • Add source_metadata: Option<String> to CreateAssertionRequest DTO (JSON string in API, converted to bytes internally).
    • Add source_metadata: Option<String> to AssertionResponse DTO (bytes converted to JSON string).
    • Wire through create handler and query handler.
    • Tests:
      • Serialization roundtrip with metadata present and absent.
      • API roundtrip: POST with metadata JSON, GET returns same JSON.
    • Note: Metadata is stored but NOT indexed in Phase 3. Indexing individual metadata fields is Phase 4+.

3B. Time & Decay (Core Query Features)

  • 3B.1 Time-Travel Engine: as_of parameter for historical queries.

    • Problem: All 4 use cases reference as_of for historical state. "What was the consensus when I made my decision?" The append-only model stores all history, but there's no way to query a past state.
    • Current state: Query at query.rs:14-29 has no as_of field. try_fast_path() returns current MV regardless.
    • Add as_of: Option<u64> to Query struct in crates/stemedb-query/src/query.rs:14.
    • Add .as_of(timestamp: u64) to QueryBuilder.
    • In Query::matches() at query.rs:43: if as_of is Some(ts), check assertion.timestamp <= ts. Assertions created after as_of are excluded.
    • In QueryEngine::execute() at engine.rs:47: if query.as_of is set, skip the fast path entirely (MVs reflect current state, not historical). Add early check before the try_fast_path call: 
      if query.as_of.is_none() {
    if let (Some(subject), Some(predicate)) = (&query.subject, &query.predicate) {
        if let Some(result) = self.try_fast_path(subject, predicate, query).await? {
            return Ok(result);
        }
    }
}
    • Add as_of: Option<u64> to QueryParams DTO in crates/stemedb-api/src/dto.rs:102.
    • Wire in query handler.
    • Tests:
      • test_as_of_excludes_future_assertions: 3 assertions at t=1000, t=2000, t=3000. Query as_of=2500. Returns only first 2.
      • test_as_of_bypasses_fast_path: MV exists, but as_of is set. Slow path used, MV ignored.
      • test_as_of_none_uses_fast_path: Normal query still uses fast path (backwards-compatible).
      • test_as_of_with_lens: Time-travel + lens = resolve only among pre-as_of candidates.

  • 3B.2 Semantic Decay: Confidence Half-Life at query time.

    • Problem: Medical knowledge decays at different rates. A Reddit post from 2022 shouldn't compete equally with a 2024 RCT. No assertion-level decay exists. TrustRank has agent-level decay but not assertion-level.
    • Current state: TrustRank.apply_decay() at trust_rank_store.rs does agent decay. No assertion decay. No decay query parameter.
    • Add decay_halflife: Option<u64> to Query struct (seconds). Represents how quickly old assertions lose effective confidence.
    • Add .decay_halflife(seconds: u64) to QueryBuilder.
    • Add decay_halflife to QueryParams DTO.
    • Implementation strategy: Apply decay in QueryEngine before passing to lens. In execute(), after fetching and filtering candidates, if decay_halflife is set:
      1. Get current time (or as_of if set).
      2. For each candidate, compute age = now - assertion.timestamp.
      3. Compute effective_confidence = confidence * 2_f32.powf(-(age as f32) / (halflife as f32)).
      4. Clone the assertion with the decayed confidence.
      5. Pass decayed candidates to the lens.
    • Add helper apply_decay(assertions: &[Assertion], halflife: u64, now: u64) -> Vec<Assertion> in a new crates/stemedb-query/src/decay.rs module.
    • Source-class-aware decay (Phase 3 stretch): If assertion has source_class, use per-tier half-lives from SourceClass::default_decay_days(). This is already implemented on the enum.
    • Tests:
      • test_decay_reduces_old_assertion_confidence: Assertion 1yr old, halflife 1yr. Effective confidence ~= original * 0.5.
      • test_decay_preserves_fresh_assertions: Assertion 1hr old, halflife 1yr. Effective confidence ~= original.
      • test_decay_interacts_with_lens: Two assertions, older has higher base confidence but after decay, newer wins via RecencyLens.
      • test_source_aware_decay: Tier 0 doesn't decay. Tier 5 decays rapidly.

3C. New Lenses

  • 3C.1 Skeptic Lens: Surface disagreement, not winners. ✅ COMPLETED

    • Status: ✅ COMPLETE
    • Implementation:
      • crates/stemedb-lens/src/skeptic.rs - Full implementation.
      • AnalysisLens trait for lenses that analyze conflict instead of resolving it.
      • SkepticLens uses normalized Shannon entropy for conflict scoring.
      • Returns ConflictAnalysis with:
        • conflict_score: f32 (0.0 = unanimous, 1.0 = chaos)
        • status: ResolutionStatus (Unanimous, Agreed, Contested)
        • claims: Vec<ClaimSummary> - all claims ranked by weight
      • SkepticResolver + SkepticView in stemedb-query/src/skeptic.rs.
      • GET /v1/skeptic?subject=X&predicate=Y API endpoint.
      • Core types in stemedb-core/src/types.rs:
        • ResolutionStatus enum
        • ConflictAnalysis struct
        • ClaimSummary, SourceSummary, AgentSummary
      • Comprehensive test coverage (21 test cases).

  • 3C.2 Layered Consensus Lens: Per-source-class consensus.

    • Problem: Current lenses return a single winner. The Consumer Health use case needs per-tier consensus: "What does Tier 0 say? What does Tier 1 say? What does Tier 5 say?" This is the core differentiator.
    • Depends on: Source-Class Field (3A.1) ✅ COMPLETE.
    • New file: crates/stemedb-lens/src/layered_consensus.rs.
    • New type in crates/stemedb-lens/src/traits.rs: 
      /// Per-tier resolution result.
pub struct TierResolution {
    pub tier: u8,
    pub winner: Option<Assertion>,
    pub candidates_count: usize,
    pub conflict_score: f32,
}

/// Multi-tier resolution result.
pub struct LayeredResolution {
    /// Per-tier consensus results, ordered by tier (0 = highest authority).
    pub tiers: Vec<TierResolution>,
    /// Overall winner (highest-tier with a winner).
    pub overall_winner: Option<Assertion>,
    /// Overall conflict score (cross-tier disagreement).
    pub overall_conflict_score: f32,
}
    • LayeredConsensusLens implements a new LayeredLens trait: 
      pub trait LayeredLens: Send + Sync {
    fn resolve_layered(&self, candidates: &[Assertion]) -> LayeredResolution;
    fn name(&self) -> &'static str;
}
    • resolve_layered() logic:
      1. Group candidates by source_class (use SourceClass::tier() method).
      2. For each tier group, run ConsensusLens::resolve() to get within-tier winner.
      3. Compute per-tier conflict_score.
      4. Overall winner = winner from the highest-authority tier that has candidates (lowest tier number).
      5. Overall conflict_score = cross-tier disagreement (do tier winners agree on the same object value?).
    • Also implement standard Lens trait on LayeredConsensusLens to maintain compatibility: resolve() returns overall_winner as a regular Resolution. The richer LayeredResolution is accessible via resolve_layered().
    • Add LayeredConsensus to LensDto enum.
    • New API response type: LayeredQueryResponse with per-tier results. Wire as a variant in the query handler when lens=LayeredConsensus.
    • Export from crates/stemedb-lens/src/lib.rs.
    • Tests:
      • test_layered_single_tier: All candidates same source_class. Returns one tier result.
      • test_layered_multi_tier_agreement: Tier 0 and Tier 5 agree on same object. Low cross-tier conflict.
      • test_layered_multi_tier_disagreement: Tier 1 says "safe", Tier 5 says "dangerous". High cross-tier conflict. Overall winner from Tier 1.
      • test_layered_overall_winner_from_highest_authority: Tier 0 present -> its winner is overall winner even if Tier 5 has 1000x more assertions.

  • 3C.3 Constraints Lens: Pre-flight check for must_use/forbidden.

    • Problem: Agile Agent Team use case needs lens=constraints returning { must_use: "axios", forbidden: "requests" }. Central to "persistent learning" — agents query constraints before acting.
    • New file: crates/stemedb-lens/src/constraints.rs.
    • Design: Not a traditional lens (doesn't pick one winner from candidates). Instead, it categorizes candidates by predicate pattern:
      • Assertions with predicate matching must_use:* -> must_use list.
      • Assertions with predicate matching forbidden:* -> forbidden list.
      • Assertions with predicate matching prefer:* -> prefer list.
    • Implements Lens trait for compatibility: resolve() returns the highest-confidence must_use assertion as the "winner" (or forbidden if no must_use exists). The richer result is accessible via a dedicated method.
    • Add dedicated resolve_constraints() method: 
      pub struct ConstraintSet {
    pub must_use: Vec<Assertion>,
    pub forbidden: Vec<Assertion>,
    pub prefer: Vec<Assertion>,
}

impl ConstraintsLens {
    pub fn resolve_constraints(&self, candidates: &[Assertion]) -> ConstraintSet { ... }
}
    • Add Constraints to LensDto enum.
    • New API response type: ConstraintResponse with categorized assertions.
    • Export from crates/stemedb-lens/src/lib.rs.
    • Tests:
      • test_constraints_categorizes_by_predicate: Mixed predicates sorted into must_use/forbidden/prefer.
      • test_constraints_empty_categories: No must_use predicates -> empty must_use list.
      • test_constraints_lens_trait_picks_must_use_winner: Standard resolve() returns highest-confidence must_use.
      • test_constraints_non_constraint_predicates_ignored: Regular predicates not categorized.

3D. Epoch Enhancement

  • 3D.1 Epoch Cascade Logic (enhancement of Phase 2.5 EpochAwareLens):
    • Problem: Phase 2.5 EpochAwareLens walks the supersedes chain at query time by reading E: keys. This is O(chain_length) per query. For long supersession chains, this is expensive.
    • Depends on: Phase 2.5 EpochAwareLens (basic version).
    • In IngestWorker::process_epoch() at crates/stemedb-ingest/src/worker.rs:
      1. When an epoch with supersedes = Some(old_epoch_id) is ingested, write a marker key SUPERSEDED:{old_epoch_id} with the new epoch ID as value.
      2. Walk the chain: if old_epoch itself superseded another epoch, write SUPERSEDED:{grandparent_id} too (transitive closure).
    • Update EpochAwareLens to check SUPERSEDED:{epoch_id} keys (O(1) lookup per candidate) instead of walking the chain at query time.
    • Tests:
      • test_cascade_writes_superseded_marker: Ingest epoch B superseding A. Verify SUPERSEDED:A key exists.
      • test_cascade_transitive: A supersedes B, B supersedes C. Verify both SUPERSEDED:B and SUPERSEDED:C exist.
      • test_epoch_aware_uses_marker: Query with EpochAwareLens uses marker key, not chain walk.

3E. Similarity Search

  • 3E.1 Vector Search: Semantic k-NN queries via embeddings.

    • Current state: vector: Option<Vec<f32>> on Assertion. Stored, returned by API. No index, no search.
    • Add hnsw-rs (or lance) as a dependency in stemedb-storage/Cargo.toml.
    • New module: crates/stemedb-storage/src/vector_index.rs.
    • VectorIndex trait: insert(hash: Hash, vector: &[f32]), search(query: &[f32], k: usize) -> Vec<(Hash, f32)>.
    • Implementation backed by HNSW graph stored alongside KV data.
    • IngestWorker: if assertion has vector, insert into vector index after KV write.
    • Add vector_near: Option<Vec<f32>> and k: Option<usize> to Query struct and API QueryParams.
    • QueryEngine: if vector_near is set, use vector index for candidate retrieval instead of SP/S index.
    • Tests: insert 100 vectors, query nearest 5, verify correct neighbors.

  • 3E.2 Visual Hash Index: VP-tree or BK-tree for O(log N) visual similarity.

    • Current state: Phase 2.5 adds brute-force hamming scan. This replaces it with an indexed approach.
    • New module: crates/stemedb-storage/src/visual_index.rs.
    • BK-tree implementation over hamming distance on PHash ([u8; 8]).
    • IngestWorker: if assertion has visual_hash, insert into BK-tree.
    • QueryEngine: if visual_near is set, use BK-tree for O(log N) candidate retrieval instead of brute-force scan.
    • Tests: insert 1000 assertions with visual hashes, query similar, verify correct matches within threshold.

3F. Provenance

  • 3F.1 Citation Recall Benchmarking: Verify 100% provenance tracking.
    • Benchmark: for every assertion in the store, verify source_hash resolves to a retrievable source document (or at minimum, the hash is non-zero and unique).
    • Add GET /v1/provenance/{hash} endpoint that looks up source by hash.
    • Add POST /v1/source endpoint to store source documents by content hash.
    • Source storage at SRC:{blake3_hash} keys.

3G. API Cleanup

  • 3G.1 Document epoch supersession via existing endpoint: No new /epoch/supersede endpoint needed.
    • Current state: POST /v1/epoch already accepts supersedes field. Use cases show POST /epoch/supersede as if it's a separate endpoint.
    • Update use case docs (consumer-health-intelligence.md, glp1-living-review.md) to use POST /v1/epoch with supersedes field.
    • Add OpenAPI examples showing the supersession flow.
    • No code change. Documentation fix only.

Phase 4: The Hive (Trust & Scale)

Goal: Change tracking, metadata indexing, and the database primitives for training pipelines.

  • TrustRank Engine: Foundation for trust-based resolution.

    • TrustRankStore for per-agent reputation storage.
    • TrustAwareAuthorityLens for reputation-weighted resolution.
    • Confidence Half-Life: Implement decay calculation engine.
    • Learning loop: record_outcome() for accuracy tracking.

  • 4.1 "Since" Parameter: Change tracking for returning consumers.

    • Problem: Consumer Health shows GET /query?since=2023-10-01 returning changes_since_query with dated change entries. The "returning consumer" story: "What changed since I last looked?"
    • Depends on: Time-Travel (3B.1) and Materializer.
    • Add since: Option<u64> to Query struct and QueryBuilder in crates/stemedb-query/src/query.rs.
    • Add since to QueryParams DTO.
    • MV Changelog: Track when materialized views change.
      • New key pattern: MVC:{subject}:{predicate}:{timestamp} storing the previous winner hash and new winner hash.
      • In Materializer::materialize_pair() at materializer.rs:164: before overwriting MV, read the existing MV. If the winner changed (different assertion hash), write a changelog entry.
    • In QueryEngine: if since is set, scan MVC:{subject}:{predicate}:* for entries with timestamp > since. Return these as a changes_since list alongside the normal query result.
    • New response field on QueryResponse: 
      pub struct ChangeEntry {
    pub timestamp: u64,
    pub previous_winner_hash: Option<String>,
    pub new_winner_hash: String,
}
      Add changes_since: Option<Vec<ChangeEntry>> to QueryResponse DTO.
    • Tests:
      • test_since_returns_changes: Materialize, change winner, re-materialize. Query with since returns the change.
      • test_since_no_changes: No MV changes since timestamp. Empty changes list.
      • test_since_multiple_changes: 3 winner changes over time. All returned in order.

  • 4.2 Source Metadata Indexing (extension of 3A.3): Index key metadata fields.

    • Problem: Phase 3 stores source_metadata as an opaque blob. Phase 4 makes key fields queryable.
    • Depends on: Rich Source Metadata (3A.3).
    • Define a set of indexed metadata keys: journal, doi, platform, study_design.
    • New key pattern: SM:{field}:{value}:{assertion_hash} for metadata field indexes.
    • IngestWorker: on ingestion, if source_metadata is present, parse JSON, extract indexed fields, write index entries.
    • Add metadata field filters to QueryParams (e.g., ?source_journal=NEJM).
    • Tests: store assertions with metadata, query by journal name, verify correct filtering.

  • 4.3 Batch TrustRank Decay API: Expose scheduled decay for external orchestration.

    • Current state: decay_all_trust_ranks() exists on TrustRankStore. No API endpoint.
    • Add POST /v1/admin/decay-trust-ranks endpoint.
    • Accepts now: u64 parameter (or uses current time).
    • Returns count of decayed agents and summary stats.
    • Note: The Gardener (Camp 5.2, app layer) calls this endpoint on a schedule. The database just exposes the primitive.

Note: The following items were reclassified as Application Layer responsibilities (see tmp/ambition-vs-reality.md, Camp 5). They are not Episteme database features. They consume the Episteme API and are built by integrators or vertical-specific teams.

  • The Simulator (Training Data Pipeline) -> Camp 5.3
  • The Super Curator (Reviewer Agent swarm) -> Camp 5.4
  • Background Gardener (Cluster detection, signal processing) -> Camp 5.2
  • Agent Wallet (Key management sidecar) -> Camp 5.1

Tracking

Active Tasks

  • Phase 2.5 Hardening: Camp 2 fixes (staleness, epoch behavior, lens rename, visual query, E2E test).

Next Up (Phase 3 sequencing)

  • 3A.2 Conflict Score: Add to Resolution. Propagate through all lenses.
  • 3A.3 Rich Source Metadata: Structured provenance field.
  • 3B.1 Time-Travel: as_of parameter. Unblocks all 4 use cases.
  • 3C.2 Layered Consensus Lens: Per-tier resolution (now unblocked by 3A.1).

Recently Completed

  • SkepticLens + SkepticView (3C.1): "Trust but Verify" conflict analysis that surfaces all claims with conflict scores.
    • AnalysisLens trait for lenses that map conflict instead of resolving it.
    • SkepticLens using normalized Shannon entropy for conflict scoring.
    • SkepticResolver + SkepticView in stemedb-query.
    • GET /v1/skeptic?subject=X&predicate=Y API endpoint.
    • Types: ResolutionStatus, ConflictAnalysis, ClaimSummary, SourceSummary, AgentSummary.
  • Source-Class Field (3A.1): 6-tier SourceClass enum with authority weighting and decay rates.
    • SourceClass enum: Regulatory, Clinical, Observational, Expert, Community, Anecdotal.
    • tier(), default_decay_days(), authority_weight() methods.
    • Field on Assertion struct with full serialization support.
  • The Meter: Token bucket quota system with MeterLayer middleware (10K tokens/agent/hour).
  • Query Audit Trail: Every query logged with provenance at AUD:{query_id}. X-Agent-Id header for attribution.
  • Event-Driven Materialization: run_notified() + IngestWorker Notify integration.
  • Fast-Path MV Lookup: QueryEngine::try_fast_path() for O(1) reads.
  • Materializer: Background worker for O(1) MV reads via AsyncLens.
  • VoteAwareConsensusLens: Real vote-based consensus resolution.
  • Compound SP Index: O(1) subject+predicate lookups.
  • TrustRank System: Agent reputation with decay and learning loop.
  • API Surface: axum HTTP server with 7 endpoints + OpenAPI docs.

Blockers

  • None.

Dependency Graph

Phase 2.5 (Hardening)          Phase 3 (The Pilot)                    Phase 4 (The Hive)
========================       ========================               ==================

[2.1 MV Staleness]  ---------> [3B.1 Time-Travel] -----+
                                       |                |
[2.2 Confidence Rename] -----> (API clarity for all)    +----------> [4.1 "Since" Param]
                                                        |
[2.3 EpochAwareLens] --------> [3D.1 Epoch Cascade] ---|----------> Invalidation Cascades pillar
                                                        |
[2.4 Visual Hash Query] -----> [3E.2 Visual Hash Index] |
                                                        |
[2.6 E2E Integration] -------> (pipeline confidence)    |
                                                        |
                               [3A.1 Source-Class] ✅ --+----------> [3C.2 Layered Consensus]
                                       |                             [3B.2 Source-Aware Decay]
                                       +-----------------------------[4.2 Metadata Indexing]
                                                        |
                               [3A.2 Conflict Score] ---> (enhance Resolution)
                                                        |
                               [3A.3 Source Metadata] ---> [4.2 Metadata Indexing]
                                                        |
                               [3C.1 Skeptic Lens] ✅    (standalone, COMPLETE)
                               [3C.3 Constraints Lens]   (standalone)
                               [3E.1 Vector Search]      (standalone)
                               [3F.1 Citation Recall]     (standalone)

Critical Path for Consumer Health Demo

[3A.1 Source-Class] ✅ --> [3A.2 Conflict Score] --> [3C.2 Layered Consensus]
                                                              |
                                                              +----> CONSUMER HEALTH MVP
                                                              |
[3B.1 Time-Travel] ------------------------------------------+
                                                              |
[3A.3 Source Metadata] --------------------------------------+
                                                              |
[3C.1 Skeptic Lens] ✅ --------------------------------------+

Critical Path for Financial DD Demo

[3A.2 Conflict Score] --> [3C.1 Skeptic Lens] ✅ -------+
                                                         |
[3B.1 Time-Travel] -------------------------------------+----> FINANCIAL DD MVP
                                                         |
[2.3 EpochAwareLens] --> [3D.1 Epoch Cascade] ----------+
                                                         |
[3B.2 Semantic Decay] ----------------------------------+

Critical Path for Agile Agent Team Demo

[3C.3 Constraints Lens] (standalone) ------+
                                            |
[3B.1 Time-Travel] -----------------------+----> AGENT TEAM MVP
                                            |
[2.3 EpochAwareLens] ---------------------+
                                            |
[Query Audit (Phase 2)] ✅ ----------------+