jordan c59066949a feat: Add quickstart "Beyond Hello World" sections with Skeptic and Layered endpoints

- Add Layered() method to Go SDK for per-source-class consensus queries
- Add LayeredQueryParams, LayeredResult, TierResolution types to Go SDK
- Create conflict example demonstrating Skeptic and Layered endpoints
- Update quickstart.md with sections 6 (conflict detection) and 7 (authority tiers)
- Remove tracked Go binary and add data/ to .gitignore

The new quickstart sections demonstrate Episteme's differentiating features:
- Skeptic endpoint shows "Trust but Verify" conflict analysis
- Layered endpoint shows per-tier resolution (Clinical vs Anecdotal)

Note: Pre-existing large files flagged by pre-commit hook (technical debt from prior sessions)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

2026-02-01 21:00:59 -07:00

44 KiB

Raw Blame History

Episteme (StemeDB) Roadmap

Goal: Build the "Git for Truth" substrate for autonomous AI research. Current Phase: Phase 4 (The Hive) Target Vertical: BioTech/Pharma ("The Living Review")

High-Level Timeline

Phase	Codename	Focus	Key Deliverable
1	The Spine	Storage & Safety	Append-only WAL + KV Store
2	The Lattice	Indexing & Async	Materialized Views + Ballot Box
2.5	Hardening	Camp 2 Fixes	MV staleness, epoch behavior, lens cleanup
3	The Pilot	Vertical Integration	Pharma Ingestion + Living Review Agent
4	The Hive	Trust & Learning	The Simulator + Super Curator

Detailed Milestones

Phase 1: The Spine (Foundation)

Goal: Securely ingest assertions and persist them without data loss.

Project Scaffold: Initialize Rust workspace, set up linting/CI (clippy, fmt).
Assertion Schema: Define the Assertion struct with rkyv serialization.
- Add dependencies: rkyv, blake3, ed25519-dalek, image_hasher.
- Define Assertion struct (Subject, Predicate, Object, Confidence, SourceHash).
- Multi-Sig Expansion: Implement SignatureEntry struct and signatures: Vec<SignatureEntry> field.
- Visual Expansion: Add visual_hash: Option<pHash> field for image provenance.
- Test serialization round-trips.
Ballot Schema: Define the Vote struct for multi-agent consensus.
- Add Vote struct: assertion_hash, agent_id, weight, signature.
- Test serialization round-trips.
Paradigm Schema (Epochs): Define the Epoch and SupersessionType structs.
- Add epoch: Option<EpochId> to Assertion.
- Implement Epoch struct with supersedes and SupersessionType.
- Test serialization round-trips.
WAL Integration: Implement the Quarantine Pattern for write-ahead logging.
- Create stemedb-wal crate.
- Port FsyncGuard and Record logic from established durability patterns.
- Implement Record format with BLAKE3 checksums and Headers.
- Verify fsync behavior with tests.
Storage Engine: Implement the Store trait using sled (embedded KV).
- Add sled dependency.
- Define KVStore trait (put, get, delete, scan_prefix, flush).
- Implement SledStore wrapper.
Basic Ingestor: Background worker that tails WAL and writes to KV.
- Implement async loop reading from WAL.
- Write deserialized assertions, votes, and epochs to sled.
- Ed25519 signature verification during ingestion.
- Maintains S: and SP: indexes on ingest.
- Persistent cursor/checkpoint (resumes from __CURSOR__:ingest in KV store).
Verification: Crash recovery tests (write -> crash -> restart -> read).
- Single and multi-record crash recovery.
- Multiple crash cycles tested.

Phase 2: The Lattice (Connectivity)

Goal: Query data with sub-millisecond latency using Materialized Views.

Lifecycle Schema: Add LifecycleStage to Assertion.
- Define enum: Proposed, UnderReview, Approved, Deprecated, Rejected.
- Update Assertion struct and serialization tests.
The Ballot Box: Implement high-velocity vote ingestion.
- VoteStore trait and implementation.
- VoteAwareConsensusLens for real vote-based resolution.
Index Infrastructure: Compound indexes for O(1) queries.
- IndexStore trait with S: and SP: indexes.
- QueryEngine smart routing (SP -> S -> scan).
Materializer: Background worker for O(1) Read Performance.
- MaterializedView type in stemedb-core.
- Materializer worker in stemedb-query with step() and run().
- Aggregates Votes via VoteAwareConsensusLens (or any AsyncLens).
- Updates MV:{Subject}:{Predicate} with the winning Assertion + metadata.
- Event-driven mode via run_notified() with tokio::sync::Notify.
- Fast-path MV lookup in QueryEngine::try_fast_path().
The Meter: Implement Economic Throttling (TAN).
- QuotaStore trait and GenericQuotaStore implementation.
- Token Bucket algorithm with per-agent per-hour quotas.
- MeterLayer tower middleware for request cost tracking.
- Cost model: Assert=10, Vote=1, Query=5+lens, +1/KB payload.
- GET /v1/meter/quota endpoint to check remaining quota.
- POST /v1/meter/quota/limit admin endpoint to set custom limits.
API Surface: axum HTTP server with OpenAPI (utoipa).
- POST /v1/assert -> Accepts JSON, writes to WAL.
- POST /v1/vote -> High-throughput vote endpoint.
- POST /v1/epoch -> Create epoch with optional supersession.
- GET /v1/query -> Subject/Predicate/Lens/Lifecycle/Epoch filtering.
- GET /v1/health -> Health check with assertion count.
- GET /swagger-ui -> Interactive API docs.
- 5 lens types available: Recency, Consensus, Authority, VoteAwareConsensus, TrustAwareAuthority.
Query Audit: Log every read with provenance.
- Define QueryAudit struct: query_id, agent_id, timestamp, params, result_hash, contributing_assertions.
- Storage at AUD:{query_id} with agent index at AUDA:{agent_id}:{timestamp}:{query_id}.
- GET /v1/audit/queries -> Returns history of agent decisions.
- GET /v1/audit/query/{id} -> Full reasoning trace for a single query.
- Auto-logging on every query via X-Agent-Id header.

Phase 2.5: Hardening (Camp 2 Fixes)

Goal: Close the gaps between "built" and "works right." Every item here addresses a feature that exists but doesn't fully deliver on its promise.

2.1 MV Staleness Detection: Make the fast-path aware of stale materialized views.
- Status: ✅ COMPLETE
- Implementation:
  - Added max_stale: Option<u64> to Query struct in crates/stemedb-query/src/query.rs.
  - Added .max_stale(secs) builder method to QueryBuilder.
  - In try_fast_path(): if query.max_stale is set and MV age exceeds threshold, falls through to slow path with debug! log.
  - Added max_stale to API QueryParams DTO in crates/stemedb-api/src/dto.rs.
  - Wired through query handler in crates/stemedb-api/src/handlers/query.rs.
- Tests:
  - test_fast_path_stale_view_falls_back: MV 1000 seconds old, max_stale = 60 → slow path used.
  - test_fast_path_fresh_view_used: Fresh MV, max_stale = 300 → fast path used.
  - test_fast_path_no_max_stale_always_uses_mv: No max_stale → any MV age accepted (backward compatible).
  - test_fast_path_max_stale_zero_rejects_old_mv: max_stale = 0, MV 1 second old → slow path.
  - test_fast_path_max_stale_zero_accepts_brand_new_mv: max_stale = 0, brand new MV → fast path.
2.2 AuthorityLens -> ConfidenceLens Rename: Eliminate the misleading name.
- Problem: AuthorityLens selects by confidence field, not by agent reputation. TrustAwareAuthorityLens is the real authority lens. The name creates confusion about what "Authority" means.
- Solution implemented:
  - Renamed authority.rs → confidence.rs, AuthorityLens → ConfidenceLens
  - Added LensDto::Confidence for the confidence-field selector
  - Changed LensDto::Authority to route to TrustAwareAuthorityLens (the real authority lens)
  - Updated query handler routing
  - Updated ai-lookup/services/lens.md and skill documentation
2.3 EpochAwareLens: Give epoch supersession runtime behavior.
- Status: ✅ COMPLETE
- Implementation:
  - EpochAwareLens in crates/stemedb-lens/src/epoch_aware.rs
  - Decorator pattern wrapping any inner lens (default: RecencyLens)
  - Walks supersession chain from E:{epoch_id} keys
  - Cycle detection + max depth guard (100)
  - Fail-open on missing epochs
  - LensDto::EpochAware added to API
  - 11 tests: excludes_superseded, chain_supersession, no_epochs_passes_all, missing_epoch_includes, cycle_detection, consensus_lens_inner, mixed_epochs, etc.
  - Documentation updated in ai-lookup/services/lens.md
- Known Limitation: Filtering only occurs when assertions from the superseding epoch are present in candidates. If all candidates are from old epoch (no new epoch assertions), they pass through (fail-open behavior).
2.4 Visual Hash Query Support: Make the stored visual_hash queryable.
- Status: ✅ COMPLETE
- Implementation:
  - hamming_distance(a: &PHash, b: &PHash) -> u32 in crates/stemedb-query/src/query.rs (lines 26-28)
  - visual_near: Option<String> and visual_threshold: Option<u32> in Query struct (lines 84-90)
  - .visual_near(hash, threshold) builder method
  - Query::matches() computes hamming distance when visual_near is set
  - API QueryParams DTO has visual_near and visual_threshold
  - 10+ tests: exact_match, within_threshold, exceeds_threshold, skips_without_hash, invalid_hex, wrong_length, combines_with_subject, default_threshold, max_threshold, threshold_63_rejects
- Note: Brute-force O(N) scan. VP-tree/BK-tree index is Phase 3+.
2.5 Vector Field: No changes needed. Already roadmapped for Phase 3.
- Status: ✅ N/A (No Phase 2 work required)
- Current state: vector: Option<Vec<f32>> on Assertion. Stored and returned by API. No index, no search.
- Phase 3 plan: Integrate hnsw-rs or lance for k-NN search.
2.6 E2E Integration Test (Write -> Materialize -> Read): Prove the full pipeline works end-to-end.
- Status: ✅ COMPLETE
- Implementation:
  - crates/stemedb-query/tests/e2e_pipeline.rs with 5 comprehensive tests:
    - test_e2e_write_materialize_read - Basic happy path
    - test_e2e_vote_consensus - Vote-weighted resolution
    - test_e2e_update_winner - Winner changes on re-materialize
    - test_e2e_cursor_persistence - Cursor survives worker restart
    - test_e2e_notify_integration - Event-driven notification channel
  - stemedb-wal and stemedb-ingest added as dev-dependencies
  - Helper functions: create_signed_assertion(), compute_assertion_hash(), create_vote()
  - Uses Ed25519 signing for authentic signature verification
- Also: crates/stemedb-api/tests/e2e_flow_test.rs tests the HTTP API layer end-to-end.

Phase 3: The Pilot (BioTech/Pharma)

Goal: Prove value in the "High-Liability" beachhead. Close every Camp 4 gap that blocks a credible demo.

3A. Schema Expansion (Prerequisite for everything below)

3A.1 Source-Class Field: Add source_class: SourceClass to Assertion.
- Status: ✅ COMPLETE
- Implementation:
  - SourceClass enum in crates/stemedb-core/src/types.rs (lines 68-88).
  - 6-tier system: Regulatory (0), Clinical (1), Observational (2), Expert (3), Community (4), Anecdotal (5).
  - tier() method returns tier number for ordering.
  - default_decay_days() method for tier-specific confidence decay.
  - authority_weight() method for conflict resolution weighting.
  - Field on Assertion struct at line 152.
  - Full serialization and indexing support.
3A.2 Conflict Score on Resolution: Add conflict_score: f32 to Resolution.
- Status: ✅ COMPLETE
- Implementation:
  - Added conflict_score: f32 field to Resolution in crates/stemedb-lens/src/traits.rs.
  - Updated Resolution::empty() to set conflict_score: 0.0.
  - Updated Resolution::with_winner() to accept conflict_score parameter.
  - Added compute_conflict_score(candidates: &[Assertion]) -> f32 utility function:
    - Uses normalized variance of confidence values.
    - 0 or 1 candidates → 0.0 (no conflict possible).
    - All same confidence → 0.0 (unanimous).
    - Max variance (0.0 vs 1.0) → 1.0 (maximum conflict).
    - Defensive NaN handling (returns 0.0 for malformed data).
  - Updated all lens implementations to compute and pass conflict score:
    - crates/stemedb-lens/src/recency.rs
    - crates/stemedb-lens/src/consensus.rs
    - crates/stemedb-lens/src/confidence.rs
    - crates/stemedb-lens/src/vote_aware_consensus.rs
    - crates/stemedb-lens/src/trust_aware_authority.rs
  - Added conflict_score: f32 to MaterializedView in crates/stemedb-core/src/types.rs.
  - Updated Materializer::materialize_pair() to write conflict_score from resolution.
  - Added conflict_score: Option<f32> and resolution_confidence: Option<f32> to QueryResponse DTO in crates/stemedb-api/src/dto.rs (only present when lens is applied).
  - Wired through query handler in crates/stemedb-api/src/handlers/query.rs.
- Tests:
  - test_conflict_score_zero_for_empty: Empty candidates → 0.0.
  - test_conflict_score_zero_for_single: 1 candidate → 0.0.
  - test_conflict_score_zero_for_agreement: All same confidence → near 0.0.
  - test_conflict_score_high_for_disagreement: Candidates at 0.1, 0.5, 0.9 → score > 0.3.
  - test_conflict_score_max_for_extremes: 0.0 vs 1.0 → score ≈ 1.0.
  - test_conflict_score_handles_nan_defensively: NaN confidences → 0.0 (fail-safe).
3A.3 Rich Source Metadata: Add structured provenance beyond source_hash.
- Status: ✅ COMPLETE
- Implementation:
  - Added source_metadata: Option<Vec<u8>> field to Assertion in crates/stemedb-core/src/types.rs (after epoch, before lifecycle).
  - Uses Vec<u8> (not String) for rkyv zero-copy compatibility. Callers encode/decode JSON on their side.
  - Added source_metadata: Option<Vec<u8>> to AssertionBuilder in crates/stemedb-core/src/testing.rs.
  - Added .source_metadata_json(json: &str) and .source_metadata(bytes) builder methods.
  - Added source_metadata: Option<String> to CreateAssertionRequest DTO (JSON string in API, converted to bytes internally).
  - Added source_metadata: Option<String> to AssertionResponse DTO (bytes converted to JSON string with defensive UTF-8 handling).
  - Wired through create handler (dto_to_assertion()) and query handler (assertion_to_dto()).
- Tests:
  - test_serialize_deserialize_assertion_with_metadata: Serialization roundtrip with metadata present.
  - test_serialize_deserialize_assertion_without_metadata: Serialization roundtrip with metadata absent.
- Note: Metadata is stored but NOT indexed in Phase 3. Indexing individual metadata fields is Phase 4+.

3B. Time & Decay (Core Query Features)

3B.1 Time-Travel Engine: as_of parameter for historical queries.
- Status: ✅ COMPLETE
- Implementation:
  - Added as_of: Option<u64> to Query struct in crates/stemedb-query/src/query.rs:92-99.
  - Added .as_of(timestamp: u64) to QueryBuilder.
  - In Query::matches(): if as_of is Some(ts), check assertion.timestamp <= ts. Assertions created after as_of are excluded.
  - In QueryEngine::execute(): if query.as_of is set, skip the fast path entirely (MVs reflect current state, not historical).
  - Added as_of: Option<u64> to QueryParams DTO in crates/stemedb-api/src/dto.rs.
  - Wired through query handler.
- Tests:
  - test_as_of_excludes_future_assertions: Assertions filtered by timestamp.
  - test_as_of_bypasses_fast_path: MV exists, but as_of is set. Slow path used.
  - test_as_of_none_uses_fast_path: Normal query still uses fast path (backwards-compatible).
  - test_as_of_with_lens_resolves_among_historical_candidates: Time-travel + lens = resolve only among pre-as_of candidates.
  - test_as_of_returns_empty_when_all_assertions_are_future: All assertions are future, returns empty.
  - test_as_of_with_exact_timestamp_match: Edge case where assertion.timestamp == as_of.
3B.2 Semantic Decay: Confidence Half-Life at query time.
- Status: ✅ COMPLETE
- Implementation:
  - Added decay_halflife: Option<u64> to Query struct in crates/stemedb-query/src/query.rs.
  - Added .decay_halflife(seconds: u64) and .source_class_decay(enabled: bool) to QueryBuilder.
  - Added decay_halflife and source_class_decay to QueryParams DTO.
  - Created new crates/stemedb-query/src/decay.rs module with:
    - apply_decay(): Uniform decay using formula confidence * 2^(-(age / halflife)).
    - apply_source_class_decay(): Tier-specific decay (Regulatory=none, Clinical=2yr, Anecdotal=30d).
    - compute_decayed_confidence(): Core decay calculation with clamping.
  - Integrated in QueryEngine::execute(): decay applied after filtering, before lens resolution.
  - Time-travel compatible: uses as_of timestamp if set, otherwise current time.
  - Source-class-aware decay fully implemented using SourceClass::default_decay_days().
- Tests: (11 unit tests in decay.rs + 1 E2E test)
  - test_decay_reduces_old_assertion_confidence: 1yr old, 1yr halflife → ~50% confidence.
  - test_decay_preserves_fresh_assertions: 1hr old, 1yr halflife → ~100% confidence.
  - test_decay_interacts_with_lens: Older high-confidence loses to newer low-confidence after decay.
  - test_source_aware_decay_tier0_no_decay: Regulatory never decays.
  - test_source_aware_decay_tier5_rapid_decay: Anecdotal decays rapidly (30-day halflife).
  - test_source_aware_decay_mixed_tiers: Clinical vs Anecdotal tier comparison.
  - test_decay_zero_halflife_no_change: Zero halflife skips decay (avoids div-by-zero).
  - test_decay_future_assertion_no_change: Future assertions don't decay.
  - test_decay_empty_assertions: Empty input returns empty output.
  - test_decay_confidence_clamps_to_valid_range: Very old assertions clamp to [0.0, 1.0].
  - test_decay_preserves_other_fields: Only confidence changes; other fields preserved.
  - test_e2e_decay_reduces_old_confidence: Full pipeline E2E test in e2e_pipeline.rs.
- Note: When decay is enabled, materialized views (fast path) are bypassed because MVs store pre-computed winners without decay applied.

3C. New Lenses

3C.1 Skeptic Lens: Surface disagreement, not winners. ✅ COMPLETED
- Status: ✅ COMPLETE
- Implementation:
  - crates/stemedb-lens/src/skeptic.rs - Full implementation.
  - AnalysisLens trait for lenses that analyze conflict instead of resolving it.
  - SkepticLens uses normalized Shannon entropy for conflict scoring.
  - Returns ConflictAnalysis with:
    - conflict_score: f32 (0.0 = unanimous, 1.0 = chaos)
    - status: ResolutionStatus (Unanimous, Agreed, Contested)
    - claims: Vec<ClaimSummary> - all claims ranked by weight
  - SkepticResolver + SkepticView in stemedb-query/src/skeptic.rs.
  - GET /v1/skeptic?subject=X&predicate=Y API endpoint.
  - Core types in stemedb-core/src/types.rs:
    - ResolutionStatus enum
    - ConflictAnalysis struct
    - ClaimSummary, SourceSummary, AgentSummary
  - Comprehensive test coverage (21 test cases).
3C.2 Layered Consensus Lens: Per-source-class consensus.
- Status: ✅ COMPLETE
- Implementation:
  - crates/stemedb-lens/src/layered_consensus.rs - Full implementation.
  - TierResolution struct: per-tier result with tier, source_class, winner, candidates_count, conflict_score, resolution_confidence.
  - LayeredResolution struct: multi-tier result with tiers vec, overall_winner, overall_conflict_score, total_candidates.
  - LayeredLens trait: resolve_layered(&[Assertion]) -> LayeredResolution, name() -> &'static str.
  - LayeredConsensusLens implements both LayeredLens and Lens traits.
  - Cross-tier conflict score uses normalized Shannon entropy of tier winner object values.
  - LensDto::LayeredConsensus variant (redirects to /v1/layered endpoint).
  - GET /v1/layered?subject=X&predicate=Y API endpoint with LayeredQueryResponse.
  - Exported from crates/stemedb-lens/src/lib.rs.
- Tests:
  - test_layered_empty_candidates: Empty input returns empty resolution.
  - test_layered_single_tier: All same source_class, returns one tier result.
  - test_layered_multi_tier_agreement: Tier 0 and Tier 5 agree, low cross-tier conflict.
  - test_layered_multi_tier_disagreement: Tier 1 vs Tier 5 disagree, high conflict, Tier 1 wins.
  - test_layered_overall_winner_from_highest_authority: Tier 0 wins despite fewer assertions.
  - test_layered_lens_trait_compatibility: Standard Lens trait works.
  - test_layered_within_tier_conflict: High internal conflict within a tier.
  - test_layered_all_tiers_present: One assertion from each tier.
  - test_layered_lens_name: Both trait names work.
  - test_layered_numeric_values: Works with numeric object values.
3C.3 Constraints Lens: Pre-flight check for must_use/forbidden.
- Status: ✅ COMPLETE
- Implementation:
  - crates/stemedb-lens/src/constraints.rs - Full implementation.
  - ConstraintSet struct: holds categorized assertions (must_use, forbidden, prefer) with conflict_score.
  - ConstraintsLens struct with resolve_constraints(&[Assertion]) -> ConstraintSet method.
  - Categorizes by predicate pattern: must_use:*, forbidden:*, prefer:*.
  - Implements Lens trait for compatibility (priority: must_use > forbidden > prefer).
  - Sorted by confidence (highest first), with timestamp as tiebreaker.
  - LensDto::Constraints added (redirects to /v1/constraints endpoint).
  - GET /v1/constraints?subject=X API endpoint with ConstraintsResponse.
  - DTOs: ConstraintsQueryParams, ConstraintEntryDto, ConstraintsResponse.
  - Exported from crates/stemedb-lens/src/lib.rs.
- Tests: (16 test cases)
  - test_constraints_categorizes_by_predicate: Mixed predicates sorted into must_use/forbidden/prefer.
  - test_constraints_empty_categories: Only prefer, no must_use/forbidden.
  - test_constraints_non_constraint_predicates_ignored: Regular predicates filtered out.
  - test_constraints_sorted_by_confidence: Within-category confidence ordering.
  - test_constraints_empty_candidates: Empty input returns empty set.
  - test_constraints_has_constraints_true: Helper method works.
  - test_constraints_all_regular_predicates: All non-constraint predicates returns no constraints.
  - test_lens_trait_picks_must_use_winner: Standard Lens trait picks must_use first.
  - test_lens_trait_falls_back_to_forbidden: Falls back to forbidden when no must_use.
  - test_lens_trait_falls_back_to_prefer: Falls back to prefer when no must_use/forbidden.
  - test_lens_trait_empty_for_no_constraints: Returns empty when no constraint predicates.
  - test_lens_name: Name returns "Constraints".
  - test_lens_empty_candidates: Empty input to Lens trait returns empty resolution.
  - test_multiple_must_use_picks_highest_confidence: Multiple must_use picks highest confidence.
  - test_confidence_tiebreaker_uses_timestamp: Same confidence uses newer timestamp.
  - test_predicate_pattern_exact_prefix: must_use_something not matched (only must_use:*).

3D. Epoch Enhancement

3D.1 Epoch Cascade Logic (enhancement of Phase 2.5 EpochAwareLens):
- Status: ✅ COMPLETE
- Implementation:
  - write_supersession_cascade() in crates/stemedb-ingest/src/worker.rs:
    - Writes SUPERSEDED:{old_epoch_id} markers for full transitive closure.
    - All markers point to the LATEST superseding epoch.
    - Max depth guard (100 levels) and cycle detection via visited set.
  - is_epoch_superseded() in crates/stemedb-lens/src/epoch_aware.rs:
    - O(1) marker lookup instead of O(chain_length) chain walks.
    - Fail-open semantics: missing marker = not superseded.
  - compute_superseded_epochs() uses marker lookups for filtering.
- Tests:
  - test_cascade_writes_superseded_marker: Epoch B supersedes A → SUPERSEDED:A exists.
  - test_cascade_transitive: C→B→A chain → both SUPERSEDED:A and SUPERSEDED:B point to C.
  - test_cascade_cycle_detection: Mutual supersession handled gracefully.
  - test_epoch_aware_uses_marker: EpochAwareLens uses O(1) marker lookup.
  - test_superseded_epoch_filtered_even_without_new_assertions: Marker-based filtering works.

3E. Similarity Search

3E.1 Vector Search: Semantic k-NN queries via embeddings.
- Status: ✅ COMPLETE
- Implementation:
  - Added hnsw_rs = "0.3" and parking_lot = "0.12" to stemedb-storage/Cargo.toml.
  - New module: crates/stemedb-storage/src/vector_index.rs.
  - VectorIndex trait: insert(hash: &Hash, vector: &[f32]), search(query: &[f32], k: usize) -> Vec<(Hash, f32)>, dimension(), len(), is_empty().
  - HnswVectorIndex implementation with HNSW graph, RwLock protection, hash↔ID mappings.
  - Input validation: dimension mismatch, NaN, Infinite values rejected.
  - Idempotent insert (same hash twice = no-op).
  - Arc<dyn VectorIndex> trait object support for sharing.
  - IngestWorker::with_vector_index() builder method for index attachment.
  - IngestWorker: if assertion has vector, inserts into vector index after KV write.
  - Added vector_near: Option<Vec<f32>> and k: Option<usize> to Query struct in crates/stemedb-query/src/query.rs.
  - Added .vector_near(vector, k) builder method to QueryBuilder.
  - Added vector_near and k to API QueryParams DTO.
  - QueryEngine::with_vector_index() builder method for index attachment.
  - QueryEngine: if vector_near is set and index configured, uses O(log N) HNSW lookup for candidates.
  - Falls back to standard path if no index configured (with debug log).
- Tests: (12 unit tests for VectorIndex + 4 integration tests in engine.rs)
  - test_create_index, test_insert_and_search, test_dimension_mismatch.
  - test_idempotent_insert, test_search_empty_index, test_search_k_zero.
  - test_nan_rejection, test_infinite_rejection, test_contains.
  - test_larger_scale (100 vectors, exact match first), test_custom_params, test_zero_dimension_panics.
  - test_vector_search_returns_nearest_neighbors, test_vector_search_with_subject_filter.
  - test_vector_search_without_index_falls_back, test_vector_search_with_as_of_filter.
- Note: Index is in-memory only. Persistence is Phase 4+.
3E.2 Visual Hash Index: BK-tree for O(log N) visual similarity.
- Status: ✅ COMPLETE
- Implementation:
  - New module: crates/stemedb-storage/src/visual_index.rs.
  - VisualIndex trait: insert(hash: &Hash, phash: &PHash), search(query: &PHash, threshold: u32) -> Vec<(Hash, u32)>, len(), is_empty().
  - BkTreeVisualIndex implementation using BK-tree over hamming distance.
  - hamming_distance(a: &PHash, b: &PHash) -> u32 utility function.
  - Threshold clamped to 0-64 range (max 64 bits).
  - Results sorted by distance ascending.
  - Idempotent insert (same hash twice = no-op).
  - Arc<dyn VisualIndex> trait object support for sharing.
  - IngestWorker::with_visual_index() builder method for index attachment.
  - IngestWorker: if assertion has visual_hash, inserts into BK-tree after KV write.
  - QueryEngine::with_visual_index() builder method for index attachment.
  - QueryEngine: if visual_near is set and index configured, uses O(log N) BK-tree lookup.
  - Falls back to brute-force scan (via query.matches()) if no index configured.
  - Invalid hex input returns QueryError::InvalidInput with clear message.
- Tests: (14 unit tests for VisualIndex + 6 integration tests in engine.rs)
  - test_hamming_distance_zero, test_hamming_distance_max, test_hamming_distance_partial.
  - test_create_index, test_insert_and_search_exact, test_search_within_threshold.
  - test_search_no_matches, test_search_empty_index, test_idempotent_insert.
  - test_contains, test_results_sorted_by_distance, test_threshold_clamped_to_64.
  - test_larger_scale (1000 hashes), test_default_impl.
  - test_visual_search_returns_similar_images, test_visual_search_with_lifecycle_filter.
  - test_visual_search_invalid_hex_returns_error, test_visual_search_without_index_uses_brute_force.
  - test_visual_search_with_limit, test_vector_search_empty_index.
- Note: Index is in-memory only. Persistence is Phase 4+.

3F. Provenance

3F.1 Source Document Storage & Provenance Lookup: Enable 100% citation recall.
- Status: ✅ COMPLETE
- Implementation:
  - POST /v1/source endpoint to store source documents by BLAKE3 content hash.
  - GET /v1/provenance/{hash} endpoint to retrieve source documents by hash.
  - Source storage at SRC:{hash} keys with format: [content_type_len:4][content_type][content].
  - Base64 encoding for binary-safe JSON transport.
  - 10MB size limit per document.
  - Content-addressed storage: same content → same hash (idempotent uploads).
  - DTOs: StoreSourceRequest, StoreSourceResponse, ProvenanceResponse.
  - OpenAPI documentation under "provenance" tag.
- Tests: (5 test cases)
  - test_store_and_retrieve_source: Happy path store + retrieve.
  - test_store_source_invalid_base64: Bad base64 → 400.
  - test_get_provenance_not_found: Unknown hash → 404.
  - test_get_provenance_invalid_hash: Bad hash format → 400.
  - test_store_source_idempotent: Same content twice → same hash.
- Note: Benchmark utility for verifying all assertions have retrievable sources is future work.

3G. API Cleanup

3G.1 Document epoch supersession via existing endpoint: No new /epoch/supersede endpoint needed.
- Status: ✅ COMPLETE
- Implementation:
  - Updated use case docs (consumer-health-intelligence.md, glp1-living-review.md) to use POST /v1/epoch with supersedes field.
  - Added OpenAPI examples showing both new epoch and supersession flows in handlers/epoch.rs.
  - Documented all 5 supersession types: Invalidate, Temporal, Refinement, RequiresReview, Additive.
- No code change. Documentation fix only.

Phase 4: The Hive (Trust & Scale)

Goal: Change tracking, metadata indexing, and the database primitives for training pipelines.

TrustRank Engine: Foundation for trust-based resolution.
- TrustRankStore for per-agent reputation storage.
- TrustAwareAuthorityLens for reputation-weighted resolution.
- Confidence Half-Life: Implement decay calculation engine.
- Learning loop: record_outcome() for accuracy tracking.
4.1 "Since" Parameter: Change tracking for returning consumers.
- Problem: Consumer Health shows GET /query?since=2023-10-01 returning changes_since_query with dated change entries. The "returning consumer" story: "What changed since I last looked?"
- Depends on: Time-Travel (3B.1) and Materializer.
- Add since: Option<u64> to Query struct and QueryBuilder in crates/stemedb-query/src/query.rs.
- Add since to QueryParams DTO.
- MV Changelog: Track when materialized views change.
  - New key pattern: MVC:{subject}:{predicate}:{timestamp} storing the previous winner hash and new winner hash.
  - In Materializer::materialize_pair() at materializer.rs:164: before overwriting MV, read the existing MV. If the winner changed (different assertion hash), write a changelog entry.
- In QueryEngine: if since is set, scan MVC:{subject}:{predicate}:* for entries with timestamp > since. Return these as a changes_since list alongside the normal query result.
- New response field on QueryResponse:
```
pub struct ChangeEntry {
    pub timestamp: u64,
    pub previous_winner_hash: Option<String>,
    pub new_winner_hash: String,
}
```
  Add changes_since: Option<Vec<ChangeEntry>> to QueryResponse DTO.
- Tests:
  - test_since_returns_changes: Materialize, change winner, re-materialize. Query with since returns the change.
  - test_since_no_changes: No MV changes since timestamp. Empty changes list.
  - test_since_multiple_changes: 3 winner changes over time. All returned in order.
4.2 Source Metadata Indexing (extension of 3A.3): Index key metadata fields.
- Problem: Phase 3 stores source_metadata as an opaque blob. Phase 4 makes key fields queryable.
- Depends on: Rich Source Metadata (3A.3).
- Define a set of indexed metadata keys: journal, doi, platform, study_design.
- New key pattern: SM:{field}:{value}:{assertion_hash} for metadata field indexes.
- IngestWorker: on ingestion, if source_metadata is present, parse JSON, extract indexed fields, write index entries.
- Add metadata field filters to QueryParams (e.g., ?source_journal=NEJM).
- Tests: store assertions with metadata, query by journal name, verify correct filtering.
4.3 Batch TrustRank Decay API: Expose scheduled decay for external orchestration.
- Current state: decay_all_trust_ranks() exists on TrustRankStore. No API endpoint.
- Add POST /v1/admin/decay-trust-ranks endpoint.
- Accepts now: u64 parameter (or uses current time).
- Returns count of decayed agents and summary stats.
- Note: The Gardener (Camp 5.2, app layer) calls this endpoint on a schedule. The database just exposes the primitive.

Note: The following items were reclassified as Application Layer responsibilities (see tmp/ambition-vs-reality.md, Camp 5). They are not Episteme database features. They consume the Episteme API and are built by integrators or vertical-specific teams.

The Simulator (Training Data Pipeline) -> Camp 5.3

The Super Curator (Reviewer Agent swarm) -> Camp 5.4

Background Gardener (Cluster detection, signal processing) -> Camp 5.2

Agent Wallet (Key management sidecar) -> Camp 5.1

Tracking

Active Tasks

Phase 3 The Pilot: Consumer Health vertical integration. ✅ COMPLETE
Phase 4 The Hive: Trust & Scale features.

Next Up (Phase 3 sequencing)

Phase 4 Items: "Since" parameter (4.1), Metadata indexing (4.2), Batch TrustRank decay (4.3).

Recently Completed

Source Document Storage (3F.1): Provenance lookup for 100% citation recall.
- POST /v1/source stores source documents by BLAKE3 content hash.
- GET /v1/provenance/{hash} retrieves source documents.
- Content-addressed storage at SRC:{hash} keys.
- Base64 encoding, 10MB limit, idempotent uploads.
- 5 unit tests covering happy path and error cases.
Epoch Cascade Logic (3D.1): O(1) supersession lookup via pre-computed markers.
- write_supersession_cascade() writes SUPERSEDED: markers for full transitive closure at ingest time.
- is_epoch_superseded() uses O(1) marker lookup instead of chain walking.
- Cycle detection and max depth guard (100 levels).
- 5 tests covering markers, transitive closure, cycles, and marker-based filtering.
Semantic Decay (3B.2): Confidence half-life at query time.
- decay_halflife: Option<u64> and source_class_decay: bool on Query.
- New decay.rs module with apply_decay() and apply_source_class_decay().
- Formula: effective_confidence = confidence * 2^(-(age / halflife)).
- Tier-specific decay: Regulatory=none, Clinical=2yr, Anecdotal=30d.
- 11 unit tests + 1 E2E integration test.
Layered Consensus Lens (3C.2): Per-source-class consensus with tier-by-tier visibility.
- LayeredConsensusLens with LayeredLens trait.
- TierResolution and LayeredResolution types.
- GET /v1/layered?subject=X&predicate=Y endpoint.
- Cross-tier conflict score using Shannon entropy.
- 10 comprehensive tests.
Time-Travel Engine (3B.1): as_of parameter for historical queries.
- as_of: Option<u64> field on Query for querying historical state.
- Bypasses fast path (MVs reflect current state).
- Query::matches() filters by assertion.timestamp <= as_of.
- 6 tests covering edge cases.
Rich Source Metadata (3A.3): Structured provenance beyond source_hash.
- source_metadata: Option<Vec<u8>> field on Assertion.
- Vec<u8> for rkyv zero-copy compatibility, callers handle JSON encoding.
- Builder methods: .source_metadata_json() and .source_metadata().
- API exposes as Option<String> with defensive UTF-8 handling.
- 2 serialization tests.
Conflict Score on Resolution (3A.2): Numeric disagreement metric across all lenses.
- conflict_score: f32 field on Resolution (0.0 = unanimous, 1.0 = max conflict).
- compute_conflict_score() utility using normalized variance.
- Updated all 5 lens implementations to compute and propagate conflict score.
- MaterializedView now stores conflict score.
- API QueryResponse exposes conflict_score and resolution_confidence when lens is applied.
- 7 unit tests including NaN handling.
SkepticLens + SkepticView (3C.1): "Trust but Verify" conflict analysis that surfaces all claims with conflict scores.
- AnalysisLens trait for lenses that map conflict instead of resolving it.
- SkepticLens using normalized Shannon entropy for conflict scoring.
- SkepticResolver + SkepticView in stemedb-query.
- GET /v1/skeptic?subject=X&predicate=Y API endpoint.
- Types: ResolutionStatus, ConflictAnalysis, ClaimSummary, SourceSummary, AgentSummary.
Source-Class Field (3A.1): 6-tier SourceClass enum with authority weighting and decay rates.
- SourceClass enum: Regulatory, Clinical, Observational, Expert, Community, Anecdotal.
- tier(), default_decay_days(), authority_weight() methods.
- Field on Assertion struct with full serialization support.
The Meter: Token bucket quota system with MeterLayer middleware (10K tokens/agent/hour).
Query Audit Trail: Every query logged with provenance at AUD:{query_id}. X-Agent-Id header for attribution.
Event-Driven Materialization: run_notified() + IngestWorker Notify integration.
Fast-Path MV Lookup: QueryEngine::try_fast_path() for O(1) reads.
Materializer: Background worker for O(1) MV reads via AsyncLens.
VoteAwareConsensusLens: Real vote-based consensus resolution.
Compound SP Index: O(1) subject+predicate lookups.
TrustRank System: Agent reputation with decay and learning loop.
API Surface: axum HTTP server with 7 endpoints + OpenAPI docs.

Blockers

None.

Dependency Graph

Phase 2.5 (Hardening)          Phase 3 (The Pilot)                    Phase 4 (The Hive)
========================       ========================               ==================

[2.1 MV Staleness]  ---------> [3B.1 Time-Travel] ✅ --+
                                       |                |
[2.2 Confidence Rename] -----> (API clarity for all)    +----------> [4.1 "Since" Param]
                                                        |
[2.3 EpochAwareLens] --------> [3D.1 Epoch Cascade] ✅ |----------> Invalidation Cascades pillar
                                                        |
[2.4 Visual Hash Query] -----> [3E.2 Visual Hash Index] ✅
                                                        |
[2.6 E2E Integration] -------> (pipeline confidence)    |
                                                        |
                               [3A.1 Source-Class] ✅ --+----------> [3C.2 Layered Consensus] ✅
                                       |                             [3B.2 Semantic Decay] ✅
                                       +-----------------------------[4.2 Metadata Indexing]
                                                        |
                               [3A.2 Conflict Score] ✅ --> (enhance Resolution)
                                                        |
                               [3A.3 Source Metadata] ✅ --> [4.2 Metadata Indexing]
                                                        |
                               [3C.1 Skeptic Lens] ✅    (standalone, COMPLETE)
                               [3C.3 Constraints Lens] ✅ (standalone, COMPLETE)
                               [3E.1 Vector Search] ✅    (standalone, COMPLETE)
                               [3E.2 Visual Hash Index] ✅ (standalone, COMPLETE)
                               [3F.1 Provenance] ✅       (standalone, COMPLETE)

Critical Path for Consumer Health Demo

[3A.1 Source-Class] ✅ --> [3A.2 Conflict Score] ✅ --> [3C.2 Layered Consensus] ✅
                                                              |
                                                              +----> CONSUMER HEALTH MVP ✅
                                                              |
[3B.1 Time-Travel] ✅ ---------------------------------------+
                                                              |
[3A.3 Source Metadata] ✅ -----------------------------------+
                                                              |
[3C.1 Skeptic Lens] ✅ --------------------------------------+

Critical Path for Financial DD Demo

[3A.2 Conflict Score] ✅ --> [3C.1 Skeptic Lens] ✅ ---+
                                                         |
[3B.1 Time-Travel] ✅ ----------------------------------+----> FINANCIAL DD MVP
                                                         |
[2.3 EpochAwareLens] --> [3D.1 Epoch Cascade] ✅ -------+
                                                         |
[3B.2 Semantic Decay] ✅ -------------------------------+

Critical Path for Agile Agent Team Demo

[3C.3 Constraints Lens] (standalone) ------+
                                            |
[3B.1 Time-Travel] ✅ --------------------+----> AGENT TEAM MVP
                                            |
[2.3 EpochAwareLens] ✅ ------------------+
                                            |
[Query Audit (Phase 2)] ✅ ----------------+

44 KiB Raw Blame History