jordan/stemedb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
42d4e09508
stemedb/roadmap.md
jordan 42d4e09508 feat: Index persistence (Phase 5C) - vector hot/cold, visual checkpoint 
Phase 5C (Index Persistence) implementation:
- PersistentVectorIndex with hot/cold architecture
  - Hot: in-memory HNSW for recent vectors
  - Cold: memory-mapped HNSW loaded from disk
  - Background builder for WAL replay and atomic swap
  - BLAKE3 integrity verification
- PersistentVisualIndex with checkpoint persistence
  - BkTreeSnapshot with rkyv serialization
  - CRC32C corruption detection
  - Atomic write pattern (temp → fsync → rename)
- Key codec additions for vector index metadata
- Split large files into modules (<500 lines each)
  - battery_pre_sentinel.rs → battery/ directory
  - visual_index.rs → visual_index/ directory
  - persistent.rs → persistent/ directory
- Refactored ingest worker tests for clarity
- Updated roadmap to mark Phase 5 complete

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-02 15:43:18 -07:00

1300 lines
83 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Episteme (StemeDB) Roadmap
> **Goal:** Build the "Git for Truth" substrate for autonomous AI research.
> **Current Phase:** Phase 5 (Foundation Hardening)
> **Target Vertical:** BioTech/Pharma ("The Living Review")
> **Endgame:** Distributed multi-writer cluster for millions of concurrent agents
---
## High-Level Timeline
| Phase | Codename | Focus | Key Deliverable |
| :--- | :--- | :--- | :--- |
| **1** | **The Spine** | Storage & Safety | Append-only WAL + KV Store |
| **2** | **The Lattice** | Indexing & Async | Materialized Views + Ballot Box |
| **2.5** | **Hardening** | Camp 2 Fixes | MV staleness, epoch behavior, lens cleanup |
| **3** | **The Pilot** | Vertical Integration | Pharma Ingestion + Living Review Agent |
| **4** | **The Hive** | Trust & Learning | TrustRank, metadata indexing, change tracking |
| **5** | **The Forge** | Foundation Hardening | Replace sled, fix WAL, persist indices, concept hierarchy |
| **6** | **The Mesh** | Distributed Writes | CRDT replication, Raft coordination, cluster membership |
| **7** | **The Shield** | Trust at Scale | EigenTrust, PoW admission, anti-spam, quarantine |
| **8** | **The Swarm** | Production Cluster | Chaos testing, observability, geo-distribution |
---
## Detailed Milestones
### Phase 1: The Spine (Foundation)
*Goal: Securely ingest assertions and persist them without data loss.*
- [x] **Project Scaffold**: Initialize Rust workspace, set up linting/CI (clippy, fmt).
- [x] **Assertion Schema**: Define the `Assertion` struct with `rkyv` serialization.
- [x] Add dependencies: `rkyv`, `blake3`, `ed25519-dalek`, `image_hasher`.
- [x] Define `Assertion` struct (Subject, Predicate, Object, Confidence, SourceHash).
- [x] **Multi-Sig Expansion**: Implement `SignatureEntry` struct and `signatures: Vec<SignatureEntry>` field.
- [x] **Visual Expansion**: Add `visual_hash: Option<pHash>` field for image provenance.
- [x] Test serialization round-trips.
- [x] **Ballot Schema**: Define the `Vote` struct for multi-agent consensus.
- [x] Add `Vote` struct: `assertion_hash`, `agent_id`, `weight`, `signature`.
- [x] Test serialization round-trips.
- [x] **Paradigm Schema (Epochs)**: Define the `Epoch` and `SupersessionType` structs.
- [x] Add `epoch: Option<EpochId>` to `Assertion`.
- [x] Implement `Epoch` struct with `supersedes` and `SupersessionType`.
- [x] Test serialization round-trips.
- [x] **WAL Integration**: Implement the Quarantine Pattern for write-ahead logging.
- [x] Create `stemedb-wal` crate.
- [x] Port `FsyncGuard` and `Record` logic from established durability patterns.
- [x] Implement Record format with BLAKE3 checksums and Headers.
- [x] Verify `fsync` behavior with tests.
- [x] **Storage Engine**: Implement the `Store` trait using `sled` (embedded KV).
- [x] Add `sled` dependency.
- [x] Define `KVStore` trait (put, get, delete, scan_prefix, flush).
- [x] Implement `SledStore` wrapper.
- [x] **Basic Ingestor**: Background worker that tails WAL and writes to KV.
- [x] Implement async loop reading from WAL.
- [x] Write deserialized assertions, votes, and epochs to `sled`.
- [x] Ed25519 signature verification during ingestion.
- [x] Maintains S: and SP: indexes on ingest.
- [x] Persistent cursor/checkpoint (resumes from `__CURSOR__:ingest` in KV store).
- [x] **Verification**: Crash recovery tests (write -> crash -> restart -> read).
- [x] Single and multi-record crash recovery.
- [x] Multiple crash cycles tested.
### Phase 2: The Lattice (Connectivity)
*Goal: Query data with sub-millisecond latency using Materialized Views.*
- [x] **Lifecycle Schema**: Add `LifecycleStage` to Assertion.
- [x] Define enum: `Proposed`, `UnderReview`, `Approved`, `Deprecated`, `Rejected`.
- [x] Update `Assertion` struct and serialization tests.
- [x] **The Ballot Box**: Implement high-velocity vote ingestion.
- [x] `VoteStore` trait and implementation.
- [x] `VoteAwareConsensusLens` for real vote-based resolution.
- [x] **Index Infrastructure**: Compound indexes for O(1) queries.
- [x] `IndexStore` trait with S: and SP: indexes.
- [x] `QueryEngine` smart routing (SP -> S -> scan).
- [x] **Materializer**: Background worker for O(1) Read Performance.
- [x] `MaterializedView` type in `stemedb-core`.
- [x] `Materializer` worker in `stemedb-query` with `step()` and `run()`.
- [x] Aggregates Votes via `VoteAwareConsensusLens` (or any `AsyncLens`).
- [x] Updates `MV:{Subject}:{Predicate}` with the winning Assertion + metadata.
- [x] Event-driven mode via `run_notified()` with `tokio::sync::Notify`.
- [x] Fast-path MV lookup in `QueryEngine::try_fast_path()`.
- [x] **The Meter**: Implement Economic Throttling (TAN).
- [x] `QuotaStore` trait and `GenericQuotaStore` implementation.
- [x] Token Bucket algorithm with per-agent per-hour quotas.
- [x] `MeterLayer` tower middleware for request cost tracking.
- [x] Cost model: Assert=10, Vote=1, Query=5+lens, +1/KB payload.
- [x] `GET /v1/meter/quota` endpoint to check remaining quota.
- [x] `POST /v1/meter/quota/limit` admin endpoint to set custom limits.
- [x] **API Surface**: `axum` HTTP server with OpenAPI (utoipa).
- [x] `POST /v1/assert` -> Accepts JSON, writes to WAL.
- [x] `POST /v1/vote` -> High-throughput vote endpoint.
- [x] `POST /v1/epoch` -> Create epoch with optional supersession.
- [x] `GET /v1/query` -> Subject/Predicate/Lens/Lifecycle/Epoch filtering.
- [x] `GET /v1/health` -> Health check with assertion count.
- [x] `GET /swagger-ui` -> Interactive API docs.
- [x] 5 lens types available: Recency, Consensus, Authority, VoteAwareConsensus, TrustAwareAuthority.
- [x] **Query Audit**: Log every read with provenance.
- [x] Define `QueryAudit` struct: query_id, agent_id, timestamp, params, result_hash, contributing_assertions.
- [x] Storage at `AUD:{query_id}` with agent index at `AUDA:{agent_id}:{timestamp}:{query_id}`.
- [x] `GET /v1/audit/queries` -> Returns history of agent decisions.
- [x] `GET /v1/audit/query/{id}` -> Full reasoning trace for a single query.
- [x] Auto-logging on every query via `X-Agent-Id` header.
### Phase 2.5: Hardening (Camp 2 Fixes)
*Goal: Close the gaps between "built" and "works right." Every item here addresses a feature that exists but doesn't fully deliver on its promise.*
- [x] **2.1 MV Staleness Detection**: Make the fast-path aware of stale materialized views.
- **Status:** ✅ COMPLETE
- **Implementation:**
- [x] Added `max_stale: Option<u64>` to `Query` struct in `crates/stemedb-query/src/query.rs`.
- [x] Added `.max_stale(secs)` builder method to `QueryBuilder`.
- [x] In `try_fast_path()`: if `query.max_stale` is set and MV age exceeds threshold, falls through to slow path with `debug!` log.
- [x] Added `max_stale` to API `QueryParams` DTO in `crates/stemedb-api/src/dto.rs`.
- [x] Wired through query handler in `crates/stemedb-api/src/handlers/query.rs`.
- **Tests:**
- [x] `test_fast_path_stale_view_falls_back`: MV 1000 seconds old, `max_stale = 60` → slow path used.
- [x] `test_fast_path_fresh_view_used`: Fresh MV, `max_stale = 300` → fast path used.
- [x] `test_fast_path_no_max_stale_always_uses_mv`: No `max_stale` → any MV age accepted (backward compatible).
- [x] `test_fast_path_max_stale_zero_rejects_old_mv`: `max_stale = 0`, MV 1 second old → slow path.
- [x] `test_fast_path_max_stale_zero_accepts_brand_new_mv`: `max_stale = 0`, brand new MV → fast path.
- [x] **2.2 AuthorityLens -> ConfidenceLens Rename**: Eliminate the misleading name.
- **Problem:** `AuthorityLens` selects by `confidence` field, not by agent reputation. `TrustAwareAuthorityLens` is the real authority lens. The name creates confusion about what "Authority" means.
- **Solution implemented:**
- [x] Renamed `authority.rs` → `confidence.rs`, `AuthorityLens` → `ConfidenceLens`
- [x] Added `LensDto::Confidence` for the confidence-field selector
- [x] Changed `LensDto::Authority` to route to `TrustAwareAuthorityLens` (the real authority lens)
- [x] Updated query handler routing
- [x] Updated ai-lookup/services/lens.md and skill documentation
- [x] **2.3 EpochAwareLens**: Give epoch supersession runtime behavior.
- **Status:** ✅ COMPLETE
- **Implementation:**
- [x] `EpochAwareLens` in `crates/stemedb-lens/src/epoch_aware.rs`
- [x] Decorator pattern wrapping any inner lens (default: RecencyLens)
- [x] Walks supersession chain from `E:{epoch_id}` keys
- [x] Cycle detection + max depth guard (100)
- [x] Fail-open on missing epochs
- [x] `LensDto::EpochAware` added to API
- [x] 11 tests: excludes_superseded, chain_supersession, no_epochs_passes_all, missing_epoch_includes, cycle_detection, consensus_lens_inner, mixed_epochs, etc.
- [x] Documentation updated in `ai-lookup/services/lens.md`
- **Known Limitation:** Filtering only occurs when assertions from the superseding epoch are present in candidates. If all candidates are from old epoch (no new epoch assertions), they pass through (fail-open behavior).
- [x] **2.4 Visual Hash Query Support**: Make the stored `visual_hash` queryable.
- **Status:** ✅ COMPLETE
- **Implementation:**
- [x] `hamming_distance(a: &PHash, b: &PHash) -> u32` in `crates/stemedb-query/src/query.rs` (lines 26-28)
- [x] `visual_near: Option<String>` and `visual_threshold: Option<u32>` in `Query` struct (lines 84-90)
- [x] `.visual_near(hash, threshold)` builder method
- [x] `Query::matches()` computes hamming distance when `visual_near` is set
- [x] API `QueryParams` DTO has `visual_near` and `visual_threshold`
- [x] 10+ tests: exact_match, within_threshold, exceeds_threshold, skips_without_hash, invalid_hex, wrong_length, combines_with_subject, default_threshold, max_threshold, threshold_63_rejects
- **Note:** Brute-force O(N) scan. VP-tree/BK-tree index is Phase 3+.
- [x] **2.5 Vector Field**: No changes needed. Already roadmapped for Phase 3.
- **Status:** ✅ N/A (No Phase 2 work required)
- **Current state:** `vector: Option<Vec<f32>>` on `Assertion`. Stored and returned by API. No index, no search.
- **Phase 3 plan:** Integrate `hnsw-rs` or `lance` for k-NN search.
- [x] **2.6 E2E Integration Test (Write -> Materialize -> Read)**: Prove the full pipeline works end-to-end.
- **Status:** ✅ COMPLETE
- **Implementation:**
- [x] `crates/stemedb-query/tests/e2e_pipeline.rs` with 5 comprehensive tests:
- `test_e2e_write_materialize_read` - Basic happy path
- `test_e2e_vote_consensus` - Vote-weighted resolution
- `test_e2e_update_winner` - Winner changes on re-materialize
- `test_e2e_cursor_persistence` - Cursor survives worker restart
- `test_e2e_notify_integration` - Event-driven notification channel
- [x] `stemedb-wal` and `stemedb-ingest` added as dev-dependencies
- [x] Helper functions: `create_signed_assertion()`, `compute_assertion_hash()`, `create_vote()`
- [x] Uses Ed25519 signing for authentic signature verification
- [x] Also: `crates/stemedb-api/tests/e2e_flow_test.rs` tests the HTTP API layer end-to-end.
### Phase 3: The Pilot (BioTech/Pharma)
*Goal: Prove value in the "High-Liability" beachhead. Close every Camp 4 gap that blocks a credible demo.*
#### 3A. Schema Expansion (Prerequisite for everything below)
- [x] **3A.1 Source-Class Field**: Add `source_class: SourceClass` to Assertion.
- **Status:** ✅ COMPLETE
- **Implementation:**
- [x] `SourceClass` enum in `crates/stemedb-core/src/types.rs` (lines 68-88).
- [x] 6-tier system: `Regulatory` (0), `Clinical` (1), `Observational` (2), `Expert` (3), `Community` (4), `Anecdotal` (5).
- [x] `tier()` method returns tier number for ordering.
- [x] `default_decay_days()` method for tier-specific confidence decay.
- [x] `authority_weight()` method for conflict resolution weighting.
- [x] Field on `Assertion` struct at line 152.
- [x] Full serialization and indexing support.
- [x] **3A.2 Conflict Score on Resolution**: Add `conflict_score: f32` to Resolution.
- **Status:** ✅ COMPLETE
- **Implementation:**
- [x] Added `conflict_score: f32` field to `Resolution` in `crates/stemedb-lens/src/traits.rs`.
- [x] Updated `Resolution::empty()` to set `conflict_score: 0.0`.
- [x] Updated `Resolution::with_winner()` to accept `conflict_score` parameter.
- [x] Added `compute_conflict_score(candidates: &[Assertion]) -> f32` utility function:
- Uses normalized variance of confidence values.
- 0 or 1 candidates → 0.0 (no conflict possible).
- All same confidence → 0.0 (unanimous).
- Max variance (0.0 vs 1.0) → 1.0 (maximum conflict).
- Defensive NaN handling (returns 0.0 for malformed data).
- [x] Updated all lens implementations to compute and pass conflict score:
- `crates/stemedb-lens/src/recency.rs`
- `crates/stemedb-lens/src/consensus.rs`
- `crates/stemedb-lens/src/confidence.rs`
- `crates/stemedb-lens/src/vote_aware_consensus.rs`
- `crates/stemedb-lens/src/trust_aware_authority.rs`
- [x] Added `conflict_score: f32` to `MaterializedView` in `crates/stemedb-core/src/types.rs`.
- [x] Updated `Materializer::materialize_pair()` to write `conflict_score` from resolution.
- [x] Added `conflict_score: Option<f32>` and `resolution_confidence: Option<f32>` to `QueryResponse` DTO in `crates/stemedb-api/src/dto.rs` (only present when lens is applied).
- [x] Wired through query handler in `crates/stemedb-api/src/handlers/query.rs`.
- **Tests:**
- [x] `test_conflict_score_zero_for_empty`: Empty candidates → 0.0.
- [x] `test_conflict_score_zero_for_single`: 1 candidate → 0.0.
- [x] `test_conflict_score_zero_for_agreement`: All same confidence → near 0.0.
- [x] `test_conflict_score_high_for_disagreement`: Candidates at 0.1, 0.5, 0.9 → score > 0.3.
- [x] `test_conflict_score_max_for_extremes`: 0.0 vs 1.0 → score ≈ 1.0.
- [x] `test_conflict_score_handles_nan_defensively`: NaN confidences → 0.0 (fail-safe).
- [x] **3A.3 Rich Source Metadata**: Add structured provenance beyond `source_hash`.
- **Status:** ✅ COMPLETE
- **Implementation:**
- [x] Added `source_metadata: Option<Vec<u8>>` field to `Assertion` in `crates/stemedb-core/src/types.rs` (after `epoch`, before `lifecycle`).
- [x] Uses `Vec<u8>` (not `String`) for rkyv zero-copy compatibility. Callers encode/decode JSON on their side.
- [x] Added `source_metadata: Option<Vec<u8>>` to `AssertionBuilder` in `crates/stemedb-core/src/testing.rs`.
- [x] Added `.source_metadata_json(json: &str)` and `.source_metadata(bytes)` builder methods.
- [x] Added `source_metadata: Option<String>` to `CreateAssertionRequest` DTO (JSON string in API, converted to bytes internally).
- [x] Added `source_metadata: Option<String>` to `AssertionResponse` DTO (bytes converted to JSON string with defensive UTF-8 handling).
- [x] Wired through create handler (`dto_to_assertion()`) and query handler (`assertion_to_dto()`).
- **Tests:**
- [x] `test_serialize_deserialize_assertion_with_metadata`: Serialization roundtrip with metadata present.
- [x] `test_serialize_deserialize_assertion_without_metadata`: Serialization roundtrip with metadata absent.
- **Note:** Metadata is stored but NOT indexed in Phase 3. Indexing individual metadata fields is Phase 4+.
#### 3B. Time & Decay (Core Query Features)
- [x] **3B.1 Time-Travel Engine**: `as_of` parameter for historical queries.
- **Status:** ✅ COMPLETE
- **Implementation:**
- [x] Added `as_of: Option<u64>` to `Query` struct in `crates/stemedb-query/src/query.rs:92-99`.
- [x] Added `.as_of(timestamp: u64)` to `QueryBuilder`.
- [x] In `Query::matches()`: if `as_of` is `Some(ts)`, check `assertion.timestamp <= ts`. Assertions created after `as_of` are excluded.
- [x] In `QueryEngine::execute()`: if `query.as_of` is set, **skip the fast path entirely** (MVs reflect current state, not historical).
- [x] Added `as_of: Option<u64>` to `QueryParams` DTO in `crates/stemedb-api/src/dto.rs`.
- [x] Wired through query handler.
- **Tests:**
- [x] `test_as_of_excludes_future_assertions`: Assertions filtered by timestamp.
- [x] `test_as_of_bypasses_fast_path`: MV exists, but `as_of` is set. Slow path used.
- [x] `test_as_of_none_uses_fast_path`: Normal query still uses fast path (backwards-compatible).
- [x] `test_as_of_with_lens_resolves_among_historical_candidates`: Time-travel + lens = resolve only among pre-as_of candidates.
- [x] `test_as_of_returns_empty_when_all_assertions_are_future`: All assertions are future, returns empty.
- [x] `test_as_of_with_exact_timestamp_match`: Edge case where assertion.timestamp == as_of.
- [x] **3B.2 Semantic Decay**: Confidence Half-Life at query time.
- **Status:** ✅ COMPLETE
- **Implementation:**
- [x] Added `decay_halflife: Option<u64>` to `Query` struct in `crates/stemedb-query/src/query.rs`.
- [x] Added `.decay_halflife(seconds: u64)` and `.source_class_decay(enabled: bool)` to `QueryBuilder`.
- [x] Added `decay_halflife` and `source_class_decay` to `QueryParams` DTO.
- [x] Created new `crates/stemedb-query/src/decay.rs` module with:
- `apply_decay()`: Uniform decay using formula `confidence * 2^(-(age / halflife))`.
- `apply_source_class_decay()`: Tier-specific decay (Regulatory=none, Clinical=2yr, Anecdotal=30d).
- `compute_decayed_confidence()`: Core decay calculation with clamping.
- [x] Integrated in `QueryEngine::execute()`: decay applied after filtering, before lens resolution.
- [x] Time-travel compatible: uses `as_of` timestamp if set, otherwise current time.
- [x] Source-class-aware decay fully implemented using `SourceClass::default_decay_days()`.
- **Tests:** (11 unit tests in decay.rs + 1 E2E test)
- [x] `test_decay_reduces_old_assertion_confidence`: 1yr old, 1yr halflife → ~50% confidence.
- [x] `test_decay_preserves_fresh_assertions`: 1hr old, 1yr halflife → ~100% confidence.
- [x] `test_decay_interacts_with_lens`: Older high-confidence loses to newer low-confidence after decay.
- [x] `test_source_aware_decay_tier0_no_decay`: Regulatory never decays.
- [x] `test_source_aware_decay_tier5_rapid_decay`: Anecdotal decays rapidly (30-day halflife).
- [x] `test_source_aware_decay_mixed_tiers`: Clinical vs Anecdotal tier comparison.
- [x] `test_decay_zero_halflife_no_change`: Zero halflife skips decay (avoids div-by-zero).
- [x] `test_decay_future_assertion_no_change`: Future assertions don't decay.
- [x] `test_decay_empty_assertions`: Empty input returns empty output.
- [x] `test_decay_confidence_clamps_to_valid_range`: Very old assertions clamp to [0.0, 1.0].
- [x] `test_decay_preserves_other_fields`: Only confidence changes; other fields preserved.
- [x] `test_e2e_decay_reduces_old_confidence`: Full pipeline E2E test in e2e_pipeline.rs.
- **Note:** When decay is enabled, materialized views (fast path) are bypassed because MVs store pre-computed winners without decay applied.
#### 3C. New Lenses
- [x] **3C.1 Skeptic Lens**: Surface disagreement, not winners. ✅ **COMPLETED**
- **Status:** ✅ COMPLETE
- **Implementation:**
- [x] `crates/stemedb-lens/src/skeptic.rs` - Full implementation.
- [x] `AnalysisLens` trait for lenses that analyze conflict instead of resolving it.
- [x] `SkepticLens` uses normalized Shannon entropy for conflict scoring.
- [x] Returns `ConflictAnalysis` with:
- `conflict_score: f32` (0.0 = unanimous, 1.0 = chaos)
- `status: ResolutionStatus` (Unanimous, Agreed, Contested)
- `claims: Vec<ClaimSummary>` - all claims ranked by weight
- [x] `SkepticResolver` + `SkepticView` in `stemedb-query/src/skeptic.rs`.
- [x] `GET /v1/skeptic?subject=X&predicate=Y` API endpoint.
- [x] Core types in `stemedb-core/src/types.rs`:
- `ResolutionStatus` enum
- `ConflictAnalysis` struct
- `ClaimSummary`, `SourceSummary`, `AgentSummary`
- [x] Comprehensive test coverage (21 test cases).
- [x] **3C.2 Layered Consensus Lens**: Per-source-class consensus.
- **Status:** ✅ COMPLETE
- **Implementation:**
- [x] `crates/stemedb-lens/src/layered_consensus.rs` - Full implementation.
- [x] `TierResolution` struct: per-tier result with tier, source_class, winner, candidates_count, conflict_score, resolution_confidence.
- [x] `LayeredResolution` struct: multi-tier result with tiers vec, overall_winner, overall_conflict_score, total_candidates.
- [x] `LayeredLens` trait: `resolve_layered(&[Assertion]) -> LayeredResolution`, `name() -> &'static str`.
- [x] `LayeredConsensusLens` implements both `LayeredLens` and `Lens` traits.
- [x] Cross-tier conflict score uses normalized Shannon entropy of tier winner object values.
- [x] `LensDto::LayeredConsensus` variant (redirects to `/v1/layered` endpoint).
- [x] `GET /v1/layered?subject=X&predicate=Y` API endpoint with `LayeredQueryResponse`.
- [x] Exported from `crates/stemedb-lens/src/lib.rs`.
- **Tests:**
- [x] `test_layered_empty_candidates`: Empty input returns empty resolution.
- [x] `test_layered_single_tier`: All same source_class, returns one tier result.
- [x] `test_layered_multi_tier_agreement`: Tier 0 and Tier 5 agree, low cross-tier conflict.
- [x] `test_layered_multi_tier_disagreement`: Tier 1 vs Tier 5 disagree, high conflict, Tier 1 wins.
- [x] `test_layered_overall_winner_from_highest_authority`: Tier 0 wins despite fewer assertions.
- [x] `test_layered_lens_trait_compatibility`: Standard Lens trait works.
- [x] `test_layered_within_tier_conflict`: High internal conflict within a tier.
- [x] `test_layered_all_tiers_present`: One assertion from each tier.
- [x] `test_layered_lens_name`: Both trait names work.
- [x] `test_layered_numeric_values`: Works with numeric object values.
- [x] **3C.3 Constraints Lens**: Pre-flight check for must_use/forbidden.
- **Status:** ✅ COMPLETE
- **Implementation:**
- [x] `crates/stemedb-lens/src/constraints.rs` - Full implementation.
- [x] `ConstraintSet` struct: holds categorized assertions (must_use, forbidden, prefer) with conflict_score.
- [x] `ConstraintsLens` struct with `resolve_constraints(&[Assertion]) -> ConstraintSet` method.
- [x] Categorizes by predicate pattern: `must_use:*`, `forbidden:*`, `prefer:*`.
- [x] Implements `Lens` trait for compatibility (priority: must_use > forbidden > prefer).
- [x] Sorted by confidence (highest first), with timestamp as tiebreaker.
- [x] `LensDto::Constraints` added (redirects to `/v1/constraints` endpoint).
- [x] `GET /v1/constraints?subject=X` API endpoint with `ConstraintsResponse`.
- [x] DTOs: `ConstraintsQueryParams`, `ConstraintEntryDto`, `ConstraintsResponse`.
- [x] Exported from `crates/stemedb-lens/src/lib.rs`.
- **Tests:** (16 test cases)
- [x] `test_constraints_categorizes_by_predicate`: Mixed predicates sorted into must_use/forbidden/prefer.
- [x] `test_constraints_empty_categories`: Only prefer, no must_use/forbidden.
- [x] `test_constraints_non_constraint_predicates_ignored`: Regular predicates filtered out.
- [x] `test_constraints_sorted_by_confidence`: Within-category confidence ordering.
- [x] `test_constraints_empty_candidates`: Empty input returns empty set.
- [x] `test_constraints_has_constraints_true`: Helper method works.
- [x] `test_constraints_all_regular_predicates`: All non-constraint predicates returns no constraints.
- [x] `test_lens_trait_picks_must_use_winner`: Standard Lens trait picks must_use first.
- [x] `test_lens_trait_falls_back_to_forbidden`: Falls back to forbidden when no must_use.
- [x] `test_lens_trait_falls_back_to_prefer`: Falls back to prefer when no must_use/forbidden.
- [x] `test_lens_trait_empty_for_no_constraints`: Returns empty when no constraint predicates.
- [x] `test_lens_name`: Name returns "Constraints".
- [x] `test_lens_empty_candidates`: Empty input to Lens trait returns empty resolution.
- [x] `test_multiple_must_use_picks_highest_confidence`: Multiple must_use picks highest confidence.
- [x] `test_confidence_tiebreaker_uses_timestamp`: Same confidence uses newer timestamp.
- [x] `test_predicate_pattern_exact_prefix`: `must_use_something` not matched (only `must_use:*`).
#### 3D. Epoch Enhancement
- [x] **3D.1 Epoch Cascade Logic** (enhancement of Phase 2.5 EpochAwareLens):
- **Status:** ✅ COMPLETE
- **Implementation:**
- [x] `write_supersession_cascade()` in `crates/stemedb-ingest/src/worker.rs`:
- Writes `SUPERSEDED:{old_epoch_id}` markers for full transitive closure.
- All markers point to the LATEST superseding epoch.
- Max depth guard (100 levels) and cycle detection via visited set.
- [x] `is_epoch_superseded()` in `crates/stemedb-lens/src/epoch_aware.rs`:
- O(1) marker lookup instead of O(chain_length) chain walks.
- Fail-open semantics: missing marker = not superseded.
- [x] `compute_superseded_epochs()` uses marker lookups for filtering.
- **Tests:**
- [x] `test_cascade_writes_superseded_marker`: Epoch B supersedes A → `SUPERSEDED:A` exists.
- [x] `test_cascade_transitive`: C→B→A chain → both `SUPERSEDED:A` and `SUPERSEDED:B` point to C.
- [x] `test_cascade_cycle_detection`: Mutual supersession handled gracefully.
- [x] `test_epoch_aware_uses_marker`: EpochAwareLens uses O(1) marker lookup.
- [x] `test_superseded_epoch_filtered_even_without_new_assertions`: Marker-based filtering works.
#### 3E. Similarity Search
- [x] **3E.1 Vector Search**: Semantic k-NN queries via embeddings.
- **Status:** ✅ COMPLETE
- **Implementation:**
- [x] Added `hnsw_rs = "0.3"` and `parking_lot = "0.12"` to `stemedb-storage/Cargo.toml`.
- [x] New module: `crates/stemedb-storage/src/vector_index.rs`.
- [x] `VectorIndex` trait: `insert(hash: &Hash, vector: &[f32])`, `search(query: &[f32], k: usize) -> Vec<(Hash, f32)>`, `dimension()`, `len()`, `is_empty()`.
- [x] `HnswVectorIndex` implementation with HNSW graph, RwLock protection, hash↔ID mappings.
- [x] Input validation: dimension mismatch, NaN, Infinite values rejected.
- [x] Idempotent insert (same hash twice = no-op).
- [x] `Arc<dyn VectorIndex>` trait object support for sharing.
- [x] `IngestWorker::with_vector_index()` builder method for index attachment.
- [x] IngestWorker: if assertion has `vector`, inserts into vector index after KV write.
- [x] Added `vector_near: Option<Vec<f32>>` and `k: Option<usize>` to `Query` struct in `crates/stemedb-query/src/query.rs`.
- [x] Added `.vector_near(vector, k)` builder method to `QueryBuilder`.
- [x] Added `vector_near` and `k` to API `QueryParams` DTO.
- [x] `QueryEngine::with_vector_index()` builder method for index attachment.
- [x] QueryEngine: if `vector_near` is set and index configured, uses O(log N) HNSW lookup for candidates.
- [x] Falls back to standard path if no index configured (with debug log).
- **Tests:** (12 unit tests for VectorIndex + 4 integration tests in engine.rs)
- [x] `test_create_index`, `test_insert_and_search`, `test_dimension_mismatch`.
- [x] `test_idempotent_insert`, `test_search_empty_index`, `test_search_k_zero`.
- [x] `test_nan_rejection`, `test_infinite_rejection`, `test_contains`.
- [x] `test_larger_scale` (100 vectors, exact match first), `test_custom_params`, `test_zero_dimension_panics`.
- [x] `test_vector_search_returns_nearest_neighbors`, `test_vector_search_with_subject_filter`.
- [x] `test_vector_search_without_index_falls_back`, `test_vector_search_with_as_of_filter`.
- **Note:** Index is in-memory only. Persistence is Phase 4+.
- [x] **3E.2 Visual Hash Index**: BK-tree for O(log N) visual similarity.
- **Status:** ✅ COMPLETE
- **Implementation:**
- [x] New module: `crates/stemedb-storage/src/visual_index.rs`.
- [x] `VisualIndex` trait: `insert(hash: &Hash, phash: &PHash)`, `search(query: &PHash, threshold: u32) -> Vec<(Hash, u32)>`, `len()`, `is_empty()`.
- [x] `BkTreeVisualIndex` implementation using BK-tree over hamming distance.
- [x] `hamming_distance(a: &PHash, b: &PHash) -> u32` utility function.
- [x] Threshold clamped to 0-64 range (max 64 bits).
- [x] Results sorted by distance ascending.
- [x] Idempotent insert (same hash twice = no-op).
- [x] `Arc<dyn VisualIndex>` trait object support for sharing.
- [x] `IngestWorker::with_visual_index()` builder method for index attachment.
- [x] IngestWorker: if assertion has `visual_hash`, inserts into BK-tree after KV write.
- [x] `QueryEngine::with_visual_index()` builder method for index attachment.
- [x] QueryEngine: if `visual_near` is set and index configured, uses O(log N) BK-tree lookup.
- [x] Falls back to brute-force scan (via `query.matches()`) if no index configured.
- [x] Invalid hex input returns `QueryError::InvalidInput` with clear message.
- **Tests:** (14 unit tests for VisualIndex + 6 integration tests in engine.rs)
- [x] `test_hamming_distance_zero`, `test_hamming_distance_max`, `test_hamming_distance_partial`.
- [x] `test_create_index`, `test_insert_and_search_exact`, `test_search_within_threshold`.
- [x] `test_search_no_matches`, `test_search_empty_index`, `test_idempotent_insert`.
- [x] `test_contains`, `test_results_sorted_by_distance`, `test_threshold_clamped_to_64`.
- [x] `test_larger_scale` (1000 hashes), `test_default_impl`.
- [x] `test_visual_search_returns_similar_images`, `test_visual_search_with_lifecycle_filter`.
- [x] `test_visual_search_invalid_hex_returns_error`, `test_visual_search_without_index_uses_brute_force`.
- [x] `test_visual_search_with_limit`, `test_vector_search_empty_index`.
- **Note:** Index is in-memory only. Persistence is Phase 4+.
#### 3F. Provenance
- [x] **3F.1 Source Document Storage & Provenance Lookup**: Enable 100% citation recall.
- **Status:** ✅ COMPLETE
- **Implementation:**
- [x] `POST /v1/source` endpoint to store source documents by BLAKE3 content hash.
- [x] `GET /v1/provenance/{hash}` endpoint to retrieve source documents by hash.
- [x] Source storage at `SRC:{hash}` keys with format: `[content_type_len:4][content_type][content]`.
- [x] Base64 encoding for binary-safe JSON transport.
- [x] 10MB size limit per document.
- [x] Content-addressed storage: same content → same hash (idempotent uploads).
- [x] DTOs: `StoreSourceRequest`, `StoreSourceResponse`, `ProvenanceResponse`.
- [x] OpenAPI documentation under "provenance" tag.
- **Tests:** (5 test cases)
- [x] `test_store_and_retrieve_source`: Happy path store + retrieve.
- [x] `test_store_source_invalid_base64`: Bad base64 → 400.
- [x] `test_get_provenance_not_found`: Unknown hash → 404.
- [x] `test_get_provenance_invalid_hash`: Bad hash format → 400.
- [x] `test_store_source_idempotent`: Same content twice → same hash.
- **Note:** Benchmark utility for verifying all assertions have retrievable sources is future work.
#### 3G. API Cleanup
- [x] **3G.1 Document epoch supersession via existing endpoint**: No new `/epoch/supersede` endpoint needed.
- **Status:** ✅ COMPLETE
- **Implementation:**
- [x] Updated use case docs (consumer-health-intelligence.md, glp1-living-review.md) to use `POST /v1/epoch` with `supersedes` field.
- [x] Added OpenAPI examples showing both new epoch and supersession flows in handlers/epoch.rs.
- [x] Documented all 5 supersession types: Invalidate, Temporal, Refinement, RequiresReview, Additive.
- **No code change.** Documentation fix only.
### Phase 4: The Hive (Trust & Scale)
*Goal: Change tracking, metadata indexing, and the database primitives for training pipelines.*
- [x] **TrustRank Engine**: Foundation for trust-based resolution.
- [x] `TrustRankStore` for per-agent reputation storage.
- [x] `TrustAwareAuthorityLens` for reputation-weighted resolution.
- [x] **Confidence Half-Life**: Implement decay calculation engine.
- [x] Learning loop: `record_outcome()` for accuracy tracking.
- [x] **4.1 "Since" Parameter**: Change tracking for returning consumers.
- **Status:** ✅ COMPLETE
- **Problem:** Consumer Health shows `GET /query?since=2023-10-01` returning `changes_since` with dated change entries. The "returning consumer" story: "What changed since I last looked?"
- **Depends on:** Time-Travel (3B.1) and Materializer.
- **Implementation:**
- [x] Added `since: Option<u64>` to `Query` struct and `QueryBuilder` in `crates/stemedb-query/src/query.rs`.
- [x] Added `since` to `QueryParams` DTO in `crates/stemedb-api/src/dto.rs`.
- [x] **MV Changelog**: Track when materialized views change.
- [x] New key pattern: `MVC:{subject}:{predicate}:{timestamp_nanos}` using nanosecond precision to prevent collisions.
- [x] `ChangeEntry` struct in `crates/stemedb-core/src/types.rs` with: `timestamp`, `previous_winner_hash`, `new_winner_hash`, `subject`, `predicate`, `lens_name`.
- [x] In `Materializer::materialize_pair()`: before overwriting MV, read existing MV. If winner changed (different hash), write changelog entry.
- [x] `get_previous_winner_hash()` and `write_changelog_entry()` helper methods.
- [x] In QueryEngine: if `since` is set, skip fast path (MVs reflect current state). `fetch_changes_since()` scans `MVC:` keys and returns entries >= since timestamp.
- [x] `ChangeEntryDto` in `crates/stemedb-api/src/dto.rs` with hex-encoded hashes.
- [x] `changes_since: Option<Vec<ChangeEntryDto>>` added to `QueryResponse` DTO.
- [x] `fetch_changelog_if_needed()` helper in query handler.
- [x] Re-exported `ChangeEntry` from `stemedb-core` crate root.
- **Tests:**
- [x] `test_changelog_written_on_first_materialization`: First MV creates changelog with `previous_winner_hash: None`.
- [x] `test_changelog_written_on_winner_change`: Winner change creates changelog with previous hash.
- [x] `test_no_changelog_when_winner_unchanged`: Re-materialize same winner = no new changelog.
- [x] `test_fetch_changes_since_returns_entries`: Fetch filtered by timestamp, sorted ascending.
- [x] `test_fetch_changes_since_empty_on_no_entries`: No entries returns empty vec.
- [x] `test_since_bypasses_fast_path`: Query with `since` uses slow path, returns all assertions.
- [x] **4.2 Source Metadata Indexing** (extension of 3A.3): Index key metadata fields.
- **Status:** ✅ COMPLETE
- **Problem:** Phase 3 stores `source_metadata` as an opaque blob. Phase 4 makes key fields queryable.
- **Depends on:** Rich Source Metadata (3A.3).
- **Implementation:**
- [x] Defined indexed metadata keys: `journal`, `doi`, `platform`, `study_design` in `INDEXED_METADATA_FIELDS`.
- [x] New module: `crates/stemedb-storage/src/source_metadata_index.rs`.
- [x] Key pattern: `SMV:{field}:{value}` storing `Vec<Hash>` (rkyv-serialized) for efficient batch lookup.
- [x] `SourceMetadataIndexStore` trait with `add_to_metadata_indexes()` and `get_by_metadata_field()`.
- [x] `GenericSourceMetadataIndexStore` implementation with case-insensitive value normalization.
- [x] IngestWorker: on ingestion, if `source_metadata` is present, parse JSON, extract indexed fields, write index entries.
- [x] Added `source_journal`, `source_doi`, `source_platform`, `source_study_design` to `Query` struct.
- [x] Added `.source_journal()`, `.source_doi()`, `.source_platform()`, `.source_study_design()` builder methods.
- [x] Added `matches_metadata_filters()` helper with AND semantics and case-insensitive matching.
- [x] Added metadata field filters to `QueryParams` DTO (e.g., `?source_journal=NEJM`).
- [x] Wired through query handler in `crates/stemedb-api/src/handlers/query.rs`.
- **Tests:** (9 unit tests in source_metadata_index.rs + 9 unit tests in query.rs)
- [x] `test_add_and_get_by_metadata_field`: Basic round-trip.
- [x] `test_case_insensitive_indexing`: "NEJM" stored, "nejm" query matches.
- [x] `test_multiple_assertions_same_field`: Vec accumulates correctly.
- [x] `test_malformed_json_gracefully_skipped`: Invalid JSON logs warning, doesn't error.
- [x] `test_missing_field_not_indexed`: Only indexed fields processed.
- [x] `test_idempotent_insert`: Same assertion twice = no duplicates.
- [x] `test_all_indexed_fields`: All 4 fields indexed correctly.
- [x] `test_empty_index_returns_empty_vec`: Non-existent field returns empty.
- [x] `test_non_string_values_ignored`: Non-string JSON values skipped.
- [x] **4.3 Batch TrustRank Decay API**: Expose scheduled decay for external orchestration.
- **Status:** ✅ COMPLETE
- **Implementation:**
- [x] New module: `crates/stemedb-api/src/handlers/admin.rs`
- [x] `POST /v1/admin/decay-trust-ranks` endpoint.
- [x] Request accepts `now: Option<u64>` (defaults to current time) and `half_life_seconds: Option<u64>` (defaults to 30 days).
- [x] Response includes `decayed_count`, `timestamp_used`, `half_life_used`, `status`.
- [x] OpenAPI docs with "admin" tag.
- [x] Thin handler delegates to `TrustRankStore::decay_trust_ranks()`.
- **Tests:** (3 integration tests in http_integration.rs)
- [x] `test_decay_trust_ranks_empty_store`: Empty store returns 0 decayed.
- [x] `test_decay_trust_ranks_with_custom_params`: Custom timestamp and half-life honored.
- [x] `test_decay_trust_ranks_response_structure`: All 4 fields present in response.
- **Note:** The Gardener (Camp 5.2, app layer) calls this endpoint on a schedule. The database just exposes the primitive.
- [x] **4.4 Vote Provenance Witness**: Transform votes from opinions into cryptographic witnesses.
- **Status:** ✅ COMPLETE
- **Problem:** Votes record "I agree" but the browser extension product needs "I saw this exact text at this URL at this time." Without provenance, votes are Tier 5 noise.
- **Implementation:**
- [x] Added `source_url: Option<String>` to Vote struct — URL where claim was observed.
- [x] Added `observed_context: Option<Vec<u8>>` to Vote struct — page context bytes (rkyv zero-copy pattern).
- [x] API DTOs: `source_url` and `observed_context` on CreateVoteRequest/VoteResponse.
- [x] Input validation: source_url max 2048 chars (non-empty if provided), observed_context max 64KB.
- [x] Backward compatible: existing votes without provenance remain valid.
- **Tests:** Serialization roundtrip (with/without provenance, mixed fields), storage roundtrip, API integration (validation).
- [x] **4.5 Conflict Score Filtering**: Query-time filtering by conflict score.
- **Status:** ✅ COMPLETE
- **Problem:** The browser extension needs "only show me claims where conflict > 0.7" to implement contradiction-only overlays. Conflict score was computed but not filterable.
- **Implementation:**
- [x] Added `min_conflict_score: Option<f32>` and `max_conflict_score: Option<f32>` to Query struct.
- [x] Builder methods: `.min_conflict_score()`, `.max_conflict_score()`.
- [x] Fast-path filtering: checks MV conflict_score against thresholds.
- [x] API validation: scores must be 0.0-1.0, finite (rejects NaN/Inf).
- [x] QueryParams DTO wired through query handler.
- **Tests:** 13 engine tests (min/max thresholds, exact boundary, range, combination with lifecycle), API validation tests.
- **Note:** Filtering works on fast path (MV reads) only. Slow path returns raw assertions without conflict scores.
- [x] **4.6 Escalation Triggers**: Active safety system for high-conflict assertions.
- **Status:** ✅ COMPLETE
- **Problem:** High conflict was invisible. Episteme should be an active safety system that fires escalations when disagreement exceeds thresholds.
- **Implementation:**
- [x] `EscalationLevel` enum: Low, Medium, High, Critical.
- [x] `EscalationEvent` struct: content-addressed (BLAKE3 ID), subject, predicate, conflict_score, level, reason, timestamp, resolved.
- [x] `EscalationPolicy` struct: configurable threshold + level + optional predicate pattern.
- [x] `EscalationStore` trait + `GenericEscalationStore`: write, query since, resolve, get pending.
- [x] Key pattern: `ESC:{timestamp_nanos}:{id_hex}`.
- [x] Materializer integration: after computing conflict_score, checks policies and writes events.
- [x] `with_escalation(store, policies)` builder on Materializer.
- [x] `GET /v1/admin/escalations` and `POST /v1/admin/escalations/{id}/resolve` endpoints.
- **Tests:** Core type tests, storage roundtrip, materializer integration (high/low conflict, predicate matching, no store configured).
- [x] **4.7 Gold Standard Verification**: Sybil defense via proof of knowledge.
- **Status:** ✅ COMPLETE
- **Problem:** New agents need a way to earn TrustRank. Gold standards are admin-verified assertions that agents can be tested against.
- **Implementation:**
- [x] `GoldStandard` struct with rkyv serialization: assertion_hash, subject, predicate, expected_object, created_at, created_by.
- [x] `GoldStandardStore` trait + `GenericGoldStandardStore`: set, get, list, remove.
- [x] Key pattern: `GS:{subject}:{predicate}`.
- [x] `TrustAdjustment` enum: Rewarded(+0.05), Penalized(-0.1), AlreadyVerified.
- [x] `verify_agent_against_gold_standard()` on TrustRankStore with deduplication (agents can only verify each gold standard once).
- [x] Verification markers at `GS_VERIFIED:{agent_id}:{subject}:{predicate}` prevent gaming.
- [x] API: `POST/GET/DELETE /v1/admin/gold-standards`, `POST /v1/admin/verify-agent`.
- **Tests:** Core (creation, matching, case-sensitivity), storage CRUD, trust adjustment (reward, penalty, clamping, dedup), API integration (5 tests).
> **Note:** The following items were reclassified as **Application Layer** responsibilities (see `tmp/ambition-vs-reality.md`, Camp 5). They are not Episteme database features. They consume the Episteme API and are built by integrators or vertical-specific teams.
>
> - **The Simulator** (Training Data Pipeline) -> Camp 5.3
> - **The Super Curator** (Reviewer Agent swarm) -> Camp 5.4
> - **Background Gardener** (Cluster detection, signal processing) -> Camp 5.2
> - **Agent Wallet** (Key management sidecar) -> Camp 5.1
### Phase 5: The Forge (Foundation Hardening)
*Goal: Replace abandoned dependencies, fix WAL gaps, persist indices. Prerequisite for distribution.*
> **Research:** [docs/research/wal-crash-recovery-research.md](docs/research/wal-crash-recovery-research.md)
#### 5A. Storage Engine Replacement
- [x] **5A.1 Replace sled with redb + fjall**: sled is abandoned (author recommends alternatives).
- **Problem:** sled is alpha-stage with known performance regressions and no active development. Our entire storage layer depends on it.
- **Solution:** HybridStore routes keys by prefix — **fjall** (LSM) for write-heavy paths (`H:`, `V:`, `VC:`, `VW:`, `E:`, `SUPERSEDED:`, `__CURSOR__:`) and **redb** (B-tree) for read-heavy paths (`S:`, `SP:`, `MV:`, `TR:`, `QA:`, `QT:`, `TP:`, `GS:`, `ESC:`).
- **Tasks:**
- [x] Generalize `StorageError::Sled` to `StorageError::Backend(String)`.
- [x] Implement `FjallStore` backend with DashMap per-key locks for atomics.
- [x] Implement `RedbStore` backend with ACID transactions.
- [x] Implement `HybridStore` routing layer with prefix-based dispatch.
- [x] Migrate all ~500 tests from `SledStore` to `HybridStore`.
- [x] Remove sled dependency entirely.
- [x] Add criterion benchmarks (sequential put, random get, prefix scan, atomic increment, mixed workload).
- **Crates:** `redb = "2"`, `fjall = "2"`, `dashmap = "6"`
- [x] **5A.2 Key Layout Redesign**: Prepare keys for subject-prefix range sharding.
- **Problem:** Current keys (`H:{hash}`, `S:{subject}`, `MV:{subject}:{predicate}`) scatter related data across the keyspace. Distributed sharding needs co-location.
- **Solution:** Subject-prefix key layout with `\x00` separator for subject-scoped keys, `\x00` prefix for global keys (sort-first):
```
Subject-prefixed (co-located):
{subject}\x00H:{hash} → Assertion data
{subject}\x00S:{hash_list} → Subject index
{subject}\x00SP:{predicate} → Compound index
{subject}\x00MV:{predicate} → Materialized view
{subject}\x00V:{hash}:{vh} → Votes
{subject}\x00VC:{hash} → Vote count cache
{subject}\x00VW:{hash} → Vote weight cache
{subject}\x00GS:{predicate} → Gold standards
Global (sort first via \x00 prefix):
\x00TRUST:{agent_id} → Trust ranks
\x00QUOTA:{agent_id}:{win} → Quota records
\x00QLIMIT:{agent_id} → Quota limits
\x00E:{epoch_id} → Epochs
\x00SUPERSEDED:{epoch_id} → Supersession markers
\x00SUP:{hash} → Supersession records
\x00AUD:{query_id} → Audit records
\x00ESC:{ts}:{id} → Escalation events
\x00TP:{pack_id} → Trust packs
\x00META:{key} → System metadata
\x00HASH_SUBJECT:{hash} → Reverse lookup index
\x00SUBJECTS:{subject} → Known subjects index
\x00GS_LIST:{subj}:{pred} → Gold standard listing
```
- **Implementation:**
- [x] `key_codec.rs` (573 lines): 40+ key builder functions, subject validation, extraction utilities, 30+ unit tests.
- [x] All stores migrated to `key_codec::` functions (91 call sites across 10 store files, zero hardcoded key patterns).
- [x] Ingestion pipeline uses `key_codec` (11 usages across 3 files).
- [x] Query engine uses `key_codec` (34 usages across 7 files).
- [x] Subject co-location verified by `test_subject_colocation` test.
- [x] Global key sort-first verified by `test_global_keys_sort_first` test.
#### 5B. WAL Hardening
- [x] **5B.1 CRC32C Checksums**: Add hardware-accelerated torn write detection.
- **Status:** ✅ COMPLETE
- **Implementation:**
- [x] Added `crc32c = "0.6"` crate dependency.
- [x] Updated WAL record format with CRC32C in `format.rs`.
- [x] CRC32C verified on read before deserializing.
- [x] Hardware-accelerated via SSE 4.2 on supported CPUs.
- [x] **5B.2 Crash Recovery Implementation**: Replace recovery stub with production recovery.
- **Status:** ✅ COMPLETE
- **Implementation:**
- [x] `recovery/mod.rs` (236 lines): Full sequential record scan with CRC32C validation.
- [x] Truncates file at first corrupted/incomplete record.
- [x] `RecoveryMetrics` struct tracks: valid_records, invalid_records, bytes_truncated, recovery_duration.
- [x] Comprehensive test suite in `recovery/tests.rs`.
- [x] **5B.3 Group Commit**: Batch fsync for throughput.
- **Status:** ✅ COMPLETE
- **Implementation:**
- [x] `group_commit.rs` (342 lines): `GroupCommitBuffer` with configurable `max_writes` and `max_duration`.
- [x] Writers append to buffer and wait on `Notify`.
- [x] Background flusher calls fsync and notifies all waiters.
- [x] `GroupCommitConfig` for tuning batch size and flush interval.
- [x] **5B.4 Log Rotation**: Bounded WAL disk usage.
- **Status:** ✅ COMPLETE
- **Implementation:**
- [x] `segment.rs` (368 lines): Full segment management.
- [x] Segment naming: `{seq:08x}.wal`.
- [x] Rotation when segment exceeds configurable threshold.
- [x] `SegmentManager` tracks active and archived segments.
- [x] Safe deletion of segments after cursor passes them.
#### 5C. Index Persistence
- [x] **5C.1 Persistent Vector Index**: Move HNSW from in-memory to disk-backed.
- **Status:** ✅ COMPLETE
- **Implementation:**
- [x] `PersistentVectorIndex` in `crates/stemedb-storage/src/vector_index/persistent.rs`.
- [x] Hybrid hot/cold architecture:
- **Hot:** In-memory HNSW for recent vectors.
- **Cold:** Disk-backed HNSW loaded from checkpoint files.
- [x] `persistence/format.rs`: `SnapshotMetadata`, `IdMappingTable` with rkyv serialization.
- [x] `hot_cold.rs`: `merge_search_results()` for combining hot and cold query results.
- [x] Background checkpoint task with atomic write pattern.
- [x] CRC32C integrity verification on load.
- [x] MAX_PAYLOAD_SIZE (1GB) validation to prevent memory exhaustion.
- **Crates:** `memmap2 = "0.9"`, `crc32c = "0.6"`, `byteorder = "1.5"`
- **Known Limitation:** Cold index currently stores ID mappings only; full vector persistence with mmap'd HNSW graphs planned for future phase.
- [x] **5C.2 Persistent Visual Index**: Persist BK-tree to disk.
- **Status:** ✅ COMPLETE
- **Implementation:**
- [x] `PersistentVisualIndex` in `crates/stemedb-storage/src/visual_index.rs`.
- [x] `BkTreeSnapshot` with rkyv serialization for BK-tree state.
- [x] Checkpoint file format: `[MAGIC:4][VERSION:1][RESERVED:3][PAYLOAD_LEN:u64][CRC32C:u32][PAYLOAD]`.
- [x] Atomic write pattern: temp file → fsync → rename → fsync parent.
- [x] Background checkpoint task with configurable interval.
- [x] CRC32C integrity verification on load.
- [x] Shared `checkpoint_format.rs` module for common read/write utilities.
#### 5D. Concept Hierarchy
> **Spec:** [docs/specs/concept-hierarchy.md](docs/specs/concept-hierarchy.md)
> **Purpose:** Hierarchical, scheme-qualified subject identifiers with cross-scheme alias resolution. Enables applications like Aphoria that need to connect `code://` paths to `rfc://` paths.
- [ ] **5D.1 ConceptPath Type**: Structured subject identifiers.
- **Tasks:**
- [ ] Add `ConceptPath` struct to `stemedb-core/src/types/concept.rs`.
- [ ] Wire format: `{scheme}://{segment_0}/{segment_1}/.../{leaf}`.
- [ ] `parse()`, `to_string()`, `leaf()`, `parent()`, `is_prefix_of()`.
- [ ] Backward-compatible: bare strings parse as `custom://{string}`.
- [ ] Unit tests for parsing, round-trip, prefix matching.
- **Crate:** `stemedb-core`
- [ ] **5D.2 Source Scheme Registry**: Map schemes to default source tiers.
- **Tasks:**
- [ ] Add `SourceScheme` enum to `stemedb-core`.
- [ ] Scheme → default `SourceClass` mapping (e.g., `rfc://` → Tier 0, `code://` → Tier 3).
- [ ] `ConceptPath::default_source_class()` method.
- **Crate:** `stemedb-core`
- [ ] **5D.3 Alias Store**: Cross-scheme entity resolution.
- **Tasks:**
- [ ] Add `ConceptAlias` struct to `stemedb-core`.
- [ ] Add `AliasStore` trait to `stemedb-storage`.
- [ ] Key prefixes: `CA:{alias_path}` → canonical, `CAR:{canonical}` → all aliases.
- [ ] Transitive alias resolution.
- [ ] `GenericAliasStore` implementation over `KVStore`.
- **Crates:** `stemedb-core`, `stemedb-storage`
- [ ] **5D.4 Hierarchical Query**: Prefix-based subject queries.
- **Tasks:**
- [ ] Add `hierarchical: bool` to `QueryParams`.
- [ ] `fetch_by_subject_prefix()` using `scan_prefix` in query engine.
- [ ] Trailing `/` handling to prevent `auth` matching `authentication`.
- **Crate:** `stemedb-query`
- [ ] **5D.5 Alias Resolution in Queries**: Expand queries to aliased paths.
- **Tasks:**
- [ ] Add `resolve_aliases: bool` to `QueryParams`.
- [ ] Resolve aliases before candidate fetch.
- [ ] Merge results from all aliased paths, deduplicate by hash.
- **Crate:** `stemedb-query`
- [ ] **5D.6 Source Class Inference**: Infer tier from scheme at ingestion.
- **Tasks:**
- [ ] If no explicit `source_class`, infer from `ConceptPath` scheme.
- [ ] `rfc://` → Regulatory (Tier 0), `code://` → Expert (Tier 3), etc.
- **Crate:** `stemedb-ingest`
- [ ] **5D.7 Concept API Endpoints**: CRUD for aliases and hierarchy browsing.
- **Tasks:**
- [ ] `POST /v1/concepts/alias` — Create alias.
- [ ] `GET /v1/concepts/aliases/{path}` — List aliases for a path.
- [ ] `DELETE /v1/concepts/alias` — Remove alias.
- [ ] `GET /v1/concepts/tree/{prefix}` — Browse hierarchy under prefix.
- [ ] `GET /v1/concepts/suggest` — Suggested aliases (shared leaf detection).
- **Crate:** `stemedb-api`
- [ ] **5D.8 Battery Tests**: Validate concept hierarchy end-to-end.
- **Tasks:**
- [ ] Battery 7: ConceptPath parsing round-trip, backward compat.
- [ ] Battery 8: Alias resolution (query `code://x/y/z` returns aliased `rfc://a/b/z`).
- [ ] Battery 9: Source class inference from scheme.
- [ ] Battery 10: Cross-scheme conflict score (`code://` Tier 3 vs `rfc://` Tier 0).
- **Crate:** `stemedb-query/tests/battery_pre_sentinel.rs`
### Phase 6: The Mesh (Distributed Writes)
*Goal: Multi-node cluster with CRDT replication and Raft coordination. The endgame.*
> **Research:** [docs/research/distributed-write-path.md](docs/research/distributed-write-path.md)
> **Agent:** `distributed-systems-engineer`
> **Key Insight:** Episteme's append-only model eliminates ~75% of CockroachDB complexity. Assertions are a G-Set CRDT. Votes are G-Counters. No distributed transactions needed.
#### 6A. CRDT Foundation (Single-Node Validation)
- [ ] **6A.1 Integrate CRDT Crate**: Wrap assertion storage in G-Set semantics.
- **Tasks:**
- [ ] Add `crdts = "7.4"` dependency to `stemedb-storage`.
- [ ] Implement `CrdtAssertionStore` wrapping assertions as `GSet<Hash>`.
- [ ] Implement `CrdtVoteStore` wrapping votes as `GCounter<(Hash, [u8; 32])>`.
- [ ] Property tests: commutativity (`merge(A,B) == merge(B,A)`), associativity, idempotence.
- [ ] Verify existing tests still pass with CRDT wrapper.
- [ ] **6A.2 Hybrid Logical Clocks**: Add causal ordering to supersessions.
- **Tasks:**
- [ ] Add `uhlc = "0.7"` dependency to `stemedb-core`.
- [ ] Replace `timestamp: u64` in `Supersession` with `hlc_timestamp: uhlc::Timestamp`.
- [ ] Update `IngestWorker` to generate HLC timestamps.
- [ ] Update `EpochAwareLens` to use HLC comparison for ordering.
- [ ] Test: concurrent supersessions from different nodes converge to same order.
- [ ] **6A.3 Merkle Tree Over Assertions**: Efficient diff detection.
- **Tasks:**
- [ ] Implement `MerkleTree` over assertion hashes using BLAKE3.
- [ ] Incremental update: insert new hash, recompute affected path.
- [ ] Root comparison: O(1) check if two nodes have same assertions.
- [ ] Recursive diff: O(log N) to find divergent subtrees.
- [ ] Serialize tree state for exchange over network.
#### 6B. Two-Node Replication (Proof of Concept)
- [ ] **6B.1 RPC Layer**: Node-to-node communication.
- **Tasks:**
- [ ] Create `stemedb-rpc` crate.
- [ ] Define protobuf messages: `SyncRequest`, `SyncResponse`, `FetchAssertions`, `GossipBroadcast`.
- [ ] Implement gRPC services with `tonic`.
- [ ] Connection pooling and retry with exponential backoff.
- **Crates:** `tonic = "0.12"`, `prost = "0.13"`
- [ ] **6B.2 Gossip Broadcast**: Push new assertions to peers.
- **Tasks:**
- [ ] On write: gossip new assertion hash + data to N peers (fanout = 3-5).
- [ ] Peers merge into local G-Set.
- [ ] Deduplicate: content-addressed hashes mean receiving same assertion twice is a no-op.
- [ ] Track gossip metrics: `gossip_messages_sent`, `gossip_duplicates_received`.
- [ ] **6B.3 Merkle Anti-Entropy Sync**: Background convergence.
- **Tasks:**
- [ ] Every 60 seconds per peer: exchange Merkle roots.
- [ ] If roots differ: recursive diff to find missing hashes.
- [ ] Fetch missing assertions from peer.
- [ ] Merge into local store + trigger MV recompute.
- [ ] Track: `sync_lag_seconds`, `merkle_diff_size`, `convergence_latency_p99`.
- [ ] **6B.4 Integration Test: Two-Node Convergence**:
- [ ] Write assertion to Node A → appears on Node B within 5 seconds.
- [ ] Write to Node A during partition → Node B converges after healing.
- [ ] Concurrent writes to both nodes → both converge to same state.
#### 6C. Multi-Node Cluster
- [ ] **6C.1 Cluster Membership (SWIM Gossip)**: Node discovery and failure detection.
- **Tasks:**
- [ ] Add `memberlist = "0.4"` dependency.
- [ ] Implement `ClusterMembership` with SWIM protocol.
- [ ] Seed-node based discovery (bootstrap nodes in config).
- [ ] Failure detection: ping, indirect probe, suspicion.
- [ ] Membership change events trigger anti-entropy with new peers.
- **Crate:** `memberlist = "0.4"`
- [ ] **6C.2 Subject-Prefix Range Sharding**: Distribute data across nodes.
- **Tasks:**
- [ ] Implement `RangeRouter`: map subject → range → node.
- [ ] Range descriptor: start key, end key, replica nodes.
- [ ] Automatic range split when size exceeds 64MB threshold.
- [ ] Range merge when adjacent ranges shrink below 20MB.
- [ ] Meta-range: store range descriptors, gossip to all nodes.
- [ ] **6C.3 Raft for MV Coordination (Optional)**: Deterministic MV computation.
- **Problem:** Without ordering, different nodes may compute different MV winners during convergence.
- **Solution:** Lightweight Raft group per subject-range for MV coordinator election.
- **Tasks:**
- [ ] Add `openraft = "0.10"` dependency.
- [ ] Implement `RaftLogStorage` backed by fjall.
- [ ] Implement `RaftStateMachine` delegating to `Materializer`.
- [ ] Leader coordinates MV recomputation order.
- [ ] Followers serve reads from local MVs.
- **Note:** This is optional. Without Raft, MVs are eventually consistent (converge once assertions sync). With Raft, MVs are strongly consistent per range.
- **Crate:** `openraft = "0.10"`
- [ ] **6C.4 Gateway**: Stateless request routing.
- **Tasks:**
- [ ] Implement `Gateway` HTTP service (axum).
- [ ] Route writes by subject → range → node.
- [ ] Route reads to nearest replica.
- [ ] Health checking and failover.
- [ ] Load balancing across replicas.
#### 6D. Consistency Guarantees
| Property | Guarantee | Mechanism |
|----------|-----------|-----------|
| **Convergence** | Eventually consistent | G-Set merge (CRDT) |
| **Causality** | Supersessions ordered | HLC timestamps |
| **Partition Tolerance** | Writes never blocked | Any node accepts via CRDT |
| **Availability** | Reads/writes always succeed | Every node is master for CRDTs |
| **Durability** | WAL + fsync per node | Existing WAL infra |
| **Conflict Resolution** | Deterministic | Lens algorithms (same inputs → same output) |
### Phase 7: The Shield (Trust at Scale)
*Goal: Defend against spam, Sybil attacks, and knowledge poisoning when open to millions of agents.*
> **Key Insight:** Open agent access requires layered defense. PoW for admission, EigenTrust for reputation, circuit breakers for misbehavior, quarantine for suspicious content.
#### 7A. Admission Control
- [ ] **7A.1 Proof-of-Work Admission**: BLAKE3 hashcash for new agents.
- New agents (trust < 0.3) must solve PoW puzzle before first N assertions accepted.
- Graduated difficulty: first 10 assertions = 16 sec, 11-50 = 1 sec, 50+ or trust > 0.6 = none.
- Puzzle: `BLAKE3(nonce || agent_id || timestamp)` must have `difficulty` leading zero bits.
- [ ] **7A.2 Graduated Trust Tiers**: Privilege escalation based on reputation.
- Untrusted (0.0-0.3): Quarantine mode, assertions hidden by default, 0.1x quota.
- Limited (0.3-0.5): Low quota, PoW required, 0.5x quota.
- Verified (0.5-0.7): Standard quota, normal privileges.
- Trusted (0.7-0.9): 2x quota, skip PoW.
- Authority (0.9-1.0): 10x quota, no limits.
#### 7B. EigenTrust
- [ ] **7B.1 Trust Graph Store**: Store direct trust relationships for propagation.
- Key pattern: `TG:{agent_a}:{agent_b}` → trust weight (0.0-1.0).
- Methods: `add_trust_edge()`, `get_trusted_by()`, `compute_eigentrust()`.
- [ ] **7B.2 EigenTrust Computation**: Global trust via power iteration (daily batch).
- Formula: `T = (1-α)CT + αP` where C = normalized trust matrix, P = seed trust, α = 0.1.
- Convergence: 10-20 iterations for 10K agents.
- Sybil resistance: isolated rings have low trust unless connected to real agents.
- Crates: `petgraph` (graph structures), `nalgebra` (linear algebra).
- [ ] **7B.3 Domain-Specific Trust**: Per-predicate-namespace reputation.
- Agent may be expert in medicine but novice in astronomy.
- Track accuracy by predicate namespace.
- Lens can weight by domain trust.
#### 7C. Content Defense
- [ ] **7C.1 MinHash Deduplication**: Near-duplicate detection with LSH bucketing.
- Compute MinHash signature for `{subject}:{predicate}` pairs.
- LSH buckets for O(1) average-case lookup.
- Bloom filter pre-check for fast "definitely not duplicate" path.
- Threshold: 0.9 Jaccard similarity = duplicate.
- [ ] **7C.2 Content Quality Scoring**: Heuristic-based spam detection.
- Shannon entropy check (high entropy = likely random noise).
- Minimum subject/predicate length.
- Structured data bonus (JSON objects, numbers, URLs).
- Untrusted agent + high confidence = suspicious.
- [ ] **7C.3 Quarantine Store**: Suspicious assertions held for review.
- Key pattern: `QUAR:{hash}` → assertion data.
- Quarantined assertions NOT indexed (invisible to queries).
- Triggers: quality < 0.4, duplicate, untrusted high-confidence.
- Admin API: `GET /v1/admin/quarantine`, `POST /v1/admin/quarantine/{hash}/approve`.
#### 7D. Circuit Breakers
- [ ] **7D.1 Per-Agent Circuit Breakers**: Ban misbehaving agents temporarily.
- States: Closed (normal), Open (banned), HalfOpen (testing recovery).
- 5 failures → Open for 30 seconds → HalfOpen → 1 success → Closed.
- Metrics: `circuit_breaker_state`, `circuit_breaker_trips_total`.
- Crate: `governor` for GCRA rate limiting.
### Phase 8: The Swarm (Production Cluster)
*Goal: Production-ready distributed deployment with chaos testing and observability.*
#### 8A. Chaos Testing
- [ ] **8A.1 Partition Testing**: Verify convergence after network partitions.
- 5-node cluster, kill 2 nodes, verify remaining nodes converge.
- Network partition between groups, verify both sides accept writes, verify convergence after healing.
- Message reordering and duplication tests.
- [ ] **8A.2 Jepsen-Style Consistency Testing**: Formal verification.
- Linearizability checks for Raft-coordinated operations.
- Eventual consistency checks for CRDT-replicated data.
- Clock skew injection (verify HLC handles drift).
#### 8B. Observability
- [ ] **8B.1 Distributed Metrics**: Per-node, per-range, per-agent metrics.
- `sync_lag_seconds{peer}`, `merkle_diff_size{peer}`, `convergence_latency_p99`.
- `raft_leader_changes{range}`, `raft_commit_latency{range}`.
- `assertions_total{node}`, `writes_per_second{node}`.
- Crate: `metrics` + `metrics-exporter-prometheus`.
- [ ] **8B.2 Admin Dashboard**: Cluster health visibility.
- `GET /v1/admin/cluster` → node list, range assignments, leader locations.
- `GET /v1/admin/ranges` → range sizes, split/merge history.
- `POST /v1/admin/sync` → force anti-entropy sync.
#### 8C. Production Hardening
- [ ] **8C.1 Snapshot/Restore**: Fast replica bootstrap.
- Serialize full node state as snapshot.
- New nodes join by restoring snapshot + replaying recent WAL.
- Faster than full Merkle sync for new nodes.
- [ ] **8C.2 Backpressure**: Don't overwhelm slow nodes.
- Track per-peer sync queue depth.
- Throttle gossip to slow peers.
- Alert when peer is consistently behind.
- [ ] **8C.3 Geo-Distribution**: Multi-region deployment.
- Regional clusters with CRDT federation between regions.
- Lazy cross-region sync (background, low priority).
- Locality-aware reads (query nearest replica).
- Regional compliance (GDPR data residency).
---
## Tracking
### Active Tasks
* [x] **Phase 3 The Pilot**: Consumer Health vertical integration. ✅ COMPLETE
* [x] **Phase 4 The Hive**: Trust & Scale + Extension Primitives. ✅ COMPLETE
* [ ] **Phase 5 The Forge**: Foundation hardening — replace sled, fix WAL, persist indices.
* [x] **5A.1**: Replace sled with redb/fjall (HybridStore). ✅ COMPLETE
* [x] **5A.2**: Key layout redesign with subject-prefix co-location (`key_codec.rs`). ✅ COMPLETE
### Next Up
* **Phase 5B.2**: Implement real crash recovery (current recovery is a stub).
* **Phase 5B.3**: Group commit for WAL throughput.
* **Phase 6**: Distributed writes via CRDT replication + Raft coordination.
* **Phase 7A-7B** (Extension blocker): PoW admission + EigenTrust for Phase 2 extension launch.
### App Layer (External)
* **Browser Extension Phase 1** (Read-Only Overlay) -> All DB dependencies complete. Extension is app layer.
* **Browser Extension Phase 2** (Active Layer / Vote-to-See) -> Blocked by Phase 7A-7B (Sybil defense).
* **The Simulator** (Training Data Pipeline) -> App layer, consumes Episteme API.
* **The Super Curator** (Reviewer Agent swarm) -> App layer.
* **Background Gardener** (Cluster detection, signal processing) -> App layer.
* **Agent Wallet** (Key management sidecar) -> App layer.
### Recently Completed
* [x] **Gold Standard Verification** (4.7): Sybil defense via proof of knowledge.
* `GoldStandard` struct with rkyv serialization, `GoldStandardStore` trait + implementation.
* `TrustAdjustment` enum: Rewarded(+0.05), Penalized(-0.1), AlreadyVerified.
* Anti-gaming: agents can only verify each gold standard once (dedup markers).
* API: `POST/GET/DELETE /v1/admin/gold-standards`, `POST /v1/admin/verify-agent`.
* 21 trust rank tests including 4 new verification tests.
* [x] **Escalation Triggers** (4.6): Active safety system for high-conflict assertions.
* `EscalationPolicy` with configurable conflict_score thresholds and predicate matching.
* `EscalationEvent` (content-addressed BLAKE3 ID) written by Materializer when policies fire.
* `EscalationStore` at `ESC:{timestamp_nanos}:{id_hex}` with resolve workflow.
* `GET /v1/admin/escalations`, `POST /v1/admin/escalations/{id}/resolve`.
* [x] **Conflict Score Filtering** (4.5): Query-time filtering by conflict score.
* `min_conflict_score` and `max_conflict_score` on Query + QueryParams.
* Fast-path MV filtering with 0.0-1.0 validation.
* 13 engine tests covering thresholds, boundaries, ranges, combinations.
* [x] **Vote Provenance Witness** (4.4): Transform votes into cryptographic witnesses.
* `source_url: Option<String>` and `observed_context: Option<Vec<u8>>` on Vote.
* Input validation: URL max 2048 chars, context max 64KB.
* Backward compatible with existing votes.
* [x] **"Since" Parameter** (4.1): Change tracking for returning consumers.
* `since: Option<u64>` on Query for incremental sync.
* MV Changelog at `MVC:{subject}:{predicate}:{timestamp_nanos}` with nanosecond precision.
* `ChangeEntry` struct with `previous_winner_hash`, `new_winner_hash`, `lens_name`.
* `fetch_changes_since()` in QueryEngine with timestamp filtering.
* `changes_since: Option<Vec<ChangeEntryDto>>` on QueryResponse.
* 6 tests covering changelog creation, fetching, and fast path bypass.
* [x] **Batch TrustRank Decay API** (4.3): Admin endpoint for scheduled trust decay.
* `POST /v1/admin/decay-trust-ranks` with optional `now` and `half_life_seconds`.
* Returns `decayed_count`, `timestamp_used`, `half_life_used`, `status`.
* 3 integration tests.
### Recently Completed
* [x] **Source Metadata Indexing** (4.2): Index key source metadata fields for filtered queries.
* New module: `crates/stemedb-storage/src/source_metadata_index.rs`.
* Indexed fields: `journal`, `doi`, `platform`, `study_design`.
* Key pattern: `SMV:{field}:{value}` -> `Vec<Hash>` (case-insensitive).
* Query filters: `?source_journal=NEJM&source_platform=PubMed`.
* AND semantics for multiple metadata filters.
* 18 unit tests covering indexing and query filtering.
* [x] **Source Document Storage** (3F.1): Provenance lookup for 100% citation recall.
* `POST /v1/source` stores source documents by BLAKE3 content hash.
* `GET /v1/provenance/{hash}` retrieves source documents.
* Content-addressed storage at `SRC:{hash}` keys.
* Base64 encoding, 10MB limit, idempotent uploads.
* 5 unit tests covering happy path and error cases.
* [x] **Epoch Cascade Logic** (3D.1): O(1) supersession lookup via pre-computed markers.
* `write_supersession_cascade()` writes `SUPERSEDED:` markers for full transitive closure at ingest time.
* `is_epoch_superseded()` uses O(1) marker lookup instead of chain walking.
* Cycle detection and max depth guard (100 levels).
* 5 tests covering markers, transitive closure, cycles, and marker-based filtering.
* [x] **Semantic Decay** (3B.2): Confidence half-life at query time.
* `decay_halflife: Option<u64>` and `source_class_decay: bool` on Query.
* New `decay.rs` module with `apply_decay()` and `apply_source_class_decay()`.
* Formula: `effective_confidence = confidence * 2^(-(age / halflife))`.
* Tier-specific decay: Regulatory=none, Clinical=2yr, Anecdotal=30d.
* 11 unit tests + 1 E2E integration test.
* [x] **Layered Consensus Lens** (3C.2): Per-source-class consensus with tier-by-tier visibility.
* `LayeredConsensusLens` with `LayeredLens` trait.
* `TierResolution` and `LayeredResolution` types.
* `GET /v1/layered?subject=X&predicate=Y` endpoint.
* Cross-tier conflict score using Shannon entropy.
* 10 comprehensive tests.
* [x] **Time-Travel Engine** (3B.1): `as_of` parameter for historical queries.
* `as_of: Option<u64>` field on `Query` for querying historical state.
* Bypasses fast path (MVs reflect current state).
* `Query::matches()` filters by `assertion.timestamp <= as_of`.
* 6 tests covering edge cases.
* [x] **Rich Source Metadata** (3A.3): Structured provenance beyond `source_hash`.
* `source_metadata: Option<Vec<u8>>` field on `Assertion`.
* `Vec<u8>` for rkyv zero-copy compatibility, callers handle JSON encoding.
* Builder methods: `.source_metadata_json()` and `.source_metadata()`.
* API exposes as `Option<String>` with defensive UTF-8 handling.
* 2 serialization tests.
* [x] **Conflict Score on Resolution** (3A.2): Numeric disagreement metric across all lenses.
* `conflict_score: f32` field on `Resolution` (0.0 = unanimous, 1.0 = max conflict).
* `compute_conflict_score()` utility using normalized variance.
* Updated all 5 lens implementations to compute and propagate conflict score.
* `MaterializedView` now stores conflict score.
* API `QueryResponse` exposes `conflict_score` and `resolution_confidence` when lens is applied.
* 7 unit tests including NaN handling.
* [x] **SkepticLens + SkepticView** (3C.1): "Trust but Verify" conflict analysis that surfaces all claims with conflict scores.
* `AnalysisLens` trait for lenses that map conflict instead of resolving it.
* `SkepticLens` using normalized Shannon entropy for conflict scoring.
* `SkepticResolver` + `SkepticView` in stemedb-query.
* `GET /v1/skeptic?subject=X&predicate=Y` API endpoint.
* Types: `ResolutionStatus`, `ConflictAnalysis`, `ClaimSummary`, `SourceSummary`, `AgentSummary`.
* [x] **Source-Class Field** (3A.1): 6-tier `SourceClass` enum with authority weighting and decay rates.
* `SourceClass` enum: Regulatory, Clinical, Observational, Expert, Community, Anecdotal.
* `tier()`, `default_decay_days()`, `authority_weight()` methods.
* Field on Assertion struct with full serialization support.
* [x] **The Meter**: Token bucket quota system with MeterLayer middleware (10K tokens/agent/hour).
* [x] **Query Audit Trail**: Every query logged with provenance at `AUD:{query_id}`. `X-Agent-Id` header for attribution.
* [x] **Event-Driven Materialization**: `run_notified()` + IngestWorker Notify integration.
* [x] **Fast-Path MV Lookup**: `QueryEngine::try_fast_path()` for O(1) reads.
* [x] **Materializer**: Background worker for O(1) MV reads via `AsyncLens`.
* [x] **VoteAwareConsensusLens**: Real vote-based consensus resolution.
* [x] **Compound SP Index**: O(1) subject+predicate lookups.
* [x] **TrustRank System**: Agent reputation with decay and learning loop.
* [x] **API Surface**: axum HTTP server with 7 endpoints + OpenAPI docs.
### Research Documents
* [docs/research/wal-crash-recovery-research.md](docs/research/wal-crash-recovery-research.md) — WAL patterns from CockroachDB, TiKV, FoundationDB, SQLite.
* [docs/research/distributed-write-path.md](docs/research/distributed-write-path.md) — Spanner/CockroachDB-style distributed writes adapted for append-only model.
### Key Architectural Decisions
* **sled → redb/fjall**: sled is abandoned. HybridStore routes by key prefix: redb for reads, fjall for writes. ✅ COMPLETE
* **Raft log = WAL**: TiKV eliminated duplicate WAL in v5.4. We should too.
* **CRDT for data, Raft for coordination**: Assertions are a G-Set CRDT (merge = set union). Only cluster metadata needs Raft.
* **Subject-prefix ranges**: Co-locate all data for a subject on one shard. Split hot subjects via range split.
* **HLC over TrueTime**: Hybrid Logical Clocks work on commodity hardware. No GPS/atomic clocks needed.
* **AP model**: Writes never blocked during partitions. Eventual consistency via CRDT convergence.
### Blockers
* **Phase 5**: None. Can start immediately.
* **Phase 6**: Blocked by Phase 5 (need stable storage + key layout before distributing).
* **Phase 7**: Blocked by Phase 6 (trust at scale requires distributed infra).
* **Phase 8**: Blocked by Phase 6 + 7 (chaos testing requires working cluster).
---
## Dependency Graph
```
Phase 2.5 (Hardening) Phase 3 (The Pilot) Phase 4 (The Hive)
======================== ======================== ==================
[2.1 MV Staleness] ---------> [3B.1 Time-Travel] ✅ --+
| |
[2.2 Confidence Rename] -----> (API clarity for all) +----------> [4.1 "Since" Param] ✅
|
[2.3 EpochAwareLens] --------> [3D.1 Epoch Cascade] ✅ |----------> Invalidation Cascades pillar
|
[2.4 Visual Hash Query] -----> [3E.2 Visual Hash Index] ✅
|
[2.6 E2E Integration] -------> (pipeline confidence) |
|
[3A.1 Source-Class] ✅ --+----------> [3C.2 Layered Consensus] ✅
| [3B.2 Semantic Decay] ✅
+-----------------------------[4.2 Metadata Indexing] ✅
|
[3A.2 Conflict Score] ✅ --> (enhance Resolution)
|
[3A.3 Source Metadata] ✅ --> [4.2 Metadata Indexing] ✅
|
[3C.1 Skeptic Lens] ✅ (standalone, COMPLETE)
[3C.3 Constraints Lens] ✅ (standalone, COMPLETE)
[3E.1 Vector Search] ✅ (standalone, COMPLETE)
[3E.2 Visual Hash Index] ✅ (standalone, COMPLETE)
[3F.1 Provenance] ✅ (standalone, COMPLETE)
```
### Critical Path for Consumer Health Demo
```
[3A.1 Source-Class] ✅ --> [3A.2 Conflict Score] ✅ --> [3C.2 Layered Consensus] ✅
|
+----> CONSUMER HEALTH MVP ✅
|
[3B.1 Time-Travel] ✅ ---------------------------------------+
|
[3A.3 Source Metadata] ✅ -----------------------------------+
|
[3C.1 Skeptic Lens] ✅ --------------------------------------+
```
### Critical Path for Financial DD Demo
```
[3A.2 Conflict Score] ✅ --> [3C.1 Skeptic Lens] ✅ ---+
|
[3B.1 Time-Travel] ✅ ----------------------------------+----> FINANCIAL DD MVP
|
[2.3 EpochAwareLens] --> [3D.1 Epoch Cascade] ✅ -------+
|
[3B.2 Semantic Decay] ✅ -------------------------------+
```
### Critical Path for Agile Agent Team Demo
```
[3C.3 Constraints Lens] (standalone) ------+
|
[3B.1 Time-Travel] ✅ --------------------+----> AGENT TEAM MVP
|
[2.3 EpochAwareLens] ✅ ------------------+
|
[Query Audit (Phase 2)] ✅ ----------------+
```
### Critical Path for Browser Extension
```
Phase 3 (Data Foundation) Phase 4 (Extension Primitives) Extension Product
========================= ============================== =================
[3A.1 Source-Class] ✅ ──────────> [4.5 Conflict Filtering] ✅ ──┐
|
[3A.2 Conflict Score] ✅ ─────────────────────────────────────────┤
|
[3C.1 Skeptic Lens] ✅ ──────────────────────────────────────────-┤
├──> PHASE 1: Read-Only Overlay
[3B.2 Semantic Decay] ✅ ────────────────────────────────────────-┤ (Benign Layer)
|
[3C.2 Layered Consensus] ✅ ─────────────────────────────────────-┘
[Phase 2 Vote System] ✅ ────────> [4.4 Vote Provenance] ✅ ─────┐
|
[TrustRank Engine] ✅ ───────────> [4.7 Gold Standards] ✅ ───────┤
├──> PHASE 2: Active Layer
[3A.2 Conflict Score] ✅ ────────> [4.6 Escalation Triggers] ✅ ──┤ (Vote to See)
|
[7A PoW Admission] ─────────────┤
[7B EigenTrust] ────────────────┘
```
**Phase 1 (Read-Only)** requires: Source tiers, conflict scores, conflict filtering, skeptic lens, decay, layered consensus. **All complete.**
**Phase 2 (Active)** requires: Vote provenance, gold standards, escalation triggers (all complete), PLUS Phase 7 Sybil defense (PoW, EigenTrust — not started).
### Critical Path to Distributed Cluster
```
Phase 5 (The Forge) Phase 6 (The Mesh) Phase 7+8
======================= ======================= ==================
[5A.1 Replace sled ✅] ───────────> [6A.1 CRDT Foundation] ──┐
| |
[5A.2 Key Layout] ───────────────> [6C.2 Range Sharding] ──> |
|
[5B.1 CRC32C Checksums] ──┐ |
[5B.2 Crash Recovery] ────┼──────> [6B.1 RPC Layer] ─────────┤
[5B.3 Group Commit] ──────┘ | |
v |
[5C.1 Persistent Vector] ─────── (independent, no blocker) |
[5C.2 Persistent Visual] ─────── (independent, no blocker) |
|
[6A.2 HLC Timestamps] ────┤
[6A.3 Merkle Tree] ───────┤
| |
v v
[6B.2 Gossip] ──> [6B.3 Anti-Entropy] ──> [6B.4 Two-Node Test]
|
v
[6C.1 SWIM Membership] ──> [6C.3 Raft MV Coord]
[6C.4 Gateway] ──────────> │
v
DISTRIBUTED CLUSTER
|
[7A PoW Admission] ──┐
[7B EigenTrust] ─────┤──> THE SHIELD
[7C Content Defense] ┤
[7D Circuit Breakers]┘
|
[8A Chaos Testing] ──┐
[8B Observability] ──┤──> THE SWARM
[8C Geo-Distribution]┘
```
### New Crates (Phases 5-8)
```
stemedb-rpc (Phase 6B) ── gRPC services for node-to-node communication
stemedb-cluster (Phase 6C) ── Cluster membership, range routing, gateway
stemedb-sync (Phase 6B) ── Merkle sync, gossip broadcast, anti-entropy
```
Reference in New Issue View Git Blame Copy Permalink