1ce4004807
This commit adds the read path (Cortex) to complement the write path (Spine): ## Crates - stemedb-api: HTTP API with axum + utoipa OpenAPI - /v1/assert, /v1/query, /v1/epoch, /v1/skeptic, /v1/trace, /v1/audit - Metered endpoints with quota enforcement - Ed25519 signature verification - stemedb-lens: Truth resolution lenses - RecencyLens, ConsensusLens, ConfidenceLens - VoteAwareConsensusLens (Ballot Box pattern) - TrustAwareAuthorityLens (The Hive pattern) - SkepticLens (conflict analysis) - EpochAwareLens (paradigm-safe queries) - stemedb-query: Query engine with materialized views ## Storage Extensions - VoteStore: Vote aggregation with cached counts - TrustRankStore: Agent reputation with decay - AuditStore: Query audit trail - IndexStore: SP/P/S index structures - SupersessionStore: Epoch supersession chains ## SDKs - sdk/go/steme: Go HTTP client with Ed25519 signing - sdk/go/adk: ADK-Go tools for AI agents ## Documentation - Updated CLAUDE.md, architecture.md, roadmap.md - New ai-lookup entries for all services - Use case docs for consumer health intelligence - Arena roadmap for simulation advancement Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
597 lines
40 KiB
Markdown
597 lines
40 KiB
Markdown
# Episteme (StemeDB) Roadmap
|
|
|
|
> **Goal:** Build the "Git for Truth" substrate for autonomous AI research.
|
|
> **Current Phase:** Phase 2.5 (Hardening)
|
|
> **Target Vertical:** BioTech/Pharma ("The Living Review")
|
|
|
|
---
|
|
|
|
## High-Level Timeline
|
|
|
|
| Phase | Codename | Focus | Key Deliverable |
|
|
| :--- | :--- | :--- | :--- |
|
|
| **1** | **The Spine** | Storage & Safety | Append-only WAL + KV Store |
|
|
| **2** | **The Lattice** | Indexing & Async | Materialized Views + Ballot Box |
|
|
| **2.5** | **Hardening** | Camp 2 Fixes | MV staleness, epoch behavior, lens cleanup |
|
|
| **3** | **The Pilot** | Vertical Integration | Pharma Ingestion + Living Review Agent |
|
|
| **4** | **The Hive** | Trust & Learning | The Simulator + Super Curator |
|
|
|
|
---
|
|
|
|
## Detailed Milestones
|
|
|
|
### Phase 1: The Spine (Foundation)
|
|
*Goal: Securely ingest assertions and persist them without data loss.*
|
|
|
|
- [x] **Project Scaffold**: Initialize Rust workspace, set up linting/CI (clippy, fmt).
|
|
- [x] **Assertion Schema**: Define the `Assertion` struct with `rkyv` serialization.
|
|
- [x] Add dependencies: `rkyv`, `blake3`, `ed25519-dalek`, `image_hasher`.
|
|
- [x] Define `Assertion` struct (Subject, Predicate, Object, Confidence, SourceHash).
|
|
- [x] **Multi-Sig Expansion**: Implement `SignatureEntry` struct and `signatures: Vec<SignatureEntry>` field.
|
|
- [x] **Visual Expansion**: Add `visual_hash: Option<pHash>` field for image provenance.
|
|
- [x] Test serialization round-trips.
|
|
- [x] **Ballot Schema**: Define the `Vote` struct for multi-agent consensus.
|
|
- [x] Add `Vote` struct: `assertion_hash`, `agent_id`, `weight`, `signature`.
|
|
- [x] Test serialization round-trips.
|
|
- [x] **Paradigm Schema (Epochs)**: Define the `Epoch` and `SupersessionType` structs.
|
|
- [x] Add `epoch: Option<EpochId>` to `Assertion`.
|
|
- [x] Implement `Epoch` struct with `supersedes` and `SupersessionType`.
|
|
- [x] Test serialization round-trips.
|
|
- [x] **WAL Integration**: Implement the Quarantine Pattern for write-ahead logging.
|
|
- [x] Create `stemedb-wal` crate.
|
|
- [x] Port `FsyncGuard` and `Record` logic from established durability patterns.
|
|
- [x] Implement Record format with BLAKE3 checksums and Headers.
|
|
- [x] Verify `fsync` behavior with tests.
|
|
- [x] **Storage Engine**: Implement the `Store` trait using `sled` (embedded KV).
|
|
- [x] Add `sled` dependency.
|
|
- [x] Define `KVStore` trait (put, get, delete, scan_prefix, flush).
|
|
- [x] Implement `SledStore` wrapper.
|
|
- [x] **Basic Ingestor**: Background worker that tails WAL and writes to KV.
|
|
- [x] Implement async loop reading from WAL.
|
|
- [x] Write deserialized assertions, votes, and epochs to `sled`.
|
|
- [x] Ed25519 signature verification during ingestion.
|
|
- [x] Maintains S: and SP: indexes on ingest.
|
|
- [x] Persistent cursor/checkpoint (resumes from `__CURSOR__:ingest` in KV store).
|
|
- [x] **Verification**: Crash recovery tests (write -> crash -> restart -> read).
|
|
- [x] Single and multi-record crash recovery.
|
|
- [x] Multiple crash cycles tested.
|
|
|
|
### Phase 2: The Lattice (Connectivity)
|
|
*Goal: Query data with sub-millisecond latency using Materialized Views.*
|
|
|
|
- [x] **Lifecycle Schema**: Add `LifecycleStage` to Assertion.
|
|
- [x] Define enum: `Proposed`, `UnderReview`, `Approved`, `Deprecated`, `Rejected`.
|
|
- [x] Update `Assertion` struct and serialization tests.
|
|
- [x] **The Ballot Box**: Implement high-velocity vote ingestion.
|
|
- [x] `VoteStore` trait and implementation.
|
|
- [x] `VoteAwareConsensusLens` for real vote-based resolution.
|
|
- [x] **Index Infrastructure**: Compound indexes for O(1) queries.
|
|
- [x] `IndexStore` trait with S: and SP: indexes.
|
|
- [x] `QueryEngine` smart routing (SP -> S -> scan).
|
|
- [x] **Materializer**: Background worker for O(1) Read Performance.
|
|
- [x] `MaterializedView` type in `stemedb-core`.
|
|
- [x] `Materializer` worker in `stemedb-query` with `step()` and `run()`.
|
|
- [x] Aggregates Votes via `VoteAwareConsensusLens` (or any `AsyncLens`).
|
|
- [x] Updates `MV:{Subject}:{Predicate}` with the winning Assertion + metadata.
|
|
- [x] Event-driven mode via `run_notified()` with `tokio::sync::Notify`.
|
|
- [x] Fast-path MV lookup in `QueryEngine::try_fast_path()`.
|
|
- [x] **The Meter**: Implement Economic Throttling (TAN).
|
|
- [x] `QuotaStore` trait and `GenericQuotaStore` implementation.
|
|
- [x] Token Bucket algorithm with per-agent per-hour quotas.
|
|
- [x] `MeterLayer` tower middleware for request cost tracking.
|
|
- [x] Cost model: Assert=10, Vote=1, Query=5+lens, +1/KB payload.
|
|
- [x] `GET /v1/meter/quota` endpoint to check remaining quota.
|
|
- [x] `POST /v1/meter/quota/limit` admin endpoint to set custom limits.
|
|
- [x] **API Surface**: `axum` HTTP server with OpenAPI (utoipa).
|
|
- [x] `POST /v1/assert` -> Accepts JSON, writes to WAL.
|
|
- [x] `POST /v1/vote` -> High-throughput vote endpoint.
|
|
- [x] `POST /v1/epoch` -> Create epoch with optional supersession.
|
|
- [x] `GET /v1/query` -> Subject/Predicate/Lens/Lifecycle/Epoch filtering.
|
|
- [x] `GET /v1/health` -> Health check with assertion count.
|
|
- [x] `GET /swagger-ui` -> Interactive API docs.
|
|
- [x] 5 lens types available: Recency, Consensus, Authority, VoteAwareConsensus, TrustAwareAuthority.
|
|
- [x] **Query Audit**: Log every read with provenance.
|
|
- [x] Define `QueryAudit` struct: query_id, agent_id, timestamp, params, result_hash, contributing_assertions.
|
|
- [x] Storage at `AUD:{query_id}` with agent index at `AUDA:{agent_id}:{timestamp}:{query_id}`.
|
|
- [x] `GET /v1/audit/queries` -> Returns history of agent decisions.
|
|
- [x] `GET /v1/audit/query/{id}` -> Full reasoning trace for a single query.
|
|
- [x] Auto-logging on every query via `X-Agent-Id` header.
|
|
|
|
### Phase 2.5: Hardening (Camp 2 Fixes)
|
|
*Goal: Close the gaps between "built" and "works right." Every item here addresses a feature that exists but doesn't fully deliver on its promise.*
|
|
|
|
- [x] **2.1 MV Staleness Detection**: Make the fast-path aware of stale materialized views.
|
|
- **Status:** ✅ COMPLETE
|
|
- **Implementation:**
|
|
- [x] Added `max_stale: Option<u64>` to `Query` struct in `crates/stemedb-query/src/query.rs`.
|
|
- [x] Added `.max_stale(secs)` builder method to `QueryBuilder`.
|
|
- [x] In `try_fast_path()`: if `query.max_stale` is set and MV age exceeds threshold, falls through to slow path with `debug!` log.
|
|
- [x] Added `max_stale` to API `QueryParams` DTO in `crates/stemedb-api/src/dto.rs`.
|
|
- [x] Wired through query handler in `crates/stemedb-api/src/handlers/query.rs`.
|
|
- **Tests:**
|
|
- [x] `test_fast_path_stale_view_falls_back`: MV 1000 seconds old, `max_stale = 60` → slow path used.
|
|
- [x] `test_fast_path_fresh_view_used`: Fresh MV, `max_stale = 300` → fast path used.
|
|
- [x] `test_fast_path_no_max_stale_always_uses_mv`: No `max_stale` → any MV age accepted (backward compatible).
|
|
- [x] `test_fast_path_max_stale_zero_rejects_old_mv`: `max_stale = 0`, MV 1 second old → slow path.
|
|
- [x] `test_fast_path_max_stale_zero_accepts_brand_new_mv`: `max_stale = 0`, brand new MV → fast path.
|
|
|
|
- [x] **2.2 AuthorityLens -> ConfidenceLens Rename**: Eliminate the misleading name.
|
|
- **Problem:** `AuthorityLens` selects by `confidence` field, not by agent reputation. `TrustAwareAuthorityLens` is the real authority lens. The name creates confusion about what "Authority" means.
|
|
- **Solution implemented:**
|
|
- [x] Renamed `authority.rs` → `confidence.rs`, `AuthorityLens` → `ConfidenceLens`
|
|
- [x] Added `LensDto::Confidence` for the confidence-field selector
|
|
- [x] Changed `LensDto::Authority` to route to `TrustAwareAuthorityLens` (the real authority lens)
|
|
- [x] Updated query handler routing
|
|
- [x] Updated ai-lookup/services/lens.md and skill documentation
|
|
|
|
- [x] **2.3 EpochAwareLens**: Give epoch supersession runtime behavior.
|
|
- **Status:** ✅ COMPLETE
|
|
- **Implementation:**
|
|
- [x] `EpochAwareLens` in `crates/stemedb-lens/src/epoch_aware.rs`
|
|
- [x] Decorator pattern wrapping any inner lens (default: RecencyLens)
|
|
- [x] Walks supersession chain from `E:{epoch_id}` keys
|
|
- [x] Cycle detection + max depth guard (100)
|
|
- [x] Fail-open on missing epochs
|
|
- [x] `LensDto::EpochAware` added to API
|
|
- [x] 11 tests: excludes_superseded, chain_supersession, no_epochs_passes_all, missing_epoch_includes, cycle_detection, consensus_lens_inner, mixed_epochs, etc.
|
|
- [x] Documentation updated in `ai-lookup/services/lens.md`
|
|
- **Known Limitation:** Filtering only occurs when assertions from the superseding epoch are present in candidates. If all candidates are from old epoch (no new epoch assertions), they pass through (fail-open behavior).
|
|
|
|
- [ ] **2.4 Visual Hash Query Support**: Make the stored `visual_hash` queryable.
|
|
- **Problem:** `visual_hash: Option<PHash>` exists on `Assertion` and is stored/returned by the API, but there is no way to query by visual similarity. The field is write-only from a query perspective.
|
|
- **Current state:** `PHash` is `[u8; 8]` (perceptual hash). Stored on assertions. API accepts/returns it. No query parameter. No similarity computation.
|
|
- [ ] Add `visual_near: Option<String>` (hex-encoded pHash) and `visual_threshold: Option<u32>` (max hamming distance, default: 8) to `Query` struct.
|
|
- [ ] Add `.visual_near(hash, threshold)` to `QueryBuilder`.
|
|
- [ ] In `Query::matches()`: if `visual_near` is set and assertion has `visual_hash`, compute hamming distance (XOR + popcount on the 8 bytes). If distance <= threshold, match. If assertion has no `visual_hash`, don't match.
|
|
- [ ] Add `visual_near` and `visual_threshold` to API `QueryParams` DTO.
|
|
- [ ] Implement `hamming_distance(a: &[u8; 8], b: &[u8; 8]) -> u32` utility function.
|
|
- [ ] Tests:
|
|
- [ ] `test_visual_near_exact_match`: Same hash, threshold 0. Matches.
|
|
- [ ] `test_visual_near_within_threshold`: Hashes differ by 3 bits, threshold 5. Matches.
|
|
- [ ] `test_visual_near_exceeds_threshold`: Hashes differ by 10 bits, threshold 5. No match.
|
|
- [ ] `test_visual_near_skips_assertions_without_hash`: Assertion has no visual_hash. Not matched.
|
|
- [ ] **Note:** This is a brute-force scan approach (O(N) with hamming distance check). A proper VP-tree or BK-tree index is Phase 3. This gives immediate queryability.
|
|
|
|
- [ ] **2.5 Vector Field**: No changes needed. Already roadmapped for Phase 3.
|
|
- **Current state:** `vector: Option<Vec<f32>>` on `Assertion`. Stored and returned by API. No index, no search.
|
|
- **Phase 3 plan:** Integrate `hnsw-rs` or `lance` for k-NN search.
|
|
- **No Phase 2.5 work required.** The field scaffolding is correct.
|
|
|
|
- [ ] **2.6 E2E Integration Test (Write -> Materialize -> Read)**: Prove the full pipeline works end-to-end.
|
|
- **Problem:** The IngestWorker, Materializer, and QueryEngine have been tested in isolation. The Notify integration between IngestWorker and Materializer is tested with a single notification. No test wires all three components together to verify the full write-materialize-read loop.
|
|
- **Current state:** IngestWorker has `with_notify(Arc<Notify>)` (`worker.rs`). Materializer has `run_notified(Arc<Notify>, Duration)` (`materializer.rs`). QueryEngine has `try_fast_path()` (`engine.rs`). Never tested together.
|
|
- [ ] Create integration test in `crates/stemedb-query/tests/e2e_pipeline.rs`:
|
|
- [ ] Setup: Create temp WAL + SledStore + VoteStore + TrustRankStore.
|
|
- [ ] Wire: IngestWorker with `with_notify(notify)`, Materializer with `run_notified(notify)`, QueryEngine with same store.
|
|
- [ ] Test steps:
|
|
1. Write a signed assertion to WAL.
|
|
2. Run IngestWorker.step() -> verifies assertion ingested to KV.
|
|
3. Verify IngestWorker triggered Notify.
|
|
4. Run Materializer.step() -> verifies MV written.
|
|
5. Execute QueryEngine.execute() with subject+predicate -> verifies fast-path returns MV winner.
|
|
- [ ] Variant: Write 2 competing assertions, add votes favoring one, run pipeline, verify correct winner in MV.
|
|
- [ ] Variant: Write assertion, materialize, write new assertion with higher timestamp, re-materialize, verify MV updated, verify query returns new winner.
|
|
- [ ] Add `stemedb-wal` and `stemedb-ingest` as dev-dependencies in `stemedb-query/Cargo.toml`.
|
|
|
|
### Phase 3: The Pilot (BioTech/Pharma)
|
|
*Goal: Prove value in the "High-Liability" beachhead. Close every Camp 4 gap that blocks a credible demo.*
|
|
|
|
#### 3A. Schema Expansion (Prerequisite for everything below)
|
|
|
|
- [x] **3A.1 Source-Class Field**: Add `source_class: SourceClass` to Assertion.
|
|
- **Status:** ✅ COMPLETE
|
|
- **Implementation:**
|
|
- [x] `SourceClass` enum in `crates/stemedb-core/src/types.rs` (lines 68-88).
|
|
- [x] 6-tier system: `Regulatory` (0), `Clinical` (1), `Observational` (2), `Expert` (3), `Community` (4), `Anecdotal` (5).
|
|
- [x] `tier()` method returns tier number for ordering.
|
|
- [x] `default_decay_days()` method for tier-specific confidence decay.
|
|
- [x] `authority_weight()` method for conflict resolution weighting.
|
|
- [x] Field on `Assertion` struct at line 152.
|
|
- [x] Full serialization and indexing support.
|
|
|
|
- [ ] **3A.2 Conflict Score on Resolution**: Add `conflict_score: f32` to Resolution.
|
|
- **Problem:** All 4 use cases reference `conflict_score` in query responses. `Resolution` has `resolution_confidence` but no conflict metric. Without a conflict score, the "disagreement is the information" thesis has no numeric representation.
|
|
- **Current state:** `Resolution` at `traits.rs:10-20` has `winner`, `candidates_count`, `resolution_confidence`. No variance computation.
|
|
- [ ] Add field to `Resolution` in `crates/stemedb-lens/src/traits.rs:10`:
|
|
```rust
|
|
/// Degree of disagreement among candidates (0.0 = full agreement, 1.0 = max conflict).
|
|
/// Computed as normalized variance of candidate confidence values.
|
|
pub conflict_score: f32,
|
|
```
|
|
- [ ] Update `Resolution::empty()` at `traits.rs:24`: set `conflict_score: 0.0`.
|
|
- [ ] Update `Resolution::with_winner()` at `traits.rs:29`: accept `conflict_score` parameter.
|
|
- [ ] Add utility function `compute_conflict_score(candidates: &[Assertion]) -> f32`:
|
|
- Compute mean confidence: `mean = sum(c.confidence) / N`.
|
|
- Compute variance: `var = sum((c.confidence - mean)^2) / N`.
|
|
- Normalize to [0.0, 1.0]: `conflict = (4.0 * var).min(1.0)` (variance of [0,1] range has max 0.25, so 4x normalizes).
|
|
- Edge cases: 0 or 1 candidates -> 0.0.
|
|
- [ ] Update every `Lens::resolve()` and `AsyncLens::resolve_async()` implementation to call `compute_conflict_score()` and pass it to `Resolution::with_winner()`:
|
|
- `crates/stemedb-lens/src/recency.rs`
|
|
- `crates/stemedb-lens/src/consensus.rs`
|
|
- `crates/stemedb-lens/src/authority.rs` (or `confidence.rs` after rename)
|
|
- `crates/stemedb-lens/src/vote_aware_consensus.rs`
|
|
- `crates/stemedb-lens/src/trust_aware_authority.rs`
|
|
- [ ] Add `conflict_score: f32` to `MaterializedView` in `crates/stemedb-core/src/types.rs:143`.
|
|
- [ ] Update `Materializer::materialize_pair()` at `materializer.rs:164` to write `conflict_score` from resolution.
|
|
- [ ] Add `conflict_score` to `QueryResponse` DTO (or a new `ResolutionMeta` sub-object) in `crates/stemedb-api/src/dto.rs`.
|
|
- [ ] Wire `conflict_score` through in the query handler's `apply_lens()` at `handlers/query.rs:111`.
|
|
- [ ] Tests:
|
|
- [ ] `test_conflict_score_zero_for_agreement`: 3 candidates all confidence 0.9. Score near 0.0.
|
|
- [ ] `test_conflict_score_high_for_disagreement`: Candidates at 0.1, 0.5, 0.9. Score > 0.5.
|
|
- [ ] `test_conflict_score_zero_for_single`: 1 candidate. Score = 0.0.
|
|
- [ ] `test_conflict_score_on_materialized_view`: MV stores conflict_score after materialization.
|
|
|
|
- [ ] **3A.3 Rich Source Metadata**: Add structured provenance beyond `source_hash`.
|
|
- **Problem:** `Assertion` has `source_hash: Hash` (a 32-byte hash) but no structured metadata. Consumer Health use case shows POST bodies with `journal`, `DOI`, `sample_size`, `subreddit`, `upvotes`. Without this, assertions are opaque about their provenance.
|
|
- **Current state:** `source_hash: Hash` at `types.rs:64`. No metadata field. API accepts/returns only the hash.
|
|
- [ ] Add field to `Assertion` in `crates/stemedb-core/src/types.rs:65` (after `source_hash`):
|
|
```rust
|
|
/// Structured source metadata as a JSON-encoded byte string.
|
|
/// Schema is domain-specific (journal info, social metrics, etc.).
|
|
pub source_metadata: Option<Vec<u8>>,
|
|
```
|
|
- [ ] Use `Vec<u8>` (not `String`) for rkyv zero-copy compatibility. Callers encode/decode JSON on their side.
|
|
- [ ] Add `source_metadata: Option<Vec<u8>>` to `AssertionBuilder`. Add `.source_metadata_json(json: &str)` builder method that stores `json.as_bytes().to_vec()`.
|
|
- [ ] Add `source_metadata: Option<String>` to `CreateAssertionRequest` DTO (JSON string in API, converted to bytes internally).
|
|
- [ ] Add `source_metadata: Option<String>` to `AssertionResponse` DTO (bytes converted to JSON string).
|
|
- [ ] Wire through create handler and query handler.
|
|
- [ ] Tests:
|
|
- [ ] Serialization roundtrip with metadata present and absent.
|
|
- [ ] API roundtrip: POST with metadata JSON, GET returns same JSON.
|
|
- **Note:** Metadata is stored but NOT indexed in Phase 3. Indexing individual metadata fields is Phase 4+.
|
|
|
|
#### 3B. Time & Decay (Core Query Features)
|
|
|
|
- [ ] **3B.1 Time-Travel Engine**: `as_of` parameter for historical queries.
|
|
- **Problem:** All 4 use cases reference `as_of` for historical state. "What was the consensus when I made my decision?" The append-only model stores all history, but there's no way to query a past state.
|
|
- **Current state:** `Query` at `query.rs:14-29` has no `as_of` field. `try_fast_path()` returns current MV regardless.
|
|
- [ ] Add `as_of: Option<u64>` to `Query` struct in `crates/stemedb-query/src/query.rs:14`.
|
|
- [ ] Add `.as_of(timestamp: u64)` to `QueryBuilder`.
|
|
- [ ] In `Query::matches()` at `query.rs:43`: if `as_of` is `Some(ts)`, check `assertion.timestamp <= ts`. Assertions created after `as_of` are excluded.
|
|
- [ ] In `QueryEngine::execute()` at `engine.rs:47`: if `query.as_of` is set, **skip the fast path entirely** (MVs reflect current state, not historical). Add early check before the `try_fast_path` call:
|
|
```rust
|
|
if query.as_of.is_none() {
|
|
if let (Some(subject), Some(predicate)) = (&query.subject, &query.predicate) {
|
|
if let Some(result) = self.try_fast_path(subject, predicate, query).await? {
|
|
return Ok(result);
|
|
}
|
|
}
|
|
}
|
|
```
|
|
- [ ] Add `as_of: Option<u64>` to `QueryParams` DTO in `crates/stemedb-api/src/dto.rs:102`.
|
|
- [ ] Wire in query handler.
|
|
- [ ] Tests:
|
|
- [ ] `test_as_of_excludes_future_assertions`: 3 assertions at t=1000, t=2000, t=3000. Query `as_of=2500`. Returns only first 2.
|
|
- [ ] `test_as_of_bypasses_fast_path`: MV exists, but `as_of` is set. Slow path used, MV ignored.
|
|
- [ ] `test_as_of_none_uses_fast_path`: Normal query still uses fast path (backwards-compatible).
|
|
- [ ] `test_as_of_with_lens`: Time-travel + lens = resolve only among pre-as_of candidates.
|
|
|
|
- [ ] **3B.2 Semantic Decay**: Confidence Half-Life at query time.
|
|
- **Problem:** Medical knowledge decays at different rates. A Reddit post from 2022 shouldn't compete equally with a 2024 RCT. No assertion-level decay exists. TrustRank has agent-level decay but not assertion-level.
|
|
- **Current state:** `TrustRank.apply_decay()` at `trust_rank_store.rs` does agent decay. No assertion decay. No `decay` query parameter.
|
|
- [ ] Add `decay_halflife: Option<u64>` to `Query` struct (seconds). Represents how quickly old assertions lose effective confidence.
|
|
- [ ] Add `.decay_halflife(seconds: u64)` to `QueryBuilder`.
|
|
- [ ] Add `decay_halflife` to `QueryParams` DTO.
|
|
- [ ] Implementation strategy: **Apply decay in QueryEngine before passing to lens.** In `execute()`, after fetching and filtering candidates, if `decay_halflife` is set:
|
|
1. Get current time (or `as_of` if set).
|
|
2. For each candidate, compute `age = now - assertion.timestamp`.
|
|
3. Compute `effective_confidence = confidence * 2_f32.powf(-(age as f32) / (halflife as f32))`.
|
|
4. Clone the assertion with the decayed confidence.
|
|
5. Pass decayed candidates to the lens.
|
|
- [ ] Add helper `apply_decay(assertions: &[Assertion], halflife: u64, now: u64) -> Vec<Assertion>` in a new `crates/stemedb-query/src/decay.rs` module.
|
|
- [ ] **Source-class-aware decay** (Phase 3 stretch): If assertion has `source_class`, use per-tier half-lives from `SourceClass::default_decay_days()`. This is already implemented on the enum.
|
|
- [ ] Tests:
|
|
- [ ] `test_decay_reduces_old_assertion_confidence`: Assertion 1yr old, halflife 1yr. Effective confidence ~= original * 0.5.
|
|
- [ ] `test_decay_preserves_fresh_assertions`: Assertion 1hr old, halflife 1yr. Effective confidence ~= original.
|
|
- [ ] `test_decay_interacts_with_lens`: Two assertions, older has higher base confidence but after decay, newer wins via RecencyLens.
|
|
- [ ] `test_source_aware_decay`: Tier 0 doesn't decay. Tier 5 decays rapidly.
|
|
|
|
#### 3C. New Lenses
|
|
|
|
- [x] **3C.1 Skeptic Lens**: Surface disagreement, not winners. ✅ **COMPLETED**
|
|
- **Status:** ✅ COMPLETE
|
|
- **Implementation:**
|
|
- [x] `crates/stemedb-lens/src/skeptic.rs` - Full implementation.
|
|
- [x] `AnalysisLens` trait for lenses that analyze conflict instead of resolving it.
|
|
- [x] `SkepticLens` uses normalized Shannon entropy for conflict scoring.
|
|
- [x] Returns `ConflictAnalysis` with:
|
|
- `conflict_score: f32` (0.0 = unanimous, 1.0 = chaos)
|
|
- `status: ResolutionStatus` (Unanimous, Agreed, Contested)
|
|
- `claims: Vec<ClaimSummary>` - all claims ranked by weight
|
|
- [x] `SkepticResolver` + `SkepticView` in `stemedb-query/src/skeptic.rs`.
|
|
- [x] `GET /v1/skeptic?subject=X&predicate=Y` API endpoint.
|
|
- [x] Core types in `stemedb-core/src/types.rs`:
|
|
- `ResolutionStatus` enum
|
|
- `ConflictAnalysis` struct
|
|
- `ClaimSummary`, `SourceSummary`, `AgentSummary`
|
|
- [x] Comprehensive test coverage (21 test cases).
|
|
|
|
- [ ] **3C.2 Layered Consensus Lens**: Per-source-class consensus.
|
|
- **Problem:** Current lenses return a single winner. The Consumer Health use case needs per-tier consensus: "What does Tier 0 say? What does Tier 1 say? What does Tier 5 say?" This is the core differentiator.
|
|
- **Depends on:** Source-Class Field (3A.1) ✅ COMPLETE.
|
|
- [ ] New file: `crates/stemedb-lens/src/layered_consensus.rs`.
|
|
- [ ] New type in `crates/stemedb-lens/src/traits.rs`:
|
|
```rust
|
|
/// Per-tier resolution result.
|
|
pub struct TierResolution {
|
|
pub tier: u8,
|
|
pub winner: Option<Assertion>,
|
|
pub candidates_count: usize,
|
|
pub conflict_score: f32,
|
|
}
|
|
|
|
/// Multi-tier resolution result.
|
|
pub struct LayeredResolution {
|
|
/// Per-tier consensus results, ordered by tier (0 = highest authority).
|
|
pub tiers: Vec<TierResolution>,
|
|
/// Overall winner (highest-tier with a winner).
|
|
pub overall_winner: Option<Assertion>,
|
|
/// Overall conflict score (cross-tier disagreement).
|
|
pub overall_conflict_score: f32,
|
|
}
|
|
```
|
|
- [ ] `LayeredConsensusLens` implements a new `LayeredLens` trait:
|
|
```rust
|
|
pub trait LayeredLens: Send + Sync {
|
|
fn resolve_layered(&self, candidates: &[Assertion]) -> LayeredResolution;
|
|
fn name(&self) -> &'static str;
|
|
}
|
|
```
|
|
- [ ] `resolve_layered()` logic:
|
|
1. Group candidates by `source_class` (use `SourceClass::tier()` method).
|
|
2. For each tier group, run `ConsensusLens::resolve()` to get within-tier winner.
|
|
3. Compute per-tier `conflict_score`.
|
|
4. Overall winner = winner from the highest-authority tier that has candidates (lowest tier number).
|
|
5. Overall conflict_score = cross-tier disagreement (do tier winners agree on the same object value?).
|
|
- [ ] Also implement standard `Lens` trait on `LayeredConsensusLens` to maintain compatibility: `resolve()` returns `overall_winner` as a regular `Resolution`. The richer `LayeredResolution` is accessible via `resolve_layered()`.
|
|
- [ ] Add `LayeredConsensus` to `LensDto` enum.
|
|
- [ ] New API response type: `LayeredQueryResponse` with per-tier results. Wire as a variant in the query handler when `lens=LayeredConsensus`.
|
|
- [ ] Export from `crates/stemedb-lens/src/lib.rs`.
|
|
- [ ] Tests:
|
|
- [ ] `test_layered_single_tier`: All candidates same source_class. Returns one tier result.
|
|
- [ ] `test_layered_multi_tier_agreement`: Tier 0 and Tier 5 agree on same object. Low cross-tier conflict.
|
|
- [ ] `test_layered_multi_tier_disagreement`: Tier 1 says "safe", Tier 5 says "dangerous". High cross-tier conflict. Overall winner from Tier 1.
|
|
- [ ] `test_layered_overall_winner_from_highest_authority`: Tier 0 present -> its winner is overall winner even if Tier 5 has 1000x more assertions.
|
|
|
|
- [ ] **3C.3 Constraints Lens**: Pre-flight check for must_use/forbidden.
|
|
- **Problem:** Agile Agent Team use case needs `lens=constraints` returning `{ must_use: "axios", forbidden: "requests" }`. Central to "persistent learning" — agents query constraints before acting.
|
|
- [ ] New file: `crates/stemedb-lens/src/constraints.rs`.
|
|
- [ ] Design: Not a traditional lens (doesn't pick one winner from candidates). Instead, it categorizes candidates by predicate pattern:
|
|
- Assertions with predicate matching `must_use:*` -> must_use list.
|
|
- Assertions with predicate matching `forbidden:*` -> forbidden list.
|
|
- Assertions with predicate matching `prefer:*` -> prefer list.
|
|
- [ ] Implements `Lens` trait for compatibility: `resolve()` returns the highest-confidence `must_use` assertion as the "winner" (or `forbidden` if no must_use exists). The richer result is accessible via a dedicated method.
|
|
- [ ] Add dedicated `resolve_constraints()` method:
|
|
```rust
|
|
pub struct ConstraintSet {
|
|
pub must_use: Vec<Assertion>,
|
|
pub forbidden: Vec<Assertion>,
|
|
pub prefer: Vec<Assertion>,
|
|
}
|
|
|
|
impl ConstraintsLens {
|
|
pub fn resolve_constraints(&self, candidates: &[Assertion]) -> ConstraintSet { ... }
|
|
}
|
|
```
|
|
- [ ] Add `Constraints` to `LensDto` enum.
|
|
- [ ] New API response type: `ConstraintResponse` with categorized assertions.
|
|
- [ ] Export from `crates/stemedb-lens/src/lib.rs`.
|
|
- [ ] Tests:
|
|
- [ ] `test_constraints_categorizes_by_predicate`: Mixed predicates sorted into must_use/forbidden/prefer.
|
|
- [ ] `test_constraints_empty_categories`: No must_use predicates -> empty must_use list.
|
|
- [ ] `test_constraints_lens_trait_picks_must_use_winner`: Standard `resolve()` returns highest-confidence must_use.
|
|
- [ ] `test_constraints_non_constraint_predicates_ignored`: Regular predicates not categorized.
|
|
|
|
#### 3D. Epoch Enhancement
|
|
|
|
- [ ] **3D.1 Epoch Cascade Logic** (enhancement of Phase 2.5 EpochAwareLens):
|
|
- **Problem:** Phase 2.5 EpochAwareLens walks the `supersedes` chain at query time by reading `E:` keys. This is O(chain_length) per query. For long supersession chains, this is expensive.
|
|
- **Depends on:** Phase 2.5 EpochAwareLens (basic version).
|
|
- [ ] In `IngestWorker::process_epoch()` at `crates/stemedb-ingest/src/worker.rs`:
|
|
1. When an epoch with `supersedes = Some(old_epoch_id)` is ingested, write a marker key `SUPERSEDED:{old_epoch_id}` with the new epoch ID as value.
|
|
2. Walk the chain: if old_epoch itself superseded another epoch, write `SUPERSEDED:{grandparent_id}` too (transitive closure).
|
|
- [ ] Update `EpochAwareLens` to check `SUPERSEDED:{epoch_id}` keys (O(1) lookup per candidate) instead of walking the chain at query time.
|
|
- [ ] Tests:
|
|
- [ ] `test_cascade_writes_superseded_marker`: Ingest epoch B superseding A. Verify `SUPERSEDED:A` key exists.
|
|
- [ ] `test_cascade_transitive`: A supersedes B, B supersedes C. Verify both `SUPERSEDED:B` and `SUPERSEDED:C` exist.
|
|
- [ ] `test_epoch_aware_uses_marker`: Query with EpochAwareLens uses marker key, not chain walk.
|
|
|
|
#### 3E. Similarity Search
|
|
|
|
- [ ] **3E.1 Vector Search**: Semantic k-NN queries via embeddings.
|
|
- **Current state:** `vector: Option<Vec<f32>>` on Assertion. Stored, returned by API. No index, no search.
|
|
- [ ] Add `hnsw-rs` (or `lance`) as a dependency in `stemedb-storage/Cargo.toml`.
|
|
- [ ] New module: `crates/stemedb-storage/src/vector_index.rs`.
|
|
- [ ] `VectorIndex` trait: `insert(hash: Hash, vector: &[f32])`, `search(query: &[f32], k: usize) -> Vec<(Hash, f32)>`.
|
|
- [ ] Implementation backed by HNSW graph stored alongside KV data.
|
|
- [ ] IngestWorker: if assertion has `vector`, insert into vector index after KV write.
|
|
- [ ] Add `vector_near: Option<Vec<f32>>` and `k: Option<usize>` to `Query` struct and API `QueryParams`.
|
|
- [ ] QueryEngine: if `vector_near` is set, use vector index for candidate retrieval instead of SP/S index.
|
|
- [ ] Tests: insert 100 vectors, query nearest 5, verify correct neighbors.
|
|
|
|
- [ ] **3E.2 Visual Hash Index**: VP-tree or BK-tree for O(log N) visual similarity.
|
|
- **Current state:** Phase 2.5 adds brute-force hamming scan. This replaces it with an indexed approach.
|
|
- [ ] New module: `crates/stemedb-storage/src/visual_index.rs`.
|
|
- [ ] BK-tree implementation over hamming distance on `PHash` ([u8; 8]).
|
|
- [ ] IngestWorker: if assertion has `visual_hash`, insert into BK-tree.
|
|
- [ ] QueryEngine: if `visual_near` is set, use BK-tree for O(log N) candidate retrieval instead of brute-force scan.
|
|
- [ ] Tests: insert 1000 assertions with visual hashes, query similar, verify correct matches within threshold.
|
|
|
|
#### 3F. Provenance
|
|
|
|
- [ ] **3F.1 Citation Recall Benchmarking**: Verify 100% provenance tracking.
|
|
- [ ] Benchmark: for every assertion in the store, verify `source_hash` resolves to a retrievable source document (or at minimum, the hash is non-zero and unique).
|
|
- [ ] Add `GET /v1/provenance/{hash}` endpoint that looks up source by hash.
|
|
- [ ] Add `POST /v1/source` endpoint to store source documents by content hash.
|
|
- [ ] Source storage at `SRC:{blake3_hash}` keys.
|
|
|
|
#### 3G. API Cleanup
|
|
|
|
- [ ] **3G.1 Document epoch supersession via existing endpoint**: No new `/epoch/supersede` endpoint needed.
|
|
- **Current state:** `POST /v1/epoch` already accepts `supersedes` field. Use cases show `POST /epoch/supersede` as if it's a separate endpoint.
|
|
- [ ] Update use case docs (consumer-health-intelligence.md, glp1-living-review.md) to use `POST /v1/epoch` with `supersedes` field.
|
|
- [ ] Add OpenAPI examples showing the supersession flow.
|
|
- **No code change.** Documentation fix only.
|
|
|
|
### Phase 4: The Hive (Trust & Scale)
|
|
*Goal: Change tracking, metadata indexing, and the database primitives for training pipelines.*
|
|
|
|
- [x] **TrustRank Engine**: Foundation for trust-based resolution.
|
|
- [x] `TrustRankStore` for per-agent reputation storage.
|
|
- [x] `TrustAwareAuthorityLens` for reputation-weighted resolution.
|
|
- [x] **Confidence Half-Life**: Implement decay calculation engine.
|
|
- [x] Learning loop: `record_outcome()` for accuracy tracking.
|
|
|
|
- [ ] **4.1 "Since" Parameter**: Change tracking for returning consumers.
|
|
- **Problem:** Consumer Health shows `GET /query?since=2023-10-01` returning `changes_since_query` with dated change entries. The "returning consumer" story: "What changed since I last looked?"
|
|
- **Depends on:** Time-Travel (3B.1) and Materializer.
|
|
- [ ] Add `since: Option<u64>` to `Query` struct and `QueryBuilder` in `crates/stemedb-query/src/query.rs`.
|
|
- [ ] Add `since` to `QueryParams` DTO.
|
|
- [ ] **MV Changelog**: Track when materialized views change.
|
|
- [ ] New key pattern: `MVC:{subject}:{predicate}:{timestamp}` storing the previous winner hash and new winner hash.
|
|
- [ ] In `Materializer::materialize_pair()` at `materializer.rs:164`: before overwriting MV, read the existing MV. If the winner changed (different assertion hash), write a changelog entry.
|
|
- [ ] In QueryEngine: if `since` is set, scan `MVC:{subject}:{predicate}:*` for entries with timestamp > since. Return these as a `changes_since` list alongside the normal query result.
|
|
- [ ] New response field on `QueryResponse`:
|
|
```rust
|
|
pub struct ChangeEntry {
|
|
pub timestamp: u64,
|
|
pub previous_winner_hash: Option<String>,
|
|
pub new_winner_hash: String,
|
|
}
|
|
```
|
|
Add `changes_since: Option<Vec<ChangeEntry>>` to `QueryResponse` DTO.
|
|
- [ ] Tests:
|
|
- [ ] `test_since_returns_changes`: Materialize, change winner, re-materialize. Query with `since` returns the change.
|
|
- [ ] `test_since_no_changes`: No MV changes since timestamp. Empty changes list.
|
|
- [ ] `test_since_multiple_changes`: 3 winner changes over time. All returned in order.
|
|
|
|
- [ ] **4.2 Source Metadata Indexing** (extension of 3A.3): Index key metadata fields.
|
|
- **Problem:** Phase 3 stores `source_metadata` as an opaque blob. Phase 4 makes key fields queryable.
|
|
- **Depends on:** Rich Source Metadata (3A.3).
|
|
- [ ] Define a set of indexed metadata keys: `journal`, `doi`, `platform`, `study_design`.
|
|
- [ ] New key pattern: `SM:{field}:{value}:{assertion_hash}` for metadata field indexes.
|
|
- [ ] IngestWorker: on ingestion, if `source_metadata` is present, parse JSON, extract indexed fields, write index entries.
|
|
- [ ] Add metadata field filters to `QueryParams` (e.g., `?source_journal=NEJM`).
|
|
- [ ] Tests: store assertions with metadata, query by journal name, verify correct filtering.
|
|
|
|
- [ ] **4.3 Batch TrustRank Decay API**: Expose scheduled decay for external orchestration.
|
|
- **Current state:** `decay_all_trust_ranks()` exists on TrustRankStore. No API endpoint.
|
|
- [ ] Add `POST /v1/admin/decay-trust-ranks` endpoint.
|
|
- [ ] Accepts `now: u64` parameter (or uses current time).
|
|
- [ ] Returns count of decayed agents and summary stats.
|
|
- **Note:** The Gardener (Camp 5.2, app layer) calls this endpoint on a schedule. The database just exposes the primitive.
|
|
|
|
> **Note:** The following items were reclassified as **Application Layer** responsibilities (see `tmp/ambition-vs-reality.md`, Camp 5). They are not Episteme database features. They consume the Episteme API and are built by integrators or vertical-specific teams.
|
|
>
|
|
> - **The Simulator** (Training Data Pipeline) -> Camp 5.3
|
|
> - **The Super Curator** (Reviewer Agent swarm) -> Camp 5.4
|
|
> - **Background Gardener** (Cluster detection, signal processing) -> Camp 5.2
|
|
> - **Agent Wallet** (Key management sidecar) -> Camp 5.1
|
|
|
|
---
|
|
|
|
## Tracking
|
|
|
|
### Active Tasks
|
|
* [ ] **Phase 2.5 Hardening**: Camp 2 fixes (~~staleness~~, epoch behavior, lens rename, visual query, E2E test).
|
|
|
|
### Next Up (Phase 3 sequencing)
|
|
* [ ] **3A.2 Conflict Score**: Add to Resolution. Propagate through all lenses.
|
|
* [ ] **3A.3 Rich Source Metadata**: Structured provenance field.
|
|
* [ ] **3B.1 Time-Travel**: `as_of` parameter. Unblocks all 4 use cases.
|
|
* [ ] **3C.2 Layered Consensus Lens**: Per-tier resolution (now unblocked by 3A.1).
|
|
|
|
### Recently Completed
|
|
* [x] **SkepticLens + SkepticView** (3C.1): "Trust but Verify" conflict analysis that surfaces all claims with conflict scores.
|
|
* `AnalysisLens` trait for lenses that map conflict instead of resolving it.
|
|
* `SkepticLens` using normalized Shannon entropy for conflict scoring.
|
|
* `SkepticResolver` + `SkepticView` in stemedb-query.
|
|
* `GET /v1/skeptic?subject=X&predicate=Y` API endpoint.
|
|
* Types: `ResolutionStatus`, `ConflictAnalysis`, `ClaimSummary`, `SourceSummary`, `AgentSummary`.
|
|
* [x] **Source-Class Field** (3A.1): 6-tier `SourceClass` enum with authority weighting and decay rates.
|
|
* `SourceClass` enum: Regulatory, Clinical, Observational, Expert, Community, Anecdotal.
|
|
* `tier()`, `default_decay_days()`, `authority_weight()` methods.
|
|
* Field on Assertion struct with full serialization support.
|
|
* [x] **The Meter**: Token bucket quota system with MeterLayer middleware (10K tokens/agent/hour).
|
|
* [x] **Query Audit Trail**: Every query logged with provenance at `AUD:{query_id}`. `X-Agent-Id` header for attribution.
|
|
* [x] **Event-Driven Materialization**: `run_notified()` + IngestWorker Notify integration.
|
|
* [x] **Fast-Path MV Lookup**: `QueryEngine::try_fast_path()` for O(1) reads.
|
|
* [x] **Materializer**: Background worker for O(1) MV reads via `AsyncLens`.
|
|
* [x] **VoteAwareConsensusLens**: Real vote-based consensus resolution.
|
|
* [x] **Compound SP Index**: O(1) subject+predicate lookups.
|
|
* [x] **TrustRank System**: Agent reputation with decay and learning loop.
|
|
* [x] **API Surface**: axum HTTP server with 7 endpoints + OpenAPI docs.
|
|
|
|
### Blockers
|
|
* None.
|
|
|
|
---
|
|
|
|
## Dependency Graph
|
|
|
|
```
|
|
Phase 2.5 (Hardening) Phase 3 (The Pilot) Phase 4 (The Hive)
|
|
======================== ======================== ==================
|
|
|
|
[2.1 MV Staleness] ---------> [3B.1 Time-Travel] -----+
|
|
| |
|
|
[2.2 Confidence Rename] -----> (API clarity for all) +----------> [4.1 "Since" Param]
|
|
|
|
|
[2.3 EpochAwareLens] --------> [3D.1 Epoch Cascade] ---|----------> Invalidation Cascades pillar
|
|
|
|
|
[2.4 Visual Hash Query] -----> [3E.2 Visual Hash Index] |
|
|
|
|
|
[2.6 E2E Integration] -------> (pipeline confidence) |
|
|
|
|
|
[3A.1 Source-Class] ✅ --+----------> [3C.2 Layered Consensus]
|
|
| [3B.2 Source-Aware Decay]
|
|
+-----------------------------[4.2 Metadata Indexing]
|
|
|
|
|
[3A.2 Conflict Score] ---> (enhance Resolution)
|
|
|
|
|
[3A.3 Source Metadata] ---> [4.2 Metadata Indexing]
|
|
|
|
|
[3C.1 Skeptic Lens] ✅ (standalone, COMPLETE)
|
|
[3C.3 Constraints Lens] (standalone)
|
|
[3E.1 Vector Search] (standalone)
|
|
[3F.1 Citation Recall] (standalone)
|
|
```
|
|
|
|
### Critical Path for Consumer Health Demo
|
|
|
|
```
|
|
[3A.1 Source-Class] ✅ --> [3A.2 Conflict Score] --> [3C.2 Layered Consensus]
|
|
|
|
|
+----> CONSUMER HEALTH MVP
|
|
|
|
|
[3B.1 Time-Travel] ------------------------------------------+
|
|
|
|
|
[3A.3 Source Metadata] --------------------------------------+
|
|
|
|
|
[3C.1 Skeptic Lens] ✅ --------------------------------------+
|
|
```
|
|
|
|
### Critical Path for Financial DD Demo
|
|
|
|
```
|
|
[3A.2 Conflict Score] --> [3C.1 Skeptic Lens] ✅ -------+
|
|
|
|
|
[3B.1 Time-Travel] -------------------------------------+----> FINANCIAL DD MVP
|
|
|
|
|
[2.3 EpochAwareLens] --> [3D.1 Epoch Cascade] ----------+
|
|
|
|
|
[3B.2 Semantic Decay] ----------------------------------+
|
|
```
|
|
|
|
### Critical Path for Agile Agent Team Demo
|
|
|
|
```
|
|
[3C.3 Constraints Lens] (standalone) ------+
|
|
|
|
|
[3B.1 Time-Travel] -----------------------+----> AGENT TEAM MVP
|
|
|
|
|
[2.3 EpochAwareLens] ---------------------+
|
|
|
|
|
[Query Audit (Phase 2)] ✅ ----------------+
|
|
```
|