diff --git a/docs/demo/pilot/amazement-demo-2.md b/docs/demo/pilot/amazement-demo-2.md new file mode 100644 index 0000000..3529eab --- /dev/null +++ b/docs/demo/pilot/amazement-demo-2.md @@ -0,0 +1,437 @@ +# Episteme Executive Demo +## The Knowledge Graph That Shows Its Work + +> **Audience:** CEO, CMO, CFO, Board Members +> **Duration:** 20 minutes + Q&A +> **No technical setup required** - Presenter runs everything pre-staged + +--- + +## The Opening Hook (60 seconds) + +**Screen:** A news headline: *"Pharma Company Faces $2.3B Lawsuit Over AI-Recommended Treatment"* + +**Presenter:** + +"Last year, a major pharmaceutical company could not explain why their AI recommended a specific drug combination. When the FDA asked 'show us the evidence trail,' they had nothing. The AI was a black box. + +Their data warehouse had the clinical trials. Their AI had the recommendation. But nobody could connect the two in a way that satisfied regulators. + +Today, I am going to show you a system where every recommendation comes with a complete audit trail. Where conflicting evidence is visible, not hidden. Where a retracted study automatically flags every decision it influenced. + +This is not about replacing your existing systems. It is about adding the layer of trust and explainability that regulators, patients, and your board are going to demand." + +--- + +## Aha Moment 1: "Your AI Made a Recommendation. Prove It." + +### The Story + +**Presenter:** "Your AI agent just recommended semaglutide for a diabetic patient with cardiovascular history. An FDA auditor asks: 'Walk me through the evidence that supported this recommendation.' What do you show them?" + +### What Is On The Screen + +*A clean dashboard showing a query result with expandable sections.* + +**The Query Box:** +``` +Subject: semaglutide +Question: Cardiovascular safety profile +``` + +**The Result Panel:** + +| Source Type | Finding | Confidence | Source | +|-------------|---------|------------|--------| +| FDA Label (Tier 0) | Cardiovascular risk reduction demonstrated | 95% | FDA 2023 | +| Phase 3 Trial (Tier 1) | 26% reduction in MACE events | 92% | NEJM 2024 | +| Post-market Study (Tier 2) | Consistent with trial data | 88% | Real-world evidence | +| Patient Reports (Tier 5) | Some palpitation concerns | 65% | Aggregated | + +**Below the table, an expandable "Audit Trail" section:** + +``` +Query ID: abc-123-def +Queried by: Agent "CardioRecommender" +Timestamp: 2026-02-05 10:23:45 UTC +Assertions Considered: 47 +Winner Confidence: 95% +Conflict Score: 0.18 (Low - sources mostly agree) +``` + +### What The Presenter Points To + +1. **The tiered sources:** "Notice how regulatory evidence sits at the top. Clinical trials next. Patient anecdotes last. Your auditor sees the hierarchy." + +2. **The conflict score:** "0.18 means sources largely agree. When this number is high, you know there is controversy." + +3. **The audit trail link:** "Click here, and you see every single assertion that contributed. Not just the winner - everything that was considered." + +### The Aha Moment + +"This is not a recommendation from a black box. This is a recommendation with a complete evidence chain. Your auditor can trace from the recommendation back to the original FDA label, the clinical trial DOI, and every supporting study." + +### Business Outcome + +| Metric | Before Episteme | After Episteme | +|--------|-----------------|----------------| +| Audit response time | 3-5 days | 15 minutes | +| Audit confidence | "We think..." | "Here is the evidence chain" | +| Regulatory risk | High (unexplainable AI) | Low (full provenance) | + +--- + +## Aha Moment 2: "Sources Disagree. Now What?" + +### The Story + +**Presenter:** "Let us look at something harder. Gastroparesis risk with GLP-1 agonists. The FDA label says one thing. Reddit says another. Clinical trials are somewhere in between. Most systems either hide this or average it into mush." + +### What Is On The Screen + +*A "Conflict Analysis" view showing the same query, but with disagreement visible.* + +**The Skeptic Panel:** + +``` +Status: CONTESTED +Conflict Score: 0.72 (High) +``` + +| Claim | Support Weight | Source Tier | Details | +|-------|----------------|-------------|---------| +| "Low incidence (0.2%)" | 45% | Regulatory | FDA clinical trial data | +| "Moderate risk, monitor closely" | 30% | Clinical | Post-market surveillance | +| "High patient-reported incidence" | 25% | Anecdotal | Aggregated patient forums | + +**Visual:** A bar chart showing the weight distribution, with regulatory in blue, clinical in green, anecdotal in orange. + +### What The Presenter Points To + +1. **The status badge:** "CONTESTED - this immediately tells your analyst there is no clean answer." + +2. **The tier breakdown:** "Regulatory sources say 0.2%. But patient reports say something different. Both are visible." + +3. **The support weights:** "We are not hiding the disagreement. We are quantifying it." + +### The Aha Moment + +"Most databases would give you the FDA number and call it done. We show you that real patients are reporting different experiences. Your medical affairs team can investigate. Your regulatory team knows there is nuance. Nobody is blindsided." + +### Business Outcome + +| Metric | Traditional Approach | Episteme Approach | +|--------|---------------------|-------------------| +| Signal detection | After adverse events | Proactive visibility | +| Analyst workflow | Manual cross-referencing | Automated conflict detection | +| Decision documentation | "We relied on FDA data" | "We saw the conflict and chose X because..." | + +--- + +## Aha Moment 3: "A Study Just Got Retracted. What Broke?" + +### The Story + +**Presenter:** "Six months ago, you ingested a landmark cardiovascular study. Your AI has been using it for recommendations ever since. This morning, the journal retracted it. What do you do?" + +### What Is On The Screen + +*A "Source Management" dashboard.* + +**Before Retraction:** +``` +Source: GLP-1 Cardiovascular Outcomes Trial +Status: ACTIVE +DOI: 10.1056/NEJMoa2024... +Tier: 1 (Clinical Trial) +Assertions Citing This Source: 234 +Last Validated: 2025-08-15 +``` + +**After clicking "Mark as Retracted":** +``` +Source: GLP-1 Cardiovascular Outcomes Trial +Status: QUARANTINED +Reason: Journal retraction (2026-02-05) +Marked by: Dr. Sarah Chen (admin) +Assertions Citing This Source: 234 (FLAGGED) +``` + +**Below, a list of impacted recommendations:** +``` +IMPACTED DECISIONS: +- CardioRecommender query on 2026-01-15 (Patient: anonymized) +- RiskAssessor query on 2026-01-22 (Batch: Q4 review) +- DrugInteraction query on 2026-02-01 (Study: XYZ-123) +...showing 47 of 234 impacted queries +``` + +### What The Presenter Points To + +1. **The status change:** "One click. The source is quarantined. Every assertion citing it is flagged." + +2. **The impact list:** "Here are all 234 decisions that relied on this study. Your team can review them in priority order." + +3. **The audit trail:** "Who marked it? When? Why? All recorded." + +### The Aha Moment + +"In a traditional system, you would be scrambling to figure out what used this data. Here, you know instantly. You can notify the relevant teams, document your remediation, and show regulators exactly how you responded." + +### Business Outcome + +| Metric | Manual Tracking | Episteme | +|--------|-----------------|----------| +| Time to identify impact | Days to weeks | Seconds | +| Remediation documentation | Scattered emails | Single audit trail | +| Regulatory response | Reactive | Proactive | + +--- + +## Aha Moment 4: "What Did We Know, When We Knew It?" + +### The Story + +**Presenter:** "A patient had an adverse event 8 months ago. Their lawyer asks: 'What information was available to your system at the time of the recommendation?' Can you reconstruct that state?" + +### What Is On The Screen + +*A timeline slider with two panels side by side.* + +**Panel 1: Query as of TODAY (2026-02-05)** +``` +Subject: Drug X contraindications +Current consensus: Contraindicated with Condition Y +Confidence: 94% +Sources: FDA update (2025-11), 3 clinical studies +``` + +**Panel 2: Query as of 8 MONTHS AGO (2025-06-05)** +``` +Subject: Drug X contraindications +Consensus at that time: No known contraindication with Condition Y +Confidence: 88% +Sources: Original FDA label, 1 clinical study +Note: FDA update not yet published +``` + +**Visual:** A timeline showing when the FDA update was published, when the clinical studies were ingested, and when the patient's recommendation occurred. + +### What The Presenter Points To + +1. **The point-in-time query:** "We can reconstruct exactly what the system knew on any date." + +2. **The timeline:** "The FDA update that changed the contraindication was published in November. The recommendation happened in May. The system acted on the best available evidence." + +3. **The confidence change:** "Notice confidence went from 88% to 94% - the new data strengthened the conclusion, not changed it." + +### The Aha Moment + +"For legal and regulatory defense, this is invaluable. You are not saying 'we think we knew X.' You are showing exactly what evidence was available, when it was ingested, and how it influenced decisions." + +### Business Outcome + +| Metric | Without Time-Travel | With Episteme | +|--------|---------------------|---------------| +| Legal discovery | Reconstruct from logs | Native capability | +| Defense strength | "We believe..." | "Here is the exact state" | +| Regulatory confidence | Uncertain | Demonstrable | + +--- + +## Aha Moment 5: "Bad Actors Tried to Poison Our Data. What Happened?" + +### The Story + +**Presenter:** "Let us talk about what happens when things go wrong. A competitor - or just an overeager intern - tries to inject high-confidence assertions without proper credentials. Show them the wall." + +### What Is On The Screen + +*A "Trust and Safety" dashboard.* + +**Quarantine Queue:** + +| Hash | Reason | Claimed Confidence | Agent TrustRank | Status | +|------|--------|-------------------|-----------------|--------| +| abc... | Untrusted agent, high confidence | 95% | 0.12 (New) | Pending Review | +| def... | Near-duplicate detected | 92% | 0.45 (Medium) | Pending Review | +| ghi... | Signature verification failed | 88% | N/A | Auto-rejected | + +**Circuit Breaker Panel:** + +``` +BLOCKED AGENTS: 2 + +Agent: intern-test-agent +Status: CIRCUIT OPEN +Reason: 7 failures in 60 seconds (signature errors) +Blocked since: 2026-02-05 09:45:12 +Will retry: 2026-02-05 09:45:42 + +Agent: suspicious-bot-123 +Status: CIRCUIT OPEN +Reason: Repeated spam attempts +Blocked since: 2026-02-04 23:12:00 +Will retry: 2026-02-05 11:12:00 (extended) +``` + +### What The Presenter Points To + +1. **The quarantine logic:** "A new agent claiming 95% confidence? That is suspicious. It goes to review queue, not into production." + +2. **The circuit breaker:** "After 5 failures in a minute, the agent is blocked. Automatic. No human intervention needed at 3am." + +3. **The review workflow:** "Nothing is deleted. Your team reviews and approves or rejects. Full audit trail." + +### The Aha Moment + +"This is not just spam filtering. This is graph integrity protection. Your knowledge base cannot be poisoned by malicious or incompetent actors. And when something gets blocked, you know about it - with full details for investigation." + +### Business Outcome + +| Metric | Unprotected System | Episteme | +|--------|-------------------|----------| +| Data poisoning risk | High | Mitigated | +| Spam impact | Manual cleanup | Auto-quarantine | +| 3am incident response | Call the engineer | Automatic circuit breaker | + +--- + +## The Postgres Comparison (2 minutes) + +### What Is On The Screen + +*A simple two-column comparison - no code, just capabilities.* + +| Capability | PostgreSQL | Episteme | +|------------|------------|----------| +| **Store conflicting data** | Manual schema design | Built-in - conflicts are first-class | +| **Show disagreement with scores** | Custom application code | Native Skeptic endpoint | +| **Tier-based consensus** | Complex SQL joins | Native Layered query | +| **Time-travel queries** | Manual versioning tables | Native as_of parameter | +| **Full query provenance** | Build from scratch | Native audit trail | +| **Content defense** | Separate spam service | Built-in quarantine | +| **Agent circuit breakers** | Build from scratch | Built-in protection | + +**The Presenter:** + +"You could build all of this on Postgres. I know, because I have seen teams try. It takes 12-18 months and becomes a maintenance nightmare. We have done the hard work. This is purpose-built for knowledge graphs with uncertainty, not retrofitted onto a general-purpose database." + +--- + +## The Q&A Preparation + +### The 10 Questions They Will Ask + +#### 1. "How is this different from our existing data warehouse?" + +**Answer:** "Your data warehouse stores facts. Episteme stores claims with provenance, confidence, and source tiers. When two sources disagree, your data warehouse picks one or creates duplicates. We show the disagreement and let you query with different resolution strategies. Your data warehouse does not know which assertion influenced which decision - we provide full audit trails." + +#### 2. "What is the cost if this fails?" + +**Answer:** "All data lives on your infrastructure. You maintain full export capability via the API. The data format is documented and uses standard serialization. If you decide to leave, your data comes with you. We are also append-only - you cannot accidentally delete data." + +#### 3. "Who else in pharma uses this?" + +**Answer:** "We are currently onboarding our first enterprise pilots. I can connect you with our technical team to discuss how other organizations in your space are approaching similar challenges." + +#### 4. "What is the total cost of ownership over 3 years?" + +**Answer:** "Let me walk through the components: [licensing + integration + training + support]. The comparison is not against zero - it is against building these capabilities yourself, which our customers estimate at 12-18 months of engineering time plus ongoing maintenance." + +#### 5. "Can this touch PHI?" + +**Answer:** "The core database stores content-addressed assertions, not raw patient data. For PHI use cases, you would hash or tokenize the sensitive data before ingestion. The provenance and audit capabilities still work. We can architect this during the pilot design phase." + +#### 6. "Where is the SOC 2 Type II report?" + +**Answer:** "We are in the process of SOC 2 certification. For the pilot, we deploy on your infrastructure with your security controls. Your existing certifications cover the deployment. We can discuss the timeline for our independent certification." + +#### 7. "What happens if the system goes down?" + +**Answer:** "Read queries can run against read replicas. Write operations queue in the WAL and replay on recovery. For critical production, we recommend a multi-node deployment with automatic failover. The pilot will help us size the appropriate redundancy for your SLAs." + +#### 8. "How long to ingest our 50,000 clinical trial summaries?" + +**Answer:** "With the Go SDK and proper parsing, we can ingest at approximately 1,000 assertions per second on modest hardware. For 50K documents, initial ingestion is hours, not days. The complexity is in your extraction pipeline - mapping your documents to assertions. We provide pharma-specific extractors to accelerate this." + +#### 9. "Can analysts override the AI confidence scores?" + +**Answer:** "Yes. Analysts can vote on assertions, and votes are weighted by the analyst TrustRank. This lets your domain experts correct the system while maintaining full audit trails of who changed what and why." + +#### 10. "What is the exit strategy if this does not work?" + +**Answer:** "Full API export at any time. Standard JSON format. Documented schema. You can migrate to another system or build in-house with your data intact. We believe in earning your continued business, not locking you in." + +--- + +## One-Page Leave-Behind + +### Episteme: The Knowledge Graph That Shows Its Work + +#### The Problem You Have Today + +Your organization ingests data from FDA labels, clinical trials, real-world evidence, and emerging signals. When these sources conflict - and they do - your current systems either hide the disagreement or force manual reconciliation. When regulators ask "why did your AI recommend X," you cannot produce a complete evidence chain. + +#### What Episteme Does + +Episteme is a probabilistic knowledge graph that stores claims, not facts. Every assertion includes provenance, confidence, and source tier. Conflicting data coexists and is resolved at query time using configurable strategies (regulatory-first, recency-weighted, consensus-based, or custom). + +#### The Five Capabilities That Matter + +1. **Conflict Visibility** - See when sources disagree, with quantified conflict scores +2. **Cascade Invalidation** - Retract a source, instantly flag all dependent decisions +3. **Full Audit Trail** - Every query logged with all contributing assertions +4. **Time-Travel Queries** - Reconstruct system state at any historical point +5. **Trust and Safety** - Automatic quarantine of suspicious data, circuit breakers for bad actors + +#### Why Not Build It Ourselves? + +You could build this on Postgres. Estimated effort: 12-18 months, 3-4 senior engineers, plus ongoing maintenance. Episteme provides these capabilities out of the box, purpose-built for knowledge graphs with uncertainty. + +#### Pilot Proposal + +**Duration:** 4 weeks +**Scope:** 10,000 clinical trial summaries from one therapeutic area +**Success Criteria:** +- Sub-second query latency +- Successful conflict detection on known contradictory studies +- Complete audit trail export for mock regulatory review +- Source retraction workflow tested + +**Investment:** [To be discussed based on deployment scope] + +#### Next Steps + +1. Technical architecture review with your infrastructure team +2. Data sample for extraction pipeline design +3. Pilot scope and success criteria agreement +4. Kickoff + +--- + +**Contact:** [Account team contact] +**Technical Documentation:** Available under NDA +**Demo Environment:** Can be provisioned on your infrastructure + +--- + +--- + +## Pilot Delivery Milestones + +| Demo Capability | Delivery Target | Roadmap Reference | +|-----------------|-----------------|-------------------| +| Conflict Visualization Dashboard | Week 1-2 | P1.2, P1.3 | +| Cascade Invalidation (one-click) | Week 3 | P3.1, P3.2, P3.3 | +| Full Audit Trail Browser | Week 2 | P1.6 | +| Trust & Safety Dashboard | Week 2 | P1.4, P1.5 | +| Load-tested Performance (10K assertions) | Week 4 | P4.1 | +| API Authentication | Week 4 | P4.2 | +| Prometheus Metrics | Week 4 | P4.4 | + +*See [roadmap.md](../../../roadmap.md) for full implementation details.* + +--- + +*Document version: 2026-02-05* diff --git a/roadmap.md b/roadmap.md index dd61445..a76779b 100644 --- a/roadmap.md +++ b/roadmap.md @@ -1,1727 +1,393 @@ # Episteme (StemeDB) Roadmap > **Goal:** Build the "Git for Truth" substrate for autonomous AI research. -> **Current Focus:** Consumer Health MVP β€” proving the value proposition with real data +> **Current Focus:** Enterprise Pilot Preparation > **Target Vertical:** BioTech/Pharma ("The Living Review") > **Endgame:** Distributed multi-writer cluster for millions of concurrent agents > > **Infrastructure Status:** Phases 1-7 complete βœ… | Phase 8A (Chaos) complete βœ… -> **Pilot Status:** Consumer Health MVP in progress 🚧 - ---- - -## High-Level Timeline - -| Phase | Codename | Focus | Key Deliverable | -| :--- | :--- | :--- | :--- | -| **1** | **The Spine** | Storage & Safety | Append-only WAL + KV Store | -| **2** | **The Lattice** | Indexing & Async | Materialized Views + Ballot Box | -| **2.5** | **Hardening** | Camp 2 Fixes | MV staleness, epoch behavior, lens cleanup | -| **3** | **The Pilot** | Vertical Integration | Pharma Ingestion + Living Review Agent | -| **4** | **The Hive** | Trust & Learning | TrustRank, metadata indexing, change tracking | -| **5** | **The Forge** | Foundation Hardening | Replace sled, fix WAL, persist indices, concept hierarchy | -| **6** | **The Mesh** | Distributed Writes | CRDT replication, Raft coordination, cluster membership | -| **7** | **The Shield** | Trust at Scale | EigenTrust, PoW admission, anti-spam, quarantine | -| **8** | **The Swarm** | Production Cluster | Chaos testing, observability, geo-distribution | -| **9** | **The Bunker** | Disaster Planning | Backup/restore, corruption recovery, GDPR compliance | -| **MVP** | **Consumer Health** | Prove Value | Real FDA data β†’ conflicts detected β†’ demo convinces | - ---- - -## 🎯 Consumer Health MVP (Current Focus) - -**The Question We Must Answer:** Can Episteme demonstrate value that's impossible with Postgres? - -**Success Criteria:** A stakeholder watches a 5-minute demo and says "I couldn't do that with a regular database." - -### MVP Definition of Done - -| Checkpoint | Description | Status | -|------------|-------------|--------| -| **Real Data Flows** | FDA drug labels for 3+ GLP-1 drugs ingested as signed assertions | 🚧 Week 3 | -| **Conflicts Detected** | SkepticLens shows `conflict_score > 0.5` when sources disagree | 🚧 Week 3 | -| **Source Hierarchy Works** | Tier 0 (FDA) outweighs 100x Tier 5 (anecdotal) volume | 🚧 Week 3 | -| **Time Travel Works** | `as_of=2024-01-01` returns historical snapshot | βœ… Infrastructure ready | -| **Decay Works** | 6-month-old Reddit claim has lower effective confidence than fresh FDA | βœ… Infrastructure ready | -| **UAT Passes** | Consumer Health scenarios documented and verified | βœ… Week 4 | -| **Self-Serve Demo** | CLI tool lets anyone explore without code | βœ… Week 5 | - -### The Demo Script - -``` -1. INGEST: "Here's semaglutide data from FDA, PubMed abstract, and a Reddit thread" - β†’ Show 3 assertions with different source_class values - -2. CONFLICT: "FDA says nausea rate is 44%. Reddit anecdotes claim 80%." - β†’ Query with SkepticLens, show conflict_score = 0.72 - -3. HIERARCHY: "But FDA is Tier 0, Reddit is Tier 5." - β†’ Query with LayeredConsensusLens, show FDA wins despite volume - -4. TIME TRAVEL: "What did we know 6 months ago?" - β†’ Query with as_of, show different winner before Reddit data existed - -5. DECAY: "That Reddit data is 8 months old now." - β†’ Query with source_class_decay=true, show Reddit's effective confidence dropped - -6. PARADIGM SHIFT: "FDA just updated the label." - β†’ Create new epoch, show old assertions superseded -``` - -### MVP Workstream - -| Week | Deliverable | Owner | Depends On | -|------|-------------|-------|------------| -| **Week 1** βœ… | Domain definitions, SubjectBuilder, pharma schema | Ontology | β€” | -| **Week 2** βœ… | FDA extractor, claim-to-assertion signing | Ontology | Week 1 | -| **Week 3** βœ… | Ingest FDA claims, mock conflicts, SkepticLens demo | Ontology | Week 2 | -| **Week 4** βœ… | UAT scenarios documented and verified | Ontology | Week 3 | -| **Week 5** βœ… | `steme-pharma` CLI for self-serve exploration | Ontology | Week 3 | -| **Week 6** | Polish, factor out reusable patterns, document | Ontology | Week 4-5 | - -### What's NOT in MVP - -These are valuable but not required to prove the core value proposition: - -- Phase 8B-C (Observability, geo-distribution) β€” production concerns -- Phase 9 (Backup, GDPR, disaster recovery) β€” operational concerns -- PubMed PDF extraction β€” complex, mock data sufficient for demo -- Browser extension β€” app layer, consumes MVP - ---- - -## Detailed Milestones - -### Phase 1: The Spine (Foundation) -*Goal: Securely ingest assertions and persist them without data loss.* - -- [x] **Project Scaffold**: Initialize Rust workspace, set up linting/CI (clippy, fmt). -- [x] **Assertion Schema**: Define the `Assertion` struct with `rkyv` serialization. - - [x] Add dependencies: `rkyv`, `blake3`, `ed25519-dalek`, `image_hasher`. - - [x] Define `Assertion` struct (Subject, Predicate, Object, Confidence, SourceHash). - - [x] **Multi-Sig Expansion**: Implement `SignatureEntry` struct and `signatures: Vec` field. - - [x] **Visual Expansion**: Add `visual_hash: Option` field for image provenance. - - [x] Test serialization round-trips. -- [x] **Ballot Schema**: Define the `Vote` struct for multi-agent consensus. - - [x] Add `Vote` struct: `assertion_hash`, `agent_id`, `weight`, `signature`. - - [x] Test serialization round-trips. -- [x] **Paradigm Schema (Epochs)**: Define the `Epoch` and `SupersessionType` structs. - - [x] Add `epoch: Option` to `Assertion`. - - [x] Implement `Epoch` struct with `supersedes` and `SupersessionType`. - - [x] Test serialization round-trips. -- [x] **WAL Integration**: Implement the Quarantine Pattern for write-ahead logging. - - [x] Create `stemedb-wal` crate. - - [x] Port `FsyncGuard` and `Record` logic from established durability patterns. - - [x] Implement Record format with BLAKE3 checksums and Headers. - - [x] Verify `fsync` behavior with tests. -- [x] **Storage Engine**: Implement the `Store` trait using `sled` (embedded KV). - - [x] Add `sled` dependency. - - [x] Define `KVStore` trait (put, get, delete, scan_prefix, flush). - - [x] Implement `SledStore` wrapper. -- [x] **Basic Ingestor**: Background worker that tails WAL and writes to KV. - - [x] Implement async loop reading from WAL. - - [x] Write deserialized assertions, votes, and epochs to `sled`. - - [x] Ed25519 signature verification during ingestion. - - [x] Maintains S: and SP: indexes on ingest. - - [x] Persistent cursor/checkpoint (resumes from `__CURSOR__:ingest` in KV store). -- [x] **Verification**: Crash recovery tests (write -> crash -> restart -> read). - - [x] Single and multi-record crash recovery. - - [x] Multiple crash cycles tested. - -### Phase 2: The Lattice (Connectivity) -*Goal: Query data with sub-millisecond latency using Materialized Views.* - -- [x] **Lifecycle Schema**: Add `LifecycleStage` to Assertion. - - [x] Define enum: `Proposed`, `UnderReview`, `Approved`, `Deprecated`, `Rejected`. - - [x] Update `Assertion` struct and serialization tests. -- [x] **The Ballot Box**: Implement high-velocity vote ingestion. - - [x] `VoteStore` trait and implementation. - - [x] `VoteAwareConsensusLens` for real vote-based resolution. -- [x] **Index Infrastructure**: Compound indexes for O(1) queries. - - [x] `IndexStore` trait with S: and SP: indexes. - - [x] `QueryEngine` smart routing (SP -> S -> scan). -- [x] **Materializer**: Background worker for O(1) Read Performance. - - [x] `MaterializedView` type in `stemedb-core`. - - [x] `Materializer` worker in `stemedb-query` with `step()` and `run()`. - - [x] Aggregates Votes via `VoteAwareConsensusLens` (or any `AsyncLens`). - - [x] Updates `MV:{Subject}:{Predicate}` with the winning Assertion + metadata. - - [x] Event-driven mode via `run_notified()` with `tokio::sync::Notify`. - - [x] Fast-path MV lookup in `QueryEngine::try_fast_path()`. -- [x] **The Meter**: Implement Economic Throttling (TAN). - - [x] `QuotaStore` trait and `GenericQuotaStore` implementation. - - [x] Token Bucket algorithm with per-agent per-hour quotas. - - [x] `MeterLayer` tower middleware for request cost tracking. - - [x] Cost model: Assert=10, Vote=1, Query=5+lens, +1/KB payload. - - [x] `GET /v1/meter/quota` endpoint to check remaining quota. - - [x] `POST /v1/meter/quota/limit` admin endpoint to set custom limits. -- [x] **API Surface**: `axum` HTTP server with OpenAPI (utoipa). - - [x] `POST /v1/assert` -> Accepts JSON, writes to WAL. - - [x] `POST /v1/vote` -> High-throughput vote endpoint. - - [x] `POST /v1/epoch` -> Create epoch with optional supersession. - - [x] `GET /v1/query` -> Subject/Predicate/Lens/Lifecycle/Epoch filtering. - - [x] `GET /v1/health` -> Health check with assertion count. - - [x] `GET /swagger-ui` -> Interactive API docs. - - [x] 5 lens types available: Recency, Consensus, Authority, VoteAwareConsensus, TrustAwareAuthority. -- [x] **Query Audit**: Log every read with provenance. - - [x] Define `QueryAudit` struct: query_id, agent_id, timestamp, params, result_hash, contributing_assertions. - - [x] Storage at `AUD:{query_id}` with agent index at `AUDA:{agent_id}:{timestamp}:{query_id}`. - - [x] `GET /v1/audit/queries` -> Returns history of agent decisions. - - [x] `GET /v1/audit/query/{id}` -> Full reasoning trace for a single query. - - [x] Auto-logging on every query via `X-Agent-Id` header. - -### Phase 2.5: Hardening (Camp 2 Fixes) -*Goal: Close the gaps between "built" and "works right." Every item here addresses a feature that exists but doesn't fully deliver on its promise.* - -- [x] **2.1 MV Staleness Detection**: Make the fast-path aware of stale materialized views. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] Added `max_stale: Option` to `Query` struct in `crates/stemedb-query/src/query.rs`. - - [x] Added `.max_stale(secs)` builder method to `QueryBuilder`. - - [x] In `try_fast_path()`: if `query.max_stale` is set and MV age exceeds threshold, falls through to slow path with `debug!` log. - - [x] Added `max_stale` to API `QueryParams` DTO in `crates/stemedb-api/src/dto.rs`. - - [x] Wired through query handler in `crates/stemedb-api/src/handlers/query.rs`. - - **Tests:** - - [x] `test_fast_path_stale_view_falls_back`: MV 1000 seconds old, `max_stale = 60` β†’ slow path used. - - [x] `test_fast_path_fresh_view_used`: Fresh MV, `max_stale = 300` β†’ fast path used. - - [x] `test_fast_path_no_max_stale_always_uses_mv`: No `max_stale` β†’ any MV age accepted (backward compatible). - - [x] `test_fast_path_max_stale_zero_rejects_old_mv`: `max_stale = 0`, MV 1 second old β†’ slow path. - - [x] `test_fast_path_max_stale_zero_accepts_brand_new_mv`: `max_stale = 0`, brand new MV β†’ fast path. - -- [x] **2.2 AuthorityLens -> ConfidenceLens Rename**: Eliminate the misleading name. - - **Problem:** `AuthorityLens` selects by `confidence` field, not by agent reputation. `TrustAwareAuthorityLens` is the real authority lens. The name creates confusion about what "Authority" means. - - **Solution implemented:** - - [x] Renamed `authority.rs` β†’ `confidence.rs`, `AuthorityLens` β†’ `ConfidenceLens` - - [x] Added `LensDto::Confidence` for the confidence-field selector - - [x] Changed `LensDto::Authority` to route to `TrustAwareAuthorityLens` (the real authority lens) - - [x] Updated query handler routing - - [x] Updated ai-lookup/services/lens.md and skill documentation - -- [x] **2.3 EpochAwareLens**: Give epoch supersession runtime behavior. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] `EpochAwareLens` in `crates/stemedb-lens/src/epoch_aware.rs` - - [x] Decorator pattern wrapping any inner lens (default: RecencyLens) - - [x] Walks supersession chain from `E:{epoch_id}` keys - - [x] Cycle detection + max depth guard (100) - - [x] Fail-open on missing epochs - - [x] `LensDto::EpochAware` added to API - - [x] 11 tests: excludes_superseded, chain_supersession, no_epochs_passes_all, missing_epoch_includes, cycle_detection, consensus_lens_inner, mixed_epochs, etc. - - [x] Documentation updated in `ai-lookup/services/lens.md` - - **Known Limitation:** Filtering only occurs when assertions from the superseding epoch are present in candidates. If all candidates are from old epoch (no new epoch assertions), they pass through (fail-open behavior). - -- [x] **2.4 Visual Hash Query Support**: Make the stored `visual_hash` queryable. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] `hamming_distance(a: &PHash, b: &PHash) -> u32` in `crates/stemedb-query/src/query.rs` (lines 26-28) - - [x] `visual_near: Option` and `visual_threshold: Option` in `Query` struct (lines 84-90) - - [x] `.visual_near(hash, threshold)` builder method - - [x] `Query::matches()` computes hamming distance when `visual_near` is set - - [x] API `QueryParams` DTO has `visual_near` and `visual_threshold` - - [x] 10+ tests: exact_match, within_threshold, exceeds_threshold, skips_without_hash, invalid_hex, wrong_length, combines_with_subject, default_threshold, max_threshold, threshold_63_rejects - - **Note:** Brute-force O(N) scan. VP-tree/BK-tree index is Phase 3+. - -- [x] **2.5 Vector Field**: No changes needed. Already roadmapped for Phase 3. - - **Status:** βœ… N/A (No Phase 2 work required) - - **Current state:** `vector: Option>` on `Assertion`. Stored and returned by API. No index, no search. - - **Phase 3 plan:** Integrate `hnsw-rs` or `lance` for k-NN search. - -- [x] **2.6 E2E Integration Test (Write -> Materialize -> Read)**: Prove the full pipeline works end-to-end. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] `crates/stemedb-query/tests/e2e_pipeline.rs` with 5 comprehensive tests: - - `test_e2e_write_materialize_read` - Basic happy path - - `test_e2e_vote_consensus` - Vote-weighted resolution - - `test_e2e_update_winner` - Winner changes on re-materialize - - `test_e2e_cursor_persistence` - Cursor survives worker restart - - `test_e2e_notify_integration` - Event-driven notification channel - - [x] `stemedb-wal` and `stemedb-ingest` added as dev-dependencies - - [x] Helper functions: `create_signed_assertion()`, `compute_assertion_hash()`, `create_vote()` - - [x] Uses Ed25519 signing for authentic signature verification - - [x] Also: `crates/stemedb-api/tests/e2e_flow_test.rs` tests the HTTP API layer end-to-end. - -### Phase 3: The Pilot (BioTech/Pharma) -*Goal: Prove value in the "High-Liability" beachhead. Close every Camp 4 gap that blocks a credible demo.* - -#### 3A. Schema Expansion (Prerequisite for everything below) - -- [x] **3A.1 Source-Class Field**: Add `source_class: SourceClass` to Assertion. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] `SourceClass` enum in `crates/stemedb-core/src/types.rs` (lines 68-88). - - [x] 6-tier system: `Regulatory` (0), `Clinical` (1), `Observational` (2), `Expert` (3), `Community` (4), `Anecdotal` (5). - - [x] `tier()` method returns tier number for ordering. - - [x] `default_decay_days()` method for tier-specific confidence decay. - - [x] `authority_weight()` method for conflict resolution weighting. - - [x] Field on `Assertion` struct at line 152. - - [x] Full serialization and indexing support. - -- [x] **3A.2 Conflict Score on Resolution**: Add `conflict_score: f32` to Resolution. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] Added `conflict_score: f32` field to `Resolution` in `crates/stemedb-lens/src/traits.rs`. - - [x] Updated `Resolution::empty()` to set `conflict_score: 0.0`. - - [x] Updated `Resolution::with_winner()` to accept `conflict_score` parameter. - - [x] Added `compute_conflict_score(candidates: &[Assertion]) -> f32` utility function: - - Uses normalized variance of confidence values. - - 0 or 1 candidates β†’ 0.0 (no conflict possible). - - All same confidence β†’ 0.0 (unanimous). - - Max variance (0.0 vs 1.0) β†’ 1.0 (maximum conflict). - - Defensive NaN handling (returns 0.0 for malformed data). - - [x] Updated all lens implementations to compute and pass conflict score: - - `crates/stemedb-lens/src/recency.rs` - - `crates/stemedb-lens/src/consensus.rs` - - `crates/stemedb-lens/src/confidence.rs` - - `crates/stemedb-lens/src/vote_aware_consensus.rs` - - `crates/stemedb-lens/src/trust_aware_authority.rs` - - [x] Added `conflict_score: f32` to `MaterializedView` in `crates/stemedb-core/src/types.rs`. - - [x] Updated `Materializer::materialize_pair()` to write `conflict_score` from resolution. - - [x] Added `conflict_score: Option` and `resolution_confidence: Option` to `QueryResponse` DTO in `crates/stemedb-api/src/dto.rs` (only present when lens is applied). - - [x] Wired through query handler in `crates/stemedb-api/src/handlers/query.rs`. - - **Tests:** - - [x] `test_conflict_score_zero_for_empty`: Empty candidates β†’ 0.0. - - [x] `test_conflict_score_zero_for_single`: 1 candidate β†’ 0.0. - - [x] `test_conflict_score_zero_for_agreement`: All same confidence β†’ near 0.0. - - [x] `test_conflict_score_high_for_disagreement`: Candidates at 0.1, 0.5, 0.9 β†’ score > 0.3. - - [x] `test_conflict_score_max_for_extremes`: 0.0 vs 1.0 β†’ score β‰ˆ 1.0. - - [x] `test_conflict_score_handles_nan_defensively`: NaN confidences β†’ 0.0 (fail-safe). - -- [x] **3A.3 Rich Source Metadata**: Add structured provenance beyond `source_hash`. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] Added `source_metadata: Option>` field to `Assertion` in `crates/stemedb-core/src/types.rs` (after `epoch`, before `lifecycle`). - - [x] Uses `Vec` (not `String`) for rkyv zero-copy compatibility. Callers encode/decode JSON on their side. - - [x] Added `source_metadata: Option>` to `AssertionBuilder` in `crates/stemedb-core/src/testing.rs`. - - [x] Added `.source_metadata_json(json: &str)` and `.source_metadata(bytes)` builder methods. - - [x] Added `source_metadata: Option` to `CreateAssertionRequest` DTO (JSON string in API, converted to bytes internally). - - [x] Added `source_metadata: Option` to `AssertionResponse` DTO (bytes converted to JSON string with defensive UTF-8 handling). - - [x] Wired through create handler (`dto_to_assertion()`) and query handler (`assertion_to_dto()`). - - **Tests:** - - [x] `test_serialize_deserialize_assertion_with_metadata`: Serialization roundtrip with metadata present. - - [x] `test_serialize_deserialize_assertion_without_metadata`: Serialization roundtrip with metadata absent. - - **Note:** Metadata is stored but NOT indexed in Phase 3. Indexing individual metadata fields is Phase 4+. - -#### 3B. Time & Decay (Core Query Features) - -- [x] **3B.1 Time-Travel Engine**: `as_of` parameter for historical queries. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] Added `as_of: Option` to `Query` struct in `crates/stemedb-query/src/query.rs:92-99`. - - [x] Added `.as_of(timestamp: u64)` to `QueryBuilder`. - - [x] In `Query::matches()`: if `as_of` is `Some(ts)`, check `assertion.timestamp <= ts`. Assertions created after `as_of` are excluded. - - [x] In `QueryEngine::execute()`: if `query.as_of` is set, **skip the fast path entirely** (MVs reflect current state, not historical). - - [x] Added `as_of: Option` to `QueryParams` DTO in `crates/stemedb-api/src/dto.rs`. - - [x] Wired through query handler. - - **Tests:** - - [x] `test_as_of_excludes_future_assertions`: Assertions filtered by timestamp. - - [x] `test_as_of_bypasses_fast_path`: MV exists, but `as_of` is set. Slow path used. - - [x] `test_as_of_none_uses_fast_path`: Normal query still uses fast path (backwards-compatible). - - [x] `test_as_of_with_lens_resolves_among_historical_candidates`: Time-travel + lens = resolve only among pre-as_of candidates. - - [x] `test_as_of_returns_empty_when_all_assertions_are_future`: All assertions are future, returns empty. - - [x] `test_as_of_with_exact_timestamp_match`: Edge case where assertion.timestamp == as_of. - -- [x] **3B.2 Semantic Decay**: Confidence Half-Life at query time. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] Added `decay_halflife: Option` to `Query` struct in `crates/stemedb-query/src/query.rs`. - - [x] Added `.decay_halflife(seconds: u64)` and `.source_class_decay(enabled: bool)` to `QueryBuilder`. - - [x] Added `decay_halflife` and `source_class_decay` to `QueryParams` DTO. - - [x] Created new `crates/stemedb-query/src/decay.rs` module with: - - `apply_decay()`: Uniform decay using formula `confidence * 2^(-(age / halflife))`. - - `apply_source_class_decay()`: Tier-specific decay (Regulatory=none, Clinical=2yr, Anecdotal=30d). - - `compute_decayed_confidence()`: Core decay calculation with clamping. - - [x] Integrated in `QueryEngine::execute()`: decay applied after filtering, before lens resolution. - - [x] Time-travel compatible: uses `as_of` timestamp if set, otherwise current time. - - [x] Source-class-aware decay fully implemented using `SourceClass::default_decay_days()`. - - **Tests:** (11 unit tests in decay.rs + 1 E2E test) - - [x] `test_decay_reduces_old_assertion_confidence`: 1yr old, 1yr halflife β†’ ~50% confidence. - - [x] `test_decay_preserves_fresh_assertions`: 1hr old, 1yr halflife β†’ ~100% confidence. - - [x] `test_decay_interacts_with_lens`: Older high-confidence loses to newer low-confidence after decay. - - [x] `test_source_aware_decay_tier0_no_decay`: Regulatory never decays. - - [x] `test_source_aware_decay_tier5_rapid_decay`: Anecdotal decays rapidly (30-day halflife). - - [x] `test_source_aware_decay_mixed_tiers`: Clinical vs Anecdotal tier comparison. - - [x] `test_decay_zero_halflife_no_change`: Zero halflife skips decay (avoids div-by-zero). - - [x] `test_decay_future_assertion_no_change`: Future assertions don't decay. - - [x] `test_decay_empty_assertions`: Empty input returns empty output. - - [x] `test_decay_confidence_clamps_to_valid_range`: Very old assertions clamp to [0.0, 1.0]. - - [x] `test_decay_preserves_other_fields`: Only confidence changes; other fields preserved. - - [x] `test_e2e_decay_reduces_old_confidence`: Full pipeline E2E test in e2e_pipeline.rs. - - **Note:** When decay is enabled, materialized views (fast path) are bypassed because MVs store pre-computed winners without decay applied. - -#### 3C. New Lenses - -- [x] **3C.1 Skeptic Lens**: Surface disagreement, not winners. βœ… **COMPLETED** - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] `crates/stemedb-lens/src/skeptic.rs` - Full implementation. - - [x] `AnalysisLens` trait for lenses that analyze conflict instead of resolving it. - - [x] `SkepticLens` uses normalized Shannon entropy for conflict scoring. - - [x] Returns `ConflictAnalysis` with: - - `conflict_score: f32` (0.0 = unanimous, 1.0 = chaos) - - `status: ResolutionStatus` (Unanimous, Agreed, Contested) - - `claims: Vec` - all claims ranked by weight - - [x] `SkepticResolver` + `SkepticView` in `stemedb-query/src/skeptic.rs`. - - [x] `GET /v1/skeptic?subject=X&predicate=Y` API endpoint. - - [x] Core types in `stemedb-core/src/types.rs`: - - `ResolutionStatus` enum - - `ConflictAnalysis` struct - - `ClaimSummary`, `SourceSummary`, `AgentSummary` - - [x] Comprehensive test coverage (21 test cases). - -- [x] **3C.2 Layered Consensus Lens**: Per-source-class consensus. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] `crates/stemedb-lens/src/layered_consensus.rs` - Full implementation. - - [x] `TierResolution` struct: per-tier result with tier, source_class, winner, candidates_count, conflict_score, resolution_confidence. - - [x] `LayeredResolution` struct: multi-tier result with tiers vec, overall_winner, overall_conflict_score, total_candidates. - - [x] `LayeredLens` trait: `resolve_layered(&[Assertion]) -> LayeredResolution`, `name() -> &'static str`. - - [x] `LayeredConsensusLens` implements both `LayeredLens` and `Lens` traits. - - [x] Cross-tier conflict score uses normalized Shannon entropy of tier winner object values. - - [x] `LensDto::LayeredConsensus` variant (redirects to `/v1/layered` endpoint). - - [x] `GET /v1/layered?subject=X&predicate=Y` API endpoint with `LayeredQueryResponse`. - - [x] Exported from `crates/stemedb-lens/src/lib.rs`. - - **Tests:** - - [x] `test_layered_empty_candidates`: Empty input returns empty resolution. - - [x] `test_layered_single_tier`: All same source_class, returns one tier result. - - [x] `test_layered_multi_tier_agreement`: Tier 0 and Tier 5 agree, low cross-tier conflict. - - [x] `test_layered_multi_tier_disagreement`: Tier 1 vs Tier 5 disagree, high conflict, Tier 1 wins. - - [x] `test_layered_overall_winner_from_highest_authority`: Tier 0 wins despite fewer assertions. - - [x] `test_layered_lens_trait_compatibility`: Standard Lens trait works. - - [x] `test_layered_within_tier_conflict`: High internal conflict within a tier. - - [x] `test_layered_all_tiers_present`: One assertion from each tier. - - [x] `test_layered_lens_name`: Both trait names work. - - [x] `test_layered_numeric_values`: Works with numeric object values. - -- [x] **3C.3 Constraints Lens**: Pre-flight check for must_use/forbidden. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] `crates/stemedb-lens/src/constraints.rs` - Full implementation. - - [x] `ConstraintSet` struct: holds categorized assertions (must_use, forbidden, prefer) with conflict_score. - - [x] `ConstraintsLens` struct with `resolve_constraints(&[Assertion]) -> ConstraintSet` method. - - [x] Categorizes by predicate pattern: `must_use:*`, `forbidden:*`, `prefer:*`. - - [x] Implements `Lens` trait for compatibility (priority: must_use > forbidden > prefer). - - [x] Sorted by confidence (highest first), with timestamp as tiebreaker. - - [x] `LensDto::Constraints` added (redirects to `/v1/constraints` endpoint). - - [x] `GET /v1/constraints?subject=X` API endpoint with `ConstraintsResponse`. - - [x] DTOs: `ConstraintsQueryParams`, `ConstraintEntryDto`, `ConstraintsResponse`. - - [x] Exported from `crates/stemedb-lens/src/lib.rs`. - - **Tests:** (16 test cases) - - [x] `test_constraints_categorizes_by_predicate`: Mixed predicates sorted into must_use/forbidden/prefer. - - [x] `test_constraints_empty_categories`: Only prefer, no must_use/forbidden. - - [x] `test_constraints_non_constraint_predicates_ignored`: Regular predicates filtered out. - - [x] `test_constraints_sorted_by_confidence`: Within-category confidence ordering. - - [x] `test_constraints_empty_candidates`: Empty input returns empty set. - - [x] `test_constraints_has_constraints_true`: Helper method works. - - [x] `test_constraints_all_regular_predicates`: All non-constraint predicates returns no constraints. - - [x] `test_lens_trait_picks_must_use_winner`: Standard Lens trait picks must_use first. - - [x] `test_lens_trait_falls_back_to_forbidden`: Falls back to forbidden when no must_use. - - [x] `test_lens_trait_falls_back_to_prefer`: Falls back to prefer when no must_use/forbidden. - - [x] `test_lens_trait_empty_for_no_constraints`: Returns empty when no constraint predicates. - - [x] `test_lens_name`: Name returns "Constraints". - - [x] `test_lens_empty_candidates`: Empty input to Lens trait returns empty resolution. - - [x] `test_multiple_must_use_picks_highest_confidence`: Multiple must_use picks highest confidence. - - [x] `test_confidence_tiebreaker_uses_timestamp`: Same confidence uses newer timestamp. - - [x] `test_predicate_pattern_exact_prefix`: `must_use_something` not matched (only `must_use:*`). - -#### 3D. Epoch Enhancement - -- [x] **3D.1 Epoch Cascade Logic** (enhancement of Phase 2.5 EpochAwareLens): - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] `write_supersession_cascade()` in `crates/stemedb-ingest/src/worker.rs`: - - Writes `SUPERSEDED:{old_epoch_id}` markers for full transitive closure. - - All markers point to the LATEST superseding epoch. - - Max depth guard (100 levels) and cycle detection via visited set. - - [x] `is_epoch_superseded()` in `crates/stemedb-lens/src/epoch_aware.rs`: - - O(1) marker lookup instead of O(chain_length) chain walks. - - Fail-open semantics: missing marker = not superseded. - - [x] `compute_superseded_epochs()` uses marker lookups for filtering. - - **Tests:** - - [x] `test_cascade_writes_superseded_marker`: Epoch B supersedes A β†’ `SUPERSEDED:A` exists. - - [x] `test_cascade_transitive`: Cβ†’Bβ†’A chain β†’ both `SUPERSEDED:A` and `SUPERSEDED:B` point to C. - - [x] `test_cascade_cycle_detection`: Mutual supersession handled gracefully. - - [x] `test_epoch_aware_uses_marker`: EpochAwareLens uses O(1) marker lookup. - - [x] `test_superseded_epoch_filtered_even_without_new_assertions`: Marker-based filtering works. - -#### 3E. Similarity Search - -- [x] **3E.1 Vector Search**: Semantic k-NN queries via embeddings. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] Added `hnsw_rs = "0.3"` and `parking_lot = "0.12"` to `stemedb-storage/Cargo.toml`. - - [x] New module: `crates/stemedb-storage/src/vector_index.rs`. - - [x] `VectorIndex` trait: `insert(hash: &Hash, vector: &[f32])`, `search(query: &[f32], k: usize) -> Vec<(Hash, f32)>`, `dimension()`, `len()`, `is_empty()`. - - [x] `HnswVectorIndex` implementation with HNSW graph, RwLock protection, hash↔ID mappings. - - [x] Input validation: dimension mismatch, NaN, Infinite values rejected. - - [x] Idempotent insert (same hash twice = no-op). - - [x] `Arc` trait object support for sharing. - - [x] `IngestWorker::with_vector_index()` builder method for index attachment. - - [x] IngestWorker: if assertion has `vector`, inserts into vector index after KV write. - - [x] Added `vector_near: Option>` and `k: Option` to `Query` struct in `crates/stemedb-query/src/query.rs`. - - [x] Added `.vector_near(vector, k)` builder method to `QueryBuilder`. - - [x] Added `vector_near` and `k` to API `QueryParams` DTO. - - [x] `QueryEngine::with_vector_index()` builder method for index attachment. - - [x] QueryEngine: if `vector_near` is set and index configured, uses O(log N) HNSW lookup for candidates. - - [x] Falls back to standard path if no index configured (with debug log). - - **Tests:** (12 unit tests for VectorIndex + 4 integration tests in engine.rs) - - [x] `test_create_index`, `test_insert_and_search`, `test_dimension_mismatch`. - - [x] `test_idempotent_insert`, `test_search_empty_index`, `test_search_k_zero`. - - [x] `test_nan_rejection`, `test_infinite_rejection`, `test_contains`. - - [x] `test_larger_scale` (100 vectors, exact match first), `test_custom_params`, `test_zero_dimension_panics`. - - [x] `test_vector_search_returns_nearest_neighbors`, `test_vector_search_with_subject_filter`. - - [x] `test_vector_search_without_index_falls_back`, `test_vector_search_with_as_of_filter`. - - **Note:** Index is in-memory only. Persistence is Phase 4+. - -- [x] **3E.2 Visual Hash Index**: BK-tree for O(log N) visual similarity. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] New module: `crates/stemedb-storage/src/visual_index.rs`. - - [x] `VisualIndex` trait: `insert(hash: &Hash, phash: &PHash)`, `search(query: &PHash, threshold: u32) -> Vec<(Hash, u32)>`, `len()`, `is_empty()`. - - [x] `BkTreeVisualIndex` implementation using BK-tree over hamming distance. - - [x] `hamming_distance(a: &PHash, b: &PHash) -> u32` utility function. - - [x] Threshold clamped to 0-64 range (max 64 bits). - - [x] Results sorted by distance ascending. - - [x] Idempotent insert (same hash twice = no-op). - - [x] `Arc` trait object support for sharing. - - [x] `IngestWorker::with_visual_index()` builder method for index attachment. - - [x] IngestWorker: if assertion has `visual_hash`, inserts into BK-tree after KV write. - - [x] `QueryEngine::with_visual_index()` builder method for index attachment. - - [x] QueryEngine: if `visual_near` is set and index configured, uses O(log N) BK-tree lookup. - - [x] Falls back to brute-force scan (via `query.matches()`) if no index configured. - - [x] Invalid hex input returns `QueryError::InvalidInput` with clear message. - - **Tests:** (14 unit tests for VisualIndex + 6 integration tests in engine.rs) - - [x] `test_hamming_distance_zero`, `test_hamming_distance_max`, `test_hamming_distance_partial`. - - [x] `test_create_index`, `test_insert_and_search_exact`, `test_search_within_threshold`. - - [x] `test_search_no_matches`, `test_search_empty_index`, `test_idempotent_insert`. - - [x] `test_contains`, `test_results_sorted_by_distance`, `test_threshold_clamped_to_64`. - - [x] `test_larger_scale` (1000 hashes), `test_default_impl`. - - [x] `test_visual_search_returns_similar_images`, `test_visual_search_with_lifecycle_filter`. - - [x] `test_visual_search_invalid_hex_returns_error`, `test_visual_search_without_index_uses_brute_force`. - - [x] `test_visual_search_with_limit`, `test_vector_search_empty_index`. - - **Note:** Index is in-memory only. Persistence is Phase 4+. - -#### 3F. Provenance - -- [x] **3F.1 Source Document Storage & Provenance Lookup**: Enable 100% citation recall. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] `POST /v1/source` endpoint to store source documents by BLAKE3 content hash. - - [x] `GET /v1/provenance/{hash}` endpoint to retrieve source documents by hash. - - [x] Source storage at `SRC:{hash}` keys with format: `[content_type_len:4][content_type][content]`. - - [x] Base64 encoding for binary-safe JSON transport. - - [x] 10MB size limit per document. - - [x] Content-addressed storage: same content β†’ same hash (idempotent uploads). - - [x] DTOs: `StoreSourceRequest`, `StoreSourceResponse`, `ProvenanceResponse`. - - [x] OpenAPI documentation under "provenance" tag. - - **Tests:** (5 test cases) - - [x] `test_store_and_retrieve_source`: Happy path store + retrieve. - - [x] `test_store_source_invalid_base64`: Bad base64 β†’ 400. - - [x] `test_get_provenance_not_found`: Unknown hash β†’ 404. - - [x] `test_get_provenance_invalid_hash`: Bad hash format β†’ 400. - - [x] `test_store_source_idempotent`: Same content twice β†’ same hash. - - **Note:** Benchmark utility for verifying all assertions have retrievable sources is future work. - -#### 3G. API Cleanup - -- [x] **3G.1 Document epoch supersession via existing endpoint**: No new `/epoch/supersede` endpoint needed. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] Updated use case docs (consumer-health-intelligence.md, glp1-living-review.md) to use `POST /v1/epoch` with `supersedes` field. - - [x] Added OpenAPI examples showing both new epoch and supersession flows in handlers/epoch.rs. - - [x] Documented all 5 supersession types: Invalidate, Temporal, Refinement, RequiresReview, Additive. - - **No code change.** Documentation fix only. - -### Phase 4: The Hive (Trust & Scale) -*Goal: Change tracking, metadata indexing, and the database primitives for training pipelines.* - -- [x] **TrustRank Engine**: Foundation for trust-based resolution. - - [x] `TrustRankStore` for per-agent reputation storage. - - [x] `TrustAwareAuthorityLens` for reputation-weighted resolution. - - [x] **Confidence Half-Life**: Implement decay calculation engine. - - [x] Learning loop: `record_outcome()` for accuracy tracking. - -- [x] **4.1 "Since" Parameter**: Change tracking for returning consumers. - - **Status:** βœ… COMPLETE - - **Problem:** Consumer Health shows `GET /query?since=2023-10-01` returning `changes_since` with dated change entries. The "returning consumer" story: "What changed since I last looked?" - - **Depends on:** Time-Travel (3B.1) and Materializer. - - **Implementation:** - - [x] Added `since: Option` to `Query` struct and `QueryBuilder` in `crates/stemedb-query/src/query.rs`. - - [x] Added `since` to `QueryParams` DTO in `crates/stemedb-api/src/dto.rs`. - - [x] **MV Changelog**: Track when materialized views change. - - [x] New key pattern: `MVC:{subject}:{predicate}:{timestamp_nanos}` using nanosecond precision to prevent collisions. - - [x] `ChangeEntry` struct in `crates/stemedb-core/src/types.rs` with: `timestamp`, `previous_winner_hash`, `new_winner_hash`, `subject`, `predicate`, `lens_name`. - - [x] In `Materializer::materialize_pair()`: before overwriting MV, read existing MV. If winner changed (different hash), write changelog entry. - - [x] `get_previous_winner_hash()` and `write_changelog_entry()` helper methods. - - [x] In QueryEngine: if `since` is set, skip fast path (MVs reflect current state). `fetch_changes_since()` scans `MVC:` keys and returns entries >= since timestamp. - - [x] `ChangeEntryDto` in `crates/stemedb-api/src/dto.rs` with hex-encoded hashes. - - [x] `changes_since: Option>` added to `QueryResponse` DTO. - - [x] `fetch_changelog_if_needed()` helper in query handler. - - [x] Re-exported `ChangeEntry` from `stemedb-core` crate root. - - **Tests:** - - [x] `test_changelog_written_on_first_materialization`: First MV creates changelog with `previous_winner_hash: None`. - - [x] `test_changelog_written_on_winner_change`: Winner change creates changelog with previous hash. - - [x] `test_no_changelog_when_winner_unchanged`: Re-materialize same winner = no new changelog. - - [x] `test_fetch_changes_since_returns_entries`: Fetch filtered by timestamp, sorted ascending. - - [x] `test_fetch_changes_since_empty_on_no_entries`: No entries returns empty vec. - - [x] `test_since_bypasses_fast_path`: Query with `since` uses slow path, returns all assertions. - -- [x] **4.2 Source Metadata Indexing** (extension of 3A.3): Index key metadata fields. - - **Status:** βœ… COMPLETE - - **Problem:** Phase 3 stores `source_metadata` as an opaque blob. Phase 4 makes key fields queryable. - - **Depends on:** Rich Source Metadata (3A.3). - - **Implementation:** - - [x] Defined indexed metadata keys: `journal`, `doi`, `platform`, `study_design` in `INDEXED_METADATA_FIELDS`. - - [x] New module: `crates/stemedb-storage/src/source_metadata_index.rs`. - - [x] Key pattern: `SMV:{field}:{value}` storing `Vec` (rkyv-serialized) for efficient batch lookup. - - [x] `SourceMetadataIndexStore` trait with `add_to_metadata_indexes()` and `get_by_metadata_field()`. - - [x] `GenericSourceMetadataIndexStore` implementation with case-insensitive value normalization. - - [x] IngestWorker: on ingestion, if `source_metadata` is present, parse JSON, extract indexed fields, write index entries. - - [x] Added `source_journal`, `source_doi`, `source_platform`, `source_study_design` to `Query` struct. - - [x] Added `.source_journal()`, `.source_doi()`, `.source_platform()`, `.source_study_design()` builder methods. - - [x] Added `matches_metadata_filters()` helper with AND semantics and case-insensitive matching. - - [x] Added metadata field filters to `QueryParams` DTO (e.g., `?source_journal=NEJM`). - - [x] Wired through query handler in `crates/stemedb-api/src/handlers/query.rs`. - - **Tests:** (9 unit tests in source_metadata_index.rs + 9 unit tests in query.rs) - - [x] `test_add_and_get_by_metadata_field`: Basic round-trip. - - [x] `test_case_insensitive_indexing`: "NEJM" stored, "nejm" query matches. - - [x] `test_multiple_assertions_same_field`: Vec accumulates correctly. - - [x] `test_malformed_json_gracefully_skipped`: Invalid JSON logs warning, doesn't error. - - [x] `test_missing_field_not_indexed`: Only indexed fields processed. - - [x] `test_idempotent_insert`: Same assertion twice = no duplicates. - - [x] `test_all_indexed_fields`: All 4 fields indexed correctly. - - [x] `test_empty_index_returns_empty_vec`: Non-existent field returns empty. - - [x] `test_non_string_values_ignored`: Non-string JSON values skipped. - -- [x] **4.3 Batch TrustRank Decay API**: Expose scheduled decay for external orchestration. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] New module: `crates/stemedb-api/src/handlers/admin.rs` - - [x] `POST /v1/admin/decay-trust-ranks` endpoint. - - [x] Request accepts `now: Option` (defaults to current time) and `half_life_seconds: Option` (defaults to 30 days). - - [x] Response includes `decayed_count`, `timestamp_used`, `half_life_used`, `status`. - - [x] OpenAPI docs with "admin" tag. - - [x] Thin handler delegates to `TrustRankStore::decay_trust_ranks()`. - - **Tests:** (3 integration tests in http_integration.rs) - - [x] `test_decay_trust_ranks_empty_store`: Empty store returns 0 decayed. - - [x] `test_decay_trust_ranks_with_custom_params`: Custom timestamp and half-life honored. - - [x] `test_decay_trust_ranks_response_structure`: All 4 fields present in response. - - **Note:** The Gardener (Camp 5.2, app layer) calls this endpoint on a schedule. The database just exposes the primitive. - -- [x] **4.4 Vote Provenance Witness**: Transform votes from opinions into cryptographic witnesses. - - **Status:** βœ… COMPLETE - - **Problem:** Votes record "I agree" but the browser extension product needs "I saw this exact text at this URL at this time." Without provenance, votes are Tier 5 noise. - - **Implementation:** - - [x] Added `source_url: Option` to Vote struct β€” URL where claim was observed. - - [x] Added `observed_context: Option>` to Vote struct β€” page context bytes (rkyv zero-copy pattern). - - [x] API DTOs: `source_url` and `observed_context` on CreateVoteRequest/VoteResponse. - - [x] Input validation: source_url max 2048 chars (non-empty if provided), observed_context max 64KB. - - [x] Backward compatible: existing votes without provenance remain valid. - - **Tests:** Serialization roundtrip (with/without provenance, mixed fields), storage roundtrip, API integration (validation). - -- [x] **4.5 Conflict Score Filtering**: Query-time filtering by conflict score. - - **Status:** βœ… COMPLETE - - **Problem:** The browser extension needs "only show me claims where conflict > 0.7" to implement contradiction-only overlays. Conflict score was computed but not filterable. - - **Implementation:** - - [x] Added `min_conflict_score: Option` and `max_conflict_score: Option` to Query struct. - - [x] Builder methods: `.min_conflict_score()`, `.max_conflict_score()`. - - [x] Fast-path filtering: checks MV conflict_score against thresholds. - - [x] API validation: scores must be 0.0-1.0, finite (rejects NaN/Inf). - - [x] QueryParams DTO wired through query handler. - - **Tests:** 13 engine tests (min/max thresholds, exact boundary, range, combination with lifecycle), API validation tests. - - **Note:** Filtering works on fast path (MV reads) only. Slow path returns raw assertions without conflict scores. - -- [x] **4.6 Escalation Triggers**: Active safety system for high-conflict assertions. - - **Status:** βœ… COMPLETE - - **Problem:** High conflict was invisible. Episteme should be an active safety system that fires escalations when disagreement exceeds thresholds. - - **Implementation:** - - [x] `EscalationLevel` enum: Low, Medium, High, Critical. - - [x] `EscalationEvent` struct: content-addressed (BLAKE3 ID), subject, predicate, conflict_score, level, reason, timestamp, resolved. - - [x] `EscalationPolicy` struct: configurable threshold + level + optional predicate pattern. - - [x] `EscalationStore` trait + `GenericEscalationStore`: write, query since, resolve, get pending. - - [x] Key pattern: `ESC:{timestamp_nanos}:{id_hex}`. - - [x] Materializer integration: after computing conflict_score, checks policies and writes events. - - [x] `with_escalation(store, policies)` builder on Materializer. - - [x] `GET /v1/admin/escalations` and `POST /v1/admin/escalations/{id}/resolve` endpoints. - - **Tests:** Core type tests, storage roundtrip, materializer integration (high/low conflict, predicate matching, no store configured). - -- [x] **4.7 Gold Standard Verification**: Sybil defense via proof of knowledge. - - **Status:** βœ… COMPLETE - - **Problem:** New agents need a way to earn TrustRank. Gold standards are admin-verified assertions that agents can be tested against. - - **Implementation:** - - [x] `GoldStandard` struct with rkyv serialization: assertion_hash, subject, predicate, expected_object, created_at, created_by. - - [x] `GoldStandardStore` trait + `GenericGoldStandardStore`: set, get, list, remove. - - [x] Key pattern: `GS:{subject}:{predicate}`. - - [x] `TrustAdjustment` enum: Rewarded(+0.05), Penalized(-0.1), AlreadyVerified. - - [x] `verify_agent_against_gold_standard()` on TrustRankStore with deduplication (agents can only verify each gold standard once). - - [x] Verification markers at `GS_VERIFIED:{agent_id}:{subject}:{predicate}` prevent gaming. - - [x] API: `POST/GET/DELETE /v1/admin/gold-standards`, `POST /v1/admin/verify-agent`. - - **Tests:** Core (creation, matching, case-sensitivity), storage CRUD, trust adjustment (reward, penalty, clamping, dedup), API integration (5 tests). - -> **Note:** The following items were reclassified as **Application Layer** responsibilities (see `tmp/ambition-vs-reality.md`, Camp 5). They are not Episteme database features. They consume the Episteme API and are built by integrators or vertical-specific teams. +> **Pilot Status:** Consumer Health MVP complete βœ… | Enterprise Demo in progress > -> - **The Simulator** (Training Data Pipeline) -> Camp 5.3 -> - **The Super Curator** (Reviewer Agent swarm) -> Camp 5.4 -> - **Background Gardener** (Cluster detection, signal processing) -> Camp 5.2 -> - **Agent Wallet** (Key management sidecar) -> Camp 5.1 +> **Archive:** For completed phases 1-7, see [roadmap-archive.md](./roadmap-archive.md) -### Phase 5: The Forge (Foundation Hardening) -*Goal: Replace abandoned dependencies, fix WAL gaps, persist indices. Prerequisite for distribution.* +--- -> **Research:** [docs/research/wal-crash-recovery-research.md](docs/research/wal-crash-recovery-research.md) +## Current Status -#### 5A. Storage Engine Replacement +| Phase | Status | Summary | +|-------|--------|---------| +| **1-7** | βœ… Complete | Core infrastructure, distributed cluster, trust & safety | +| **8A** | βœ… Complete | Chaos testing, Jepsen-style verification | +| **MVP** | βœ… Complete | Consumer Health demo with real FDA data | +| **Pilot Prep** | 🎯 In Progress | Dashboard, impact analysis, production hardening | +| **8B-C** | Planned | Observability, geo-distribution | +| **9** | Planned | Disaster recovery, compliance, storage management | -- [x] **5A.1 Replace sled with redb + fjall**: sled is abandoned (author recommends alternatives). - - **Problem:** sled is alpha-stage with known performance regressions and no active development. Our entire storage layer depends on it. - - **Solution:** HybridStore routes keys by prefix β€” **fjall** (LSM) for write-heavy paths (`H:`, `V:`, `VC:`, `VW:`, `E:`, `SUPERSEDED:`, `__CURSOR__:`) and **redb** (B-tree) for read-heavy paths (`S:`, `SP:`, `MV:`, `TR:`, `QA:`, `QT:`, `TP:`, `GS:`, `ESC:`). - - **Tasks:** - - [x] Generalize `StorageError::Sled` to `StorageError::Backend(String)`. - - [x] Implement `FjallStore` backend with DashMap per-key locks for atomics. - - [x] Implement `RedbStore` backend with ACID transactions. - - [x] Implement `HybridStore` routing layer with prefix-based dispatch. - - [x] Migrate all ~500 tests from `SledStore` to `HybridStore`. - - [x] Remove sled dependency entirely. - - [x] Add criterion benchmarks (sequential put, random get, prefix scan, atomic increment, mixed workload). - - **Crates:** `redb = "2"`, `fjall = "2"`, `dashmap = "6"` +--- -- [x] **5A.2 Key Layout Redesign**: Prepare keys for subject-prefix range sharding. - - **Problem:** Current keys (`H:{hash}`, `S:{subject}`, `MV:{subject}:{predicate}`) scatter related data across the keyspace. Distributed sharding needs co-location. - - **Solution:** Subject-prefix key layout with `\x00` separator for subject-scoped keys, `\x00` prefix for global keys (sort-first): - ``` - Subject-prefixed (co-located): - {subject}\x00H:{hash} β†’ Assertion data - {subject}\x00S:{hash_list} β†’ Subject index - {subject}\x00SP:{predicate} β†’ Compound index - {subject}\x00MV:{predicate} β†’ Materialized view - {subject}\x00V:{hash}:{vh} β†’ Votes - {subject}\x00VC:{hash} β†’ Vote count cache - {subject}\x00VW:{hash} β†’ Vote weight cache - {subject}\x00GS:{predicate} β†’ Gold standards +## 🎯 Phase: Enterprise Pilot Preparation (CURRENT) - Global (sort first via \x00 prefix): - \x00TRUST:{agent_id} β†’ Trust ranks - \x00QUOTA:{agent_id}:{win} β†’ Quota records - \x00QLIMIT:{agent_id} β†’ Quota limits - \x00E:{epoch_id} β†’ Epochs - \x00SUPERSEDED:{epoch_id} β†’ Supersession markers - \x00SUP:{hash} β†’ Supersession records - \x00AUD:{query_id} β†’ Audit records - \x00ESC:{ts}:{id} β†’ Escalation events - \x00TP:{pack_id} β†’ Trust packs - \x00META:{key} β†’ System metadata - \x00HASH_SUBJECT:{hash} β†’ Reverse lookup index - \x00SUBJECTS:{subject} β†’ Known subjects index - \x00GS_LIST:{subj}:{pred} β†’ Gold standard listing - ``` - - **Implementation:** - - [x] `key_codec.rs` (573 lines): 40+ key builder functions, subject validation, extraction utilities, 30+ unit tests. - - [x] All stores migrated to `key_codec::` functions (91 call sites across 10 store files, zero hardcoded key patterns). - - [x] Ingestion pipeline uses `key_codec` (11 usages across 3 files). - - [x] Query engine uses `key_codec` (34 usages across 7 files). - - [x] Subject co-location verified by `test_subject_colocation` test. - - [x] Global key sort-first verified by `test_global_keys_sort_first` test. +> **Goal:** Make the pilot bulletproof. Amaze enterprise decision makers. +> **Timeline:** 5 weeks +> **Success Criteria:** Dr. Sarah Chen (skeptical VP of Data Infrastructure) fights her CFO for budget -#### 5B. WAL Hardening +### The 5 Amazement Moments We Must Deliver -- [x] **5B.1 CRC32C Checksums**: Add hardware-accelerated torn write detection. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] Added `crc32c = "0.6"` crate dependency. - - [x] Updated WAL record format with CRC32C in `format.rs`. - - [x] CRC32C verified on read before deserializing. - - [x] Hardware-accelerated via SSE 4.2 on supported CPUs. +| # | Moment | Current State | Gap | +|---|--------|---------------|-----| +| 1 | Contradictions visible with confidence scores | βœ… Complete | Dashboard scaffold + Skeptic Query UI βœ… | +| 2 | Cascade invalidation when source retracted | ⚠️ Manual | No automatic cascade | +| 3 | Full FDA-ready audit trail | βœ… API ready | Dashboard scaffold ready, UI pending (P1.6) | +| 4 | Point-in-time queries + decay | βœ… API ready | No timeline UI | +| 5 | Malicious agent blocked by circuit breaker | βœ… API ready | Dashboard scaffold ready, UI pending (P1.5) | -- [x] **5B.2 Crash Recovery Implementation**: Replace recovery stub with production recovery. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] `recovery/mod.rs` (236 lines): Full sequential record scan with CRC32C validation. - - [x] Truncates file at first corrupted/incomplete record. - - [x] `RecoveryMetrics` struct tracks: valid_records, invalid_records, bytes_truncated, recovery_duration. - - [x] Comprehensive test suite in `recovery/tests.rs`. +### Pilot-1: Demo Dashboard (Week 1-2) -- [x] **5B.3 Group Commit**: Batch fsync for throughput. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] `group_commit.rs` (342 lines): `GroupCommitBuffer` with configurable `max_writes` and `max_duration`. - - [x] Writers append to buffer and wait on `Notify`. - - [x] Background flusher calls fsync and notifies all waiters. - - [x] `GroupCommitConfig` for tuning batch size and flush interval. +> **Deliverable:** React admin dashboard that makes the API visual -- [x] **5B.4 Log Rotation**: Bounded WAL disk usage. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] `segment.rs` (368 lines): Full segment management. - - [x] Segment naming: `{seq:08x}.wal`. - - [x] Rotation when segment exceeds configurable threshold. - - [x] `SegmentManager` tracks active and archived segments. - - [x] Safe deletion of segments after cursor passes them. +- [x] **P1.1 Dashboard Scaffold**: Next.js + shadcn/ui project setup βœ… + - [x] Project structure: `applications/stemedb-dashboard/` + - [x] API client for StemeDB endpoints (`src/lib/api/client.ts`) + - [x] Authentication scaffold (API key header) + - [x] Dark mode (default), responsive layout with collapsible sidebar + - [x] shadcn/ui components: button, card, badge, input, separator, tabs + - [x] Live API status indicator (polls /health every 30s) + - [x] Port 18188, builds and runs successfully -#### 5C. Index Persistence +- [x] **P1.2 Skeptic Query Visualization**: Show contradictions graphically βœ… + - [x] Query builder: subject, predicate inputs + - [x] Conflict score gauge (0.0-1.0 with color coding) + - [x] Claims table with weight bars, source tier badges + - [x] "CONTESTED" / "AGREED" / "UNANIMOUS" status badges + - [x] Expandable claim rows with source details, agents, provenance hashes + - [x] Loading skeleton, empty state, error state with retry -- [x] **5C.1 Persistent Vector Index**: Move HNSW from in-memory to disk-backed. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] `PersistentVectorIndex` in `crates/stemedb-storage/src/vector_index/persistent.rs`. - - [x] Hybrid hot/cold architecture: - - **Hot:** In-memory HNSW for recent vectors. - - **Cold:** Disk-backed HNSW loaded from checkpoint files. - - [x] `persistence/format.rs`: `SnapshotMetadata`, `IdMappingTable` with rkyv serialization. - - [x] `hot_cold.rs`: `merge_search_results()` for combining hot and cold query results. - - [x] Background checkpoint task with atomic write pattern. - - [x] CRC32C integrity verification on load. - - [x] MAX_PAYLOAD_SIZE (1GB) validation to prevent memory exhaustion. - - **Crates:** `memmap2 = "0.9"`, `crc32c = "0.6"`, `byteorder = "1.5"` - - **Known Limitation:** Cold index currently stores ID mappings only; full vector persistence with mmap'd HNSW graphs planned for future phase. +- [ ] **P1.3 Layered Consensus View**: Per-tier breakdown + - [ ] Tier accordion showing each source class + - [ ] Within-tier conflict score + - [ ] Cross-tier conflict visualization + - [ ] "Overall winner" highlight with confidence -- [x] **5C.2 Persistent Visual Index**: Persist BK-tree to disk. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] `PersistentVisualIndex` in `crates/stemedb-storage/src/visual_index.rs`. - - [x] `BkTreeSnapshot` with rkyv serialization for BK-tree state. - - [x] Checkpoint file format: `[MAGIC:4][VERSION:1][RESERVED:3][PAYLOAD_LEN:u64][CRC32C:u32][PAYLOAD]`. - - [x] Atomic write pattern: temp file β†’ fsync β†’ rename β†’ fsync parent. - - [x] Background checkpoint task with configurable interval. - - [x] CRC32C integrity verification on load. - - [x] Shared `checkpoint_format.rs` module for common read/write utilities. +- [ ] **P1.4 Quarantine Admin Panel**: Content defense visibility + - [ ] Pending queue with reason, timestamp, quality score + - [ ] Approve/Reject buttons with confirmation + - [ ] Filter by reason (duplicate, spam, untrusted high-confidence) + - [ ] Metrics: pending count, approved/rejected today -#### 5D. Concept Hierarchy βœ… COMPLETE +- [ ] **P1.5 Circuit Breaker Status**: Trust & safety dashboard + - [ ] Blocked agents list with failure count, retry time + - [ ] State badges: OPEN (red), HALF_OPEN (yellow), CLOSED (green) + - [ ] Manual reset button for admin override + - [ ] Historical trip events -> **Spec:** [docs/specs/concept-hierarchy.md](docs/specs/concept-hierarchy.md) -> **Purpose:** Hierarchical, scheme-qualified subject identifiers with cross-scheme alias resolution. Enables applications like Aphoria that need to connect `code://` paths to `rfc://` paths. +- [ ] **P1.6 Audit Trail Browser**: Query provenance explorer + - [ ] Recent queries list with agent, timestamp, subject + - [ ] Drilldown: contributing assertions, weights, winner + - [ ] Filter by agent, time range, subject + - [ ] Export to JSON/CSV -- [x] **5D.1 ConceptPath Type**: Structured subject identifiers. βœ… - - **Tasks:** - - [x] Add `ConceptPath` struct to `stemedb-core/src/types/concept.rs`. - - [x] Wire format: `{scheme}://{segment_0}/{segment_1}/.../{leaf}`. - - [x] `parse()`, `to_wire_format()`, `leaf()`, `parent()`, `is_prefix_of()`. - - [x] Backward-compatible: bare strings parse as `custom://{string}`. - - [x] Unit tests for parsing, round-trip, prefix matching (Battery 8). - - **Crate:** `stemedb-core` +### Pilot-2: Demo Data Seeder (Week 2) -- [x] **5D.2 Source Scheme Registry**: Map schemes to default source tiers. βœ… - - **Tasks:** - - [x] Add `SourceScheme` enum to `stemedb-core`. - - [x] Scheme β†’ default `SourceClass` mapping (e.g., `rfc://` β†’ Tier 0, `code://` β†’ Tier 3). - - [x] `ConceptPath::default_source_class()` method. - - **Crate:** `stemedb-core` +> **Deliverable:** Pre-signed realistic demo data using Go SDK -- [x] **5D.3 Alias Store**: Cross-scheme entity resolution. βœ… - - **Tasks:** - - [x] Add `ConceptAlias` struct to `stemedb-core`. - - [x] Add `AliasStore` trait to `stemedb-storage`. - - [x] Key prefixes: `CA:{alias_path}` β†’ canonical, `CAR:{canonical}` β†’ all aliases. - - [x] Transitive alias resolution with cycle detection. - - [x] `GenericAliasStore` implementation over `KVStore`. - - **Crates:** `stemedb-core`, `stemedb-storage` +- [ ] **P2.1 Demo Keypair Management**: Reproducible demo keys + - [ ] 5 demo agents with known keys (FDA_AGENT, CLINICAL_AGENT, REDDIT_AGENT, etc.) + - [ ] Keys stored in `demo/keys/` (gitignored for real deploys) + - [ ] Go SDK script: `cmd/demo-seed/main.go` -- [x] **5D.4 Hierarchical Query**: Prefix-based subject queries. βœ… - - **Tasks:** - - [x] `fetch_by_subject_prefix()` using `scan_prefix` in query engine (already implemented in Battery 5). - - [x] Trailing `/` handling to prevent `auth` matching `authentication`. - - **Crate:** `stemedb-query` - - **Note:** Hierarchical prefix scanning was already working; Battery 5 validates it. +- [ ] **P2.2 Conflict Scenarios**: Pre-built disagreements + - [ ] Gastroparesis risk: FDA (0.2%) vs Reddit (high) + - [ ] Cardiovascular benefit: Trial A vs Trial B (conflicting results) + - [ ] Nausea rate: Label vs real-world evidence gap + - [ ] 500+ total assertions across 10 subjects -- [x] **5D.5 Alias Resolution in Queries**: Expand queries to aliased paths. βœ… - - **Tasks:** - - [x] `AliasStore::resolve_all()` for transitive alias expansion. - - [x] API endpoint `GET /v1/concepts/resolve?path=...&transitive=true`. - - **Crate:** `stemedb-query`, `stemedb-api` - - **Note:** Resolution available via API; QueryEngine integration is future work. +- [ ] **P2.3 Retractable Sources**: Set up cascade demo + - [ ] One "landmark study" source that will be retracted + - [ ] 50+ assertions citing this source + - [ ] Script to mark source as quarantined + - [ ] Script to show impact -- [x] **5D.6 Source Class Inference**: Infer tier from scheme. βœ… - - **Tasks:** - - [x] `ConceptPath::default_source_class()` returns tier based on scheme. - - [x] `SourceScheme::parse()` maps scheme strings to enum variants. - - **Crate:** `stemedb-core` - - **Note:** Inference at ingestion time would break content-addressing (signature verification). Inference is available at query time or before signing. +- [ ] **P2.4 Historical Data**: Time-travel demo data + - [ ] Assertions with timestamps spanning 12 months + - [ ] Knowledge state that changed (FDA label update scenario) + - [ ] Before/after demonstration -- [x] **5D.7 Concept API Endpoints**: CRUD for aliases and hierarchy browsing. βœ… - - **Tasks:** - - [x] `POST /v1/concepts/alias` β€” Create alias. - - [x] `GET /v1/concepts/aliases` β€” List all aliases (with optional canonical filter). - - [x] `DELETE /v1/concepts/alias` β€” Remove alias. - - [x] `GET /v1/concepts/resolve` β€” Resolve path to canonical/transitive aliases. - - [x] `GET /v1/concepts/suggest` β€” Suggested aliases (shared leaf detection). - - [x] `GET /v1/concepts/parse` β€” Parse path and return ConceptPath info. - - **Crate:** `stemedb-api` +### Pilot-3: Impact Analysis (Week 3) -- [x] **5D.8 Battery Tests**: Validate concept hierarchy end-to-end. βœ… - - **Tests:** - - [x] Battery 8 (7 tests): ConceptPath parsing, round-trip, prefix matching, source class inference. - - [x] Battery 9 (8 tests): Alias resolution, transitive resolution, cycle detection, bidirectional lookup, delete, suggestions. - - **Crate:** `stemedb-query/tests/battery_pre_sentinel.rs` +> **Deliverable:** Automatic cascade when source is retracted -### Phase 6: The Mesh (Distributed Writes) -*Goal: Multi-node cluster with CRDT replication and Raft coordination. The endgame.* +- [ ] **P3.1 Impact Analysis Endpoint**: `GET /v1/sources/{hash}/impact` + - [ ] Returns all assertions citing this source + - [ ] Returns count of queries that used those assertions + - [ ] Returns list of affected agents + - [ ] Implementation in `stemedb-api/src/handlers/source_registry/` -> **Research:** [docs/research/distributed-write-path.md](docs/research/distributed-write-path.md) -> **Agent:** `distributed-systems-engineer` -> **Key Insight:** Episteme's append-only model eliminates ~75% of CockroachDB complexity. Assertions are a G-Set CRDT. Votes are G-Counters. No distributed transactions needed. +- [ ] **P3.2 Cascade Flagging**: Automatic downstream impact + - [ ] When source status β†’ quarantined, flag citing assertions + - [ ] New field on assertion index: `source_status: Option` + - [ ] Queries can filter by `exclude_quarantined_sources=true` + - [ ] Alternative: `SourceAwareLens` that checks source status at resolution -#### 6A. CRDT Foundation (Single-Node Validation) βœ… COMPLETE +- [ ] **P3.3 Impact Dashboard Widget**: Visualize the cascade + - [ ] Source status change UI (Active β†’ Quarantined) + - [ ] Animated "impact ripple" showing affected count + - [ ] List of impacted queries with timestamp + - [ ] "Remediation status" tracking -- [x] **6A.1 Integrate CRDT Crate**: Wrap assertion storage in G-Set semantics. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] `CrdtAssertionStore` in `crates/stemedb-storage/src/crdt/assertion_store.rs` β€” G-Set semantics for assertions. - - [x] `CrdtVoteStore` in `crates/stemedb-storage/src/crdt/vote_store.rs` β€” G-Counter semantics for votes. - - [x] `CrdtMerge` trait in `crates/stemedb-storage/src/crdt/traits.rs` for generic merge operations. - - [x] Property tests: commutativity, associativity, idempotence (proptest-based). - - [x] `AssertionTransfer` type for efficient cross-node data transfer. - - **Tests:** 9 unit tests + 3 property tests (assertion_store), 6 unit tests (vote_store). - - **Note:** Did not use external `crdts` crate β€” implemented native CRDT semantics over existing storage. +### Pilot-4: Production Hardening (Week 4) -- [x] **6A.2 Hybrid Logical Clocks**: Add causal ordering to supersessions. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] `HlcTimestamp` in `crates/stemedb-core/src/types/hlc.rs` β€” serializable HLC with `uhlc` integration. - - [x] Added `uhlc = "0.8"` dependency to `stemedb-core`. - - [x] `HlcTimestamp::from_uhlc()`, `to_uhlc()`, `now()` for clock management. - - [x] Total ordering via NTP64 time + node_id tiebreaker. - - [x] `detect_clock_skew()` utility for monitoring clock drift between nodes. - - [x] `millis()`, `is_before()`, `is_concurrent_with()` helper methods. - - **Tests:** 10 unit tests covering ordering, equality, concurrency, serialization, clock skew detection. - - **Crate:** `uhlc = "0.8"` +> **Deliverable:** Load testing, authentication, backup documentation -- [x] **6A.3 Merkle Tree Over Assertions**: Efficient diff detection. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] New `stemedb-merkle` crate with BLAKE3-based Merkle tree. - - [x] `MerkleTree` struct: O(log N) insert, O(1) root, O(log N) diff. - - [x] `DiffResult::diff()` for computing missing hashes between trees. - - [x] `roots_equal()` for O(1) identity check. - - [x] Zero-copy serialization via rkyv for network transfer. - - [x] `MerkleTreeManager` in `stemedb-sync` for persistence and coordination. - - **Crate:** `crates/stemedb-merkle/` +- [ ] **P4.1 Load Testing**: Prove performance claims + - [ ] k6 or wrk load test scripts + - [ ] Benchmark: 10K assertions baseline latency + - [ ] Benchmark: 1K writes/sec sustained for 1 hour + - [ ] Benchmark: 100 concurrent readers, no degradation + - [ ] Document results in `uat/production-readiness/` -#### 6B. Two-Node Replication (Proof of Concept) βœ… COMPLETE +- [ ] **P4.2 API Authentication**: Basic security for pilot + - [ ] API key middleware (`X-API-Key` header) + - [ ] Per-key rate limiting (separate from per-agent quota) + - [ ] Admin keys vs read-only keys + - [ ] Key management: `POST /v1/admin/api-keys` -> **Why "Proof of Concept":** All primitives are implemented and unit/integration tested. The PoC validates that CRDT merge, HLC ordering, Merkle diff, gossip broadcast, and anti-entropy sync work correctly in isolation. Full network tests (two running gRPC servers, partition tolerance, concurrent writes) are deferred to 6C where cluster infrastructure provides a natural testing environment. +- [ ] **P4.3 Backup/Restore Documentation**: DR story + - [ ] Document WAL-based recovery procedure + - [ ] Script: `scripts/backup-stemedb.sh` (snapshot + WAL archive) + - [ ] Script: `scripts/restore-stemedb.sh` (restore from backup) + - [ ] Test restore procedure, document in UAT -- [x] **6B.1 RPC Layer**: Node-to-node communication. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] New `stemedb-rpc` crate with tonic gRPC. - - [x] `proto/sync.proto` defines: `GossipRequest/Response`, `RootExchangeRequest/Response`, `FetchRequest/Response`, `PingRequest/Response`, `GetLeavesRequest/Response`. - - [x] `SyncClient` in `src/client.rs` with `RetryConfig` for exponential backoff. - - [x] `SyncServiceHandler` in `src/server.rs` implementing `SyncService` trait. - - [x] `SyncStorage` trait for pluggable storage backends. - - **Crates:** `tonic = "0.12"`, `prost = "0.13"` - - **Crate:** `crates/stemedb-rpc/` +- [ ] **P4.4 Prometheus Metrics**: Observability baseline + - [ ] `GET /metrics` endpoint with prometheus format + - [ ] Key metrics: `assertions_total`, `queries_total`, `query_latency_seconds` + - [ ] Trust metrics: `quarantine_pending`, `circuit_breakers_open` + - [ ] Basic Grafana dashboard template -- [x] **6B.2 Gossip Broadcast**: Push new assertions to peers. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] `GossipBroadcaster` in `crates/stemedb-sync/src/gossip.rs`. - - [x] Configurable fanout (default: 3 peers). - - [x] Token bucket rate limiting via `with_rate_limit()`. - - [x] Enable/disable support for maintenance windows. - - [x] Metrics: `messages_sent`, `send_failures`, `rate_limited`. - - [x] Best-effort delivery: failures logged but don't block ingestion. - - [x] `GossipBroadcast` trait in `stemedb-ingest` for dependency injection. - - **Tests:** 3 unit tests (noop, no peers, enable/disable). +### Pilot-5: Operational Readiness (Week 5) -- [x] **6B.3 Merkle Anti-Entropy Sync**: Background convergence. - - **Status:** βœ… COMPLETE - - **Implementation:** - - [x] `AntiEntropyWorker` in `crates/stemedb-sync/src/anti_entropy.rs`. - - [x] Periodic root exchange via `RootExchangeRequest`. - - [x] `compute_missing_hashes()` compares local and remote leaf sets. - - [x] `FetchRequest` retrieves missing assertion data by hash. - - [x] Merge via `CrdtAssertionStore::merge_with_data()`. - - [x] Merkle tree update after merge. - - [x] Configurable interval via `SyncConfig`. - - [x] Metrics: `sync_cycles`, `sync_failures`, `assertions_synced`. - - [x] Graceful shutdown support. - - **Tests:** 1 unit test (SyncResult variants). +> **Deliverable:** Runbooks, monitoring, reference architecture -- [x] **6B.4 Integration Test: Two-Node Convergence**: - - **Status:** βœ… COMPLETE (component-level validation) - - **Implementation:** - - [x] `battery11_replication.rs` with 8 tests validating replication primitives: - - `test_identical_trees_same_root` β€” Merkle root equality. - - `test_different_trees_different_roots` β€” Merkle root divergence. - - `test_merkle_diff_finds_missing` β€” Diff algorithm correctness. - - `test_gossip_enable_disable` β€” Gossip control. - - `test_merkle_checkpoint_restore` β€” Persistence roundtrip. - - `test_content_addressed_idempotent` β€” Idempotent storage. - - `test_crdt_merge_with_data` β€” CRDT merge semantics. - - `test_sync_config_builder` β€” Configuration validation. - - **Note:** Tests validate primitives in isolation. Live network tests (real gRPC servers, partition healing, concurrent writes) deferred to 6C cluster testing. - - **Crate:** `crates/stemedb-query/tests/battery/battery11_replication.rs` +- [ ] **P5.1 Operational Runbooks**: Common procedures documented + - [ ] "Server won't start" troubleshooting + - [ ] "High query latency" investigation + - [ ] "Quarantine queue overflow" handling + - [ ] "Circuit breaker stuck open" resolution + - [ ] "Restore from backup" step-by-step -#### 6C. Multi-Node Cluster βœ… +- [ ] **P5.2 Reference Architecture**: Deployment guide + - [ ] Single-node pilot deployment diagram + - [ ] Network requirements (ports, firewall rules) + - [ ] Reverse proxy configuration (nginx/envoy with TLS) + - [ ] Resource sizing guide (CPU, memory, disk) -- [x] **6C.1 Cluster Membership (SWIM Gossip)**: Node discovery and failure detection. - - **Tasks:** - - [x] Implement `SwimMembership` with SWIM-like protocol in `stemedb-cluster`. - - [x] `NodeId` (UUID-based), `NodeInfo`, `NodeState`, `MembershipEvent` types. - - [x] Seed-node based discovery (bootstrap nodes in config). - - [x] Failure detection: ping, indirect probe, suspicion with timeouts. - - [x] Membership change events via `tokio::broadcast` channel. - - [x] Gossip queue for piggybacked membership propagation. - - [x] `ClusterConfig` with `SwimConfig` (tunable intervals, timeouts). - - **Crate:** `stemedb-cluster` +- [ ] **P5.3 Pilot Success Criteria Document**: Definition of done + - [ ] Sub-second query latency at 10K assertions: measured + - [ ] Successful conflict detection on known contradictory studies: demonstrated + - [ ] Complete audit trail export for mock regulatory review: tested + - [ ] Source retraction workflow: exercised -- [x] **6C.2 Subject-Prefix Range Sharding**: Distribute data across nodes. - - **Tasks:** - - [x] Implement `RangeRouter`: map subject β†’ shard via BLAKE3 + jump hash. - - [x] `RangeDescriptor`: start key, end key, replicas, size, generation. - - [x] `MetaRange`: collection of descriptors with version and merge logic. - - [x] Automatic range split when size exceeds threshold (configurable, default 64MB). - - [x] Range merge when adjacent ranges shrink below threshold (configurable, default 20MB). - - [x] Meta-range gossip merge for cluster-wide propagation. - - [x] `ShardingConfig` with tunable shard count, replication factor, thresholds. - - **Crate:** `stemedb-cluster` +- [ ] **P5.4 Executive Demo Script Validation**: End-to-end rehearsal + - [ ] Run through `amazement-demo-2.md` with real dashboard + - [ ] Time each segment (target: 20 minutes total) + - [ ] Anticipate and document answers to 10 tough questions + - [ ] Record demo video for async sharing + - [ ] All 5 Aha Moments demonstrable with real data (not mockups) -- [ ] **6C.3 Raft for MV Coordination (Optional)**: DEFERRED. - - **Decision:** Skipped for this delivery. MVs are eventually consistent (converge once assertions sync via anti-entropy). Lenses are deterministic: same inputs produce same output. Can add Raft later if strong MV consistency becomes a requirement. +### Pilot Prep Deliverables Summary -- [x] **6C.4 Gateway**: Stateless request routing. - - **Tasks:** - - [x] Implement `Gateway` HTTP service (axum) with full routing. - - [x] Route writes by subject hash β†’ shard β†’ leader node. - - [x] Route reads to nearest replica (prefer local). - - [x] Health check endpoint (`/v1/health`). - - [x] Cluster status endpoint (`/v1/cluster/status`). - - [x] Shard info and route test endpoints. - - [x] CORS and tracing middleware. - - **Crate:** `stemedb-cluster` +| Week | Deliverable | Owner | Acceptance Criteria | +|------|-------------|-------|---------------------| +| 1-2 | `stemedb-dashboard` | Frontend | 6 functional panels, connects to API | +| 2 | `demo-seed` | SDK | 500+ signed assertions, retractable sources | +| 3 | Impact Analysis | Backend | `/v1/sources/{hash}/impact` returns cascade | +| 4 | Load Test Results | QA | 10K assertions, 1K writes/sec documented | +| 4 | API Authentication | Backend | API keys work, rate limiting functional | +| 4 | Backup/Restore | Ops | Documented and tested procedure | +| 4 | Metrics Endpoint | Backend | `/metrics` returns Prometheus format | +| 5 | Runbooks | Ops | 5 runbooks in `docs/runbooks/` | +| 5 | Reference Architecture | Docs | Deployment guide complete | +| 5 | Demo Rehearsal | All | 20-minute demo runs smoothly | -- [x] **6C.5 Integration Tests**: 82 tests covering membership, sharding, and gateway. - - Membership: 3-node discovery, failure detection, rejoin, gossip propagation. - - Sharding: routing consistency, distribution, split/merge, meta-range gossip. - - Gateway: HTTP endpoint testing via axum `oneshot` for all routes. +--- -#### 6D. Consistency Guarantees +## Phase 8B-C: Production Observability (Planned) -| Property | Guarantee | Mechanism | -|----------|-----------|-----------| -| **Convergence** | Eventually consistent | G-Set merge (CRDT) | -| **Causality** | Supersessions ordered | HLC timestamps | -| **Partition Tolerance** | Writes never blocked | Any node accepts via CRDT | -| **Availability** | Reads/writes always succeed | Every node is master for CRDTs | -| **Durability** | WAL + fsync per node | Existing WAL infra | -| **Conflict Resolution** | Deterministic | Lens algorithms (same inputs β†’ same output) | +> **Blocked by:** Pilot Prep (need real production deployment first) -### Phase 7: The Shield (Trust at Scale) -*Goal: Defend against spam, Sybil attacks, and knowledge poisoning when open to millions of agents.* - -> **Key Insight:** Open agent access requires layered defense. PoW for admission, EigenTrust for reputation, circuit breakers for misbehavior, quarantine for suspicious content. - -#### 7A. Admission Control βœ… - -- [x] **7A.1 Proof-of-Work Admission**: BLAKE3 hashcash for new agents. - - New agents (trust < 0.3) must solve PoW puzzle before first N assertions accepted. - - Graduated difficulty: first 10 assertions = 16 bits (~16 sec), 11-50 = 1 bit (trivial), 50+ or trust > 0.6 = exempt. - - Puzzle: `BLAKE3(nonce || agent_id || timestamp)` must have `difficulty` leading zero bits. - - Implemented: `PowProof` struct, `AdmissionLayer` middleware, HTTP 428 responses. - -- [x] **7A.2 Graduated Trust Tiers**: Privilege escalation based on reputation. - - Untrusted (0.0-0.3): Quarantine mode, assertions hidden by default, 0.1x quota. - - Limited (0.3-0.5): Low quota, PoW required, 0.5x quota. - - Verified (0.5-0.7): Standard quota, normal privileges. - - Trusted (0.7-0.9): 2x quota, skip PoW. - - Authority (0.9-1.0): 10x quota, no limits. - - Implemented: `TrustTier` enum, `AdmissionStore` trait, `/v1/admission/status` endpoint. - -#### 7B. EigenTrust βœ… - -- [x] **7B.1 Trust Graph Store**: Store direct trust relationships for propagation. - - Key pattern: `TG:{from}:{to}` β†’ TrustEdge, `TGR:{to}:{from}` β†’ reverse index. - - Methods: `add_trust_edge()`, `get_trusts()`, `get_trusted_by()`, `compute_eigentrust()`. - - Seed trust: `ET:seed:{agent}` for pre-trusted agents (P vector). - -- [x] **7B.2 EigenTrust Computation**: Global trust via power iteration (daily batch). - - Formula: `T = (1-Ξ±)C^T*T + Ξ±P` where C = normalized trust matrix, P = seed trust, Ξ± = 0.1. - - Convergence: 10-100 iterations, Ξ΅ = 1e-4 threshold. - - Sybil resistance: isolated rings get near-zero trust (not connected to seeds). - - Dangling node handling: redistribute to seed vector. - -- [x] **7B.3 Domain-Specific Trust**: Per-predicate-namespace reputation. - - `DomainTrust` tracks accuracy by domain (medicine, finance, technology, etc.). - - `extract_domain()` maps predicates to domains. - - `domain_factor = 0.5 + (score Γ— 0.5)` scales weight by expertise. - - `EigenTrustAuthorityLens`: `weight = confidence Γ— eigentrust Γ— domain_factor`. - -#### 7C. Content Defense βœ… COMPLETE - -- [x] **7C.1 MinHash Deduplication**: Near-duplicate detection with LSH bucketing. - - `SimilarityIndex` trait with `GenericSimilarityIndex` implementation. - - MinHash signatures (k=128) for `{subject}:{predicate}` pairs. - - LSH buckets (16 bands Γ— 8 rows) for O(1) average-case lookup. - - Bloom filter pre-check for fast "definitely not duplicate" path. - - Threshold: 0.9 Jaccard similarity = duplicate. - - Implemented: `similarity_index/` module in `stemedb-storage`. - -- [x] **7C.2 Content Quality Scoring**: Heuristic-based spam detection. - - `ContentQualityScorer` with configurable thresholds. - - Shannon entropy check (low entropy = likely random noise/repetitive). - - Minimum subject/predicate length (3 chars default). - - Structured data bonus (JSON objects, numbers, URLs). - - Untrusted agent + high confidence (>0.8) = suspicious. - - Implemented: `content_defense/quality.rs` in `stemedb-storage`. - -- [x] **7C.3 Quarantine Store**: Suspicious assertions held for review. - - `QuarantineStore` trait with `GenericQuarantineStore` implementation. - - Key pattern: `QUAR:{timestamp}:{hash}` β†’ assertion data (time-ordered). - - Secondary index: `QUAR_IDX:{hash}` β†’ timestamp for O(1) hash lookups. - - Quarantined assertions NOT indexed (invisible to queries). - - Triggers: quality < 0.4, duplicate, untrusted high-confidence. - - Admin API: `GET /v1/admin/quarantine`, `POST .../approve`, `POST .../reject`. - - `ContentDefenseLayer` integration in `stemedb-ingest`. - -#### 7D. Circuit Breakers βœ… - -- [x] **7D.1 Per-Agent Circuit Breakers**: Ban misbehaving agents temporarily. βœ… - - States: Closed (normal), Open (banned), HalfOpen (testing recovery). - - 5 failures β†’ Open for 30 seconds β†’ HalfOpen β†’ 1 success β†’ Closed. - - Failure types: InvalidSignature, InputValidation, PowError, QuotaExceeded, ApplicationError. - - `CircuitBreakerStore` trait + `GenericCircuitBreakerStore`. - - `CircuitBreakerLayer` middleware (outermost in stack, runs first). - - Admin API: `GET /v1/admin/circuit-breaker/{agent_id}`, `POST .../reset`, `GET .../tripped`. - - 25 unit tests covering full state machine lifecycle. - -### Phase 8: The Swarm (Production Cluster) -*Goal: Production-ready distributed deployment with chaos testing and observability.* - -#### 8A. Chaos Testing βœ… - -- [x] **8A.1 Partition Testing**: Verify convergence after network partitions. - - βœ… 5-node cluster, kill 2 nodes, verify remaining nodes converge. - - βœ… Network partition between groups, verify both sides accept writes, verify convergence after healing. - - βœ… Message reordering and duplication tests. - - βœ… Cascading failure recovery. - - βœ… SWIM suspicion handling (false positive prevention). - - βœ… Asymmetric partition testing. - - βœ… Write availability during partition. - - Crate: `stemedb-chaos` with `TestCluster`, `ChaosNode`, `NetworkController`. - -- [x] **8A.2 Jepsen-Style Consistency Testing**: Formal verification. - - βœ… CRDT eventual consistency (1000 concurrent writes across 5 nodes). - - βœ… CRDT commutativity, associativity, idempotence verification. - - βœ… Clock skew injection (+5s/-5s), verify HLC handles drift. - - βœ… HLC monotonicity under partition. - - βœ… Supersession ordering with clock skew. - - βœ… Concurrent writes to same subject under partition (both survive). - - βœ… Large Merkle diff convergence (1500 vs 500 assertions). - - Crate: `stemedb-chaos::crdt_properties` with property verification functions. - -#### 8B. Observability +### 8B. Observability - [ ] **8B.1 Distributed Metrics**: Per-node, per-range, per-agent metrics. - - `sync_lag_seconds{peer}`, `merkle_diff_size{peer}`, `convergence_latency_p99`. - - `raft_leader_changes{range}`, `raft_commit_latency{range}`. - - `assertions_total{node}`, `writes_per_second{node}`. - - Crate: `metrics` + `metrics-exporter-prometheus`. + - `sync_lag_seconds{peer}`, `merkle_diff_size{peer}`, `convergence_latency_p99` + - `assertions_total{node}`, `writes_per_second{node}` + - Crate: `metrics` + `metrics-exporter-prometheus` - [ ] **8B.2 Admin Dashboard**: Cluster health visibility. - - `GET /v1/admin/cluster` β†’ node list, range assignments, leader locations. - - `GET /v1/admin/ranges` β†’ range sizes, split/merge history. - - `POST /v1/admin/sync` β†’ force anti-entropy sync. + - `GET /v1/admin/cluster` β†’ node list, range assignments, leader locations + - `GET /v1/admin/ranges` β†’ range sizes, split/merge history + - `POST /v1/admin/sync` β†’ force anti-entropy sync -#### 8C. Production Hardening +### 8C. Production Hardening - [ ] **8C.1 Snapshot/Restore**: Fast replica bootstrap. - - Serialize full node state as snapshot. - - New nodes join by restoring snapshot + replaying recent WAL. - - Faster than full Merkle sync for new nodes. + - Serialize full node state as snapshot + - New nodes join by restoring snapshot + replaying recent WAL - [ ] **8C.2 Backpressure**: Don't overwhelm slow nodes. - - Track per-peer sync queue depth. - - Throttle gossip to slow peers. - - Alert when peer is consistently behind. + - Track per-peer sync queue depth + - Throttle gossip to slow peers - [ ] **8C.3 Geo-Distribution**: Multi-region deployment. - - Regional clusters with CRDT federation between regions. - - Lazy cross-region sync (background, low priority). - - Locality-aware reads (query nearest replica). - - Regional compliance (GDPR data residency). + - Regional clusters with CRDT federation + - Locality-aware reads -### Phase 9: The Bunker (Disaster Planning) -*Goal: Survive the worst. Backup, restore, recover from corruption, comply with regulations, and plan for unbounded growth.* +--- -> **Key Insight:** Append-only CRDTs are a double-edged sword. They provide partition tolerance and conflict-free merge, but once bad data is merged, it's everywhere forever. Phase 9 addresses the failure modes that Phases 6-8 introduce. +## Phase 9: The Bunker (Disaster Planning) -#### 9A. Backup & Cold Storage +> **Goal:** Survive the worst. Backup, restore, recover from corruption, comply with regulations. -- [ ] **9A.1 Full Cluster Backup**: Point-in-time snapshot to cold storage. - - **Problem:** 8C.1 snapshots are for node bootstrap, not disaster recovery. Need immutable backups to S3/GCS. - - **Tasks:** - - [ ] `BackupCoordinator`: elect leader, pause writes, snapshot all nodes, upload to object storage. - - [ ] Incremental backups: WAL segments since last full backup. - - [ ] Backup manifest: cluster topology, Merkle roots, HLC high-water mark. - - [ ] Retention policy: 7 daily, 4 weekly, 12 monthly. - - [ ] `POST /v1/admin/backup/trigger`, `GET /v1/admin/backup/status`. +### 9A. Backup & Cold Storage -- [ ] **9A.2 Point-in-Time Recovery (PITR)**: Restore to any timestamp. - - **Problem:** "Restore yesterday's backup" isn't enough. Need "restore to 3:47pm yesterday." - - **Tasks:** - - [ ] WAL archiving to object storage (continuous). - - [ ] Restore = snapshot + replay WAL until target HLC timestamp. - - [ ] `POST /v1/admin/restore?target_hlc=`. - - [ ] Validation: Merkle root matches expected state after restore. +- [ ] **9A.1 Full Cluster Backup**: Point-in-time snapshot to S3/GCS. +- [ ] **9A.2 Point-in-Time Recovery (PITR)**: Restore to any HLC timestamp. +- [ ] **9A.3 Backup Verification**: Weekly automated restore tests. -- [ ] **9A.3 Backup Verification**: Prove backups actually work. - - **Problem:** Backups that can't restore are useless. Verify automatically. - - **Tasks:** - - [ ] Weekly "fire drill": restore backup to ephemeral cluster, run integrity checks. - - [ ] Merkle root comparison: restored cluster root == source cluster root at backup time. - - [ ] Alert on verification failure. - - [ ] `GET /v1/admin/backup/verification-history`. - -#### 9B. Data Corruption & Rollback - -- [ ] **9B.1 Corruption Detection**: Catch bad data before it spreads. - - **Problem:** Malformed assertions, invalid signatures, or logical corruption can poison the cluster via CRDT merge. - - **Tasks:** - - [ ] `IngestionValidator`: deep validation before accepting gossip (beyond signature check). - - [ ] Schema validation: required fields, type constraints, value ranges. - - [ ] Semantic validation: subject/predicate format, confidence bounds, timestamp sanity. - - [x] `QuarantineStore`: βœ… Implemented in Phase 7C. Extend with new `QuarantineReason` variants. - - [ ] Metrics: `assertions_quarantined`, `assertions_rejected`. +### 9B. Data Corruption & Rollback +- [ ] **9B.1 Corruption Detection**: Deep validation before accepting gossip. - [ ] **9B.2 Assertion Tombstones**: "Delete" in an append-only world. - - **Problem:** Can't actually delete from a G-Set. Need a way to mark assertions as invalid. - - **Tasks:** - - [ ] `TombstoneAssertion`: special assertion type that marks another assertion as dead. - - [ ] Tombstones propagate via CRDT like regular assertions. - - [ ] Lenses skip tombstoned assertions during resolution. - - [ ] `POST /v1/admin/tombstone/{assertion_hash}` (admin only). - - [ ] Tombstone reasons: `Corrupted`, `Malicious`, `Legal`, `Retracted`. - -- [ ] **9B.3 Cluster Rollback**: "Undo" a time range across all nodes. - - **Problem:** If bad data got merged cluster-wide, need to roll back the entire cluster. - - **Tasks:** - - [ ] `RollbackCoordinator`: elect leader, compute affected assertions, generate tombstones. - - [ ] Input: time range (HLC from/to) or list of assertion hashes. - - [ ] Output: batch of `TombstoneAssertion` propagated cluster-wide. - - [ ] Audit log: who triggered rollback, why, what was affected. - - [ ] `POST /v1/admin/rollback?from_hlc=X&to_hlc=Y&reason=...`. - +- [ ] **9B.3 Cluster Rollback**: Batch tombstone generation for time ranges. - [ ] **9B.4 Fork Recovery**: Heal split-brain after extended partition. - - **Problem:** Two clusters evolve independently during partition. After healing, they have divergent state that technically "merges" but may have semantic conflicts. - - **Tasks:** - - [ ] `ForkDetector`: identify assertions created during partition on each side. - - [ ] `ConflictReport`: list all subject/predicate pairs with divergent winners. - - [ ] Manual resolution: admin reviews conflicts, chooses winners, tombstones losers. - - [ ] `GET /v1/admin/fork-analysis`, `POST /v1/admin/fork-resolve`. -#### 9C. Compliance & Legal +### 9C. Compliance & Legal -- [ ] **9C.1 GDPR Right to Erasure**: Handle deletion requests in append-only system. - - **Problem:** GDPR requires "right to be forgotten." Append-only means data exists forever. Legal conflict. - - **Strategy:** Cryptographic erasure β€” encrypt agent data with per-agent key, delete key to "erase." - - **Tasks:** - - [ ] Agent data encrypted with per-agent key (AES-256-GCM). - - [ ] Key stored in `AgentKeyStore` (separate from assertion data). - - [ ] "Erasure" = delete agent's key β†’ their data becomes unreadable garbage. - - [ ] Tombstones for their assertions (semantically dead). - - [ ] `DELETE /v1/agents/{agent_id}` triggers erasure workflow. - - [ ] Audit log: erasure requests, completion timestamp, affected assertion count. +- [ ] **9C.1 GDPR Right to Erasure**: Cryptographic erasure via per-agent keys. +- [ ] **9C.2 Data Retention Policies**: Per-subject/predicate retention rules. +- [ ] **9C.3 Audit Trail for Compliance**: Immutable admin action log. +- [ ] **9C.4 SOC 2 Type II Certification**: External audit and certification. + - Gap assessment and remediation + - Evidence collection automation + - Auditor engagement + - Target: Q3 2026 -- [ ] **9C.2 Data Retention Policies**: Don't keep data forever. - - **Problem:** Append-only doesn't mean keep-forever. Old data has storage cost and legal liability. - - **Tasks:** - - [ ] `RetentionPolicy`: per-subject or per-predicate retention rules. - - [ ] Default: 7 years (financial), configurable per use case. - - [ ] `RetentionWorker`: background job generates tombstones for expired assertions. - - [ ] "Archive tier": cold storage for expired-but-not-deleted assertions. - - [ ] `GET/PUT /v1/admin/retention-policies`. - -- [ ] **9C.3 Audit Trail for Compliance**: Prove what happened when. - - **Problem:** Regulators ask "who changed what when." Need immutable audit log. - - **Tasks:** - - [ ] `AuditStore`: immutable log of admin actions (separate from assertions). - - [ ] Events: backup, restore, rollback, tombstone, erasure, policy change. - - [ ] Tamper-evident: Merkle chain over audit entries. - - [ ] `GET /v1/admin/audit?from=X&to=Y`. - - [ ] Export to external SIEM (Splunk, DataDog, etc.). - -#### 9D. Storage Management +### 9D. Storage Management - [ ] **9D.1 Compaction**: Reclaim space from tombstoned data. - - **Problem:** Tombstones don't free storage. Need compaction to actually reclaim space. - - **Tasks:** - - [ ] `CompactionWorker`: background job removes tombstoned assertions from storage. - - [ ] Compaction delay: wait N days after tombstone before physical deletion. - - [ ] Update Merkle tree after compaction (tree shrinks). - - [ ] Compaction manifest: what was removed, when. - - [ ] Metrics: `storage_reclaimed_bytes`, `assertions_compacted`. - - [ ] **9D.2 Tiered Storage**: Hot/warm/cold based on access patterns. - - **Problem:** Most queries hit recent data. Old assertions waste fast storage. - - **Tasks:** - - [ ] Hot tier: NVMe (< 30 days old, frequently accessed). - - [ ] Warm tier: SSD (30-365 days, occasionally accessed). - - [ ] Cold tier: Object storage (> 365 days, rarely accessed). - - [ ] Transparent access: queries fetch from appropriate tier. - - [ ] Migration worker: move data between tiers based on age/access. - - [ ] Metrics: `tier_hot_bytes`, `tier_warm_bytes`, `tier_cold_bytes`. +- [ ] **9D.3 Storage Quotas**: Per-agent and cluster-wide limits. -- [ ] **9D.3 Storage Quotas**: Prevent runaway growth. - - **Problem:** Open agent access + append-only = potential unbounded growth. - - **Tasks:** - - [ ] Per-agent storage quota (in bytes or assertion count). - - [ ] Per-subject storage quota (prevent subject stuffing). - - [ ] Cluster-wide storage limit with alerting. - - [ ] Rejection when quota exceeded: HTTP 429 with `Retry-After`. - - [ ] `GET /v1/admin/storage/usage`, `PUT /v1/admin/storage/quotas`. - -#### 9E. Incident Response - -- [ ] **9E.1 Alerting & Escalation**: Know when things break. - - **Tasks:** - - [ ] Alert definitions: sync lag > 5min, Merkle divergence, node unreachable, storage > 80%. - - [ ] Escalation tiers: P1 (page immediately), P2 (Slack + 15min), P3 (email). - - [ ] Integration: PagerDuty, OpsGenie, Slack, email. - - [ ] Runbook links in alerts (what to do when this fires). +### 9E. Incident Response +- [ ] **9E.1 Alerting & Escalation**: PagerDuty/Slack integration. - [ ] **9E.2 Operational Runbooks**: Documented procedures for common failures. - - **Runbooks to write:** - - [ ] Node won't start (WAL corruption, disk full, config error). - - [ ] Node behind on sync (network, slow disk, backpressure). - - [ ] Cluster split-brain (partition detection, resolution). - - [ ] Restore from backup (step-by-step with validation). - - [ ] Emergency rollback (bad data merged, need to undo). - - [ ] Capacity expansion (add nodes, rebalance ranges). - - [ ] Security incident (compromised node, leaked keys). +- [ ] **9E.3 Chaos Engineering**: Monthly "game days" with controlled failures. -- [ ] **9E.3 Chaos Engineering**: Break things on purpose. - - **Problem:** Can't trust disaster recovery you've never tested. - - **Tasks:** - - [ ] Scheduled chaos: monthly "game days" with controlled failures. - - [ ] Scenarios: node death, network partition, disk corruption, clock skew. - - [ ] Automated chaos: `chaos-monkey` style random failures in staging. - - [ ] Post-mortem template and review process. +### 9F. Security Hardening -#### 9F. Security Hardening - -- [ ] **9F.1 TLS Everywhere**: Encrypt all node-to-node traffic. - - **Tasks:** - - [ ] mTLS for gRPC (SyncService, gossip, anti-entropy). - - [ ] Certificate rotation without downtime. - - [ ] CA management: internal CA or external (Vault, ACME). - - [ ] Reject unencrypted connections. - -- [ ] **9F.2 Encryption at Rest**: Protect stored data. - - **Tasks:** - - [ ] WAL encryption (AES-256-GCM). - - [ ] KV store encryption (fjall supports this). - - [ ] Key management: external KMS (AWS KMS, Vault) or local. - - [ ] Key rotation without full re-encryption. - -- [ ] **9F.3 Node Authentication**: Verify cluster membership. - - **Tasks:** - - [ ] Node identity via Ed25519 keypair. - - [ ] Cluster join requires signed invitation from existing member. - - [ ] Revocation: remove compromised node's key, propagate via gossip. - - [ ] Audit: log all join/leave/revoke events. +- [ ] **9F.1 TLS Everywhere**: mTLS for node-to-node traffic. +- [ ] **9F.2 Encryption at Rest**: WAL and KV store encryption. +- [ ] **9F.3 Node Authentication**: Ed25519 keypair identity, signed cluster join. --- -## Tracking +## Architecture Overview -### Active Tasks -* [x] **Phase 3 The Pilot**: Consumer Health vertical integration. βœ… COMPLETE -* [x] **Phase 4 The Hive**: Trust & Scale + Extension Primitives. βœ… COMPLETE -* [x] **Phase 5 The Forge**: Foundation hardening β€” replace sled, fix WAL, persist indices. βœ… COMPLETE - * [x] **5A**: Replace sled with redb/fjall (HybridStore), key layout redesign. βœ… COMPLETE - * [x] **5B**: WAL hardening β€” CRC32C, crash recovery, group commit, log rotation. βœ… COMPLETE - * [x] **5C**: Index persistence β€” vector hot/cold, visual checkpoint. βœ… COMPLETE - * [x] **5D**: Concept hierarchy β€” ConceptPath, AliasStore, scheme-based inference. βœ… COMPLETE +``` +Write Path (Spine): Read Path (Cortex): +[Agent] -> [Ingestion] [Agent] <- [Lens Engine] + | | + v | + [WAL/Fsync] [Index Lookup] + | | + v | + [KV Store] <--------------------+ +``` -### Phase 6 Progress -* [x] **6A**: CRDT Foundation β€” G-Set/G-Counter stores, HLC timestamps, Merkle tree. βœ… COMPLETE -* [x] **6B**: Two-Node Replication (PoC) β€” RPC layer, gossip, anti-entropy. βœ… COMPLETE -* [x] **6C**: Multi-Node Cluster β€” SWIM membership, range sharding, gateway. βœ… COMPLETE +## Port Scheme (181XX) -### Phase 7 Progress -* [x] **7A**: Admission Control β€” TrustTier, PowProof, AdmissionLayer, /v1/admission/status. βœ… COMPLETE -* [x] **7B**: EigenTrust β€” TrustGraphStore, DomainTrustStore, EigenTrustAuthorityLens. βœ… COMPLETE -* [x] **7C**: Content Defense β€” SimilarityIndex, ContentQualityScorer, QuarantineStore, Admin API. βœ… COMPLETE -* [x] **7D**: Circuit Breakers β€” CircuitBreakerStore, CircuitBreakerLayer middleware, Admin API. βœ… COMPLETE +| Offset | Service | Default | Env Var | +|--------|---------|---------|---------| +| +0 | HTTP API | 18180 | `STEMEDB_BIND_ADDR` | +| +1 | Cluster Gateway | 18181 | `STEMEDB_NODE_API_ADDR` | +| +2 | Cluster RPC | 18182 | `STEMEDB_NODE_RPC_ADDR` | +| +3 | SWIM Gossip | 18183 | via `SwimConfig` | +| +4 | Metrics | 18184 | (reserved) | +| +5 | Admin | 18185 | (reserved) | +| +6 | Latent Signal | 18186 | β€” | +| +7 | Community App | 18187 | β€” | +| +8 | Admin Dashboard | 18188 | β€” | -### 🎯 Consumer Health MVP Progress (Current Focus) -* [x] **Week 1**: Domain Definition Core β€” `Domain` struct, `SubjectBuilder`, pharma definition. βœ… COMPLETE -* [x] **Week 2**: FDA Extractor + Signing β€” `FdaLabelExtractor`, `MedicalClaim::to_assertion()`, exponential backoff. βœ… COMPLETE -* [x] **Week 3**: StemeDB Integration β€” `StemeClient`, `pharma-ingest` CLI, mock conflict demo. βœ… COMPLETE -* [x] **Week 4**: UAT Scenarios β€” Document acceptance criteria, validation tests. βœ… COMPLETE -* [x] **Week 5**: CLI Tool β€” `steme-pharma` CLI for ingest/query/compare. βœ… COMPLETE -* [ ] **Week 6**: Generalization β€” Factor out reusable patterns, document "Adding a Domain". +## Crates -### Next Up -* **Week 6 MVP**: Factor out reusable patterns, document "Adding a Domain" guide. -* **Phase 8B-C** (deferred): Observability, geo-distribution β€” production concerns, not MVP blockers. +| Crate | Purpose | Status | +|-------|---------|--------| +| `stemedb-core` | Assertion, LifecycleStage, MaterializedView, types, signing | βœ… | +| `stemedb-wal` | Write-ahead log with crash recovery | βœ… | +| `stemedb-storage` | KVStore, VoteStore, IndexStore, TrustRankStore, QuarantineStore | βœ… | +| `stemedb-ingest` | Ingestion pipeline, signature verification, ContentDefenseLayer | βœ… | +| `stemedb-query` | Query engine, Materializer for O(1) MV reads | βœ… | +| `stemedb-lens` | Lenses (Recency, Consensus, Authority, Skeptic, Layered, etc.) | βœ… | +| `stemedb-api` | HTTP API with axum + utoipa OpenAPI docs | βœ… | +| `stemedb-sim` | Simulation for testing the pipeline | βœ… | +| `stemedb-merkle` | BLAKE3 Merkle tree for diff detection | βœ… | +| `stemedb-rpc` | gRPC services for node-to-node communication | βœ… | +| `stemedb-sync` | Merkle sync, gossip broadcast, anti-entropy | βœ… | +| `stemedb-cluster` | Cluster membership (SWIM), sharding, gateway | βœ… | +| `stemedb-ontology` | Domain definitions (Pharma), subject builders, medical extractors | βœ… | +| `stemedb-chaos` | Chaos testing infrastructure | βœ… | +| `stemedb-dashboard` | Admin dashboard (React/Next.js) | 🎯 In Progress (scaffold complete) | -### App Layer (External) -* **Browser Extension Phase 1** (Read-Only Overlay) -> All DB dependencies complete. Extension is app layer. -* **Browser Extension Phase 2** (Active Layer / Vote-to-See) -> βœ… All blockers resolved. 7A PoW + 7B EigenTrust complete. -* **The Simulator** (Training Data Pipeline) -> App layer, consumes Episteme API. -* **The Super Curator** (Reviewer Agent swarm) -> App layer. -* **Background Gardener** (Cluster detection, signal processing) -> App layer. -* **Agent Wallet** (Key management sidecar) -> App layer. +## SDKs -### Recently Completed -* [x] **🎯 MVP Week 5**: `steme-pharma` CLI for self-serve exploration. - * Full CLI binary with 5 subcommands: `ingest`, `query`, `compare`, `explore`, `validate`. - * Query modes: `skeptic` (default), `layered` (per-tier), and lens-based (recency, consensus, etc.). - * Table and JSON output formats via `comfy-table`. - * Client extensions: `layered()`, `query()`, `list_predicates()` methods. - * Response DTOs: `LayeredResponse`, `QueryResponse`, `AssertionDto`, `TierResolutionDto`. - * Domain validation for known pharma predicates and subject patterns. - * Modular design: cli.rs, commands.rs, helpers.rs, output.rs. -* [x] **🎯 MVP Week 4**: UAT scenarios documented and verified. - * Integration test suite: `crates/stemedb-ontology/tests/consumer_health_uat.rs` - * 4 automated UAT scenarios with real Ed25519 signing - * API readiness validator: `uat/consumer-health/validate_api_readiness.sh` - * Test isolation with unique subject prefixes per run -* [x] **🎯 MVP Week 3**: StemeDB integration with conflict demo. - * `StemeClient` HTTP client for assertion submission and skeptic queries. - * `pharma-ingest` CLI binary with `--with-conflicts` for mock data. - * DTO module mirroring stemedb-api types for client-side use. - * Mock trial conflicts for HbA1c, nausea rate, and weight loss. - * Health check, batch ingestion, and instrumented logging. - * 55 unit tests passing, clippy clean. -* [x] **🎯 MVP Week 2**: FDA extractor with claim-to-assertion conversion. - * `FdaLabelExtractor` fetches from api.fda.gov with exponential backoff retry. - * `MedicalClaim::to_assertion()` with Ed25519 v2 enterprise signing. - * Shared `stemedb_core::signing::compute_content_hash_v2()` utility. - * Lifecycle rules: Regulatory sources β†’ `Approved`, others β†’ `Proposed`. - * Drug name normalization handles brand names and formulation variants. - * Integration tests with `#[ignore]` for live FDA API validation. - * 10 unit tests, 5 integration tests. -* [x] **🎯 MVP Week 1**: Domain definition core for pharma vertical. - * `Domain` struct with entity types, predicate schemas, source hierarchy. - * `SubjectBuilder` for schema-driven subject construction. - * Pharma domain definition with GLP-1 drug mappings. - * `MedicalExtractor` trait and `ExtractError` enum. -* [x] **Phase 7D Circuit Breakers** (The Shield): Per-agent misbehavior isolation. - * State machine: Closed β†’ Open (5 failures) β†’ HalfOpen (30 sec timeout) β†’ Closed (1 success). - * `CircuitBreakerStore` trait with `GenericCircuitBreakerStore` implementation. - * Failure types: InvalidSignature, InputValidation, PowError, QuotaExceeded, ApplicationError. - * `CircuitBreakerLayer` Tower middleware (outermost layer, runs first). - * Admin API: `GET /v1/admin/circuit-breaker/{agent_id}`, `POST .../reset`, `GET .../tripped`. - * 503 Service Unavailable with `X-Circuit-Breaker-State`, `Retry-After` headers. - * 25 unit tests covering full state machine lifecycle and edge cases. -* [x] **Phase 7C Content Defense** (The Shield): Spam and duplicate detection with quarantine workflow. - * `SimilarityIndex` trait with MinHash (k=128) + LSH (16 bands Γ— 8 rows) for near-duplicate detection. - * Bloom filter pre-check for O(1) "definitely not duplicate" fast path. - * `ContentQualityScorer` with Shannon entropy, length checks, structured data detection. - * `QuarantineStore` with time-ordered keys + O(1) hash index for admin lookups. - * `ContentDefenseLayer` in `stemedb-ingest` orchestrating all checks. - * Admin API: `GET /v1/admin/quarantine`, `POST .../approve`, `POST .../reject`. - * Triggers: quality < 0.4, 0.9+ Jaccard similarity, untrusted + confidence > 0.8. -* [x] **Phase 6C Multi-Node Cluster** (The Mesh): Distributed cluster infrastructure. - * `SwimMembership` with SWIM gossip protocol for node discovery and failure detection. - * `RangeRouter` with BLAKE3 + jump hash for subject-prefix range sharding. - * `Gateway` HTTP service with routing, health checks, and read-your-writes. - * 82 integration tests covering membership, sharding, availability, partition tolerance. -* [x] **Phase 7B EigenTrust** (The Shield): Sybil-resistant global trust propagation. - * `TrustGraphStore` trait with edge CRUD, seed trust management, EigenTrust computation. - * Power iteration: `T = (1-Ξ±)C^T*T + Ξ±P` with dangling node handling. - * `DomainTrustStore` for per-domain expertise tracking (medicine, finance, etc.). - * `EigenTrustAuthorityLens`: `weight = confidence Γ— eigentrust Γ— domain_factor`. - * Sybil resistance: isolated rings get near-zero trust (not connected to seeds). -* [x] **Phase 7A Admission Control** (The Shield): PoW-based spam protection for new agents. - * `TrustTier` enum with 5 tiers, quota multipliers, PoW requirements. - * `PowProof` struct with BLAKE3 verification, graduated difficulty (16β†’1β†’0 bits). - * `AdmissionStore` trait + `AdmissionLayer` middleware + `/v1/admission/status` endpoint. - * Fail-open design for availability, milestone tracking for client UX. -* [x] **Phase 5D Concept Hierarchy**: Hierarchical subjects with cross-scheme alias resolution. - * `ConceptPath` struct with scheme://segments/leaf format, backward-compatible parsing. - * `SourceScheme` enum mapping schemes to source tiers (rfcβ†’Regulatory, codeβ†’Expert, etc.). - * `AliasStore` trait with transitive resolution and cycle detection. - * API: `POST/DELETE /v1/concepts/alias`, `GET /v1/concepts/resolve|aliases|suggest|parse`. - * Battery 8 (7 tests) + Battery 9 (8 tests). -* [x] **Phase 5C Index Persistence**: Vector hot/cold tiering, visual checkpoint. -* [x] **Phase 5B WAL Hardening**: CRC32C checksums, crash recovery, group commit, log rotation. -* [x] **Gold Standard Verification** (4.7): Sybil defense via proof of knowledge. - * `GoldStandard` struct with rkyv serialization, `GoldStandardStore` trait + implementation. - * `TrustAdjustment` enum: Rewarded(+0.05), Penalized(-0.1), AlreadyVerified. - * Anti-gaming: agents can only verify each gold standard once (dedup markers). - * API: `POST/GET/DELETE /v1/admin/gold-standards`, `POST /v1/admin/verify-agent`. - * 21 trust rank tests including 4 new verification tests. -* [x] **Escalation Triggers** (4.6): Active safety system for high-conflict assertions. - * `EscalationPolicy` with configurable conflict_score thresholds and predicate matching. - * `EscalationEvent` (content-addressed BLAKE3 ID) written by Materializer when policies fire. - * `EscalationStore` at `ESC:{timestamp_nanos}:{id_hex}` with resolve workflow. - * `GET /v1/admin/escalations`, `POST /v1/admin/escalations/{id}/resolve`. -* [x] **Conflict Score Filtering** (4.5): Query-time filtering by conflict score. - * `min_conflict_score` and `max_conflict_score` on Query + QueryParams. - * Fast-path MV filtering with 0.0-1.0 validation. - * 13 engine tests covering thresholds, boundaries, ranges, combinations. -* [x] **Vote Provenance Witness** (4.4): Transform votes into cryptographic witnesses. - * `source_url: Option` and `observed_context: Option>` on Vote. - * Input validation: URL max 2048 chars, context max 64KB. - * Backward compatible with existing votes. -* [x] **"Since" Parameter** (4.1): Change tracking for returning consumers. - * `since: Option` on Query for incremental sync. - * MV Changelog at `MVC:{subject}:{predicate}:{timestamp_nanos}` with nanosecond precision. - * `ChangeEntry` struct with `previous_winner_hash`, `new_winner_hash`, `lens_name`. - * `fetch_changes_since()` in QueryEngine with timestamp filtering. - * `changes_since: Option>` on QueryResponse. - * 6 tests covering changelog creation, fetching, and fast path bypass. -* [x] **Batch TrustRank Decay API** (4.3): Admin endpoint for scheduled trust decay. - * `POST /v1/admin/decay-trust-ranks` with optional `now` and `half_life_seconds`. - * Returns `decayed_count`, `timestamp_used`, `half_life_used`, `status`. - * 3 integration tests. +| SDK | Purpose | Status | +|-----|---------|--------| +| `sdk/go/steme` | Go HTTP client with Ed25519 signing and fluent builders | βœ… | +| `sdk/go/adk` | ADK-Go tools and callbacks for AI agents | βœ… | -### Recently Completed -* [x] **Source Metadata Indexing** (4.2): Index key source metadata fields for filtered queries. - * New module: `crates/stemedb-storage/src/source_metadata_index.rs`. - * Indexed fields: `journal`, `doi`, `platform`, `study_design`. - * Key pattern: `SMV:{field}:{value}` -> `Vec` (case-insensitive). - * Query filters: `?source_journal=NEJM&source_platform=PubMed`. - * AND semantics for multiple metadata filters. - * 18 unit tests covering indexing and query filtering. -* [x] **Source Document Storage** (3F.1): Provenance lookup for 100% citation recall. - * `POST /v1/source` stores source documents by BLAKE3 content hash. - * `GET /v1/provenance/{hash}` retrieves source documents. - * Content-addressed storage at `SRC:{hash}` keys. - * Base64 encoding, 10MB limit, idempotent uploads. - * 5 unit tests covering happy path and error cases. -* [x] **Epoch Cascade Logic** (3D.1): O(1) supersession lookup via pre-computed markers. - * `write_supersession_cascade()` writes `SUPERSEDED:` markers for full transitive closure at ingest time. - * `is_epoch_superseded()` uses O(1) marker lookup instead of chain walking. - * Cycle detection and max depth guard (100 levels). - * 5 tests covering markers, transitive closure, cycles, and marker-based filtering. -* [x] **Semantic Decay** (3B.2): Confidence half-life at query time. - * `decay_halflife: Option` and `source_class_decay: bool` on Query. - * New `decay.rs` module with `apply_decay()` and `apply_source_class_decay()`. - * Formula: `effective_confidence = confidence * 2^(-(age / halflife))`. - * Tier-specific decay: Regulatory=none, Clinical=2yr, Anecdotal=30d. - * 11 unit tests + 1 E2E integration test. -* [x] **Layered Consensus Lens** (3C.2): Per-source-class consensus with tier-by-tier visibility. - * `LayeredConsensusLens` with `LayeredLens` trait. - * `TierResolution` and `LayeredResolution` types. - * `GET /v1/layered?subject=X&predicate=Y` endpoint. - * Cross-tier conflict score using Shannon entropy. - * 10 comprehensive tests. -* [x] **Time-Travel Engine** (3B.1): `as_of` parameter for historical queries. - * `as_of: Option` field on `Query` for querying historical state. - * Bypasses fast path (MVs reflect current state). - * `Query::matches()` filters by `assertion.timestamp <= as_of`. - * 6 tests covering edge cases. -* [x] **Rich Source Metadata** (3A.3): Structured provenance beyond `source_hash`. - * `source_metadata: Option>` field on `Assertion`. - * `Vec` for rkyv zero-copy compatibility, callers handle JSON encoding. - * Builder methods: `.source_metadata_json()` and `.source_metadata()`. - * API exposes as `Option` with defensive UTF-8 handling. - * 2 serialization tests. -* [x] **Conflict Score on Resolution** (3A.2): Numeric disagreement metric across all lenses. - * `conflict_score: f32` field on `Resolution` (0.0 = unanimous, 1.0 = max conflict). - * `compute_conflict_score()` utility using normalized variance. - * Updated all 5 lens implementations to compute and propagate conflict score. - * `MaterializedView` now stores conflict score. - * API `QueryResponse` exposes `conflict_score` and `resolution_confidence` when lens is applied. - * 7 unit tests including NaN handling. -* [x] **SkepticLens + SkepticView** (3C.1): "Trust but Verify" conflict analysis that surfaces all claims with conflict scores. - * `AnalysisLens` trait for lenses that map conflict instead of resolving it. - * `SkepticLens` using normalized Shannon entropy for conflict scoring. - * `SkepticResolver` + `SkepticView` in stemedb-query. - * `GET /v1/skeptic?subject=X&predicate=Y` API endpoint. - * Types: `ResolutionStatus`, `ConflictAnalysis`, `ClaimSummary`, `SourceSummary`, `AgentSummary`. -* [x] **Source-Class Field** (3A.1): 6-tier `SourceClass` enum with authority weighting and decay rates. - * `SourceClass` enum: Regulatory, Clinical, Observational, Expert, Community, Anecdotal. - * `tier()`, `default_decay_days()`, `authority_weight()` methods. - * Field on Assertion struct with full serialization support. -* [x] **The Meter**: Token bucket quota system with MeterLayer middleware (10K tokens/agent/hour). -* [x] **Query Audit Trail**: Every query logged with provenance at `AUD:{query_id}`. `X-Agent-Id` header for attribution. -* [x] **Event-Driven Materialization**: `run_notified()` + IngestWorker Notify integration. -* [x] **Fast-Path MV Lookup**: `QueryEngine::try_fast_path()` for O(1) reads. -* [x] **Materializer**: Background worker for O(1) MV reads via `AsyncLens`. -* [x] **VoteAwareConsensusLens**: Real vote-based consensus resolution. -* [x] **Compound SP Index**: O(1) subject+predicate lookups. -* [x] **TrustRank System**: Agent reputation with decay and learning loop. -* [x] **API Surface**: axum HTTP server with 7 endpoints + OpenAPI docs. +## Specialized Agents -### Research Documents -* [docs/research/wal-crash-recovery-research.md](docs/research/wal-crash-recovery-research.md) β€” WAL patterns from CockroachDB, TiKV, FoundationDB, SQLite. -* [docs/research/distributed-write-path.md](docs/research/distributed-write-path.md) β€” Spanner/CockroachDB-style distributed writes adapted for append-only model. - -### Key Architectural Decisions -* **sled β†’ redb/fjall**: sled is abandoned. HybridStore routes by key prefix: redb for reads, fjall for writes. βœ… COMPLETE -* **Raft log = WAL**: TiKV eliminated duplicate WAL in v5.4. We should too. -* **CRDT for data, Raft for coordination**: Assertions are a G-Set CRDT (merge = set union). Only cluster metadata needs Raft. -* **Subject-prefix ranges**: Co-locate all data for a subject on one shard. Split hot subjects via range split. -* **HLC over TrueTime**: Hybrid Logical Clocks work on commodity hardware. No GPS/atomic clocks needed. -* **AP model**: Writes never blocked during partitions. Eventual consistency via CRDT convergence. - -### Blockers -* **Phase 5**: βœ… COMPLETE β€” All foundation hardening done. -* **Phase 6**: βœ… COMPLETE β€” CRDT foundation, two-node replication, multi-node cluster. -* **Phase 7**: βœ… COMPLETE β€” The Shield (Admission control, EigenTrust, Content Defense, Circuit Breakers). -* **Phase 8**: Unblocked. Can proceed with chaos testing, observability, geo-distribution. -* **Phase 9**: Partially blocked. 9A-9B need Phase 8 (can't backup what doesn't exist). 9C-9F can start earlier (compliance planning, security design). +| Domain | Agent | When to use | +|--------|-------|-------------| +| **Product Vision** | `episteme-product-visionary` | Use cases, "why not Postgres?", product-market fit | +| **Pilot Prep** | `enterprise-skeptic-buyer` | Pressure-test demos, find gaps, prepare for tough questions | +| General Rust | `primary-developer` | Feature implementation, refactoring | +| Code Quality | `rust-quality-engineer` | Reviews, test coverage, clippy | +| Storage | `storage-engine-architect` | WAL, LSM, crash recovery | +| Graph Engine | `rust-graph-engine-architect` | Lock-free structures, cache optimization | +| Defensive | `defensive-systems-architect` | Rate limiting, circuit breakers, hostile input | +| Distributed | `distributed-systems-engineer` | CRDT replication, Raft coordination, Merkle sync | +| Lenses | `stemedb-lens-architect` | Query resolution, ranking algorithms | +| Planning | `stemedb-planner` | Milestone planning, roadmap | --- -## Dependency Graph +## Quick Reference -``` -Phase 2.5 (Hardening) Phase 3 (The Pilot) Phase 4 (The Hive) -======================== ======================== ================== +```bash +# Build +cargo build --workspace -[2.1 MV Staleness] ---------> [3B.1 Time-Travel] βœ… --+ - | | -[2.2 Confidence Rename] -----> (API clarity for all) +----------> [4.1 "Since" Param] βœ… - | -[2.3 EpochAwareLens] --------> [3D.1 Epoch Cascade] βœ… |----------> Invalidation Cascades pillar - | -[2.4 Visual Hash Query] -----> [3E.2 Visual Hash Index] βœ… - | -[2.6 E2E Integration] -------> (pipeline confidence) | - | - [3A.1 Source-Class] βœ… --+----------> [3C.2 Layered Consensus] βœ… - | [3B.2 Semantic Decay] βœ… - +-----------------------------[4.2 Metadata Indexing] βœ… - | - [3A.2 Conflict Score] βœ… --> (enhance Resolution) - | - [3A.3 Source Metadata] βœ… --> [4.2 Metadata Indexing] βœ… - | - [3C.1 Skeptic Lens] βœ… (standalone, COMPLETE) - [3C.3 Constraints Lens] βœ… (standalone, COMPLETE) - [3E.1 Vector Search] βœ… (standalone, COMPLETE) - [3E.2 Visual Hash Index] βœ… (standalone, COMPLETE) - [3F.1 Provenance] βœ… (standalone, COMPLETE) +# Test +cargo test --workspace + +# Lint (must pass before commit) +cargo clippy --workspace -- -D warnings +cargo fmt --check + +# Run API server +cargo run --bin stemedb-api + +# Run demo script +./scripts/demo-consumer-health.sh ``` -### 🎯 Critical Path for Consumer Health MVP (Current Focus) +--- -``` -INFRASTRUCTURE (Complete) VERTICAL INTEGRATION (In Progress) -========================= =================================== +## Related Documents -[3A.1 Source-Class] βœ… ──────────────────┐ -[3A.2 Conflict Score] βœ… ───────────────── -[3C.2 Layered Consensus] βœ… ────────────── -[3B.1 Time-Travel] βœ… ───────────────────┼───> [MVP Week 3: Ingest + Conflicts] βœ… -[3A.3 Source Metadata] βœ… ──────────────── | -[3C.1 Skeptic Lens] βœ… ─────────────────── v -[3B.2 Semantic Decay] βœ… β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ [MVP Week 4: UAT Scenarios] βœ… - | -[stemedb-ontology Weeks 1-3] βœ… β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ - (Domain defs, FDA extractor, StemeClient) | - v - [MVP Week 5: CLI Tool] βœ… - | - v - [MVP Week 6: Polish & Docs] [ ] - | - v - 🎯 CONSUMER HEALTH MVP - "5-minute demo that convinces" -``` - -### Critical Path for Financial DD Demo - -``` -[3A.2 Conflict Score] βœ… --> [3C.1 Skeptic Lens] βœ… ---+ - | -[3B.1 Time-Travel] βœ… ----------------------------------+----> FINANCIAL DD MVP - | -[2.3 EpochAwareLens] --> [3D.1 Epoch Cascade] βœ… -------+ - | -[3B.2 Semantic Decay] βœ… -------------------------------+ -``` - -### Critical Path for Agile Agent Team Demo - -``` -[3C.3 Constraints Lens] (standalone) ------+ - | -[3B.1 Time-Travel] βœ… --------------------+----> AGENT TEAM MVP - | -[2.3 EpochAwareLens] βœ… ------------------+ - | -[Query Audit (Phase 2)] βœ… ----------------+ -``` - -### Critical Path for Browser Extension - -``` -Phase 3 (Data Foundation) Phase 4 (Extension Primitives) Extension Product -========================= ============================== ================= - -[3A.1 Source-Class] βœ… ──────────> [4.5 Conflict Filtering] βœ… ──┐ - | -[3A.2 Conflict Score] βœ… ────────────────────────────────────────── - | -[3C.1 Skeptic Lens] βœ… ──────────────────────────────────────────-─ - β”œβ”€β”€> PHASE 1: Read-Only Overlay -[3B.2 Semantic Decay] βœ… ────────────────────────────────────────-─ (Benign Layer) - | -[3C.2 Layered Consensus] βœ… ─────────────────────────────────────-β”˜ - -[Phase 2 Vote System] βœ… ────────> [4.4 Vote Provenance] βœ… ─────┐ - | -[TrustRank Engine] βœ… ───────────> [4.7 Gold Standards] βœ… ──────── - β”œβ”€β”€> PHASE 2: Active Layer -[3A.2 Conflict Score] βœ… ────────> [4.6 Escalation Triggers] βœ… ─── (Vote to See) - | - [7A PoW Admission] βœ… ──────────── - [7B EigenTrust] βœ… β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ -``` - -**Phase 1 (Read-Only)** requires: Source tiers, conflict scores, conflict filtering, skeptic lens, decay, layered consensus. **All complete.** - -**Phase 2 (Active)** requires: Vote provenance, gold standards, escalation triggers, PLUS Phase 7 Sybil defense. **βœ… All complete. Ready to build.** - -### Critical Path to Distributed Cluster - -``` -Phase 5 (The Forge) βœ… Phase 6 (The Mesh) βœ… Phase 7+8 -======================= ======================= ================== - -[5A.1 Replace sled βœ…] ───────────> [6A.1 CRDT Foundation βœ…] ──┐ - | | -[5A.2 Key Layout βœ…] ────────────> [6C.2 Range Sharding βœ…] ───> | - | -[5B.1 CRC32C Checksums βœ…] ──┐ | -[5B.2 Crash Recovery βœ…] ────┼───> [6B.1 RPC Layer βœ…] ────────── -[5B.3 Group Commit βœ…] β”€β”€β”€β”€β”€β”€β”˜ | | - v | -[5C.1 Persistent Vector βœ…] ─── (independent, no blocker) | -[5C.2 Persistent Visual βœ…] ─── (independent, no blocker) | - | - [6A.2 HLC Timestamps βœ…] ───── - [6A.3 Merkle Tree βœ…] ──────── - | | - v v - [6B.2 Gossip βœ…] ──> [6B.3 Anti-Entropy βœ…] ──> [6B.4 PoC Tests βœ…] - | - v - [6C.1 SWIM Membership βœ…] ───> [6C.3 Raft MV Coord] (DEFERRED) - [6C.4 Gateway βœ…] ───────────> β”‚ - v - DISTRIBUTED CLUSTER βœ… - | - [7A PoW Admission βœ…] ┐ - [7B EigenTrust βœ…] ─────> THE SHIELD βœ… - [7C Content Defense βœ…]─ - [7D Circuit Breakers βœ…]β”˜ - | - [8A Chaos Testing] ──┐ - [8B Observability] ─────> THE SWARM - [8C Geo-Distribution]β”˜ - | - [9A Backup/PITR] ─────┐ - [9B Corruption/Rollback]─ - [9C GDPR/Retention] ─────> THE BUNKER - [9D Storage Mgmt] ───── - [9E Incident Response]─ - [9F Security Hardening]β”˜ -``` - -### New Crates (Phases 5-9) - -``` -stemedb-merkle (Phase 6A) ── BLAKE3 Merkle tree for diff detection βœ… IMPLEMENTED -stemedb-rpc (Phase 6B) ── gRPC services for node-to-node communication βœ… IMPLEMENTED -stemedb-sync (Phase 6B) ── Merkle sync, gossip broadcast, anti-entropy βœ… IMPLEMENTED -stemedb-cluster (Phase 6C) ── Cluster membership, range routing, gateway βœ… IMPLEMENTED -stemedb-ontology (Parallel) ── Domain definitions, subject builders, medical extractors βœ… IMPLEMENTED -stemedb-backup (Phase 9A) ── Backup coordination, PITR, verification (PLANNED) -stemedb-admin (Phase 9B) ── Tombstones, rollback, fork recovery, compliance (PLANNED) -``` +- [CLAUDE.md](./CLAUDE.md) β€” AI assistant instructions and project rules +- [roadmap-archive.md](./roadmap-archive.md) β€” Completed phases 1-7 detail +- [docs/demo/pilot/amazement-demo.md](./docs/demo/pilot/amazement-demo.md) β€” Technical demo script +- [docs/demo/pilot/amazement-demo-2.md](./docs/demo/pilot/amazement-demo-2.md) β€” Executive demo script +- [uat/production-readiness/README.md](./uat/production-readiness/README.md) β€” Production verification checklist +- [.claude/agents/enterprise-skeptic-buyer.md](./.claude/agents/enterprise-skeptic-buyer.md) β€” Dr. Sarah Chen persona