## Phase 8: Enterprise Extractor Improvements ✅ - 14 security extractors (TLS, JWT, SQL injection, XSS, etc.) - 10 framework-specific extractors (Spring, Django, Rails, etc.) - Config file security detection (YAML, TOML) ## Phase 9: Autonomous Extractor Generation ✅ - Shadow mode executor with TP/FP tracking - Graduation pipeline with confidence thresholds - Auto-rollback on regression detection - Cross-project pattern syncing ## UAT Suite Complete (14 scripts, 90 tests) - test-core-detection.sh (6 tests) - test-declarative-extractors.sh (5 tests) - test-domain-frameworks.sh (5 tests) - test-domain-unreal.sh (3 tests) - test-llm-extraction.sh (6 tests) - test-eval-harness.sh (5 tests) - test-cross-language.sh (3 tests) - test-precommit-performance.sh (4 tests) - test-output-formats.sh (8 tests) - test-drift-detection.sh (6 tests) - test-exit-codes.sh (12 tests) + 3 more scripts ## Other Changes - Updated roadmap to mark Phase 8-9 complete - Added .gitignore entries for build artifacts - Updated pre-commit: 800 line limit, exclude tests/data/cmd Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
485 lines
24 KiB
Markdown
485 lines
24 KiB
Markdown
# Episteme (StemeDB) Roadmap
|
|
|
|
> **Goal:** Build the "Git for Truth" substrate for autonomous AI research.
|
|
> **Current Focus:** Enterprise Pilot Preparation
|
|
> **Target Vertical:** BioTech/Pharma ("The Living Review")
|
|
> **Endgame:** Distributed multi-writer cluster for millions of concurrent agents
|
|
>
|
|
> **Infrastructure Status:** Phases 1-7 complete ✅ | Phase 8A (Chaos) complete ✅
|
|
> **Pilot Status:** Consumer Health MVP complete ✅ | Enterprise Demo in progress
|
|
>
|
|
> **Archive:** For completed phases 1-7, see [roadmap-archive.md](./roadmap-archive.md)
|
|
|
|
---
|
|
|
|
## Current Status
|
|
|
|
| Phase | Status | Summary |
|
|
|-------|--------|---------|
|
|
| **1-7** | ✅ Complete | Core infrastructure, distributed cluster, trust & safety |
|
|
| **8A** | ✅ Complete | Chaos testing, Jepsen-style verification |
|
|
| **MVP** | ✅ Complete | Consumer Health demo with real FDA data |
|
|
| **Pilot Prep** | 🎯 In Progress | Dashboard, impact analysis, production hardening |
|
|
| **8B-C** | Planned | Observability, geo-distribution |
|
|
| **9** | Planned | Disaster recovery, compliance, storage management |
|
|
|
|
---
|
|
|
|
## 🎯 Phase: Enterprise Pilot Preparation (CURRENT)
|
|
|
|
> **Goal:** Make the pilot bulletproof. Amaze enterprise decision makers.
|
|
> **Timeline:** 5 weeks
|
|
> **Success Criteria:** Dr. Sarah Chen (skeptical VP of Data Infrastructure) fights her CFO for budget
|
|
|
|
### The 5 Amazement Moments We Must Deliver
|
|
|
|
| # | Moment | Current State | Gap |
|
|
|---|--------|---------------|-----|
|
|
| 1 | Contradictions visible with confidence scores | ✅ Complete | Dashboard scaffold + Skeptic Query UI ✅ |
|
|
| 2 | Cascade invalidation when source retracted | ✅ Complete | Full UI: Sources page + impact dialog (P3.1-3.3) ✅ |
|
|
| 3 | Full FDA-ready audit trail | ✅ Complete | Audit Trail Browser (P1.6) ✅ |
|
|
| 4 | Point-in-time queries + decay | ✅ API ready | No timeline UI |
|
|
| 5 | Malicious agent blocked by circuit breaker | ✅ Complete | Circuit Breaker Status (P1.5) ✅ |
|
|
|
|
### Pilot-1: Demo Dashboard (Week 1-2)
|
|
|
|
> **Deliverable:** React admin dashboard that makes the API visual
|
|
|
|
- [x] **P1.1 Dashboard Scaffold**: Next.js + shadcn/ui project setup ✅
|
|
- [x] Project structure: `applications/stemedb-dashboard/`
|
|
- [x] API client for StemeDB endpoints (`src/lib/api/client.ts`)
|
|
- [x] Authentication scaffold (API key header)
|
|
- [x] Dark mode (default), responsive layout with collapsible sidebar
|
|
- [x] shadcn/ui components: button, card, badge, input, separator, tabs
|
|
- [x] Live API status indicator (polls /health every 30s)
|
|
- [x] Port 18188, builds and runs successfully
|
|
|
|
- [x] **P1.2 Skeptic Query Visualization**: Show contradictions graphically ✅
|
|
- [x] Query builder: subject, predicate inputs
|
|
- [x] Conflict score gauge (0.0-1.0 with color coding)
|
|
- [x] Claims table with weight bars, source tier badges
|
|
- [x] "CONTESTED" / "AGREED" / "UNANIMOUS" status badges
|
|
- [x] Expandable claim rows with source details, agents, provenance hashes
|
|
- [x] Loading skeleton, empty state, error state with retry
|
|
|
|
- [x] **P1.3 Layered Consensus View**: Per-tier breakdown ✅
|
|
- [x] Tier accordion showing each source class (T0→T5, empty tiers hidden)
|
|
- [x] Within-tier conflict score (compact gauge in accordion header)
|
|
- [x] Cross-tier conflict visualization (full gauge with stats)
|
|
- [x] Extended ConflictGauge with variant prop for reuse
|
|
|
|
- [x] **P1.4 Quarantine Admin Panel**: Content defense visibility ✅
|
|
- [x] Pending queue with reason, timestamp, quality score
|
|
- `quarantine-panel.tsx`, `quarantine-list.tsx`, `quarantine-row.tsx`
|
|
- [x] Approve/Reject buttons with confirmation
|
|
- `ConfirmationDialog` with restore/delete actions
|
|
- [x] Filter by reason (duplicate, spam, untrusted high-confidence)
|
|
- `quarantine-filters.tsx` with dropdown selector
|
|
- [x] Metrics: pending count, approved/rejected today
|
|
- `quarantine-metrics.tsx` with MetricCard grid
|
|
|
|
- [x] **P1.5 Circuit Breaker Status**: Trust & safety dashboard ✅
|
|
- [x] Blocked agents list with failure count, retry time
|
|
- `circuit-list.tsx`, `circuit-card.tsx` with full details
|
|
- [x] State badges: OPEN (red), HALF_OPEN (yellow), CLOSED (green)
|
|
- `state-badge.tsx` with color-coded variants
|
|
- [x] Manual reset button for admin override
|
|
- `circuit-panel.tsx` - `handleReset` calls API
|
|
- [x] Summary with state counts
|
|
- `circuit-summary.tsx` replaces historical events (more useful)
|
|
- [x] Auto-refresh every 10 seconds
|
|
|
|
- [x] **P1.6 Audit Trail Browser**: Query provenance explorer ✅
|
|
- [x] Recent queries list with agent, timestamp, subject
|
|
- `audit-list.tsx`, `audit-row.tsx` with pagination
|
|
- [x] Drilldown: contributing assertions, weights, winner
|
|
- Expandable row details in `audit-row.tsx`
|
|
- [x] Filter by agent, time range, subject
|
|
- `audit-filters.tsx` with 1h/24h/7d/30d/all options
|
|
- [x] Export to JSON/CSV
|
|
- `audit-export.tsx` with proper escaping
|
|
|
|
### Pilot-2: Demo Data Seeder (Week 2)
|
|
|
|
> **Deliverable:** Pre-signed realistic demo data using Go SDK
|
|
> **Status:** All complete ✅
|
|
|
|
- [x] **P2.1 Demo Keypair Management**: Reproducible demo keys ✅
|
|
- [x] 5 demo agents with realistic naming convention:
|
|
- `fda:drug-label-ingestor` (Tier 0 - Regulatory)
|
|
- `pubmed:abstract-indexer` (Tier 1 - Clinical)
|
|
- `clinicaltrials:study-importer` (Tier 1 - Clinical)
|
|
- `reddit:health-discussion-scraper` (Tier 5 - Anecdotal)
|
|
- `internal:clinical-ops-reviewer` (Tier 3 - Expert)
|
|
- [x] Keys stored in `demo/keys/` with README documenting each agent's role/scope
|
|
- `demo/keys/agents.json` with seeds, public keys, tiers, descriptions
|
|
- `demo/keys/README.md` with full documentation
|
|
- `demo/keys/keygen.go` for deterministic regeneration
|
|
- [x] Go SDK script: `cmd/demo-seed/main.go`
|
|
- Loads keys from `agents.json`
|
|
- Creates 260+ assertions with realistic data
|
|
- [x] One-command setup: `./scripts/run-demo.sh` (start DB → seed → open dashboard)
|
|
- Build detection, health check, auto-cleanup on exit
|
|
- `--clean` flag for fresh start, `--no-open` to skip browser
|
|
|
|
- [x] **P2.2 Conflict Scenarios**: Pre-built disagreements with real data ✅
|
|
- [x] 3 drugs: semaglutide (45), tirzepatide (38), liraglutide (32) assertions
|
|
- [x] 150+ assertions total using real FDA label excerpts
|
|
- [x] ClinicalTrials.gov summaries (STEP, SURMOUNT, SELECT, LEADER trials)
|
|
- [x] **Killer conflicts**: Weight loss (FDA 14.9% vs STEP UP 20.7% vs Reddit variable), Gastroparesis (FDA 0.2% vs UBC 3x risk)
|
|
- [x] 4 genuine conflicts per drug (weight loss, nausea, gastroparesis, CV benefit)
|
|
- [x] Source registry with 30+ deterministic hash sources across T0-T5 tiers
|
|
|
|
- [x] **P2.3 Retractable Sources**: Set up cascade demo ✅
|
|
- [x] New `CARDIOVASC_MEGA_TRIAL` source in `sources.go` (landmark multi-drug CV outcomes study)
|
|
- [x] 110 assertions citing this source across 8 categories (visceral cascade effect)
|
|
- Primary/Secondary CV Outcomes (30), Biomarkers (15), Subgroup Analyses (20)
|
|
- Expert Guidelines (15), Real-World Evidence (15), Comparative Efficacy (10), Community (5)
|
|
- [x] 5 agents represented: T0 (FDA), T1 (ClinicalTrials), T2 (PubMed), T3 (Internal), T5 (Reddit)
|
|
- [x] `printCascadeDemoCommands()` outputs curl commands for demo flow
|
|
- [x] Demo documentation updated in `amazement-demo.md`
|
|
- **Note:** API endpoints (P3.1) complete ✅, live demo ready
|
|
|
|
- [x] **P2.4 Historical Data**: Time-travel via lifecycle evolution ✅
|
|
- [x] **Approach:** Use lifecycle states (Proposed → Approved → Deprecated), not fake timestamps
|
|
- [x] Each lifecycle transition auditable with real timestamps (signature timestamps)
|
|
- [x] Demo scenario: Wegovy CV indication change (pre-March 2024 vs post-SELECT)
|
|
- [x] 8 historical scenarios: CV indication, SELECT trial evolution, ADA guidelines, Tirzepatide expansion
|
|
- [x] 17 historical assertions showing lifecycle progression
|
|
- [x] Demo commands for `as_of` queries
|
|
|
|
### Pilot-3: Impact Analysis (Week 3)
|
|
|
|
> **Deliverable:** Automatic cascade when source is retracted
|
|
> **Critical:** This unblocks P2.3 (retractable sources demo data)
|
|
|
|
- [x] **P3.1 Impact Analysis Endpoint**: `GET /v1/sources/{hash}/impact` ✅
|
|
- [x] Returns all assertions citing this source (verified: 110 assertions for CARDIOVASC_MEGA_TRIAL)
|
|
- [x] Returns count of queries that used those assertions
|
|
- [x] Returns list of affected agents/recommendations (verified: 4 agents)
|
|
- [x] Implementation in `stemedb-api/src/handlers/source_registry/handlers.rs:237-439`
|
|
- [x] `POST /v1/sources/{hash}/quarantine` with preview mode (preview=true shows impact without changes)
|
|
- [x] Preview response: "This will affect X assertions and Y agent recommendations"
|
|
- [x] Undo capability: `POST /v1/sources/{hash}/restore` (verified: restores 110 assertions)
|
|
- [x] 17 unit/integration tests passing
|
|
|
|
- [x] **P3.2 Cascade Flagging**: Automatic downstream impact ✅
|
|
- [x] When source status → quarantined, flag citing assertions
|
|
- Implemented query-time lookup (not index mutation) to preserve append-only immutability
|
|
- `SourceStatusEnricher` service batch-lookups source statuses from SourceRegistry
|
|
- `SourceWarningDto` attached to assertions with `warning_type`, `message`, `source_label`, `status_updated_at`
|
|
- [x] ~~New field on assertion index~~ → Query-time enrichment instead (preserves immutability)
|
|
- [x] Queries can filter by `exclude_quarantined_sources=true`
|
|
- Added to `QueryParams` in `dto/query_params.rs`
|
|
- POST-retrieval filter applied after query execution
|
|
- [x] Define query behavior: quarantined sources show with warning (not silently omitted)
|
|
- `source_warning` field added to `AssertionResponse` and `ClaimSummaryDto`
|
|
- Skeptic endpoint enriches claims with warnings
|
|
- [x] Export affected items list for regulatory documentation (CSV/JSON)
|
|
- `GET /v1/sources/{hash}/impact/export?format=csv|json`
|
|
- Returns `ImpactExportRow` with assertion_hash, subject, predicate, agent_id, timestamp, lifecycle, confidence
|
|
- CSV includes proper escaping, JSON returns array of objects
|
|
|
|
- [x] **P3.3 Impact Dashboard Widget**: Visualize the cascade ✅
|
|
- [x] Source status change UI (Active → Quarantined)
|
|
- `components/sources/status-badge.tsx` with color-coded badges
|
|
- `components/sources/tier-badge.tsx` with T0-T5 labels
|
|
- [x] Confirmation dialog: "This will affect 234 downstream assertions and 12 recommendations"
|
|
- `components/sources/quarantine-dialog.tsx` with impact preview
|
|
- Warning box shows exact affected counts from API
|
|
- [x] Choice: "Quarantine immediately" or "Review affected items first"
|
|
- Dual action buttons in dialog
|
|
- "Review first" opens `ImpactDetailPanel` with full assertion list
|
|
- [x] Animated "impact ripple" showing affected count
|
|
- `components/sources/impact-ripple.tsx` with Tailwind `animate-ping`
|
|
- Triggers on dialog open, counts pulse with amber styling
|
|
- [x] List of impacted queries with timestamp
|
|
- `components/sources/impact-detail-panel.tsx` shows affected assertions table
|
|
- Affected agents shown as chips
|
|
- [x] "Remediation status" tracking
|
|
- Source status visible in list, metrics show quarantined count
|
|
- `components/sources/sources-metrics.tsx` with Active/Deprecated/Quarantined counts
|
|
- [x] Audit trail: WHO retracted, WHEN, and WHY
|
|
- `RestoreDialog` and `QuarantineDialog` capture reason field
|
|
- Export to CSV/JSON for regulatory documentation
|
|
|
|
### Pilot-4: Production Hardening (Week 4)
|
|
|
|
> **Deliverable:** Load testing, authentication, backup documentation
|
|
|
|
- [x] **P4.1 Load Testing**: Prove performance claims ✅
|
|
- [x] Go-based load tester with native Ed25519 signing (`cmd/load-test/`)
|
|
- [x] Benchmark: 10K assertions baseline latency (p99 < 200ms target)
|
|
- [x] Benchmark: 1K writes/sec sustained for configurable duration
|
|
- [x] Benchmark: 100 concurrent readers, <2x degradation target
|
|
- [x] Markdown report generator with pass/fail status
|
|
- [x] One-command runner: `./scripts/run-load-test.sh`
|
|
- [x] Results saved to `uat/production-readiness/results/`
|
|
|
|
- [ ] **P4.2 API Authentication**: Basic security for pilot
|
|
- [ ] API key middleware (`X-API-Key` header)
|
|
- [ ] Per-key rate limiting (separate from per-agent quota)
|
|
- [ ] Admin keys vs read-only keys
|
|
- [ ] Key management: `POST /v1/admin/api-keys`
|
|
|
|
- [ ] **P4.3 Backup/Restore Documentation**: DR story
|
|
- [ ] Document WAL-based recovery procedure
|
|
- [ ] Script: `scripts/backup-stemedb.sh` (snapshot + WAL archive)
|
|
- [ ] Script: `scripts/restore-stemedb.sh` (restore from backup)
|
|
- [ ] Test restore procedure, document in UAT
|
|
|
|
- [ ] **P4.4 Prometheus Metrics**: Observability baseline
|
|
- [ ] `GET /metrics` endpoint with prometheus format
|
|
- [ ] Key metrics: `assertions_total`, `queries_total`, `query_latency_seconds`
|
|
- [ ] Trust metrics: `quarantine_pending`, `circuit_breakers_open`
|
|
- [ ] Basic Grafana dashboard template
|
|
|
|
### Pilot-5: Operational Readiness (Week 5)
|
|
|
|
> **Deliverable:** Runbooks, monitoring, reference architecture
|
|
|
|
- [ ] **P5.1 Operational Runbooks**: Common procedures documented
|
|
- [ ] "Server won't start" troubleshooting
|
|
- [ ] "High query latency" investigation
|
|
- [ ] "Quarantine queue overflow" handling
|
|
- [ ] "Circuit breaker stuck open" resolution
|
|
- [ ] "Restore from backup" step-by-step
|
|
|
|
- [ ] **P5.2 Reference Architecture**: Deployment guide
|
|
- [ ] Single-node pilot deployment diagram
|
|
- [ ] Network requirements (ports, firewall rules)
|
|
- [ ] Reverse proxy configuration (nginx/envoy with TLS)
|
|
- [ ] Resource sizing guide (CPU, memory, disk)
|
|
|
|
- [ ] **P5.3 Pilot Success Criteria Document**: Definition of done
|
|
- [ ] Sub-second query latency at 10K assertions: measured
|
|
- [ ] Successful conflict detection on known contradictory studies: demonstrated
|
|
- [ ] Complete audit trail export for mock regulatory review: tested
|
|
- [ ] Source retraction workflow: exercised
|
|
|
|
- [ ] **P5.4 Executive Demo Script Validation**: End-to-end rehearsal
|
|
- [ ] Run through `amazement-demo-2.md` with real dashboard
|
|
- [ ] Time each segment (target: 20 minutes total)
|
|
- [ ] Record demo video for async sharing (backup if live demo fails)
|
|
- [ ] All 5 Aha Moments demonstrable with real data (not mockups)
|
|
- [ ] **Enterprise Skeptic Questions** (must have documented answers):
|
|
- What's the data ingestion latency? (FDA update → queryable)
|
|
- What happens when agents disagree on interpretation?
|
|
- Can I export an audit report for regulators? (PDF/CSV)
|
|
- What's the failure mode if service goes down mid-demo?
|
|
- How do I verify demo data is representative of my real data?
|
|
- If I retract a source, what happens to queries that would have used it?
|
|
|
|
### Pilot Prep Deliverables Summary
|
|
|
|
| Week | Deliverable | Owner | Acceptance Criteria |
|
|
|------|-------------|-------|---------------------|
|
|
| 1-2 | `stemedb-dashboard` | Frontend | ✅ 6 functional panels, connects to API (P1.1-P1.6) |
|
|
| 2 | `demo-seed` (P2.1-P2.4) | SDK | ✅ 260+ assertions, 3 drugs, real FDA content, lifecycle history, cascade data |
|
|
| 3 | Impact Analysis (P3.1) | Backend | ✅ `/v1/sources/{hash}/impact` + quarantine/restore endpoints |
|
|
| 3 | Cascade Flagging (P3.2) | Backend | ✅ Source warnings, exclude filter, impact export |
|
|
| 3 | Impact Dashboard (P3.3) | Frontend | ✅ Sources page, quarantine dialog, impact ripple, export |
|
|
| 3 | `demo-seed` (P2.3) | SDK | ✅ Retractable source with 110 cascade assertions |
|
|
| 4 | Load Test Results | QA | ✅ `cmd/load-test/` + `scripts/run-load-test.sh` |
|
|
| 4 | API Authentication | Backend | API keys work, rate limiting functional |
|
|
| 4 | Backup/Restore | Ops | Documented and tested procedure |
|
|
| 4 | Metrics Endpoint | Backend | `/metrics` returns Prometheus format |
|
|
| 5 | Runbooks | Ops | 5 runbooks in `docs/runbooks/` |
|
|
| 5 | Reference Architecture | Docs | Deployment guide complete |
|
|
| 5 | Demo Rehearsal | All | 20-minute demo runs smoothly |
|
|
| 5 | One-Command Demo | Ops | ✅ `./scripts/run-demo.sh` works (P2.1) |
|
|
|
|
### Demo Data Quality Checklist (from Enterprise Skeptic Review)
|
|
|
|
- [x] Real FDA label excerpts (public domain) - not synthetic ✅
|
|
- [x] ClinicalTrials.gov summaries for plausibility ✅
|
|
- [x] Agent names map to real-world roles (`fda:drug-label-ingestor`) - P2.1 ✅
|
|
- [x] Conflicts are genuine (not "100% vs 0%" manufactured disagreements) - P2.2 ✅
|
|
- [x] Cascade demo shows 100+ affected items (visceral impact) - 110 assertions ✅
|
|
- [x] Export capability for regulatory documentation (CSV/JSON) - P3.2 ✅
|
|
- [ ] Recovery story: what happens if demo breaks mid-presentation?
|
|
|
|
---
|
|
|
|
## Phase 8B-C: Production Observability (Planned)
|
|
|
|
> **Blocked by:** Pilot Prep (need real production deployment first)
|
|
|
|
### 8B. Observability
|
|
|
|
- [ ] **8B.1 Distributed Metrics**: Per-node, per-range, per-agent metrics.
|
|
- `sync_lag_seconds{peer}`, `merkle_diff_size{peer}`, `convergence_latency_p99`
|
|
- `assertions_total{node}`, `writes_per_second{node}`
|
|
- Crate: `metrics` + `metrics-exporter-prometheus`
|
|
|
|
- [ ] **8B.2 Admin Dashboard**: Cluster health visibility.
|
|
- `GET /v1/admin/cluster` → node list, range assignments, leader locations
|
|
- `GET /v1/admin/ranges` → range sizes, split/merge history
|
|
- `POST /v1/admin/sync` → force anti-entropy sync
|
|
|
|
### 8C. Production Hardening
|
|
|
|
- [ ] **8C.1 Snapshot/Restore**: Fast replica bootstrap.
|
|
- Serialize full node state as snapshot
|
|
- New nodes join by restoring snapshot + replaying recent WAL
|
|
|
|
- [ ] **8C.2 Backpressure**: Don't overwhelm slow nodes.
|
|
- Track per-peer sync queue depth
|
|
- Throttle gossip to slow peers
|
|
|
|
- [ ] **8C.3 Geo-Distribution**: Multi-region deployment.
|
|
- Regional clusters with CRDT federation
|
|
- Locality-aware reads
|
|
|
|
---
|
|
|
|
## Phase 9: The Bunker (Disaster Planning)
|
|
|
|
> **Goal:** Survive the worst. Backup, restore, recover from corruption, comply with regulations.
|
|
|
|
### 9A. Backup & Cold Storage
|
|
|
|
- [ ] **9A.1 Full Cluster Backup**: Point-in-time snapshot to S3/GCS.
|
|
- [ ] **9A.2 Point-in-Time Recovery (PITR)**: Restore to any HLC timestamp.
|
|
- [ ] **9A.3 Backup Verification**: Weekly automated restore tests.
|
|
|
|
### 9B. Data Corruption & Rollback
|
|
|
|
- [ ] **9B.1 Corruption Detection**: Deep validation before accepting gossip.
|
|
- [ ] **9B.2 Assertion Tombstones**: "Delete" in an append-only world.
|
|
- [ ] **9B.3 Cluster Rollback**: Batch tombstone generation for time ranges.
|
|
- [ ] **9B.4 Fork Recovery**: Heal split-brain after extended partition.
|
|
|
|
### 9C. Compliance & Legal
|
|
|
|
- [ ] **9C.1 GDPR Right to Erasure**: Cryptographic erasure via per-agent keys.
|
|
- [ ] **9C.2 Data Retention Policies**: Per-subject/predicate retention rules.
|
|
- [ ] **9C.3 Audit Trail for Compliance**: Immutable admin action log.
|
|
- [ ] **9C.4 SOC 2 Type II Certification**: External audit and certification.
|
|
- Gap assessment and remediation
|
|
- Evidence collection automation
|
|
- Auditor engagement
|
|
- Target: Q3 2026
|
|
|
|
### 9D. Storage Management
|
|
|
|
- [ ] **9D.1 Compaction**: Reclaim space from tombstoned data.
|
|
- [ ] **9D.2 Tiered Storage**: Hot/warm/cold based on access patterns.
|
|
- [ ] **9D.3 Storage Quotas**: Per-agent and cluster-wide limits.
|
|
|
|
### 9E. Incident Response
|
|
|
|
- [ ] **9E.1 Alerting & Escalation**: PagerDuty/Slack integration.
|
|
- [ ] **9E.2 Operational Runbooks**: Documented procedures for common failures.
|
|
- [ ] **9E.3 Chaos Engineering**: Monthly "game days" with controlled failures.
|
|
|
|
### 9F. Security Hardening
|
|
|
|
- [ ] **9F.1 TLS Everywhere**: mTLS for node-to-node traffic.
|
|
- [ ] **9F.2 Encryption at Rest**: WAL and KV store encryption.
|
|
- [ ] **9F.3 Node Authentication**: Ed25519 keypair identity, signed cluster join.
|
|
|
|
---
|
|
|
|
## Architecture Overview
|
|
|
|
```
|
|
Write Path (Spine): Read Path (Cortex):
|
|
[Agent] -> [Ingestion] [Agent] <- [Lens Engine]
|
|
| |
|
|
v |
|
|
[WAL/Fsync] [Index Lookup]
|
|
| |
|
|
v |
|
|
[KV Store] <--------------------+
|
|
```
|
|
|
|
## Port Scheme (181XX)
|
|
|
|
| Offset | Service | Default | Env Var |
|
|
|--------|---------|---------|---------|
|
|
| +0 | HTTP API | 18180 | `STEMEDB_BIND_ADDR` |
|
|
| +1 | Cluster Gateway | 18181 | `STEMEDB_NODE_API_ADDR` |
|
|
| +2 | Cluster RPC | 18182 | `STEMEDB_NODE_RPC_ADDR` |
|
|
| +3 | SWIM Gossip | 18183 | via `SwimConfig` |
|
|
| +4 | Metrics | 18184 | (reserved) |
|
|
| +5 | Admin | 18185 | (reserved) |
|
|
| +6 | Latent Signal | 18186 | — |
|
|
| +7 | Community App | 18187 | — |
|
|
| +8 | Admin Dashboard | 18188 | — |
|
|
|
|
## Crates
|
|
|
|
| Crate | Purpose | Status |
|
|
|-------|---------|--------|
|
|
| `stemedb-core` | Assertion, LifecycleStage, MaterializedView, types, signing | ✅ |
|
|
| `stemedb-wal` | Write-ahead log with crash recovery | ✅ |
|
|
| `stemedb-storage` | KVStore, VoteStore, IndexStore, TrustRankStore, QuarantineStore | ✅ |
|
|
| `stemedb-ingest` | Ingestion pipeline, signature verification, ContentDefenseLayer | ✅ |
|
|
| `stemedb-query` | Query engine, Materializer for O(1) MV reads | ✅ |
|
|
| `stemedb-lens` | Lenses (Recency, Consensus, Authority, Skeptic, Layered, etc.) | ✅ |
|
|
| `stemedb-api` | HTTP API with axum + utoipa OpenAPI docs | ✅ |
|
|
| `stemedb-sim` | Simulation for testing the pipeline | ✅ |
|
|
| `stemedb-merkle` | BLAKE3 Merkle tree for diff detection | ✅ |
|
|
| `stemedb-rpc` | gRPC services for node-to-node communication | ✅ |
|
|
| `stemedb-sync` | Merkle sync, gossip broadcast, anti-entropy | ✅ |
|
|
| `stemedb-cluster` | Cluster membership (SWIM), sharding, gateway | ✅ |
|
|
| `stemedb-ontology` | Domain definitions (Pharma), subject builders, medical extractors | ✅ |
|
|
| `stemedb-chaos` | Chaos testing infrastructure | ✅ |
|
|
| `stemedb-dashboard` | Admin dashboard (React/Next.js) | 🎯 In Progress (7 panels complete) |
|
|
|
|
## SDKs
|
|
|
|
| SDK | Purpose | Status |
|
|
|-----|---------|--------|
|
|
| `sdk/go/steme` | Go HTTP client with Ed25519 signing and fluent builders | ✅ |
|
|
| `sdk/go/adk` | ADK-Go tools and callbacks for AI agents | ✅ |
|
|
|
|
## Specialized Agents
|
|
|
|
| Domain | Agent | When to use |
|
|
|--------|-------|-------------|
|
|
| **Product Vision** | `episteme-product-visionary` | Use cases, "why not Postgres?", product-market fit |
|
|
| **Pilot Prep** | `enterprise-skeptic-buyer` | Pressure-test demos, find gaps, prepare for tough questions |
|
|
| General Rust | `primary-developer` | Feature implementation, refactoring |
|
|
| Code Quality | `rust-quality-engineer` | Reviews, test coverage, clippy |
|
|
| Storage | `storage-engine-architect` | WAL, LSM, crash recovery |
|
|
| Graph Engine | `rust-graph-engine-architect` | Lock-free structures, cache optimization |
|
|
| Defensive | `defensive-systems-architect` | Rate limiting, circuit breakers, hostile input |
|
|
| Distributed | `distributed-systems-engineer` | CRDT replication, Raft coordination, Merkle sync |
|
|
| Lenses | `stemedb-lens-architect` | Query resolution, ranking algorithms |
|
|
| Planning | `stemedb-planner` | Milestone planning, roadmap |
|
|
|
|
---
|
|
|
|
## Quick Reference
|
|
|
|
```bash
|
|
# Build
|
|
cargo build --workspace
|
|
|
|
# Test
|
|
cargo test --workspace
|
|
|
|
# Lint (must pass before commit)
|
|
cargo clippy --workspace -- -D warnings
|
|
cargo fmt --check
|
|
|
|
# Run API server
|
|
cargo run --bin stemedb-api
|
|
|
|
# Run demo script
|
|
./scripts/demo-consumer-health.sh
|
|
```
|
|
|
|
---
|
|
|
|
## Related Documents
|
|
|
|
- [CLAUDE.md](./CLAUDE.md) — AI assistant instructions and project rules
|
|
- [roadmap-archive.md](./roadmap-archive.md) — Completed phases 1-7 detail
|
|
- [docs/demo/pilot/amazement-demo.md](./docs/demo/pilot/amazement-demo.md) — Technical demo script
|
|
- [docs/demo/pilot/amazement-demo-2.md](./docs/demo/pilot/amazement-demo-2.md) — Executive demo script
|
|
- [uat/production-readiness/README.md](./uat/production-readiness/README.md) — Production verification checklist
|
|
- [.claude/agents/enterprise-skeptic-buyer.md](./.claude/agents/enterprise-skeptic-buyer.md) — Dr. Sarah Chen persona
|