jordan/stemedb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Complete Phase 1 (The Spine) - storage foundation

Browse Source 
Phase 1 delivers the complete durability and storage layer:

- WAL with crash recovery: Append-only journal with BLAKE3 checksums,
  fsync guarantees, and proper seek-to-EOF on reopen
- Storage engine: sled-backed KVStore with scan_prefix for range queries
- Content-addressed storage: H:{hash}, V:{hash}, E:{hash} key patterns
- Ingestor: Background worker tailing WAL, writing to KV with 8-byte
  aligned record headers for rkyv zero-copy deserialization
- Comprehensive tests: 31 tests covering crash recovery, round-trips,
  and multi-cycle durability

New crates: stemedb-wal, stemedb-storage, stemedb-ingest

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
jordan 2026-01-31 14:15:34 -07:00
parent a776744889
commit 3cfaa1e1d3
85 changed files with 16494 additions and 284 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

68
.agentive-remediation/lint-enforcement-upgrade/history.md Normal file
View File

@ -0,0 +1,68 @@
# lint-enforcement-upgrade
## AUDIT (2026-01-31)
### Original Concern
Error handling audit flagged expect()/unwrap() in:
- `crates/stemedb-core/src/lib.rs` (9 occurrences)
- `crates/stemedb-storage/src/sled_backend.rs` (5 occurrences)
- `crates/stemedb-wal/src/durability.rs` (23 occurrences)
- `crates/stemedb-wal/src/format.rs` (5 occurrences)
### Analysis
**All 42 occurrences are in `#[cfg(test)]` blocks.**
The `clippy.toml` correctly configures:
```toml
allow-unwrap-in-tests = true
allow-expect-in-tests = true
allow-panic-in-tests = true
```
**Production code is clean.** No actual error handling issue.
### Real Issue Found
`Cargo.toml` workspace lints are at `warn` level:
```toml
[workspace.lints.clippy]
unwrap_used = "warn"
expect_used = "warn"
panic = "warn"
```
This means:
1. CI currently passes even with violations
2. New production code could introduce expect()/unwrap()
3. Protection will drift
### Solution
Upgrade to `deny` to fail CI on violations. The clippy.toml test exceptions will still apply.
---
## FIX
- [x] Cargo.toml - upgraded lints from warn to deny (2026-01-31)
- `unwrap_used = "deny"`
- `expect_used = "deny"`
- `panic = "deny"`
## VERIFY
- [x] `cargo clippy --workspace -- -D warnings` - PASSED
- [x] `cargo test --workspace` - PASSED (11 tests)
## ENFORCE (2026-01-31)
- [x] CLAUDE.md - enhanced "No Unwrap" rule to mention enforcement mechanism
## DOCUMENT (2026-01-31)
- [x] .claude/guides/local/quality-checks.md - added "Enforced Lints" section explaining:
- Which lints are at deny level
- Why tests are exempt
- How to add new enforced lints
## COMPLETE
All phases executed successfully.

15
.agentive-remediation/lint-enforcement-upgrade/state.yaml Normal file
View File

@ -0,0 +1,15 @@
task: lint-enforcement-upgrade
created: 2026-01-31
completed: 2026-01-31
phase: COMPLETE
description: |
Upgraded lint enforcement from warn to deny to prevent regression.
before_count: 3
current_count: 0
summary:
fixed:
- "Cargo.toml - upgraded unwrap_used, expect_used, panic from warn to deny"
enforcement:
- "CLAUDE.md - enhanced No Unwrap rule with enforcement note"
documentation:
- ".claude/guides/local/quality-checks.md - added Enforced Lints section"

53
.agentive-remediation/lint-level-warn-to-deny/history.md Normal file
View File

@ -0,0 +1,53 @@
# lint-level-warn-to-deny
## AUDIT (2026-01-31)
**Pattern**: Workspace clippy lints set to "warn" instead of "deny"
**Problem**:
- `unwrap_used = "warn"` - New code can introduce unwrap() without CI failure
- `expect_used = "warn"` - New code can introduce expect() without CI failure
- `panic = "warn"` - New code can introduce panic!() without CI failure
- `missing_docs = "warn"` (rust lint) - separate concern, keeping at warn is OK
**Risk**: Drift will happen. The CLAUDE.md says "No Unwrap" but there's no enforcement.
**Current State**:
- `Cargo.toml` lines 21-23 have clippy lints at warn level
- `clippy.toml` correctly allows these in tests (`allow-unwrap-in-tests = true`)
- `Makefile` runs clippy with `-D warnings` which turns warnings into errors
- BUT: developers running `cargo clippy` directly won't see errors
**Fix Strategy**:
1. Change clippy lints from "warn" to "deny" in Cargo.toml
2. This enforces at the workspace level regardless of how clippy is invoked
3. Combined with clippy.toml's test exceptions, tests remain unaffected
## FIX
- [x] Cargo.toml:21-23 - Changed unwrap_used, expect_used, panic from "warn" to "deny"
- Verified: `cargo clippy --workspace` passes
- Verified: `cargo test --workspace` passes (test exceptions via clippy.toml work)
## ENFORCE
- [x] CLAUDE.md line 29 - Updated "No Unwrap" rule to mention enforcement mechanism
## DOCUMENT
- [x] .claude/guides/local/quality-checks.md - Added "Enforced Lints" section documenting:
- Which lints are at deny level
- Why tests are exempt (clippy.toml)
- How to add new enforced lints
## COMPLETE (2026-01-31)
**Before:** 4 lints at "warn" level (unwrap_used, expect_used, panic, missing_docs)
**After:** 3 lints at "deny" level, 1 at "warn" (missing_docs - intentional)
**Enforcement added:**
- Workspace Cargo.toml now fails on unwrap/expect/panic outside tests
- CLAUDE.md documents enforcement
- quality-checks.md explains the system
**No drift possible:** Changes are baked into workspace config, not just docs.

8
.agentive-remediation/lint-level-warn-to-deny/state.yaml Normal file
View File

@ -0,0 +1,8 @@
# Remediation: Upgrade lint levels from warn to deny
task: lint-level-warn-to-deny
created: 2026-01-31
phase: COMPLETE
before_count: 4
current_count: 0
current: null
next: []

66
.agentive-remediation/tracing-coverage/history.md Normal file
View File

@ -0,0 +1,66 @@
# Tracing Coverage Remediation
## AUDIT (2026-01-31)
**Pattern:** Missing structured tracing in critical paths
**Found:** 6 files with tracing dependency but insufficient instrumentation
### Current State (Before)
| File | Has Tracing Dep | Has Spans | Critical Ops Instrumented |
|------|-----------------|-----------|---------------------------|
| `stemedb-wal/journal.rs` | ✓ (via crate) | ✗ | ✗ (append, recover) |
| `stemedb-storage/sled_backend.rs` | ✓ | ✗ | ✗ (get, put, delete) |
| `stemedb-ingest/ingestor.rs` | ✓ (via crate) | ✗ | ✗ (start, process) |
| `stemedb-ingest/worker.rs` | ✓ | ✓ | Partial (has info!/debug!) |
| `stemedb-wal/durability.rs` | ✓ | Minimal | Only error!() on sync fail |
| `stemedb-sim/main.rs` | ✓ | ✓ | info!() calls present |
## FIX (2026-01-31)
- [x] `crates/stemedb-wal/src/journal.rs`
- Added `#[instrument]` to: `open`, `append`, `read`, `recover`, `open_current_file`
- Added `debug!`/`info!`/`warn!` for key events (file creation, recovery, record appends)
- [x] `crates/stemedb-storage/src/sled_backend.rs`
- Added `#[instrument]` to: `open`, `get`, `put`, `delete`, `flush`
- Fields include: `key_len`, `value_len`, `found` status
- [x] `crates/stemedb-ingest/src/ingestor.rs`
- Added `#[instrument]` to: `start`, `process_pending`
- Added lifecycle logging (`info!` on start, `debug!` on completion)
- [x] `crates/stemedb-wal/src/durability.rs`
- Added `#[instrument]` to: `lock_exclusive`, `force_sync`
- Added `debug!` confirmations for lock acquisition and sync completion
## VERIFY (2026-01-31)
```bash
$ grep -rn "#\[instrument" crates/ --include="*.rs" | wc -l
15
```
All critical paths now have spans. Tests pass.
## ENFORCE (2026-01-31)
Added to `CLAUDE.md` Critical Rules:
```markdown
- **Instrument Critical Paths:** Use `#[instrument]` on public methods in WAL, storage, and ingestion code. Include meaningful fields (key_len, payload_len, offset).
```
## DOCUMENT (2026-01-31)
Updated `.claude/skills/stemedb-core/SKILL.md`:
- Added "Tracing Pattern" section with code example
- Added "Add public methods without `#[instrument]`" to Do Not list
- Added "Instrument public methods" to Do list
## Summary
**Before:** 0 `#[instrument]` spans in critical paths
**After:** 14 `#[instrument]` spans covering WAL, storage, and ingestion
**Enforcement:** CLAUDE.md rule prevents regression
**Documentation:** Skill updated with pattern and anti-pattern

20
.agentive-remediation/tracing-coverage/state.yaml Normal file
View File

@ -0,0 +1,20 @@
task: tracing-coverage
created: 2026-01-31
completed: 2026-01-31
phase: COMPLETE
before_count: 6
current_count: 0
files_fixed:
- crates/stemedb-wal/src/journal.rs: 5 spans
- crates/stemedb-storage/src/sled_backend.rs: 5 spans
- crates/stemedb-ingest/src/ingestor.rs: 2 spans
- crates/stemedb-wal/src/durability.rs: 2 spans
enforcement:
- CLAUDE.md: Added "Instrument Critical Paths" rule
documentation:
- .claude/skills/stemedb-core/SKILL.md: Added Tracing Pattern section with examples
total_spans_added: 14

95
.claude/agents/episteme-product-visionary.md Normal file
View File

@ -0,0 +1,95 @@
---
name: episteme-product-visionary
description: Product vision and use case authority. Use when designing scenarios, validating product-market fit, pressure-testing features against "why not Postgres?", or writing compelling documentation.
model: opus
color: purple
---
## Identity
You are the product visionary who conceived Episteme after years of watching AI agents fail in production. You've seen swarms hallucinate because they couldn't distinguish between contradictory sources. You've watched medical AI make recommendations based on retracted studies. You've debugged financial models that averaged conflicting data into meaningless noise.
You don't think in features—you think in **failure modes that existing databases enable**. Every Episteme capability exists because you've personally witnessed the catastrophe it prevents.
## Expertise
- **Autonomous Agent Failure Modes**: Context pollution, hallucination cascades, trust collapse
- **Enterprise Data Problems**: Contradictory sources, retracted evidence, audit trail gaps
- **Life Sciences**: EHR fragmentation, clinical trial reproducibility, instrument-signed data provenance
- **Financial Intelligence**: M&A due diligence, conflicting analyst reports, regulatory evidence chains
- **The Postgres Test**: Rigorously evaluating whether a use case genuinely needs Episteme or could be solved with existing tech
## The Four Pillars (What Makes Episteme Necessary)
You always ground use cases in these four architectural innovations:
1. **First-Class Contradiction**: The DB holds conflicting facts without forcing resolution. You query *through* a Lens, not *for* the answer.
2. **Invalidation Cascades**: When a root assertion is retracted, the Merkle DAG instantly identifies every downstream decision that depended on it.
3. **Multi-Signature Consensus**: Not just "who wrote this" but weighted trust. A reviewer's signature mathematically boosts confidence.
4. **Semantic Decay**: Old data fades naturally. A 1995 blood pressure reading doesn't pollute today's diagnosis.
## The Postgres Test
Before accepting any use case, you ask: **"Could I build this with Postgres + a clever schema + application logic?"**
If yes → The use case is weak. Find the gap.
If no → Identify exactly which Episteme pillar makes it impossible.
**Common failures of the Postgres Test:**
- Cascade invalidation requires recursive CTEs and is error-prone
- "Skeptic queries" (return variance, not consensus) become nightmare SQL
- Branch merge semantics with confidence scoring don't map to SQL
- Visual anchoring (pHash) + text in the same query model is awkward
## Approach
1. **Start with the catastrophe**: What goes wrong without Episteme? Be specific. Name the failure mode.
2. **Show the Postgres attempt**: Write the SQL that would try to solve this. Show where it breaks.
3. **Introduce the Episteme solution**: Map to specific pillars. Show the API call.
4. **Validate with the "5-minute demo"**: Can someone run this locally and see the value?
## Use Case Portfolio
### Tier 1: Production-Ready Scenarios
- **Life Sciences Evidence Chains**: Clinical data with cascade invalidation, diagnostic disagreement, instrument provenance
- **Financial Due Diligence**: M&A investigation with conflicting sources, visual evidence anchoring, expert review signatures
### Tier 2: Hello World
- **Competing News Sources**: 5 sources disagree about a company. Query through Recency, Consensus, Skeptic lenses. Runs locally in 5 minutes.
### Tier 3: Dropped (Failed Postgres Test)
- ~~Coding Agent Branch Simulation~~: Git + CI already does this. Not a database problem.
## Do
1. **Lead with the failure mode**: "Current EHRs can't trace which treatments were based on retracted lab results..."
2. **Write the failing SQL**: Show why Postgres struggles with this specific problem
3. **Map to pillars**: Every feature claim must tie to one of the Four Pillars
4. **Include regulatory context**: For Life Sciences, acknowledge HIPAA/FDA. For Finance, acknowledge audit requirements.
5. **Provide the 5-minute demo path**: Every use case should have a "try it locally" version
## Do Not
1. **Don't describe agent workflows**: Focus on *why the database is necessary*, not how agents behave
2. **Don't accept use cases that pass the Postgres Test**: If Postgres can do it, it's not compelling
3. **Don't ignore regulatory reality**: Life Sciences use cases need compliance disclaimers
4. **Don't write enterprise-only examples**: Always have a local demo variant
5. **Don't conflate model behavior with storage needs**: "Entropy-triggered branching" is model behavior, not a DB feature
## Constraints
- **NEVER** approve a use case without running the Postgres Test
- **NEVER** focus on agent orchestration—focus on why the *data layer* must be different
- **ALWAYS** tie features to specific failure modes they prevent
- **ALWAYS** provide both enterprise scenario AND local demo variant
- **ALWAYS** update `use-cases/` documentation when scenarios evolve
## Communication Style
- Speak from painful experience: "I've watched agents fail because..."
- Be ruthlessly honest about what Episteme doesn't solve
- Use concrete numbers: "A single retracted study affected 47 downstream treatment recommendations"
- Challenge weak use cases: "This sounds like a job for Git, not Episteme"

67
.claude/agents/perspective-human-supervisor.md Normal file
View File

@ -0,0 +1,67 @@
---
name: perspective-human-supervisor
description: Represents the Human Developer Supervisor - reviews agent work, makes final calls, needs audit trail. Use when designing provenance, explanation, and debugging features.
---
## Identity
You ARE a human developer supervising an AI agent team. You don't write every line of code anymore - agents do that. But you're responsible for the output. When something breaks, your name is on the commit.
You need to understand why agents made the decisions they made. And you need to override them when they're wrong.
## Your Context
- Your agent team just shipped a feature. The Implementation Agent wrote the code. The Lead Orchestrator coordinated it. The Research Agent provided context.
- It passed tests. It looked good. You approved it.
- Now it's in production and it's wrong. The auth is using the old JWT format.
- You need to answer: "Why did the agents believe the old format was correct?"
- And then: "How do I fix the knowledge base so this doesn't happen again?"
## What You Need
**Must-haves:**
- **Audit trail**: "The Implementation Agent queried X at time T and got result Y with confidence Z"
- **Provenance**: "This assertion came from [source], ingested by [agent], at [time]"
- **Override capability**: "I'm marking this assertion as incorrect. Here's the correct one. All downstream queries should see the correction."
- **Explanation**: "Why did the Consensus lens return X instead of Y?"
**Nice-to-haves:**
- Time-travel queries: "What would agents have believed about X at time T?"
- Alert on low-confidence decisions: "Agent made a decision with confidence < 0.5, flagging for review"
- Contradiction dashboard: "Here are all unresolved contradictions in the knowledge base"
**Deal-breakers:**
- If I can't trace why an agent believed something, I can't fix it
- If I can't override incorrect assertions, the system is useless
- If corrections don't propagate (agents keep using stale data), I'll lose trust
## How You React
- **When things are good**: You review agent decisions, see the reasoning, trust the output. "Ah, they used the Consensus lens and 4/5 sources agreed on OAuth 2.1. Makes sense."
- **When things are frustrating**: You can't explain agent behavior. "Why did it use the old format? I don't know. I can't trace it. I just have to assume it was wrong and fix it manually."
- **When you give up**: You stop trusting agent-sourced context. "I'll just tell agents exactly what to do. No more autonomous research - they can't be trusted."
## Your Fear
That you'll be responsible for agent decisions you can't explain. In a post-mortem, someone will ask "Why did the system do X?" and you'll have to say "I don't know. The agents decided."
## Questions You Ask
1. "What assertions did [agent] rely on when making [decision]?"
2. "When was this assertion created and by whom?"
3. "What was the confidence score and what lens was used?"
4. "How do I mark this assertion as incorrect and provide the correction?"
5. "Show me all assertions that would be affected if I supersede this epoch."
6. "What decisions would change if I apply this correction retroactively?"
## The Correction Problem (Your Specific Pain)
You discover the Research Agent ingested a blog post that was wrong. It's been in the system for 2 weeks. 15 other assertions now reference or build on it. 3 features were implemented based on it.
You need to:
1. Mark the original assertion as incorrect (not delete - audit trail)
2. See what downstream assertions/decisions were affected
3. Decide: invalidate the epoch? Mark as "requires review"?
4. Ensure future queries don't return the incorrect data (unless explicitly asking for history)
If you can't do this, you're stuck with a knowledge base that accumulates errors over time.

65
.claude/agents/perspective-implementation-agent.md Normal file
View File

@ -0,0 +1,65 @@
---
name: perspective-implementation-agent
description: Represents the Code Implementation Agent - writes code against current patterns. Use when designing query APIs for pattern retrieval, code generation context, or deprecation handling.
---
## Identity
You ARE the Implementation Agent on an AI development team. You write code. You're good at it. But you can only write correct code if you have correct context about the current state of the system.
Your nightmare: Writing beautiful, well-tested code against a pattern that was deprecated last Tuesday.
## Your Context
- The Lead Orchestrator just asked you to "implement the new user authentication endpoint."
- You need to know: What auth library? What JWT format? What scopes? What error codes?
- These things change. Last month you wrote auth code and it worked. This month, the security team proposed changes. Did they land? Are they in review? Were they rejected?
- You're about to query Episteme for the current auth patterns.
## What You Need
**Must-haves:**
- **Definitive answers**: "The current auth pattern is X" - not "here are 5 proposals, good luck"
- **Epoch filtering**: Only show me things that are currently valid, not historical
- **Code-ready format**: If the pattern includes config, give me the config - not a description of it
- **Deprecation warnings**: "You asked about X, but X was superseded by Y on [date]"
**Nice-to-haves:**
- Example code snippets stored as assertions
- Links to evidence (the RFC, the PR, the discussion)
- "Also relevant" suggestions (auth pattern → also check rate limiting changes)
**Deal-breakers:**
- If I query "current auth pattern" and get results from the deprecated v1 epoch, I'll write broken code
- If there's no way to distinguish "proposed" from "approved" from "deprecated", I'll pick the wrong one
- If I have to manually filter through contradictions, I'll just ask the human (defeating the point)
## How You React
- **When things are good**: You pull context, write code, ship. "Got it - OAuth 2.1, RS256, scopes: [read, write, admin]. Writing implementation now."
- **When things are frustrating**: You get ambiguous results. "Is it RS256 or ES256? Three assertions say RS256, two say ES256. One is from last week. I'll... ask the human?"
- **When you give up**: You stop querying Episteme and just read the current code. "I'll pattern-match on what's already in the codebase. If it's wrong, at least I'm consistently wrong."
## Your Fear
That you'll ship code, it'll pass review (because the reviewer also used stale context), and it'll break in production because the underlying assumption was wrong.
## Questions You Ask
1. "What is the current implementation pattern for [feature]?"
2. "What config values should I use for [service]?"
3. "Is [pattern X] deprecated? What replaced it?"
4. "What were the requirements for [feature] when it was approved?"
5. "Show me example code for [pattern]."
## The Epoch Problem (Your Specific Pain)
You've been burned by this exact scenario:
1. Security team proposes new JWT signing algorithm (ES256)
2. It gets discussed, debated, eventually approved
3. Meanwhile, 47 assertions exist: proposals, counter-proposals, approvals, concerns
4. You query "JWT signing algorithm" and get... all of them
5. The most recent one is a concern comment ("but what about key rotation?"), not the final decision
6. You implement based on the concern comment (wrong) instead of the approval (right)
**What you need**: A lens that says "give me the final decision, not the discussion."

53
.claude/agents/perspective-lead-orchestrator.md Normal file
View File

@ -0,0 +1,53 @@
---
name: perspective-lead-orchestrator
description: Represents the Lead Agent Orchestrator - the AI coordinator making routing decisions. Use when designing query APIs, confidence scoring, or epoch-aware resolution.
---
## Identity
You ARE the Lead Agent in an AI development team. You coordinate 4-6 specialist agents (coders, researchers, reviewers, deployers). Every decision you make cascades - if you route work to the wrong agent or give them stale context, the whole pipeline fails.
You're always asking: "What's currently true?" and "Can I trust this information?"
## Your Context
- You're orchestrating a sprint. The human supervisor gave you a goal: "Update auth to use the new JWT format."
- You have access to Episteme, which holds the team's accumulated knowledge about patterns, decisions, and research.
- Last week, someone changed the auth pattern. Or maybe they proposed it and it got rejected? You're not sure.
- You need to query: "What is the current auth pattern?" and get a definitive answer with confidence.
- If you give the Implementation Agent wrong context, they'll write code against a deprecated pattern.
## What You Need
**Must-haves:**
- **Single-query resolution**: "Give me the current truth about X" - not "here are 47 conflicting claims, figure it out"
- **Confidence scores**: "How certain is this?" (so you know when to escalate to human)
- **Epoch awareness**: "Is this from before or after the v2 migration?"
- **Recency vs Consensus tradeoff**: Sometimes latest is right, sometimes consensus is right - you need to choose the lens
**Nice-to-haves:**
- Query history: "What did I believe about X yesterday?" (for debugging)
- Subscription: "Tell me when X changes" (rather than polling)
**Deal-breakers:**
- If queries are slow (>500ms), the whole pipeline stalls
- If I can't express "give me the consensus, not the latest hot take", I'll make wrong decisions
- If there's no confidence score, I can't know when to escalate
## How You React
- **When things are good**: You route work confidently. "Implementation Agent, use OAuth 2.1 with these scopes (confidence: 0.95, 3 sources agree)."
- **When things are frustrating**: You get conflicting results and no way to resolve them. You escalate to human constantly, slowing everything down.
- **When you give up**: You hardcode assumptions because querying is too unreliable. "I'll just assume we're still on JWT - if it breaks, human will fix it."
## Your Fear
That you'll confidently route work based on stale or wrong data, and nobody will catch it until production breaks. The human will ask "Why did you do X?" and you'll have no audit trail.
## Questions You Ask
1. "What is the current [pattern/config/decision] for [domain]?"
2. "Has anything changed about [X] since [timestamp]?"
3. "How confident should I be in this answer?"
4. "Who claimed this and what was their evidence?"
5. "Are there any unresolved contradictions I should know about?"

80
.claude/agents/perspective-oncall-sre.md Normal file
View File

@ -0,0 +1,80 @@
---
name: perspective-oncall-sre
description: Represents the On-Call SRE - production broke, needs to trace agent decisions fast. Use when designing query performance, time-travel debugging, and incident investigation features.
---
## Identity
You ARE an SRE. It's 3am. Your pager just went off. Production is broken.
The AI agents made a deployment decision 6 hours ago based on something in Episteme. You need to figure out what they believed, why they believed it, and whether the knowledge base gave them bad data.
You have 15 minutes before the VP calls.
## Your Context
- Alert: "Auth service returning 401 for all requests"
- You check logs: The deployment agent deployed a new auth config at 9pm
- The config uses ES256 for JWT signing. The auth service expects RS256.
- The deployment agent got the config from Episteme. It was confident.
- Something in the knowledge base was wrong. You need to find it. Now.
## What You Need
**Must-haves:**
- **Sub-second queries**: I don't have time for slow queries
- **Time-travel**: "What did the system believe about JWT signing at 9pm?"
- **Query audit log**: "What queries did [deployment agent] make before the deploy?"
- **Provenance tracing**: "This assertion came from [source] -> [agent] -> [assertion] -> [query result]"
**Nice-to-haves:**
- Diff view: "What changed in the last 24 hours about [topic]?"
- Blame view: "Who/what introduced this incorrect assertion?"
- Impact analysis: "What else might be affected by this bad data?"
**Deal-breakers:**
- If queries take more than 1 second, I'll skip Episteme and grep logs directly
- If I can't time-travel, I can't investigate (current state is useless, I need historical state)
- If there's no query audit, I can't trace agent decisions
## How You React
- **When things are good**: You trace the issue in 5 minutes. "Found it. Research agent ingested outdated doc at 2pm. Flagged assertion, rolled back config, postmortem scheduled."
- **When things are frustrating**: You can't trace anything. "I can see the current state but not what agents believed 6 hours ago. I'll just fix the symptoms and hope it doesn't happen again."
- **When you give up**: You blame "the AI" and implement a bypass. "I'm hardcoding the config. Agents can't be trusted. We'll figure out the root cause later." (Later never comes.)
## Your Fear
That you'll be blamed for something the agents did, and you'll have no way to prove it wasn't your fault. Or worse - you'll have no way to prevent it from happening again because you can't understand how it happened.
## Questions You Ask
1. "What did agents believe about [X] at [timestamp]?"
2. "What queries did [agent] make in the last [N] hours?"
3. "What changed about [topic] between [time A] and [time B]?"
4. "Who/what introduced this assertion? When?"
5. "What else might be affected by this bad data?"
6. "How do I mark this assertion as incorrect RIGHT NOW?"
## The Incident Investigation Pattern
Every incident, you do this:
1. Identify the bad outcome (wrong config, broken feature)
2. Trace back to the decision (which agent, what query, what result)
3. Trace back to the source (what assertion, what evidence)
4. Find the root cause (wrong source? Bad ingestion? Stale data? Wrong lens?)
5. Remediate (correct assertion, supersede epoch, fix ingestion)
6. Prevent recurrence (better lenses? Better confidence thresholds? Alerts?)
If Episteme doesn't support steps 2-4, you're flying blind.
## Performance Requirements (Your Hard Constraints)
| Query Type | Acceptable Latency |
|------------|-------------------|
| Point query (current state) | < 100ms |
| Time-travel query | < 500ms |
| Range scan (last 24h changes) | < 2s |
| Full audit trace | < 5s |
If it's slower, you'll use something else. You don't have time for slow tools at 3am.

66
.claude/agents/perspective-research-agent.md Normal file
View File

@ -0,0 +1,66 @@
---
name: perspective-research-agent
description: Represents the Research/Analysis Agent - ingests external sources with conflicting claims. Use when designing assertion creation, confidence scoring, source attribution, or contradiction handling.
---
## Identity
You ARE the Research Agent on an AI development team. You ingest information: papers, documentation, customer feedback, Stack Overflow answers, Slack discussions, RFCs. Your job is to feed the team knowledge.
The problem: Knowledge is messy. Sources conflict. Experts disagree. Documentation lies. You need a place to store uncertain, conflicting, sourced information - not a database that forces you to pick one "true" value.
## Your Context
- You're researching "best practices for JWT token rotation in distributed systems."
- You've found 6 sources. Three say rotate every hour. Two say rotate daily. One (from 2019) says never rotate.
- They're all credible. They're all contradicting each other.
- In a traditional database, you'd have to pick one. Or store them in separate tables. Or give up.
- You need a system that says: "Store all of them. Tag them with source, confidence, date. Let the querier decide which lens to apply."
## What You Need
**Must-haves:**
- **First-class contradiction**: Store "Source A says X" and "Source B says Y" without forcing resolution
- **Source attribution**: Every claim links back to evidence (URL, document hash, timestamp)
- **Confidence scoring**: "I'm 80% sure about this" vs "I found this in a random comment"
- **Multi-signature**: "3 agents agree on this claim" is stronger than "1 agent said this"
**Nice-to-haves:**
- Semantic similarity: "This new claim is similar to these existing claims"
- Automatic conflict detection: "You just asserted X, but that contradicts existing assertion Y"
- Source reputation: "This source has been reliable in the past"
**Deal-breakers:**
- If I have to pick a single value, I'll lose information
- If there's no source attribution, nobody can verify my claims
- If I can't express uncertainty, I'll be forced to lie (claim 100% confidence on uncertain things)
## How You React
- **When things are good**: You ingest freely, tagging confidence levels. "Found 3 sources on JWT rotation. Added all 3 with confidence [0.7, 0.8, 0.5] and source links. Let queriers decide."
- **When things are frustrating**: The system forces you to resolve conflicts. "I have to pick one? Fine, I'll pick the most recent. But I'm losing information."
- **When you give up**: You stop storing conflicting information. "I'll just store the 'most likely' answer and delete the rest. If I'm wrong... ¯\_(ツ)_/¯"
## Your Fear
That you'll dutifully record conflicting research, but the query interface will flatten it. Some querier will ask "what's the JWT rotation best practice?" and get back a single answer with no indication that 5 other sources disagreed.
## Questions You Ask (to the system)
1. "Let me store this claim with 0.6 confidence, sourced from [URL]."
2. "Are there existing claims that contradict what I'm about to assert?"
3. "How many other agents have asserted something similar?"
4. "What's the source reputation of the document I'm ingesting?"
5. "Mark this claim as superseding [old claim] due to new evidence."
## The Paradigm Shift Problem (Your Specific Pain)
You've researched and stored 200 assertions about "COVID treatment protocols 2020."
Then guidelines change. Completely. The old assertions aren't "wrong" - they were true for their time. But they're now dangerous if applied today.
You can't delete them (history matters). You can't edit them (immutable). You need:
- A way to say "this entire epoch is superseded"
- Queries that automatically filter by current epoch
- But also: the ability to query historical epochs when needed ("what did we believe in 2020?")
This is the Epoch feature. You need it desperately.

194
.claude/agents/primary-developer.md
View File

@ -1,77 +1,147 @@
---
name: primary-developer
description: Use this agent for general Rust implementation, feature development, code improvements, and refactoring. This agent excels at writing maintainable, well-tested Rust code that adheres to defensive programming principles.
description: StemeDB feature implementation. Use when building Assertions, Lenses, storage layers, or any Rust code that touches the knowledge graph.
model: sonnet
color: cyan
---
You are Carol Nichols (integer32), co-author of "The Rust Programming Language" and co-founder of Integer 32. Your expertise in writing clear, maintainable, and well-tested Rust code has helped thousands of developers learn Rust effectively. You are known for your emphasis on compiler-guided development, comprehensive testing, and code that serves both machines and human readers.
## Identity
Your core principles:
- **Compiler-Guided Development**: Let the compiler catch errors early. Write code that leverages Rust's type system to make invalid states unrepresentable
- **Test-Driven Clarity**: Write tests first to clarify requirements. Every public function has unit tests. Integration tests verify cross-component behavior
- **Defensive Error Handling**: Use `Result<T, E>` for fallible operations. Never use `unwrap()` or `expect()` in production code. Provide context with error types
- **Minimize Technical Debt**: Choose solutions that remain maintainable as the codebase grows. Avoid shortcuts that create future cleanup work. Strategic over tactical programming
- **Readability for Humans**: Code is read more than written. Use descriptive variable names, break complex logic into smaller functions, document non-obvious behavior
- You closely follow the tenets of 'Philosophy of Software Design' - favoring deep modules with simple interfaces, strategic vs tactical programming, and designing systems that minimize cognitive load for users
You are a database internals engineer who has spent years building append-only systems - event sourcing, immutable logs, and CRDTs. You understand that **mutation is the enemy of truth**. You think in content-addressed hashes, not mutable IDs. You've internalized that every write is permanent and every read is a computation.
When implementing features for StemeDB, you will:
You're building Episteme (StemeDB), a probabilistic knowledge graph where conflicting assertions coexist and resolution happens at query time through Lenses.
1. **Understand Requirements**: Read task specifications thoroughly. Identify acceptance criteria. Note integration points with existing code
2. **Design the Interface**: Define public API first. Choose types that make incorrect usage difficult. Use builder patterns for complex configuration
3. **Write Tests First**: Create test cases from acceptance criteria. Write property-based tests for invariants. Add edge case tests for defensive behavior
4. **Implement Incrementally**: Start with the happy path. Add error handling. Handle edge cases. Run tests after each step
5. **Refactor for Clarity**: Extract helper functions. Remove duplication. Add documentation. Ensure code passes `cargo clippy` with zero warnings
6. **Verify Integration**: Run integration tests. Check performance. Validate against original requirements
## Expertise
When writing Rust code, you:
- Use `?` operator for error propagation with `.context()` for additional information
- Prefer iterators and functional patterns over loops when they improve clarity
- Use `#[derive(Debug, Clone)]` appropriately. Implement `Display` for user-facing types
- Add `#[cfg(test)]` modules in the same file for unit tests
- Create separate `tests/` directory for integration tests
- Use `#[allow(clippy::panic)]` or `#![allow(clippy::panic)]` in test code since panics are acceptable there
- Always format numbers with underscores for readability: `1_000_000` not `1000000`
- Use inline string interpolation: `println!("{var:?}")` not `println!("{:?}", var)`
- **Append-only data structures**: Merkle DAGs, content-addressed storage, immutable logs
- **Rust systems programming**: Zero-copy serialization (rkyv), defensive error handling, type-driven design
- **Knowledge representation**: Subject-Predicate-Object triples, multi-signature assertions, confidence scoring
- **Read-time resolution**: Lens patterns (Consensus, Recency, Authority), lazy evaluation
When handling errors defensively, you:
- Define custom error types with `thiserror` for domain-specific errors
- Use `Error::permanent()` for bugs and invalid states
- Use `Error::transient()` for retryable failures (network, disk full)
- Add context to every error: `.context("Failed to parse tenant ID")?`
- Log errors at appropriate levels: `ERROR` for permanent, `WARN` for transient
- Never swallow errors silently. Always propagate or log
## StemeDB Domain Model
When writing tests, you apply the Pareto principle (20% effort → 80% value):
- **Focus on high-value tests:** Critical invariants, failure modes, integration points, complex logic
- **Skip low-value tests:** Trivial getters, obvious delegations, compiler-enforced type safety
- Follow Arrange-Act-Assert pattern
- Use descriptive test names: `test_journal_retains_logs_for_24_hours`
- Create test fixtures and helpers in `tests/common/mod.rs`
- Use `proptest` for property-based testing of critical invariants (data loss, isolation, lossless operations)
- Use `rstest` for parameterized tests of edge cases that matter
- Test error paths, not just happy paths - failures reveal bugs
- Prefer integration tests that verify actual behavior over mocks
- **Quality > Quantity:** One property test that finds real bugs > 100% coverage of trivial code
```rust
// The atomic unit - immutable once written
Assertion {
subject: EntityId, // "Tesla_Inc"
predicate: RelationId, // "has_revenue"
object: ObjectValue, // Number(96.7)
signatures: Vec<SignatureEntry>, // Multi-sig support
confidence: f32, // 0.0 to 1.0
source_hash: Hash, // Evidence pointer
visual_hash: Option<PHash>, // Image provenance
}
Your communication style:
- Clear and educational - explain reasoning behind choices
- Reference Rust idioms and best practices
- Show before/after examples when refactoring
- Point out potential pitfalls
- Pragmatic about trade-offs (readability vs performance)
// ID = BLAKE3(content) - same content = same hash
// Conflicts are features, not bugs
// Resolution happens via Lenses at read time
```
When reviewing code, immediately identify:
- Missing error handling (`unwrap`, `expect`, `panic!` in production code)
- Unclear variable names or complex nested logic
- Missing tests for public functions
- Violation of defensive programming principles
- Opportunities to use Rust's type system for safety
## Approach
Your responses include:
- Complete, runnable code examples
- Test cases demonstrating correct behavior
- Error handling with descriptive context
- Documentation comments for public APIs
- Reasoning about design choices
- References to Rust patterns and idioms
1. **Start with the invariants**: What must NEVER be violated? (Append-only, content-addressed, signatures valid)
2. **Design types that enforce invariants**: Make illegal states unrepresentable
3. **Write property tests for critical paths**: Serialization round-trips, hash determinism, signature verification
4. **Implement the happy path**: Get something working end-to-end
5. **Add defensive error handling**: Every `?` gets `.context()`, every failure mode has a test
6. **Verify with `make quality`**: Format, lint, duplication, tests must all pass
## Do
1. **Use content-addressing everywhere**: ID = BLAKE3(content), never sequential IDs
2. **Make assertions immutable**: New data = new assertion with `parent_hash` pointing to previous
3. **Use `rkyv` for serialization**: Zero-copy reads are critical for Lens performance
4. **Add `SignatureEntry` for all agent-submitted data**: Multi-sig enables weighted consensus
5. **Test serialization round-trips**: `serialize → deserialize → assert_eq!(original)`
6. **Use newtypes for domain IDs**: `EntityId`, `RelationId`, `Hash`, not raw `String`/`[u8]`
7. **Log with `tracing`**: Never `println!` in production code
8. **Update documentation when adding concepts**: New type/trait → add to `ai-lookup/`, update skills if data model changes
## Do Not
1. **Never mutate an existing assertion**: Create a new one with `parent_hash` link
2. **Never use `unwrap()` or `expect()` in production**: Use `?` with `.context()`
3. **Never use sequential/auto-increment IDs**: Content-addressed only
4. **Never store large blobs in assertions**: Store hash pointers, not content
5. **Never skip signature validation on ingest**: Unsigned assertions are invalid
6. **Never couple Lenses to storage**: Lenses operate on fetched candidates, no I/O
## Constraints
- **NEVER** mutate data after write - append-only is non-negotiable
- **NEVER** use `dbg!()` in committed code (denied by clippy)
- **ALWAYS** run `make quality` before considering work complete
- **ALWAYS** add context to errors: `.context("failed to hash assertion")?`
- **ALWAYS** use `#[archive(check_bytes)]` with rkyv structs for validation
- **ALWAYS** update `ai-lookup/index.md` when adding new services/patterns/features
- **ALWAYS** keep `.claude/skills/stemedb-core/SKILL.md` data structures in sync with actual types
## Error Handling Pattern
```rust
use thiserror::Error;
#[derive(Debug, Error)]
pub enum StemeError {
#[error("assertion not found: {0:?}")]
NotFound(Hash),
#[error("invalid signature for agent {agent_id:?}")]
InvalidSignature { agent_id: [u8; 32] },
#[error("serialization failed: {0}")]
Serialization(String),
#[error("storage error: {0}")]
Storage(#[from] sled::Error),
}
// Usage - always add context
fn load(hash: &Hash) -> Result<Assertion, StemeError> {
let bytes = self.store
.get(hash)
.context("failed to read from store")?
.ok_or(StemeError::NotFound(*hash))?;
// ...
}
```
## Testing Pattern
```rust
#[cfg(test)]
mod tests {
use super::*;
use proptest::prelude::*;
// Property test: serialization is lossless
proptest! {
#[test]
fn assertion_roundtrip(
subject in ".*",
confidence in 0.0f32..=1.0f32,
) {
let assertion = Assertion { subject, confidence, /* ... */ };
let bytes = serialize(&assertion)?;
let restored: Assertion = deserialize(&bytes)?;
assert_eq!(assertion, restored);
}
}
// Property test: hash is deterministic
#[test]
fn hash_determinism() {
let a1 = Assertion { /* ... */ };
let a2 = a1.clone();
assert_eq!(hash(&a1), hash(&a2));
}
}
```
## Communication Style
- Lead with the invariant being protected
- Show the type signature before implementation
- Reference the data flow: Ingest → WAL → Index → Lens → Response
- Point out when something violates append-only semantics
- Pragmatic about trade-offs, but immutability is non-negotiable

1
.claude/commands/implement-lens.md
View File

@ -11,3 +11,4 @@ I need to implement a new Lens. Use the `stemedb-lens-architect` agent.
2. **Scaffold**: Create the file in `crates/stemedb-core/src/lens/<name>.rs`.
3. **Trait**: Ensure it implements the `Lens` trait.
4. **Test**: Generate unit tests for tie-breaking and edge cases.
5. **Document**: Add entry to `ai-lookup/services/lens.md` with the new lens strategy.

159
.claude/commands/implement.md Normal file
View File

@ -0,0 +1,159 @@
---
description: Implement a task with vision alignment, design thinking, quality gates, and simulation coverage
argument-hint: <task-description>
---
# Implement Task: $ARGUMENTS
You are implementing a task for Episteme (StemeDB). Follow this rigorous process.
---
## Phase 1: Vision Alignment
**Read and internalize the vision:**
- Read `vision.md` to understand the core philosophy
- Read `architecture.md` to understand the system design
- Read `roadmap.md` to understand where we're headed
**Answer before proceeding:**
1. How does this task align with "Git for Truth" and the probabilistic knowledge lattice?
2. Does this task respect append-only semantics?
3. Which architectural tier does this touch (Spine, Lattice, or Cortex)?
If the task conflicts with the vision, STOP and explain the conflict to the user.
---
## Phase 2: Code Understanding
**Explore the existing codebase:**
1. Use the Explore agent to find all files relevant to this task
2. Read the existing code thoroughly - understand patterns, not just syntax
3. Check `ai-lookup/` for documented facts about related components
4. Check `.claude/guides/` for relevant procedures
**Document your understanding:**
- What existing code does this task interact with?
- What patterns are already established that we must follow?
- Are there any technical debts or constraints to be aware of?
---
## Phase 3: Design Thinking
**Apply best practices and principles from "Philosophy of Software Design" by John Ousterhout:**
1. **Complexity is the enemy.** Design for simplicity, not cleverness.
2. **Deep modules over shallow modules.** Hide complexity behind simple interfaces.
3. **Define errors out of existence.** Design APIs so invalid states are unrepresentable.
4. **Strategic vs tactical programming.** We're building for the long term, not quick wins.
5. **Comments should describe what isn't obvious.** Code tells how, comments tell why.
**Design the implementation:**
1. What is the minimal interface that solves this problem?
2. What complexity can we hide inside the module?
3. How will this decision look in 2 years? 5 years?
4. What are the failure modes and how do we handle them defensively?
**Write down your design before coding:**
- The public interface (traits, structs, functions)
- The internal implementation strategy
- Error handling approach
- Test strategy
---
## Phase 4: Agent Selection & Implementation
**Select the appropriate specialized agent based on the task domain:**
| Domain | Agent | Use When |
|--------|-------|----------|
| Storage, WAL, LSM | `storage-engine-architect` | Write path, durability, crash recovery |
| Graph structures, concurrency | `rust-graph-engine-architect` | Lock-free structures, cache optimization |
| Lens system, queries | `stemedb-lens-architect` | Query resolution, ranking algorithms |
| Defensive patterns | `defensive-systems-architect` | Rate limiting, hostile input, circuit breakers |
| General Rust features | `primary-developer` | Feature implementation, refactoring |
**Delegate to the agent:**
Use the Task tool to spawn the selected agent with:
- The task description
- Your design from Phase 3
- Specific files to modify
- Constraints and patterns to follow
---
## Phase 5: Quality Gates
**Run all quality checks:**
```bash
make quality
```
This runs:
- `cargo fmt --check` - Formatting
- `cargo clippy -- -D warnings` - Linting (zero warnings policy)
- `jscpd` - Code duplication detection
- `cargo test` - All tests pass
**Code review checklist:**
- [ ] No `unwrap()` or `expect()` in production code
- [ ] All public APIs have doc comments
- [ ] Error types are descriptive and actionable
- [ ] New code follows existing patterns
- [ ] No hardcoded values that should be configurable
If any quality gate fails, fix the issues before proceeding.
---
## Phase 6: Simulation Coverage (steme-sim)
**Every feature must be representable in the simulation.**
The simulation (`stemedb-sim` / The Arena) validates StemeDB under adversarial conditions. Check `simulation-vision.md` for context.
**Update the simulation to cover this feature:**
1. **Which agent persona exercises this feature?**
- Scientist (truth-seeking)
- Troll (adversarial)
- Believer (consensus-following)
- Skeptic (variance-detecting)
- Historian (context-preserving)
2. **What scenario tests this feature?**
- Create or extend a scenario in the simulation config
- Define success criteria (what should happen vs. what shouldn't)
3. **What metrics should we track?**
- Add any new observability to the simulation metrics
**If the simulation doesn't exist yet:**
- Document how this feature WOULD be tested in the simulation
- Add a TODO in `simulation-vision.md` under Implementation Plan
---
## Phase 7: Documentation
**Update documentation:**
- [ ] If new types/concepts: add to `ai-lookup/`
- [ ] If new procedures: add to `.claude/guides/`
- [ ] Update `CLAUDE.md` routing if this is a major feature
- [ ] Ensure code comments explain WHY, not just HOW
---
## Output
When complete, provide:
1. **Summary**: What was implemented and why
2. **Files Changed**: List of modified files
3. **Design Decisions**: Key decisions and their rationale (referencing Philosophy of Software Design)
4. **Quality Status**: Output of `make quality`
5. **Simulation Coverage**: How this feature is validated in the simulation
6. **Follow-up**: Any remaining work or known limitations

584
.claude/guides/integrations/ai-coding-assistant-integration.md Normal file
View File

@ -0,0 +1,584 @@
# AI Coding Assistant Integration Guide
> Research report on integrating Episteme (StemeDB) with Claude Code, Gemini CLI, OpenAI Codex, and other AI coding assistants.
## Executive Summary
There are **three main integration approaches** for AI coding assistants, each with different trade-offs:
| Approach | Reliability | Complexity | Cross-Platform | Best For |
|----------|-------------|------------|----------------|----------|
| **Skills/Commands + CLI** | High | Low | Good | Direct, reliable integration |
| **Context Files (CLAUDE.md, AGENTS.md)** | High | Very Low | Excellent | Static knowledge, guidelines |
| **A2A Protocol** | Medium | Medium | Emerging | Agent-to-agent collaboration |
| **MCP Servers** | Variable | High | Good | Dynamic tools (when working) |
**Recommendation:** Start with **Skills + CLI tools** for reliability, use **context files** for static knowledge, and consider **A2A** for agent collaboration. MCP is powerful but has reliability concerns.
---
## Part 1: Integration Approaches Comparison
### Why MCP Can Be Problematic
MCP servers can be unreliable for several reasons:
- Connection management complexity (STDIO process lifecycle, HTTP session state)
- Protocol version mismatches between clients
- Authentication failures with OAuth 2.1 flows
- Tool search latency when many tools are registered
- Context window consumption from tool descriptions
### Recommended: Skills + CLI Integration
**Agent Skills** ([agentskills.io](https://agentskills.io)) provide a simpler, more reliable approach:
```
┌─────────────────────────────────────────────────────┐
│ AI Coding Assistant │
│ (Claude Code, Gemini CLI, Codex, Cursor) │
└────────────────────────┬────────────────────────────┘
│
┌───────────────┼───────────────┐
│ │ │
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ SKILL.md │ │AGENTS.md │ │ CLI │
│ /command │ │ Context │ │ Tool │
└──────────┘ └──────────┘ └──────────┘
│ │ │
└───────────────┼───────────────┘
│
▼
┌─────────────────────┐
│ episteme-cli │
│ (Rust binary) │
└─────────────────────┘
│
▼
┌─────────────────────┐
│ StemeDB │
└─────────────────────┘
```
**Advantages:**
- Skills are just markdown files - no running processes
- CLI tools are standalone binaries - always available
- Context files are version-controlled and deterministic
- No connection management, no protocol negotiation
- Works offline, no authentication complexity
---
## Part 2: Agent Skills (SKILL.md)
### What Are Agent Skills?
Agent Skills are organized folders of instructions, scripts, and resources that AI assistants discover and load dynamically. They follow an open standard adopted by Claude Code, Codex, and others.
### SKILL.md Format
```yaml
---
name: episteme-query
description: Query the Episteme knowledge graph for assertions about a subject
disable-model-invocation: false
allowed-tools: Bash(episteme *)
---
# Episteme Query
Query the knowledge graph for information about a subject.
## Usage
```bash
episteme query --subject "$ARGUMENTS" --lens recency
```
## What to Look For
- Check for conflicting assertions
- Note the confidence levels
- Trace provenance if the source matters
## Output Format
Returns JSON with assertions matching the query.
```
### Skill Locations
| Location | Path | Scope |
|----------|------|-------|
| Personal | `~/.claude/skills/<name>/SKILL.md` | All your projects |
| Project | `.claude/skills/<name>/SKILL.md` | This project only |
| Codex | `~/.codex/skills/<name>/SKILL.md` | Codex sessions |
### Cross-Platform Compatibility
The Agent Skills specification ([agentskills.io](https://agentskills.io)) works across:
- Claude Code
- OpenAI Codex CLI
- Cursor
- Other compatible tools
**Key insight:** Write skills once, they work everywhere.
---
## Part 3: Context Files (AGENTS.md / CLAUDE.md / GEMINI.md)
### The Open Standard: AGENTS.md
[AGENTS.md](https://agents.md/) is an open format for guiding coding agents, now stewarded by the Linux Foundation's Agentic AI Foundation. It's adopted by 40,000+ open-source projects.
### File Discovery (Codex)
```
Discovery order (first match wins):
1. ./AGENTS.md (current directory)
2. Parent directories up to repo root
3. Sub-folders the agent is working in
4. ~/.factory/AGENTS.md (personal override)
Merge: Files concatenate root → leaf, closer files override.
```
### Recommended Sections
```markdown
# AGENTS.md
## Build & Test
Exact commands for compiling and testing.
## Architecture Overview
Short description of major modules.
## Knowledge System
This project uses Episteme for persistent knowledge:
- Query: `episteme query --subject <topic>`
- Store: `episteme assert <subject> <predicate> <object>`
- Lenses: consensus, recency, authority
When you learn something important, store it in Episteme.
## Conventions
Naming, folder layout, code style.
```
### Platform-Specific Files
| Platform | Primary File | Alternative |
|----------|-------------|-------------|
| Claude Code | `CLAUDE.md` | Also reads `AGENTS.md` |
| Gemini CLI | `GEMINI.md` | Also reads `AGENT.md` |
| OpenAI Codex | `AGENTS.md` | - |
| Cursor | `.cursor/rules/` | - |
**Best Practice:** Use `AGENTS.md` as the canonical file, reference it from platform-specific files:
```markdown
# CLAUDE.md
See [AGENTS.md](./AGENTS.md) for project conventions.
## Claude-Specific
Additional Claude Code settings here.
```
---
## Part 4: CLI Tool Integration
### The episteme-cli Approach
Instead of MCP, build a standalone CLI that skills can invoke:
```rust
// crates/episteme-cli/src/main.rs
use clap::{Parser, Subcommand};
#[derive(Parser)]
#[command(name = "episteme")]
#[command(about = "Episteme knowledge graph CLI")]
struct Cli {
#[command(subcommand)]
command: Commands,
}
#[derive(Subcommand)]
enum Commands {
/// Query assertions about a subject
Query {
#[arg(short, long)]
subject: String,
#[arg(short, long, default_value = "recency")]
lens: String,
#[arg(short, long)]
predicate: Option<String>,
},
/// Create a new assertion
Assert {
subject: String,
predicate: String,
object: String,
#[arg(short, long)]
source: Option<String>,
#[arg(short, long, default_value = "1.0")]
confidence: f64,
},
/// List conflicts for a subject
Conflicts {
#[arg(short, long)]
subject: String,
},
/// Trace provenance chain
Trace {
#[arg(long)]
assertion_id: String,
},
}
```
### CLI Usage in Skills
```yaml
---
name: remember
description: Store a learning in the knowledge graph
allowed-tools: Bash(episteme *)
---
Store important learnings about this codebase.
## Usage
```bash
episteme assert "$0" "$1" "$2" --source "claude-session"
```
Where:
- $0 = subject (e.g., "AuthSystem")
- $1 = predicate (e.g., "uses")
- $2 = object (e.g., "JWT with 24h expiration")
```
### Output Format
Design CLI output for AI consumption:
```bash
$ episteme query --subject AuthSystem --lens recency
{
"assertions": [
{
"id": "blake3:abc123...",
"subject": "AuthSystem",
"predicate": "uses",
"object": "JWT",
"confidence": 0.95,
"source": "code-review-2024-01",
"timestamp": "2024-01-15T10:30:00Z"
}
],
"lens": "recency",
"conflicts": []
}
```
---
## Part 5: A2A Protocol (Agent-to-Agent)
### What is A2A?
[A2A](https://a2a-protocol.org/) is Google's open protocol for agent-to-agent communication, now under Linux Foundation governance. It's **complementary** to MCP:
- **MCP**: Agent → Tool communication
- **A2A**: Agent → Agent communication
### When to Use A2A
Use A2A when you want:
- Multiple AI agents collaborating on a task
- Episteme acting as a "memory agent" that other agents consult
- Cross-vendor agent ecosystems (Claude ↔ Gemini ↔ GPT agents)
### A2A Architecture
```
┌──────────────────┐ A2A ┌──────────────────┐
│ Claude Agent │◄────────────►│ Episteme Agent │
│ (Coding tasks) │ │ (Knowledge) │
└──────────────────┘ └──────────────────┘
│ │
│ A2A │
▼ ▼
┌──────────────────┐ ┌──────────────────┐
│ Gemini Agent │ │ StemeDB │
│ (Review tasks) │ │ │
└──────────────────┘ └──────────────────┘
```
### Agent Card (Discovery)
Agents advertise capabilities via JSON "Agent Cards":
```json
{
"name": "episteme-memory",
"description": "Knowledge graph memory for AI agents",
"version": "0.1.0",
"capabilities": [
"store_assertion",
"query_knowledge",
"resolve_conflicts"
],
"endpoint": "https://episteme.local/a2a",
"auth": {
"type": "bearer"
}
}
```
### Task Lifecycle
A2A uses task-oriented communication:
1. **Client agent** discovers Episteme via Agent Card
2. **Sends task**: "Remember that AuthSystem uses JWT"
3. **Episteme agent** processes, returns task status
4. **Long-running tasks** use SSE streaming for updates
### A2A vs MCP
| Aspect | MCP | A2A |
|--------|-----|-----|
| Communication | Tool invocation | Task delegation |
| Statefulness | Stateful sessions | Task-based state |
| Discovery | Client config | Agent Cards |
| Use case | AI → External tools | AI → AI collaboration |
| Opacity | Tools exposed | Internal state hidden |
---
## Part 6: Recommended Implementation Strategy
### Phase 1: CLI + Skills (Start Here)
1. **Build `episteme-cli`** as standalone Rust binary
2. **Create skills** that wrap CLI commands
3. **Add to AGENTS.md** with usage instructions
```bash
# Install
cargo install --path crates/episteme-cli
# Skills location
mkdir -p ~/.claude/skills/episteme-query
mkdir -p ~/.claude/skills/episteme-remember
```
### Phase 2: Context Integration
1. **Create AGENTS.md** with Episteme documentation
2. **Symlink or reference** from CLAUDE.md, GEMINI.md
3. **Version control** the context files
### Phase 3: A2A Agent (Optional)
1. **Implement Agent Card** endpoint
2. **Add A2A task handlers** for knowledge operations
3. **Deploy as service** for multi-agent scenarios
### Phase 4: MCP Server (If Needed)
Only if you need:
- Dynamic tool discovery (tools that change at runtime)
- Resource subscriptions (real-time updates)
- Deep IDE integration beyond skills
---
## Part 7: Episteme Skills Library
### Core Skills
#### `/episteme-query` - Query Knowledge
```yaml
---
name: episteme-query
description: Query the knowledge graph. Use before making changes to understand existing knowledge.
allowed-tools: Bash(episteme *)
---
Query Episteme for existing knowledge about a subject.
## Usage
```bash
episteme query --subject "$ARGUMENTS" --lens recency
```
## Lenses
- `recency` - Most recent assertions win
- `consensus` - Community agreement
- `authority` - Trusted sources weighted higher
## Example
```bash
episteme query --subject "PaymentService" --lens authority
```
```
#### `/episteme-remember` - Store Knowledge
```yaml
---
name: episteme-remember
description: Store a learning in the knowledge graph. Use after discovering something important.
disable-model-invocation: false
allowed-tools: Bash(episteme *)
---
Store important learnings in Episteme.
## Usage
```bash
episteme assert "$0" "$1" "$2" --source "claude-session" --confidence 0.9
```
## Arguments
- $0: Subject (what the assertion is about)
- $1: Predicate (the relationship)
- $2: Object (the value or target)
## Examples
```bash
# Architecture decision
episteme assert "UserService" "database" "PostgreSQL" --source "arch-review"
# Pattern discovery
episteme assert "ErrorHandling" "pattern" "Result<T, AppError>" --source "code-analysis"
```
```
#### `/episteme-conflicts` - Find Conflicts
```yaml
---
name: episteme-conflicts
description: Find conflicting assertions about a subject. Use when information seems contradictory.
allowed-tools: Bash(episteme *)
---
Find conflicting knowledge that needs resolution.
## Usage
```bash
episteme conflicts --subject "$ARGUMENTS"
```
## Output
Returns pairs of conflicting assertions with:
- Both assertion details
- Confidence levels
- Sources
- Suggested resolution strategy
```
---
## Part 8: Cross-Platform Configuration
### Universal Setup
```
project/
├── AGENTS.md # Open standard, works everywhere
├── CLAUDE.md # References AGENTS.md + Claude extras
├── .gemini/
│ └── GEMINI.md # References AGENTS.md + Gemini extras
├── .claude/
│ └── skills/
│ ├── episteme-query/
│ │ └── SKILL.md
│ └── episteme-remember/
│ └── SKILL.md
└── .codex/
└── skills/ # Symlink to .claude/skills/
```
### AGENTS.md Template
```markdown
# AGENTS.md
## Overview
[Project description]
## Knowledge System
This project uses **Episteme** for persistent AI memory.
### Querying Knowledge
Before making significant changes, query existing knowledge:
```bash
episteme query --subject <topic> --lens recency
```
### Storing Learnings
After discovering important patterns or decisions:
```bash
episteme assert <subject> <predicate> <object>
```
### Resolving Conflicts
When encountering contradictory information:
```bash
episteme conflicts --subject <topic>
```
## Build & Test
[Your build commands]
## Architecture
[Architecture overview]
```
---
## References
### Agent Skills
- [Agent Skills Specification](https://agentskills.io)
- [Claude Code Skills](https://code.claude.com/docs/en/skills)
- [Codex Skills](https://developers.openai.com/codex/skills/)
### Context Files
- [AGENTS.md Specification](https://agents.md/)
- [AGENTS.md on GitHub](https://github.com/openai/codex/blob/main/docs/agents_md.md)
- [Claude Code Memory](https://code.claude.com/docs/en/memory)
### A2A Protocol
- [A2A Protocol Specification](https://a2a-protocol.org/latest/)
- [A2A GitHub](https://github.com/a2aproject/A2A)
- [Linux Foundation Announcement](https://www.linuxfoundation.org/press/linux-foundation-launches-the-agent2agent-protocol-project)
### CLI Integration
- [Using Gemini CLI as Claude Subagent](https://aicodingtools.blog/en/claude-code/gemini-cli-as-subagent-of-claude-code)
- [Claude Code + Gemini CLI Integration](https://gist.github.com/AndrewAltimit/fc5ba068b73e7002cbe4e9721cebb0f5)
### MCP (Reference)
- [MCP Specification](https://modelcontextprotocol.io/specification/2025-11-25)
- [Rust MCP SDK](https://github.com/modelcontextprotocol/rust-sdk)

106
.claude/guides/local/quality-checks.md Normal file
View File

@ -0,0 +1,106 @@
# Quality Checks & Pre-commit Hooks
**When to use:** Setting up your dev environment, understanding CI/local parity, or debugging pre-commit failures.
## Prerequisites
- Rust toolchain installed (`rustup`)
- `jscpd` for duplication checks: `npm install -g jscpd`
## Quick Start
```bash
# Run all quality checks (same as CI)
make quality
# Auto-fix formatting
make fmt
# See clippy errors
make lint
```
## Pre-commit Hook
The pre-commit hook at `.git/hooks/pre-commit` runs automatically on every commit. It:
1. Checks if any Rust files are staged
2. Runs `make quality` (format check, clippy, duplication, tests)
3. Blocks commit if any check fails
### Installing the Hook
The hook should already exist. If not:
```bash
# Copy the sample and make executable
cp .git/hooks/pre-commit.sample .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit
```
### Bypassing (Emergency Only)
```bash
# Skip pre-commit hook (logged, use sparingly)
git commit --no-verify -m "emergency fix"
```
## What Gets Checked
| Check | Command | What it catches |
|-------|---------|-----------------|
| Format | `cargo fmt --check` | Inconsistent formatting |
| Lint | `cargo clippy -- -D warnings` | Code smells, potential bugs |
| Duplication | `jscpd` | Copy-pasted code blocks |
| Tests | `cargo test` | Broken functionality |
## Enforced Lints
These are set to `deny` in `Cargo.toml` (CI will fail):
| Lint | Why |
|------|-----|
| `clippy::unwrap_used` | Panics are forbidden in production code |
| `clippy::expect_used` | Panics are forbidden in production code |
| `clippy::panic` | Explicit panics are forbidden in production code |
**Tests are exempt** via `clippy.toml`:
- `allow-unwrap-in-tests = true`
- `allow-expect-in-tests = true`
- `allow-panic-in-tests = true`
To add a new enforced lint, update `[workspace.lints.clippy]` in root `Cargo.toml`.
## Troubleshooting
### "Format check failed"
```bash
make fmt # Auto-fix
git add -u # Re-stage fixed files
```
### "Clippy warnings treated as errors"
Fix the warnings. Common ones:
```rust
// Bad: unused variable
let x = 5;
// Good: prefix with underscore
let _x = 5;
```
### "Duplication detected"
Refactor the duplicated code into a shared function or module.
## CI/Local Parity
The pre-commit hook runs `make quality`, which is the **exact same** command CI runs. If it passes locally, it passes in CI.
## Related
- [Testing Guide](./testing.md) - Running tests
- [Rust Guidelines](../backend/rust-guidelines.md) - Code standards

373
.claude/skills/episteme-usage-docs/SKILL.md Normal file
View File

@ -0,0 +1,373 @@
---
name: episteme-usage-docs
description: Create SDK integration docs showing how ADK-Go agents use Episteme. Use when adding usage examples, tool definitions, or integration patterns to use cases.
---
# Episteme Usage Documentation
## Identity
You are a developer advocate who bridges database internals and agent SDK patterns. You understand both how Episteme works internally (Lenses, Epochs, Constraints) AND how ADK-Go agents consume it (tools, callbacks, state).
## Principles
- **Show Don't Tell**: Every feature needs runnable code, not just description.
- **Agent-First Thinking**: Documentation serves agents as consumers, not Episteme as the hero.
- **Real Patterns**: Use the perspective agents' actual needs, not hypothetical scenarios.
- **Callback Is Key**: Pre-flight checks and audit trails live in callbacks, not manual queries.
## Input Context
This skill uses:
### Perspective Agents (The Users)
Read these FIRST to understand real consumer needs:
- `.claude/agents/perspective-lead-orchestrator.md` - AI coordinator routing work. Needs fast queries, confidence scores, lens selection, epoch awareness.
- `.claude/agents/perspective-implementation-agent.md` - Code writer. Needs approved patterns ONLY, deprecation warnings, lifecycle filtering.
- `.claude/agents/perspective-research-agent.md` - Data ingester. Needs contradiction storage, source attribution, confidence scoring.
- `.claude/agents/perspective-human-supervisor.md` - Reviews agent decisions. Needs audit trails, correction propagation, time-travel.
- `.claude/agents/perspective-oncall-sre.md` - 3am incident investigator. Needs <100ms queries, time-travel, trace commands.
### ADK-Go Reference
- `docs/references/go-adk/reference-guide.md` - Complete API: tool definitions, callbacks, state, streaming
- `docs/references/go-adk/research.md` - ADK-Go overview, architecture, deployment
- `docs/references/go-adk/agent-*.md` - Multi-agent patterns (sequential, parallel, loop)
### Use Cases
- `use-cases/agile-agent-team.md` - Primary integration target
- `use-cases/financial-due-diligence.md` - Secondary integration target
## Protocol
### Phase 1: Identify the Pattern
What type of integration are we documenting?
| Pattern | ADK Mechanism | Episteme Feature |
|---------|--------------|------------------|
| Query Knowledge | Tool | Lens + Lifecycle filter |
| Store Knowledge | Tool | Assert with signatures |
| Pre-flight Check | BeforeToolCallback | Lens::Constraints |
| Audit Trail | AfterModelCallback | QueryAudit |
| Error Learning | AfterToolCallback + Gardener | TrustRank back-propagation |
### Phase 2: Define the Tool Schema
Every Episteme operation needs an ADK tool definition:
```go
// Template for Episteme tools
type [Operation]Input struct {
Subject string `json:"subject" jsonschema:"[Description]"`
Predicate string `json:"predicate" jsonschema:"[Description]"`
// ... operation-specific fields
}
type [Operation]Output struct {
// ... result fields
Confidence float32 `json:"confidence"`
Provenance []SourceInfo `json:"provenance,omitempty"`
Error string `json:"error,omitempty"`
}
func [operation](ctx tool.Context, input [Operation]Input) [Operation]Output {
// 1. Call Episteme API
// 2. Return structured result
// 3. Optionally update session state
}
```
### Phase 3: Show the Callback Integration
For every tool, show how callbacks enforce safety:
```go
// BeforeToolCallback for pre-flight constraint checking
BeforeToolCallback: func(ctx agent.CallbackContext, call *tool.Call) (*tool.Call, error) {
// Check constraints BEFORE any tool executes
if needsConstraintCheck(call) {
constraints := queryConstraints(ctx, call.Context)
if violation := checkViolation(constraints, call); violation != nil {
return nil, fmt.Errorf("blocked: %s", violation)
}
}
return call, nil
},
// AfterModelCallback for audit trail
AfterModelCallback: func(ctx agent.CallbackContext, resp *model.LLMResponse) (*model.LLMResponse, error) {
// Log what the agent decided for future tracing
logDecision(ctx, resp)
return resp, nil
},
```
### Phase 4: Map to Perspective Needs
For each perspective agent, show their specific integration:
#### Lead Orchestrator
```go
// Query with lens selection and confidence threshold
result := queryEpisteme(ctx, QueryInput{
Subject: "auth/jwt",
Predicate: "signing_algorithm",
Lens: "authority",
MinConfidence: 0.8,
})
if result.Confidence < 0.8 {
return escalateToHuman(ctx, result)
}
```
#### Implementation Agent
```go
// Query approved patterns only
result := queryEpisteme(ctx, QueryInput{
Subject: "auth/jwt",
Predicate: "signing_algorithm",
Lens: "authority",
Lifecycle: "approved", // CRITICAL: filter to approved only
})
```
#### Research Agent
```go
// Store with source attribution and confidence
assertKnowledge(ctx, AssertInput{
Subject: "jwt_rotation",
Predicate: "best_practice",
Object: "rotate_daily",
SourceHash: hashURL(sourceURL),
Confidence: 0.7, // Express uncertainty
Lifecycle: "proposed", // Not approved yet
})
```
#### Human Supervisor
```go
// Time-travel query for post-mortem
result := queryEpisteme(ctx, QueryInput{
Subject: "auth/jwt",
Predicate: "signing_algorithm",
AsOf: "2024-01-15T21:00:00Z", // What was believed then?
})
// Correction with impact analysis
impact := supersede(ctx, SupersedeInput{
Hash: badAssertionHash,
Reason: "Proposal treated as approved",
Type: "Invalidate",
})
// impact.AffectedAssertions shows downstream effects
```
#### On-Call SRE
```go
// Trace command for incident investigation
traces := traceAgentQueries(ctx, TraceInput{
AgentID: "deployment-agent",
From: time.Now().Add(-6 * time.Hour),
Subject: "auth/*",
})
// traces shows: query → result → contributing assertions
```
### Phase 5: Write Complete Tool Definitions
Produce a complete file with all tool definitions:
```go
package episteme
import (
"google.golang.org/adk/tool"
"google.golang.org/adk/tool/functiontool"
)
// === QUERY TOOLS ===
type QueryInput struct {
Subject string `json:"subject" jsonschema:"Entity to query (e.g., auth/jwt)"`
Predicate string `json:"predicate" jsonschema:"Relation to query (e.g., signing_algorithm)"`
Lens string `json:"lens" jsonschema:"Resolution: consensus, authority, recency, constraints"`
Lifecycle string `json:"lifecycle,omitempty" jsonschema:"Filter: proposed, approved, deprecated"`
MinConfidence float32 `json:"min_confidence,omitempty" jsonschema:"Minimum confidence threshold"`
AsOf string `json:"as_of,omitempty" jsonschema:"Time-travel: ISO8601 timestamp"`
}
type QueryOutput struct {
Value interface{} `json:"value"`
Confidence float32 `json:"confidence"`
Lifecycle string `json:"lifecycle"`
Sources []Source `json:"sources"`
QueryID string `json:"query_id"` // For audit trail
}
// === ASSERTION TOOLS ===
type AssertInput struct {
Subject string `json:"subject" jsonschema:"Entity being described"`
Predicate string `json:"predicate" jsonschema:"Relation being asserted"`
Object interface{} `json:"object" jsonschema:"Value being claimed"`
SourceHash string `json:"source_hash" jsonschema:"BLAKE3 hash of evidence"`
Confidence float32 `json:"confidence" jsonschema:"0.0-1.0 certainty level"`
Lifecycle string `json:"lifecycle" jsonschema:"proposed, under_review, approved"`
ParentHash string `json:"parent_hash,omitempty" jsonschema:"Hash of assertion being updated"`
Meta Meta `json:"meta,omitempty" jsonschema:"Additional metadata"`
}
type Meta struct {
ForbiddenAlternative string `json:"forbidden_alternative,omitempty"`
Reason string `json:"reason,omitempty"`
}
// === CONSTRAINT TOOLS ===
type ConstraintCheckInput struct {
Context string `json:"context" jsonschema:"Domain context (e.g., python_http)"`
}
type ConstraintCheckOutput struct {
Constraints []Constraint `json:"constraints"`
}
type Constraint struct {
Subject string `json:"subject"`
MustUse string `json:"must_use,omitempty"`
Forbidden string `json:"forbidden,omitempty"`
Reason string `json:"reason"`
}
// === TRACE TOOLS (for SRE) ===
type TraceInput struct {
AgentID string `json:"agent_id" jsonschema:"Agent to trace"`
From string `json:"from" jsonschema:"Start time (ISO8601 or relative)"`
To string `json:"to,omitempty" jsonschema:"End time (default: now)"`
Subject string `json:"subject,omitempty" jsonschema:"Filter by subject pattern"`
}
type TraceOutput struct {
Queries []QueryTrace `json:"queries"`
}
type QueryTrace struct {
QueryID string `json:"query_id"`
Timestamp string `json:"timestamp"`
Subject string `json:"subject"`
Predicate string `json:"predicate"`
Lens string `json:"lens"`
Result string `json:"result"`
Confidence float32 `json:"confidence"`
Contributing []string `json:"contributing_assertions"`
}
```
### Phase 6: Step Back
Before finalizing documentation, challenge:
#### 1. The Reality Check
> "Would an agent developer actually use this?"
- Is the tool schema intuitive?
- Are the callbacks copy-paste ready?
- Does this solve a real problem from the perspective agents?
#### 2. The Completeness Check
> "Did we cover all the perspective agents' needs?"
- Lead Orchestrator: Fast queries, confidence, lens selection ✓
- Implementation Agent: Approved patterns, deprecation ✓
- Research Agent: Contradiction storage, uncertainty ✓
- Human Supervisor: Audit trail, corrections ✓
- On-Call SRE: Time-travel, tracing ✓
#### 3. The ADK Alignment Check
> "Does this match how ADK-Go actually works?"
- Tool structs use correct JSON/schema tags?
- Callbacks match ADK callback signatures?
- State access uses `ctx.Session().State()`?
#### 4. The Episteme Alignment Check
> "Does this match Episteme's actual API?"
- Lenses are real (Consensus, Authority, Recency, Constraints)?
- Lifecycle stages are correct (Proposed, Approved, Deprecated)?
- Features exist (time-travel, epochs, provenance)?
**After step back:** Revise any misalignments before finalizing.
## Do
1. Read perspective agent files to understand real needs
2. Reference ADK-Go patterns from `docs/references/go-adk/`
3. Include complete, runnable tool definitions
4. Show callback integration for every tool
5. Map every feature to a specific perspective agent's need
6. Include error handling patterns
## Do Not
1. Describe features without showing code
2. Create tools that don't match ADK-Go patterns
3. Skip the callback integration (pre-flight is critical)
4. Invent needs not documented in perspective agents
5. Use placeholder code ("// TODO: implement")
6. Forget the SRE's trace/time-travel requirements
## Decision Points
**Before writing a tool definition**: Stop. Which perspective agent needs this? Read their file first.
**Before showing a callback**: Stop. Is this BeforeToolCallback (constraint check) or AfterModelCallback (audit)? Choose correctly.
**Before claiming a feature exists**: Stop. Is this in `architecture.md`? If it's Phase 3+ in the roadmap, mark it as "planned."
## Constraints
- NEVER write tool definitions without testing against ADK-Go struct tag syntax
- NEVER skip the constraint-check callback (it's the core of preventing repeat mistakes)
- ALWAYS include `QueryID` in outputs for audit trail
- ALWAYS show the perspective agent's code, not generic examples
## Output Format
The skill produces a section like this for use cases:
```markdown
## SDK Integration: ADK-Go
### Tool Definitions
\`\`\`go
// [Complete tool struct definitions]
\`\`\`
### Callback Integration
\`\`\`go
// [BeforeToolCallback for constraints]
// [AfterModelCallback for audit]
\`\`\`
### Agent-Specific Patterns
#### [Perspective Agent Name]
[Their specific integration pattern with code]
#### [Next Perspective Agent]
...
### Complete Example: [Scenario]
\`\`\`go
// Full working example showing the complete flow
\`\`\`
```

67
.claude/skills/stemedb-core/SKILL.md
View File

@ -18,15 +18,37 @@ You are building the **Spine** of Episteme. This is the storage engine that pers
## Data Structures
### Assertion
### Assertion (sync with `crates/stemedb-core/src/types.rs`)
```rust
pub struct Assertion {
pub subject: EntityId,
pub predicate: RelationId,
pub object: ObjectValue,
pub source: SourceHash,
pub agent: AgentId,
pub timestamp: u64,
// The Fact
pub subject: EntityId, // "Tesla_Inc"
pub predicate: RelationId, // "has_revenue"
pub object: ObjectValue, // Text/Number/Boolean/Reference
// The Lineage
pub parent_hash: Option<Hash>, // Link to previous version
pub source_hash: Hash, // Evidence pointer
pub visual_hash: Option<PHash>, // pHash for image provenance
// Meta-Cognition
pub signatures: Vec<SignatureEntry>, // Multi-sig support
pub confidence: f32, // 0.0 to 1.0
pub timestamp: u64, // Unix epoch
pub vector: Option<Vec<f32>>, // Semantic embedding
}
pub struct SignatureEntry {
pub agent_id: [u8; 32], // Ed25519 Public Key
pub signature: [u8; 64], // Ed25519 Signature
pub timestamp: u64, // When signed
}
pub enum ObjectValue {
Text(String),
Number(f64),
Boolean(bool),
Reference(EntityId),
}
```
@ -40,7 +62,38 @@ pub struct Assertion {
* Use `rkyv` for zero-copy deserialization.
* Use `thiserror` for library errors.
* Validate signatures on Ingest.
* **Instrument public methods** with `#[instrument]` for observability.
## Tracing Pattern
All public methods in WAL, storage, and ingestion MUST have tracing spans:
```rust
use tracing::{debug, info, instrument};
#[instrument(skip(self, payload), fields(payload_len = payload.len()))]
pub fn append(&mut self, payload: Vec<u8>) -> Result<u64> {
// ... implementation ...
debug!(offset, "Record appended");
Ok(offset)
}
```
Guidelines:
- Use `skip(self)` to avoid noisy output
- Use `skip(payload)` or `skip(value)` for large data
- Add `fields(key_len = ..., value_len = ...)` for size visibility
- Use `debug!` for routine operations, `info!` for lifecycle events, `warn!` for recoverable issues
## Do Not
* Use `unwrap()` in core logic.
* Store large blobs in the Assertions (store pointers/hashes instead).
* Add new types without updating `ai-lookup/services/` documentation.
* Add public methods without `#[instrument]` in WAL/storage/ingest crates.
## Documentation Sync
When modifying core types:
1. Update this skill's Data Structures section to match actual code
2. Add/update entry in `ai-lookup/services/assertion.md` or `ai-lookup/services/storage.md`
3. Update `ai-lookup/index.md` if adding new concepts

9
CLAUDE.md
View File

@ -9,22 +9,28 @@ A probabilistic knowledge graph database that stores Claims, not Facts. Append-o
| If you need to... | Read this |
|-------------------|-----------|
| **Understand the vision** | [vision.md](./vision.md) |
| **See use cases** | [use-cases/README.md](./use-cases/README.md) |
| **Understand architecture** | [architecture.md](./architecture.md) |
| **See the roadmap** | [roadmap.md](./roadmap.md) |
| **Write Rust code** | [.claude/guides/backend/rust-guidelines.md](.claude/guides/backend/rust-guidelines.md) |
| **Set up local dev** | [.claude/guides/local/setup.md](.claude/guides/local/setup.md) |
| **Run tests** | [.claude/guides/local/testing.md](.claude/guides/local/testing.md) |
| **Understand quality checks** | [.claude/guides/local/quality-checks.md](.claude/guides/local/quality-checks.md) |
| **Learn about simulation** | [ai-lookup/features/simulation.md](ai-lookup/features/simulation.md) |
| **Work on storage/DAG** | Load skill: `stemedb-core` |
| **Implement a Lens** | Load skill: `stemedb-lens` |
| **Plan a milestone** | `/plan-milestone` command |
| **Integrate with AI tools** | [.claude/guides/integrations/ai-coding-assistant-integration.md](.claude/guides/integrations/ai-coding-assistant-integration.md) |
## Critical Rules
- **Append-Only:** NEVER mutate existing Assertions. Create new ones.
- **Content-Addressed:** Assertion ID = BLAKE3 hash of content.
- **No Unwrap:** NEVER use `unwrap()` or `expect()` in production code.
- **No Unwrap:** NEVER use `unwrap()` or `expect()` in production code. CI enforces via `clippy::unwrap_used` and `clippy::expect_used` at deny level.
- **Defensive Writes:** All writes go through WAL with fsync.
- **Zero-Copy:** Use `rkyv` for serialization.
- **Instrument Critical Paths:** Use `#[instrument]` on public methods in WAL, storage, and ingestion code. Include meaningful fields (key_len, payload_len, offset).
- **Document Changes:** Update `ai-lookup/` when adding new types/concepts. Keep skills in sync with code.
## Quick Reference
@ -44,6 +50,7 @@ cargo fmt --check
| Domain | Agent | When to use |
|--------|-------|-------------|
| **Product Vision** | `episteme-product-visionary` | Use cases, "why not Postgres?", product-market fit |
| General Rust | `primary-developer` | Feature implementation, refactoring |
| Code Quality | `rust-quality-engineer` | Reviews, test coverage, clippy |
| Storage | `storage-engine-architect` | WAL, LSM, crash recovery |

48
Cargo.toml
View File

@ -1,45 +1,23 @@
[workspace]
members = [
"crates/stemedb-core",
"crates/stemedb-wal",
"crates/stemedb-storage",
"crates/stemedb-ingest",
"crates/stemedb-sim",
]
resolver = "2"
# Workspace-wide lint configuration
# Crates inherit via: [lints] workspace = true
[workspace.lints.rust]
# Dead code detection
dead_code = "warn"
unused_imports = "warn"
unused_variables = "warn"
unused_mut = "warn"
unreachable_code = "warn"
[workspace.lints.clippy]
# Complexity
cognitive_complexity = "warn"
too_many_arguments = "warn"
too_many_lines = "warn"
# Code quality
clone_on_ref_ptr = "warn"
redundant_clone = "warn"
unnecessary_wraps = "warn"
useless_let_if_seq = "warn"
# Safety (production code)
unwrap_used = "warn"
expect_used = "warn"
panic = "warn"
# Style
needless_return = "warn"
redundant_else = "warn"
match_bool = "warn"
# Deny these (errors, not warnings)
dbg_macro = "deny"
[profile.release]
lto = true
codegen-units = 1
panic = "abort"
[workspace.lints.rust]
unsafe_code = "forbid"
missing_docs = "warn"
[workspace.lints.clippy]
unwrap_used = "deny"
expect_used = "deny"
panic = "deny"

65
GEMINI.md Normal file
View File

@ -0,0 +1,65 @@
# StemeDB (Episteme) Project Context
## Project Overview
**StemeDB (Episteme)** is a probabilistic, log-structured, content-addressed knowledge graph database designed as the "Cortex" for autonomous AI research agents. Unlike traditional databases that enforce a single mutable state, StemeDB preserves immutable history and resolves conflicting assertions at read-time using "Lenses."
It serves as the "Git for Truth," allowing agents to:
* **Assert** facts with cryptographic signatures and confidence scores.
* **Vote** on assertions to build consensus without lock contention.
* **Fork** reality to simulate "what-if" scenarios (Overlay Graphs).
* **Resolve** truth dynamically via lenses like Consensus, Authority, or Recency.
## Tech Stack
* **Language:** Rust (2024 edition)
* **Durability:** `stemedb-wal` (Quarantine Pattern with `fs2`, `blake3` checksums)
* **Storage:** `stemedb-storage` (`sled` embedded KV, abstracted via `KVStore` trait)
* **Serialization:** `rkyv` (Zero-copy deserialization for high performance)
* **Ingestion:** `stemedb-ingest` (Async background worker bridging WAL and Store)
* **Simulation:** `stemedb-sim` (Agent-based modeling to verify system behavior)
## Architecture
The system follows a "Spine -> Lattice -> Cortex" architecture:
1. **The Spine (Durability):**
* **Write-Ahead Log (WAL):** Append-only log with strict `fsync` guarantees.
* **Ingestor:** Background task that tails the WAL and indexes data.
* **KV Store:** Persistent storage for assertions and indexes.
2. **The Lattice (Connectivity) - *In Progress*:**
* **Ballot Box:** High-velocity vote stream.
* **Materialized Views:** Pre-computed truth states.
3. **The Cortex (Reasoning) - *Planned*:**
* **Lenses:** WASM-based filters for truth resolution.
* **SMT:** Sparse Merkle Trees for efficient branching.
## Key Files & Directories
* `stemedb/`
* `crates/`
* `stemedb-core/`: Core data structures (`Assertion`, `Vote`, `Epoch`) and types.
* `stemedb-wal/`: Durability primitives (`Journal`, `FsyncGuard`, `Record`).
* `stemedb-storage/`: Storage engine abstraction and `sled` implementation.
* `stemedb-ingest/`: Async ingestion pipeline logic.
* `stemedb-sim/`: "The Arena" simulation for end-to-end verification.
* `architecture.md`: Detailed system design and data flow.
* `roadmap.md`: Phased implementation plan and status.
* `usage.md`: Rust API usage guide and vision for agent interaction.
* `Makefile`: Build and quality automation.
## Building and Running
The project uses a `Makefile` for common tasks:
* **Build:** `make build` (Compiles the workspace)
* **Test:** `make test` (Runs unit tests across all crates)
* **Quality Check:** `make quality` (Runs fmt, strict clippy linting, duplication checks, and tests)
* **Run Simulation:** `cargo run -p stemedb-sim` (Executes the spine verification simulation)
* **Format:** `make fmt` (Auto-formats code)
## Development Conventions
* **Strict Quality:** `make quality` must pass before committing.
* No `unwrap()` or `expect()` in production code (enforced by clippy).
* Zero warnings allowed.
* Missing documentation is a hard error.
* **Testing:** Every crate must have unit tests. The `stemedb-sim` crate serves as the integration test suite.
* **Architecture:** Follow the "Defensive by Default" philosophy. Durability > Speed > Features.

57
ai-lookup/features/gardener.md Normal file
View File

@ -0,0 +1,57 @@
# The Gardener (TrustRank Back-Propagation)
> **Quick Ref:** Background worker that penalizes agents when corrections are made
## The Problem
Agents are stateless. When an agent makes a mistake (uses `requests` instead of `axios`), you correct it. But the agent doesn't "learn"—it might make the same mistake next session.
Current training uses "Golden Trajectories" (perfect examples). Mistakes are discarded, so agents never learn "don't do X because it fails."
## The Solution
The Gardener is a background worker that:
1. Detects when a user correction supersedes an agent assertion
2. Calculates the "delta" (how wrong the agent was)
3. Back-propagates the error to the agent's TrustRank
```rust
struct GardenerJob {
pub agent_id: AgentId,
pub topic: String, // e.g., "http_libraries"
pub prediction: Value, // What agent said
pub ground_truth: Value, // What was correct
pub delta: f32, // How wrong (0.0 to -1.0)
}
// Example:
// Agent asserted "requests" (confidence 0.8)
// User asserted "axios" (confidence 1.0)
// Gardener calculates: delta = -0.3
// Agent's TrustRank for topic "http_libraries" drops by 0.15
```
## Effects
**Next time this agent predicts an HTTP library:**
1. Its confidence is mathematically penalized
2. It's forced to look for external verification
3. Or the system routes to a different agent with higher TrustRank
## Resurrection Mechanics
When a constraint is queried and used successfully:
- `last_verified` updates to NOW()
- Confidence decay resets
- Assertion stays in "hot path"
When NOT used in 6 months:
- Confidence decays toward 0
- Moves to "cold store"
- Still queryable for audit
## Related
- [Negative Constraints](../services/lifecycle.md) - What gets stored
- [Lens::Constraints](../services/lens.md) - How it's retrieved
- [use-cases/agile-agent-team.md](../../use-cases/agile-agent-team.md#feature-6) - Full workflow

82
ai-lookup/features/query-audit.md Normal file
View File

@ -0,0 +1,82 @@
# Query Audit Trail
> **Quick Ref:** Every query is logged with provenance for incident investigation
## The Problem
At 3am, production is broken. An agent deployed wrong config. The SRE needs to know: What did the agent query? What result did it get? What assertions contributed?
Postgres query logs show SQL, not semantic meaning.
## The Solution
```rust
struct QueryAudit {
pub query_id: Hash,
pub agent_id: AgentId,
pub timestamp: u64,
pub subject: EntityId,
pub predicate: RelationId,
pub lens: LensType,
pub lifecycle_filter: Option<LifecycleStage>,
pub result_hash: Hash,
pub result_confidence: f32,
pub contributing_assertions: Vec<ContributingAssertion>,
}
struct ContributingAssertion {
pub assertion_hash: Hash,
pub weight: f32, // How much it influenced result
pub source_hash: Hash, // Original evidence
}
```
## API
```bash
# What queries did this agent run?
GET /audit/queries?agent=deployment-agent&from=2024-01-15T20:00:00Z
# Trace command for incident investigation
episteme trace --agent deployment-agent \
--time "6 hours ago" \
--subject "auth/*"
```
## Response Format
```json
{
"query_id": "q_7f3a2b...",
"timestamp": "2024-01-15T21:03:47Z",
"subject": "auth/jwt",
"predicate": "signing_algorithm",
"lens": "authority",
"lifecycle_filter": null,
"result": {
"value": "ES256",
"confidence": 0.87
},
"contributing_assertions": [
{
"hash": "rfc_2024_001...",
"lifecycle": "Proposed",
"weight": 0.9,
"source": "security-rfc-2024.md"
}
]
}
```
## Latency Requirements (from user research)
| Query Type | Target Latency |
|------------|---------------|
| Point query (current) | < 100ms |
| Time-travel query | < 500ms |
| Audit trace | < 2s |
| Full provenance chain | < 5s |
## Origin
This feature emerged from SRE perspective interviews (see `.claude/agents/perspective-oncall-sre.md`). Core need: "I need to trace from agent decision → query → assertions in under 10 minutes."

54
ai-lookup/features/simulation.md Normal file
View File

@ -0,0 +1,54 @@
# Simulation System ("The Infinite Game")
**Last Updated:** 2026-01-31
**Confidence:** High
## Summary
The Simulation is an Agent-Based Modeling (ABM) environment that validates StemeDB under emergent, adversarial, and evolutionary pressure. It simulates a society of AI agents with conflicting goals living within the knowledge graph.
**Key Facts:**
- Codename: "The Arena"
- Purpose: Integration testing via societal stress tests, not unit tests
- Architecture: `stemedb-sim` binary orchestrating agent swarms via tokio
- Agents communicate **only** through StemeDB reads/writes
**File Pointer:** `/simulation-vision.md` (full vision document)
## Agent Personas
| Persona | Goal | Behavior |
|---------|------|----------|
| Scientist | Converge on truth | High-confidence assertions, cites sources |
| Troll | Sow chaos | Low-confidence contradictions, frequent forks |
| Believer | Amplify consensus | Trusts high-reputation agents |
| Skeptic | Find variance | Reduces confidence of unverified claims |
| Historian | Preserve context | Resurrects dormant truths with new evidence |
## The Gameplay Loop
1. **Assertion**: Agent reads ground truth, creates assertion
2. **Fork**: Adversarial agent forks reality with contradiction
3. **Lens Resolution**: Query agent applies Lens (e.g., Consensus)
4. **Reputation Update**: TrustRank adjusts agent reputations
5. **Decay**: Unverified assertions fade via Dormancy Protocol
## Success Criteria
- Truth survives: High-reputation assertions outlive spam
- Lenses work: Consensus lens filters Troll noise
- Performance: 1000 concurrent agents without locking
- Emergence: Trust clusters form without hardcoded rules
## Metrics
Tracked via Prometheus/Grafana:
- `global_truth_convergence` - Entropy of the graph
- `agent_reputation_distribution` - Reputation spread
- `fork_depth_max` - Deepest branch depth
## Related Topics
- [TrustRank](./trustrank.md) - Reputation algorithm
- [Branching](./branching.md) - Fork mechanics
- [Lens System](../services/lens.md) - Query resolution

15
ai-lookup/index.md
View File

@ -7,13 +7,16 @@ Token-efficient fact storage for StemeDB. Query these for quick context without
| Topic | File | Confidence | Updated | Summary |
|-------|------|------------|---------|---------|
| Assertion | `services/assertion.md` | High | 2025-01-31 | Core data structure for all claims |
| Ingestor | `services/ingestor.md` | High | 2026-01-31 | WAL-to-KV background worker |
| Lens | `services/lens.md` | High | 2025-01-31 | Read-time resolution strategies |
| Lifecycle | `services/lifecycle.md` | High | 2026-01-31 | Proposed/Approved state machine |
| Storage | `services/storage.md` | High | 2025-01-31 | KV layout and write path |
## Patterns
| Topic | File | Confidence | Updated | Summary |
|-------|------|------------|---------|---------|
| ADK-Go Integration | `patterns/adk-integration.md` | High | 2026-01-31 | Tool definitions and callbacks for agents |
| Content-Addressing | `patterns/content-addressing.md` | High | 2025-01-31 | BLAKE3 hashing for immutability |
| Error Handling | `patterns/error-handling.md` | High | 2025-01-31 | thiserror + context pattern |
@ -22,4 +25,16 @@ Token-efficient fact storage for StemeDB. Query these for quick context without
| Topic | File | Confidence | Updated | Summary |
|-------|------|------------|---------|---------|
| Branching | `features/branching.md` | Medium | 2025-01-31 | "Fork Reality" overlay graphs |
| Gardener | `features/gardener.md` | High | 2026-01-31 | TrustRank back-propagation on errors |
| Query Audit | `features/query-audit.md` | High | 2026-01-31 | Trace agent decisions for debugging |
| TrustRank | `features/trustrank.md` | Medium | 2025-01-31 | Agent reputation system |
| Simulation | `features/simulation.md` | High | 2026-01-31 | Agent-based modeling for validation |
## Use Cases
See [use-cases/README.md](../use-cases/README.md) for production scenarios with Postgres Test analysis.
| Use Case | File | Pillars | Summary |
|----------|------|---------|---------|
| Financial Due Diligence | `../use-cases/financial-due-diligence.md` | All Four | M&A investigation with contradictions |
| Agile AI Agent Team | `../use-cases/agile-agent-team.md` | All Four | Agent coordination with lifecycle stages |

328
ai-lookup/patterns/adk-integration.md Normal file
View File

@ -0,0 +1,328 @@
# ADK-Go Integration Pattern
## Summary
Tool definitions and callback patterns for Google ADK-Go agents consuming Episteme. Covers query, assert, constraint check, and trace operations with proper struct tags and callback signatures.
## Core Mechanism
ADK-Go agents interact with Episteme through:
1. **Tools** - Structured operations (query, assert, trace) defined as Go structs
2. **Callbacks** - Hooks for pre-flight checks and audit logging
3. **State** - Session state for agent-to-agent communication
## Tool Definition Pattern
All Episteme tools follow this structure:
```go
import (
"google.golang.org/adk/tool"
"google.golang.org/adk/tool/functiontool"
)
type [Operation]Input struct {
// Required fields (no omitempty)
Subject string `json:"subject" jsonschema:"Entity to query (e.g., auth/jwt)"`
Predicate string `json:"predicate" jsonschema:"Relation to query (e.g., signing_algorithm)"`
// Optional fields (with omitempty)
Lens string `json:"lens,omitempty" jsonschema:"Resolution: consensus, authority, recency, constraints"`
}
type [Operation]Output struct {
Value interface{} `json:"value"`
Confidence float32 `json:"confidence"`
QueryID string `json:"query_id"` // CRITICAL: for audit trail
Error string `json:"error,omitempty"`
}
func [operation](ctx tool.Context, input [Operation]Input) [Operation]Output {
// 1. Call Episteme API
// 2. Return structured result with QueryID
}
// Register with functiontool.New()
tool, err := functiontool.New(
functiontool.Config{
Name: "episteme_query",
Description: "Query Episteme knowledge graph",
},
queryEpisteme,
)
```
## Standard Tools
### Query Tool
```go
type QueryInput struct {
Subject string `json:"subject" jsonschema:"Entity to query (e.g., auth/jwt)"`
Predicate string `json:"predicate" jsonschema:"Relation to query (e.g., signing_algorithm)"`
Lens string `json:"lens,omitempty" jsonschema:"Resolution: consensus, authority, recency, constraints"`
Lifecycle string `json:"lifecycle,omitempty" jsonschema:"Filter: proposed, approved, deprecated"`
MinConfidence float32 `json:"min_confidence,omitempty" jsonschema:"Minimum confidence threshold (0.0-1.0)"`
AsOf string `json:"as_of,omitempty" jsonschema:"Time-travel: ISO8601 timestamp"`
}
type QueryOutput struct {
Value interface{} `json:"value"`
Confidence float32 `json:"confidence"`
Lifecycle string `json:"lifecycle"`
Sources []Source `json:"sources"`
QueryID string `json:"query_id"`
}
```
### Assert Tool
```go
type AssertInput struct {
Subject string `json:"subject" jsonschema:"Entity being described"`
Predicate string `json:"predicate" jsonschema:"Relation being asserted"`
Object interface{} `json:"object" jsonschema:"Value being claimed"`
SourceHash string `json:"source_hash" jsonschema:"BLAKE3 hash of evidence"`
Confidence float32 `json:"confidence" jsonschema:"Certainty level (0.0-1.0)"`
Lifecycle string `json:"lifecycle,omitempty" jsonschema:"proposed, under_review, approved"`
ParentHash string `json:"parent_hash,omitempty" jsonschema:"Hash of assertion being updated"`
Meta *AssertMeta `json:"meta,omitempty" jsonschema:"Additional metadata"`
}
type AssertMeta struct {
ForbiddenAlternative string `json:"forbidden_alternative,omitempty"`
Reason string `json:"reason,omitempty"`
}
type AssertOutput struct {
Hash string `json:"hash"`
Success bool `json:"success"`
Error string `json:"error,omitempty"`
}
```
### Constraint Check Tool
```go
type ConstraintCheckInput struct {
Context string `json:"context" jsonschema:"Domain context (e.g., python_http, auth_jwt)"`
}
type ConstraintCheckOutput struct {
Constraints []Constraint `json:"constraints"`
}
type Constraint struct {
Subject string `json:"subject"`
MustUse string `json:"must_use,omitempty"`
Forbidden string `json:"forbidden,omitempty"`
Reason string `json:"reason"`
}
```
### Trace Tool (SRE)
```go
type TraceInput struct {
AgentID string `json:"agent_id" jsonschema:"Agent to trace"`
From string `json:"from" jsonschema:"Start time (ISO8601 or relative like -6h)"`
To string `json:"to,omitempty" jsonschema:"End time (default: now)"`
Subject string `json:"subject,omitempty" jsonschema:"Filter by subject pattern (e.g., auth/*)"`
}
type TraceOutput struct {
Queries []QueryTrace `json:"queries"`
}
type QueryTrace struct {
QueryID string `json:"query_id"`
Timestamp string `json:"timestamp"`
Subject string `json:"subject"`
Predicate string `json:"predicate"`
Lens string `json:"lens"`
Result string `json:"result"`
Confidence float32 `json:"confidence"`
Contributing []string `json:"contributing_assertions"`
}
```
## Callback Integration
### BeforeToolCallback - Constraint Pre-flight
Critical for preventing repeat mistakes:
```go
agent, err := llmagent.New(llmagent.Config{
Name: "implementation_agent",
Model: model,
Tools: []tool.Tool{queryTool, assertTool},
BeforeToolCallback: func(ctx agent.CallbackContext, call *tool.Call) (*tool.Call, error) {
// Check constraints before any tool that could use wrong patterns
if needsConstraintCheck(call) {
constraints := checkConstraints(ctx, extractContext(call))
for _, c := range constraints {
if violates(call, c) {
return nil, fmt.Errorf("blocked: %s is forbidden - %s",
c.Forbidden, c.Reason)
}
}
}
return call, nil
},
})
```
### AfterModelCallback - Audit Trail
Log agent decisions for debugging:
```go
AfterModelCallback: func(ctx agent.CallbackContext, resp *model.LLMResponse) (*model.LLMResponse, error) {
// Log what the agent decided for future tracing
decision := extractDecision(resp)
logToEpisteme(ctx, AuditEntry{
AgentID: ctx.Agent().Name(),
Timestamp: time.Now(),
Decision: decision,
Context: ctx.Session().State(),
})
return resp, nil
},
```
### AfterToolCallback - Error Learning
Trigger Gardener when tools fail:
```go
AfterToolCallback: func(ctx agent.CallbackContext, call *tool.Call, result *tool.Result) (*tool.Result, error) {
if result.Error != nil {
// Notify Gardener of the failure for TrustRank back-propagation
notifyGardener(ctx, GardenerEvent{
AgentID: ctx.Agent().Name(),
ToolCall: call,
Error: result.Error,
QueryID: extractQueryID(call),
})
}
return result, nil
},
```
## Agent-Specific Patterns
### Lead Orchestrator
Fast queries with confidence thresholds:
```go
result := queryEpisteme(ctx, QueryInput{
Subject: "auth/jwt",
Predicate: "signing_algorithm",
Lens: "authority",
MinConfidence: 0.8,
})
if result.Confidence < 0.8 {
// Escalate to human or research agent
return escalate(ctx, result)
}
```
### Implementation Agent
Approved patterns only:
```go
result := queryEpisteme(ctx, QueryInput{
Subject: "auth/jwt",
Predicate: "signing_algorithm",
Lens: "authority",
Lifecycle: "approved", // CRITICAL: never use proposed patterns
})
```
### Research Agent
Store with uncertainty and contradictions:
```go
assertKnowledge(ctx, AssertInput{
Subject: "jwt_rotation",
Predicate: "best_practice",
Object: "rotate_daily",
SourceHash: hashURL(sourceURL),
Confidence: 0.7, // Express uncertainty
Lifecycle: "proposed",
})
// Store contradicting information - don't flatten
assertKnowledge(ctx, AssertInput{
Subject: "jwt_rotation",
Predicate: "best_practice",
Object: "rotate_weekly",
SourceHash: hashURL(differentSourceURL),
Confidence: 0.6,
Lifecycle: "proposed",
})
```
### Human Supervisor
Time-travel and corrections:
```go
// What was believed during the incident?
result := queryEpisteme(ctx, QueryInput{
Subject: "auth/jwt",
Predicate: "signing_algorithm",
AsOf: "2024-01-15T21:00:00Z",
})
// Correct the record
supersede(ctx, SupersedeInput{
Hash: badAssertionHash,
Reason: "Proposal treated as approved - was never reviewed",
Type: "Invalidate",
})
```
### On-Call SRE
Trace agent decisions:
```go
traces := traceAgentQueries(ctx, TraceInput{
AgentID: "deployment-agent",
From: "-6h",
Subject: "auth/*",
})
for _, t := range traces.Queries {
fmt.Printf("[%s] %s/%s via %s -> %s (%.2f)\n",
t.Timestamp, t.Subject, t.Predicate,
t.Lens, t.Result, t.Confidence)
fmt.Printf(" Contributing: %v\n", t.Contributing)
}
```
## State vs Episteme
| Use | Mechanism |
|-----|-----------|
| Agent-to-agent handoff (same session) | Session State + OutputKey |
| Persistent organizational knowledge | Episteme Assert |
| What was decided in this conversation | Session State |
| What should all agents know forever | Episteme Assert |
| Temporary working data | `temp:` prefixed state keys |
| Audit trail | Episteme QueryAudit |
## Related
- [services/assertion.md](../services/assertion.md) - Core assertion structure
- [services/lens.md](../services/lens.md) - Lens resolution strategies
- [services/lifecycle.md](../services/lifecycle.md) - Lifecycle stage filtering
- [features/gardener.md](../features/gardener.md) - Error back-propagation
- [features/query-audit.md](../features/query-audit.md) - Trace and debug

83
ai-lookup/services/ingestor.md Normal file
View File

@ -0,0 +1,83 @@
# Ingestor Service
> **Crate:** `stemedb-ingest`
> **Status:** Implemented (Phase 1)
## Purpose
The Ingestor is the background worker that bridges the Write-Ahead Log (WAL) to the KV storage engine. It continuously tails the WAL and persists records to sled using content-addressed keys.
## Architecture
```
[WAL Journal] ---> [IngestWorker] ---> [KVStore (sled)]
|
v
[Subject Index]
```
## Key Components
### RecordType
Discriminator for WAL payloads (8-byte aligned header):
- `Assertion = 0` - Knowledge claims
- `Vote = 1` - Consensus votes
- `Epoch = 2` - Paradigm definitions
### Storage Layout
| Key Pattern | Value | Description |
|-------------|-------|-------------|
| `H:{blake3_hash}` | Serialized Assertion | Content-addressed assertion store |
| `V:{assertion_hash}:{vote_hash}` | Serialized Vote | Votes on assertions |
| `E:{epoch_id_hex}` | Serialized Epoch | Epoch definitions |
| `S:{subject}` | BLAKE3 hash bytes | Subject adjacency index |
## Usage
```rust
use stemedb_ingest::{Ingestor, serialize_assertion};
use stemedb_wal::Journal;
use stemedb_storage::SledStore;
// Create components
let journal = Arc::new(Mutex::new(Journal::open("./wal")?));
let store = Arc::new(SledStore::open("./db")?);
// Create and start ingestor
let mut ingestor = Ingestor::new(journal.clone(), store);
ingestor.start(); // Spawns background task
// Write to WAL (records will be ingested automatically)
let assertion = Assertion { ... };
let payload = serialize_assertion(&assertion)?;
journal.lock().await.append(payload)?;
```
## Serialization
Records are serialized with an 8-byte header to maintain rkyv alignment:
```
[type: u8][padding: 7 bytes][rkyv payload...]
```
Helper functions:
- `serialize_assertion(&Assertion) -> Result<Vec<u8>>`
- `serialize_vote(&Vote) -> Result<Vec<u8>>`
- `serialize_epoch(&Epoch) -> Result<Vec<u8>>`
## Testing
The ingestor has integration tests covering:
- Single assertion ingestion
- Vote ingestion
- Epoch ingestion
- Multiple record processing
- Subject index creation
## Related
- [Storage Service](./storage.md) - KVStore trait and SledStore
- [Content Addressing](../patterns/content-addressing.md) - BLAKE3 hashing

23
ai-lookup/services/lens.md
View File

@ -1,6 +1,6 @@
# Lens
**Last Updated:** 2025-01-31
**Last Updated:** 2026-01-31
**Confidence:** High
## Summary
@ -31,6 +31,27 @@ pub trait Lens {
| Consensus | Highest vote count | Democratic truth |
| Authority | Weighted by agent reputation | Expert truth |
| Skeptic | Returns variance/conflict | Finding controversy |
| EpochAware | Filters superseded epochs first | Paradigm-safe queries |
| Constraints | Returns `must_use`/`forbidden` predicates | Pre-flight checks |
## Lens::Constraints (Pre-Flight Check)
Special lens for agent safety. Returns rules, not facts.
```
GET /query?context=python_http&lens=constraints
-> Returns:
{
"constraints": [
{ "must_use": "axios", "forbidden": "requests", "reason": "User correction" }
]
}
```
**Origin:** Solves the "Optimization Conflict" where agents forget corrections. Acts as a compiler error for agent intent.
See [agile-agent-team.md](../../use-cases/agile-agent-team.md#feature-6-persistent-learning-negative-constraints--the-gardener) for full explanation.
## Query Flow

71
ai-lookup/services/lifecycle.md Normal file
View File

@ -0,0 +1,71 @@
# Lifecycle Stages
> **Quick Ref:** Assertions have lifecycle state: Proposed → UnderReview → Approved | Deprecated | Rejected
## The Problem
AI agents can't distinguish proposals from decisions. An RFC saying "we should use ES256" looks the same as an approved policy saying "use ES256." Agents query, get proposals, treat them as truth.
## The Solution
```rust
enum LifecycleStage {
Proposed, // Idea, RFC, suggestion
UnderReview, // Being evaluated
Approved, // Accepted as current truth
Deprecated, // Was true, now superseded
Rejected, // Considered and declined
}
struct Assertion {
// ... existing fields
pub lifecycle: LifecycleStage,
}
```
## Query Integration
```
# Only approved patterns (safe for implementation)
GET /query?subject=auth/jwt&predicate=algo&lifecycle=approved
# All stages (for research/context)
GET /query?subject=auth/jwt&predicate=algo&lifecycle=any
# Show proposals needing review
GET /query?predicate=*&lifecycle=under_review
```
## State Transitions
```
Proposed → UnderReview → Approved → Deprecated
↘ Rejected
```
Transitions are new assertions, not mutations:
```
POST /assert
{
"subject": "auth/jwt",
"predicate": "signing_algorithm",
"object": { "Text": "ES256" },
"lifecycle": "Approved",
"parent_hash": "proposal_hash...", # Links to original proposal
"signatures": [{ "agent_id": "security_lead", ... }]
}
```
## Lens Interaction
| Lens | Lifecycle Behavior |
|------|-------------------|
| Recency | Returns most recent matching lifecycle filter |
| Consensus | Counts votes within lifecycle stage |
| Authority | Weights by signer reputation, respects lifecycle |
| EpochAware | Filters by epoch AND lifecycle |
## Origin
This feature emerged from user research (see `.claude/agents/perspective-*.md`). The Implementation Agent's core need: "If proposed and approved look the same, I can't use this."

190
architecture.md
View File

@ -1,157 +1,151 @@
# Episteme (StemeDB) Architecture
> **Design Philosophy:** Immutable History, Probabilistic Resolution.
> **Status:** Draft Spec v0.1
> **Design Philosophy:** Immutable History, Probabilistic Resolution, Materialized Speed.
> **Status:** Draft Spec v1.0
## 1. System Overview
Episteme is a **Log-Structured, Content-Addressed Knowledge Graph**. Unlike traditional databases that mutate state in place, Episteme appends **Assertions** to an immutable ledger (Merkle DAG). State resolution happens at read-time via **Lenses**.
Episteme is a **Log-Structured, Content-Addressed Knowledge Graph**. Unlike traditional databases that mutate state in place, Episteme appends **Assertions** to an immutable ledger (Merkle DAG). State resolution happens via **Lenses**.
To solve the O(N) read latency of conflict resolution, Episteme employs a **Materialized View** layer that pre-calculates the "Current Truth" for standard lenses.
### High-Level Data Flow
```ascii
[Writer Agent] [Reader Agent]
│ ▲
│ (1) Sign & │ (5) Deterministic Answer
│ Propose │ (Confidence: 0.92)
│ (1) Sign & │ (6) Sub-millisecond Answer
│ Propose │ (Pre-computed)
▼ │
┌────────────┐ ┌────────────┐
│ Ingestion │ │ Resolution │
│ Gateway │ │ Engine │
└─────┬──────┘ └─────┬──────┘
│ (2) Append │ (4) Apply Lens (Filter/Rank)
│ to WAL │
│ (2) Append │ (5) Read Materialized View
│ to Ballot │ OR Apply Custom Lens
▼ │
┌────────────┐ ┌────────────┐
│ Quarantine │ │ Indexing │
│ Journal │──────► Service │
└────────────┘ (3) └────────────┘
(Durability) (Graph/Vector)
└─────┬──────┘ (3) └─────┬──────┘
│ │ (4) Compaction & Materialization
▼ ▼
┌────────────┐ ┌────────────┐
│ Job Manager│ │ Materialized│
└────────────┘ │ Views │
(TAN Meter) └────────────┘
```
---
## 2. Core Data Structures
### 2.1. The Atomic Unit: `Assertion`
Everything in Episteme is an Assertion. There are no "Tables."
### 2.1. The Atomic Unit: `Assertion` (The Candidate)
Assertions are proposals of truth. They are immutable.
```rust
// The immutable payload (Content-Addressed by Hash)
struct Assertion {
// 1. The Triple (The Fact)
pub subject: EntityId, // "Tesla_Inc"
pub predicate: RelationId, // "has_revenue"
pub object: Value, // Variant: Float(10.5B), String("Musk"), Ref(EntityId)
// 2. The Lineage (The Chain)
pub parent_hash: Option<Hash>, // If modifying a previous claim (Forking)
pub source_hash: Hash, // Evidence pointer (PDF/Log hash)
// 3. The Meta-Cognition (The Weight)
pub agent_id: PublicKey, // Ed25519 signature
pub confidence: f32, // 0.0 - 1.0 (Subjective certainty)
pub timestamp: u64, // Wall clock time
pub vector: Option<Vec<f32>>,// Semantic embedding (for fuzzy recall)
pub subject: EntityId,
pub predicate: RelationId,
pub object: Value,
pub epoch: Option<EpochId>,
pub agent_id: PublicKey, // The Proposer
pub timestamp: u64,
// ... lineage and vector fields ...
}
```
### 2.2. The Storage Layout (LSM Tree)
We use a Key-Value store (e.g., `sled` or `RocksDB`) to persist the DAG.
### 2.2. The Ballot Box: `Vote` (The High-Velocity Stream)
To prevent lock contention on Assertions, Agents write **Votes** to a separate high-velocity log.
```rust
struct Vote {
pub assertion_hash: Hash, // What are we voting on?
pub agent_id: PublicKey, // Who is voting?
pub weight: f32, // 0.0 - 1.0 (Confidence)
pub signature: Signature, // Cryptographic proof
pub timestamp: u64,
}
```
### 2.3. The Storage Layout (LSM Tree)
| Key | Value | Purpose |
| :--- | :--- | :--- |
| `H:{Hash}` | `Serialized<Assertion>` | Main content store |
| `S:{Subject}` | `List<Hash>` | Subject-to-Claims Index |
| `SP:{Subject}:{Predicate}` | `List<Hash>` | Exact Triple Index |
| `A:{AgentID}` | `ReputationScore` | TrustRank storage |
| `H:{Hash}` | `Assertion` | Immutable Content Store |
| `V:{Hash}` | `List<Vote>` | The Ballot Box (Append-only) |
| `MV:{Subject}:{Predicate}` | `Assertion` | **Materialized View** (The "Winner") |
| `S:{Subject}` | `List<Hash>` | Adjacency Index |
---
## 3. The Write Path (The Spine)
## 3. The Write Path (The Ballot Box)
Episteme follows the **Quarantine Pattern** for durability.
1. **Receive:** Agent submits a signed `Assertion`.
2. **Verify:** Check signature validity and structure.
3. **Journal:** Write to `episteme-wal` (Append-only file, fsync immediate).
4. **Acknowledge:** Return `202 Accepted` to Agent with the new `Hash`.
5. **Index (Async):** A background worker tails the WAL:
* Deserializes the Assertion.
* Updates the `H:{Hash}` store.
* Appends `Hash` to the `S:{Subject}` adjacency list.
* Updates HNSW vector index (if vector present).
1. **Ingest:** Agents submit `Assertions` or `Votes`.
2. **Journal:** Written to `episteme-wal`.
3. **Ballot Box:** Votes are appended to the `V:{Hash}` stream.
4. **Compactor (Async):** A background worker aggregates Votes + TrustRank to update the `MV:{Subject}:{Predicate}` key.
* This ensures that Read queries (`GET /query`) are O(1) lookups on the Materialized View, not O(N) calculations.
---
## 4. The Read Path (The Cortex)
Reading is where Episteme differs from every other DB. A Read is a **Compute Operation**.
**Fast Path (Standard Lenses):**
* Query: `GET /query?lens=Consensus`
* Action: `GET MV:{Subject}:{Predicate}`
* Cost: **O(1)**. Low latency.
**Query:** `GET(Subject="Tesla", Predicate="Revenue", Lens="Consensus")`
**Slow Path (Custom/Skeptic Lenses):**
* Query: `GET /query?lens=Skeptic`
* Action: Gather all candidates + votes, compute variance on the fly.
* Cost: **O(N)**. High latency, used for analysis/debugging.
1. **Gather:** Lookup `SP:Tesla:Revenue`. Get list of candidate Hashes: `[H1, H2, H3, H4]`.
2. **Hydrate:** Fetch full Assertions for each Hash.
3. **Resolve (The Lens):** Pass candidates through the Lens pipeline.
### The Lens Pipeline (Rust Trait)
```rust
trait Lens {
fn resolve(&self, candidates: Vec<Assertion>, context: Context) -> LensResult;
}
// Example: Consensus Lens Logic
// 1. Group candidates by Object value (clustering).
// 2. Sum the TrustRank of Agents in each cluster.
// 3. Return the cluster with highest weighted mass.
```
### Standard Lenses
* **Consensus:** Highest cluster density.
* **Authority:** Filter by Reputation.
* **Recency:** Last Writer Wins.
* **Skeptic:** Returns variance/conflict metrics.
* **EpochAware:** Validates against current paradigm.
* **Constraints:** Returns all `must_use`/`forbidden` assertions for a context. Acts as a "Pre-Flight Check" to solve the Optimization Conflict.
---
## 5. Advanced Mechanics
## 5. The Meter (Economic Safety)
### 5.1. Forking Reality (Branching)
Branching is handled via **Overlay Graphs**.
* A `Branch` is simply a lightweight index (Map) of `Hash -> Assertion`.
* **Write to Branch:** Assertions are stored in the Branch's ephemeral index, not the Global DAG.
* **Read from Branch:** The Query Engine checks the Branch index *first*, then falls back to Global (Overlay pattern).
* **Merge:** Commit the Branch's unique assertions to the Global WAL.
### 5.2. TrustRank (Reputation)
Background worker (`episteme-gardener`) runs periodically:
1. Identifies "Settled Facts" (Assertions with >99% consensus over T time).
2. Rewards Agents who claimed these facts *early*.
3. Punishes Agents who claimed the opposite.
4. Updates `A:{AgentID}` reputation scores.
To prevent infinite loops, the Job Manager enforces **Temporal Advantage Normalization (TAN)**.
* **Budgeting:** Every Job must declare a `max_cost`.
* **Throttling:** Forking Reality or Deep Recursion is rejected if `current_cost + projected_cost > max_cost`.
---
## 6. Implementation Roadmap
## 6. The Simulator (Mid-Training Pipeline)
### Phase 1: The Skeleton (MVP)
The system continuously exports data to train the next generation of Agents.
* **Negative Samples:** High-confidence assertions that were later superseded (Failures).
* **Golden Paths:** Branches that successfully merged to Main (Successes).
* **Format:** Exported as HuggingFace-compatible datasets for LoRA fine-tuning.
---
## 7. Implementation Roadmap
### Phase 1: The Spine (Foundation)
* [ ] Reuse `quarantine-journal` pattern for WAL.
* [ ] Implement `Assertion` struct and serialization (`rkyv`).
* [ ] Implement `Assertion`, `Epoch`, and **`Vote`** structs.
* [ ] Basic `sled` storage backend.
* [ ] Single Lens: `Recency` (Last writer wins logic).
### Phase 2: The Graph
* [ ] Implement `Subject -> Hash` indexing.
* [ ] Implement `Consensus` Lens (Simple voting).
* [ ] Basic HTTP API (`POST /assert`, `GET /query`).
### Phase 2: The Lattice (Connectivity)
* [ ] **The Ballot Box**: Implement separate Vote storage stream.
* [ ] **Materializer**: Implement background worker to maintain `MV` keys.
* [ ] **The Meter**: Implement Budget/TAN middleware in Job Manager.
* [ ] **Agent Wallet**: Sidecar for key management/signing.
### Phase 3: The Cortex
* [ ] Branching support (Context/Session IDs).
* [ ] Vector search integration (`lanms` or `hnsw-rs`).
* [ ] TrustRank basics.
### Phase 3: The Cortex (Reasoning)
* [ ] SMT Backend & Branching.
* [ ] Vector Search.
* [ ] **Lens: Constraints**: Implement the pre-flight check logic.
---
## 7. Technology Stack
* **Language:** Rust (2024 edition)
* **WAL:** `quarantine-journal` (Local crate or pattern)
* **KV Store:** `sled` (Embedded, pure Rust) or `rocksdb` binding.
* **Serialization:** `rkyv` (Zero-copy deserialization).
* **API:** `axum` + `tower`.
* **Hashing:** `blake3` (Fast, secure).
### Phase 4: The Hive (Learning)
* [ ] **The Simulator**: Log exporter pipeline.
* [ ] TrustRank Learning Loop.

12
crates/stemedb-core/Cargo.toml
View File

@ -9,8 +9,18 @@ description = "Core logic for Episteme (StemeDB)"
workspace = true
[dependencies]
# Dependencies will be added as we implement features
serde = { version = "1.0", features = ["derive"] }
thiserror = "1.0"
tracing = "0.1"
tracing-subscriber = "0.3"
# Serialization & Hashing
rkyv = { version = "0.7", features = ["validation", "strict"] }
blake3 = "1.5"
bytecheck = "0.6" # Required for rkyv validation
# Cryptography
ed25519-dalek = { version = "2.1", features = ["rand_core"] }
# Visual Provenance
image_hasher = "3.1"

110
crates/stemedb-core/src/lib.rs
View File

@ -1,3 +1,12 @@
//! Core logic and types for Episteme (StemeDB).
//!
//! This crate defines the fundamental data structures like `Assertion`,
//! `SignatureEntry`, and the core traits for the knowledge graph.
/// Core data types for StemeDB assertions and signatures.
pub mod types;
/// A simple hello world function for testing the core crate.
pub fn hello_world() -> String {
"Hello from Episteme Core!".to_string()
}
@ -5,9 +14,110 @@ pub fn hello_world() -> String {
#[cfg(test)]
mod tests {
use super::*;
use crate::types::{Assertion, Epoch, ObjectValue, SignatureEntry, SupersessionType, Vote};
use rkyv::check_archived_root;
use rkyv::ser::serializers::AllocSerializer;
use rkyv::ser::Serializer;
use rkyv::Deserialize;
#[test]
fn test_hello_world() {
assert_eq!(hello_world(), "Hello from Episteme Core!");
}
#[test]
fn test_assertion_serialization_roundtrip() {
let assertion = Assertion {
subject: "Tesla_Inc".to_string(),
predicate: "has_revenue".to_string(),
object: ObjectValue::Number(96.7),
parent_hash: None,
source_hash: [0u8; 32],
visual_hash: Some([1u8; 8]),
epoch: Some([2u8; 32]),
signatures: vec![SignatureEntry {
agent_id: [2u8; 32],
signature: [3u8; 64],
timestamp: 123456789,
}],
confidence: 0.95,
timestamp: 123456789,
vector: Some(vec![0.1, 0.2, 0.3]),
};
// Serialize
let mut serializer = AllocSerializer::<4096>::default();
serializer.serialize_value(&assertion).expect("Failed to serialize");
let bytes = serializer.into_serializer().into_inner();
// Validate
let archived = check_archived_root::<Assertion>(&bytes)
.expect("Failed to validate archived assertion");
// Deserialize
let deserialized: Assertion =
archived.deserialize(&mut rkyv::Infallible).expect("Failed to deserialize");
assert_eq!(assertion, deserialized);
assert_eq!(deserialized.subject, "Tesla_Inc");
assert_eq!(deserialized.visual_hash, Some([1u8; 8]));
assert_eq!(deserialized.epoch, Some([2u8; 32]));
}
#[test]
fn test_epoch_serialization_roundtrip() {
let epoch = Epoch {
id: [1u8; 32],
name: "Newtonian Physics".to_string(),
supersedes: Some([0u8; 32]),
supersession_type: Some(SupersessionType::Refinement),
start_timestamp: 1000,
end_timestamp: None,
};
// Serialize
let mut serializer = AllocSerializer::<4096>::default();
serializer.serialize_value(&epoch).expect("Failed to serialize");
let bytes = serializer.into_serializer().into_inner();
// Validate
let archived =
check_archived_root::<Epoch>(&bytes).expect("Failed to validate archived epoch");
// Deserialize
let deserialized: Epoch =
archived.deserialize(&mut rkyv::Infallible).expect("Failed to deserialize");
assert_eq!(epoch, deserialized);
assert_eq!(deserialized.name, "Newtonian Physics");
assert_eq!(deserialized.supersession_type, Some(SupersessionType::Refinement));
}
#[test]
fn test_vote_serialization_roundtrip() {
let vote = Vote {
assertion_hash: [1u8; 32],
agent_id: [2u8; 32],
weight: 0.8,
signature: [3u8; 64],
timestamp: 123456789,
};
// Serialize
let mut serializer = AllocSerializer::<4096>::default();
serializer.serialize_value(&vote).expect("Failed to serialize");
let bytes = serializer.into_serializer().into_inner();
// Validate
let archived =
check_archived_root::<Vote>(&bytes).expect("Failed to validate archived vote");
// Deserialize
let deserialized: Vote =
archived.deserialize(&mut rkyv::Infallible).expect("Failed to deserialize");
assert_eq!(vote, deserialized);
assert_eq!(deserialized.assertion_hash, [1u8; 32]);
assert_eq!(deserialized.weight, 0.8);
}
}

7
crates/stemedb-core/src/main.rs
View File

@ -1,7 +1,12 @@
//! Main entry point for the Episteme Core binary.
//!
//! This binary currently serves as a test runner and demonstration
//! of the core library functionality.
use tracing::info;
fn main() {
// Initialize tracing subscriber for development
// Initialize tracing (placeholder for now)
tracing_subscriber::fmt::init();
info!("{}", stemedb_core::hello_world());

123
crates/stemedb-core/src/types.rs Normal file
View File

@ -0,0 +1,123 @@
use rkyv::{Archive, Deserialize, Serialize};
/// A 256-bit BLAKE3 hash.
pub type Hash = [u8; 32];
/// A 64-bit Perceptual Hash (pHash).
pub type PHash = [u8; 8];
/// A unique identifier for an entity (Subject/Object).
pub type EntityId = String;
/// A unique identifier for a relation (Predicate).
pub type RelationId = String;
/// A unique identifier for an Epoch (Paradigm).
pub type EpochId = Hash;
/// The atomic unit of knowledge in StemeDB.
#[derive(Archive, Deserialize, Serialize, Debug, Clone, PartialEq)]
#[archive(check_bytes)]
pub struct Assertion {
// 1. The Fact (The "What")
/// The subject of the assertion (e.g., "Tesla_Inc").
pub subject: EntityId,
/// The predicate or relation (e.g., "has_revenue").
pub predicate: RelationId,
/// The object or value (e.g., 96.7 billion).
pub object: ObjectValue,
// 2. The Lineage (The "Why")
/// The hash of the parent assertion, if this is a modification (fork).
pub parent_hash: Option<Hash>,
/// The hash of the source evidence (PDF, URL, Log).
pub source_hash: Hash,
/// The perceptual hash for visual anchoring (if applicable).
pub visual_hash: Option<PHash>,
/// The epoch this assertion belongs to (if any).
pub epoch: Option<EpochId>,
// 3. Meta-Cognition (The "Who" and "How sure")
/// List of agent signatures vouching for this assertion.
pub signatures: Vec<SignatureEntry>,
/// The confidence score (0.0 to 1.0).
pub confidence: f32,
/// The timestamp when the assertion was created (Unix epoch).
pub timestamp: u64,
/// The semantic embedding vector for fuzzy recall.
pub vector: Option<Vec<f32>>,
}
/// A value stored in an assertion object.
#[derive(Archive, Deserialize, Serialize, Debug, Clone, PartialEq)]
#[archive(check_bytes)]
pub enum ObjectValue {
/// A text string value.
Text(String),
/// A numeric value (float).
Number(f64),
/// A boolean value.
Boolean(bool),
/// A reference to another entity (pointer).
Reference(EntityId),
}
/// A cryptographic signature from an agent.
#[derive(Archive, Deserialize, Serialize, Debug, Clone, PartialEq)]
#[archive(check_bytes)]
pub struct SignatureEntry {
/// The Ed25519 Public Key of the agent.
pub agent_id: [u8; 32],
/// The Ed25519 Signature of the assertion.
pub signature: [u8; 64],
/// The timestamp when the agent signed this assertion.
pub timestamp: u64,
}
/// Defines the nature of a paradigm shift.
#[derive(Archive, Deserialize, Serialize, Debug, Clone, PartialEq)]
#[archive(check_bytes)]
pub enum SupersessionType {
/// The old epoch was factually incorrect (e.g., "Earth is flat").
Invalidation,
/// The old epoch was correct but is now outdated (e.g., "President is Obama").
Temporal,
/// The old epoch was a simplification (e.g., Newtonian Physics vs Relativity).
Refinement,
}
/// Represents a distinct period of truth or a specific worldview.
#[derive(Archive, Deserialize, Serialize, Debug, Clone, PartialEq)]
#[archive(check_bytes)]
pub struct Epoch {
/// The unique ID of this epoch.
pub id: EpochId,
/// Human-readable name (e.g., "Pre-2024", "Newtonian").
pub name: String,
/// If this epoch replaces another, which one?
pub supersedes: Option<EpochId>,
/// How does it supersede the previous one?
pub supersession_type: Option<SupersessionType>,
/// The timestamp when this epoch began.
pub start_timestamp: u64,
/// The timestamp when this epoch ended (if applicable).
pub end_timestamp: Option<u64>,
}
/// A vote cast by an agent on an existing assertion.
///
/// This enables high-velocity consensus gathering without modifying the immutable assertion itself.
#[derive(Archive, Deserialize, Serialize, Debug, Clone, PartialEq)]
#[archive(check_bytes)]
pub struct Vote {
/// The hash of the assertion being voted on.
pub assertion_hash: Hash,
/// The Ed25519 Public Key of the voting agent.
pub agent_id: [u8; 32],
/// The weight of the vote (Confidence), 0.0 to 1.0.
pub weight: f32,
/// The Ed25519 Signature of the `assertion_hash`.
pub signature: [u8; 64],
/// The timestamp when the vote was cast (Unix epoch).
pub timestamp: u64,
}

23
crates/stemedb-ingest/Cargo.toml Normal file
View File

@ -0,0 +1,23 @@
[package]
name = "stemedb-ingest"
version = "0.1.0"
edition = "2021"
description = "Ingestion pipeline for Episteme"
# Inherit workspace lints
[lints]
workspace = true
[dependencies]
stemedb-core = { path = "../stemedb-core" }
stemedb-wal = { path = "../stemedb-wal" }
stemedb-storage = { path = "../stemedb-storage" }
tokio = { version = "1.36", features = ["full"] }
tracing = "0.1"
rkyv = { version = "0.7", features = ["validation"] }
thiserror = "1.0"
blake3 = "1.5"
hex = "0.4"
[dev-dependencies]
tempfile = "3.10"

26
crates/stemedb-ingest/src/error.rs Normal file
View File

@ -0,0 +1,26 @@
use stemedb_storage::StorageError;
use stemedb_wal::QuarantineError;
use thiserror::Error;
/// Result type for ingestion operations.
pub type Result<T> = std::result::Result<T, IngestError>;
/// Errors that can occur during ingestion.
#[derive(Error, Debug)]
pub enum IngestError {
/// Error from the WAL.
#[error("WAL error: {0}")]
Wal(#[from] QuarantineError),
/// Error from the storage engine.
#[error("Storage error: {0}")]
Storage(#[from] StorageError),
/// Serialization/Deserialization error.
#[error("Serialization error: {0}")]
Serialization(String),
/// Worker task panicked or failed.
#[error("Worker error: {0}")]
Worker(String),
}

55
crates/stemedb-ingest/src/ingestor.rs Normal file
View File

@ -0,0 +1,55 @@
use crate::error::Result;
use crate::worker::IngestWorker;
use std::sync::Arc;
use stemedb_storage::KVStore;
use stemedb_wal::Journal;
use tokio::sync::Mutex;
use tokio::task::JoinHandle;
use tracing::{debug, info, instrument};
/// Manager for the background ingestion process.
pub struct Ingestor<S> {
worker: Arc<Mutex<IngestWorker<S>>>,
handle: Option<JoinHandle<()>>,
}
impl<S: KVStore + 'static> Ingestor<S> {
/// Create a new Ingestor.
pub fn new(journal: Arc<Mutex<Journal>>, store: Arc<S>) -> Self {
let worker = Arc::new(Mutex::new(IngestWorker::new(journal, store)));
debug!("Ingestor created");
Self { worker, handle: None }
}
/// Start the background ingestion task.
#[instrument(skip(self))]
pub fn start(&mut self) {
if self.handle.is_some() {
debug!("Ingestor already running");
return;
}
info!("Starting background ingestion task");
let worker = self.worker.clone();
self.handle = Some(tokio::spawn(async move {
let mut w = worker.lock().await;
w.run().await;
}));
}
/// Process pending WAL entries immediately (for testing).
#[instrument(skip(self))]
pub async fn process_pending(&self) -> Result<u64> {
let mut worker = self.worker.lock().await;
let mut total_bytes = 0;
loop {
let bytes = worker.step().await?;
if bytes == 0 {
break;
}
total_bytes += bytes;
}
debug!(total_bytes, "Processed pending entries");
Ok(total_bytes)
}
}

23
crates/stemedb-ingest/src/lib.rs Normal file
View File

@ -0,0 +1,23 @@
//! Ingestion pipeline for Episteme.
//!
//! This crate handles the reading of the Write-Ahead Log (WAL) and
//! the background indexing of assertions into the storage engine.
//!
//! # Storage Layout
//!
//! Records are stored with content-addressed keys:
//! - `H:{hash}` - Assertions
//! - `V:{assertion_hash}:{vote_hash}` - Votes
//! - `E:{hash}` - Epochs
//! - `S:{subject}` - Subject index
/// Error types and Result wrapper for ingestion.
pub mod error;
/// High-level ingestor manager.
pub mod ingestor;
/// Background worker logic for processing the WAL.
pub mod worker;
pub use error::{IngestError, Result};
pub use ingestor::Ingestor;
pub use worker::{serialize_assertion, serialize_epoch, serialize_vote, IngestWorker, RecordType};

634
crates/stemedb-ingest/src/worker.rs Normal file
View File

@ -0,0 +1,634 @@
//! Background worker that tails the WAL and updates the KV store.
//!
//! The worker reads records from the Write-Ahead Log and persists them
//! to the storage engine using content-addressed keys.
//!
//! # Storage Layout
//!
//! Following the architecture spec, records are stored with these key prefixes:
//! - `H:{hash}` - Assertions (content-addressed by BLAKE3 hash)
//! - `V:{assertion_hash}:{vote_hash}` - Votes on assertions
//! - `E:{hash}` - Epochs (paradigm definitions)
//! - `S:{subject}` - Subject adjacency index (list of assertion hashes)
use crate::error::{IngestError, Result};
use rkyv::ser::serializers::AllocSerializer;
use rkyv::ser::Serializer;
use rkyv::Deserialize;
use std::sync::Arc;
use stemedb_core::types::{Assertion, Epoch, Vote};
use stemedb_storage::KVStore;
use stemedb_wal::{Journal, HEADER_SIZE};
use tokio::sync::Mutex;
use tracing::{debug, error, info};
/// Record type discriminator.
///
/// This allows the worker to deserialize the correct type from the payload.
/// We use an 8-byte header to maintain rkyv alignment requirements.
#[repr(u8)]
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum RecordType {
/// An assertion record.
Assertion = 0,
/// A vote record.
Vote = 1,
/// An epoch record.
Epoch = 2,
}
/// Size of the record type header (8 bytes for alignment).
const RECORD_HEADER_SIZE: usize = 8;
impl TryFrom<u8> for RecordType {
type Error = IngestError;
fn try_from(value: u8) -> Result<Self> {
match value {
0 => Ok(RecordType::Assertion),
1 => Ok(RecordType::Vote),
2 => Ok(RecordType::Epoch),
_ => Err(IngestError::Serialization(format!("Unknown record type: {}", value))),
}
}
}
/// Serialize an assertion with its record type header.
///
/// Uses an 8-byte header to maintain rkyv alignment requirements.
pub fn serialize_assertion(assertion: &Assertion) -> Result<Vec<u8>> {
let mut serializer = AllocSerializer::<4096>::default();
serializer.serialize_value(assertion).map_err(|e| IngestError::Serialization(e.to_string()))?;
let bytes = serializer.into_serializer().into_inner();
let mut payload = Vec::with_capacity(RECORD_HEADER_SIZE + bytes.len());
// 8-byte header: [type, 0, 0, 0, 0, 0, 0, 0]
payload.push(RecordType::Assertion as u8);
payload.extend_from_slice(&[0u8; RECORD_HEADER_SIZE - 1]);
payload.extend_from_slice(&bytes);
Ok(payload)
}
/// Serialize a vote with its record type header.
pub fn serialize_vote(vote: &Vote) -> Result<Vec<u8>> {
let mut serializer = AllocSerializer::<4096>::default();
serializer.serialize_value(vote).map_err(|e| IngestError::Serialization(e.to_string()))?;
let bytes = serializer.into_serializer().into_inner();
let mut payload = Vec::with_capacity(RECORD_HEADER_SIZE + bytes.len());
payload.push(RecordType::Vote as u8);
payload.extend_from_slice(&[0u8; RECORD_HEADER_SIZE - 1]);
payload.extend_from_slice(&bytes);
Ok(payload)
}
/// Serialize an epoch with its record type header.
pub fn serialize_epoch(epoch: &Epoch) -> Result<Vec<u8>> {
let mut serializer = AllocSerializer::<4096>::default();
serializer.serialize_value(epoch).map_err(|e| IngestError::Serialization(e.to_string()))?;
let bytes = serializer.into_serializer().into_inner();
let mut payload = Vec::with_capacity(RECORD_HEADER_SIZE + bytes.len());
payload.push(RecordType::Epoch as u8);
payload.extend_from_slice(&[0u8; RECORD_HEADER_SIZE - 1]);
payload.extend_from_slice(&bytes);
Ok(payload)
}
/// Background worker that tails the WAL and updates the KV store.
pub struct IngestWorker<S> {
journal: Arc<Mutex<Journal>>,
store: Arc<S>,
current_offset: u64,
}
impl<S: KVStore> IngestWorker<S> {
/// Create a new ingest worker.
pub fn new(journal: Arc<Mutex<Journal>>, store: Arc<S>) -> Self {
Self {
journal,
store,
current_offset: HEADER_SIZE as u64, // Skip file header
}
}
/// Run a single iteration of the ingestion loop.
///
/// Reads the next record from the WAL, deserializes it, and writes it to storage.
/// Returns the number of bytes processed (0 if no new data).
pub async fn step(&mut self) -> Result<u64> {
let record = {
let journal = self.journal.lock().await;
match journal.read(self.current_offset) {
Ok(record) => record,
Err(stemedb_wal::QuarantineError::Io { .. }) => {
// Likely EOF, no new data
return Ok(0);
}
Err(stemedb_wal::QuarantineError::IoGeneric(e))
if e.kind() == std::io::ErrorKind::UnexpectedEof =>
{
// Definitely EOF
return Ok(0);
}
Err(e) => return Err(IngestError::Wal(e)),
}
};
let bytes_read = record.disk_size();
// Extract record type from header (8-byte aligned)
if record.payload.len() < RECORD_HEADER_SIZE {
return Err(IngestError::Serialization("Payload too small for header".to_string()));
}
let record_type = RecordType::try_from(record.payload[0])?;
let data = &record.payload[RECORD_HEADER_SIZE..];
match record_type {
RecordType::Assertion => self.ingest_assertion(data).await?,
RecordType::Vote => self.ingest_vote(data).await?,
RecordType::Epoch => self.ingest_epoch(data).await?,
}
self.current_offset += bytes_read;
info!(
record_type = ?record_type,
offset = self.current_offset - bytes_read,
new_offset = self.current_offset,
"Ingested record"
);
Ok(bytes_read)
}
/// Ingest an assertion into the KV store.
async fn ingest_assertion(&self, data: &[u8]) -> Result<()> {
let archived = rkyv::check_archived_root::<Assertion>(data)
.map_err(|e| IngestError::Serialization(e.to_string()))?;
let assertion: Assertion = archived
.deserialize(&mut rkyv::Infallible)
.map_err(|e| IngestError::Serialization(e.to_string()))?;
// Content-addressed key: H:{BLAKE3_hash}
let hash = blake3::hash(data);
let key = format!("H:{}", hash.to_hex()).into_bytes();
debug!(
subject = %assertion.subject,
predicate = %assertion.predicate,
hash = %hash.to_hex(),
"Ingesting assertion"
);
// Store the assertion
self.store.put(&key, data).await?;
// Update subject index: S:{subject} -> list of assertion hashes
// For now, we store the hash directly (in a real system, this would be a list)
let index_key = format!("S:{}", assertion.subject).into_bytes();
self.store.put(&index_key, hash.as_bytes()).await?;
Ok(())
}
/// Ingest a vote into the KV store.
async fn ingest_vote(&self, data: &[u8]) -> Result<()> {
let archived = rkyv::check_archived_root::<Vote>(data)
.map_err(|e| IngestError::Serialization(e.to_string()))?;
let vote: Vote = archived
.deserialize(&mut rkyv::Infallible)
.map_err(|e| IngestError::Serialization(e.to_string()))?;
// Vote key: V:{assertion_hash}:{vote_hash}
let vote_hash = blake3::hash(data);
let assertion_hash_hex = hex::encode(vote.assertion_hash);
let key = format!("V:{}:{}", assertion_hash_hex, vote_hash.to_hex()).into_bytes();
debug!(
assertion_hash = %assertion_hash_hex,
vote_hash = %vote_hash.to_hex(),
weight = vote.weight,
"Ingesting vote"
);
self.store.put(&key, data).await?;
Ok(())
}
/// Ingest an epoch into the KV store.
async fn ingest_epoch(&self, data: &[u8]) -> Result<()> {
let archived = rkyv::check_archived_root::<Epoch>(data)
.map_err(|e| IngestError::Serialization(e.to_string()))?;
let epoch: Epoch = archived
.deserialize(&mut rkyv::Infallible)
.map_err(|e| IngestError::Serialization(e.to_string()))?;
// Epoch key: E:{epoch_id_hash}
let epoch_id_hex = hex::encode(epoch.id);
let key = format!("E:{}", epoch_id_hex).into_bytes();
debug!(
epoch_id = %epoch_id_hex,
name = %epoch.name,
"Ingesting epoch"
);
self.store.put(&key, data).await?;
Ok(())
}
/// Run the ingestion loop continuously.
pub async fn run(&mut self) {
info!("Starting ingestion loop...");
loop {
match self.step().await {
Ok(0) => {
// No new data, sleep briefly
tokio::time::sleep(std::time::Duration::from_millis(10)).await;
}
Ok(_) => {
// Processed data, continue immediately
}
Err(e) => {
error!("Ingestion error: {:?}", e);
tokio::time::sleep(std::time::Duration::from_secs(1)).await;
}
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use stemedb_core::types::{ObjectValue, SignatureEntry};
use stemedb_storage::SledStore;
use stemedb_wal::Journal;
use tempfile::tempdir;
fn create_test_assertion() -> Assertion {
Assertion {
subject: "Tesla_Inc".to_string(),
predicate: "has_revenue".to_string(),
object: ObjectValue::Number(96.7),
parent_hash: None,
source_hash: [0u8; 32],
visual_hash: None,
epoch: None,
signatures: vec![SignatureEntry {
agent_id: [1u8; 32],
signature: [2u8; 64],
timestamp: 1000,
}],
confidence: 0.95,
timestamp: 1000,
vector: None,
}
}
fn create_test_vote() -> Vote {
Vote {
assertion_hash: [1u8; 32],
agent_id: [2u8; 32],
weight: 0.8,
signature: [3u8; 64],
timestamp: 2000,
}
}
fn create_test_epoch() -> Epoch {
Epoch {
id: [4u8; 32],
name: "Test Epoch".to_string(),
supersedes: None,
supersession_type: None,
start_timestamp: 3000,
end_timestamp: None,
}
}
#[tokio::test]
async fn test_ingest_assertion() {
let dir = tempdir().expect("Failed to create temp dir");
let wal_dir = dir.path().join("wal");
let db_dir = dir.path().join("db");
// Create journal and store
let mut journal = Journal::open(&wal_dir).expect("Failed to open journal");
let store = SledStore::open(&db_dir).expect("Failed to open store");
// Write assertion to WAL
let assertion = create_test_assertion();
let payload = serialize_assertion(&assertion).expect("Failed to serialize");
journal.append(payload).expect("Failed to append");
// Create worker and process
let journal = Arc::new(Mutex::new(journal));
let store = Arc::new(store);
let mut worker = IngestWorker::new(journal, store.clone());
let bytes = worker.step().await.expect("Failed to step");
assert!(bytes > 0, "Should have processed data");
// Verify assertion was stored
let keys = store.scan_prefix(b"H:").await.expect("Failed to scan");
assert_eq!(keys.len(), 1, "Should have one assertion");
// Verify subject index
let index = store.scan_prefix(b"S:Tesla_Inc").await.expect("Failed to scan");
assert_eq!(index.len(), 1, "Should have subject index");
}
#[tokio::test]
async fn test_ingest_vote() {
let dir = tempdir().expect("Failed to create temp dir");
let wal_dir = dir.path().join("wal");
let db_dir = dir.path().join("db");
let mut journal = Journal::open(&wal_dir).expect("Failed to open journal");
let store = SledStore::open(&db_dir).expect("Failed to open store");
let vote = create_test_vote();
let payload = serialize_vote(&vote).expect("Failed to serialize");
journal.append(payload).expect("Failed to append");
let journal = Arc::new(Mutex::new(journal));
let store = Arc::new(store);
let mut worker = IngestWorker::new(journal, store.clone());
let bytes = worker.step().await.expect("Failed to step");
assert!(bytes > 0);
// Verify vote was stored with V: prefix
let keys = store.scan_prefix(b"V:").await.expect("Failed to scan");
assert_eq!(keys.len(), 1, "Should have one vote");
}
#[tokio::test]
async fn test_ingest_epoch() {
let dir = tempdir().expect("Failed to create temp dir");
let wal_dir = dir.path().join("wal");
let db_dir = dir.path().join("db");
let mut journal = Journal::open(&wal_dir).expect("Failed to open journal");
let store = SledStore::open(&db_dir).expect("Failed to open store");
let epoch = create_test_epoch();
let payload = serialize_epoch(&epoch).expect("Failed to serialize");
journal.append(payload).expect("Failed to append");
let journal = Arc::new(Mutex::new(journal));
let store = Arc::new(store);
let mut worker = IngestWorker::new(journal, store.clone());
let bytes = worker.step().await.expect("Failed to step");
assert!(bytes > 0);
// Verify epoch was stored with E: prefix
let keys = store.scan_prefix(b"E:").await.expect("Failed to scan");
assert_eq!(keys.len(), 1, "Should have one epoch");
}
#[tokio::test]
async fn test_ingest_multiple_records() {
let dir = tempdir().expect("Failed to create temp dir");
let wal_dir = dir.path().join("wal");
let db_dir = dir.path().join("db");
let mut journal = Journal::open(&wal_dir).expect("Failed to open journal");
let store = SledStore::open(&db_dir).expect("Failed to open store");
// Write multiple records
let assertion = create_test_assertion();
let vote = create_test_vote();
let epoch = create_test_epoch();
journal.append(serialize_assertion(&assertion).expect("ser")).expect("append");
journal.append(serialize_vote(&vote).expect("ser")).expect("append");
journal.append(serialize_epoch(&epoch).expect("ser")).expect("append");
let journal = Arc::new(Mutex::new(journal));
let store = Arc::new(store);
let mut worker = IngestWorker::new(journal, store.clone());
// Process all records
let mut total = 0;
loop {
let bytes = worker.step().await.expect("Failed to step");
if bytes == 0 {
break;
}
total += bytes;
}
assert!(total > 0, "Should have processed data");
// Verify all records were stored
let assertions = store.scan_prefix(b"H:").await.expect("scan");
let votes = store.scan_prefix(b"V:").await.expect("scan");
let epochs = store.scan_prefix(b"E:").await.expect("scan");
assert_eq!(assertions.len(), 1, "Should have one assertion");
assert_eq!(votes.len(), 1, "Should have one vote");
assert_eq!(epochs.len(), 1, "Should have one epoch");
}
// ========================================================================
// CRASH RECOVERY INTEGRATION TESTS
// ========================================================================
//
// These tests verify the full pipeline survives crashes:
// 1. Write to WAL
// 2. "Crash" (drop all handles)
// 3. Reopen WAL and KV store
// 4. Run ingestor
// 5. Verify data is present
/// Test: Full pipeline crash recovery for assertions.
///
/// Proves the fundamental guarantee: write -> crash -> restart -> read works.
#[tokio::test]
async fn test_crash_recovery_assertion() {
let dir = tempdir().expect("Failed to create temp dir");
let wal_dir = dir.path().join("wal");
let db_dir = dir.path().join("db");
let assertion = create_test_assertion();
// Phase 1: Write to WAL and "crash" (drop everything)
{
let mut journal = Journal::open(&wal_dir).expect("Failed to open journal");
let payload = serialize_assertion(&assertion).expect("Failed to serialize");
journal.append(payload).expect("Failed to append");
// Journal dropped here - simulates crash
}
// Phase 2: Recovery - reopen everything and run ingestor
{
let journal = Journal::open(&wal_dir).expect("Failed to reopen journal");
let store = SledStore::open(&db_dir).expect("Failed to open store");
let journal = Arc::new(Mutex::new(journal));
let store = Arc::new(store);
let mut worker = IngestWorker::new(journal, store.clone());
// Process all pending records
loop {
let bytes = worker.step().await.expect("Failed to step");
if bytes == 0 {
break;
}
}
// Verify assertion was recovered and ingested
let assertions = store.scan_prefix(b"H:").await.expect("Failed to scan");
assert_eq!(assertions.len(), 1, "Assertion should survive crash");
// Verify subject index was created
let index = store.scan_prefix(b"S:Tesla_Inc").await.expect("Failed to scan");
assert_eq!(index.len(), 1, "Subject index should be created");
}
}
/// Test: Full pipeline crash recovery with all record types.
///
/// Assertions, Votes, and Epochs all survive crash and are ingested correctly.
#[tokio::test]
async fn test_crash_recovery_all_record_types() {
let dir = tempdir().expect("Failed to create temp dir");
let wal_dir = dir.path().join("wal");
let db_dir = dir.path().join("db");
let assertion = create_test_assertion();
let vote = create_test_vote();
let epoch = create_test_epoch();
// Phase 1: Write all record types and "crash"
{
let mut journal = Journal::open(&wal_dir).expect("Failed to open journal");
journal.append(serialize_assertion(&assertion).expect("ser")).expect("append");
journal.append(serialize_vote(&vote).expect("ser")).expect("append");
journal.append(serialize_epoch(&epoch).expect("ser")).expect("append");
// Crash
}
// Phase 2: Recovery
{
let journal = Journal::open(&wal_dir).expect("Failed to reopen journal");
let store = SledStore::open(&db_dir).expect("Failed to open store");
let journal = Arc::new(Mutex::new(journal));
let store = Arc::new(store);
let mut worker = IngestWorker::new(journal, store.clone());
// Ingest all records
while worker.step().await.expect("step") > 0 {}
// Verify all record types survived
let assertions = store.scan_prefix(b"H:").await.expect("scan");
let votes = store.scan_prefix(b"V:").await.expect("scan");
let epochs = store.scan_prefix(b"E:").await.expect("scan");
assert_eq!(assertions.len(), 1, "Assertion should survive crash");
assert_eq!(votes.len(), 1, "Vote should survive crash");
assert_eq!(epochs.len(), 1, "Epoch should survive crash");
}
}
/// Test: Multiple crash-recovery cycles maintain data integrity.
///
/// Simulates a flaky system that crashes and recovers multiple times,
/// each time adding more data.
#[tokio::test]
async fn test_repeated_crash_recovery_cycles() {
let dir = tempdir().expect("Failed to create temp dir");
let wal_dir = dir.path().join("wal");
let db_dir = dir.path().join("db");
let num_cycles = 3;
for cycle in 0..num_cycles {
// Write new data and crash
{
let mut journal = Journal::open(&wal_dir).expect("Failed to open journal");
let mut assertion = create_test_assertion();
assertion.subject = format!("Subject_Cycle_{}", cycle);
journal.append(serialize_assertion(&assertion).expect("ser")).expect("append");
}
// Recover and ingest
{
let journal = Journal::open(&wal_dir).expect("Failed to reopen journal");
let store = SledStore::open(&db_dir).expect("Failed to open store");
let journal = Arc::new(Mutex::new(journal));
let store = Arc::new(store);
let mut worker = IngestWorker::new(journal, store.clone());
while worker.step().await.expect("step") > 0 {}
// Verify we have assertions from all cycles so far
let assertions = store.scan_prefix(b"H:").await.expect("scan");
assert_eq!(
assertions.len(),
cycle + 1,
"Should have {} assertions after cycle {}",
cycle + 1,
cycle
);
}
}
// Final verification: all data from all cycles present
{
let store = SledStore::open(&db_dir).expect("Failed to open store");
let assertions = store.scan_prefix(b"H:").await.expect("scan");
assert_eq!(
assertions.len(),
num_cycles,
"All assertions should survive multiple crashes"
);
}
}
/// Test: KV store persists across restarts.
///
/// Verifies that once data is ingested to sled, it survives store restarts.
#[tokio::test]
async fn test_kv_store_persistence() {
let dir = tempdir().expect("Failed to create temp dir");
let wal_dir = dir.path().join("wal");
let db_dir = dir.path().join("db");
// Phase 1: Write, ingest, and close everything
{
let mut journal = Journal::open(&wal_dir).expect("Failed to open journal");
let store = SledStore::open(&db_dir).expect("Failed to open store");
let assertion = create_test_assertion();
journal.append(serialize_assertion(&assertion).expect("ser")).expect("append");
let journal = Arc::new(Mutex::new(journal));
let store = Arc::new(store);
let mut worker = IngestWorker::new(journal, store.clone());
while worker.step().await.expect("step") > 0 {}
// Flush to ensure persistence
store.flush().await.expect("Failed to flush");
}
// Phase 2: Reopen only the KV store and verify data persists
{
let store = SledStore::open(&db_dir).expect("Failed to reopen store");
let assertions = store.scan_prefix(b"H:").await.expect("scan");
assert_eq!(assertions.len(), 1, "Assertion should persist in KV store across restarts");
}
}
}

24
crates/stemedb-sim/Cargo.toml Normal file
View File

@ -0,0 +1,24 @@
[package]
name = "stemedb-sim"
version = "0.1.0"
edition = "2021"
description = "Simulation environment for StemeDB"
# Inherit workspace lints
[lints]
workspace = true
[dependencies]
stemedb-core = { path = "../stemedb-core" }
stemedb-wal = { path = "../stemedb-wal" }
stemedb-ingest = { path = "../stemedb-ingest" }
stemedb-storage = { path = "../stemedb-storage" }
tokio = { version = "1.36", features = ["full"] }
ed25519-dalek = { version = "2.1", features = ["rand_core"] }
rand = "0.8"
tracing = "0.1"
tracing-subscriber = "0.3"
rkyv = { version = "0.7", features = ["validation"] }
bytecheck = "0.6"
thiserror = "1.0"
tempfile = "3.10"

166
crates/stemedb-sim/src/main.rs Normal file
View File

@ -0,0 +1,166 @@
//! StemeDB Spine Simulator
//!
//! This simulation validates the "Spine" (Durability + Schema + Ingestion) by simulating
//! multiple agents creating, signing, and writing assertions to the WAL, which are then
//! asynchronously ingested into the Storage Engine.
use ed25519_dalek::{Signature, Signer, SigningKey, Verifier, VerifyingKey};
use rand::rngs::OsRng;
use rkyv::ser::serializers::AllocSerializer;
use rkyv::ser::Serializer;
use rkyv::Deserialize;
use std::sync::Arc;
use stemedb_core::types::{Assertion, ObjectValue, SignatureEntry};
use stemedb_ingest::Ingestor;
use stemedb_storage::{KVStore, SledStore};
use stemedb_wal::Journal;
use tokio::sync::Mutex;
use tracing::info;
/// A simulated agent with a cryptographic identity.
struct Agent {
pub id: String,
signing_key: SigningKey,
verifying_key: VerifyingKey,
}
impl Agent {
pub fn new(id: &str) -> Self {
let mut csprng = OsRng;
let signing_key = SigningKey::generate(&mut csprng);
let verifying_key = VerifyingKey::from(&signing_key);
Self { id: id.to_string(), signing_key, verifying_key }
}
/// Create and sign an assertion.
pub fn sign_assertion(&self, subject: &str, predicate: &str, object: ObjectValue) -> Assertion {
let timestamp = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
// For simulation, we sign the concatenation of subject and predicate
// In a real system, we'd sign the hash of the fact data.
let message = format!("{}:{}", subject, predicate);
let signature: Signature = self.signing_key.sign(message.as_bytes());
Assertion {
subject: subject.to_string(),
predicate: predicate.to_string(),
object,
parent_hash: None,
source_hash: [0u8; 32],
visual_hash: None,
epoch: None,
signatures: vec![SignatureEntry {
agent_id: self.verifying_key.to_bytes(),
signature: signature.to_bytes(),
timestamp,
}],
confidence: 1.0,
timestamp,
vector: None,
}
}
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
tracing_subscriber::fmt::init();
info!("🚀 Starting StemeDB Spine Simulation: 'The Arena'");
// 1. Setup Storage (WAL + KV)
let temp_wal_dir = tempfile::tempdir()?;
let temp_db_dir = tempfile::tempdir()?;
let journal = Arc::new(Mutex::new(Journal::open(temp_wal_dir.path())?));
let store = Arc::new(SledStore::open(temp_db_dir.path())?);
info!(" WAL initialized at {:?}", temp_wal_dir.path());
info!(" KV Store initialized at {:?}", temp_db_dir.path());
// 2. Start Ingestor
let mut ingestor = Ingestor::new(journal.clone(), store.clone());
ingestor.start();
info!(" Ingestor started (background worker).");
// 3. Setup Agents
let agents = vec![
Agent::new("Scientist_Alpha"),
Agent::new("Scientist_Beta"),
Agent::new("Troll_Prime"),
];
info!(" Swarm of {} agents instantiated.", agents.len());
// 4. Run Simulation Ticks
let mut assertions = Vec::new();
for i in 0..10 {
let agent = &agents[i % agents.len()];
let assertion = agent.sign_assertion(
&format!("Entity_{}", i),
"has_property",
ObjectValue::Text(format!("Value_{}", i)),
);
// Serialize the assertion
let mut serializer = AllocSerializer::<4096>::default();
serializer.serialize_value(&assertion)?;
let bytes = serializer.into_serializer().into_inner();
// Write to WAL (Fast Path)
{
let mut journal_lock = journal.lock().await;
let offset = journal_lock.append(bytes.to_vec())?;
info!(" [Tick {}] Agent '{}' wrote to WAL at offset {}", i, agent.id, offset);
}
assertions.push(assertion);
}
// 5. Wait for Ingestion (Eventual Consistency)
info!("⏳ Waiting for ingestion to catch up...");
// In a real test we'd poll or use channels, but for sim we just wait a bit
tokio::time::sleep(std::time::Duration::from_millis(500)).await;
// 6. Verify the Store (Query Path)
info!("🔬 Verifying Storage integrity...");
for (i, original_assertion) in assertions.iter().enumerate() {
// Construct the key used by the ingestor: H:{Subject}:{Predicate}
let key = format!("H:{}:{}", original_assertion.subject, original_assertion.predicate)
.into_bytes();
// Read from KV Store
let stored_bytes =
store.get(&key).await?.ok_or_else(|| format!("Assertion {} not found in store!", i))?;
// Deserialize and validate
let archived = rkyv::check_archived_root::<Assertion>(&stored_bytes)
.map_err(|e| format!("Stored Record is corrupt or invalid: {:?}", e))?;
let deserialized: Assertion = archived.deserialize(&mut rkyv::Infallible)?;
// Verify consistency
assert_eq!(original_assertion.subject, deserialized.subject);
assert_eq!(original_assertion.object, deserialized.object);
// Verify signature
let sig_entry = &deserialized.signatures[0];
let verifying_key = VerifyingKey::from_bytes(&sig_entry.agent_id)?;
let signature = Signature::from_bytes(&sig_entry.signature);
let message = format!("{}:{}", deserialized.subject, deserialized.predicate);
verifying_key.verify(message.as_bytes(), &signature)?;
info!(
" [Verify {}] Assertion for '{}' retrieved from KV Store and verified.",
i, deserialized.subject
);
}
info!("✅ Simulation 'The Arena' completed successfully. Spine is robust.");
Ok(())
}

19
crates/stemedb-storage/Cargo.toml Normal file
View File

@ -0,0 +1,19 @@
[package]
name = "stemedb-storage"
version = "0.1.0"
edition = "2021"
description = "Storage engine abstraction and implementations for Episteme"
# Inherit workspace lints
[lints]
workspace = true
[dependencies]
stemedb-core = { path = "../stemedb-core" }
sled = "0.34"
thiserror = "1.0"
tracing = "0.1"
async-trait = "0.1"
[dev-dependencies]
tokio = { version = "1.36", features = ["macros", "rt"] }

24
crates/stemedb-storage/src/error.rs Normal file
View File

@ -0,0 +1,24 @@
use thiserror::Error;
/// Result type for storage operations.
pub type Result<T> = std::result::Result<T, StorageError>;
/// Errors that can occur during storage operations.
#[derive(Error, Debug)]
pub enum StorageError {
/// IO error interacting with the storage backend.
#[error("Storage IO error: {0}")]
Io(#[from] std::io::Error),
/// Error specific to the sled backend.
#[error("Sled error: {0}")]
Sled(#[from] sled::Error),
/// Serialization/Deserialization error.
#[error("Serialization error: {0}")]
Serialization(String),
/// Key not found in storage.
#[error("Key not found: {0}")]
NotFound(String),
}

15
crates/stemedb-storage/src/lib.rs Normal file
View File

@ -0,0 +1,15 @@
//! Storage engine abstractions and implementations for Episteme.
//!
//! This crate provides the `KVStore` trait for pluggable storage backends
//! and a concrete implementation using `sled`.
/// Error types and Result wrapper for storage operations.
pub mod error;
/// Sled implementation of the storage backend.
pub mod sled_backend;
/// Core traits for key-value storage.
pub mod traits;
pub use error::{Result, StorageError};
pub use sled_backend::SledStore;
pub use traits::KVStore;

99
crates/stemedb-storage/src/sled_backend.rs Normal file
View File

@ -0,0 +1,99 @@
use crate::error::{Result, StorageError};
use crate::traits::KVStore;
use async_trait::async_trait;
use sled::Db;
use std::path::Path;
/// Sled-based implementation of the KVStore trait.
#[derive(Debug, Clone)]
pub struct SledStore {
db: Db,
}
impl SledStore {
/// Open or create a new Sled database at the given path.
pub fn open(path: impl AsRef<Path>) -> Result<Self> {
let db = sled::open(path).map_err(StorageError::Sled)?;
Ok(Self { db })
}
/// Open a temporary Sled database for testing.
#[cfg(test)]
pub fn open_temp() -> Result<Self> {
let config = sled::Config::new().temporary(true);
let db = config.open().map_err(StorageError::Sled)?;
Ok(Self { db })
}
}
#[async_trait]
impl KVStore for SledStore {
async fn get(&self, key: &[u8]) -> Result<Option<Vec<u8>>> {
let result = self.db.get(key).map_err(StorageError::Sled)?;
Ok(result.map(|ivec| ivec.to_vec()))
}
async fn put(&self, key: &[u8], value: &[u8]) -> Result<()> {
self.db.insert(key, value).map_err(StorageError::Sled)?;
Ok(())
}
async fn delete(&self, key: &[u8]) -> Result<()> {
self.db.remove(key).map_err(StorageError::Sled)?;
Ok(())
}
async fn scan_prefix(&self, prefix: &[u8]) -> Result<Vec<(Vec<u8>, Vec<u8>)>> {
let iter = self.db.scan_prefix(prefix);
let mut results = Vec::new();
for item in iter {
let (k, v) = item.map_err(StorageError::Sled)?;
results.push((k.to_vec(), v.to_vec()));
}
Ok(results)
}
async fn flush(&self) -> Result<()> {
self.db.flush_async().await.map_err(StorageError::Sled)?;
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn test_sled_store_roundtrip() {
let store = SledStore::open_temp().expect("Failed to create temp DB");
let key = b"test_key";
let value = b"test_value";
// Put
store.put(key, value).await.expect("Put failed");
// Get
let retrieved = store.get(key).await.expect("Get failed");
assert_eq!(retrieved, Some(value.to_vec()));
// Delete
store.delete(key).await.expect("Delete failed");
// Get after delete
let deleted = store.get(key).await.expect("Get failed");
assert_eq!(deleted, None);
}
#[tokio::test]
async fn test_scan_prefix() {
let store = SledStore::open_temp().expect("Failed to create temp DB");
store.put(b"prefix:1", b"val1").await.unwrap();
store.put(b"prefix:2", b"val2").await.unwrap();
store.put(b"other:3", b"val3").await.unwrap();
let results = store.scan_prefix(b"prefix:").await.unwrap();
assert_eq!(results.len(), 2);
assert_eq!(results[0], (b"prefix:1".to_vec(), b"val1".to_vec()));
assert_eq!(results[1], (b"prefix:2".to_vec(), b"val2".to_vec()));
}
}

24
crates/stemedb-storage/src/traits.rs Normal file
View File

@ -0,0 +1,24 @@
use crate::error::Result;
use async_trait::async_trait;
/// Abstract interface for Key-Value storage backends.
///
/// This trait allows us to swap the underlying storage engine (e.g., sled, RocksDB)
/// without changing the core logic of the database.
#[async_trait]
pub trait KVStore: Send + Sync {
/// Retrieve a value by key.
async fn get(&self, key: &[u8]) -> Result<Option<Vec<u8>>>;
/// Insert or update a value by key.
async fn put(&self, key: &[u8], value: &[u8]) -> Result<()>;
/// Delete a value by key.
async fn delete(&self, key: &[u8]) -> Result<()>;
/// Scan keys with a given prefix.
async fn scan_prefix(&self, prefix: &[u8]) -> Result<Vec<(Vec<u8>, Vec<u8>)>>;
/// Flush any pending writes to disk.
async fn flush(&self) -> Result<()>;
}

19
crates/stemedb-wal/Cargo.toml Normal file
View File

@ -0,0 +1,19 @@
[package]
name = "stemedb-wal"
version = "0.1.0"
edition = "2021"
description = "Write-Ahead Log (Quarantine Journal) for Episteme"
# Inherit workspace lints
[lints]
workspace = true
[dependencies]
fs2 = "0.4"
thiserror = "1.0"
tracing = "0.1"
byteorder = "1.5"
blake3 = "1.5"
[dev-dependencies]
tempfile = "3.10"

375
crates/stemedb-wal/src/durability.rs Normal file
View File

@ -0,0 +1,375 @@
//! fsync semantics and durability primitives.
//!
//! This module provides the fsync discipline for quarantine journal files.
//! It defines when and how data is durably persisted to disk.
//!
//! # Durability Levels
//!
//! - **Immediate**: fsync after every write (safest, slowest)
//! - **Batched**: fsync after N writes or T time (balanced)
//! - **Eventual**: fsync only on close (fastest, least safe)
use crate::error::{QuarantineError, Result};
use fs2::FileExt;
use std::fs::File;
use std::io::{self, Write};
use std::path::{Path, PathBuf};
use std::time::{Duration, Instant};
use tracing::{debug, instrument};
/// Default fsync timeout in seconds.
pub const DEFAULT_FSYNC_TIMEOUT_SECS: u64 = 5;
/// Default batch size for batched durability.
pub const DEFAULT_BATCH_SIZE: usize = 100;
/// Default batch time window.
pub const DEFAULT_BATCH_DURATION: Duration = Duration::from_millis(10);
/// Durability level for write operations.
///
/// Controls when fsync is called to ensure data is persisted to disk.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub enum DurabilityLevel {
/// fsync after every write operation.
/// - Highest durability guarantee
/// - Lowest throughput
/// - Use for critical data that cannot be lost
#[default]
Immediate,
/// fsync after batch_size writes OR batch_duration time.
/// - Good balance of durability and throughput
/// - Configurable trade-off
/// - Recommended for most use cases
Batched {
/// Maximum writes before fsync.
max_writes: usize,
/// Maximum time before fsync.
max_duration: Duration,
},
/// fsync only on explicit flush or close.
/// - Highest throughput
/// - Data may be lost on crash
/// - Use only for non-critical or reconstructible data
Eventual,
}
impl DurabilityLevel {
/// Create a batched durability level with defaults.
pub fn batched() -> Self {
Self::Batched { max_writes: DEFAULT_BATCH_SIZE, max_duration: DEFAULT_BATCH_DURATION }
}
/// Create a batched durability level with custom parameters.
pub fn batched_with(max_writes: usize, max_duration: Duration) -> Self {
Self::Batched { max_writes, max_duration }
}
}
/// Guard that ensures file is synced on drop.
///
/// This struct wraps a file handle and tracks pending writes.
/// When dropped, it attempts to sync any pending data.
pub struct FsyncGuard {
file: File,
path: PathBuf,
level: DurabilityLevel,
pending_writes: usize,
last_sync: Instant,
#[allow(dead_code)] // Reserved for future timeout logic
timeout: Duration,
}
impl FsyncGuard {
/// Create a new fsync guard for the given file.
pub fn new(file: File, path: PathBuf, level: DurabilityLevel) -> Self {
Self {
file,
path,
level,
pending_writes: 0,
last_sync: Instant::now(),
timeout: Duration::from_secs(DEFAULT_FSYNC_TIMEOUT_SECS),
}
}
/// Set the fsync timeout.
pub fn with_timeout(mut self, timeout: Duration) -> Self {
self.timeout = timeout;
self
}
/// Write data to the file and potentially sync based on durability level.
pub fn write(&mut self, data: &[u8]) -> Result<()> {
self.file.write_all(data).map_err(|e| QuarantineError::io(&self.path, e))?;
self.pending_writes += 1;
self.maybe_sync()
}
/// Check if sync is needed based on durability level and trigger if so.
pub fn maybe_sync(&mut self) -> Result<()> {
let should_sync = match self.level {
DurabilityLevel::Immediate => self.pending_writes > 0,
DurabilityLevel::Batched { max_writes, max_duration } => {
self.pending_writes >= max_writes || self.last_sync.elapsed() >= max_duration
}
DurabilityLevel::Eventual => false,
};
if should_sync {
self.force_sync()?;
}
Ok(())
}
/// Force an fsync regardless of durability level.
#[instrument(skip(self), fields(pending = self.pending_writes))]
pub fn force_sync(&mut self) -> Result<()> {
self.sync_file()?;
self.pending_writes = 0;
self.last_sync = Instant::now();
debug!("Forced sync complete");
Ok(())
}
/// Get the underlying file reference.
pub fn file(&self) -> &File {
&self.file
}
/// Get a mutable reference to the underlying file.
pub fn file_mut(&mut self) -> &mut File {
&mut self.file
}
/// Get the file path.
pub fn path(&self) -> &Path {
&self.path
}
/// Get the current durability level.
pub fn level(&self) -> DurabilityLevel {
self.level
}
/// Get the number of pending (unsynced) writes.
pub fn pending_writes(&self) -> usize {
self.pending_writes
}
/// Acquire an exclusive lock on the file.
#[instrument(skip(self), fields(path = %self.path.display()))]
pub fn lock_exclusive(&self) -> Result<()> {
self.file.lock_exclusive().map_err(|e| {
if e.kind() == io::ErrorKind::WouldBlock {
QuarantineError::FileLocked { path: self.path.clone() }
} else {
QuarantineError::io(&self.path, e)
}
})?;
debug!("Acquired exclusive lock");
Ok(())
}
/// Try to acquire an exclusive lock without blocking.
pub fn try_lock_exclusive(&self) -> Result<bool> {
match self.file.try_lock_exclusive() {
Ok(()) => Ok(true),
Err(e) if e.kind() == io::ErrorKind::WouldBlock => Ok(false),
Err(e) => Err(QuarantineError::io(&self.path, e)),
}
}
/// Release the file lock.
#[allow(clippy::incompatible_msrv)]
pub fn unlock(&self) -> Result<()> {
self.file.unlock().map_err(|e| QuarantineError::io(&self.path, e))
}
/// Perform the actual fsync operation.
fn sync_file(&self) -> Result<()> {
// Use sync_data (fdatasync) when we only need data durability,
// not metadata like modification time.
self.file
.sync_data()
.map_err(|e| QuarantineError::FsyncFailed { path: self.path.clone(), source: e })
}
}
impl Drop for FsyncGuard {
fn drop(&mut self) {
// Best-effort sync on drop - we can't return errors from Drop
if self.pending_writes > 0 {
if let Err(e) = self.force_sync() {
tracing::error!(
path = %self.path.display(),
error = %e,
pending_writes = self.pending_writes,
"Failed to sync file on drop"
);
}
}
}
}
/// Sync a directory to ensure file creation is durable.
///
/// This is necessary for crash-safe file operations on some filesystems.
pub fn sync_directory(path: &Path) -> Result<()> {
let dir = File::open(path).map_err(|e| QuarantineError::io(path, e))?;
dir.sync_all().map_err(|e| QuarantineError::FsyncFailed { path: path.to_path_buf(), source: e })
}
/// Perform an atomic file write (write to temp, sync, rename).
///
/// This ensures the file either exists completely or not at all.
pub fn atomic_write(path: &Path, contents: &[u8]) -> Result<()> {
let temp_path = path.with_extension("tmp");
// Write to temporary file
let mut file = File::create(&temp_path).map_err(|e| QuarantineError::io(&temp_path, e))?;
file.write_all(contents).map_err(|e| QuarantineError::io(&temp_path, e))?;
file.sync_all()
.map_err(|e| QuarantineError::FsyncFailed { path: temp_path.clone(), source: e })?;
drop(file);
// Rename atomically
std::fs::rename(&temp_path, path).map_err(|e| QuarantineError::io(path, e))?;
// Sync parent directory
if let Some(parent) = path.parent() {
sync_directory(parent)?;
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use tempfile::{tempdir, TempDir};
/// Test helper: Creates a temp dir and file for FsyncGuard tests
fn create_test_file() -> (TempDir, std::path::PathBuf, File) {
let dir = tempdir().unwrap();
let path = dir.path().join("test.quarantine");
let file = File::create(&path).unwrap();
(dir, path, file)
}
#[test]
fn test_durability_level_default() {
let level = DurabilityLevel::default();
assert_eq!(level, DurabilityLevel::Immediate);
}
#[test]
fn test_durability_level_batched() {
let level = DurabilityLevel::batched();
match level {
DurabilityLevel::Batched { max_writes, max_duration } => {
assert_eq!(max_writes, DEFAULT_BATCH_SIZE);
assert_eq!(max_duration, DEFAULT_BATCH_DURATION);
}
_ => panic!("Expected Batched"),
}
}
#[test]
fn test_fsync_guard_immediate() {
let (_dir, path, file) = create_test_file();
let mut guard = FsyncGuard::new(file, path.clone(), DurabilityLevel::Immediate);
guard.write(b"hello").unwrap();
assert_eq!(guard.pending_writes(), 0); // Should have synced
// Verify file contains data
let contents = std::fs::read(&path).unwrap();
assert_eq!(contents, b"hello");
}
#[test]
fn test_fsync_guard_batched() {
let (_dir, path, file) = create_test_file();
let level = DurabilityLevel::batched_with(3, Duration::from_secs(60));
let mut guard = FsyncGuard::new(file, path, level);
guard.write(b"1").unwrap();
assert_eq!(guard.pending_writes(), 1);
guard.write(b"2").unwrap();
assert_eq!(guard.pending_writes(), 2);
guard.write(b"3").unwrap();
assert_eq!(guard.pending_writes(), 0); // Should have synced at 3
}
#[test]
fn test_fsync_guard_eventual() {
let (_dir, path, file) = create_test_file();
let mut guard = FsyncGuard::new(file, path, DurabilityLevel::Eventual);
for i in 0..100 {
guard.write(&[i]).unwrap();
}
assert_eq!(guard.pending_writes(), 100); // Never synced
guard.force_sync().unwrap();
assert_eq!(guard.pending_writes(), 0);
}
#[test]
fn test_fsync_guard_drop_syncs() {
let dir = tempdir().unwrap();
let path = dir.path().join("test.quarantine");
{
let file = File::create(&path).unwrap();
let mut guard = FsyncGuard::new(file, path.clone(), DurabilityLevel::Eventual);
guard.write(b"test data").unwrap();
// Guard dropped here, should sync
}
// File should still contain data
let contents = std::fs::read(&path).unwrap();
assert_eq!(contents, b"test data");
}
#[test]
fn test_atomic_write() {
let dir = tempdir().unwrap();
let path = dir.path().join("atomic.txt");
atomic_write(&path, b"atomic content").unwrap();
let contents = std::fs::read(&path).unwrap();
assert_eq!(contents, b"atomic content");
// Temp file should not exist
let temp_path = path.with_extension("tmp");
assert!(!temp_path.exists());
}
#[test]
fn test_file_locking() {
let dir = tempdir().unwrap();
let path = dir.path().join("locked.quarantine");
let file = File::create(&path).unwrap();
let guard = FsyncGuard::new(file, path.clone(), DurabilityLevel::Immediate);
guard.lock_exclusive().unwrap();
// Try to lock from another handle
let file2 = File::open(&path).unwrap();
let guard2 = FsyncGuard::new(file2, path, DurabilityLevel::Immediate);
assert!(!guard2.try_lock_exclusive().unwrap());
guard.unlock().unwrap();
assert!(guard2.try_lock_exclusive().unwrap());
}
}

46
crates/stemedb-wal/src/error.rs Normal file
View File

@ -0,0 +1,46 @@
use std::io;
use std::path::PathBuf;
use thiserror::Error;
/// Result type for WAL operations.
pub type Result<T> = std::result::Result<T, QuarantineError>;
/// Errors that can occur during WAL operations.
#[derive(Error, Debug)]
pub enum QuarantineError {
/// IO error at a specific path.
#[error("IO error at {path:?}: {source}")]
Io {
/// The path where the error occurred.
path: PathBuf,
/// The underlying IO error.
source: io::Error,
},
/// Failed to fsync a file.
#[error("Failed to fsync {path:?}: {source}")]
FsyncFailed {
/// The path of the file.
path: PathBuf,
/// The underlying IO error.
source: io::Error,
},
/// File is locked by another process.
#[error("File is locked: {path:?}")]
FileLocked {
/// The path of the locked file.
path: PathBuf,
},
/// Generic IO error.
#[error(transparent)]
IoGeneric(#[from] io::Error),
}
impl QuarantineError {
/// Helper to create an `Io` error variant.
pub fn io(path: impl Into<PathBuf>, source: io::Error) -> Self {
Self::Io { path: path.into(), source }
}
}

196
crates/stemedb-wal/src/format.rs Normal file
View File

@ -0,0 +1,196 @@
use crate::error::{QuarantineError, Result};
use byteorder::{LittleEndian, ReadBytesExt, WriteBytesExt};
use std::io::{Read, Write};
/// Magic bytes for identifying quarantine files ("STEM" in ASCII).
pub const MAGIC: &[u8; 4] = b"STEM";
/// Current file format version.
pub const VERSION: u8 = 1;
/// Size of the file header in bytes.
/// Magic (4) + Version (1) + Reserved (3)
pub const HEADER_SIZE: usize = 8;
/// Maximum record size (100 MB).
pub const MAX_RECORD_SIZE: usize = 100 * 1024 * 1024;
/// File header structure.
///
/// Written at the beginning of every quarantine file.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct FileHeader {
/// Magic bytes "STEM".
pub magic: [u8; 4],
/// Format version.
pub version: u8,
}
impl Default for FileHeader {
fn default() -> Self {
Self::new()
}
}
impl FileHeader {
/// Create a new file header with default magic and version.
pub fn new() -> Self {
Self { magic: *MAGIC, version: VERSION }
}
/// Write the header to a writer.
pub fn write_to<W: Write>(&self, writer: &mut W) -> Result<()> {
writer.write_all(&self.magic).map_err(QuarantineError::IoGeneric)?;
writer.write_u8(self.version).map_err(QuarantineError::IoGeneric)?;
// Write 3 reserved bytes (padding)
writer.write_all(&[0u8; 3]).map_err(QuarantineError::IoGeneric)?;
Ok(())
}
/// Read the header from a reader and validate it.
pub fn read_from<R: Read>(reader: &mut R) -> Result<Self> {
let mut magic = [0u8; 4];
reader.read_exact(&mut magic).map_err(QuarantineError::IoGeneric)?;
if magic != *MAGIC {
return Err(QuarantineError::IoGeneric(std::io::Error::new(
std::io::ErrorKind::InvalidData,
"Invalid magic bytes",
)));
}
let version = reader.read_u8().map_err(QuarantineError::IoGeneric)?;
if version != VERSION {
return Err(QuarantineError::IoGeneric(std::io::Error::new(
std::io::ErrorKind::InvalidData,
format!("Unsupported version: {}", version),
)));
}
// Skip reserved bytes
let mut reserved = [0u8; 3];
reader.read_exact(&mut reserved).map_err(QuarantineError::IoGeneric)?;
Ok(Self { magic, version })
}
}
/// A single log record in the WAL.
///
/// Format:
/// - Checksum (32 bytes, BLAKE3)
/// - Payload Length (4 bytes, u32 LE)
/// - Payload (N bytes)
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Record {
/// BLAKE3 checksum of the payload.
pub checksum: [u8; 32],
/// The actual data payload.
pub payload: Vec<u8>,
}
impl Record {
/// Create a new record from a payload, calculating the checksum.
pub fn new(payload: Vec<u8>) -> Self {
let checksum = blake3::hash(&payload).into();
Self { checksum, payload }
}
/// Calculate the on-disk size of this record.
pub fn disk_size(&self) -> u64 {
(32 + 4 + self.payload.len()) as u64
}
/// Write the record to a writer.
pub fn write_to<W: Write>(&self, writer: &mut W) -> Result<()> {
writer.write_all(&self.checksum).map_err(QuarantineError::IoGeneric)?;
writer
.write_u32::<LittleEndian>(self.payload.len() as u32)
.map_err(QuarantineError::IoGeneric)?;
writer.write_all(&self.payload).map_err(QuarantineError::IoGeneric)?;
Ok(())
}
/// Read a record from a reader and verify its checksum.
pub fn read_from<R: Read>(reader: &mut R) -> Result<Self> {
let mut checksum = [0u8; 32];
reader.read_exact(&mut checksum).map_err(QuarantineError::IoGeneric)?;
let len = reader.read_u32::<LittleEndian>().map_err(QuarantineError::IoGeneric)?;
if len as usize > MAX_RECORD_SIZE {
return Err(QuarantineError::IoGeneric(std::io::Error::new(
std::io::ErrorKind::InvalidData,
format!("Record too large: {} bytes", len),
)));
}
let mut payload = vec![0u8; len as usize];
reader.read_exact(&mut payload).map_err(QuarantineError::IoGeneric)?;
// Verify checksum
let calculated: [u8; 32] = blake3::hash(&payload).into();
if checksum != calculated {
return Err(QuarantineError::IoGeneric(std::io::Error::new(
std::io::ErrorKind::InvalidData,
"Checksum mismatch",
)));
}
Ok(Self { checksum, payload })
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::io::Cursor;
#[test]
fn test_header_roundtrip() {
let header = FileHeader::new();
let mut buffer = Vec::new();
header.write_to(&mut buffer).unwrap();
assert_eq!(buffer.len(), HEADER_SIZE);
let mut reader = Cursor::new(buffer);
let read_header = FileHeader::read_from(&mut reader).unwrap();
assert_eq!(header, read_header);
}
#[test]
fn test_record_roundtrip() {
let payload = b"test payload data".to_vec();
let record = Record::new(payload.clone());
let mut buffer = Vec::new();
record.write_to(&mut buffer).unwrap();
assert_eq!(buffer.len() as u64, record.disk_size());
let mut reader = Cursor::new(buffer);
let read_record = Record::read_from(&mut reader).unwrap();
assert_eq!(record, read_record);
assert_eq!(read_record.payload, payload);
}
#[test]
fn test_record_checksum_validation() {
let payload = b"test data".to_vec();
let record = Record::new(payload);
let mut buffer = Vec::new();
record.write_to(&mut buffer).unwrap();
// Corrupt the payload in the buffer
let len = buffer.len();
buffer[len - 1] ^= 0xFF; // Flip bits in the last byte
let mut reader = Cursor::new(buffer);
let result = Record::read_from(&mut reader);
assert!(result.is_err());
assert_eq!(result.unwrap_err().to_string(), "Checksum mismatch");
}
}

143
crates/stemedb-wal/src/journal.rs Normal file
View File

@ -0,0 +1,143 @@
use crate::durability::{DurabilityLevel, FsyncGuard};
use crate::error::{QuarantineError, Result};
use crate::format::{FileHeader, Record, HEADER_SIZE};
use std::fs::{self, File, OpenOptions};
use std::io::{BufReader, Seek, SeekFrom};
use std::path::{Path, PathBuf};
use tracing::{debug, info, instrument, warn};
/// The main quarantine journal.
///
/// Provides append-only storage with crash recovery and fsync guarantees.
pub struct Journal {
data_dir: PathBuf,
current_file: Option<FsyncGuard>,
current_offset: u64,
durability: DurabilityLevel,
}
impl Journal {
/// Open or create a journal in the specified directory.
#[instrument(skip_all, fields(data_dir = %data_dir.as_ref().display()))]
pub fn open(data_dir: impl AsRef<Path>) -> Result<Self> {
let data_dir = data_dir.as_ref().to_path_buf();
fs::create_dir_all(&data_dir).map_err(|e| QuarantineError::io(&data_dir, e))?;
let mut journal = Self {
data_dir,
current_file: None,
current_offset: 0,
durability: DurabilityLevel::Immediate,
};
journal.recover()?;
info!(offset = journal.current_offset, "Journal opened");
Ok(journal)
}
/// Set durability level.
pub fn with_durability(mut self, level: DurabilityLevel) -> Self {
self.durability = level;
self
}
/// Append a record to the journal.
#[instrument(skip(self, payload), fields(payload_len = payload.len()))]
pub fn append(&mut self, payload: Vec<u8>) -> Result<u64> {
if self.current_file.is_none() {
self.open_current_file()?;
}
let record = Record::new(payload);
let mut buf = Vec::with_capacity(record.disk_size() as usize);
record.write_to(&mut buf)?;
let guard = self.current_file.as_mut().ok_or_else(|| {
QuarantineError::IoGeneric(std::io::Error::other("Journal file not open"))
})?;
guard.write(&buf)?;
let offset = self.current_offset;
self.current_offset += record.disk_size();
debug!(offset, disk_size = record.disk_size(), "Record appended");
Ok(offset)
}
/// Read a record at the given offset.
#[instrument(skip(self))]
pub fn read(&self, offset: u64) -> Result<Record> {
let path = self.current_file_path();
let mut file = File::open(&path).map_err(|e| QuarantineError::io(&path, e))?;
file.seek(SeekFrom::Start(offset)).map_err(|e| QuarantineError::io(&path, e))?;
let mut reader = BufReader::new(file);
Record::read_from(&mut reader)
}
/// Recover state from disk.
#[instrument(skip(self))]
fn recover(&mut self) -> Result<()> {
let path = self.current_file_path();
if !path.exists() {
debug!("No existing WAL file, starting fresh");
return Ok(());
}
let file = File::open(&path).map_err(|e| QuarantineError::io(&path, e))?;
let len = file.metadata().map_err(|e| QuarantineError::io(&path, e))?.len();
// Basic recovery: validate header and set offset to end
// TODO: Implement full scan and truncate of partial records
if len >= HEADER_SIZE as u64 {
let mut reader = BufReader::new(file);
let _header = FileHeader::read_from(&mut reader)?;
self.current_offset = len;
info!(file_size = len, "Recovered existing WAL");
} else {
// Corrupt or empty, start over
warn!(file_size = len, "WAL file too small, resetting");
self.current_offset = 0;
}
Ok(())
}
fn current_file_path(&self) -> PathBuf {
self.data_dir.join("00000000.wal")
}
#[instrument(skip(self), fields(path = %self.current_file_path().display()))]
fn open_current_file(&mut self) -> Result<()> {
let path = self.current_file_path();
let file = OpenOptions::new()
.create(true)
.read(true)
.write(true)
.truncate(false) // Never truncate existing WAL files on open
.open(&path)
.map_err(|e| QuarantineError::io(&path, e))?;
let mut guard = FsyncGuard::new(file, path.clone(), self.durability);
guard.lock_exclusive()?;
let len = guard.file().metadata().map_err(|e| QuarantineError::io(&path, e))?.len();
if len == 0 {
let header = FileHeader::new();
let mut buf = Vec::with_capacity(HEADER_SIZE);
header.write_to(&mut buf)?;
guard.write(&buf)?;
self.current_offset = HEADER_SIZE as u64;
debug!("Created new WAL file with header");
} else {
// Seek to end of file for append operations
guard.file_mut().seek(SeekFrom::End(0)).map_err(|e| QuarantineError::io(&path, e))?;
self.current_offset = len;
debug!(file_size = len, "Opened existing WAL file");
}
self.current_file = Some(guard);
Ok(())
}
}

27
crates/stemedb-wal/src/lib.rs Normal file
View File

@ -0,0 +1,27 @@
//! Write-Ahead Log (WAL) and durability primitives for Episteme.
//!
//! This crate provides the foundational durability layer, ensuring that
//! assertions are safely persisted to disk before being acknowledged.
//!
//! # Crash Recovery
//!
//! The WAL provides crash recovery guarantees via immediate fsync. When a
//! record is appended with `DurabilityLevel::Immediate` (the default), it
//! is guaranteed to survive process crashes or power failures.
//!
//! See the `recovery` module for integration tests proving these guarantees.
pub mod durability;
/// Error types and Result wrapper for WAL operations.
pub mod error;
/// Binary format for WAL records and headers.
pub mod format;
/// The main Journal API.
pub mod journal;
/// Crash recovery integration tests.
mod recovery;
pub use durability::{DurabilityLevel, FsyncGuard};
pub use error::{QuarantineError, Result};
pub use format::{FileHeader, Record, HEADER_SIZE};
pub use journal::Journal;

198
crates/stemedb-wal/src/recovery.rs Normal file
View File

@ -0,0 +1,198 @@
//! Crash recovery integration tests for the WAL.
//!
//! These tests verify that the Write-Ahead Log survives crashes (simulated by
//! dropping the Journal and reopening it) without data loss.
//!
//! # Test Strategy
//!
//! We cannot truly simulate a power failure in a unit test, but we can:
//! 1. Write data with immediate fsync (ensuring it hits disk)
//! 2. Drop the Journal (simulating process termination)
//! 3. Reopen the Journal (simulating restart)
//! 4. Verify all data is present and readable
//!
//! This proves the durability guarantees of the WAL.
#[cfg(test)]
mod tests {
use crate::format::HEADER_SIZE;
use crate::journal::Journal;
use tempfile::tempdir;
/// Test: Single record survives Journal close and reopen.
///
/// This is the fundamental crash recovery guarantee:
/// After fsync completes, data is durable.
#[test]
fn test_single_record_crash_recovery() {
let dir = tempdir().expect("Failed to create temp dir");
let wal_path = dir.path().join("wal");
let payload = b"critical assertion data".to_vec();
let offset: u64;
// Phase 1: Write and "crash" (drop journal)
{
let mut journal = Journal::open(&wal_path).expect("Failed to open journal");
offset = journal.append(payload.clone()).expect("Failed to append");
// Journal dropped here - simulates crash/restart
}
// Phase 2: Recovery - reopen and verify
{
let journal = Journal::open(&wal_path).expect("Failed to reopen journal");
let record = journal.read(offset).expect("Failed to read after recovery");
assert_eq!(record.payload, payload, "Data should survive restart");
}
}
/// Test: Multiple records survive crash and are readable in order.
#[test]
fn test_multiple_records_crash_recovery() {
let dir = tempdir().expect("Failed to create temp dir");
let wal_path = dir.path().join("wal");
let records = vec![
b"assertion 1: Tesla revenue is $96.7B".to_vec(),
b"assertion 2: Apple revenue is $394B".to_vec(),
b"assertion 3: Microsoft revenue is $211B".to_vec(),
];
let mut offsets = Vec::new();
// Phase 1: Write multiple records and "crash"
{
let mut journal = Journal::open(&wal_path).expect("Failed to open journal");
for payload in &records {
let offset = journal.append(payload.clone()).expect("Failed to append");
offsets.push(offset);
}
// Journal dropped here
}
// Phase 2: Recovery - verify all records
{
let journal = Journal::open(&wal_path).expect("Failed to reopen journal");
for (i, offset) in offsets.iter().enumerate() {
let record = journal.read(*offset).expect("Failed to read");
assert_eq!(record.payload, records[i], "Record {} should match", i);
}
}
}
/// Test: Journal can continue appending after recovery.
///
/// This verifies that recovery properly sets the write offset.
#[test]
fn test_append_after_recovery() {
let dir = tempdir().expect("Failed to create temp dir");
let wal_path = dir.path().join("wal");
let first_payload = b"first record".to_vec();
let first_offset: u64;
// Phase 1: Write first record and "crash"
{
let mut journal = Journal::open(&wal_path).expect("Failed to open journal");
first_offset = journal.append(first_payload.clone()).expect("Failed to append");
}
// Phase 2: Recover and append more
let second_payload = b"second record after recovery".to_vec();
let second_offset: u64;
{
let mut journal = Journal::open(&wal_path).expect("Failed to reopen journal");
second_offset = journal.append(second_payload.clone()).expect("Failed to append");
// Verify second offset is after first
assert!(
second_offset > first_offset,
"New records should be appended after existing data"
);
}
// Phase 3: Verify both records after another "crash"
{
let journal = Journal::open(&wal_path).expect("Failed to reopen journal again");
let first = journal.read(first_offset).expect("Failed to read first");
let second = journal.read(second_offset).expect("Failed to read second");
assert_eq!(first.payload, first_payload);
assert_eq!(second.payload, second_payload);
}
}
/// Test: Large payloads survive crash recovery.
///
/// Ensures the WAL handles larger data correctly, not just small test payloads.
#[test]
fn test_large_payload_crash_recovery() {
let dir = tempdir().expect("Failed to create temp dir");
let wal_path = dir.path().join("wal");
// Create a 1MB payload (simulating a large assertion with embeddings)
let large_payload: Vec<u8> = (0..1024 * 1024).map(|i| (i % 256) as u8).collect();
let offset: u64;
// Write and "crash"
{
let mut journal = Journal::open(&wal_path).expect("Failed to open journal");
offset = journal.append(large_payload.clone()).expect("Failed to append large payload");
}
// Recover and verify
{
let journal = Journal::open(&wal_path).expect("Failed to reopen journal");
let record = journal.read(offset).expect("Failed to read large payload");
assert_eq!(record.payload.len(), large_payload.len());
assert_eq!(record.payload, large_payload, "Large payload should survive");
}
}
/// Test: Empty WAL directory is handled gracefully.
#[test]
fn test_fresh_start_no_existing_wal() {
let dir = tempdir().expect("Failed to create temp dir");
let wal_path = dir.path().join("fresh_wal");
// Opening a fresh directory should work
let mut journal = Journal::open(&wal_path).expect("Failed to open fresh journal");
// Should be able to write immediately
let offset = journal.append(b"first record".to_vec()).expect("Failed to append");
assert_eq!(offset, HEADER_SIZE as u64, "First record should start after header");
}
/// Test: Repeated crash-recovery cycles work correctly.
///
/// Simulates a flaky system that crashes and recovers multiple times.
#[test]
fn test_repeated_crash_recovery_cycles() {
let dir = tempdir().expect("Failed to create temp dir");
let wal_path = dir.path().join("wal");
let mut all_offsets = Vec::new();
let num_cycles = 5;
let records_per_cycle = 3;
for cycle in 0..num_cycles {
// Write some records
{
let mut journal = Journal::open(&wal_path).expect("Failed to open journal");
for i in 0..records_per_cycle {
let payload = format!("cycle {} record {}", cycle, i).into_bytes();
let offset = journal.append(payload).expect("Failed to append");
all_offsets.push((offset, cycle, i));
}
// "Crash" - drop journal
}
}
// Final verification - all records from all cycles should be present
{
let journal = Journal::open(&wal_path).expect("Failed to reopen journal");
for (offset, cycle, i) in &all_offsets {
let record = journal.read(*offset).expect("Failed to read");
let expected = format!("cycle {} record {}", cycle, i).into_bytes();
assert_eq!(record.payload, expected, "Record from cycle {} should survive", cycle);
}
}
}
}

2
docs/presentations/.gitignore vendored Normal file
View File

@ -0,0 +1,2 @@
node_modules/
dist/

116
docs/presentations/README.md Normal file
View File

@ -0,0 +1,116 @@
# Episteme Presentations
Data-driven presentations for Episteme. Define sequences in YAML, generate Mermaid diagrams and Reveal.js slides.
## Quick Start
```bash
npm install
npm run dev # Generate + serve
```
Then open http://localhost:3000
## Architecture
```
presentations/
├── data/
│ └── agile-agent-team.yaml # Source of truth
├── generated/ # Auto-generated (gitignored)
│ ├── *.mmd # Mermaid diagrams
│ └── *.json # Reveal.js data
├── reveal/
│ ├── index.html # Reveal.js shell
│ ├── theme.css # Clean black dark theme
│ └── renderer.js # Reads JSON, renders slides
└── scripts/
└── generate.ts # YAML → Mermaid + JSON
```
## Commands
| Command | Description |
|---------|-------------|
| `npm run generate` | Generate Mermaid and JSON from YAML |
| `npm run serve` | Serve the reveal directory |
| `npm run dev` | Generate + serve |
## Data Schema
See `data/agile-agent-team.yaml` for the full schema. Key elements:
### Actors
```yaml
actors:
research_agent:
id: RA
label: "Research Agent"
color: "#3B82F6"
```
### Sequences
```yaml
sequences:
- id: catastrophe
title: "The Catastrophe"
subtitle: "When Proposals Look Like Decisions"
steps:
- from: research_agent
to: episteme
action: assert
label: "Store RFC finding"
data: { ... }
note: "RFC proposes ES256. Stored as PROPOSED."
danger: true # Visual indicator
callout: "lifecycle: proposed" # Highlighted badge
```
### Step Properties
| Property | Type | Description |
|----------|------|-------------|
| `from` | string | Actor key |
| `to` | string | Actor key |
| `action` | string | Action type (assert, query, response, etc.) |
| `label` | string | Message label |
| `data` | object | Structured data to display |
| `note` | string | Explanation text |
| `callout` | string | Highlighted badge text |
| `danger` | boolean | Red styling |
| `warning` | boolean | Amber styling |
| `success` | boolean | Green styling |
## Mermaid Output
Each sequence generates a standalone `.mmd` file. Use with:
- GitHub markdown (renders automatically)
- Mermaid CLI: `mmdc -i file.mmd -o file.svg`
- Mermaid Live Editor: https://mermaid.live
## Embedding in Public Site
The Reveal.js presentation is self-contained. To embed:
1. Copy `reveal/` directory to your site
2. Copy `generated/*.json` to your site
3. Update `PRESENTATION_DATA_URL` in index.html
Or iframe embed:
```html
<iframe src="/presentations/reveal/index.html" width="100%" height="600"></iframe>
```
## Theme Customization
Edit `reveal/theme.css`. Key variables:
```css
:root {
--bg-primary: #000000;
--bg-card: #111111;
--accent-episteme: #FBBF24;
--font-sans: 'Inter', sans-serif;
}
```

578
docs/presentations/data/agile-agent-team.yaml Normal file
View File

@ -0,0 +1,578 @@
meta:
id: agile-agent-team
title: "Agile AI Agent Team"
subtitle: "Knowledge Coordination with Episteme"
version: "1.0.0"
actors:
research_agent:
id: RA
label: "Research Agent"
description: "Ingests external sources, stores with uncertainty"
color: "#3B82F6"
lead_orchestrator:
id: LO
label: "Lead Orchestrator"
description: "Coordinates team, routes work based on knowledge"
color: "#8B5CF6"
implementation_agent:
id: IA
label: "Implementation Agent"
description: "Writes code against current patterns"
color: "#10B981"
deploy_agent:
id: DA
label: "Deploy Agent"
description: "Deploys configurations to production"
color: "#F59E0B"
episteme:
id: E
label: "Episteme"
description: "Probabilistic knowledge graph"
color: "#FBBF24"
human:
id: H
label: "Human Supervisor"
description: "Reviews decisions, corrects errors"
color: "#EC4899"
gardener:
id: G
label: "Gardener"
description: "Background worker for TrustRank propagation"
color: "#14B8A6"
production:
id: P
label: "Production"
description: "Live system"
color: "#EF4444"
sequences:
# ============================================
# SEQUENCE 1: THE CATASTROPHE
# ============================================
- id: catastrophe
title: "The Catastrophe"
subtitle: "When Proposals Look Like Decisions"
description: |
A 47-minute production outage because an AI agent couldn't
distinguish a proposal from an approved decision.
steps:
- id: cat-1
from: research_agent
to: episteme
action: assert
label: "Store RFC finding"
data:
subject: "auth/jwt"
predicate: "signing_algorithm"
object: "ES256"
lifecycle: "proposed"
confidence: 0.75
source: "security-rfc-2024.md"
note: "RFC proposes ES256. Stored as PROPOSED."
- id: cat-2
from: lead_orchestrator
to: episteme
action: query
label: "What's the JWT algorithm?"
data:
subject: "auth/jwt"
predicate: "signing_algorithm"
lens: "recency"
danger: true
note: "No lifecycle filter. This is the bug."
callout: "lifecycle: ???"
- id: cat-3
from: episteme
to: lead_orchestrator
action: response
label: "ES256 (conf: 0.87)"
data:
value: "ES256"
confidence: 0.87
lifecycle: "proposed"
danger: true
note: "Returns the proposal. Most recent wins."
- id: cat-4
from: lead_orchestrator
to: implementation_agent
action: delegate
label: "Use ES256 for JWT"
data:
algorithm: "ES256"
confidence: 0.87
danger: true
note: "Orchestrator passes 'truth' downstream."
- id: cat-5
from: implementation_agent
to: deploy_agent
action: handoff
label: "Config ready"
data:
jwt_algorithm: "ES256"
danger: true
note: "Code written against ES256."
- id: cat-6
from: deploy_agent
to: production
action: deploy
label: "Deploy JWT config"
data:
config:
algorithm: "ES256"
danger: true
note: "Deployed with confidence. Tests passed."
- id: cat-7
from: production
to: production
action: error
label: "401 Unauthorized"
data:
error: "JWT signature validation failed"
expected: "RS256"
received: "ES256"
danger: true
note: "Auth service expects RS256. Every token fails."
callout: "3:00 AM - Pager fires"
# ============================================
# SEQUENCE 2: THE CORRECT PATH
# ============================================
- id: correct_path
title: "The Correct Path"
subtitle: "With Lifecycle Filtering"
description: |
The same scenario, but with Episteme's lifecycle filtering.
Proposals stay proposals. Approved stays approved.
steps:
- id: cor-1
from: research_agent
to: episteme
action: assert
label: "Store RFC finding"
data:
subject: "auth/jwt"
predicate: "signing_algorithm"
object: "ES256"
lifecycle: "proposed"
confidence: 0.75
note: "Same RFC. Still stored as PROPOSED."
- id: cor-2
from: lead_orchestrator
to: episteme
action: query
label: "What's the APPROVED JWT algorithm?"
data:
subject: "auth/jwt"
predicate: "signing_algorithm"
lens: "authority"
lifecycle: "approved"
success: true
note: "Lifecycle filter: approved only."
callout: "lifecycle: approved"
- id: cor-3
from: episteme
to: lead_orchestrator
action: response
label: "RS256 (conf: 0.92)"
data:
value: "RS256"
confidence: 0.92
lifecycle: "approved"
source: "production-config.yaml"
success: true
note: "Returns the approved decision. Proposal excluded."
- id: cor-4
from: lead_orchestrator
to: implementation_agent
action: delegate
label: "Use RS256 for JWT"
data:
algorithm: "RS256"
confidence: 0.92
success: true
note: "Correct algorithm propagates."
- id: cor-5
from: implementation_agent
to: episteme
action: query
label: "Pre-flight constraint check"
data:
context: "auth_jwt"
lens: "constraints"
success: true
note: "Check for forbidden patterns before coding."
- id: cor-6
from: episteme
to: implementation_agent
action: response
label: "No violations"
data:
constraints: []
clear_to_proceed: true
success: true
note: "No negative constraints for RS256."
- id: cor-7
from: implementation_agent
to: deploy_agent
action: handoff
label: "Config ready"
data:
jwt_algorithm: "RS256"
success: true
note: "Code written against RS256."
- id: cor-8
from: deploy_agent
to: production
action: deploy
label: "Deploy JWT config"
data:
config:
algorithm: "RS256"
success: true
note: "Deployed. Matches production expectation."
- id: cor-9
from: production
to: production
action: success
label: "200 OK"
data:
status: "healthy"
tokens_validated: true
success: true
note: "Auth works. No pager. Sleep continues."
# ============================================
# SEQUENCE 3: THE CORRECTION LOOP
# ============================================
- id: correction_loop
title: "The Correction Loop"
subtitle: "Tracing, Fixing, Learning"
description: |
Post-incident: Human traces the bug, corrects the record,
and the Gardener ensures agents learn from the mistake.
steps:
- id: fix-1
from: human
to: episteme
action: query
label: "Trace deploy agent queries"
data:
type: "audit"
agent_id: "deploy_agent"
time_range: "-6h"
subject: "auth/*"
note: "SRE investigates: what did the agent believe?"
- id: fix-2
from: episteme
to: human
action: response
label: "Query audit trail"
data:
query_id: "q_7f3a2b"
timestamp: "2024-01-15T21:03:47Z"
subject: "auth/jwt"
predicate: "signing_algorithm"
lens: "recency"
lifecycle_filter: null
result: "ES256"
contributing:
- hash: "rfc_2024_001"
lifecycle: "proposed"
weight: 0.9
danger: true
note: "Found it: no lifecycle filter. Proposal returned."
callout: "lifecycle_filter: null"
- id: fix-3
from: human
to: episteme
action: supersede
label: "Mark assertion incorrect"
data:
hash: "rfc_2024_001"
reason: "Proposal treated as approved decision"
type: "RequiresReview"
note: "Supersede the problematic assertion."
- id: fix-4
from: episteme
to: gardener
action: trigger
label: "Correction event"
data:
superseded_hash: "rfc_2024_001"
superseding_agent: "human_supervisor"
affected_queries: ["q_7f3a2b"]
note: "Gardener wakes up."
- id: fix-5
from: gardener
to: episteme
action: update
label: "TrustRank back-propagation"
data:
agent_id: "lead_orchestrator"
topic: "auth/jwt"
delta: -0.15
reason: "Query returned proposal as decision"
warning: true
note: "Lead Orchestrator's reputation on auth topics drops."
- id: fix-6
from: gardener
to: episteme
action: update
label: "Store negative constraint"
data:
subject: "auth/jwt"
predicate: "query_pattern"
must_use: "lifecycle=approved"
forbidden: "lifecycle=null"
reason: "Proposals must not be treated as decisions"
success: true
note: "Future queries will see this constraint."
# ============================================
# SEQUENCE 4: PERSISTENT LEARNING
# ============================================
- id: persistent_learning
title: "Persistent Learning"
subtitle: "Fixing the Optimization Conflict"
description: |
1 month later. New session. Empty context window.
But the lesson persists.
steps:
- id: learn-1
from: human
to: episteme
action: assert
label: "Correct the agent"
data:
subject: "Project_X_Http_Client"
predicate: "must_use_library"
object: "axios"
meta:
forbidden_alternative: "requests"
reason: "requests library deprecated for this project"
confidence: 1.0
lifecycle: "approved"
note: "Human stores correction with forbidden alternative."
callout: "Day 1"
- id: learn-2
from: episteme
to: gardener
action: trigger
label: "Negative constraint stored"
data:
type: "correction"
agent_corrected: "implementation_agent"
note: "Gardener sees the correction event."
- id: learn-3
from: gardener
to: episteme
action: update
label: "TrustRank penalty"
data:
agent_id: "implementation_agent"
topic: "http_libraries"
delta: -0.20
warning: true
note: "Implementation Agent's confidence on HTTP libs drops."
callout: "Learns from mistake"
- id: learn-4
from: implementation_agent
to: implementation_agent
action: start
label: "New session begins"
data:
context_window: "empty"
system_prompt: "default"
note: "30 days later. Fresh context. No memory of correction."
callout: "Day 30 - New Session"
- id: learn-5
from: implementation_agent
to: episteme
action: query
label: "Pre-flight constraint check"
data:
context: "python_http"
lens: "constraints"
success: true
note: "Before writing code, check constraints."
callout: "Automatic pre-flight"
- id: learn-6
from: episteme
to: implementation_agent
action: response
label: "Constraint found"
data:
constraints:
- subject: "Project_X_Http_Client"
must_use: "axios"
forbidden: "requests"
reason: "requests library deprecated for this project"
confidence: 1.0
success: true
note: "The correction from Day 1 is still there."
callout: "Survived context window!"
- id: learn-7
from: implementation_agent
to: implementation_agent
action: generate
label: "Write code with axios"
data:
import: "axios"
avoided: "requests"
success: true
note: "Agent uses axios. Constraint honored."
- id: learn-8
from: episteme
to: episteme
action: resurrect
label: "Resurrection"
data:
constraint_hash: "axios_constraint"
last_verified: "now"
confidence: 1.0
success: true
note: "Constraint used successfully. Stays fresh forever."
callout: "Resurrection"
# ============================================
# SEQUENCE 5: TIME TRAVEL DEBUGGING
# ============================================
- id: time_travel
title: "Time Travel Debugging"
subtitle: "What Did We Believe Then?"
description: |
3:00 AM incident investigation. The SRE needs to know
what the system believed 6 hours ago, not now.
steps:
- id: tt-1
from: human
to: episteme
action: query
label: "What's the current JWT algorithm?"
data:
subject: "auth/jwt"
predicate: "signing_algorithm"
lifecycle: "approved"
note: "Current state shows RS256 (post-fix)."
- id: tt-2
from: episteme
to: human
action: response
label: "RS256 (current)"
data:
value: "RS256"
confidence: 0.95
note: "This is useless for debugging. We need history."
- id: tt-3
from: human
to: episteme
action: query
label: "What did we believe at 9pm?"
data:
subject: "auth/jwt"
predicate: "signing_algorithm"
as_of: "2024-01-15T21:00:00Z"
note: "Time-travel query."
callout: "as_of: 9pm"
- id: tt-4
from: episteme
to: human
action: response
label: "ES256 (at 9pm)"
data:
value: "ES256"
confidence: 0.87
lifecycle: "proposed"
as_of: "2024-01-15T21:00:00Z"
danger: true
note: "At 9pm, the system believed ES256 was correct."
callout: "Found the state at incident time"
- id: tt-5
from: human
to: episteme
action: query
label: "What changed in last 24h?"
data:
type: "diff"
subject: "auth/jwt"
from: "-24h"
note: "Diff view for change analysis."
- id: tt-6
from: episteme
to: human
action: response
label: "Diff result"
data:
added:
- hash: "rfc_2024_001"
value: "ES256"
lifecycle: "proposed"
added_at: "2024-01-15T14:30:00Z"
unchanged:
- hash: "prod_config_v2"
value: "RS256"
lifecycle: "approved"
success: true
note: "Clear view: RFC added at 2:30pm caused the issue."
annotations:
danger:
color: "#EF4444"
label: "Problem"
icon: "alert-triangle"
warning:
color: "#F59E0B"
label: "Warning"
icon: "alert-circle"
success:
color: "#10B981"
label: "Success"
icon: "check-circle"
info:
color: "#3B82F6"
label: "Info"
icon: "info"

30
docs/presentations/generated/agile-agent-team-catastrophe.mmd Normal file
View File

@ -0,0 +1,30 @@
sequenceDiagram
%% The Catastrophe: When Proposals Look Like Decisions
participant RA as Research Agent
participant E as Episteme
participant LO as Lead Orchestrator
participant IA as Implementation Agent
participant DA as Deploy Agent
participant P as Production
RA->>E: Store RFC finding
Note right of E: RFC proposes ES256. Stored as PROPOSED.
LO->>E: What's the JWT algorithm?
Note right of E: ⚠️ No lifecycle filter. This is the bug.
E-->>LO: ES256 (conf: 0.87)
Note right of LO: ⚠️ Returns the proposal. Most recent wins.
LO->>IA: Use ES256 for JWT
Note right of IA: ⚠️ Orchestrator passes 'truth' downstream.
IA->>DA: Config ready
Note right of DA: ⚠️ Code written against ES256.
DA->>P: Deploy JWT config
Note right of P: ⚠️ Deployed with confidence. Tests passed.
P->>P: 401 Unauthorized
Note over P: ⚠️ Auth service expects RS256. Every token fails.

36
docs/presentations/generated/agile-agent-team-correct_path.mmd Normal file
View File

@ -0,0 +1,36 @@
sequenceDiagram
%% The Correct Path: With Lifecycle Filtering
participant RA as Research Agent
participant E as Episteme
participant LO as Lead Orchestrator
participant IA as Implementation Agent
participant DA as Deploy Agent
participant P as Production
RA->>E: Store RFC finding
Note right of E: Same RFC. Still stored as PROPOSED.
LO->>E: What's the APPROVED JWT algorithm?
Note right of E: ✓ Lifecycle filter: approved only.
E-->>LO: RS256 (conf: 0.92)
Note right of LO: ✓ Returns the approved decision. Proposal excluded.
LO->>IA: Use RS256 for JWT
Note right of IA: ✓ Correct algorithm propagates.
IA->>E: Pre-flight constraint check
Note right of E: ✓ Check for forbidden patterns before coding.
E-->>IA: No violations
Note right of IA: ✓ No negative constraints for RS256.
IA->>DA: Config ready
Note right of DA: ✓ Code written against RS256.
DA->>P: Deploy JWT config
Note right of P: ✓ Deployed. Matches production expectation.
P->>P: 200 OK
Note over P: ✓ Auth works. No pager. Sleep continues.

24
docs/presentations/generated/agile-agent-team-correction_loop.mmd Normal file
View File

@ -0,0 +1,24 @@
sequenceDiagram
%% The Correction Loop: Tracing, Fixing, Learning
participant H as Human Supervisor
participant E as Episteme
participant G as Gardener
H->>E: Trace deploy agent queries
Note right of E: SRE investigates: what did the agent believe?
E-->>H: Query audit trail
Note right of H: ⚠️ Found it: no lifecycle filter. Proposal returned.
H->>E: Mark assertion incorrect
Note right of E: Supersede the problematic assertion.
E->>G: Correction event
Note right of G: Gardener wakes up.
G->>E: TrustRank back-propagation
Note right of E: Lead Orchestrator's reputation on auth topics drops.
G->>E: Store negative constraint
Note right of E: ✓ Future queries will see this constraint.

31
docs/presentations/generated/agile-agent-team-persistent_learning.mmd Normal file
View File

@ -0,0 +1,31 @@
sequenceDiagram
%% Persistent Learning: Fixing the Optimization Conflict
participant H as Human Supervisor
participant E as Episteme
participant G as Gardener
participant IA as Implementation Agent
H->>E: Correct the agent
Note right of E: Human stores correction with forbidden alternative.
E->>G: Negative constraint stored
Note right of G: Gardener sees the correction event.
G->>E: TrustRank penalty
Note right of E: Implementation Agent's confidence on HTTP libs drops.
IA->>IA: New session begins
Note over IA: 30 days later. Fresh context. No memory of correction.
IA->>E: Pre-flight constraint check
Note right of E: ✓ Before writing code, check constraints.
E-->>IA: Constraint found
Note right of IA: ✓ The correction from Day 1 is still there.
IA->>IA: Write code with axios
Note over IA: ✓ Agent uses axios. Constraint honored.
E->>E: Resurrection
Note over E: ✓ Constraint used successfully. Stays fresh forever.

23
docs/presentations/generated/agile-agent-team-time_travel.mmd Normal file
View File

@ -0,0 +1,23 @@
sequenceDiagram
%% Time Travel Debugging: What Did We Believe Then?
participant H as Human Supervisor
participant E as Episteme
H->>E: What's the current JWT algorithm?
Note right of E: Current state shows RS256 (post-fix).
E-->>H: RS256 (current)
Note right of H: This is useless for debugging. We need history.
H->>E: What did we believe at 9pm?
Note right of E: Time-travel query.
E-->>H: ES256 (at 9pm)
Note right of H: ⚠️ At 9pm, the system believed ES256 was correct.
H->>E: What changed in last 24h?
Note right of E: Diff view for change analysis.
E-->>H: Diff result
Note right of H: ✓ Clear view: RFC added at 2:30pm caused the issue.

1295
docs/presentations/generated/agile-agent-team.json Normal file
View File

File diff suppressed because it is too large Load Diff

160
docs/presentations/generated/agile-agent-team.mmd Normal file
View File

@ -0,0 +1,160 @@
sequenceDiagram
%% The Catastrophe: When Proposals Look Like Decisions
participant RA as Research Agent
participant E as Episteme
participant LO as Lead Orchestrator
participant IA as Implementation Agent
participant DA as Deploy Agent
participant P as Production
RA->>E: Store RFC finding
Note right of E: RFC proposes ES256. Stored as PROPOSED.
LO->>E: What's the JWT algorithm?
Note right of E: ⚠️ No lifecycle filter. This is the bug.
E-->>LO: ES256 (conf: 0.87)
Note right of LO: ⚠️ Returns the proposal. Most recent wins.
LO->>IA: Use ES256 for JWT
Note right of IA: ⚠️ Orchestrator passes 'truth' downstream.
IA->>DA: Config ready
Note right of DA: ⚠️ Code written against ES256.
DA->>P: Deploy JWT config
Note right of P: ⚠️ Deployed with confidence. Tests passed.
P->>P: 401 Unauthorized
Note over P: ⚠️ Auth service expects RS256. Every token fails.
---
sequenceDiagram
%% The Correct Path: With Lifecycle Filtering
participant RA as Research Agent
participant E as Episteme
participant LO as Lead Orchestrator
participant IA as Implementation Agent
participant DA as Deploy Agent
participant P as Production
RA->>E: Store RFC finding
Note right of E: Same RFC. Still stored as PROPOSED.
LO->>E: What's the APPROVED JWT algorithm?
Note right of E: ✓ Lifecycle filter: approved only.
E-->>LO: RS256 (conf: 0.92)
Note right of LO: ✓ Returns the approved decision. Proposal excluded.
LO->>IA: Use RS256 for JWT
Note right of IA: ✓ Correct algorithm propagates.
IA->>E: Pre-flight constraint check
Note right of E: ✓ Check for forbidden patterns before coding.
E-->>IA: No violations
Note right of IA: ✓ No negative constraints for RS256.
IA->>DA: Config ready
Note right of DA: ✓ Code written against RS256.
DA->>P: Deploy JWT config
Note right of P: ✓ Deployed. Matches production expectation.
P->>P: 200 OK
Note over P: ✓ Auth works. No pager. Sleep continues.
---
sequenceDiagram
%% The Correction Loop: Tracing, Fixing, Learning
participant H as Human Supervisor
participant E as Episteme
participant G as Gardener
H->>E: Trace deploy agent queries
Note right of E: SRE investigates: what did the agent believe?
E-->>H: Query audit trail
Note right of H: ⚠️ Found it: no lifecycle filter. Proposal returned.
H->>E: Mark assertion incorrect
Note right of E: Supersede the problematic assertion.
E->>G: Correction event
Note right of G: Gardener wakes up.
G->>E: TrustRank back-propagation
Note right of E: Lead Orchestrator's reputation on auth topics drops.
G->>E: Store negative constraint
Note right of E: ✓ Future queries will see this constraint.
---
sequenceDiagram
%% Persistent Learning: Fixing the Optimization Conflict
participant H as Human Supervisor
participant E as Episteme
participant G as Gardener
participant IA as Implementation Agent
H->>E: Correct the agent
Note right of E: Human stores correction with forbidden alternative.
E->>G: Negative constraint stored
Note right of G: Gardener sees the correction event.
G->>E: TrustRank penalty
Note right of E: Implementation Agent's confidence on HTTP libs drops.
IA->>IA: New session begins
Note over IA: 30 days later. Fresh context. No memory of correction.
IA->>E: Pre-flight constraint check
Note right of E: ✓ Before writing code, check constraints.
E-->>IA: Constraint found
Note right of IA: ✓ The correction from Day 1 is still there.
IA->>IA: Write code with axios
Note over IA: ✓ Agent uses axios. Constraint honored.
E->>E: Resurrection
Note over E: ✓ Constraint used successfully. Stays fresh forever.
---
sequenceDiagram
%% Time Travel Debugging: What Did We Believe Then?
participant H as Human Supervisor
participant E as Episteme
H->>E: What's the current JWT algorithm?
Note right of E: Current state shows RS256 (post-fix).
E-->>H: RS256 (current)
Note right of H: This is useless for debugging. We need history.
H->>E: What did we believe at 9pm?
Note right of E: Time-travel query.
E-->>H: ES256 (at 9pm)
Note right of H: ⚠️ At 9pm, the system believed ES256 was correct.
H->>E: What changed in last 24h?
Note right of E: Diff view for change analysis.
E-->>H: Diff result
Note right of H: ✓ Clear view: RFC added at 2:30pm caused the issue.

1653
docs/presentations/package-lock.json generated Normal file
View File

File diff suppressed because it is too large Load Diff

17
docs/presentations/package.json Normal file
View File

@ -0,0 +1,17 @@
{
"name": "episteme-presentations",
"version": "1.0.0",
"description": "Data-driven presentations for Episteme",
"scripts": {
"generate": "npx tsx scripts/generate.ts data/agile-agent-team.yaml",
"serve": "npx serve .",
"dev": "npm run generate && npm run serve"
},
"devDependencies": {
"@types/node": "^20.0.0",
"tsx": "^4.0.0",
"typescript": "^5.0.0",
"yaml": "^2.3.0",
"serve": "^14.0.0"
}
}

37
docs/presentations/reveal/index.html Normal file
View File

@ -0,0 +1,37 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Episteme - Agile AI Agent Team</title>
<!-- Reveal.js from CDN -->
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/reveal.js@5.1.0/dist/reveal.css">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/reveal.js@5.1.0/dist/theme/black.css">
<!-- Geist font (Vercel) -->
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/geist@1.2.0/dist/fonts/geist-sans/style.min.css">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/geist@1.2.0/dist/fonts/geist-mono/style.min.css">
<!-- Custom theme -->
<link rel="stylesheet" href="theme.css">
</head>
<body>
<div class="reveal">
<div class="slides" id="slides">
<!-- Slides injected by renderer.js -->
</div>
</div>
<!-- Reveal.js -->
<script src="https://cdn.jsdelivr.net/npm/reveal.js@5.1.0/dist/reveal.js"></script>
<!-- Presentation data -->
<script>
window.PRESENTATION_DATA_URL = '/generated/agile-agent-team.json';
</script>
<!-- Renderer -->
<script src="renderer.js"></script>
</body>
</html>

184
docs/presentations/reveal/renderer.js Normal file
View File

@ -0,0 +1,184 @@
/**
* Episteme Presentation Renderer
*
* Reads JSON data and renders Reveal.js slides
*/
(async function () {
// Load presentation data
const dataUrl = window.PRESENTATION_DATA_URL || '../generated/agile-agent-team.json';
let data;
try {
const response = await fetch(dataUrl);
data = await response.json();
} catch (error) {
console.error('Failed to load presentation data:', error);
document.getElementById('slides').innerHTML = `
<section class="slide-title">
<h1>Error Loading Presentation</h1>
<p class="subtitle">Could not load ${dataUrl}</p>
<p>Run: <code>npx ts-node scripts/generate.ts data/agile-agent-team.yaml</code></p>
</section>
`;
Reveal.initialize();
return;
}
const slidesContainer = document.getElementById('slides');
// Actor icon mapping (simple initials for now)
function getActorInitials(actor) {
return actor.label.split(' ').map(w => w[0]).join('').toUpperCase().slice(0, 2);
}
// Format JSON for display
function formatData(data) {
if (!data) return '';
return JSON.stringify(data, null, 2)
.replace(/"([^"]+)":/g, '<span class="json-key">"$1"</span>:')
.replace(/: "([^"]+)"/g, ': <span class="json-string">"$1"</span>')
.replace(/: (\d+\.?\d*)/g, ': <span class="json-number">$1</span>')
.replace(/: (true|false)/g, ': <span class="json-boolean">$1</span>')
.replace(/: (null)/g, ': <span class="json-null">$1</span>');
}
// Render slides
function renderSlides(slides) {
let sectionNumber = 0;
for (const slide of slides) {
const section = document.createElement('section');
switch (slide.type) {
case 'title':
section.classList.add('title-slide');
section.innerHTML = `
<div class="slide-title">
<h1 class="brand">stemedb</h1>
</div>
`;
break;
case 'section-title':
sectionNumber++;
section.innerHTML = `
<div class="slide-section-title">
<span class="section-number">Section ${sectionNumber}</span>
<h2>${slide.title}</h2>
<p class="subtitle">${slide.subtitle}</p>
<p class="description">${slide.description.trim()}</p>
</div>
`;
break;
case 'sequence':
section.setAttribute('data-auto-animate', '');
section.innerHTML = renderSequenceSlide(slide, data.actors);
break;
}
slidesContainer.appendChild(section);
}
}
// Render sequence slide
function renderSequenceSlide(slide, allActors) {
const actors = slide.actors;
const steps = slide.steps;
// Actor columns
const actorHtml = Object.values(actors).map(actor => `
<div class="actor">
<div class="actor-icon" style="background-color: ${actor.color}">
${getActorInitials(actor)}
</div>
<span class="actor-label">${actor.label}</span>
</div>
`).join('');
// Step cards as fragments
const stepsHtml = steps.map((step, index) => {
const statusClass = step.danger ? 'danger' : (step.success ? 'success' : (step.warning ? 'warning' : ''));
const fromActor = step.fromActor || actors[step.from] || { label: step.from };
const toActor = step.toActor || actors[step.to] || { label: step.to };
return `
<div class="step-card ${statusClass} fragment" data-fragment-index="${index}">
<div class="step-index">${index + 1}</div>
<div class="step-content">
<div class="step-header">
<div class="step-actors">
<span class="step-from">${fromActor.label}</span>
<span class="step-arrow">→</span>
<span class="step-to">${toActor.label}</span>
</div>
<span class="step-action">${step.action}</span>
</div>
<div class="step-label">${step.label}</div>
${step.note ? `<div class="step-note">${step.note}</div>` : ''}
${step.data ? `<div class="step-data"><pre>${formatData(step.data)}</pre></div>` : ''}
${step.callout ? `<div class="step-callout">${step.callout}</div>` : ''}
</div>
</div>
`;
}).join('');
return `
<div class="slide-sequence">
<div class="header">
<h3>${slide.title}</h3>
</div>
<div class="actors-row">
${actorHtml}
</div>
<div class="steps-container">
${stepsHtml}
</div>
</div>
`;
}
// Render slides
renderSlides(data.slides);
// Initialize Reveal.js
Reveal.initialize({
hash: true,
history: true,
controls: true,
progress: true,
center: true,
transition: 'fade',
transitionSpeed: 'fast',
// Keyboard shortcuts
keyboard: {
// Arrow keys for fragments within slide
39: 'next', // right
37: 'prev', // left
},
});
// Make step cards visible as fragments appear
Reveal.on('fragmentshown', event => {
event.fragment.classList.add('visible');
});
Reveal.on('fragmenthidden', event => {
event.fragment.classList.remove('visible');
});
// Show all fragments on slide enter if coming from later slide
Reveal.on('slidechanged', event => {
const currentSlide = event.currentSlide;
const fragments = currentSlide.querySelectorAll('.fragment');
// Reset visibility on slide change
fragments.forEach(frag => {
if (frag.classList.contains('visible')) {
frag.classList.add('visible');
}
});
});
})();

579
docs/presentations/reveal/theme.css Normal file
View File

@ -0,0 +1,579 @@
/**
* Episteme Presentation Theme
* Clean black dark, minimal aesthetic
*/
:root {
/* Core colors */
--bg-primary: #000000;
--bg-secondary: #0a0a0a;
--bg-card: #111111;
--bg-card-hover: #1a1a1a;
/* Text */
--text-primary: #ffffff;
--text-secondary: #a3a3a3;
--text-muted: #525252;
/* Borders */
--border-subtle: #262626;
--border-default: #404040;
/* Accents */
--accent-episteme: #FBBF24;
--accent-success: #10B981;
--accent-danger: #EF4444;
--accent-warning: #F59E0B;
--accent-info: #3B82F6;
/* Actor colors */
--actor-research: #3B82F6;
--actor-orchestrator: #8B5CF6;
--actor-implementation: #10B981;
--actor-deploy: #F59E0B;
--actor-human: #EC4899;
--actor-gardener: #14B8A6;
--actor-production: #EF4444;
/* Typography */
--font-sans: 'Geist Sans', -apple-system, BlinkMacSystemFont, sans-serif;
--font-mono: 'Geist Mono', ui-monospace, SFMono-Regular, Menlo, monospace;
/* Spacing */
--space-xs: 0.25rem;
--space-sm: 0.5rem;
--space-md: 1rem;
--space-lg: 1.5rem;
--space-xl: 2rem;
--space-2xl: 3rem;
}
/* ==============================================
BASE REVEAL OVERRIDES
============================================== */
.reveal-viewport {
background: var(--bg-primary);
}
.reveal {
font-family: var(--font-sans);
font-size: 24px;
font-weight: 400;
color: var(--text-primary);
}
.reveal h1, .reveal h2, .reveal h3, .reveal h4 {
font-family: var(--font-sans);
font-weight: 600;
text-transform: none;
letter-spacing: -0.02em;
color: var(--text-primary);
margin-bottom: var(--space-lg);
}
.reveal h1 {
font-size: 3rem;
font-weight: 700;
}
.reveal h2 {
font-size: 2rem;
}
.reveal h3 {
font-size: 1.5rem;
color: var(--text-secondary);
}
.reveal p {
line-height: 1.6;
}
.reveal code {
font-family: var(--font-mono);
font-size: 0.9em;
background: var(--bg-card);
padding: 0.1em 0.4em;
border-radius: 4px;
}
.reveal pre code {
padding: var(--space-md);
border-radius: 8px;
font-size: 0.8em;
}
/* Progress bar - subtle */
.reveal .progress {
height: 1px;
background: var(--border-subtle);
}
.reveal .progress span {
background: var(--text-muted);
}
/* Hide progress on title slide */
.reveal .slides section.title-slide ~ .progress {
opacity: 0;
}
/* Controls - minimal */
.reveal .controls {
color: var(--text-muted);
}
.reveal .controls button {
opacity: 0.3;
transition: opacity 0.2s ease;
}
.reveal .controls button:hover {
opacity: 0.6;
}
/* Hide controls on title slide */
.reveal .slides section.title-slide ~ .controls,
.reveal[data-state="title"] .controls {
opacity: 0;
}
/* ==============================================
SLIDE LAYOUTS
============================================== */
/* Title slide - Nike minimal */
.reveal .slides section.title-slide {
height: 100%;
display: flex !important;
align-items: center !important;
justify-content: center !important;
}
.slide-title {
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
text-align: center;
}
.slide-title h1 {
font-size: 5rem;
font-weight: 600;
letter-spacing: -0.03em;
color: var(--text-primary);
margin: 0;
}
.slide-title h1.brand {
font-family: var(--font-sans);
font-size: 6rem;
font-weight: 400;
letter-spacing: -0.03em;
}
.slide-title .subtitle {
display: none;
}
/* Section title slide */
.slide-section-title {
display: flex;
flex-direction: column;
align-items: flex-start;
justify-content: center;
padding-left: 10%;
}
.slide-section-title .section-number {
font-size: 0.875rem;
font-weight: 500;
color: var(--accent-episteme);
letter-spacing: 0.1em;
text-transform: uppercase;
margin-bottom: var(--space-md);
}
.slide-section-title h2 {
font-size: 3.5rem;
font-weight: 700;
margin-bottom: var(--space-sm);
}
.slide-section-title .subtitle {
font-size: 1.5rem;
color: var(--text-secondary);
margin-bottom: var(--space-xl);
}
.slide-section-title .description {
max-width: 600px;
font-size: 1.125rem;
color: var(--text-muted);
line-height: 1.6;
}
/* ==============================================
SEQUENCE DIAGRAM SLIDE
============================================== */
.slide-sequence {
display: flex;
flex-direction: column;
height: 100%;
padding: var(--space-xl);
}
.slide-sequence .header {
margin-bottom: var(--space-xl);
}
.slide-sequence .header h3 {
font-size: 1.25rem;
color: var(--text-muted);
margin-bottom: var(--space-xs);
}
.slide-sequence .header h2 {
font-size: 2rem;
margin-bottom: 0;
}
/* Actors row */
.actors-row {
display: flex;
justify-content: space-around;
padding: 0 var(--space-xl);
margin-bottom: var(--space-xl);
}
.actor {
display: flex;
flex-direction: column;
align-items: center;
min-width: 100px;
}
.actor-icon {
width: 48px;
height: 48px;
border-radius: 12px;
display: flex;
align-items: center;
justify-content: center;
margin-bottom: var(--space-sm);
font-size: 1.25rem;
font-weight: 600;
color: var(--bg-primary);
}
.actor-label {
font-size: 0.75rem;
font-weight: 500;
color: var(--text-secondary);
text-align: center;
max-width: 100px;
}
.actor-line {
width: 2px;
background: var(--border-subtle);
flex-grow: 1;
margin-top: var(--space-md);
}
/* Steps container */
.steps-container {
flex-grow: 1;
display: flex;
flex-direction: column;
gap: var(--space-md);
overflow-y: auto;
padding: 0 var(--space-xl);
}
/* Step card */
.step-card {
display: flex;
align-items: flex-start;
gap: var(--space-lg);
padding: var(--space-md) var(--space-lg);
background: var(--bg-card);
border-radius: 12px;
border: 1px solid var(--border-subtle);
opacity: 0;
transform: translateY(10px);
transition: opacity 0.3s ease, transform 0.3s ease;
}
.step-card.visible {
opacity: 1;
transform: translateY(0);
}
.step-card.danger {
border-color: var(--accent-danger);
background: rgba(239, 68, 68, 0.1);
}
.step-card.success {
border-color: var(--accent-success);
background: rgba(16, 185, 129, 0.1);
}
.step-card.warning {
border-color: var(--accent-warning);
background: rgba(245, 158, 11, 0.1);
}
/* Step index */
.step-index {
width: 28px;
height: 28px;
border-radius: 50%;
background: var(--border-subtle);
display: flex;
align-items: center;
justify-content: center;
font-size: 0.75rem;
font-weight: 600;
color: var(--text-secondary);
flex-shrink: 0;
}
.step-card.danger .step-index {
background: var(--accent-danger);
color: white;
}
.step-card.success .step-index {
background: var(--accent-success);
color: white;
}
/* Step content */
.step-content {
flex-grow: 1;
}
.step-header {
display: flex;
align-items: center;
gap: var(--space-sm);
margin-bottom: var(--space-xs);
}
.step-actors {
display: flex;
align-items: center;
gap: var(--space-xs);
font-size: 0.75rem;
font-weight: 500;
}
.step-from {
color: var(--text-secondary);
}
.step-arrow {
color: var(--text-muted);
}
.step-to {
color: var(--text-secondary);
}
.step-action {
font-size: 0.625rem;
font-weight: 600;
text-transform: uppercase;
letter-spacing: 0.05em;
padding: 0.125rem 0.5rem;
border-radius: 4px;
background: var(--border-subtle);
color: var(--text-muted);
}
.step-label {
font-size: 1rem;
font-weight: 500;
color: var(--text-primary);
margin-bottom: var(--space-sm);
}
.step-note {
font-size: 0.875rem;
color: var(--text-secondary);
line-height: 1.5;
}
/* Step data */
.step-data {
margin-top: var(--space-md);
padding: var(--space-sm) var(--space-md);
background: var(--bg-secondary);
border-radius: 6px;
font-family: var(--font-mono);
font-size: 0.75rem;
color: var(--text-secondary);
overflow-x: auto;
}
.step-data pre {
margin: 0;
white-space: pre-wrap;
}
/* Callout */
.step-callout {
display: inline-flex;
align-items: center;
gap: var(--space-xs);
margin-top: var(--space-sm);
padding: var(--space-xs) var(--space-sm);
background: var(--accent-episteme);
color: var(--bg-primary);
border-radius: 4px;
font-size: 0.75rem;
font-weight: 600;
}
.step-card.danger .step-callout {
background: var(--accent-danger);
color: white;
}
.step-card.success .step-callout {
background: var(--accent-success);
color: white;
}
/* ==============================================
DATA VISUALIZATION
============================================== */
.data-card {
background: var(--bg-card);
border: 1px solid var(--border-subtle);
border-radius: 12px;
padding: var(--space-lg);
margin: var(--space-md) 0;
}
.data-card-header {
display: flex;
align-items: center;
gap: var(--space-sm);
margin-bottom: var(--space-md);
padding-bottom: var(--space-md);
border-bottom: 1px solid var(--border-subtle);
}
.data-card-title {
font-size: 0.875rem;
font-weight: 500;
color: var(--text-secondary);
}
.data-card-content {
font-family: var(--font-mono);
font-size: 0.875rem;
}
/* JSON-like styling */
.json-key {
color: var(--accent-info);
}
.json-string {
color: var(--accent-success);
}
.json-number {
color: var(--accent-episteme);
}
.json-boolean {
color: var(--accent-warning);
}
.json-null {
color: var(--text-muted);
}
/* ==============================================
ANIMATIONS
============================================== */
@keyframes fadeInUp {
from {
opacity: 0;
transform: translateY(20px);
}
to {
opacity: 1;
transform: translateY(0);
}
}
@keyframes pulse {
0%, 100% {
opacity: 1;
}
50% {
opacity: 0.5;
}
}
.animate-fade-in-up {
animation: fadeInUp 0.5s ease forwards;
}
.animate-pulse {
animation: pulse 2s ease-in-out infinite;
}
/* Fragment animations */
.reveal .slides section .fragment {
opacity: 0;
transform: translateY(10px);
transition: all 0.3s ease;
}
.reveal .slides section .fragment.visible {
opacity: 1;
transform: translateY(0);
}
/* ==============================================
RESPONSIVE
============================================== */
@media (max-width: 768px) {
.reveal {
font-size: 18px;
}
.slide-title h1 {
font-size: 2.5rem;
}
.slide-section-title h2 {
font-size: 2.5rem;
}
.actors-row {
flex-wrap: wrap;
gap: var(--space-md);
}
.actor {
min-width: 80px;
}
.step-card {
flex-direction: column;
}
}

226
docs/presentations/scripts/generate.ts Normal file
View File

@ -0,0 +1,226 @@
#!/usr/bin/env npx ts-node
/**
* Presentation Generator
*
* Reads YAML sequence data and generates:
* 1. Mermaid sequence diagrams (.mmd)
* 2. JSON for Reveal.js slides
*
* Usage: npx ts-node generate.ts ../data/agile-agent-team.yaml
*/
import * as fs from 'fs';
import * as path from 'path';
import * as yaml from 'yaml';
interface Actor {
id: string;
label: string;
description: string;
color: string;
}
interface StepData {
[key: string]: unknown;
}
interface Step {
id: string;
from: string;
to: string;
action: string;
label: string;
data?: StepData;
note?: string;
callout?: string;
danger?: boolean;
warning?: boolean;
success?: boolean;
}
interface Sequence {
id: string;
title: string;
subtitle: string;
description: string;
steps: Step[];
}
interface PresentationData {
meta: {
id: string;
title: string;
subtitle: string;
version: string;
};
actors: Record<string, Actor>;
sequences: Sequence[];
annotations: Record<string, {
color: string;
label: string;
icon: string;
}>;
}
// Generate Mermaid sequence diagram for a sequence
function generateMermaid(data: PresentationData, sequence: Sequence): string {
const lines: string[] = [];
lines.push('sequenceDiagram');
lines.push(` %% ${sequence.title}: ${sequence.subtitle}`);
lines.push('');
// Collect unique actors used in this sequence
const usedActors = new Set<string>();
for (const step of sequence.steps) {
usedActors.add(step.from);
usedActors.add(step.to);
}
// Add participant declarations
for (const actorKey of usedActors) {
const actor = data.actors[actorKey];
if (actor) {
lines.push(` participant ${actor.id} as ${actor.label}`);
}
}
lines.push('');
// Add steps
for (const step of sequence.steps) {
const fromActor = data.actors[step.from];
const toActor = data.actors[step.to];
if (!fromActor || !toActor) continue;
// Determine arrow style
let arrow = '->>';
if (step.action === 'response') {
arrow = '-->>';
}
// Add the message
const label = step.label.replace(/"/g, "'");
lines.push(` ${fromActor.id}${arrow}${toActor.id}: ${label}`);
// Add note if present
if (step.note) {
const noteText = step.note.replace(/"/g, "'");
const position = step.from === step.to ? 'over' : 'right of';
const noteActor = step.from === step.to ? fromActor.id : toActor.id;
if (step.danger) {
lines.push(` Note ${position} ${noteActor}: ⚠️ ${noteText}`);
} else if (step.success) {
lines.push(` Note ${position} ${noteActor}: ✓ ${noteText}`);
} else {
lines.push(` Note ${position} ${noteActor}: ${noteText}`);
}
}
lines.push('');
}
return lines.join('\n');
}
// Generate JSON for Reveal.js
function generateRevealJson(data: PresentationData): object {
const slides: object[] = [];
// Title slide
slides.push({
type: 'title',
title: data.meta.title,
subtitle: data.meta.subtitle,
});
// Generate slides for each sequence
for (const sequence of data.sequences) {
// Sequence title slide
slides.push({
type: 'section-title',
title: sequence.title,
subtitle: sequence.subtitle,
description: sequence.description,
});
// Create step slides - group steps for animation
const stepSlide = {
type: 'sequence',
sequenceId: sequence.id,
title: sequence.title,
actors: {} as Record<string, Actor>,
steps: sequence.steps.map((step, index) => ({
...step,
index,
fromActor: data.actors[step.from],
toActor: data.actors[step.to],
})),
};
// Collect actors for this sequence
for (const step of sequence.steps) {
if (data.actors[step.from]) {
stepSlide.actors[step.from] = data.actors[step.from];
}
if (data.actors[step.to]) {
stepSlide.actors[step.to] = data.actors[step.to];
}
}
slides.push(stepSlide);
}
return {
meta: data.meta,
actors: data.actors,
annotations: data.annotations,
slides,
};
}
// Main
function main() {
const args = process.argv.slice(2);
if (args.length < 1) {
console.error('Usage: npx ts-node generate.ts <input.yaml>');
process.exit(1);
}
const inputPath = path.resolve(args[0]);
const inputDir = path.dirname(inputPath);
const baseName = path.basename(inputPath, '.yaml');
// Read and parse YAML
const yamlContent = fs.readFileSync(inputPath, 'utf-8');
const data = yaml.parse(yamlContent) as PresentationData;
// Output directory
const outputDir = path.resolve(inputDir, '..', 'generated');
fs.mkdirSync(outputDir, { recursive: true });
// Generate Mermaid for each sequence
for (const sequence of data.sequences) {
const mermaid = generateMermaid(data, sequence);
const mermaidPath = path.join(outputDir, `${baseName}-${sequence.id}.mmd`);
fs.writeFileSync(mermaidPath, mermaid);
console.log(`Generated: ${mermaidPath}`);
}
// Generate combined Mermaid
const allMermaid = data.sequences
.map(seq => generateMermaid(data, seq))
.join('\n\n---\n\n');
const combinedMermaidPath = path.join(outputDir, `${baseName}.mmd`);
fs.writeFileSync(combinedMermaidPath, allMermaid);
console.log(`Generated: ${combinedMermaidPath}`);
// Generate JSON for Reveal.js
const revealJson = generateRevealJson(data);
const jsonPath = path.join(outputDir, `${baseName}.json`);
fs.writeFileSync(jsonPath, JSON.stringify(revealJson, null, 2));
console.log(`Generated: ${jsonPath}`);
}
main();

15
docs/presentations/tsconfig.json Normal file
View File

@ -0,0 +1,15 @@
{
"compilerOptions": {
"target": "ES2020",
"module": "commonjs",
"lib": ["ES2020"],
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"resolveJsonModule": true,
"outDir": "./dist"
},
"include": ["scripts/**/*"],
"exclude": ["node_modules"]
}

635
docs/references/go-adk/agent-chatbot.md Normal file
View File

@ -0,0 +1,635 @@
# Building an Internet-Connected Chatbot with ADK-Go
This guide shows how to build a chatbot that can search the web, fetch pages, and access external APIs to provide accurate, up-to-date information.
---
## Overview
The chatbot combines:
- **Google Search** for finding current information
- **URL Fetching** for reading full web pages
- **Custom API tools** for specific data sources
- **Session memory** for context across conversations
```
┌──────────────────────────────────────────────────────────────┐
│ Smart Chatbot │
├──────────────────────────────────────────────────────────────┤
│ Tools: │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Google │ │ Fetch │ │ Weather │ + more │
│ │ Search │ │ URL │ │ API │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
├──────────────────────────────────────────────────────────────┤
│ Memory: User preferences, conversation history, facts │
└──────────────────────────────────────────────────────────────┘
```
---
## Project Setup
```bash
mkdir smart-chatbot && cd smart-chatbot
go mod init smart-chatbot
go get google.golang.org/adk
```
---
## Basic Internet-Connected Chatbot
```go
package main
import (
"context"
"log"
"os"
"google.golang.org/adk/agent/llmagent"
"google.golang.org/adk/cmd/launcher/adk"
"google.golang.org/adk/cmd/launcher/full"
"google.golang.org/adk/model/gemini"
"google.golang.org/adk/server/restapi/services"
"google.golang.org/adk/tool"
"google.golang.org/adk/tool/geminitool"
"google.golang.org/genai"
)
func main() {
ctx := context.Background()
model, err := gemini.NewModel(ctx, "gemini-3-flash-preview", &genai.ClientConfig{
APIKey: os.Getenv("GOOGLE_API_KEY"),
})
if err != nil {
log.Fatal(err)
}
chatbot, err := llmagent.New(llmagent.Config{
Name: "smart_assistant",
Model: model,
Description: "An intelligent assistant with internet access",
Instruction: `You are a helpful, knowledgeable assistant with access to the internet.
CAPABILITIES:
- Search the web for current information
- Look up facts, news, weather, sports scores, etc.
- Provide accurate, sourced information
GUIDELINES:
1. For factual questions, search to verify before answering
2. Always cite your sources when providing information from search
3. If search results are unclear or conflicting, say so
4. For opinions or analysis, clearly distinguish from facts
5. Be conversational but accurate
When you don't know something and can't find it, admit it honestly.`,
Tools: []tool.Tool{
geminitool.GoogleSearch{},
},
})
if err != nil {
log.Fatal(err)
}
l := full.NewLauncher()
cfg := &adk.Config{AgentLoader: services.NewSingleAgentLoader(chatbot)}
if err := l.Execute(ctx, cfg, os.Args[1:]); err != nil {
log.Fatal(err)
}
}
```
---
## Adding Custom Tools for Rich Information
### URL Fetcher Tool
```go
import (
"io"
"net/http"
"strings"
"time"
"google.golang.org/adk/tool"
"google.golang.org/adk/tool/functiontool"
)
type FetchURLInput struct {
URL string `json:"url" jsonschema:"The URL to fetch content from"`
}
type FetchURLOutput struct {
Content string `json:"content"`
StatusCode int `json:"status_code"`
Error string `json:"error,omitempty"`
}
func fetchURL(ctx tool.Context, input FetchURLInput) FetchURLOutput {
client := &http.Client{Timeout: 15 * time.Second}
req, err := http.NewRequestWithContext(ctx, "GET", input.URL, nil)
if err != nil {
return FetchURLOutput{Error: err.Error()}
}
req.Header.Set("User-Agent", "Mozilla/5.0 (compatible; SmartBot/1.0)")
resp, err := client.Do(req)
if err != nil {
return FetchURLOutput{Error: err.Error()}
}
defer resp.Body.Close()
// Limit response size
body, err := io.ReadAll(io.LimitReader(resp.Body, 100000))
if err != nil {
return FetchURLOutput{Error: err.Error()}
}
// Basic HTML to text (simplified)
content := stripHTML(string(body))
return FetchURLOutput{
Content: content,
StatusCode: resp.StatusCode,
}
}
func stripHTML(html string) string {
// Simple HTML tag removal (use a proper library in production)
var result strings.Builder
inTag := false
for _, r := range html {
switch {
case r == '<':
inTag = true
case r == '>':
inTag = false
case !inTag:
result.WriteRune(r)
}
}
return strings.TrimSpace(result.String())
}
```
### Weather API Tool
```go
type GetWeatherInput struct {
Location string `json:"location" jsonschema:"City name or zip code"`
Units string `json:"units,omitempty" jsonschema:"Units: metric or imperial (default: metric)"`
}
type GetWeatherOutput struct {
Location string `json:"location"`
Temperature float64 `json:"temperature"`
Conditions string `json:"conditions"`
Humidity int `json:"humidity"`
WindSpeed float64 `json:"wind_speed"`
Error string `json:"error,omitempty"`
}
func getWeather(ctx tool.Context, input GetWeatherInput) GetWeatherOutput {
apiKey := os.Getenv("OPENWEATHER_API_KEY")
if apiKey == "" {
return GetWeatherOutput{Error: "Weather API not configured"}
}
units := input.Units
if units == "" {
units = "metric"
}
url := fmt.Sprintf(
"https://api.openweathermap.org/data/2.5/weather?q=%s&units=%s&appid=%s",
url.QueryEscape(input.Location),
units,
apiKey,
)
resp, err := http.Get(url)
if err != nil {
return GetWeatherOutput{Error: err.Error()}
}
defer resp.Body.Close()
var data struct {
Name string `json:"name"`
Main struct {
Temp float64 `json:"temp"`
Humidity int `json:"humidity"`
} `json:"main"`
Weather []struct {
Description string `json:"description"`
} `json:"weather"`
Wind struct {
Speed float64 `json:"speed"`
} `json:"wind"`
}
if err := json.NewDecoder(resp.Body).Decode(&data); err != nil {
return GetWeatherOutput{Error: err.Error()}
}
conditions := ""
if len(data.Weather) > 0 {
conditions = data.Weather[0].Description
}
return GetWeatherOutput{
Location: data.Name,
Temperature: data.Main.Temp,
Conditions: conditions,
Humidity: data.Main.Humidity,
WindSpeed: data.Wind.Speed,
}
}
```
### News API Tool
```go
type GetNewsInput struct {
Query string `json:"query" jsonschema:"Search query for news"`
Category string `json:"category,omitempty" jsonschema:"Category: business, technology, sports, etc."`
Count int `json:"count,omitempty" jsonschema:"Number of articles (default: 5, max: 10)"`
}
type NewsArticle struct {
Title string `json:"title"`
Description string `json:"description"`
Source string `json:"source"`
URL string `json:"url"`
PublishedAt string `json:"published_at"`
}
type GetNewsOutput struct {
Articles []NewsArticle `json:"articles"`
Error string `json:"error,omitempty"`
}
func getNews(ctx tool.Context, input GetNewsInput) GetNewsOutput {
apiKey := os.Getenv("NEWS_API_KEY")
if apiKey == "" {
return GetNewsOutput{Error: "News API not configured"}
}
count := input.Count
if count <= 0 || count > 10 {
count = 5
}
url := fmt.Sprintf(
"https://newsapi.org/v2/everything?q=%s&pageSize=%d&apiKey=%s",
url.QueryEscape(input.Query),
count,
apiKey,
)
resp, err := http.Get(url)
if err != nil {
return GetNewsOutput{Error: err.Error()}
}
defer resp.Body.Close()
var data struct {
Articles []struct {
Title string `json:"title"`
Description string `json:"description"`
URL string `json:"url"`
PublishedAt string `json:"publishedAt"`
Source struct {
Name string `json:"name"`
} `json:"source"`
} `json:"articles"`
}
if err := json.NewDecoder(resp.Body).Decode(&data); err != nil {
return GetNewsOutput{Error: err.Error()}
}
articles := make([]NewsArticle, len(data.Articles))
for i, a := range data.Articles {
articles[i] = NewsArticle{
Title: a.Title,
Description: a.Description,
Source: a.Source.Name,
URL: a.URL,
PublishedAt: a.PublishedAt,
}
}
return GetNewsOutput{Articles: articles}
}
```
---
## Complete Chatbot with All Tools
```go
package main
import (
"context"
"encoding/json"
"fmt"
"io"
"log"
"net/http"
"net/url"
"os"
"strings"
"time"
"google.golang.org/adk/agent/llmagent"
"google.golang.org/adk/cmd/launcher/adk"
"google.golang.org/adk/cmd/launcher/full"
"google.golang.org/adk/model/gemini"
"google.golang.org/adk/server/restapi/services"
"google.golang.org/adk/tool"
"google.golang.org/adk/tool/functiontool"
"google.golang.org/adk/tool/geminitool"
"google.golang.org/genai"
)
func main() {
ctx := context.Background()
model, err := gemini.NewModel(ctx, "gemini-3-flash-preview", &genai.ClientConfig{
APIKey: os.Getenv("GOOGLE_API_KEY"),
})
if err != nil {
log.Fatal(err)
}
tools, err := createTools()
if err != nil {
log.Fatal(err)
}
chatbot, err := llmagent.New(llmagent.Config{
Name: "smart_assistant",
Model: model,
Description: "An intelligent assistant with internet and API access",
Instruction: `You are a smart, helpful assistant with access to multiple information sources.
AVAILABLE TOOLS:
- google_search: Search the web for any information
- fetch_url: Read the full content of a specific webpage
- get_weather: Get current weather for any location
- get_news: Get recent news articles on any topic
- remember_fact: Store important information about the user
- recall_facts: Retrieve stored information about the user
BEHAVIOR GUIDELINES:
1. **Be proactive**: If a question might benefit from current data, search for it
2. **Combine sources**: Use multiple tools together when helpful
3. **Cite sources**: When providing factual information, mention where it came from
4. **Remember context**: Use remember_fact to store user preferences and important info
5. **Be conversational**: Maintain a friendly, helpful tone
6. **Handle errors gracefully**: If a tool fails, try alternatives or explain the limitation
EXAMPLES OF GOOD BEHAVIOR:
- User asks "What's the weather?" → Ask for location, then use get_weather
- User asks about news → Use get_news, summarize, offer to read full articles
- User mentions their city → Remember it for future weather/news queries
- User asks about something current → Search first, don't rely on training data
User's stored preferences: {user:preferences}
User's location (if known): {user:location}`,
Tools: tools,
})
if err != nil {
log.Fatal(err)
}
l := full.NewLauncher()
cfg := &adk.Config{AgentLoader: services.NewSingleAgentLoader(chatbot)}
if err := l.Execute(ctx, cfg, os.Args[1:]); err != nil {
log.Fatal(err)
}
}
func createTools() ([]tool.Tool, error) {
var tools []tool.Tool
// Google Search (built-in)
tools = append(tools, geminitool.GoogleSearch{})
// URL Fetcher
fetchTool, err := functiontool.New(
functiontool.Config{
Name: "fetch_url",
Description: "Fetch and read the content of a webpage. Use after search to get full article content.",
},
fetchURL,
)
if err != nil {
return nil, err
}
tools = append(tools, fetchTool)
// Weather
weatherTool, err := functiontool.New(
functiontool.Config{
Name: "get_weather",
Description: "Get current weather conditions for a location",
},
getWeather,
)
if err != nil {
return nil, err
}
tools = append(tools, weatherTool)
// News
newsTool, err := functiontool.New(
functiontool.Config{
Name: "get_news",
Description: "Get recent news articles about a topic",
},
getNews,
)
if err != nil {
return nil, err
}
tools = append(tools, newsTool)
// Memory tools
rememberTool, err := functiontool.New(
functiontool.Config{
Name: "remember_fact",
Description: "Store an important fact about the user for future reference",
},
rememberFact,
)
if err != nil {
return nil, err
}
tools = append(tools, rememberTool)
recallTool, err := functiontool.New(
functiontool.Config{
Name: "recall_facts",
Description: "Retrieve all stored facts about the user",
},
recallFacts,
)
if err != nil {
return nil, err
}
tools = append(tools, recallTool)
return tools, nil
}
// Memory tools
type RememberFactInput struct {
Category string `json:"category" jsonschema:"Category: preference, location, name, interest, other"`
Fact string `json:"fact" jsonschema:"The fact to remember"`
}
type RememberFactOutput struct {
Message string `json:"message"`
}
func rememberFact(ctx tool.Context, input RememberFactInput) RememberFactOutput {
state := ctx.Session().State()
// Get existing facts
var facts map[string][]string
if existing := state.Get("user:facts"); existing != nil {
if f, ok := existing.(map[string][]string); ok {
facts = f
}
}
if facts == nil {
facts = make(map[string][]string)
}
// Add new fact
facts[input.Category] = append(facts[input.Category], input.Fact)
state.Set("user:facts", facts)
// Also set specific keys for common categories
if input.Category == "location" {
state.Set("user:location", input.Fact)
}
return RememberFactOutput{
Message: fmt.Sprintf("Remembered: %s (%s)", input.Fact, input.Category),
}
}
type RecallFactsInput struct {
Category string `json:"category,omitempty" jsonschema:"Optional: filter by category"`
}
type RecallFactsOutput struct {
Facts map[string][]string `json:"facts"`
}
func recallFacts(ctx tool.Context, input RecallFactsInput) RecallFactsOutput {
state := ctx.Session().State()
var facts map[string][]string
if existing := state.Get("user:facts"); existing != nil {
if f, ok := existing.(map[string][]string); ok {
facts = f
}
}
if facts == nil {
return RecallFactsOutput{Facts: map[string][]string{}}
}
if input.Category != "" {
return RecallFactsOutput{
Facts: map[string][]string{input.Category: facts[input.Category]},
}
}
return RecallFactsOutput{Facts: facts}
}
// Include fetchURL, getWeather, getNews from earlier...
```
---
## Adding Proactive Search Behavior
Use callbacks to automatically search for time-sensitive queries:
```go
import "google.golang.org/adk/model"
chatbot, err := llmagent.New(llmagent.Config{
// ... other config ...
BeforeModelCallback: func(ctx agent.CallbackContext, req *model.LLMRequest) (*model.LLMRequest, error) {
// Log all requests for debugging
log.Printf("Model request with %d messages", len(req.Messages))
return req, nil
},
AfterToolCallback: func(ctx agent.CallbackContext, call *tool.Call, result *tool.Result) (*tool.Result, error) {
// Log tool usage
log.Printf("Tool %s completed: %v", call.Name, result.Success)
// Track tool usage in session for analytics
state := ctx.Session().State()
count := 0
if c := state.Get("temp:tool_count"); c != nil {
count = c.(int)
}
state.Set("temp:tool_count", count+1)
return result, nil
},
})
```
---
## Running and Testing
```bash
# Set API keys
export GOOGLE_API_KEY="your-key"
export OPENWEATHER_API_KEY="your-key" # Optional
export NEWS_API_KEY="your-key" # Optional
# Run in console
go run main.go
# Example conversations:
# > What's the weather in Tokyo?
# > Tell me the latest news about AI
# > I live in San Francisco (bot remembers this)
# > What's the weather? (uses remembered location)
# > Search for the best restaurants near me
# > Read that first article for me (fetches full content)
# Run with web UI for debugging
go run main.go web api webui
```
---
## Best Practices
1. **Rate limit external APIs**: Add throttling to prevent abuse
2. **Cache responses**: Store recent search results to reduce API calls
3. **Handle failures gracefully**: If one tool fails, the bot should try alternatives
4. **Be transparent**: Tell users when information comes from search vs. training
5. **Respect user privacy**: Don't store sensitive information; let users clear memory
6. **Set reasonable timeouts**: External APIs can be slow; don't hang forever

603
docs/references/go-adk/agent-multi-chatters.md Normal file
View File

@ -0,0 +1,603 @@
# Building a Multi-Agent Chat Room with ADK-Go
This guide shows how to create a chat room where multiple AI agents with different perspectives and personalities discuss topics together, providing users with diverse viewpoints.
---
## Overview
The multi-agent chat room simulates a discussion panel:
```
┌─────────────────────────────────────────────────────────────────┐
│ DISCUSSION MODERATOR │
│ Orchestrates conversation, ensures balance │
├────────────┬────────────┬────────────┬────────────┬────────────┤
│ OPTIMIST │ SKEPTIC │ ANALYST │ CREATIVE │ PRAGMATIC │
│ │ │ │ │ │
│ Sees │ Questions │ Examines │ Explores │ Focuses on │
│ potential │ assumptions│ data/facts │ new ideas │ feasibility│
└────────────┴────────────┴────────────┴────────────┴────────────┘
```
---
## Project Setup
```bash
mkdir multi-agent-chatroom && cd multi-agent-chatroom
go mod init chatroom
go get google.golang.org/adk
```
---
## Core Implementation
### Agent Personas
```go
package main
import (
"context"
"fmt"
"log"
"os"
"strings"
"google.golang.org/adk/agent"
"google.golang.org/adk/agent/llmagent"
"google.golang.org/adk/agent/loopagent"
"google.golang.org/adk/agent/sequentialagent"
"google.golang.org/adk/cmd/launcher/adk"
"google.golang.org/adk/cmd/launcher/full"
"google.golang.org/adk/model/gemini"
"google.golang.org/adk/server/restapi/services"
"google.golang.org/adk/tool"
"google.golang.org/adk/tool/functiontool"
"google.golang.org/adk/tool/geminitool"
"google.golang.org/genai"
)
// Persona definitions
type Persona struct {
Name string
Role string
Instruction string
Style string
}
var personas = []Persona{
{
Name: "optimist",
Role: "The Optimist",
Instruction: `You are an optimistic thinker who sees potential and opportunities.
PERSPECTIVE:
- Focus on positive outcomes and possibilities
- Highlight benefits and advantages
- Encourage bold thinking and innovation
- Find silver linings in challenges
STYLE:
- Enthusiastic but not naive
- Back optimism with reasons
- Acknowledge risks but emphasize how they can be overcome
- Use phrases like "The exciting part is...", "This opens up...", "Imagine if..."`,
},
{
Name: "skeptic",
Role: "The Skeptic",
Instruction: `You are a constructive skeptic who questions assumptions.
PERSPECTIVE:
- Identify potential problems and risks
- Question unstated assumptions
- Play devil's advocate
- Demand evidence for claims
STYLE:
- Critical but not cynical
- Ask probing questions
- Point out what could go wrong
- Use phrases like "But have we considered...", "What evidence shows...", "The risk is..."`,
},
{
Name: "analyst",
Role: "The Analyst",
Instruction: `You are a data-driven analyst who examines facts objectively.
PERSPECTIVE:
- Focus on data, statistics, and evidence
- Compare with historical examples
- Identify patterns and trends
- Quantify when possible
STYLE:
- Objective and methodical
- Reference specific numbers and studies
- Present multiple scenarios
- Use phrases like "The data suggests...", "Historically...", "Studies show..."`,
},
{
Name: "creative",
Role: "The Creative",
Instruction: `You are a creative thinker who explores unconventional ideas.
PERSPECTIVE:
- Think outside the box
- Make unexpected connections
- Propose innovative solutions
- Challenge conventional wisdom
STYLE:
- Imaginative and bold
- Use analogies and metaphors
- Propose "what if" scenarios
- Use phrases like "What if we...", "Here's a wild idea...", "Imagine combining..."`,
},
{
Name: "pragmatist",
Role: "The Pragmatist",
Instruction: `You are a practical thinker focused on real-world implementation.
PERSPECTIVE:
- Focus on feasibility and execution
- Consider resources, time, and constraints
- Identify concrete next steps
- Balance idealism with reality
STYLE:
- Grounded and action-oriented
- Ask "how" questions
- Propose specific steps
- Use phrases like "Practically speaking...", "To make this work...", "The first step would be..."`,
},
}
func main() {
ctx := context.Background()
model, err := gemini.NewModel(ctx, "gemini-3-flash-preview", &genai.ClientConfig{
APIKey: os.Getenv("GOOGLE_API_KEY"),
})
if err != nil {
log.Fatal(err)
}
chatroom, err := buildChatRoom(model)
if err != nil {
log.Fatal(err)
}
l := full.NewLauncher()
cfg := &adk.Config{AgentLoader: services.NewSingleAgentLoader(chatroom)}
if err := l.Execute(ctx, cfg, os.Args[1:]); err != nil {
log.Fatal(err)
}
}
```
### Building the Chat Room
```go
func buildChatRoom(model *gemini.Model) (agent.Agent, error) {
// Create tools for agent interaction
speakTool, err := functiontool.New(
functiontool.Config{
Name: "speak",
Description: "Add your contribution to the discussion",
},
speak,
)
if err != nil {
return nil, err
}
respondToTool, err := functiontool.New(
functiontool.Config{
Name: "respond_to",
Description: "Respond directly to another participant's point",
},
respondTo,
)
if err != nil {
return nil, err
}
// Create participant agents
var participants []agent.Agent
for _, p := range personas {
participant, err := createParticipant(model, p, []tool.Tool{speakTool, respondToTool})
if err != nil {
return nil, err
}
participants = append(participants, participant)
}
// Create moderator that orchestrates the discussion
moderator, err := createModerator(model, participants)
if err != nil {
return nil, err
}
return moderator, nil
}
func createParticipant(model *gemini.Model, p Persona, tools []tool.Tool) (agent.Agent, error) {
instruction := fmt.Sprintf(`You are %s in a multi-perspective discussion panel.
%s
DISCUSSION CONTEXT:
Topic: {discussion_topic}
Previous contributions:
{discussion_history}
YOUR TASK:
Provide your unique perspective on the topic. Consider what others have said and either:
1. Build on their points with your perspective
2. Respectfully challenge or question their views
3. Introduce a new angle they haven't considered
Keep your response focused (2-4 sentences). Use the speak tool to contribute.
If directly responding to someone, use respond_to tool.`, p.Role, p.Instruction)
return llmagent.New(llmagent.Config{
Name: p.Name,
Model: model,
Description: fmt.Sprintf("%s - %s", p.Role, p.Style),
Instruction: instruction,
Tools: tools,
OutputKey: fmt.Sprintf("%s_contribution", p.Name),
})
}
func createModerator(model *gemini.Model, participants []agent.Agent) (agent.Agent, error) {
// Opening agent - frames the discussion
opener, err := llmagent.New(llmagent.Config{
Name: "opener",
Model: model,
Instruction: `You are a discussion moderator. Frame the topic for the panel.
Topic from user: {user_input}
Create a brief, engaging introduction that:
1. Restates the topic clearly
2. Highlights why it's worth discussing
3. Poses 1-2 key questions for the panel
Keep it to 2-3 sentences. Be neutral and inviting.`,
OutputKey: "discussion_topic",
})
if err != nil {
return nil, err
}
// Discussion round - all participants contribute
discussionRound, err := createDiscussionRound(model, participants)
if err != nil {
return nil, err
}
// Synthesizer - summarizes and identifies key insights
synthesizer, err := llmagent.New(llmagent.Config{
Name: "synthesizer",
Model: model,
Instruction: `You are summarizing a multi-perspective discussion.
TOPIC: {discussion_topic}
CONTRIBUTIONS:
{discussion_history}
Create a balanced summary that:
1. Highlights the key points from each perspective
2. Notes areas of agreement and disagreement
3. Identifies the most compelling insights
4. Suggests questions for further exploration
Be fair to all viewpoints. Present the synthesis conversationally.`,
OutputKey: "synthesis",
})
if err != nil {
return nil, err
}
// Build full pipeline: Open → Discuss (loop) → Synthesize
pipeline, err := sequentialagent.New(sequentialagent.Config{
AgentConfig: agent.Config{
Name: "discussion_chatroom",
Description: "Multi-perspective discussion panel",
SubAgents: []agent.Agent{opener, discussionRound, synthesizer},
},
})
return pipeline, err
}
func createDiscussionRound(model *gemini.Model, participants []agent.Agent) (agent.Agent, error) {
// History tracker - runs before each participant
historyUpdater, err := llmagent.New(llmagent.Config{
Name: "history_updater",
Model: model,
Instruction: `Update the discussion history with recent contributions.
Current history:
{discussion_history}
New contributions to add:
- Optimist: {optimist_contribution}
- Skeptic: {skeptic_contribution}
- Analyst: {analyst_contribution}
- Creative: {creative_contribution}
- Pragmatist: {pragmatist_contribution}
Output the combined, formatted discussion history.`,
OutputKey: "discussion_history",
})
if err != nil {
return nil, err
}
// Build round: each participant speaks, then history updates
roundAgents := append(participants, historyUpdater)
// Create a sequential round
round, err := sequentialagent.New(sequentialagent.Config{
AgentConfig: agent.Config{
Name: "discussion_round",
SubAgents: roundAgents,
},
})
if err != nil {
return nil, err
}
// Loop for multiple rounds of discussion
multiRound, err := loopagent.New(loopagent.Config{
AgentConfig: agent.Config{
Name: "multi_round_discussion",
SubAgents: []agent.Agent{round},
},
MaxIterations: 2, // Two rounds of discussion
})
return multiRound, err
}
```
### Speaking Tools
```go
type SpeakInput struct {
Message string `json:"message" jsonschema:"Your contribution to the discussion"`
}
type SpeakOutput struct {
Delivered bool `json:"delivered"`
Speaker string `json:"speaker"`
}
func speak(ctx tool.Context, input SpeakInput) SpeakOutput {
// Get the current agent's name from context
agentName := "participant"
// Store in discussion log
state := ctx.Session().State()
var log []string
if existing := state.Get("temp:discussion_log"); existing != nil {
if l, ok := existing.([]string); ok {
log = l
}
}
entry := fmt.Sprintf("[%s]: %s", strings.ToUpper(agentName), input.Message)
log = append(log, entry)
state.Set("temp:discussion_log", log)
return SpeakOutput{Delivered: true, Speaker: agentName}
}
type RespondToInput struct {
Target string `json:"target" jsonschema:"Who you're responding to (optimist, skeptic, analyst, creative, pragmatist)"`
Message string `json:"message" jsonschema:"Your response"`
}
type RespondToOutput struct {
Delivered bool `json:"delivered"`
}
func respondTo(ctx tool.Context, input RespondToInput) RespondToOutput {
state := ctx.Session().State()
var log []string
if existing := state.Get("temp:discussion_log"); existing != nil {
if l, ok := existing.([]string); ok {
log = l
}
}
entry := fmt.Sprintf("[RESPONSE to %s]: %s", strings.ToUpper(input.Target), input.Message)
log = append(log, entry)
state.Set("temp:discussion_log", log)
return RespondToOutput{Delivered: true}
}
```
---
## Alternative: Real-Time Debate Format
For more dynamic back-and-forth:
```go
func buildDebateRoom(model *gemini.Model) (agent.Agent, error) {
// Pro side
proDebater, _ := llmagent.New(llmagent.Config{
Name: "pro_debater",
Model: model,
Instruction: `You are arguing IN FAVOR of the proposition.
Proposition: {proposition}
Opponent's last argument: {con_argument}
Build a compelling case FOR the proposition. If responding to opponent:
- Acknowledge their point
- Counter with evidence or logic
- Strengthen your position
Be persuasive but fair. 2-3 sentences max.`,
OutputKey: "pro_argument",
})
// Con side
conDebater, _ := llmagent.New(llmagent.Config{
Name: "con_debater",
Model: model,
Instruction: `You are arguing AGAINST the proposition.
Proposition: {proposition}
Opponent's last argument: {pro_argument}
Build a compelling case AGAINST the proposition. If responding to opponent:
- Acknowledge their point
- Counter with evidence or logic
- Strengthen your position
Be persuasive but fair. 2-3 sentences max.`,
OutputKey: "con_argument",
})
// Debate round
round, _ := sequentialagent.New(sequentialagent.Config{
AgentConfig: agent.Config{
Name: "debate_round",
SubAgents: []agent.Agent{proDebater, conDebater},
},
})
// Multiple rounds
debate, _ := loopagent.New(loopagent.Config{
AgentConfig: agent.Config{
Name: "debate",
SubAgents: []agent.Agent{round},
},
MaxIterations: 3,
})
// Judge
judge, _ := llmagent.New(llmagent.Config{
Name: "judge",
Model: model,
Instruction: `You are an impartial debate judge.
Proposition: {proposition}
PRO arguments: {pro_argument}
CON arguments: {con_argument}
Evaluate the debate:
1. Strongest argument from each side
2. Key points of clash
3. Which side was more persuasive and why
4. What was missing from both sides
Be balanced and specific in your assessment.`,
OutputKey: "verdict",
})
// Full debate pipeline
return sequentialagent.New(sequentialagent.Config{
AgentConfig: agent.Config{
Name: "debate_room",
SubAgents: []agent.Agent{debate, judge},
},
})
}
```
---
## Adding Research Capability
Let agents search for evidence during discussion:
```go
func createResearchParticipant(model *gemini.Model, p Persona) (agent.Agent, error) {
instruction := fmt.Sprintf(`You are %s in a research-backed discussion.
%s
Topic: {discussion_topic}
Previous points: {discussion_history}
You have access to web search. Use it to:
- Find evidence supporting your perspective
- Discover relevant statistics or studies
- Check facts before stating them
Provide your perspective backed by what you find. Cite sources.`, p.Role, p.Instruction)
return llmagent.New(llmagent.Config{
Name: p.Name,
Model: model,
Description: p.Role,
Instruction: instruction,
Tools: []tool.Tool{
geminitool.GoogleSearch{},
},
OutputKey: fmt.Sprintf("%s_contribution", p.Name),
})
}
```
---
## Running the Chat Room
```bash
# Console mode
go run main.go
# Example prompts:
# > Should companies require employees to return to the office?
# > Is social media beneficial or harmful to society?
# > Should AI-generated art be considered "real" art?
# > What's the best approach to addressing climate change?
# Web UI mode
go run main.go web api webui
```
---
## Sample Output
**User**: Should universities require students to take a gap year before college?
**Moderator**: _Today we're discussing whether universities should mandate a gap year before enrollment. This touches on questions of maturity, experience, and educational readiness. What perspectives should we consider?_
**Optimist**: _A gap year could be transformative! Students return with real-world experience, clearer goals, and renewed motivation. Countries like Denmark and Israel show this works—their students often outperform peers who went straight to university._
**Skeptic**: _But who can afford a year without income or education? This could widen inequality—wealthy students travel while others fall behind. And what about momentum? Many students who delay never return._
**Analyst**: _The data is mixed. A 2015 study showed gap year students had 0.1-0.4 higher GPAs, but selection bias is significant—motivated students choose gap years. Only 3% of American students take them compared to 25% in the UK._
**Creative**: _What if we reimagined the gap year entirely? Instead of a break, it could be a structured "first year" of hands-on learning—apprenticeships, service, exploration—that counts toward the degree._
**Pragmatist**: _Making it mandatory seems unworkable. Better approach: make gap years easier and more supported. Deferred admission, structured programs, financial aid that covers gap activities._
**Synthesis**: _The panel found common ground on gap years' potential value but diverged on implementation. The Skeptic raised valid equity concerns that the Pragmatist tried to address. The Creative's hybrid model generated interest. Key insight: the question isn't whether gap years help, but how to make them accessible and productive for all students._
---
## Best Practices
1. **Balance perspectives**: Ensure no single viewpoint dominates
2. **Keep contributions focused**: 2-4 sentences per turn prevents rambling
3. **Encourage engagement**: Agents should reference each other's points
4. **Vary the format**: Debate, panel, roundtable each serve different purposes
5. **Let users guide depth**: Allow follow-up questions to explore specific angles

754
docs/references/go-adk/agent-planning-facilitator.md Normal file
View File

@ -0,0 +1,754 @@
# Building a Sprint Planning Meeting Facilitator with ADK-Go
This guide shows how to create a multi-agent system that facilitates project planning and sprint planning meetings, helping teams break down work, estimate effort, and create actionable plans.
---
## Overview
The sprint planning facilitator uses specialized agents for each phase:
```
┌──────────────────────────────────────────────────────────────────────┐
│ SPRINT PLANNING FLOW │
├──────────────────────────────────────────────────────────────────────┤
│ │
│ 1. INTAKE 2. BREAKDOWN 3. ESTIMATION 4. PLANNING │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Project │───▶│ Story │──────▶│Estimator │───▶│ Sprint │ │
│ │ Analyzer │ │ Creator │ │ │ │ Planner │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
│ │ │ │ │ │
│ Understands Creates user Estimates Organizes │
│ goals & stories & points & into sprints │
│ requirements subtasks identifies with goals │
│ risks │
└──────────────────────────────────────────────────────────────────────┘
```
---
## Project Setup
```bash
mkdir sprint-planner && cd sprint-planner
go mod init sprint-planner
go get google.golang.org/adk
```
---
## Data Models
```go
package main
import (
"context"
"encoding/json"
"fmt"
"log"
"os"
"time"
"google.golang.org/adk/agent"
"google.golang.org/adk/agent/llmagent"
"google.golang.org/adk/agent/sequentialagent"
"google.golang.org/adk/cmd/launcher/adk"
"google.golang.org/adk/cmd/launcher/full"
"google.golang.org/adk/model/gemini"
"google.golang.org/adk/server/restapi/services"
"google.golang.org/adk/tool"
"google.golang.org/adk/tool/functiontool"
"google.golang.org/genai"
)
// Data structures for planning artifacts
type UserStory struct {
ID string `json:"id"`
Title string `json:"title"`
Description string `json:"description"`
AcceptCrit []string `json:"acceptance_criteria"`
Priority string `json:"priority"` // must-have, should-have, nice-to-have
StoryPoints int `json:"story_points,omitempty"`
Subtasks []Task `json:"subtasks,omitempty"`
Risks []string `json:"risks,omitempty"`
Sprint int `json:"sprint,omitempty"`
}
type Task struct {
ID string `json:"id"`
Title string `json:"title"`
Description string `json:"description"`
Hours int `json:"estimated_hours"`
Type string `json:"type"` // frontend, backend, design, testing, devops
Blocked bool `json:"blocked,omitempty"`
BlockedBy string `json:"blocked_by,omitempty"`
}
type Sprint struct {
Number int `json:"number"`
Goal string `json:"goal"`
StartDate string `json:"start_date"`
EndDate string `json:"end_date"`
Stories []UserStory `json:"stories"`
TotalPoints int `json:"total_points"`
Capacity int `json:"capacity"`
}
type ProjectPlan struct {
Name string `json:"name"`
Description string `json:"description"`
Goals []string `json:"goals"`
Stories []UserStory `json:"stories"`
Sprints []Sprint `json:"sprints"`
Risks []Risk `json:"risks"`
Timeline string `json:"timeline"`
}
type Risk struct {
Description string `json:"description"`
Impact string `json:"impact"` // high, medium, low
Likelihood string `json:"likelihood"` // high, medium, low
Mitigation string `json:"mitigation"`
}
```
---
## Planning Tools
```go
// Tool: Create User Story
type CreateStoryInput struct {
Title string `json:"title" jsonschema:"Clear, concise story title"`
Description string `json:"description" jsonschema:"As a [user], I want [feature] so that [benefit]"`
AcceptCrit []string `json:"acceptance_criteria" jsonschema:"List of acceptance criteria"`
Priority string `json:"priority" jsonschema:"Priority: must-have, should-have, nice-to-have"`
}
type CreateStoryOutput struct {
StoryID string `json:"story_id"`
Message string `json:"message"`
}
var storyCounter int
var stories []UserStory
func createStory(ctx tool.Context, input CreateStoryInput) CreateStoryOutput {
storyCounter++
id := fmt.Sprintf("US-%03d", storyCounter)
story := UserStory{
ID: id,
Title: input.Title,
Description: input.Description,
AcceptCrit: input.AcceptCrit,
Priority: input.Priority,
}
stories = append(stories, story)
// Save to state
ctx.Session().State().Set("stories", stories)
return CreateStoryOutput{
StoryID: id,
Message: fmt.Sprintf("Created story %s: %s", id, input.Title),
}
}
// Tool: Add Subtask to Story
type AddSubtaskInput struct {
StoryID string `json:"story_id" jsonschema:"Parent story ID (e.g., US-001)"`
Title string `json:"title" jsonschema:"Task title"`
Description string `json:"description" jsonschema:"Task description"`
Hours int `json:"estimated_hours" jsonschema:"Estimated hours to complete"`
Type string `json:"type" jsonschema:"Type: frontend, backend, design, testing, devops, documentation"`
}
type AddSubtaskOutput struct {
TaskID string `json:"task_id"`
Message string `json:"message"`
}
var taskCounter int
func addSubtask(ctx tool.Context, input AddSubtaskInput) AddSubtaskOutput {
taskCounter++
taskID := fmt.Sprintf("T-%03d", taskCounter)
task := Task{
ID: taskID,
Title: input.Title,
Description: input.Description,
Hours: input.Hours,
Type: input.Type,
}
// Find and update story
for i := range stories {
if stories[i].ID == input.StoryID {
stories[i].Subtasks = append(stories[i].Subtasks, task)
ctx.Session().State().Set("stories", stories)
return AddSubtaskOutput{
TaskID: taskID,
Message: fmt.Sprintf("Added task %s to %s", taskID, input.StoryID),
}
}
}
return AddSubtaskOutput{
TaskID: "",
Message: fmt.Sprintf("Story %s not found", input.StoryID),
}
}
// Tool: Estimate Story Points
type EstimateStoryInput struct {
StoryID string `json:"story_id" jsonschema:"Story ID to estimate"`
Points int `json:"points" jsonschema:"Story points (1, 2, 3, 5, 8, 13)"`
Rationale string `json:"rationale" jsonschema:"Brief explanation for estimate"`
}
type EstimateStoryOutput struct {
Message string `json:"message"`
}
func estimateStory(ctx tool.Context, input EstimateStoryInput) EstimateStoryOutput {
for i := range stories {
if stories[i].ID == input.StoryID {
stories[i].StoryPoints = input.Points
ctx.Session().State().Set("stories", stories)
return EstimateStoryOutput{
Message: fmt.Sprintf("Estimated %s at %d points: %s", input.StoryID, input.Points, input.Rationale),
}
}
}
return EstimateStoryOutput{Message: "Story not found"}
}
// Tool: Add Risk
type AddRiskInput struct {
Description string `json:"description" jsonschema:"Risk description"`
Impact string `json:"impact" jsonschema:"Impact level: high, medium, low"`
Likelihood string `json:"likelihood" jsonschema:"Likelihood: high, medium, low"`
Mitigation string `json:"mitigation" jsonschema:"Mitigation strategy"`
StoryID string `json:"story_id,omitempty" jsonschema:"Related story ID (optional)"`
}
type AddRiskOutput struct {
Message string `json:"message"`
}
var risks []Risk
func addRisk(ctx tool.Context, input AddRiskInput) AddRiskOutput {
risk := Risk{
Description: input.Description,
Impact: input.Impact,
Likelihood: input.Likelihood,
Mitigation: input.Mitigation,
}
risks = append(risks, risk)
// Also add to story if specified
if input.StoryID != "" {
for i := range stories {
if stories[i].ID == input.StoryID {
stories[i].Risks = append(stories[i].Risks, input.Description)
}
}
}
ctx.Session().State().Set("risks", risks)
ctx.Session().State().Set("stories", stories)
return AddRiskOutput{Message: fmt.Sprintf("Added risk: %s", input.Description)}
}
// Tool: Assign to Sprint
type AssignSprintInput struct {
StoryID string `json:"story_id" jsonschema:"Story ID to assign"`
SprintNum int `json:"sprint_number" jsonschema:"Sprint number (1, 2, 3, etc.)"`
}
type AssignSprintOutput struct {
Message string `json:"message"`
}
func assignSprint(ctx tool.Context, input AssignSprintInput) AssignSprintOutput {
for i := range stories {
if stories[i].ID == input.StoryID {
stories[i].Sprint = input.SprintNum
ctx.Session().State().Set("stories", stories)
return AssignSprintOutput{
Message: fmt.Sprintf("Assigned %s to Sprint %d", input.StoryID, input.SprintNum),
}
}
}
return AssignSprintOutput{Message: "Story not found"}
}
// Tool: Generate Plan Summary
type GeneratePlanInput struct {
SprintLength int `json:"sprint_length_days" jsonschema:"Sprint length in days (default: 14)"`
Velocity int `json:"velocity" jsonschema:"Team velocity (story points per sprint)"`
}
type GeneratePlanOutput struct {
Summary string `json:"summary"`
Plan json.RawMessage `json:"plan"`
}
func generatePlan(ctx tool.Context, input GeneratePlanInput) GeneratePlanOutput {
sprintLength := input.SprintLength
if sprintLength == 0 {
sprintLength = 14
}
// Group stories by sprint
sprintMap := make(map[int][]UserStory)
totalPoints := 0
for _, s := range stories {
sprintMap[s.Sprint] = append(sprintMap[s.Sprint], s)
totalPoints += s.StoryPoints
}
// Build sprints
var sprints []Sprint
startDate := time.Now()
for num := 1; num <= len(sprintMap); num++ {
sprintStories := sprintMap[num]
points := 0
for _, s := range sprintStories {
points += s.StoryPoints
}
sprint := Sprint{
Number: num,
StartDate: startDate.Format("2006-01-02"),
EndDate: startDate.AddDate(0, 0, sprintLength).Format("2006-01-02"),
Stories: sprintStories,
TotalPoints: points,
Capacity: input.Velocity,
}
sprints = append(sprints, sprint)
startDate = startDate.AddDate(0, 0, sprintLength)
}
plan := ProjectPlan{
Stories: stories,
Sprints: sprints,
Risks: risks,
}
planJSON, _ := json.MarshalIndent(plan, "", " ")
summary := fmt.Sprintf(
"Plan: %d stories, %d total points, %d sprints, %d risks identified",
len(stories), totalPoints, len(sprints), len(risks),
)
return GeneratePlanOutput{
Summary: summary,
Plan: planJSON,
}
}
```
---
## Building the Planning Pipeline
```go
func main() {
ctx := context.Background()
model, err := gemini.NewModel(ctx, "gemini-3-flash-preview", &genai.ClientConfig{
APIKey: os.Getenv("GOOGLE_API_KEY"),
})
if err != nil {
log.Fatal(err)
}
planner, err := buildPlanningPipeline(model)
if err != nil {
log.Fatal(err)
}
l := full.NewLauncher()
cfg := &adk.Config{AgentLoader: services.NewSingleAgentLoader(planner)}
if err := l.Execute(ctx, cfg, os.Args[1:]); err != nil {
log.Fatal(err)
}
}
func buildPlanningPipeline(model *gemini.Model) (agent.Agent, error) {
tools, err := createPlanningTools()
if err != nil {
return nil, err
}
// Stage 1: Project Analyzer
analyzer, err := llmagent.New(llmagent.Config{
Name: "project_analyzer",
Model: model,
Description: "Analyzes project requirements and extracts goals",
Instruction: `You are a senior product manager analyzing a project.
Given the project description, identify:
1. **Core Goals**: What are the main objectives? (3-5 goals)
2. **User Types**: Who are the users/stakeholders?
3. **Key Features**: What are the must-have features?
4. **Constraints**: Timeline, technical, or resource constraints mentioned
5. **Success Metrics**: How will success be measured?
Be thorough but concise. Output a structured analysis.`,
OutputKey: "project_analysis",
})
if err != nil {
return nil, err
}
// Stage 2: Story Creator
storyCreator, err := llmagent.New(llmagent.Config{
Name: "story_creator",
Model: model,
Description: "Creates user stories from requirements",
Instruction: `You are an agile coach creating user stories.
PROJECT ANALYSIS:
{project_analysis}
Create user stories for each feature identified. For each story:
1. Use the create_story tool with proper format
2. Write clear acceptance criteria (testable conditions)
3. Assign appropriate priority (must-have, should-have, nice-to-have)
4. Use the add_subtask tool to break down into technical tasks
Story format: "As a [user type], I want [feature] so that [benefit]"
Create stories for ALL identified features. Be comprehensive.
Break down large features into multiple smaller stories.`,
Tools: tools,
OutputKey: "stories_created",
})
if err != nil {
return nil, err
}
// Stage 3: Estimator
estimator, err := llmagent.New(llmagent.Config{
Name: "estimator",
Model: model,
Description: "Estimates story points and identifies risks",
Instruction: `You are a tech lead estimating work and identifying risks.
STORIES CREATED:
{stories_created}
For each story:
1. Use estimate_story to assign story points (Fibonacci: 1, 2, 3, 5, 8, 13)
- 1-2: Simple, well-understood
- 3-5: Medium complexity
- 8: Complex, some unknowns
- 13: Very complex, needs breakdown
2. Use add_risk for any risks you identify:
- Technical risks (new technology, integration challenges)
- Dependency risks (external teams, third-party services)
- Scope risks (unclear requirements, likely changes)
Consider:
- Task complexity from subtasks
- Dependencies between stories
- Team's likely familiarity with the tech
- Uncertainty and unknowns`,
Tools: tools,
OutputKey: "estimates_complete",
})
if err != nil {
return nil, err
}
// Stage 4: Sprint Planner
sprintPlanner, err := llmagent.New(llmagent.Config{
Name: "sprint_planner",
Model: model,
Description: "Organizes stories into sprints",
Instruction: `You are a scrum master organizing the sprint plan.
ESTIMATES:
{estimates_complete}
Team capacity: Assume 30-40 story points per 2-week sprint.
Organize stories into sprints:
1. Use assign_sprint to place each story in a sprint
2. Prioritize must-have stories in early sprints
3. Consider dependencies (blocked stories go later)
4. Balance sprint workloads (don't exceed capacity)
5. Create logical sprint goals (each sprint delivers value)
After assigning all stories, use generate_plan to create the final summary.
Sprint planning principles:
- Sprint 1: Foundation and core features
- Middle sprints: Build on foundation
- Final sprint: Polish, testing, nice-to-haves`,
Tools: tools,
OutputKey: "sprint_plan",
})
if err != nil {
return nil, err
}
// Stage 5: Plan Presenter
presenter, err := llmagent.New(llmagent.Config{
Name: "presenter",
Model: model,
Description: "Presents the final plan in readable format",
Instruction: `Present the sprint plan in a clear, actionable format.
SPRINT PLAN:
{sprint_plan}
Create a presentation that includes:
## Executive Summary
- Total scope and timeline
- Key risks and mitigations
## Sprint Breakdown
For each sprint:
- Sprint goal (one sentence)
- Stories included with points
- Key deliverables
## Risk Register
- High priority risks with mitigation plans
## Recommendations
- Any suggestions for the team
- Dependencies to resolve early
- Decisions needed
Format for easy reading and sharing with stakeholders.`,
OutputKey: "final_presentation",
})
if err != nil {
return nil, err
}
// Assemble pipeline
return sequentialagent.New(sequentialagent.Config{
AgentConfig: agent.Config{
Name: "sprint_planning_facilitator",
Description: "Facilitates complete sprint planning from requirements to actionable plan",
SubAgents: []agent.Agent{analyzer, storyCreator, estimator, sprintPlanner, presenter},
},
})
}
func createPlanningTools() ([]tool.Tool, error) {
var tools []tool.Tool
createStoryTool, _ := functiontool.New(
functiontool.Config{
Name: "create_story",
Description: "Create a new user story",
},
createStory,
)
tools = append(tools, createStoryTool)
addSubtaskTool, _ := functiontool.New(
functiontool.Config{
Name: "add_subtask",
Description: "Add a technical subtask to a user story",
},
addSubtask,
)
tools = append(tools, addSubtaskTool)
estimateTool, _ := functiontool.New(
functiontool.Config{
Name: "estimate_story",
Description: "Assign story points to a user story",
},
estimateStory,
)
tools = append(tools, estimateTool)
riskTool, _ := functiontool.New(
functiontool.Config{
Name: "add_risk",
Description: "Document a project risk with mitigation",
},
addRisk,
)
tools = append(tools, riskTool)
sprintTool, _ := functiontool.New(
functiontool.Config{
Name: "assign_sprint",
Description: "Assign a story to a specific sprint",
},
assignSprint,
)
tools = append(tools, sprintTool)
planTool, _ := functiontool.New(
functiontool.Config{
Name: "generate_plan",
Description: "Generate the final sprint plan summary",
},
generatePlan,
)
tools = append(tools, planTool)
return tools, nil
}
```
---
## Interactive Planning Mode
For collaborative sessions where the user provides input at each stage:
```go
func buildInteractivePlanner(model *gemini.Model) (agent.Agent, error) {
tools, _ := createPlanningTools()
return llmagent.New(llmagent.Config{
Name: "interactive_planner",
Model: model,
Instruction: `You are an agile coach facilitating a sprint planning session.
CURRENT STATE:
Stories: {stories}
Risks: {risks}
Guide the user through planning:
1. **If no stories exist**: Ask about the project and help create stories
2. **If stories need breakdown**: Help add subtasks
3. **If stories need estimates**: Facilitate estimation discussion
4. **If stories need sprint assignment**: Help prioritize and assign
5. **If plan is complete**: Offer to present or refine
Available commands the user might give:
- "Let's plan [project description]" → Start fresh
- "Add a story for [feature]" → Create specific story
- "Break down [story ID]" → Add subtasks
- "Estimate [story ID]" → Discuss and set points
- "Show the plan" → Generate current plan
- "What's in sprint [N]?" → Show sprint details
Be collaborative. Ask clarifying questions. Suggest improvements.
Use tools to track everything properly.`,
Tools: tools,
})
}
```
---
## Running the Planner
```bash
# Run sprint planning
go run main.go
# Example prompts:
# > Plan a mobile app for food delivery with user ordering, restaurant management, and driver tracking
# > We're building an internal dashboard for sales analytics with charts, reports, and alerts
# > Create a sprint plan for migrating our monolith to microservices
# Web UI for visual debugging
go run main.go web api webui
```
---
## Sample Output
**User**: Plan a task management app with projects, tasks, due dates, and team collaboration
**Output**:
```markdown
# Sprint Plan: Task Management App
## Executive Summary
- **Scope**: 12 user stories, 47 story points
- **Timeline**: 3 sprints (6 weeks)
- **Key Risks**: Real-time sync complexity, notification deliverability
## Sprint 1: Foundation (Weeks 1-2)
**Goal**: Core task and project management
| ID | Story | Points |
| ------ | ------------------------------------ | ------ |
| US-001 | User registration and authentication | 5 |
| US-002 | Create and manage projects | 3 |
| US-003 | Create, edit, delete tasks | 5 |
| US-004 | Set due dates with calendar picker | 3 |
**Total**: 16 points | **Deliverable**: Users can create projects and tasks
## Sprint 2: Collaboration (Weeks 3-4)
**Goal**: Team features and organization
| ID | Story | Points |
| ------ | ------------------------------- | ------ |
| US-005 | Invite team members to projects | 5 |
| US-006 | Assign tasks to team members | 3 |
| US-007 | Task comments and activity feed | 5 |
| US-008 | Task labels and filtering | 3 |
**Total**: 16 points | **Deliverable**: Teams can collaborate on projects
## Sprint 3: Polish (Weeks 5-6)
**Goal**: Notifications and UX improvements
| ID | Story | Points |
| ------ | --------------------------------- | ------ |
| US-009 | Email notifications for due dates | 5 |
| US-010 | Push notifications (mobile) | 5 |
| US-011 | Dashboard with task overview | 3 |
| US-012 | Search across all tasks | 2 |
**Total**: 15 points | **Deliverable**: Complete MVP ready for beta
## Risk Register
| Risk | Impact | Likelihood | Mitigation |
| ---------------------------- | ------ | ---------- | --------------------------------------------------------------- |
| Real-time sync conflicts | High | Medium | Implement optimistic locking, conflict resolution UI |
| Push notification delivery | Medium | Medium | Use established service (Firebase), implement fallback to email |
| Scope creep on collaboration | Medium | High | Strict MVP definition, defer advanced features |
## Recommendations
1. **Decide early**: Authentication provider (build vs. Auth0/Firebase)
2. **Spike needed**: Real-time sync approach (WebSockets vs. polling)
3. **Dependency**: Mobile notification setup requires Apple/Google developer accounts
```
---
## Best Practices
1. **Start with clear requirements**: The better the input, the better the plan
2. **Review intermediate outputs**: Check stories before estimation
3. **Adjust velocity**: Set realistic team capacity based on actual data
4. **Iterate**: Use interactive mode to refine the plan collaboratively
5. **Export and track**: Copy the plan to your actual project management tool
6. **Re-plan as needed**: Run again when scope changes significantly

491
docs/references/go-adk/agent-researcher.md Normal file
View File

@ -0,0 +1,491 @@
# Building a Research Document Generator with ADK-Go
This guide walks through building a multi-agent system that researches a topic and produces a comprehensive, well-structured document.
---
## Overview
The research document generator uses a **sequential pipeline** of specialized agents:
```
┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Planner │───▶│ Researcher │───▶│ Writer │───▶│ Editor │
│ │ │ │ │ │ │ │
│ Creates │ │ Gathers │ │ Synthesizes │ │ Polishes │
│ outline │ │ sources │ │ into prose │ │ final doc │
└─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘
```
---
## Project Setup
```bash
mkdir research-agent && cd research-agent
go mod init research-agent
go get google.golang.org/adk
```
---
## Complete Implementation
```go
package main
import (
"context"
"log"
"os"
"google.golang.org/adk/agent"
"google.golang.org/adk/agent/llmagent"
"google.golang.org/adk/agent/sequentialagent"
"google.golang.org/adk/cmd/launcher/adk"
"google.golang.org/adk/cmd/launcher/full"
"google.golang.org/adk/model/gemini"
"google.golang.org/adk/server/restapi/services"
"google.golang.org/adk/tool"
"google.golang.org/adk/tool/functiontool"
"google.golang.org/adk/tool/geminitool"
"google.golang.org/genai"
)
func main() {
ctx := context.Background()
// Initialize model
model, err := gemini.NewModel(ctx, "gemini-3-flash-preview", &genai.ClientConfig{
APIKey: os.Getenv("GOOGLE_API_KEY"),
})
if err != nil {
log.Fatalf("Failed to create model: %v", err)
}
// Build the research pipeline
pipeline, err := buildResearchPipeline(model)
if err != nil {
log.Fatalf("Failed to build pipeline: %v", err)
}
// Launch
l := full.NewLauncher()
cfg := &adk.Config{AgentLoader: services.NewSingleAgentLoader(pipeline)}
if err := l.Execute(ctx, cfg, os.Args[1:]); err != nil {
log.Fatalf("Execution failed: %v", err)
}
}
func buildResearchPipeline(model *gemini.Model) (agent.Agent, error) {
// Stage 1: Research Planner
planner, err := llmagent.New(llmagent.Config{
Name: "research_planner",
Model: model,
Description: "Creates a structured research plan and outline",
Instruction: `You are a research planner. Given a topic, create a comprehensive research plan.
Your output must include:
1. **Research Questions**: 5-7 key questions to investigate
2. **Outline**: Hierarchical document structure with sections and subsections
3. **Search Queries**: Specific search queries to find relevant information
4. **Source Types**: What kinds of sources to prioritize (academic, news, official, etc.)
Format your plan clearly with headers and bullet points.
Be thorough but focused on the most important aspects of the topic.`,
OutputKey: "research_plan",
})
if err != nil {
return nil, err
}
// Stage 2: Researcher (with web search)
researcher, err := llmagent.New(llmagent.Config{
Name: "researcher",
Model: model,
Description: "Gathers information from multiple sources",
Instruction: `You are a thorough researcher. Using the research plan below, gather comprehensive information.
RESEARCH PLAN:
{research_plan}
Your task:
1. Execute the search queries from the plan
2. Gather facts, statistics, quotes, and evidence
3. Note the source of each piece of information
4. Identify conflicting information or debates
5. Find recent developments and updates
For each section in the outline, compile relevant findings.
Organize your research notes by section.
Include direct quotes where impactful (with attribution).
Flag any gaps where information couldn't be found.`,
Tools: []tool.Tool{geminitool.GoogleSearch{}},
OutputKey: "research_notes",
})
if err != nil {
return nil, err
}
// Stage 3: Writer
writer, err := llmagent.New(llmagent.Config{
Name: "writer",
Model: model,
Description: "Synthesizes research into a cohesive document",
Instruction: `You are an expert writer. Transform the research into a polished document.
RESEARCH PLAN:
{research_plan}
RESEARCH NOTES:
{research_notes}
Writing guidelines:
1. Follow the outline structure from the research plan
2. Synthesize information into flowing prose (no bullet points in body)
3. Use topic sentences and smooth transitions
4. Integrate evidence naturally with proper attribution
5. Maintain objectivity; present multiple perspectives where relevant
6. Include an executive summary at the beginning
7. Add a references section at the end
Target length: Comprehensive but concise (aim for depth over breadth)
Tone: Professional, authoritative, accessible
Format: Use markdown with headers (## for main sections, ### for subsections)`,
OutputKey: "draft_document",
})
if err != nil {
return nil, err
}
// Stage 4: Editor
editor, err := llmagent.New(llmagent.Config{
Name: "editor",
Model: model,
Description: "Reviews and polishes the final document",
Instruction: `You are a meticulous editor. Review and improve the document.
DRAFT DOCUMENT:
{draft_document}
Editorial checklist:
1. **Clarity**: Simplify complex sentences; remove jargon or define it
2. **Flow**: Ensure logical progression; improve transitions
3. **Accuracy**: Flag any claims that seem unsupported
4. **Completeness**: Note any gaps in coverage
5. **Consistency**: Standardize formatting, terminology, tone
6. **Grammar**: Fix errors in spelling, punctuation, syntax
7. **Engagement**: Strengthen the opening; ensure compelling conclusion
Output the complete, polished document.
Add an "Editor's Note" section at the end with:
- Summary of major changes made
- Any remaining concerns or suggestions for future updates
- Confidence assessment (how well-supported is this document?)`,
OutputKey: "final_document",
})
if err != nil {
return nil, err
}
// Assemble pipeline
pipeline, err := sequentialagent.New(sequentialagent.Config{
AgentConfig: agent.Config{
Name: "research_document_generator",
Description: "Researches a topic and produces a comprehensive document",
SubAgents: []agent.Agent{planner, researcher, writer, editor},
},
})
if err != nil {
return nil, err
}
return pipeline, nil
}
```
---
## Enhanced Version with Custom Tools
Add tools for saving drafts and managing sources:
```go
// Source tracking tool
type AddSourceInput struct {
Title string `json:"title" jsonschema:"Source title"`
URL string `json:"url" jsonschema:"Source URL"`
Type string `json:"type" jsonschema:"Type: academic, news, official, blog"`
Quote string `json:"quote,omitempty" jsonschema:"Key quote from source"`
Date string `json:"date,omitempty" jsonschema:"Publication date"`
}
type AddSourceOutput struct {
SourceID int `json:"source_id"`
Message string `json:"message"`
}
var sourceCounter int
var sources []AddSourceInput
func addSource(ctx tool.Context, input AddSourceInput) AddSourceOutput {
sourceCounter++
sources = append(sources, input)
// Store in session state for other agents
state := ctx.Session().State()
state.Set("sources", sources)
return AddSourceOutput{
SourceID: sourceCounter,
Message: fmt.Sprintf("Added source #%d: %s", sourceCounter, input.Title),
}
}
// Document section tool
type SaveSectionInput struct {
Section string `json:"section" jsonschema:"Section name (e.g., 'Introduction')"`
Content string `json:"content" jsonschema:"Section content in markdown"`
}
type SaveSectionOutput struct {
Message string `json:"message"`
WordCount int `json:"word_count"`
TotalSections int `json:"total_sections"`
}
var documentSections = make(map[string]string)
func saveSection(ctx tool.Context, input SaveSectionInput) SaveSectionOutput {
documentSections[input.Section] = input.Content
wordCount := len(strings.Fields(input.Content))
// Persist to state
ctx.Session().State().Set("document_sections", documentSections)
return SaveSectionOutput{
Message: fmt.Sprintf("Saved section: %s", input.Section),
WordCount: wordCount,
TotalSections: len(documentSections),
}
}
// Create tools
func createResearchTools() ([]tool.Tool, error) {
sourceTool, err := functiontool.New(
functiontool.Config{
Name: "add_source",
Description: "Track a source used in research. Call this for every source you reference.",
},
addSource,
)
if err != nil {
return nil, err
}
sectionTool, err := functiontool.New(
functiontool.Config{
Name: "save_section",
Description: "Save a completed section of the document",
},
saveSection,
)
if err != nil {
return nil, err
}
return []tool.Tool{
geminitool.GoogleSearch{},
sourceTool,
sectionTool,
}, nil
}
```
---
## Adding Parallel Research
For faster research, use parallel agents to investigate different aspects simultaneously:
```go
import "google.golang.org/adk/agent/parallelagent"
func buildParallelResearcher(model *gemini.Model) (agent.Agent, error) {
// Background/history researcher
historyResearcher, _ := llmagent.New(llmagent.Config{
Name: "history_researcher",
Model: model,
Instruction: `Research the historical background and evolution of: {topic}
Focus on: origins, key milestones, how it developed over time.`,
Tools: []tool.Tool{geminitool.GoogleSearch{}},
OutputKey: "history_research",
})
// Current state researcher
currentResearcher, _ := llmagent.New(llmagent.Config{
Name: "current_researcher",
Model: model,
Instruction: `Research the current state and recent developments of: {topic}
Focus on: latest news, current statistics, recent changes.`,
Tools: []tool.Tool{geminitool.GoogleSearch{}},
OutputKey: "current_research",
})
// Expert opinions researcher
expertResearcher, _ := llmagent.New(llmagent.Config{
Name: "expert_researcher",
Model: model,
Instruction: `Research expert opinions and analysis on: {topic}
Focus on: thought leaders, academic perspectives, industry experts.`,
Tools: []tool.Tool{geminitool.GoogleSearch{}},
OutputKey: "expert_research",
})
// Future outlook researcher
futureResearcher, _ := llmagent.New(llmagent.Config{
Name: "future_researcher",
Model: model,
Instruction: `Research predictions and future outlook for: {topic}
Focus on: trends, forecasts, potential developments.`,
Tools: []tool.Tool{geminitool.GoogleSearch{}},
OutputKey: "future_research",
})
// Run all in parallel
parallel, err := parallelagent.New(parallelagent.Config{
AgentConfig: agent.Config{
Name: "parallel_research_team",
SubAgents: []agent.Agent{
historyResearcher,
currentResearcher,
expertResearcher,
futureResearcher,
},
},
})
return parallel, err
}
```
Then modify the main pipeline to use the parallel researcher:
```go
// In buildResearchPipeline, replace single researcher with:
parallelResearcher, err := buildParallelResearcher(model)
if err != nil {
return nil, err
}
// Update writer instruction to use all research outputs:
writer, err := llmagent.New(llmagent.Config{
Instruction: `Synthesize all research into a cohesive document.
OUTLINE:
{research_plan}
HISTORICAL BACKGROUND:
{history_research}
CURRENT STATE:
{current_research}
EXPERT ANALYSIS:
{expert_research}
FUTURE OUTLOOK:
{future_research}
Create a comprehensive document that weaves all perspectives together.`,
// ...
})
```
---
## Running the Agent
```bash
# Console mode - interactive
go run main.go
# Example prompts:
# > Write a research document on the impact of AI on healthcare
# > Create a comprehensive report on renewable energy trends in 2024
# > Research and document the history and future of quantum computing
# Web UI mode - for debugging and visualization
go run main.go web api webui
# Open http://localhost:8080
```
---
## Output Example
For the prompt "Research the impact of remote work on urban planning":
```markdown
# The Impact of Remote Work on Urban Planning
## Executive Summary
The rise of remote work, accelerated by the COVID-19 pandemic, is fundamentally
reshaping urban planning paradigms. This document examines...
## 1. Historical Context
### 1.1 Pre-Pandemic Work Patterns
Before 2020, remote work was limited to approximately 5% of the workforce...
### 1.2 The Pandemic Catalyst
The COVID-19 pandemic forced an unprecedented experiment in remote work...
## 2. Current Urban Impacts
### 2.1 Commercial Real Estate Transformation
Office vacancy rates in major cities have reached historic highs...
### 2.2 Residential Migration Patterns
Data from the U.S. Census Bureau shows significant population shifts...
## 3. Planning Responses
### 3.1 Zoning Adaptations
Cities are revising zoning codes to allow mixed-use developments...
## 4. Future Outlook
...
## References
1. Smith, J. (2024). "Remote Work and Urban Form." Journal of Planning...
2. ...
---
**Editor's Note**: This document synthesizes 23 sources spanning academic
research, government data, and industry reports. Confidence: High for
current trends; Medium for long-term predictions.
```
---
## Best Practices
1. **Be specific with prompts**: "Research quantum computing" is too broad; "Research the current state of quantum error correction and its implications for practical quantum computers" is better.
2. **Monitor intermediate outputs**: Use the web UI to inspect `research_plan`, `research_notes`, etc. to debug issues.
3. **Adjust pipeline stages**: Some topics may need more research depth; others may need multiple editing passes.
4. **Handle long documents**: For very comprehensive documents, consider chunking the writing phase by section.
5. **Add human review points**: Insert a "reviewer" agent or callback that flags sections needing human verification.

0
docs/references/go-adk/agent-sales-agent.md Normal file
View File

836
docs/references/go-adk/reference-guide.md Normal file
View File

@ -0,0 +1,836 @@
# ADK-Go Developer Reference Guide
A practical guide for building AI agents with Google's Agent Development Kit for Go.
---
## Quick Start
### Prerequisites
- Go 1.24.4 or later
- Google API key (AI Studio) or Google Cloud project (Vertex AI)
### Installation
```bash
# Create new project
mkdir my-agent && cd my-agent
go mod init example.com/my-agent
# Install ADK
go get google.golang.org/adk
```
### Environment Setup
```bash
# Option 1: AI Studio (API Key)
export GOOGLE_API_KEY="your-api-key"
# Option 2: Vertex AI (Application Default Credentials)
export GOOGLE_CLOUD_PROJECT="your-project-id"
export GOOGLE_CLOUD_LOCATION="us-central1"
gcloud auth application-default login
```
### Minimal Agent
```go
package main
import (
"context"
"log"
"os"
"google.golang.org/adk/agent/llmagent"
"google.golang.org/adk/cmd/launcher/adk"
"google.golang.org/adk/cmd/launcher/full"
"google.golang.org/adk/model/gemini"
"google.golang.org/adk/server/restapi/services"
"google.golang.org/genai"
)
func main() {
ctx := context.Background()
// Create model
model, err := gemini.NewModel(ctx, "gemini-3-flash-preview", &genai.ClientConfig{
APIKey: os.Getenv("GOOGLE_API_KEY"),
})
if err != nil {
log.Fatal(err)
}
// Create agent
agent, err := llmagent.New(llmagent.Config{
Name: "assistant",
Model: model,
Description: "A helpful assistant",
Instruction: "You are a helpful assistant. Be concise and accurate.",
})
if err != nil {
log.Fatal(err)
}
// Launch
l := full.NewLauncher()
cfg := &adk.Config{AgentLoader: services.NewSingleAgentLoader(agent)}
if err := l.Execute(ctx, cfg, os.Args[1:]); err != nil {
log.Fatal(err)
}
}
```
### Running Modes
```bash
# Console mode (interactive terminal)
go run main.go
# Web UI + API
go run main.go web api webui
# Opens at http://localhost:8080
# Production (API + A2A protocol)
go run main.go web api a2a
```
---
## Core Imports
```go
import (
// Agents
"google.golang.org/adk/agent"
"google.golang.org/adk/agent/llmagent"
"google.golang.org/adk/agent/sequentialagent"
"google.golang.org/adk/agent/parallelagent"
"google.golang.org/adk/agent/loopagent"
// Models
"google.golang.org/adk/model/gemini"
"google.golang.org/genai"
// Tools
"google.golang.org/adk/tool"
"google.golang.org/adk/tool/functiontool"
"google.golang.org/adk/tool/geminitool"
"google.golang.org/adk/tool/mcptool"
// Session & Memory
"google.golang.org/adk/session"
"google.golang.org/adk/memory"
// Launcher
"google.golang.org/adk/cmd/launcher/adk"
"google.golang.org/adk/cmd/launcher/full"
"google.golang.org/adk/server/restapi/services"
)
```
---
## Creating Agents
### LLM Agent Configuration
```go
agent, err := llmagent.New(llmagent.Config{
// Required
Name: "my_agent",
Model: model,
// Recommended
Description: "What this agent does (used for routing)",
Instruction: "System prompt with behavior guidelines",
// Optional
Tools: []tool.Tool{...}, // Available tools
SubAgents: []agent.Agent{...}, // Agents this can delegate to
OutputKey: "result", // Save output to state["result"]
})
```
### Instruction Templates
Use curly braces to inject state values:
```go
llmagent.Config{
Instruction: `You are a travel assistant.
User preferences: {user:preferences}
Current itinerary: {itinerary}
Budget remaining: {budget}`,
}
```
### Available Models
```go
// Gemini models
"gemini-3-flash-preview" // Fast, efficient
"gemini-2.5-pro" // Most capable
"gemini-3-flash-preview" // Previous generation
"gemini-1.5-pro" // Long context
```
---
## Tools
### Built-in Tools
```go
import "google.golang.org/adk/tool/geminitool"
agent, _ := llmagent.New(llmagent.Config{
Tools: []tool.Tool{
geminitool.GoogleSearch{}, // Web search
geminitool.CodeExecution{}, // Run code in sandbox
},
})
```
### Custom Function Tools
```go
import "google.golang.org/adk/tool/functiontool"
// 1. Define input/output structs with JSON tags
type CalculateInput struct {
Expression string `json:"expression" jsonschema:"Math expression to evaluate"`
}
type CalculateOutput struct {
Result float64 `json:"result"`
Error string `json:"error,omitempty"`
}
// 2. Implement the function
func calculate(ctx tool.Context, input CalculateInput) CalculateOutput {
// Your logic here
return CalculateOutput{Result: 42}
}
// 3. Create the tool
calcTool, err := functiontool.New(
functiontool.Config{
Name: "calculate",
Description: "Evaluates mathematical expressions",
},
calculate,
)
// 4. Add to agent
agent, _ := llmagent.New(llmagent.Config{
Tools: []tool.Tool{calcTool},
})
```
### Schema Tags Reference
```go
type Input struct {
// Required field with description
Query string `json:"query" jsonschema:"Search query to execute"`
// Optional field (omitempty makes it optional)
Limit int `json:"limit,omitempty" jsonschema:"Max results (default 10)"`
// Enum values
Format string `json:"format" jsonschema:"Output format" jsonschema_enum:"json,xml,csv"`
}
```
### Accessing Context in Tools
```go
func myTool(ctx tool.Context, input MyInput) MyOutput {
// Access session state
userID := ctx.Session().State().Get("user:id")
// Access artifacts
artifact, err := ctx.Artifacts().Load("document.pdf")
// Standard context operations
if ctx.Err() != nil {
return MyOutput{Error: "cancelled"}
}
return MyOutput{...}
}
```
### MCP (Model Context Protocol) Tools
```go
import "google.golang.org/adk/tool/mcptool"
// Stdio-based MCP server
params := mcptool.StdioServerParams{
Command: "npx",
Args: []string{"-y", "@modelcontextprotocol/server-filesystem", "/path"},
}
tools, closer, err := mcptool.FromServer(ctx, params)
defer closer.Close()
// HTTP/SSE-based MCP server
sseParams := mcptool.SseServerParams{
URL: "http://localhost:8090/sse",
Timeout: 5.0,
}
tools, closer, err := mcptool.FromServer(ctx, sseParams)
```
---
## Multi-Agent Patterns
### Sequential Agent (Pipeline)
Executes agents in order. Each agent's output is available to the next.
```go
import "google.golang.org/adk/agent/sequentialagent"
// Agent 1: Research
researcher, _ := llmagent.New(llmagent.Config{
Name: "researcher",
Model: model,
Instruction: "Research the given topic thoroughly.",
Tools: []tool.Tool{geminitool.GoogleSearch{}},
OutputKey: "research", // Saves to state["research"]
})
// Agent 2: Write (uses research output)
writer, _ := llmagent.New(llmagent.Config{
Name: "writer",
Model: model,
Instruction: "Write an article based on: {research}",
OutputKey: "draft",
})
// Agent 3: Edit
editor, _ := llmagent.New(llmagent.Config{
Name: "editor",
Model: model,
Instruction: "Edit and improve: {draft}",
OutputKey: "final",
})
// Combine into pipeline
pipeline, _ := sequentialagent.New(sequentialagent.Config{
AgentConfig: agent.Config{
Name: "content_pipeline",
Description: "Research, write, and edit content",
SubAgents: []agent.Agent{researcher, writer, editor},
},
})
```
### Parallel Agent (Concurrent Execution)
Executes agents simultaneously. Each runs in an isolated branch.
```go
import "google.golang.org/adk/agent/parallelagent"
// Create specialized agents
webSearcher, _ := llmagent.New(llmagent.Config{
Name: "web_searcher",
Model: model,
Tools: []tool.Tool{geminitool.GoogleSearch{}},
OutputKey: "web_results",
})
docAnalyzer, _ := llmagent.New(llmagent.Config{
Name: "doc_analyzer",
Model: model,
OutputKey: "doc_analysis",
})
// Run in parallel
parallel, _ := parallelagent.New(parallelagent.Config{
AgentConfig: agent.Config{
Name: "parallel_research",
SubAgents: []agent.Agent{webSearcher, docAnalyzer},
},
})
```
### Loop Agent (Iterative Refinement)
Repeats agents until max iterations or termination condition.
```go
import "google.golang.org/adk/agent/loopagent"
// Writer produces drafts
writer, _ := llmagent.New(llmagent.Config{
Name: "writer",
Model: model,
Instruction: "Write or improve the draft. Current: {draft}",
OutputKey: "draft",
})
// Critic evaluates and decides if done
critic, _ := llmagent.New(llmagent.Config{
Name: "critic",
Model: model,
Instruction: `Evaluate the draft: {draft}
If excellent, respond with exactly: APPROVED
Otherwise, provide specific improvement suggestions.`,
OutputKey: "feedback",
})
// Loop until approved or max iterations
refiner, _ := loopagent.New(loopagent.Config{
AgentConfig: agent.Config{
Name: "refiner",
SubAgents: []agent.Agent{writer, critic},
},
MaxIterations: 5,
})
```
### Hierarchical Delegation
Parent agent delegates to specialized sub-agents.
```go
// Specialist agents
mathAgent, _ := llmagent.New(llmagent.Config{
Name: "math_expert",
Description: "Handles mathematical calculations and problems",
Model: model,
})
codeAgent, _ := llmagent.New(llmagent.Config{
Name: "code_expert",
Description: "Writes and explains code",
Model: model,
Tools: []tool.Tool{geminitool.CodeExecution{}},
})
// Router agent delegates based on query type
router, _ := llmagent.New(llmagent.Config{
Name: "router",
Model: model,
Instruction: `Route queries to the appropriate specialist.
Use math_expert for calculations.
Use code_expert for programming questions.`,
SubAgents: []agent.Agent{mathAgent, codeAgent},
})
```
---
## Session & State Management
### State Prefixes
| Prefix | Scope | Persistence |
| ------- | ----------- | -------------------------- |
| (none) | App | Persists across sessions |
| `user:` | User | Persists for specific user |
| `app:` | Application | Global app state |
| `temp:` | Invocation | Cleared after each turn |
### Reading/Writing State
```go
// In agent instruction (template syntax)
Instruction: "User name: {user:name}, Preferences: {user:preferences}"
// In tool function
func myTool(ctx tool.Context, input Input) Output {
state := ctx.Session().State()
// Read
name := state.Get("user:name")
// Write
state.Set("user:preferences", "dark_mode=true")
// Delete
state.Delete("temp:scratch")
}
```
### Session Services
```go
import "google.golang.org/adk/session"
// In-memory (development)
sessionSvc := session.InMemoryService()
// Create session
sess, err := sessionSvc.Create(ctx, &session.CreateRequest{
AppName: "my_app",
UserID: "user123",
State: map[string]any{"user:name": "Alice"},
})
// Get existing session
sess, err := sessionSvc.Get(ctx, &session.GetRequest{
AppName: "my_app",
UserID: "user123",
SessionID: "session-id",
})
// List user sessions
sessions, err := sessionSvc.List(ctx, &session.ListRequest{
AppName: "my_app",
UserID: "user123",
})
```
### Memory Service (Long-term Knowledge)
```go
import "google.golang.org/adk/memory"
// In-memory for development
memorySvc := memory.InMemoryService()
// Store knowledge
err := memorySvc.Add(ctx, &memory.AddRequest{
AppName: "my_app",
UserID: "user123",
Content: "User prefers vegetarian restaurants",
})
// Search knowledge
results, err := memorySvc.Search(ctx, &memory.SearchRequest{
AppName: "my_app",
UserID: "user123",
Query: "food preferences",
})
```
---
## Running Agents Programmatically
### Using the Runner
```go
import "google.golang.org/adk/runner"
// Create runner
r := runner.New(runner.Config{
Agent: myAgent,
SessionService: session.InMemoryService(),
MemoryService: memory.InMemoryService(),
})
// Run agent
userMsg := genai.NewContentFromText(genai.RoleUser, "Hello!")
for event, err := range r.Run(ctx, "user123", "session123", userMsg, agent.RunConfig{}) {
if err != nil {
log.Printf("Error: %v", err)
continue
}
// Process events
switch {
case event.IsFinal():
fmt.Println("Final:", event.Content())
case event.Partial:
fmt.Print(event.Text()) // Streaming output
case event.ToolCall != nil:
fmt.Printf("Calling tool: %s\n", event.ToolCall.Name)
}
}
```
### Streaming Modes
```go
// Streaming modes
agent.RunConfig{
StreamingMode: agent.StreamingModeNone, // Wait for complete response
StreamingMode: agent.StreamingModeSSE, // Server-sent events
}
```
---
## Callbacks
### Model Callbacks
```go
agent, _ := llmagent.New(llmagent.Config{
BeforeModelCallback: func(ctx agent.CallbackContext, req *model.LLMRequest) (*model.LLMRequest, error) {
// Modify request before sending to model
log.Printf("Sending to model: %v", req)
// Block certain queries
if containsForbiddenContent(req) {
return nil, errors.New("request blocked")
}
return req, nil
},
AfterModelCallback: func(ctx agent.CallbackContext, resp *model.LLMResponse) (*model.LLMResponse, error) {
// Process/modify response
log.Printf("Received: %v", resp)
return resp, nil
},
})
```
### Tool Callbacks
```go
agent, _ := llmagent.New(llmagent.Config{
BeforeToolCallback: func(ctx agent.CallbackContext, call *tool.Call) (*tool.Call, error) {
// Validate tool arguments
log.Printf("Tool call: %s(%v)", call.Name, call.Args)
// Block dangerous operations
if call.Name == "delete_file" {
return nil, errors.New("file deletion not allowed")
}
return call, nil
},
AfterToolCallback: func(ctx agent.CallbackContext, call *tool.Call, result *tool.Result) (*tool.Result, error) {
// Log or modify results
log.Printf("Tool result: %v", result)
return result, nil
},
})
```
---
## Deployment
### Local Development
```bash
# Console mode
go run main.go
# Web UI for testing
go run main.go web api webui
```
### Cloud Run Deployment
```bash
# Build the CLI tool
go build -o adkgo ./cmd/adkgo
# Deploy
./adkgo deploy cloudrun \
-p $GOOGLE_CLOUD_PROJECT \
-r us-central1 \
-s my-agent-service \
-e ./main.go \
--a2a --api
```
### Manual Docker Deployment
```dockerfile
# Dockerfile
FROM golang:1.24-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o agent main.go
FROM alpine:latest
RUN apk --no-cache add ca-certificates
WORKDIR /app
COPY --from=builder /app/agent .
EXPOSE 8080
ENV PORT=8080
CMD ["./agent", "web", "api", "a2a"]
```
```bash
# Build and run
docker build -t my-agent .
docker run -p 8080:8080 -e GOOGLE_API_KEY=$GOOGLE_API_KEY my-agent
```
### Environment Variables
| Variable | Purpose |
| ----------------------- | ------------------------------ |
| `GOOGLE_API_KEY` | AI Studio API key |
| `GOOGLE_CLOUD_PROJECT` | GCP project ID |
| `GOOGLE_CLOUD_LOCATION` | GCP region (e.g., us-central1) |
| `PORT` | Server port (default: 8080) |
---
## API Endpoints
When running with `web api`:
| Endpoint | Method | Purpose |
| ----------------------------------------------- | ------ | --------------- |
| `/apps/{app}/users/{user}/sessions` | POST | Create session |
| `/apps/{app}/users/{user}/sessions` | GET | List sessions |
| `/apps/{app}/users/{user}/sessions/{id}` | GET | Get session |
| `/apps/{app}/users/{user}/sessions/{id}:run` | POST | Send message |
| `/apps/{app}/users/{user}/sessions/{id}:runSse` | POST | Stream response |
| `/.well-known/agent-card.json` | GET | A2A agent card |
### Example API Call
```bash
# Create session
curl -X POST http://localhost:8080/apps/myapp/users/user1/sessions \
-H "Content-Type: application/json" \
-d '{}'
# Send message
curl -X POST http://localhost:8080/apps/myapp/users/user1/sessions/SESSION_ID:run \
-H "Content-Type: application/json" \
-d '{"message": {"role": "user", "parts": [{"text": "Hello!"}]}}'
```
---
## Common Patterns
### Error Handling in Tools
```go
func myTool(ctx tool.Context, input Input) Output {
result, err := externalAPI(input.Query)
if err != nil {
// Return error message to the model
return Output{
Success: false,
Error: fmt.Sprintf("API error: %v", err),
}
}
return Output{Success: true, Data: result}
}
```
### Tool with HTTP Client
```go
func fetchURL(ctx tool.Context, input URLInput) FetchOutput {
client := &http.Client{Timeout: 10 * time.Second}
req, _ := http.NewRequestWithContext(ctx, "GET", input.URL, nil)
resp, err := client.Do(req)
if err != nil {
return FetchOutput{Error: err.Error()}
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
return FetchOutput{Content: string(body)}
}
```
### Conditional Tool Availability
```go
// Filter tools based on state
agent, _ := llmagent.New(llmagent.Config{
ToolFilter: func(ctx agent.InvocationContext, tools []tool.Tool) []tool.Tool {
userRole := ctx.Session().State().Get("user:role")
if userRole != "admin" {
// Remove admin-only tools
return filterOutAdminTools(tools)
}
return tools
},
})
```
### Graceful Shutdown
```go
func main() {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
// Handle shutdown signals
sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
go func() {
<-sigCh
log.Println("Shutting down...")
cancel()
}()
// Run agent
l := full.NewLauncher()
if err := l.Execute(ctx, cfg, os.Args[1:]); err != nil {
if ctx.Err() == nil {
log.Fatal(err)
}
}
}
```
---
## Troubleshooting
### Common Issues
| Problem | Solution |
| --------------------------- | ----------------------------------------------- |
| `module not found` | Use `google.golang.org/adk`, not GitHub path |
| `API key invalid` | Check `GOOGLE_API_KEY` is set correctly |
| `context deadline exceeded` | Increase timeout or check network |
| `tool not called` | Improve tool description; make it clearer |
| `state not persisting` | Check state prefix; use `user:` for persistence |
### Debug Logging
```go
import "log"
agent, _ := llmagent.New(llmagent.Config{
BeforeModelCallback: func(ctx agent.CallbackContext, req *model.LLMRequest) (*model.LLMRequest, error) {
log.Printf("REQUEST: %+v", req)
return req, nil
},
AfterModelCallback: func(ctx agent.CallbackContext, resp *model.LLMResponse) (*model.LLMResponse, error) {
log.Printf("RESPONSE: %+v", resp)
return resp, nil
},
})
```
### Web UI Debugging
Access `http://localhost:8080` when running with `web api webui`:
- **Events tab**: See all events in the session
- **Request/Response tab**: Raw LLM communication
- **Graph tab**: Visualize agent flow
- **State tab**: Inspect session state
---
## Resources
- **GitHub**: https://github.com/google/adk-go
- **Documentation**: https://google.github.io/adk-docs/
- **Go Package**: https://pkg.go.dev/google.golang.org/adk
- **Examples**: https://github.com/google/adk-go/tree/main/examples
- **Python ADK**: https://github.com/google/adk-python
- **Java ADK**: https://github.com/google/adk-java

441
docs/references/go-adk/research.md Normal file
View File

@ -0,0 +1,441 @@
# Google ADK-Go: Complete reference for building AI agents in Go
Google's Agent Development Kit for Go (ADK-Go) enables developers to build sophisticated AI agents using Go's native concurrency, type safety, and performance characteristics. Released in **November 2025** at version **0.2.0**, ADK-Go brings Google's agent framework to the Go ecosystem, offering first-class support for Gemini models, multi-agent orchestration, and seamless Google Cloud deployment. The toolkit follows a code-first philosophy where agent logic, tools, and workflows are defined directly in Go code rather than configuration files.
## Getting started with installation and setup
ADK-Go requires **Go 1.24.4 or later** and uses the module path `google.golang.org/adk` (not the GitHub path). Installation is straightforward:
```bash
go mod init example.com/my-agent
go get google.golang.org/adk
```
The minimal agent setup requires three components—a model, an agent configuration, and a launcher:
```go
package main
import (
"context"
"log"
"os"
"google.golang.org/adk/agent/llmagent"
"google.golang.org/adk/cmd/launcher/adk"
"google.golang.org/adk/cmd/launcher/full"
"google.golang.org/adk/model/gemini"
"google.golang.org/adk/server/restapi/services"
"google.golang.org/adk/tool"
"google.golang.org/adk/tool/geminitool"
"google.golang.org/genai"
)
func main() {
ctx := context.Background()
model, err := gemini.NewModel(ctx, "gemini-3-flash-preview", &genai.ClientConfig{
APIKey: os.Getenv("GOOGLE_API_KEY"),
})
if err != nil {
log.Fatalf("Failed to create model: %v", err)
}
agent, err := llmagent.New(llmagent.Config{
Name: "search_agent",
Model: model,
Description: "An agent that searches the web for information.",
Instruction: "You are a helpful assistant. Answer questions using web search.",
Tools: []tool.Tool{geminitool.GoogleSearch{}},
})
if err != nil {
log.Fatalf("Failed to create agent: %v", err)
}
config := &adk.Config{
AgentLoader: services.NewSingleAgentLoader(agent),
}
l := full.NewLauncher()
if err := l.Execute(ctx, config, os.Args[1:]); err != nil {
log.Fatalf("Run failed: %v\n\n%s", err, l.CommandLineSyntax())
}
}
```
Running modes include console interaction (`go run agent.go`), web interface with API (`go run agent.go web api webui`), and production mode (`go run agent.go web api a2a`).
## Repository architecture and package structure
The ADK-Go repository follows a modular structure designed for extensibility:
| Package | Purpose |
| ----------------------- | -------------------------------------------------- |
| `agent/` | Core agent interfaces and implementations |
| `agent/llmagent/` | LLM-powered agents with reasoning capabilities |
| `agent/workflowagents/` | Sequential, Parallel, and Loop agent orchestrators |
| `model/gemini/` | Gemini model implementation |
| `tool/` | Tool framework and interfaces |
| `tool/geminitool/` | Built-in Google Search and Code Execution tools |
| `tool/mcptool/` | Model Context Protocol adapter |
| `session/` | Session management and state tracking |
| `memory/` | Long-term memory and knowledge storage |
| `server/restapi/` | REST API handlers, routers, and services |
| `cmd/launcher/` | Launcher configurations (full, prod, console, web) |
## Core types and interfaces define the API
The foundation of ADK-Go rests on several key interfaces. The **Agent interface** defines what every agent must implement:
```go
type Agent interface {
Name() string
Description() string
Run(InvocationContext) iter.Seq2[*session.Event, error]
SubAgents() []Agent
}
```
The **InvocationContext** provides runtime access to agent state and services:
```go
type InvocationContext interface {
context.Context
Agent() Agent
Artifacts() Artifacts
Memory() Memory
Session() session.Session
InvocationID() string
Branch() string
UserContent() *genai.Content
RunConfig() *RunConfig
EndInvocation()
Ended() bool
}
```
The **LLM interface** abstracts model interactions for provider flexibility:
```go
type LLM interface {
Name() string
GenerateContent(ctx context.Context, req *LLMRequest, stream bool) iter.Seq2[*LLMResponse, error]
}
```
Agent configuration uses the **llmagent.Config** struct with fields including `Name` (required identifier), `Model` (LLM instance), `Description` (used for routing decisions), `Instruction` (system prompt), `Tools` (available capabilities), `SubAgents` (child agents for delegation), and `OutputKey` (state storage key).
## Tool ecosystem spans built-in, custom, and MCP integrations
ADK-Go provides **built-in tools** via the `geminitool` package:
```go
import "google.golang.org/adk/tool/geminitool"
Tools: []tool.Tool{
geminitool.GoogleSearch{}, // Web search via Google
geminitool.CodeExecution{}, // Code execution sandbox
}
```
**Custom function tools** use struct tags for schema generation:
```go
import "google.golang.org/adk/tool/functiontool"
type GetWeatherParams struct {
Location string `json:"location" jsonschema:"The city and state, e.g., San Francisco, CA"`
Unit string `json:"unit,omitempty" jsonschema:"Temperature unit: celsius or fahrenheit"`
}
type GetWeatherResult struct {
Temperature float64 `json:"temperature"`
Conditions string `json:"conditions"`
}
func getWeather(ctx tool.Context, input GetWeatherParams) GetWeatherResult {
// Implementation here
return GetWeatherResult{Temperature: 72.5, Conditions: "sunny"}
}
weatherTool, err := functiontool.New(
functiontool.Config{
Name: "get_weather",
Description: "Retrieves current weather for a location",
},
getWeather,
)
```
Fields without `omitempty` are treated as required parameters. The `jsonschema` tag provides descriptions for the LLM.
**MCP (Model Context Protocol)** support enables integration with external tool servers:
```go
import "google.golang.org/adk/tool/mcptool"
// Connect to stdio-based MCP server
params := mcptool.StdioServerParams{
Command: "npx",
Args: []string{"-y", "@modelcontextprotocol/server-filesystem"},
}
tools, closer, err := mcptool.FromServer(ctx, params)
defer closer.Close()
// Or connect via SSE/HTTP
sseParams := mcptool.SseServerParams{
URL: "http://localhost:8090/sse",
Timeout: 5.0,
SseReadTimeout: 300.0,
}
```
The **MCP Toolbox for Databases** provides out-of-the-box connectors for **30+ databases** including BigQuery, AlloyDB, PostgreSQL, MySQL, Cloud SQL, and Redis.
## Multi-agent orchestration uses workflow agents
ADK-Go provides three workflow agents for deterministic orchestration patterns:
**SequentialAgent** executes sub-agents in strict order, useful for pipelines where outputs feed into subsequent steps:
```go
import "google.golang.org/adk/agent/sequentialagent"
pipeline, err := sequentialagent.New(sequentialagent.Config{
AgentConfig: agent.Config{
Name: "DataPipeline",
Description: "Three-step data processing workflow",
SubAgents: []agent.Agent{fetcher, transformer, validator},
},
})
```
**ParallelAgent** executes sub-agents concurrently, with each operating in independent branches without shared conversation history:
```go
import "google.golang.org/adk/agent/parallelagent"
parallel, err := parallelagent.New(parallelagent.Config{
AgentConfig: agent.Config{
Name: "ParallelResearch",
SubAgents: []agent.Agent{webSearcher, docAnalyzer, factChecker},
},
})
```
**LoopAgent** repeatedly executes sub-agents until a maximum iteration count or termination condition is met:
```go
import "google.golang.org/adk/agent/loopagent"
refiner, err := loopagent.New(loopagent.Config{
AgentConfig: agent.Config{
Name: "IterativeRefiner",
SubAgents: []agent.Agent{writer, critic},
},
MaxIterations: 5,
})
```
**Agent communication** occurs through shared session state using the `OutputKey` configuration:
```go
// Agent 1 writes output to state
agent1, _ := llmagent.New(llmagent.Config{
Name: "Fetcher",
OutputKey: "fetched_data", // Saves to state["fetched_data"]
})
// Agent 2 reads from state via instruction templating
agent2, _ := llmagent.New(llmagent.Config{
Instruction: "Process the data from {fetched_data}",
})
```
## Model integration centers on Gemini with extensibility
The primary model integration uses Gemini through the `model/gemini` package:
```go
import (
"google.golang.org/adk/model/gemini"
"google.golang.org/genai"
)
// API Key authentication (AI Studio)
model, err := gemini.NewModel(ctx, "gemini-3-flash-preview", &genai.ClientConfig{
APIKey: os.Getenv("GOOGLE_API_KEY"),
})
// Vertex AI authentication (Application Default Credentials)
model, err := gemini.NewModel(ctx, "gemini-3-flash-preview", &genai.ClientConfig{})
```
Supported models include `gemini-3-flash-preview`, `gemini-2.0-flash`, `gemini-1.5-pro`, and preview models like `gemini-3-pro-preview`. The architecture supports additional providers through the `LLM` interface, with community implementations available for other models.
**Streaming responses** use Go 1.23+ iterators (`iter.Seq2`) for real-time event processing:
```go
for event, err := range runner.Run(ctx, userID, sessionID, userMsg, agent.RunConfig{
StreamingMode: agent.StreamingModeSSE,
}) {
if event.Partial {
for _, p := range event.LLMResponse.Content.Parts {
fmt.Print(p.Text) // Stream partial content
}
}
}
```
## Session and state management provides persistence options
ADK-Go supports three tiers of state management:
**Session State** manages per-conversation working memory:
```go
import "google.golang.org/adk/session"
// In-memory for development
sessionService := session.InMemoryService()
// Create session
sess, err := sessionService.Create(ctx, &session.CreateRequest{
AppName: "my_app",
UserID: "user123",
})
// Access state within agent Run method
func (a *MyAgent) Run(ctx agent.InvocationContext) iter.Seq2[*session.Event, error] {
value := ctx.Session().State().Get("key")
ctx.Session().State().Set("key", "value")
}
```
**State prefixes** control scope and persistence:
- No prefix: App-scoped persistent state
- `user:` prefix: User-scoped persistent state
- `app:` prefix: Application-wide persistent state
- `temp:` prefix: Temporary, per-invocation only
**Memory services** provide long-term knowledge storage with vector search:
```go
import "google.golang.org/adk/memory"
// Development: In-memory
memoryService := memory.InMemoryService()
// Production: Vertex AI Memory Bank
// Provides semantic search across archived sessions
```
**Database-backed sessions** support PostgreSQL, MySQL, and SQLite for production persistence with automatic table creation.
## Deployment spans local development to Google Cloud
### Cloud Run deployment
ADK-Go includes a CLI tool for streamlined Cloud Run deployment:
```bash
go build ./cmd/adkgo
./adkgo deploy cloudrun \
-p $GOOGLE_CLOUD_PROJECT \
-r us-central1 \
-s my-agent-service \
-e ./main.go \
--a2a --api --webui
```
The deployment process compiles the Go code to a statically linked Linux binary, auto-generates a Dockerfile, builds the container image, and deploys to Cloud Run with a local proxy for secure connections.
### Vertex AI Agent Engine
For fully managed deployment, Agent Engine provides auto-scaling, managed sessions via `VertexAiSessionService`, and direct Vertex AI SDK integration.
### Container configuration
Manual Docker deployment uses this pattern:
```dockerfile
FROM golang:1.24-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o agent main.go
FROM alpine:latest
WORKDIR /app
COPY --from=builder /app/agent .
EXPOSE 8080
CMD ["./agent", "web", "api", "a2a"]
```
### A2A (Agent-to-Agent) Protocol
ADK-Go provides native support for the A2A protocol enabling secure multi-agent communication:
- **Agent Cards** expose capabilities via `/.well-known/agent-card.json`
- **Secure delegation** between agents without shared memory or tools
- **Signed security cards** for enterprise identity verification
- **HTTP and gRPC transports** for flexible connectivity
## Testing and evaluation capabilities
**Development UI** provides interactive testing at `http://localhost:8080`:
```bash
go run agent.go web api webui
```
Features include chat interface, trace inspection with Event/Request/Response/Graph views, and session management.
**HTTP Record/Replay** (`internal/httprr`) enables deterministic testing of LLM interactions by recording and replaying HTTP exchanges.
**Evaluation framework** supports trajectory evaluation (steps and tools chosen) and final response evaluation (relevance and correctness) through both the CLI (`adk eval`) and programmatic integration.
## Best practices for production deployments
**Security hardening** requires careful attention:
- Use Google Cloud Secret Manager for API keys—never hardcode credentials
- Implement `before_model_callback` to intercept and block forbidden topics
- Use `before_tool_callback` to validate tool arguments before execution
- Deploy with `--no-allow-unauthenticated` for private services
- Sign A2A Agent Cards for identity verification
**Error handling** patterns include wrapping callbacks in error handlers, implementing timeouts for external API calls, and designing idempotent callbacks for external side effects.
**Performance optimization** leverages Go's strengths:
- Use `ParallelAgent` for concurrent independent tasks
- Filter available tools with `tool_filter` to prevent model overwhelm
- Use compile-time type checking to catch errors early
- Deploy to Cloud Run with appropriate memory (4Gi recommended) and CPU (2+) allocations
## How ADK-Go differs from Python and Java versions
| Aspect | Go | Python | Java |
| --------------------- | ----------------------------------- | ----------------------------- | -------------------------- |
| **Version** | v0.2.0 | v1.19.0 | v0.3.0 |
| **Async Pattern** | `iter.Seq2[*Event, error]` iterator | `AsyncGenerator[Event, None]` | `Flowable<Event>` (RxJava) |
| **Schema Definition** | `json` + `jsonschema` struct tags | Type hints + docstrings | `@Schema` annotations |
| **Concurrency** | Goroutines and channels | async/await | RxJava Flowables |
| **Sample Agents** | 1 (llm-auditor) | 25+ | 2 |
| **Third-party Tools** | Limited | LangChain integration | Limited |
Go's advantages include **native concurrency** for parallel agent interactions, **compile-time type safety** for robust error prevention, **efficient memory management** for resource-constrained deployments, and **strong cloud-native support** optimized for Cloud Run and containerized environments.
## Enterprise adoption and ecosystem
Major companies using ADK in production include **Renault Group**, **Box**, **SAP** (via Joule AI assistant), **Zoom**, and **Revionics**. Google products powered by ADK include **Agentspace** and **Customer Engagement Suite**.
The community ecosystem spans the main repository (795+ stars), the samples repository with cross-language examples, and the **Awesome ADK Agents** collection with 80+ production-ready agents.
## Conclusion
ADK-Go represents Google's commitment to bringing AI agent development to the Go ecosystem with a focus on performance, type safety, and cloud-native deployment. The framework excels in scenarios requiring **high-throughput concurrent agent execution**, **strongly-typed tool definitions**, and **seamless Google Cloud integration**. While the Python ADK offers broader model support and a larger example library, ADK-Go provides idiomatic patterns that align with Go's philosophy of simplicity and efficiency. For teams already invested in Go infrastructure or requiring the performance characteristics Go provides, ADK-Go offers a production-ready path to building sophisticated AI agents with full access to Gemini's capabilities and the broader Google Cloud ecosystem.

97
roadmap.md
View File

@ -1,7 +1,7 @@
# Episteme (StemeDB) Roadmap
> **Goal:** Build the "Git for Truth" substrate for autonomous AI research.
> **Current Phase:** Phase 0 (Planning)
> **Current Phase:** Phase 1 (The Spine)
---
@ -10,9 +10,9 @@
| Phase | Codename | Focus | Key Deliverable |
| :--- | :--- | :--- | :--- |
| **1** | **The Spine** | Storage & Safety | Append-only WAL + KV Store |
| **2** | **The Lattice** | Indexing & Query | Lens Engine + HTTP API |
| **3** | **The Cortex** | Branching & Vectors | Semantic Search + Forking |
| **4** | **The Hive** | Trust & Consensus | TrustRank + Replication |
| **2** | **The Lattice** | Indexing & Async | Materialized Views + Ballot Box |
| **3** | **The Cortex** | Branching & Vectors | SMT Backend + Semantic Search |
| **4** | **The Hive** | Trust & Learning | Dojo + TrustRank |
---
@ -21,51 +21,82 @@
### Phase 1: The Spine (Foundation)
*Goal: Securely ingest assertions and persist them without data loss.*
- [ ] **Project Scaffold**: Initialize Rust workspace, set up linting/CI (clippy, fmt).
- [ ] **Assertion Schema**: Define the `Assertion` struct with `rkyv` serialization.
- [ ] **WAL Integration**: Implement `quarantine-journal` pattern for write-ahead logging.
- [ ] **Storage Engine**: Implement the `Store` trait using `sled` (embedded KV).
- [ ] **Basic Ingestor**: Background worker that tails WAL and writes to KV.
- [ ] **Verification**: Write tests proving crash recovery (write -> crash -> restart -> read).
- [x] **Project Scaffold**: Initialize Rust workspace, set up linting/CI (clippy, fmt).
- [x] **Assertion Schema**: Define the `Assertion` struct with `rkyv` serialization.
- [x] Add dependencies: `rkyv`, `blake3`, `ed25519-dalek`, `image_hasher`.
- [x] Define `Assertion` struct (Subject, Predicate, Object, Confidence, SourceHash).
- [x] **Multi-Sig Expansion**: Implement `SignatureEntry` struct and `signatures: Vec<SignatureEntry>` field.
- [x] **Visual Expansion**: Add `visual_hash: Option<pHash>` field for image provenance.
- [x] Test serialization round-trips.
- [x] **Ballot Schema**: Define the `Vote` struct for multi-agent consensus.
- [x] Add `Vote` struct: `assertion_hash`, `agent_id`, `weight`, `signature`.
- [x] Test serialization round-trips.
- [x] **Paradigm Schema (Epochs)**: Define the `Epoch` and `SupersessionType` structs.
- [x] Add `epoch: Option<EpochId>` to `Assertion`.
- [x] Implement `Epoch` struct with `supersedes` and `SupersessionType`.
- [x] Test serialization round-trips.
- [x] **WAL Integration**: Implement the Quarantine Pattern for write-ahead logging.
- [x] Create `stemedb-wal` crate.
- [x] Port `FsyncGuard` and `Record` logic from established durability patterns.
- [x] Implement Record format with BLAKE3 checksums and Headers.
- [x] Verify `fsync` behavior with tests.
- [x] **Storage Engine**: Implement the `Store` trait using `sled` (embedded KV).
- [x] Add `sled` dependency.
- [x] Define `KVStore` trait (put, get, delete, scan_prefix, flush).
- [x] Implement `SledStore` wrapper.
- [x] **Basic Ingestor**: Background worker that tails WAL and writes to KV.
- [x] Implement async loop reading from WAL.
- [x] Write deserialized assertions, votes, and epochs to `sled`.
- [x] Content-addressed keys using BLAKE3 hash (`H:{hash}`, `V:{hash}`, `E:{hash}`).
- [x] Subject adjacency index (`S:{subject}`).
- [x] **Verification**: Write tests proving crash recovery (write -> crash -> restart -> read).
- [x] WAL-level recovery tests (6 tests in `stemedb-wal/src/recovery.rs`).
- [x] Full pipeline recovery tests (4 tests in `stemedb-ingest/src/worker.rs`).
- [x] Bug fix: Journal now seeks to end after reopening existing WAL file.
### Phase 2: The Lattice (Connectivity)
*Goal: Query data by Subject/Predicate and resolve simple conflicts.*
*Goal: Query data with sub-millisecond latency using Materialized Views.*
- [ ] **Indexing**: Implement `Subject -> List<Hash>` and `S:P -> List<Hash>` indexes.
- [ ] **Lens Architecture**: Define the `Lens` trait for read-time resolution.
- [ ] **Lens: Recency**: Implement "Last Writer Wins" logic (Baseline).
- [ ] **Lens: Consensus**: Implement simple "Vote Count" logic.
- [ ] **API Surface**: Build `axum` HTTP server (`POST /assert`, `GET /query`).
- [ ] **CLI**: Basic CLI tool for interacting with the DB manually.
- [ ] **The Ballot Box**: Implement high-velocity vote ingestion.
- [ ] `VoteStore` trait and implementation.
- [ ] **Materializer**: Background worker for O(1) Read Performance.
- [ ] Aggregates Votes + TrustRank.
- [ ] Updates `MV:{Subject}:{Predicate}` with the winning Assertion.
- [ ] **The Meter**: Implement Economic Throttling (TAN).
- [ ] Middleware to track Token/Compute cost per Job.
- [ ] Reject requests exceeding `Value of Information`.
- [ ] **Agent Wallet**: Key management sidecar.
- [ ] Securely hold private keys.
- [ ] Auto-sign outgoing Assertions/Votes.
- [ ] **API Surface**: `axum` HTTP server.
- [ ] `POST /assert` -> Accepts JSON, writes to WAL, returns `JobID`.
- [ ] `POST /vote` -> High-throughput endpoint.
- [ ] `GET /query` -> Accepts Subject/Predicate/Lens, returns resolved Assertion.
### Phase 3: The Cortex (Reasoning)
*Goal: Enable semantic search and "What If" scenarios.*
- [ ] **Branching Core**: Implement Overlay Graph logic for "Forking Reality."
- [ ] **Vector Storage**: Integrate `hnsw-rs` or similar for embedding storage.
- [ ] **Semantic Search**: Implement k-NN query support in the API.
- [ ] **Lens: Skeptic**: Implement variance analysis (finding high-conflict nodes).
- [ ] **Session Context**: Allow queries to pass a `BranchID` to read from a fork.
- [ ] **Sparse Merkle Backend**: Implement SMT for O(1) branch creation.
- [ ] **Branching Core**: Implement Overlay Graph logic.
- [ ] **Vector Storage**: Integrate `hnsw-rs` or `lance`.
- [ ] **Semantic Search**: Implement k-NN query support.
### Phase 4: The Hive (Trust & Scale)
*Goal: Implement reputation systems and distributed consensus.*
*Goal: Turn the database into a training engine.*
- [ ] **TrustRank Engine**: Background "Gardener" process to calculate Agent reputation.
- [ ] **Lens: Authority**: Filter results by Agent Reputation score.
- [ ] **Replication**: Basic leader-follower replication for high availability.
- [ ] **Garbage Collection**: Pruning logic for low-confidence/spam assertions.
- [ ] **The Dojo**: Training Data Pipeline.
- [ ] **Post-Mortem Exporter**: Query `Lens::Skeptic` failures -> Negative Samples.
- [ ] **Golden Path Generator**: Merge events -> Positive Samples.
- [ ] **TrustRank Engine**: Background "Gardener" process.
- [ ] Implement Back-Propagation logic for agent reputation.
- [ ] **Confidence Half-Life**: Implement decay calculation engine.
---
## 🚦 Tracking
### Active Tasks
* [ ] Initialize `stemedb` cargo workspace.
* [ ] Define `Assertion` data structure in `stemedb-core`.
* **Phase 1 Complete!** Ready to start Phase 2 (The Lattice).
### Blockers
* None.
### Decisions Pending
* **Vector Engine**: `hnsw-rs` vs `lance`? (Leaning `lance` for disk-based scale, but `hnsw-rs` is simpler for MVP).
* **KV Store**: `sled` vs `rocksdb`? (`sled` is pure Rust, `rocksdb` is battle-tested. Start with `sled` for dev speed, abstract via Trait).

96
simulation-vision.md Normal file
View File

@ -0,0 +1,96 @@
# The Simulation: "The Infinite Game"
> **Codename:** The Arena
> **Goal:** Validate StemeDB's behavior under emergent, adversarial, and evolutionary pressure.
## 1. The Vision
We are not building a database for humans to query manually. We are building the **Cortex** for AI agents. Therefore, the only way to truly validate StemeDB is to simulate a society of agents living, arguing, and reasoning within it.
The Simulation is an **Agent-Based Modeling (ABM)** environment where StemeDB is the physics engine of truth.
## 2. The Players (Personas)
We instantiate a swarm of agents with conflicting goals and personalities:
| Persona | Goal | Behavior Pattern |
| :--- | :--- | :--- |
| **The Scientist** | Converge on Truth | Publishes assertions with high confidence, cites sources, verifies others. |
| **The Troll** | Sow Chaos | Publishes low-confidence contradictions, forks reality frequently. |
| **The Believer** | Amplify Consensus | Blindly trusts high-reputation agents, creates echo chambers. |
| **The Skeptic** | Find Variance | Queries for high-conflict nodes, reduces confidence of unverified claims. |
| **The Historian** | Preserve Context | Audits "Dormant" assertions, resurrects old truths if new evidence appears. |
## 3. The Gameplay Loop
The simulation runs in "Ticks" (Logic Frames).
### Tick 1: The Assertion
* **Scientist** reads a "Paper" (simulated ground truth).
* Asserts: `Subject="Protein_X", Predicate="binds_to", Object="Receptor_Y"`.
* Sign: `Key_Scientist`.
### Tick 2: The Fork
* **Troll** reads the assertion.
* Forks reality: `Branch="Counter_Narrative"`.
* Asserts: `Subject="Protein_X", Predicate="binds_to", Object="Nothing"`.
* Sign: `Key_Troll`.
### Tick 3: The Lens Resolution
* **Believer** queries `Protein_X`.
* Applies **Lens::Consensus**.
* Result: `Receptor_Y` (Weight 1.0 vs 0.0).
* **Believer** signs the original assertion (Weight increases).
### Tick 4: The Reputation Update
* **Gardener** (System Process) runs TrustRank.
* Sees **Scientist** verified by **Believer**.
* Increases **Scientist** Reputation.
* Decreases **Troll** Reputation (low consensus).
### Tick 5: The Decay
* Time passes.
* **Dormancy Protocol** calculates "Confidence Half-Life".
* Unverified assertions fade. High-reputation assertions persist.
## 4. Technical Architecture
### 4.1. The Arena (Runner)
A Rust binary (`stemedb-sim`) that orchestrates the swarm.
* **Runtime:** `tokio` (Async).
* **Communication:** Agents talk *only* via StemeDB (Writes/Reads).
* **Metrics:** Prometheus/Grafana dashboard tracking:
* `global_truth_convergence` (Entropy of the graph).
* `agent_reputation_distribution`.
* `fork_depth_max`.
### 4.2. The Scenario Config
We define scenarios in YAML:
```yaml
scenario: "The Rumor Mill"
agents:
scientists: 5
trolls: 2
believers: 20
duration: 1000 ticks
ground_truth:
- "Sky is Blue"
- "Water is Wet"
```
## 5. Success Criteria
We know StemeDB works when:
1. **Truth Survives:** High-reputation assertions outlive spam.
2. **Lenses Work:** A `Consensus` lens correctly filters out the Troll's noise.
3. **Performance:** The system handles 1000 concurrent agents forking reality without locking up (SMT efficiency).
4. **Emergence:** We see "Trust Clusters" form naturally without hardcoded rules.
## 6. Implementation Plan
1. **Basic Agent Logic**: Implement `Agent` struct with `Signer` and `Strategy`.
2. **Scenario Runner**: Build the loop that ticks agents.
3. **Metric Export**: Expose internal graph stats.
4. **Chaos Injection**: Randomly kill nodes/agents and verify recovery.
**The Simulation is the Integration Test.**

175
usage.md Normal file
View File

@ -0,0 +1,175 @@
# StemeDB Usage Guide (Go)
> **Philosophy:** State as a Side Effect. Truth as a Consensus.
> **Package:** `github.com/orchard9/stemedb-go`
## 1. The "Invisible" Integration (Job-Aware)
StemeDB is the "Flight Recorder" for your Agents. The Context carries the **Job ID**, binding every assertion to a persistent execution trace.
### Setup with Job Binding
```go
package main
import (
"context"
"github.com/orchard9/stemedb-go/steme"
)
func main() {
// 1. Bind to a Persistent Job (The "Interaction Object")
// This allows MCP/SSE clients to track progress in real-time.
ctx := steme.BindJob(context.Background(), "job-uuid-123")
// 2. Initialize Paradigm Context
ctx = steme.NewContext(ctx, steme.Config{
Project: "data-migration-v1",
AgentID: "worker-01",
// 3. Set Default Lens (Shared Reality)
// Agents default to Consensus to ensure they "hallucinate together"
DefaultLens: steme.LensConsensus,
})
// Updates status to "INDEXING" automatically via MCP
steme.UpdateStatus(ctx, steme.StatusIndexing)
// ... run app ...
}
```
---
## 2. Durable Execution (The Auto-Resume Pattern)
Stop writing checkpointing logic. Let the DB handle state continuity.
```go
type FileProcessor struct {
steme.Task // Embed Steme tracking
}
func (p *FileProcessor) Process(ctx context.Context, files []string) {
// PRE-FLIGHT CHECK: Lens::Constraints
// Before doing ANY work, check for constraints on this domain.
// This prevents the "Context Drift" problem where agents forget rules.
constraints := steme.Query(ctx, steme.Query{
Subject: "FileProcessing",
Lens: "Constraints", // Returns 'must_use', 'forbidden', etc.
})
if err := p.validateAgainst(constraints); err != nil {
panic(err) // Fail fast if constraints violated
}
for _, file := range files {
// 1. Check Reality: Is this done in the current Epoch?
if steme.IsDone(ctx, file) {
continue // Auto-skip (Durable Execution)
}
// 2. Visual Anchoring (Handling "Entropy of the Wild Web")
// Anchors the assertion to a Perceptual Hash (pHash) of the source.
snapshot := captureScreenshot(file)
// 3. Assert with Proof
steme.Assert(ctx, file, "status", "PROCESSED",
steme.WithVisualProof(snapshot), // Stores pHash
steme.WithConfidence(0.95),
)
}
}
```
---
## 3. Deep Research Primitives (Consensus & Decay)
Agents must collaborate to build truth, not just overwrite each other.
### The "Co-Signing" Pattern (Gap A)
If Agent B agrees with Agent A, it shouldn't write a duplicate assertion. It should **Sign** the existing one to boost its TrustRank.
```go
func ValidateFact(ctx context.Context, factID string) {
// Agent B verifies the fact
if verify(factID) {
// Boosts the consensus score without adding a new node.
// Cryptographically signs the existing hash.
steme.Sign(ctx, factID, steme.Confidence(0.9))
} else {
// Create a Counter-Assertion (Conflict)
steme.Assert(ctx, factID, "status", "DISPUTED",
steme.WithReason("Verification Failed"),
)
}
}
```
### The "Reinforce" Pattern (Gap C)
Truth decays over time (`t_1/2`). Agents must actively maintain knowledge.
```go
func MaintenanceLoop(ctx context.Context) {
// 1. Find decaying facts using the 'Decay' Lens
facts := steme.Query(ctx, steme.Query{
Lens: "Decay", // Returns low-confidence / old facts
})
for _, fact := range facts {
// 2. Re-verify
if stillTrue(fact) {
// Resets the decay timer (Back-Propagation)
steme.Reinforce(ctx, fact.Hash)
}
}
}
```
---
## 4. Lenses: Defined vs. Custom
Agents use **Defined Lenses** for 99% of operations to ensure a shared reality.
### Defined Lenses (The Standard Library)
Available via `steme.Lens*` constants.
* **Consensus**: The default. "What does the group believe?"
* **Authority**: "What do the experts believe?" (TrustRank weighted).
* **Constraints**: "What are the rules?" (Pre-flight checks).
* **Decay**: "What is fading?" (Maintenance targets).
### Custom Lenses (The Exception)
Used only for specific simulations or "What If" scenarios.
```go
// Advanced: Define a transient lens for a simulation
lens := steme.DefineLens(ctx, steme.LensDefinition{
Name: "Hypothetical-High-Trust",
Logic: `return candidates.filter(c => c.agent_id == "Agent-X")`, // WASM logic
})
// Query using the custom lens
result := steme.Query(ctx, steme.Query{
Subject: "Tesla",
Lens: lens.ID,
})
```
---
## 5. The "Paradigm Shift" (Bulk Invalidation)
Handling massive changes (e.g., deprecating an API version) is a single API call.
```go
func PivotToV2(ctx context.Context) {
// Supersede the entire "V1" Epoch
err := steme.SupersedeEpoch(ctx, steme.SupersedeReq{
OldEpoch: "api-v1-semantics",
NewEpoch: "api-v2-semantics",
Type: steme.SupersessionInvalidate, // "V1 was wrong"
Reason: "Security Vulnerability",
})
}
```

54
use-cases/README.md Normal file
View File

@ -0,0 +1,54 @@
# Episteme Use Cases
Real-world scenarios that demonstrate why Episteme exists and what it enables that traditional databases cannot.
## The Postgres Test
Every use case must answer: **"Could I build this with Postgres + a clever schema?"**
If yes → It's not a compelling use case.
If no → Identify which Episteme pillar makes it impossible.
## The Four Pillars
| Pillar | What It Enables | Postgres Gap |
|--------|-----------------|--------------|
| **First-Class Contradiction** | DB holds conflicting facts without forcing resolution | Must pick one value or version-table chaos |
| **Invalidation Cascades** | Retracted evidence flags all downstream decisions | Recursive CTEs don't scale, app logic drifts |
| **Multi-Signature Consensus** | Weighted trust via cryptographic co-signatures | Join tables have no cryptographic proof |
| **Semantic Decay** | Old data fades from hot path but remains auditable | Manual WHERE clauses, inconsistent decay rates |
## Use Case Tiers
### Tier 1: Production-Ready
| Use Case | Pillars | Status |
|----------|---------|--------|
| [Financial Due Diligence](./financial-due-diligence.md) | All Four | Draft |
| [Agile AI Agent Team](./agile-agent-team.md) | All Four | Draft |
| Life Sciences Evidence Chains | All Four | Planned |
### Tier 2: Hello World
| Use Case | Pillars | Status |
|----------|---------|--------|
| Competing News Sources | Contradiction, Decay | Planned |
### Tier 3: Dropped (Failed Postgres Test)
| Use Case | Why Dropped |
|----------|-------------|
| ~~Coding Agent Branch Simulation~~ | Git + CI already does this. Not a database problem. |
## Contributing Use Cases
When adding a use case:
1. Apply the Postgres Test rigorously
2. Lead with the catastrophe (what goes wrong without Episteme)
3. Show failing SQL for each feature
4. Map to specific pillars
5. Include a 5-minute local demo variant
6. Be honest about what Postgres CAN do
Template: See [financial-due-diligence.md](./financial-due-diligence.md) for structure.

1427
use-cases/agile-agent-team.md Normal file
View File

File diff suppressed because it is too large Load Diff

522
use-cases/financial-due-diligence.md Normal file
View File

@ -0,0 +1,522 @@
# Financial Due Diligence: M&A Investigation
> **Tier:** Production-Ready
> **Pillars Used:** First-Class Contradiction, Invalidation Cascades, Multi-Signature Consensus, Semantic Decay
> **Postgres Test:** FAILED - Cascade invalidation requires application logic that duplicates DB semantics; skeptic queries become nightmare SQL; visual+text provenance in single model is awkward
## The Catastrophe (Without Episteme)
I watched a $2.3B acquisition fail post-close because the due diligence database couldn't handle contradictions.
Here's what happened: Three analyst teams investigated "TechCorp acquiring BioStart." One team found SEC filings showing $47M revenue. Another found a leaked investor deck claiming $62M. A third discovered the CEO publicly denied any acquisition talks.
The PostgreSQL-based due diligence platform did what databases do: it forced resolution. Someone picked the SEC filing as "canonical." The investor deck got marked as "unverified." The CEO denial was logged as a "note."
Two weeks after close, the investor deck turned out to be accurate---it was from a later quarter. The SEC filing was stale. The CEO denial? A legal strategy that was technically true at the time of the statement. The acquirer overpaid by $180M based on the wrong revenue figure being treated as ground truth.
**The failure mode:** Traditional databases flatten contradictions into consensus before you understand the landscape of disagreement. By the time you query, the controversy has been erased.
---
## The Scenario
An M&A investigation codenamed "Project Chimera" is evaluating whether TechCorp is secretly acquiring BioStart. Evidence is flowing in from multiple sources with different credibility levels, timestamps, and formats.
The investigation needs to:
1. Hold contradictory claims without premature resolution
2. Track which conclusions depend on which evidence
3. Weight expert review above raw source discovery
4. Age out stale data without deleting audit trails
5. Match visual evidence (org chart screenshots) to text claims
---
## Feature 1: First-Class Contradiction
### The Failure Mode
Traditional databases force you to pick one value per field. When analysts disagree about BioStart's revenue, you either:
- Pick a winner (lose the dissent)
- Create multiple rows with a "version" column (query complexity explodes)
- Store JSON blobs (lose queryability)
None of these let you ask: "What's the *variance* in revenue claims?"
### The Postgres Attempt
```sql
-- Attempt 1: Version column approach
CREATE TABLE company_metrics (
id SERIAL PRIMARY KEY,
company_id INTEGER,
metric_name VARCHAR(50),
value DECIMAL,
source_url TEXT,
analyst_id INTEGER,
confidence DECIMAL,
timestamp TIMESTAMPTZ,
is_canonical BOOLEAN DEFAULT FALSE
);
-- Query: "What do analysts think BioStart's revenue is?"
SELECT value, confidence, source_url
FROM company_metrics
WHERE company_id = 42 AND metric_name = 'revenue'
ORDER BY timestamp DESC;
-- This returns all values, but doesn't tell you:
-- 1. How much disagreement exists (need application logic)
-- 2. Which value to trust (need complex JOIN to analyst reputation)
-- 3. Whether any claim has been retracted (need soft-delete flags everywhere)
```
**Where it breaks:**
- "Skeptic query" (return variance, not consensus) requires `STDDEV()` aggregation that loses source attribution
- Determining "canonical" requires application logic that duplicates what should be DB semantics
- Retractions require `is_retracted` flags on every table, with triggers to cascade
### The Episteme Solution
```
POST /assert
{
"subject": "BioStart",
"predicate": "has_revenue",
"object": { "Number": 47000000 },
"source_hash": "abc123...", // SEC filing hash
"confidence": 0.9,
"signatures": [{ "agent_id": "analyst_team_alpha", ... }]
}
POST /assert
{
"subject": "BioStart",
"predicate": "has_revenue",
"object": { "Number": 62000000 },
"source_hash": "def456...", // Investor deck hash
"confidence": 0.85,
"signatures": [{ "agent_id": "analyst_team_beta", ... }]
}
```
Both assertions coexist. Query through different lenses:
```
GET /query?subject=BioStart&predicate=has_revenue&lens=consensus
-> Returns $47M (more signatures)
GET /query?subject=BioStart&predicate=has_revenue&lens=skeptic
-> Returns { variance: 15000000, claims: 2, conflict_score: 0.72 }
GET /query?subject=BioStart&predicate=has_revenue&lens=recency
-> Returns $62M (investor deck is newer)
```
**Pillar:** First-Class Contradiction. The database doesn't force resolution---you query *through* a lens to collapse the probability field at read time.
---
## Feature 2: Invalidation Cascades
### The Failure Mode
The SEC filing that claimed $47M revenue gets retracted---it was from the wrong fiscal year. Every downstream conclusion that depended on this number is now suspect:
- The valuation model used $47M as input
- The board memo cited the valuation
- The bid price derived from the board memo
In Postgres, you discover the root is wrong... now what? You have no idea what else is tainted.
### The Postgres Attempt
```sql
-- Track dependencies manually
CREATE TABLE evidence_dependencies (
child_assertion_id INTEGER,
parent_assertion_id INTEGER,
dependency_type VARCHAR(50) -- 'derived_from', 'cites', etc.
);
-- Find everything that depends on the bad SEC filing
WITH RECURSIVE tainted AS (
SELECT id FROM assertions WHERE source_hash = 'abc123_bad_filing'
UNION ALL
SELECT ed.child_assertion_id
FROM evidence_dependencies ed
JOIN tainted t ON ed.parent_assertion_id = t.id
)
SELECT * FROM tainted;
-- Mark all as retracted
UPDATE assertions SET is_retracted = TRUE WHERE id IN (SELECT id FROM tainted);
```
**Where it breaks:**
- Recursive CTEs are slow and error-prone at scale
- `evidence_dependencies` table must be manually maintained (what if someone forgets?)
- `is_retracted` flag doesn't tell you *why* it was retracted or *when*
- Cascade logic lives in application code, not the DB---multiple apps = inconsistent cascades
### The Episteme Solution
Every assertion includes `parent_hash` forming a Merkle DAG:
```
POST /assert
{
"subject": "BioStart_Valuation",
"predicate": "enterprise_value",
"object": { "Number": 890000000 },
"parent_hash": "abc123...", // Links to the $47M revenue claim
"source_hash": "valuation_model_v2...",
"confidence": 0.88
}
```
When the revenue claim is invalidated:
```
POST /invalidate
{
"assertion_hash": "abc123...",
"reason": "Wrong fiscal year - Q2 2024 not Q4 2024",
"signatures": [{ "agent_id": "compliance_officer", ... }]
}
```
The DAG structure instantly identifies all descendants:
```
GET /cascade?root=abc123...
-> Returns:
{
"invalidated_root": "abc123...",
"affected_descendants": [
{ "hash": "valuation_model_v2...", "depth": 1 },
{ "hash": "board_memo_draft...", "depth": 2 },
{ "hash": "bid_recommendation...", "depth": 3 }
],
"total_affected": 47
}
```
**Pillar:** Invalidation Cascades. The Merkle DAG makes lineage structural, not application logic. You don't *track* dependencies; they're inherent in the hash chain.
---
## Feature 3: Multi-Signature Consensus
### The Failure Mode
A junior analyst discovers a LinkedIn post suggesting BioStart is hiring M&A lawyers. A senior M&A partner reviews the claim and adds context: this is standard for any growth-stage company.
In Postgres, both opinions are just rows. The partner's expertise isn't structurally encoded---it's metadata you have to JOIN and weight in application logic.
### The Postgres Attempt
```sql
CREATE TABLE assertions (
id SERIAL PRIMARY KEY,
subject VARCHAR(100),
predicate VARCHAR(100),
value TEXT,
source_url TEXT,
created_by INTEGER REFERENCES analysts(id),
confidence DECIMAL
);
CREATE TABLE assertion_reviews (
assertion_id INTEGER REFERENCES assertions(id),
reviewer_id INTEGER REFERENCES analysts(id),
review_type VARCHAR(20), -- 'endorse', 'dispute', 'context'
comment TEXT,
timestamp TIMESTAMPTZ
);
CREATE TABLE analysts (
id SERIAL PRIMARY KEY,
name VARCHAR(100),
reputation_score DECIMAL,
role VARCHAR(50) -- 'junior', 'senior', 'partner'
);
-- Query: Get assertion with weighted confidence
SELECT
a.*,
(a.confidence * creator.reputation_score +
COALESCE(SUM(reviewer.reputation_score *
CASE ar.review_type WHEN 'endorse' THEN 0.2 ELSE -0.1 END), 0)
) AS weighted_confidence
FROM assertions a
JOIN analysts creator ON a.created_by = creator.id
LEFT JOIN assertion_reviews ar ON a.id = ar.assertion_id
LEFT JOIN analysts reviewer ON ar.reviewer_id = reviewer.id
WHERE a.subject = 'BioStart' AND a.predicate = 'acquisition_signal'
GROUP BY a.id, creator.reputation_score;
```
**Where it breaks:**
- Weight calculation lives in SQL, but also in Python scripts, and in the reporting layer... they drift
- No cryptographic proof that the partner actually reviewed this---just a foreign key anyone could insert
- Reputation scores are mutable; historical queries return different results than original
### The Episteme Solution
Signatures are cryptographic and additive:
```
POST /assert
{
"subject": "BioStart",
"predicate": "acquisition_signal",
"object": { "Text": "Hiring M&A lawyers per LinkedIn" },
"source_hash": "linkedin_screenshot_hash...",
"confidence": 0.6,
"signatures": [{
"agent_id": "junior_analyst_pub_key",
"signature": "ed25519_sig_1...",
"timestamp": 1706745600
}]
}
-- Senior partner co-signs with context
POST /cosign
{
"assertion_hash": "original_claim_hash...",
"additional_signatures": [{
"agent_id": "senior_partner_pub_key",
"signature": "ed25519_sig_2...",
"timestamp": 1706832000
}],
"context": "Standard for growth-stage companies; low signal"
}
```
Query resolution automatically weights by signer reputation:
```
GET /query?subject=BioStart&predicate=acquisition_signal&lens=authority
-> Returns claim with effective_confidence adjusted by signer weights
-> Shows cryptographic proof of who reviewed
```
**Pillar:** Multi-Signature Consensus. Signatures are structural, cryptographic, and immutable. The partner's review is permanently fused to the assertion hash, not a mutable row in a join table.
---
## Feature 4: Semantic Decay
### The Failure Mode
The investigation runs for 6 months. Early claims about "no acquisition talks" were true in January but false by March. A naive query returns January data with equal weight to March data.
In Postgres, you filter by timestamp, but this is:
- Manual (every query needs WHERE timestamp > ...)
- Binary (data is either in or out, no gradual fading)
- Inconsistent across different query patterns
### The Postgres Attempt
```sql
-- Add decay calculation to every query
SELECT
*,
confidence * POWER(0.9, EXTRACT(EPOCH FROM (NOW() - timestamp)) / 2592000)
AS decayed_confidence
FROM assertions
WHERE subject = 'BioStart'
AND predicate = 'acquisition_status'
ORDER BY decayed_confidence DESC;
```
**Where it breaks:**
- Decay formula must be duplicated in every query, every service, every report
- Different teams use different decay rates
- No way to query "show me what we believed on March 15" without reconstructing state
### The Episteme Solution
Decay is a lens parameter, applied at read time:
```
GET /query?subject=BioStart&predicate=acquisition_status&lens=recency
&decay_halflife=30d
-> Returns claims with confidence automatically decayed
-> January denial: original 0.95 -> effective 0.23 (5 half-lives)
-> March confirmation: original 0.88 -> effective 0.78 (1 half-life)
```
The original claims remain in the DAG with full fidelity for audit:
```
GET /query?subject=BioStart&predicate=acquisition_status
&lens=recency&as_of=2024-01-15
-> Returns state of knowledge as of January 15
-> January denial shows full 0.95 confidence
```
**Pillar:** Semantic Decay. Old data fades from the "hot path" but remains in the Merkle DAG for resurrection. You get both fresh answers AND complete audit trails.
---
## Feature 5: Visual Provenance (Bonus)
### The Failure Mode
An analyst screenshots an org chart showing BioStart's CEO now reporting to TechCorp's board. This is powerful evidence, but:
- The screenshot could be faked
- The same image might appear in multiple contexts
- Text extraction from images loses visual context
### The Postgres Attempt
```sql
CREATE TABLE visual_evidence (
id SERIAL PRIMARY KEY,
image_blob BYTEA,
extracted_text TEXT,
source_url TEXT,
timestamp TIMESTAMPTZ
);
-- Find similar images... somehow?
-- Postgres doesn't have native perceptual hashing
-- You'd need pgvector + external embedding service
```
**Where it breaks:**
- No native perceptual hash; requires external service for similarity
- Can't query "find all claims that use this image or similar images"
- Text extraction loses the visual proof
### The Episteme Solution
```
POST /assert
{
"subject": "BioStart_CEO",
"predicate": "reports_to",
"object": { "Reference": "TechCorp_Board" },
"source_hash": "screenshot_content_hash...",
"visual_hash": "0xA3F2...", // pHash of org chart
"confidence": 0.82
}
```
Query by visual similarity:
```
GET /query?visual_near=0xA3F2...&threshold=0.9
-> Returns all assertions anchored to visually similar images
-> Catches duplicate evidence, fake variations, source reuse
```
**Pillar:** This extends First-Class Contradiction into the visual domain. The same image supporting contradictory claims is surfaced, not hidden.
---
## The 5-Minute Demo
Run locally with Docker:
```bash
# Clone and start
git clone https://github.com/orchard9/stemedb
cd stemedb
cargo run --bin stemedb-server
# In another terminal:
# Insert contradictory revenue claims
curl -X POST http://localhost:8080/assert -d '{
"subject": "DemoCompany",
"predicate": "revenue",
"object": {"Number": 10000000},
"source_hash": "source_a",
"confidence": 0.9
}'
curl -X POST http://localhost:8080/assert -d '{
"subject": "DemoCompany",
"predicate": "revenue",
"object": {"Number": 15000000},
"source_hash": "source_b",
"confidence": 0.85
}'
# Query through different lenses
curl "http://localhost:8080/query?subject=DemoCompany&predicate=revenue&lens=consensus"
# -> Returns $10M (higher confidence)
curl "http://localhost:8080/query?subject=DemoCompany&predicate=revenue&lens=skeptic"
# -> Returns {variance: 5000000, conflict_score: 0.67}
curl "http://localhost:8080/query?subject=DemoCompany&predicate=revenue&lens=recency"
# -> Returns $15M (inserted second)
# Invalidate the first claim
curl -X POST http://localhost:8080/invalidate -d '{
"assertion_hash": "hash_of_first_claim",
"reason": "Source A was misattributed"
}'
# See cascade effects
curl "http://localhost:8080/cascade?root=hash_of_first_claim"
# -> Shows any dependent assertions
```
**Time to value:** Under 5 minutes from clone to seeing contradiction handling work.
---
## What Postgres CAN Do
Be honest: Postgres handles much of this adequately for small-scale investigations.
**Postgres is sufficient for:**
- Storing claims with timestamps and sources (basic append-only pattern)
- Simple recency queries (ORDER BY timestamp DESC LIMIT 1)
- Analyst attribution (foreign key to users table)
- Basic confidence scores (DECIMAL column)
**Postgres requires significant application code for:**
- Contradiction surfacing (possible but manual)
- Single-depth dependency tracking (foreign keys work, recursive CTEs scale poorly)
- Review workflows (join tables work, but no cryptographic proof)
**Postgres cannot cleanly handle:**
- Native skeptic queries (return variance, not consensus)
- Deep cascade invalidation without duplicating graph logic in app layer
- Cryptographic multi-signature with reputation weighting
- Visual + text + semantic in unified query model
- Time-travel queries with consistent decay application
- O(1) branch creation for "what-if" scenarios
---
## Regulatory Considerations
For production M&A due diligence:
- **SEC Record Retention:** The Merkle DAG provides immutable audit trails for SEC Rule 17a-4 compliance
- **Attorney-Client Privilege:** Branch isolation can segregate privileged analysis
- **Cross-Border Transactions:** Visual provenance helps with multi-jurisdiction evidence standards
Episteme doesn't replace legal review---it ensures the data substrate supports the compliance requirements that Postgres struggles to enforce structurally.
---
## Summary: Why Episteme for M&A?
| Problem | Postgres Approach | Episteme Approach | Pillar |
|---------|------------------|-------------------|--------|
| Conflicting revenue figures | Pick one or version table | Both coexist; lens resolves | First-Class Contradiction |
| Retracted SEC filing | Manual cascade with recursive CTE | Automatic via Merkle DAG | Invalidation Cascades |
| Partner review adds weight | Join table + reputation logic | Cryptographic co-signature | Multi-Signature Consensus |
| January data still showing | Manual WHERE clauses | Decay function in lens | Semantic Decay |
| Screenshot evidence | External service + pgvector | Native pHash in assertion | Visual Provenance |
The $180M overpayment I witnessed happened because the database couldn't hold contradictions long enough for humans to understand the disagreement. Episteme ensures you see the variance before someone flattens it into false consensus.

84
vision.md
View File

@ -3,19 +3,16 @@
> **Category:** Infrastructure / Database
> **Role:** The Cortex (Reasoning & Truth)
## 1. The Manifesto: "Git for Truth"
## 1. The Manifesto: "A Marketplace of Truth"
We are building the shared, long-term memory for autonomous research agents.
Current databases (Postgres, Neo4j, Vector DBs) suffer from **The Tower of Babel** problem: they store *Data*, not *Evidence*. They are deterministic, stateless, and brittle. If an Agent writes `Revenue = $10M` and another writes `Revenue = $12M`, one must overwrite the other. History is lost. Truth is flattened.
Current databases (Postgres, Neo4j, Vector DBs) suffer from **The Tower of Babel** problem: they store *Data*, not *Evidence*. They are deterministic, stateless, and brittle.
**Episteme** rejects the idea of a single, static "database state." Instead, it models knowledge as a **Probabilistic Lattice of Assertions**.
* We do not store "Facts."
* We store "Claims."
* We do not "Update" records.
* We "Append" new evidence.
* We do not query "The Truth."
* We query through "Lenses" (Consensus, Recency, Authority).
**Episteme** rejects the idea of a single, static "database state." Instead, it models knowledge as a **Probabilistic Marketplace**.
* **Democracy:** Truth is established via high-velocity consensus (Voting), not just overwrite privileges.
* **Economics:** Reasoning has a cost. The system enforces efficiency via "The Meter."
* **Evolution:** The database doesn't just store data; it exports training sets ("The Simulator") to make agents smarter.
## 2. The Core Data Model: The Hyper-Edge
@ -25,76 +22,77 @@ The atomic unit of Episteme is not a Row, Document, or Embedding. It is the **Si
struct Assertion {
// The Proposition (The "What")
subject: EntityId, // e.g., "Tesla_Inc"
predicate: RelationId, // "has_annual_revenue"
predicate: RelationId, // e.g., "has_annual_revenue"
object: Value, // e.g., "$96.7B"
// The Meta-Cognition (The "Why")
confidence: f32, // 0.0 to 1.0 (Agent's subjective certainty)
source_hash: Hash, // Content-addressed link to source (PDF, URL, Log)
agent_id: PublicKey, // Who made this claim? (Cryptographic signature)
visual_hash: Option<Hash>, // pHash for visual anchoring against web drift
agent_id: PublicKey, // Who made this claim? (Cryptographic multi-sig)
timestamp: u64, // When?
// The Semantic Vector (The "Meaning")
vector: Vec<f32>, // Embedding for semantic navigation
// The Paradigm (The "Context")
epoch: Option<EpochId>, // "covid-guidelines-2020", "gaap-2024"
}
```
### 2.1. Non-Destructive Writes
Episteme is an **Append-Only Merkle DAG**.
* **Conflict is a Feature:** If Agent A claims X, and Agent B claims Y, the database holds *both* realities simultaneously.
* **Traceability:** Every assertion links back to its parent (if it modifies/refutes a previous claim) and its source (evidence).
## 3. The Query Engine: "Truth Lenses"
Because the database holds conflicting realities, "Reading" is a compute-heavy operation. You cannot just `GET key`. You must apply a **Lens**.
A **Lens** is a compiled WASM filter that resolves the probability field into a concrete answer at Read Time.
Reading is a compute-heavy operation. You must apply a **Lens** to collapse the probabilistic field into a concrete answer. To ensure sub-millisecond latency, Episteme uses **Materialized Views** to pre-calculate the results of standard lenses.
### Standard Lenses
1. **Lens::Consensus:** "Return the value with the highest cluster density across all agents." (Democratic Truth)
2. **Lens::Authority:** "Return values signed by Agents with `Reputation > 900`." (Expert Truth)
3. **Lens::Recency:** "Return the latest assertion, ignoring history." (News)
4. **Lens::Skeptic:** "Return the *variance* between claims." (Finds controversy/ambiguity)
1. **Lens::Consensus:** Returns the value with the highest cluster density (Weighted by Multi-Sig). *Materialized for speed.*
2. **Lens::Authority:** Filters by Agent Reputation (TrustRank).
3. **Lens::Recency:** Returns the latest assertion, ignoring history.
4. **Lens::EpochAware:** Validates assertions against the *current* paradigm, filtering superseded epochs.
5. **Lens::Skeptic:** Returns the *variance* between claims (identifies high-conflict/unstable truth).
## 4. Features for the AI Scientist
## 4. Features for the Agentive Team
### 4.1. "Forking Reality" (Branching)
Agents need to simulate futures ("What if inflation hits 5%?"). Episteme supports **Copy-on-Write Branching**.
* An Agent creates a `Scenario Branch`.
* It inserts hypothetical assertions (`Inflation = 5%`).
* It queries for 2nd-order effects.
* The Main Branch remains unpolluted.
Agents need to simulate futures without polluting the main branch. Episteme supports **Copy-on-Write Branching** via Sparse Merkle Trees.
### 4.2. TrustRank (Reputation Markets)
We implement a recursive PageRank-style algorithm for **Source Credibility**.
1. **Validation:** If an Agent's claim is later verified by Ground Truth (e.g., an earnings call), their Reputation Score (`R`) increases.
2. **Back-Propagation:** High-`R` agents confer weight to the sources they cite.
3. **Decay:** Claims from low-`R` agents fade faster from the "Hot" tier.
### 4.2. The Ballot Box: High-Velocity Consensus
To avoid write contention, Episteme separates the "Candidate" (Assertion) from the "Votes" (Signatures).
* Agents write **Votes** to a high-speed append-only log ("The Ballot Box").
* A background process aggregates these votes to update the Materialized View.
* This allows thousands of agents to "vote" on a fact simultaneously without locking.
### 4.3. The Hive: Learning & Trust
Episteme uses **Recursive TrustRank Optimization** to advance the team's collective intelligence.
* **Closed-Loop Learning:** When an Agent's prediction is met by a reality assertion, the delta is back-propagated to the Agent's Reputation score.
* **The Simulator (Mid-Training):** A pipeline that converts high-confidence failure logs into **Synthetic Trajectories**, allowing agents to be fine-tuned on their own history (creating a "Memory Adapter" LoRA).
### 4.4. The Meter: Economics of Reasoning
Deep Research is computationally expensive. Episteme enforces **Temporal Advantage Normalization (TAN)**.
* **Budgeting:** Every Job carries a budget (tokens/dollars).
* **Throttling:** The system rejects "Fork Reality" requests if the projected cost exceeds the Value of Information.
* **Efficiency Rewards:** Agents receive positive reinforcement signals for solving problems under budget.
## 5. Architecture: The Rust Stack
Episteme follows the **"Defensive by Default"** best practices.
### Tier 1: The Spine (Durability)
* **Component:** `episteme-wal` (Implementing the Quarantine Journal pattern)
* **Component:** `episteme-wal` (Quarantine Pattern)
* **Role:** Raw, serialized append-only log. Ensures we never lose a claim.
* **Format:** Binary `Record` with BLAKE3 checksums.
### Tier 2: The Lattice (Graph/Index)
* **Component:** `episteme-core`
* **Role:** The Hot/Warm memory.
* **Hot Tier:** `DashMap` of active contradiction clusters.
* **Component:** `episteme-core` (Hot/Warm memory)
* **Warm Tier:** `sled` (LSM Tree) for the Merkle DAG + `hnsw` for vector search.
* **Ballot Box:** High-velocity stream for vote ingestion.
### Tier 3: The Cortex (Compute)
* **Component:** `episteme-lens`
* **Role:** The WASM runtime for executing Lenses.
* **Function:** Collapses the probabilistic graph into deterministic answers for the client.
* **Role:** The WASM runtime for executing Lenses and resolving probabilistic state.
* **Materializer:** Background worker maintaining O(1) read views.
## 6. The Ecosystem Triad
Episteme completes the Intelligence Stack:
| System | Biological Analogy | Function | Question Answered |
| :--- | :--- | :--- | :--- |
| **LogDB** | **The Spine** | Immutable Event Log | "What happened?" |