stemedb/applications/aphoria/roadmap.md

Aphoria Roadmap

---

Phase 0: StemeDB Foundation ✅

Tracked in: roadmap.md § 5D. Concept Hierarchy

Changes to the core database that Aphoria depends on. Shipped as Phase 5D of the main StemeDB roadmap.

Aphoria Phase 0	StemeDB Phase 5D	Status
0.1 ConceptPath Type	5D.1 ConceptPath Type	✅
0.2 ConceptPath in Assertion	(implicit in 5D.1)	✅
0.3 Hierarchical Index	5D.4 Hierarchical Query	✅
0.4 Alias Store	5D.3 Alias Store + 5D.5 Alias Resolution	✅
0.5 Source Class Inference	5D.6 Source Class Inference	✅
0.6 Concept API Endpoints	5D.7 Concept API Endpoints	✅

Spec: docs/specs/concept-hierarchy.md

---

Phase 2: CLI Core ✅

Phase 2 was built before Phase 1 (authoritative corpus expansion). The CLI pipeline works end-to-end with a bootstrapped corpus of 11 hardcoded assertions covering TLS, JWT, CORS, secrets, and rate limiting.

Task	Status
2.1 Project Walker	✅ walker/mod.rs, walker/path_mapper.rs, walker/language.rs
2.2 Extractors (10)	✅ tls_verify, jwt_config, hardcoded_secrets, timeout_config, dep_versions, cors_config, rate_limit, weak_crypto, command_injection, sql_injection
2.3 Ingestion Bridge	✅ bridge.rs — BLAKE3 hashing, Ed25519 signing, claim→assertion conversion
2.4 Conflict Query	✅ episteme.rs — LocalEpisteme with check_conflicts()
2.5 Report Output	✅ report/ — table (comfy-table), JSON, SARIF 2.1.0, markdown
2.6 Acknowledge Command	✅ lib.rs acknowledge()
Baseline & Diff	✅ lib.rs set_baseline(), show_diff()
Status Command	✅ lib.rs show_status()

183 tests pass. Clippy and fmt clean.

Phase 2 Code Quality Fixes ✅

Code review improvements to extractors:

Issue	Fix	Status
DES/RC4 concept path misclassification	Split check_pattern() into check_hash_pattern() and check_encryption_pattern(); DES/RC4 now use crypto/encryption/algorithm path	✅
SHA1 edge case undocumented	Added comments and test documenting that SHA1 detection is intentionally broad (triggers for git hashes, etc.)	✅
JS exec() regex overly broad	Tightened regex to require child_process. prefix or non-word/non-dot preceding character; prevents RegExp.exec() false positives	✅

---

Phase 2A: Concept Matching ✅

Status: Complete. Tail-path matching (2A.1), alias-aware queries (2A.2), and auto-alias creation (2A.3) all implemented.

2A.1 Leaf-Based Concept Matching (Aphoria-side fix) ✅

Implemented in episteme.rs via ConceptIndex:

• make_key(subject, predicate) extracts tail 2 path segments + predicate
• build(assertions) creates in-memory index keyed by tail path
• lookup(subject, predicate) finds matching authoritative assertions
• check_conflicts() uses ConceptIndex instead of QueryEngine for cross-scheme matching

Integration tests prove TLS and JWT conflicts are detected correctly.

2A.2 Alias Resolution in QueryEngine (StemeDB-side fix) ✅

Wired AliasStore into QueryEngine.execute():

• Added resolve_aliases: bool field to Query (defaults to false)
• Added alias_store: Option<Arc<dyn AliasStore>> to QueryEngine
• Added .with_alias_store() builder method
• When resolve_aliases: true, expands subject via AliasStore.resolve_all() before index lookup
• Added fetch_by_subjects() and fetch_by_subjects_predicate() for multi-subject deduplication
• Modified Query.matches() to skip subject filtering when aliases are resolved
• Skips fast path (MV lookup) when resolve_aliases: true
• Gracefully degrades when no alias store is configured

7 unit tests in engine/tests/alias_resolution.rs. This is the architecturally correct long-term fix that complements leaf matching.

2A.3 Auto-Alias Creation ✅

When Aphoria ingests authoritative assertions and code claims that share leaf names, automatically create aliases:

• code://rust/myapp/tls/cert_verification ↔ rfc://5246/tls/cert_verification
• code://rust/myapp/auth/jwt/audience_validation ↔ rfc://7519/jwt/audience_validation

This bridges 2A.1 (leaf matching) with 2A.2 (alias resolution) — leaf matching identifies candidates, aliases persist the relationship.

Implementation:

• Added auto_create_aliases: bool config option to AliasConfig (defaults to true)
• Added AliasOrigin::AutoDetected variant to stemedb-core for tracking auto-created aliases
• Wired GenericAliasStore into LocalEpisteme for alias persistence
• In check_conflicts(), when a code claim matches an authoritative claim by leaf, calls AliasStore.set_alias() to persist the relationship with AliasOrigin::AutoDetected
• Alias creation is idempotent (skips if alias already exists)
• 4 unit tests verify: alias creation on conflict, no creation when disabled, correct origin, idempotency

---

Phase 1: Authoritative Corpus Expansion ✅

Expanded from 11 hardcoded assertions to a pluggable corpus system with RFC, OWASP, and Vendor sources.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                     aphoria corpus build                         │
│                                                                  │
│  ┌──────────────┐  ┌──────────────┐  ┌───────────────────────┐  │
│  │ RFC Ingester │  │ OWASP        │  │ Vendor Bootstrapper   │  │
│  │ (Tier 0)     │  │ Ingester     │  │ (Tier 2)              │  │
│  │              │  │ (Tier 1)     │  │                       │  │
│  └──────┬───────┘  └──────┬───────┘  └───────────┬───────────┘  │
│         │                 │                      │              │
│         └─────────────────┼──────────────────────┘              │
│                           ▼                                     │
│                  ┌─────────────────┐                            │
│                  │ CorpusRegistry  │                            │
│                  └────────┬────────┘                            │
│                           ▼                                     │
│                  ┌─────────────────┐                            │
│                  │ LocalEpisteme   │                            │
│                  │ ingest_         │                            │
│                  │ authoritative() │                            │
│                  └─────────────────┘                            │
└─────────────────────────────────────────────────────────────────┘

1.1 CorpusBuilder Trait ✅

Task	Status
CorpusBuilder trait	✅ corpus/mod.rs — name, scheme, default_tier, build, requires_network
CorpusRegistry	✅ Manages multiple builders, build_all(), list_builders()
CorpusBuildResult	✅ Stats per builder, total assertions, success/fail/skip counts

1.2 RFC Ingester ✅

Task	Status
RfcCorpusBuilder	✅ corpus/rfc.rs
HTTP fetching	✅ Via ureq, cached to ~/.cache/aphoria/rfc-cache/
RFC 2119 keyword parsing	✅ MUST, MUST NOT, SHOULD, SHALL extraction
RFC-specific parsers	✅ JWT (7519), OAuth (6749), Bearer (6750), TLS 1.3 (8446), TLS BCP (7525), TOTP (6238), Basic Auth (7617), HTTP (9110)
Concept mapping	✅ rfc://{number}/{topic} at Tier 0 (Regulatory)

1.3 OWASP Ingester ✅

Task	Status
OwaspCorpusBuilder	✅ corpus/owasp.rs
HTTP fetching	✅ From GitHub raw content, cached to ~/.cache/aphoria/owasp-cache/
Markdown parsing	✅ MUST/SHOULD statements, section context
Cheat sheet parsers	✅ Authentication, JWT, TLS, Secrets, Input Validation, Session, CSRF, Password Storage, HTTP Headers
Concept mapping	✅ owasp://cheatsheet/{topic}/{claim} at Tier 1 (Clinical)

1.4 Vendor Docs ✅

Task	Status
VendorCorpusBuilder	✅ corpus/vendor.rs
PostgreSQL claims	✅ pool_size, idle_timeout, ssl_mode
Redis claims	✅ timeout, max_retries, tls
reqwest claims	✅ cert_verification, connect_timeout, request_timeout
hyper claims	✅ keep_alive_timeout, max_concurrent_streams
Go net/http claims	✅ read_timeout, write_timeout, idle_timeout, min_tls_version
tokio-postgres claims	✅ pool_size, ssl_mode
SQLx claims	✅ max_connections, idle_timeout
Concept mapping	✅ vendor://{product}/{topic}/{claim} at Tier 2 (Observational)

1.5 Hardcoded Refactor ✅

Task	Status
HardcodedCorpusBuilder	✅ corpus/hardcoded.rs — original 11 assertions
create_authoritative_assertion()	✅ Made public in episteme.rs for corpus builders

1.6 CLI Integration ✅

Task	Status
aphoria corpus build	✅ Fetches and ingests from all sources
--only rfc,owasp,vendor	✅ Filter to specific sources
--offline	✅ Skip network-requiring sources
--clear-cache	✅ Clear cache before building
aphoria corpus list	✅ List available corpus sources
CorpusConfig	✅ cache_dir, include_*, rfc_list options

1.7 Error Handling ✅

Task	Status
RfcFetch error	✅ Per-RFC fetch failures with context
OwaspFetch error	✅ Per-cheat-sheet fetch failures with context
CorpusBuild error	✅ General corpus build failures
Graceful degradation	✅ Continue with other sources if one fails

Files: corpus/mod.rs, corpus/hardcoded.rs, corpus/rfc.rs, corpus/owasp.rs, corpus/vendor.rs

---

Phase 3: Skill Integration ✅

Complete. Aphoria is now usable in Claude Code agent workflows.

3.1 Claude Code Skill ✅

Task	Status
skill/SKILL.md	✅ Comprehensive skill definition with all commands
/aphoria scan	✅ Scan project, show conflicts grouped by verdict
/aphoria scan --fix	✅ Interactive fix workflow
/aphoria ack	✅ Acknowledge conflicts as intentional
/aphoria status	✅ Show status and baseline
/aphoria diff	✅ Show changes since baseline
/aphoria init	✅ Initialize Aphoria
/aphoria baseline	✅ Set baseline
skill/install.sh	✅ Install script for ~/.claude/skills/aphoria/

Files: skill/SKILL.md, skill/install.sh, skill/hooks.json

3.2 Agent Pre-Flight Hook ✅

Task	Status
--exit-code flag	✅ Returns 2 for BLOCK, 1 for FLAG only, 0 for clean
--strict flag	✅ Lower thresholds (FLAG at 0.3, BLOCK at 0.5)
Hook template	✅ skill/hooks.json with PreCommit and PrePush examples

Usage:

{
  "hooks": {
    "PreCommit": [{"command": "aphoria scan --format sarif --exit-code"}],
    "PrePush": [{"command": "aphoria scan --strict --exit-code"}]
  }
}

3.3 Alias Suggestion Workflow ✅

Auto-alias creation is now automatic (Phase 2A.3). When Aphoria scans:

1. Tail-path matching finds authoritative assertions
2. Aliases are auto-created with AliasOrigin::AutoDetected
3. Future queries use the alias automatically

The skill documents the suggestion flow for manual alias management:

• y (Accept): Creates alias
• n (Reject): Records intentional difference
• defer: Flags for later review

---

Phase 4: Pre-Commit Integration ⬜

Depends on Phase 3 (skill validates the UX before hook automation).

4.1 Git Pre-Commit Hook ⬜

A git pre-commit hook that runs Aphoria before every commit:

#!/bin/sh
# .git/hooks/pre-commit

aphoria scan --exit-code --format table

if [ $? -eq 2 ]; then
    echo "BLOCKED: Fix conflicts before committing"
    exit 1
fi

Or using pre-commit framework (.pre-commit-config.yaml):

repos:
  - repo: local
    hooks:
      - id: aphoria
        name: Aphoria Security Lint
        entry: aphoria scan --exit-code
        language: system
        pass_filenames: false

4.2 Baseline Mode ✅

Already implemented in Phase 2. For existing projects with many conflicts:

$ aphoria baseline
Baseline recorded: 12 existing conflicts frozen.
Future scans will only report new conflicts.

4.3 Diff-Only Scanning ⬜

Scan only changed files instead of the whole project:

# Scan only staged files
aphoria scan --staged

# Scan only files changed since baseline
aphoria scan --since-baseline

This makes pre-commit hooks fast even in large projects.

---

Phase 4.5: Ephemeral Scan Mode ✅

Performance optimization: 40x faster scans by skipping Episteme storage when persistence isn't needed.

Problem

Every aphoria scan was slow because it initialized the full Episteme stack:

• WAL recovery (O(n) on every startup)
• Dual backend initialization (fjall + redb)
• Store and index initialization

But conflict detection is actually 100% in-memory — it never reads from the KV store. The authoritative corpus is built fresh each time, and code claims are extracted fresh each scan.

Solution

Added ScanMode enum with two modes:

Mode	Use Case	Storage	Performance
Ephemeral (default)	CI, pre-commit, quick checks	None	~0.25 seconds
Persistent	Baseline/diff tracking, alias creation	WAL + store	~1-2 seconds

Implementation ✅

Task	Status
ScanMode enum	✅ types.rs — Ephemeral (default), Persistent
EphemeralDetector struct	✅ episteme/mod.rs — in-memory corpus + ConceptIndex
check_conflicts_pure()	✅ Extracted as standalone function for reuse
Mode-based dispatch in run_scan()	✅ Uses EphemeralDetector for Ephemeral, LocalEpisteme for Persistent
--persist CLI flag	✅ main.rs — opt-in to persistent mode
Tests for both modes	✅ test_ephemeral_scan_no_storage_created, test_persistent_scan_creates_storage, test_scan_modes_produce_same_conflicts

Usage

# Fast ephemeral scan (default) — no storage created
aphoria scan .

# Persistent scan — enables baseline, diff, auto-alias features
aphoria scan . --persist

Performance

Mode	Time	Storage
Ephemeral	~0.25s	None
Persistent	~1-2s	WAL + store directories

Files: types.rs, episteme/mod.rs, lib.rs, main.rs, tests.rs

---

Phase 5: Research Agent Loop ✅

Research agent fills gaps in authoritative coverage by researching official documentation.

5.1 Gap Detection ✅

Task	Status
Gap struct	✅ research/gap_detector.rs — concept_path, topic, predicate, source info
detect_gaps()	✅ Compares claims against ConceptIndex, identifies missing coverage
Topic normalization	✅ Extracts last 2 path segments for cross-scheme matching
Deduplication	✅ Deduplicates gaps by topic+predicate key

5.2 Gap Storage ✅

Task	Status
GapRecord	✅ research/gap_store.rs — tracking metadata, project count, research status
GapStore	✅ JSON-backed persistent storage with atomic saves
Project tracking	✅ Records which projects reported each gap
Research eligibility	✅ is_eligible_for_research() with threshold and cooldown
Gap pruning	✅ prune_old_gaps() removes stale entries

5.3 Quality Validation ✅

Task	Status
QualityValidator	✅ research/quality.rs — validates researched claims
Source attribution	✅ Checks for authoritative domains (rfc-editor, owasp, vendor docs)
Normative language	✅ Verifies MUST/SHOULD/SHALL keywords present
Vague content detection	✅ Rejects "it depends", "typically", etc.
Consistency scoring	✅ Detects conflicting claims on same subject
QualityReport	✅ Detailed per-claim validation results
filter_passed()	✅ Returns only claims meeting quality threshold

5.4 Research Execution ✅

Task	Status
Researcher	✅ research/researcher.rs — orchestrates research pipeline
DocumentationSource	✅ Configurable sources with URL patterns and topics
Default sources	✅ Redis, PostgreSQL, Go, Rust, OWASP, Kafka, MongoDB
Content fetching	✅ HTTP with timeout and size limits
Normative extraction	✅ Regex-based MUST/SHOULD/SHALL extraction
Section tracking	✅ Extracts heading context for attribution
Confidence scoring	✅ Based on keyword strength, statement length, content size

5.5 CLI Integration ✅

Task	Status
aphoria research run	✅ Run research agent with configurable threshold
aphoria research status	✅ Show gap statistics and research progress
aphoria research gaps	✅ List gaps by project count
--threshold	✅ Minimum projects before researching (default: 3)
--strict	✅ Use strict quality validation
--prune	✅ Remove stale gaps before researching
--ready	✅ Show only gaps ready for research

Files: research/mod.rs, research/gap_detector.rs, research/gap_store.rs, research/quality.rs, research/researcher.rs, research/tests.rs

5.7 Security Extractors ✅

Extended Phase 2 extractors with OWASP-aligned security vulnerability detection:

Extractor	Detects	Languages
weak_crypto	MD5, SHA1, DES, RC4 usage	Rust, Go, Python, JS/TS
command_injection	Shell execution, os.system, subprocess shell=True	Rust, Go, Python, JS/TS
sql_injection	String concatenation in SQL queries	Rust, Go, Python, JS/TS

Concept paths:

• crypto/hashing/algorithm — MD5, SHA1
• crypto/encryption/algorithm — DES, RC4
• os/command/input, os/shell_mode — command injection
• db/query/input — SQL injection

5.6 Community Corpus Contributions ⬜

Future: Users can opt in to contribute patterns anonymously.

• "Every Rust project has this JWT pattern" → pre-built alias set
• "This Redis config is always acknowledged" → adjust default threshold
• "This TLS pattern is always a real bug" → elevate threshold

---

Phase 6: Federated Policy & Trust Packs ✅

Allow teams to define their own authoritative truths and distribute them as signed Trust Packs. This enables "Enterprise Grade" compliance across distributed teams.

6.1 Trust Pack Format ✅

Task	Status
TrustPack schema	✅ policy.rs — Assertions, Aliases, Metadata, Signature
PackHeader	✅ Name, version, issuer, timestamp
Serialization	✅ rkyv for zero-copy efficiency
Signing	✅ ed25519-dalek signing and verification

6.2 Policy Management ✅

Task	Status
PolicyManager	✅ Loads local and remote (HTTP/HTTPS) policies
Caching	✅ Caches remote policies in ~/.cache/aphoria/policies/
aphoria.toml config	✅ policies list support

6.3 Core Integration ✅

Task	Status
EphemeralDetector integration	✅ Ingests policies into memory corpus/index
check_conflicts_pure update	✅ Resolves policy aliases before authoritative lookup
LocalEpisteme export helpers	✅ fetch_acknowledgments, fetch_manual_aliases

6.4 CLI Commands ✅

Task	Status
aphoria policy export	✅ Exports local ack decisions as a Trust Pack
aphoria scan policy loading	✅ Auto-loads policies from config

Files: policy.rs, config.rs, episteme/mod.rs, lib.rs, main.rs

---

Phase 7: Declarative Extractors ⬜

Enable users to define new extractors in config/policy files (YAML/TOML) without writing Rust code. This removes the recompilation bottleneck for custom pattern enforcement.

7.1 Declarative Schema ⬜

Define a schema for pattern-based extraction:

extractors:
  - name: "api_style"
    language: "go"
    pattern: 'func \w+\(.*\) \[\]\w+'
    claim:
      subject: "api/response_format"
      predicate: "structure"
      object: "raw_array"

7.2 Implementation Tasks ⬜

Task	Description
DeclarativeExtractor	Generic extractor implementation reading from config
ExtractorConfig update	Load declarative definitions from aphoria.toml and Trust Packs
Regex optimization	Pre-compile all declarative patterns
Validation	Ensure valid regex and claim structure at load time

---

Milestone Summary

Phase	Deliverable	Depends On	Status
0	ConceptPath in StemeDB	concept-hierarchy spec	✅
2	Aphoria CLI (scan, report, ack)	Phase 0	✅
2A.1	Leaf-based concept matching	Phase 2	✅
2A.2	Alias resolution in QueryEngine	Phase 2	✅
2A.3	Auto-alias creation	Phase 2A.2	✅
1	Authoritative corpus expansion	Phase 0	✅
3	Claude Code skill + hooks	Phase 2A	✅
4.5	Ephemeral scan mode (40x faster)	Phase 2	✅
5	Research agent loop	Phase 3	✅
5.7	Security extractors (weak_crypto, command_injection, sql_injection)	Phase 2	✅
6	Federated Policy & Trust Packs	Phase 4.5	✅
7	Declarative Extractors	Phase 6	⬜ NEXT
4	Pre-commit integration (git hooks, diff scanning)	Phase 3, 4.5	⬜ PLANNED

Current state:

• Phase 1 is complete: RFC, OWASP, and Vendor corpus builders with aphoria corpus build CLI
• Phase 2 expanded: 10 extractors including weak_crypto, command_injection, sql_injection
• Phase 2 code quality: DES/RC4 concept paths fixed, SHA1 behavior documented, JS exec() regex tightened
• Phase 2A is complete: conflict detection via tail-path matching, alias-aware QueryEngine, and auto-alias creation
• Phase 3 is complete: /aphoria skill installed to ~/.claude/skills/aphoria/, hook templates ready
• Phase 4.5 is complete: Ephemeral scan mode with 40x faster performance for CI/pre-commit use
• Phase 5 is complete: Research agent with gap detection, quality validation, and official doc research
• Phase 6 is complete: Federated Policy & Trust Packs implemented (export, import, signing, remote loading)
• 183 tests pass. Clippy and fmt clean.

Next: Phase 7 — Declarative Extractors. This will allow Trust Packs to ship both the Policy (Assertion) and the Detection Logic (Extractor) in a single file.