41c676a78e
Enterprise Features: - Hosted mode with remote sync for team pattern aggregation - Community sharing with privacy-preserving anonymization - LLM-based semantic claim extraction with Gemini integration - Pattern learning with promotion to declarative extractors - High-entropy secrets extractor with configurable thresholds - Auth bypass and insecure cookies extractors Module Refactoring: - Split oversized files to comply with 500-line limit - Config split: types/core.rs, types/extractors.rs, types/hosted.rs, etc. - Handlers split: scan.rs, policy.rs, report.rs modules - Extractors split: declarative/, high_entropy_secrets/, insecure_cookies/ - Learning split: store modules with metrics and persistence SDK & Ontology: - stemedb-ontology SDK with fluent builders and StemeDB client - Pharma domain extractors for FDA Orange Book data - Consumer health UAT test infrastructure Code Quality: - Fixed clippy warnings (needless_borrows_for_generic_args) - Added KVStore trait imports where needed - Fixed utoipa path re-exports for OpenAPI docs Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2107 lines
75 KiB
Markdown
2107 lines
75 KiB
Markdown
# Aphoria Roadmap
|
|
|
|
---
|
|
|
|
## Phase 0: StemeDB Foundation ✅
|
|
|
|
> **Tracked in:** [roadmap.md § 5D. Concept Hierarchy](../../roadmap.md)
|
|
|
|
Changes to the core database that Aphoria depends on. Shipped as **Phase 5D** of the main StemeDB roadmap.
|
|
|
|
| Aphoria Phase 0 | StemeDB Phase 5D | Status |
|
|
|-----------------|------------------|--------|
|
|
| 0.1 ConceptPath Type | 5D.1 ConceptPath Type | ✅ |
|
|
| 0.2 ConceptPath in Assertion | (implicit in 5D.1) | ✅ |
|
|
| 0.3 Hierarchical Index | 5D.4 Hierarchical Query | ✅ |
|
|
| 0.4 Alias Store | 5D.3 Alias Store + 5D.5 Alias Resolution | ✅ |
|
|
| 0.5 Source Class Inference | 5D.6 Source Class Inference | ✅ |
|
|
| 0.6 Concept API Endpoints | 5D.7 Concept API Endpoints | ✅ |
|
|
|
|
**Spec:** [docs/specs/concept-hierarchy.md](../../docs/specs/concept-hierarchy.md)
|
|
|
|
---
|
|
|
|
## Phase 2: CLI Core ✅
|
|
|
|
> Phase 2 was built before Phase 1 (authoritative corpus expansion). The CLI pipeline works end-to-end with a bootstrapped corpus of 11 hardcoded assertions covering TLS, JWT, CORS, secrets, and rate limiting.
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| 2.1 Project Walker | ✅ `walker/mod.rs`, `walker/path_mapper.rs`, `walker/language.rs` |
|
|
| 2.2 Extractors (10) | ✅ `tls_verify`, `jwt_config`, `hardcoded_secrets`, `timeout_config`, `dep_versions`, `cors_config`, `rate_limit`, `weak_crypto`, `command_injection`, `sql_injection` |
|
|
| 2.3 Ingestion Bridge | ✅ `bridge.rs` — BLAKE3 hashing, Ed25519 signing, claim→assertion conversion |
|
|
| 2.4 Conflict Query | ✅ `episteme.rs` — LocalEpisteme with check_conflicts() |
|
|
| 2.5 Report Output | ✅ `report/` — table (comfy-table), JSON, SARIF 2.1.0, markdown |
|
|
| 2.6 Acknowledge Command | ✅ `lib.rs` acknowledge() |
|
|
| Baseline & Diff | ✅ `lib.rs` set_baseline(), show_diff() |
|
|
| Status Command | ✅ `lib.rs` show_status() |
|
|
|
|
183 tests pass. Clippy and fmt clean.
|
|
|
|
### Phase 2 Code Quality Fixes ✅
|
|
|
|
Code review improvements to extractors:
|
|
|
|
| Issue | Fix | Status |
|
|
|-------|-----|--------|
|
|
| DES/RC4 concept path misclassification | Split `check_pattern()` into `check_hash_pattern()` and `check_encryption_pattern()`; DES/RC4 now use `crypto/encryption/algorithm` path | ✅ |
|
|
| SHA1 edge case undocumented | Added comments and test documenting that SHA1 detection is intentionally broad (triggers for git hashes, etc.) | ✅ |
|
|
| JS exec() regex overly broad | Tightened regex to require `child_process.` prefix or non-word/non-dot preceding character; prevents `RegExp.exec()` false positives | ✅ |
|
|
|
|
---
|
|
|
|
## Phase 2A: Concept Matching ✅
|
|
|
|
> **Status:** Complete. Tail-path matching (2A.1), alias-aware queries (2A.2), and auto-alias creation (2A.3) all implemented.
|
|
|
|
### 2A.1 Leaf-Based Concept Matching (Aphoria-side fix) ✅
|
|
|
|
Implemented in `episteme.rs` via `ConceptIndex`:
|
|
- `make_key(subject, predicate)` extracts tail 2 path segments + predicate
|
|
- `build(assertions)` creates in-memory index keyed by tail path
|
|
- `lookup(subject, predicate)` finds matching authoritative assertions
|
|
- `check_conflicts()` uses `ConceptIndex` instead of `QueryEngine` for cross-scheme matching
|
|
|
|
Integration tests prove TLS and JWT conflicts are detected correctly.
|
|
|
|
### 2A.2 Alias Resolution in QueryEngine (StemeDB-side fix) ✅
|
|
|
|
Wired `AliasStore` into `QueryEngine.execute()`:
|
|
- Added `resolve_aliases: bool` field to `Query` (defaults to `false`)
|
|
- Added `alias_store: Option<Arc<dyn AliasStore>>` to `QueryEngine`
|
|
- Added `.with_alias_store()` builder method
|
|
- When `resolve_aliases: true`, expands subject via `AliasStore.resolve_all()` before index lookup
|
|
- Added `fetch_by_subjects()` and `fetch_by_subjects_predicate()` for multi-subject deduplication
|
|
- Modified `Query.matches()` to skip subject filtering when aliases are resolved
|
|
- Skips fast path (MV lookup) when `resolve_aliases: true`
|
|
- Gracefully degrades when no alias store is configured
|
|
|
|
7 unit tests in `engine/tests/alias_resolution.rs`. This is the architecturally correct long-term fix that complements leaf matching.
|
|
|
|
### 2A.3 Auto-Alias Creation ✅
|
|
|
|
When Aphoria ingests authoritative assertions and code claims that share leaf names, automatically create aliases:
|
|
- `code://rust/myapp/tls/cert_verification` ↔ `rfc://5246/tls/cert_verification`
|
|
- `code://rust/myapp/auth/jwt/audience_validation` ↔ `rfc://7519/jwt/audience_validation`
|
|
|
|
This bridges 2A.1 (leaf matching) with 2A.2 (alias resolution) — leaf matching identifies candidates, aliases persist the relationship.
|
|
|
|
**Implementation:**
|
|
- Added `auto_create_aliases: bool` config option to `AliasConfig` (defaults to `true`)
|
|
- Added `AliasOrigin::AutoDetected` variant to `stemedb-core` for tracking auto-created aliases
|
|
- Wired `GenericAliasStore` into `LocalEpisteme` for alias persistence
|
|
- In `check_conflicts()`, when a code claim matches an authoritative claim by leaf, calls `AliasStore.set_alias()` to persist the relationship with `AliasOrigin::AutoDetected`
|
|
- Alias creation is idempotent (skips if alias already exists)
|
|
- 4 unit tests verify: alias creation on conflict, no creation when disabled, correct origin, idempotency
|
|
|
|
---
|
|
|
|
## Phase 1: Authoritative Corpus Expansion ✅
|
|
|
|
> Expanded from 11 hardcoded assertions to a pluggable corpus system with RFC, OWASP, and Vendor sources.
|
|
|
|
### Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ aphoria corpus build │
|
|
│ │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌───────────────────────┐ │
|
|
│ │ RFC Ingester │ │ OWASP │ │ Vendor Bootstrapper │ │
|
|
│ │ (Tier 0) │ │ Ingester │ │ (Tier 2) │ │
|
|
│ │ │ │ (Tier 1) │ │ │ │
|
|
│ └──────┬───────┘ └──────┬───────┘ └───────────┬───────────┘ │
|
|
│ │ │ │ │
|
|
│ └─────────────────┼──────────────────────┘ │
|
|
│ ▼ │
|
|
│ ┌─────────────────┐ │
|
|
│ │ CorpusRegistry │ │
|
|
│ └────────┬────────┘ │
|
|
│ ▼ │
|
|
│ ┌─────────────────┐ │
|
|
│ │ LocalEpisteme │ │
|
|
│ │ ingest_ │ │
|
|
│ │ authoritative() │ │
|
|
│ └─────────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### 1.1 CorpusBuilder Trait ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `CorpusBuilder` trait | ✅ `corpus/mod.rs` — name, scheme, default_tier, build, requires_network |
|
|
| `CorpusRegistry` | ✅ Manages multiple builders, build_all(), list_builders() |
|
|
| `CorpusBuildResult` | ✅ Stats per builder, total assertions, success/fail/skip counts |
|
|
|
|
### 1.2 RFC Ingester ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `RfcCorpusBuilder` | ✅ `corpus/rfc.rs` |
|
|
| HTTP fetching | ✅ Via `ureq`, cached to `~/.cache/aphoria/rfc-cache/` |
|
|
| RFC 2119 keyword parsing | ✅ MUST, MUST NOT, SHOULD, SHALL extraction |
|
|
| RFC-specific parsers | ✅ JWT (7519), OAuth (6749), Bearer (6750), TLS 1.3 (8446), TLS BCP (7525), TOTP (6238), Basic Auth (7617), HTTP (9110) |
|
|
| Concept mapping | ✅ `rfc://{number}/{topic}` at Tier 0 (Regulatory) |
|
|
|
|
### 1.3 OWASP Ingester ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `OwaspCorpusBuilder` | ✅ `corpus/owasp.rs` |
|
|
| HTTP fetching | ✅ From GitHub raw content, cached to `~/.cache/aphoria/owasp-cache/` |
|
|
| Markdown parsing | ✅ MUST/SHOULD statements, section context |
|
|
| Cheat sheet parsers | ✅ Authentication, JWT, TLS, Secrets, Input Validation, Session, CSRF, Password Storage, HTTP Headers |
|
|
| Concept mapping | ✅ `owasp://cheatsheet/{topic}/{claim}` at Tier 1 (Clinical) |
|
|
|
|
### 1.4 Vendor Docs ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `VendorCorpusBuilder` | ✅ `corpus/vendor.rs` |
|
|
| PostgreSQL claims | ✅ pool_size, idle_timeout, ssl_mode |
|
|
| Redis claims | ✅ timeout, max_retries, tls |
|
|
| reqwest claims | ✅ cert_verification, connect_timeout, request_timeout |
|
|
| hyper claims | ✅ keep_alive_timeout, max_concurrent_streams |
|
|
| Go net/http claims | ✅ read_timeout, write_timeout, idle_timeout, min_tls_version |
|
|
| tokio-postgres claims | ✅ pool_size, ssl_mode |
|
|
| SQLx claims | ✅ max_connections, idle_timeout |
|
|
| Concept mapping | ✅ `vendor://{product}/{topic}/{claim}` at Tier 2 (Observational) |
|
|
|
|
### 1.5 Hardcoded Refactor ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `HardcodedCorpusBuilder` | ✅ `corpus/hardcoded.rs` — original 11 assertions |
|
|
| `create_authoritative_assertion()` | ✅ Made public in `episteme.rs` for corpus builders |
|
|
|
|
### 1.6 CLI Integration ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `aphoria corpus build` | ✅ Fetches and ingests from all sources |
|
|
| `--only rfc,owasp,vendor` | ✅ Filter to specific sources |
|
|
| `--offline` | ✅ Skip network-requiring sources |
|
|
| `--clear-cache` | ✅ Clear cache before building |
|
|
| `aphoria corpus list` | ✅ List available corpus sources |
|
|
| `CorpusConfig` | ✅ cache_dir, include_*, rfc_list options |
|
|
|
|
### 1.7 Error Handling ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `RfcFetch` error | ✅ Per-RFC fetch failures with context |
|
|
| `OwaspFetch` error | ✅ Per-cheat-sheet fetch failures with context |
|
|
| `CorpusBuild` error | ✅ General corpus build failures |
|
|
| Graceful degradation | ✅ Continue with other sources if one fails |
|
|
|
|
**Files:** `corpus/mod.rs`, `corpus/hardcoded.rs`, `corpus/rfc.rs`, `corpus/owasp.rs`, `corpus/vendor.rs`
|
|
|
|
---
|
|
|
|
## Phase 3: Skill Integration ✅
|
|
|
|
> Complete. Aphoria is now usable in Claude Code agent workflows.
|
|
|
|
### 3.1 Claude Code Skill ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `skill/SKILL.md` | ✅ Comprehensive skill definition with all commands |
|
|
| `/aphoria scan` | ✅ Scan project, show conflicts grouped by verdict |
|
|
| `/aphoria scan --fix` | ✅ Interactive fix workflow |
|
|
| `/aphoria ack` | ✅ Acknowledge conflicts as intentional |
|
|
| `/aphoria status` | ✅ Show status and baseline |
|
|
| `/aphoria diff` | ✅ Show changes since baseline |
|
|
| `/aphoria init` | ✅ Initialize Aphoria |
|
|
| `/aphoria baseline` | ✅ Set baseline |
|
|
| `skill/install.sh` | ✅ Install script for `~/.claude/skills/aphoria/` |
|
|
|
|
**Files:** `skill/SKILL.md`, `skill/install.sh`, `skill/hooks.json`
|
|
|
|
### 3.2 Agent Pre-Flight Hook ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `--exit-code` flag | ✅ Returns 2 for BLOCK, 1 for FLAG only, 0 for clean |
|
|
| `--strict` flag | ✅ Lower thresholds (FLAG at 0.3, BLOCK at 0.5) |
|
|
| Hook template | ✅ `skill/hooks.json` with PreCommit and PrePush examples |
|
|
|
|
**Usage:**
|
|
```json
|
|
{
|
|
"hooks": {
|
|
"PreCommit": [{"command": "aphoria scan --format sarif --exit-code"}],
|
|
"PrePush": [{"command": "aphoria scan --strict --exit-code"}]
|
|
}
|
|
}
|
|
```
|
|
|
|
### 3.3 Alias Suggestion Workflow ✅
|
|
|
|
Auto-alias creation is now automatic (Phase 2A.3). When Aphoria scans:
|
|
1. Tail-path matching finds authoritative assertions
|
|
2. Aliases are auto-created with `AliasOrigin::AutoDetected`
|
|
3. Future queries use the alias automatically
|
|
|
|
The skill documents the suggestion flow for manual alias management:
|
|
- **y (Accept)**: Creates alias
|
|
- **n (Reject)**: Records intentional difference
|
|
- **defer**: Flags for later review
|
|
|
|
---
|
|
|
|
## Phase 4: Full-Cycle Pre-Commit (Scan + Sync) ✅
|
|
|
|
> **Vision:** The pre-commit hook is a **bidirectional knowledge sync**, not just a read-only linter. Every commit extracts claims, checks authority, detects drift from prior observations, and records new observations back.
|
|
|
|
**Spec:** [uat/2026-02-04-full-cycle-precommit-vision.md](uat/2026-02-04-full-cycle-precommit-vision.md)
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ PRE-COMMIT FLOW │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ 1. EXTRACT → What claims does this code make? │
|
|
│ 2. CHECK → Against authority + own prior claims │
|
|
│ 3. CLASSIFY → Authority conflict | Self conflict | Novel │
|
|
│ 4. UPDATE → Record observations to local Episteme │
|
|
│ 5. GATE → Exit code (BLOCK=2, FLAG=1, PASS=0) │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### 4.1 Git Pre-Commit Hook ✅
|
|
|
|
All flags needed for pre-commit integration are implemented:
|
|
|
|
```bash
|
|
#!/bin/sh
|
|
# .git/hooks/pre-commit
|
|
aphoria scan --staged --sync --exit-code
|
|
```
|
|
|
|
Or using pre-commit framework:
|
|
|
|
```yaml
|
|
repos:
|
|
- repo: local
|
|
hooks:
|
|
- id: aphoria
|
|
name: Aphoria Truth Sync
|
|
entry: aphoria scan --staged --sync --exit-code
|
|
language: system
|
|
pass_filenames: false
|
|
```
|
|
|
|
### 4.2 Baseline Mode ✅
|
|
|
|
Already implemented in Phase 2.
|
|
|
|
### 4A: Observational Claims ✅
|
|
|
|
Record code claims as Tier 4 (Community) assertions when no authority conflict exists:
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `sync: bool` in ScanArgs | ✅ `types/command.rs` |
|
|
| `observations_recorded: usize` in ScanResult | ✅ `types/result.rs` |
|
|
| `--sync` CLI flag | ✅ `cli.rs` — requires `--persist` |
|
|
| `claim_to_observation()` | ✅ `bridge.rs` — creates Tier 4 (Community, 0.3 weight) assertions |
|
|
| `ingest_observations()` in LocalEpisteme | ✅ `episteme/local.rs` — writes to WAL + predicate index |
|
|
| Scan flow integration | ✅ `scan.rs` — splits claims by conflict status, writes novel claims as observations |
|
|
| Handler validation | ✅ `handlers.rs` — `--sync requires --persist` error |
|
|
| Report output | ✅ `report/table.rs`, `report/json.rs` — shows observation count |
|
|
| Tests | ✅ 5 new tests for observation write-back |
|
|
|
|
```
|
|
Code: connection_pool.max_size = 25
|
|
Authority: (nothing)
|
|
Action: Record as Tier 4 observation (project memory)
|
|
```
|
|
|
|
**Usage:**
|
|
```bash
|
|
# Scan with observation write-back
|
|
aphoria scan --persist --sync
|
|
|
|
# Output:
|
|
# Recorded 45 observations (project memory)
|
|
```
|
|
|
|
### 4B: Self-Conflict Detection ✅
|
|
|
|
Detect drift from the project's own prior observations:
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| Query prior claims before conflict check | ✅ `fetch_observations_for_concept()` |
|
|
| Compare current vs stored observations | ✅ `check_drift()` compares values |
|
|
| Report changes as SELF-CONFLICT | ✅ DriftResult with prior/current values |
|
|
| New verdict: `Drift` (distinct from Block/Flag) | ✅ `Verdict::Drift` |
|
|
| Drift reporting in all formats | ✅ table, json, markdown, sarif |
|
|
| Exit code includes drift | ✅ `--exit-code` returns 1 for drift |
|
|
|
|
```
|
|
Prior: db/pool_size = 25 (recorded 2026-01-15)
|
|
Now: db/pool_size = 100
|
|
Result: DRIFT — "You changed pool_size from 25 to 100. Intentional?"
|
|
```
|
|
|
|
**Files:** `types/result.rs`, `types/verdict.rs`, `episteme/local.rs`, `scan.rs`, `report/*.rs`
|
|
|
|
### 4C: Diff-Only Scanning ✅
|
|
|
|
Fast scanning for pre-commit hooks:
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `FileSource` enum (All, Staged) | ✅ `types/command.rs` |
|
|
| `--staged` flag (git diff --cached) | ✅ `cli.rs`, `handlers.rs` |
|
|
| `walker/git.rs` git utilities | ✅ `find_repo_root()`, `get_staged_files()` |
|
|
| `walk_staged_files()` | ✅ `walker/mod.rs` — filters to scan root, applies same filters |
|
|
| Scan dispatch by file_source | ✅ `scan.rs` |
|
|
| Error handling (NotGitRepo, GitCommand) | ✅ `error.rs` |
|
|
| Tests | ✅ 9 tests in `tests/staged_scanning.rs` |
|
|
| Target: < 500ms for staged-only | ✅ |
|
|
|
|
**Files:** `types/command.rs`, `walker/git.rs`, `walker/mod.rs`, `scan.rs`, `cli.rs`, `handlers.rs`, `error.rs`
|
|
|
|
**Usage:**
|
|
```bash
|
|
# Pre-commit hook (fast, staged files only)
|
|
aphoria scan --staged --exit-code
|
|
|
|
# Full cycle with observation sync
|
|
aphoria scan --staged --persist --sync --exit-code
|
|
```
|
|
|
|
### 4D: Enhanced Ack ✅
|
|
|
|
Acknowledgments with rationale and policy updates:
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `--reason "text"` flag | ✅ `cli.rs` — required on `ack`, `bless`, `update` commands |
|
|
| Store rationale in assertion metadata | ✅ `policy_ops.rs` — stored in value/description fields |
|
|
| `aphoria update` for intentional drift | ✅ `policy_ops.rs` — creates `policy_update` assertion |
|
|
| Policy update assertions | ✅ `types/mod.rs` — `predicates::POLICY_UPDATE` |
|
|
|
|
**Files:** `cli.rs`, `handlers.rs`, `policy_ops.rs`, `types/command.rs`, `types/mod.rs`
|
|
|
|
```bash
|
|
$ aphoria ack db/pool_size --reason "Scaling for Black Friday"
|
|
$ aphoria update db/pool_size 100 --reason "New baseline after load test"
|
|
```
|
|
|
|
### 4E: Hosted Mode ✅
|
|
|
|
Organizations run their own StemeDB server and all team members automatically sync observations:
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `HostedConfig` in config.rs | ✅ `url`, `project_id`, `team_id`, `sync_mode`, `offline_fallback`, `api_key_env` |
|
|
| `SyncMode` enum | ✅ `remote-only` (default), `local-and-remote` |
|
|
| `OfflineFallback` enum | ✅ `skip` (default), `fail`, `queue` |
|
|
| `HostedClient` HTTP client | ✅ `hosted.rs` — retry logic, auth headers, observation push |
|
|
| `POST /v1/aphoria/observations` endpoint | ✅ Server receives observations with project/team metadata |
|
|
| Scan integration | ✅ Auto-enables sync when `[hosted]` configured |
|
|
| `Hosted(String)` error variant | ✅ For connection/auth failures |
|
|
| Graceful offline fallback | ✅ Based on `offline_fallback` config |
|
|
| Tests | ✅ Config parsing, client creation, assertion conversion |
|
|
|
|
```toml
|
|
# aphoria.toml
|
|
[hosted]
|
|
url = "https://episteme.acme.corp" # Enables hosted mode
|
|
project_id = "billing-service" # Optional, defaults to [project.name]
|
|
team_id = "platform-team" # Optional, for multi-team servers
|
|
sync_mode = "remote-only" # "remote-only" | "local-and-remote"
|
|
offline_fallback = "skip" # "skip" | "fail" | "queue"
|
|
api_key_env = "APHORIA_API_KEY" # Env var for auth token
|
|
```
|
|
|
|
**Architecture:**
|
|
```
|
|
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
|
|
│ Developer A │ │ Developer B │ │ Developer C │
|
|
│ aphoria scan │ │ aphoria scan │ │ aphoria scan │
|
|
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘
|
|
│ │ │
|
|
└─────────────────┼─────────────────┘
|
|
▼
|
|
┌─────────────────────┐
|
|
│ Team StemeDB Server │
|
|
│ POST /v1/aphoria/ │
|
|
│ observations │
|
|
└─────────────────────┘
|
|
│
|
|
▼
|
|
Aggregated team patterns
|
|
```
|
|
|
|
**Files:** `config.rs`, `hosted.rs`, `scan.rs`, `error.rs`, `lib.rs`, `crates/stemedb-api/src/handlers/aphoria.rs`, `crates/stemedb-api/src/dto/aphoria.rs`
|
|
|
|
---
|
|
|
|
## Phase 4.5: Ephemeral Scan Mode ✅
|
|
|
|
> Performance optimization: 40x faster scans by skipping Episteme storage when persistence isn't needed.
|
|
|
|
### Problem
|
|
|
|
Every `aphoria scan` was slow because it initialized the full Episteme stack:
|
|
- WAL recovery (O(n) on every startup)
|
|
- Dual backend initialization (fjall + redb)
|
|
- Store and index initialization
|
|
|
|
But conflict detection is actually 100% in-memory — it never reads from the KV store. The authoritative corpus is built fresh each time, and code claims are extracted fresh each scan.
|
|
|
|
### Solution
|
|
|
|
Added `ScanMode` enum with two modes:
|
|
|
|
| Mode | Use Case | Storage | Performance |
|
|
|------|----------|---------|-------------|
|
|
| **Ephemeral** (default) | CI, pre-commit, quick checks | None | ~0.25 seconds |
|
|
| **Persistent** | Baseline/diff tracking, alias creation | WAL + store | ~1-2 seconds |
|
|
|
|
### Implementation ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `ScanMode` enum | ✅ `types.rs` — Ephemeral (default), Persistent |
|
|
| `EphemeralDetector` struct | ✅ `episteme/mod.rs` — in-memory corpus + ConceptIndex |
|
|
| `check_conflicts_pure()` | ✅ Extracted as standalone function for reuse |
|
|
| Mode-based dispatch in `run_scan()` | ✅ Uses `EphemeralDetector` for Ephemeral, `LocalEpisteme` for Persistent |
|
|
| `--persist` CLI flag | ✅ `main.rs` — opt-in to persistent mode |
|
|
| Tests for both modes | ✅ `test_ephemeral_scan_no_storage_created`, `test_persistent_scan_creates_storage`, `test_scan_modes_produce_same_conflicts` |
|
|
|
|
### Usage
|
|
|
|
```bash
|
|
# Fast ephemeral scan (default) — no storage created
|
|
aphoria scan .
|
|
|
|
# Persistent scan — enables baseline, diff, auto-alias features
|
|
aphoria scan . --persist
|
|
```
|
|
|
|
### Performance
|
|
|
|
| Mode | Time | Storage |
|
|
|------|------|---------|
|
|
| Ephemeral | ~0.25s | None |
|
|
| Persistent | ~1-2s | WAL + store directories |
|
|
|
|
**Files:** `types.rs`, `episteme/mod.rs`, `lib.rs`, `main.rs`, `tests.rs`
|
|
|
|
---
|
|
|
|
## Phase 5: Research Agent Loop ✅
|
|
|
|
> Research agent fills gaps in authoritative coverage by researching official documentation.
|
|
|
|
### 5.1 Gap Detection ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `Gap` struct | ✅ `research/gap_detector.rs` — concept_path, topic, predicate, source info |
|
|
| `detect_gaps()` | ✅ Compares claims against ConceptIndex, identifies missing coverage |
|
|
| Topic normalization | ✅ Extracts last 2 path segments for cross-scheme matching |
|
|
| Deduplication | ✅ Deduplicates gaps by topic+predicate key |
|
|
|
|
### 5.2 Gap Storage ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `GapRecord` | ✅ `research/gap_store.rs` — tracking metadata, project count, research status |
|
|
| `GapStore` | ✅ JSON-backed persistent storage with atomic saves |
|
|
| Project tracking | ✅ Records which projects reported each gap |
|
|
| Research eligibility | ✅ `is_eligible_for_research()` with threshold and cooldown |
|
|
| Gap pruning | ✅ `prune_old_gaps()` removes stale entries |
|
|
|
|
### 5.3 Quality Validation ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `QualityValidator` | ✅ `research/quality.rs` — validates researched claims |
|
|
| Source attribution | ✅ Checks for authoritative domains (rfc-editor, owasp, vendor docs) |
|
|
| Normative language | ✅ Verifies MUST/SHOULD/SHALL keywords present |
|
|
| Vague content detection | ✅ Rejects "it depends", "typically", etc. |
|
|
| Consistency scoring | ✅ Detects conflicting claims on same subject |
|
|
| `QualityReport` | ✅ Detailed per-claim validation results |
|
|
| `filter_passed()` | ✅ Returns only claims meeting quality threshold |
|
|
|
|
### 5.4 Research Execution ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `Researcher` | ✅ `research/researcher.rs` — orchestrates research pipeline |
|
|
| `DocumentationSource` | ✅ Configurable sources with URL patterns and topics |
|
|
| Default sources | ✅ Redis, PostgreSQL, Go, Rust, OWASP, Kafka, MongoDB |
|
|
| Content fetching | ✅ HTTP with timeout and size limits |
|
|
| Normative extraction | ✅ Regex-based MUST/SHOULD/SHALL extraction |
|
|
| Section tracking | ✅ Extracts heading context for attribution |
|
|
| Confidence scoring | ✅ Based on keyword strength, statement length, content size |
|
|
|
|
### 5.5 CLI Integration ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `aphoria research run` | ✅ Run research agent with configurable threshold |
|
|
| `aphoria research status` | ✅ Show gap statistics and research progress |
|
|
| `aphoria research gaps` | ✅ List gaps by project count |
|
|
| `--threshold` | ✅ Minimum projects before researching (default: 3) |
|
|
| `--strict` | ✅ Use strict quality validation |
|
|
| `--prune` | ✅ Remove stale gaps before researching |
|
|
| `--ready` | ✅ Show only gaps ready for research |
|
|
|
|
**Files:** `research/mod.rs`, `research/gap_detector.rs`, `research/gap_store.rs`, `research/quality.rs`, `research/researcher.rs`, `research/tests.rs`
|
|
|
|
### 5.7 Security Extractors ✅
|
|
|
|
Extended Phase 2 extractors with OWASP-aligned security vulnerability detection:
|
|
|
|
| Extractor | Detects | Languages |
|
|
|-----------|---------|-----------|
|
|
| `weak_crypto` | MD5, SHA1, DES, RC4 usage | Rust, Go, Python, JS/TS |
|
|
| `command_injection` | Shell execution, os.system, subprocess shell=True | Rust, Go, Python, JS/TS |
|
|
| `sql_injection` | String concatenation in SQL queries | Rust, Go, Python, JS/TS |
|
|
|
|
**Concept paths:**
|
|
- `crypto/hashing/algorithm` — MD5, SHA1
|
|
- `crypto/encryption/algorithm` — DES, RC4
|
|
- `os/command/input`, `os/shell_mode` — command injection
|
|
- `db/query/input` — SQL injection
|
|
|
|
### 5.6 Community Corpus Contributions ✅
|
|
|
|
> Users can opt in to contribute patterns anonymously to a central corpus, enabling community consensus to adjust default thresholds.
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `CommunityConfig` | ✅ `config/mod.rs` — enabled (false), anonymize (true), exclude, include, min_confidence |
|
|
| `AnonymizedObservation` | ✅ `community/types.rs` — privacy-preserving observation without file/line/text |
|
|
| `CommunityObjectValue` | ✅ `community/types.rs` — serde-compatible version of ObjectValue |
|
|
| `PatternAggregate` | ✅ `community/types.rs` — server-side aggregation with project counts |
|
|
| `anonymize_claim()` | ✅ `community/anonymizer.rs` — wildcards project names, strips file/line, rounds timestamps |
|
|
| `compute_anon_hash()` | ✅ Hash computed WITHOUT file/line/text (privacy-critical) |
|
|
| `wildcard_project_path()` | ✅ `code://rust/myapp/tls` → `code://rust/*/tls` |
|
|
| `--community-preview` flag | ✅ `cli.rs` — dry-run showing what WOULD be shared |
|
|
| `PatternAggregateStore` | ✅ `stemedb-storage` — server-side pattern aggregation |
|
|
| Project deduplication | ✅ Uses project_hash to prevent double-counting |
|
|
| `POST /v1/aphoria/community/observations` | ✅ Push anonymized observations |
|
|
| `GET /v1/aphoria/patterns` | ✅ Retrieve high-confidence community patterns |
|
|
|
|
**Privacy Model:**
|
|
- Project names wildcarded: `myapp` → `*`
|
|
- File paths, line numbers, matched text NEVER shared
|
|
- Timestamps rounded to hour (k-anonymity)
|
|
- Server receives `project_hash`, not raw project names
|
|
- `enabled` defaults to `false` (explicit opt-in required)
|
|
- `anonymize` defaults to `true` (privacy-preserving by default)
|
|
|
|
**Usage:**
|
|
```bash
|
|
# Preview what would be shared (no network)
|
|
aphoria scan --community-preview
|
|
|
|
# Enable in aphoria.toml:
|
|
[community]
|
|
enabled = true
|
|
anonymize = true
|
|
min_confidence = 0.8
|
|
exclude = ["vendor://acme/internal/*"]
|
|
|
|
# Scan with sync to share patterns
|
|
aphoria scan --persist --sync
|
|
```
|
|
|
|
**Files:** `community/mod.rs`, `community/types.rs`, `community/anonymizer.rs`, `config/mod.rs`, `cli.rs`, `handlers.rs`, `stemedb-storage/src/pattern_aggregate_store/`
|
|
|
|
---
|
|
|
|
## Phase 6: Federated Policy & Trust Packs ✅
|
|
|
|
> Allow teams to define their own authoritative truths and distribute them as signed Trust Packs. This enables "Enterprise Grade" compliance across distributed teams.
|
|
|
|
### 6.1 Trust Pack Format ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `TrustPack` schema | ✅ `policy.rs` — Assertions, Aliases, Metadata, Signature |
|
|
| `PackHeader` | ✅ Name, version, issuer, timestamp |
|
|
| Serialization | ✅ `rkyv` for zero-copy efficiency |
|
|
| Signing | ✅ `ed25519-dalek` signing and verification |
|
|
|
|
### 6.2 Policy Management ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `PolicyManager` | ✅ Loads local and remote (HTTP/HTTPS) policies |
|
|
| Caching | ✅ Caches remote policies in `~/.cache/aphoria/policies/` |
|
|
| `aphoria.toml` config | ✅ `policies` list support |
|
|
|
|
### 6.3 Core Integration ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `EphemeralDetector` integration | ✅ Ingests policies into memory corpus/index |
|
|
| `check_conflicts_pure` update | ✅ Resolves policy aliases before authoritative lookup |
|
|
| `LocalEpisteme` export helpers | ✅ `fetch_acknowledgments`, `fetch_manual_aliases` |
|
|
|
|
### 6.4 CLI Commands ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `aphoria policy export` | ✅ Exports local `ack` decisions as a Trust Pack |
|
|
| `aphoria scan` policy loading | ✅ Auto-loads policies from config |
|
|
|
|
**Files:** `policy.rs`, `config.rs`, `episteme/mod.rs`, `lib.rs`, `main.rs`
|
|
|
|
---
|
|
|
|
## Phase 6.5: Trust Pack Extensions ⬜
|
|
|
|
> Enhancements to Trust Packs based on enterprise pilot feedback. Deferred until real-world usage patterns emerge.
|
|
|
|
### 6.5.1 Predicate Aliases ⬜
|
|
|
|
**Status:** Deferred pending enterprise feedback
|
|
**Trigger:** When enterprises report predicate naming conflicts between policy and extractors
|
|
|
|
**User Story:**
|
|
> As a security architect, when my policy uses `required=true` but the extractor emits `enabled=true`, I need them to match semantically.
|
|
|
|
**Problem:**
|
|
- Policy blesses: `code://standard/tls/cert_verification` with predicate `required`, value `true`
|
|
- Extractor emits: `code://config/tls/cert_verification` with predicate `enabled`, value `false`
|
|
- Tail-path matching finds the concept (`tls/cert_verification`) ✓
|
|
- But predicates differ: `required` vs `enabled` — no conflict detected ✗
|
|
|
|
**Solution:**
|
|
|
|
| Task | Description |
|
|
|------|-------------|
|
|
| `predicate_aliases` field | Add to Trust Pack schema |
|
|
| Default aliases | `enabled` ↔ `required` ↔ `mandatory` ↔ `enforced` |
|
|
| ConceptIndex update | Check aliases during lookup |
|
|
| Pack-defined aliases | Allow packs to specify custom alias sets |
|
|
|
|
**Trust Pack Schema Extension:**
|
|
```toml
|
|
# In Trust Pack
|
|
[predicate_aliases]
|
|
security_enabled = ["enabled", "required", "mandatory", "enforced", "active"]
|
|
version_minimum = ["min_version", "minimum_version", "tls_min_version"]
|
|
```
|
|
|
|
**Implementation Plan:**
|
|
1. Add `predicate_aliases: HashMap<String, Vec<String>>` to `TrustPack`
|
|
2. Store aliases alongside assertions during import
|
|
3. Update `ConceptIndex.make_key()` to normalize predicates via aliases
|
|
4. Match during conflict detection: if `predicate_a` aliases to `predicate_b`, treat as same concept
|
|
|
|
### 6.5.2 Pack Signing Key Rotation ⬜
|
|
|
|
**Status:** Deferred pending security key management requirements
|
|
**Trigger:** Enterprise security requirements for key rotation
|
|
|
|
**User Story:**
|
|
> As a security admin, when our signing key is rotated, I need to re-sign all packs without losing policy content.
|
|
|
|
**Problem:**
|
|
- Trust Packs are signed with Ed25519 keys
|
|
- When keys are rotated (security best practice), existing packs become unverifiable
|
|
- Need to re-sign packs with new key while preserving content hash
|
|
|
|
**Solution:**
|
|
|
|
| Task | Description |
|
|
|------|-------------|
|
|
| `aphoria policy resign` | CLI command to re-sign pack with new key |
|
|
| Content hash preservation | Keep `content_hash` unchanged, only update signature |
|
|
| Key rotation audit | Log key rotation events |
|
|
| Old signature archival | Optionally keep old signature for audit trail |
|
|
|
|
**CLI:**
|
|
```bash
|
|
# Re-sign pack with new key
|
|
aphoria policy resign my-standards.pack --key-file new-private-key.pem
|
|
|
|
# Re-sign with signature chain (audit trail)
|
|
aphoria policy resign my-standards.pack --key-file new-key.pem --chain-signatures
|
|
```
|
|
|
|
**Trust Pack Schema Extension:**
|
|
```rust
|
|
pub struct TrustPack {
|
|
// Existing fields...
|
|
pub signature: Signature,
|
|
|
|
// New field for key rotation audit
|
|
pub signature_chain: Option<Vec<SignatureRecord>>,
|
|
}
|
|
|
|
pub struct SignatureRecord {
|
|
pub issuer_public_key: [u8; 32],
|
|
pub signature: Signature,
|
|
pub signed_at: DateTime<Utc>,
|
|
pub reason: Option<String>, // "Key rotation", "Security incident", etc.
|
|
}
|
|
```
|
|
|
|
### 6.5.3 Priority
|
|
|
|
| Feature | Priority | Trigger |
|
|
|---------|----------|---------|
|
|
| Predicate Aliases | Medium | Enterprise feedback showing predicate naming conflicts |
|
|
| Key Rotation | Low | Enterprise security key management requirements |
|
|
|
|
**Documented in:** [uat/future-scenarios.md](uat/future-scenarios.md)
|
|
|
|
---
|
|
|
|
## Phase 7: Declarative Extractors ✅
|
|
|
|
> Enable users to define new extractors in config/policy files (TOML) without writing Rust code. This removes the recompilation bottleneck for custom pattern enforcement.
|
|
|
|
**User Outcome:** "I added a custom extractor to my aphoria.toml that detects our company's deprecated API patterns. Now every scan flags files using the old pattern without me writing any Rust code."
|
|
|
|
### 7.1 Core Types ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `DeclarativeExtractorDef` | ✅ `extractors/declarative.rs` — name, description, languages, pattern, claim, confidence |
|
|
| `DeclarativeClaimDef` | ✅ subject, predicate, value specification |
|
|
| `DeclarativeValue` enum | ✅ MatchedText, Boolean, Text variants |
|
|
| `DeclarativeExtractor` | ✅ Compiled extractor with `Extractor` trait impl |
|
|
|
|
### 7.2 Configuration ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `ExtractorConfig.declarative` | ✅ `config/mod.rs` — `Vec<DeclarativeExtractorDef>` |
|
|
| TOML parsing | ✅ Serde deserialization with `#[serde(untagged)]` for value types |
|
|
| Example config | ✅ Documented in module and config docs |
|
|
|
|
**Example aphoria.toml:**
|
|
```toml
|
|
[[extractors.declarative]]
|
|
name = "deprecated_api_v1"
|
|
description = "Detects usage of deprecated v1 API endpoints"
|
|
languages = ["go", "rust", "python"]
|
|
pattern = '/api/v1/\w+'
|
|
claim.subject = "api/deprecated_endpoint"
|
|
claim.predicate = "version"
|
|
claim.value = "v1"
|
|
confidence = 1.0
|
|
|
|
[[extractors.declarative]]
|
|
name = "legacy_encryption"
|
|
description = "Detects legacy encryption algorithms"
|
|
languages = ["rust", "go", "python", "javascript"]
|
|
pattern = '(?i)blowfish|twofish|cast5'
|
|
claim.subject = "crypto/encryption/algorithm"
|
|
claim.predicate = "algorithm"
|
|
claim.value_from_match = true
|
|
confidence = 0.9
|
|
```
|
|
|
|
### 7.3 Validation & Security ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| Name validation | ✅ Non-empty required |
|
|
| Subject/predicate validation | ✅ Non-empty required |
|
|
| Confidence validation | ✅ Must be 0.0-1.0 |
|
|
| Regex validation | ✅ Compiled at load time, not scan time |
|
|
| ReDoS protection | ✅ `RegexBuilder` with 10MB size limits |
|
|
| Language parsing | ✅ `Language::from_str()` with `FromStr` trait |
|
|
| Graceful failure | ✅ Invalid extractors logged as warnings, don't block others |
|
|
|
|
### 7.4 Registry Integration ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| Module export | ✅ `extractors/mod.rs` — public types |
|
|
| Registry registration | ✅ `ExtractorRegistry::new()` loads from config |
|
|
| Enable/disable support | ✅ Declarative extractors respect `disabled` list |
|
|
| Runtime addition | ✅ `add_from_definitions()` for Trust Pack integration |
|
|
|
|
### 7.5 Error Handling ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `DeclarativeExtractor` error variant | ✅ `error.rs` — name + message |
|
|
| Validation errors | ✅ Clear messages for each failure mode |
|
|
| Structured logging | ✅ `tracing::warn!` for compilation failures |
|
|
|
|
### 7.6 Tests ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| Unit tests | ✅ 22 tests in `declarative.rs` |
|
|
| Registry tests | ✅ 7 tests for integration |
|
|
| Validation tests | ✅ Empty name, subject, predicate; invalid confidence, regex, language |
|
|
| Extraction tests | ✅ Boolean, text, matched_text value types |
|
|
| Deserialization tests | ✅ TOML parsing for all value types |
|
|
|
|
**Files:** `extractors/declarative.rs`, `extractors/mod.rs`, `config/mod.rs`, `types/language.rs`, `error.rs`
|
|
|
|
---
|
|
|
|
## Phase 7.5: LLM-in-the-Loop Extraction ✅
|
|
|
|
> Use LLM (Gemini) to extract claims semantically during persistent scans. This fills gaps that regex extractors can't catch, providing immediate value while the learning system builds up pattern knowledge.
|
|
|
|
### Vision
|
|
|
|
```
|
|
Code file → Regex extractors → Claims found
|
|
↓
|
|
High-value files (auth, config, crypto)
|
|
↓
|
|
LLM Extractor → Additional semantic claims
|
|
↓
|
|
Combined claims → Conflict detection
|
|
```
|
|
|
|
### 7.5.1 LLM Extractor Implementation ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `GeminiClient` struct | ✅ `llm/client.rs` — Gemini API client using ureq |
|
|
| `LlmExtractor` struct | ✅ `llm/extractor.rs` — orchestrates extraction with budget tracking |
|
|
| Prompt engineering | ✅ Security-focused extraction prompt with structured JSON output |
|
|
| Response parsing | ✅ Parse Gemini's JSON response into `ExtractedClaim` format |
|
|
| Error handling | ✅ Graceful degradation when API unavailable or key missing |
|
|
|
|
### 7.5.2 Selective Triggering ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `is_high_value_file()` | ✅ `llm/extractor.rs` — auth/, config/, crypto/, security/, secrets/, certs/, ssl/, tls/, keys/, credentials/ directories |
|
|
| High-value file names | ✅ secret, password, credential, token, auth, login, session, jwt, tls, ssl, cert, key, config, settings, security, crypto, encrypt, decrypt, oauth, saml, ldap, api_key, apikey, access_key, private |
|
|
| Token budget | ✅ `max_tokens_per_scan` (default 50k), `max_tokens_per_file` (default 4k) |
|
|
| Skip conditions | ✅ Only runs when regex extractors found nothing AND file is high-value |
|
|
|
|
### 7.5.3 Cost Controls ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| Token tracking | ✅ `Arc<AtomicUsize>` for thread-safe budget tracking across files |
|
|
| BLAKE3 caching | ✅ `llm/cache.rs` — content hash + model + prompt version for cache key |
|
|
| Cache location | ✅ `~/.cache/aphoria/llm-cache/` |
|
|
| Budget enforcement | ✅ `within_budget()` check before each LLM call |
|
|
|
|
### 7.5.4 Configuration ✅
|
|
|
|
```toml
|
|
# aphoria.toml
|
|
[llm]
|
|
enabled = true # Enable LLM extraction (default: false)
|
|
provider = "gemini" # Only "gemini" supported
|
|
# model defaults to DEFAULT_LLM_MODEL (currently "gemini-3-flash-preview")
|
|
api_key_env = "GEMINI_API_KEY" # Environment variable for API key
|
|
max_tokens_per_scan = 50000 # Budget per scan
|
|
max_tokens_per_file = 4000 # Budget per file (for max_output_tokens)
|
|
high_value_only = true # Only use on auth/config/crypto files
|
|
cache_responses = true # Cache by content hash
|
|
timeout_secs = 60 # API timeout
|
|
min_confidence = 0.7 # Filter claims below this confidence
|
|
```
|
|
|
|
**Files:** `llm/mod.rs`, `llm/client.rs`, `llm/extractor.rs`, `llm/cache.rs`, `config/mod.rs`, `scan.rs`, `error.rs`
|
|
|
|
---
|
|
|
|
## Phase 7.6: Pattern Learning Store ✅
|
|
|
|
> When LLM extracts something that regex extractors missed, remember the pattern. Track which patterns recur across projects to identify candidates for promotion to declarative extractors.
|
|
|
|
### Vision
|
|
|
|
```
|
|
LLM extracts claim from code
|
|
↓
|
|
Pattern not in learned store?
|
|
↓
|
|
Store: { example_code, claim, project_hash }
|
|
↓
|
|
Same pattern seen in 5+ projects?
|
|
↓
|
|
Flag for promotion to declarative extractor
|
|
```
|
|
|
|
### 7.6.1 LearnedPattern Schema ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `ValueType` enum | ✅ `learning/types.rs` — Text, Number, Boolean |
|
|
| `ClaimTemplate` struct | ✅ `learning/types.rs` — subject_template, predicate, value_type, description |
|
|
| `LearnedPattern` struct | ✅ `learning/types.rs` — full schema with timestamps, project hashes, confidence tracking |
|
|
| Serde serialization | ✅ JSON serialization with chrono timestamps |
|
|
| Tests | ✅ 5 unit tests for types |
|
|
|
|
### 7.6.2 PatternStore Implementation ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `PatternStore` trait | ✅ `learning/store.rs` — abstract storage interface |
|
|
| `LocalPatternStore` | ✅ JSON-backed local storage at `~/.aphoria/learning/patterns.json` |
|
|
| `RwLock` thread safety | ✅ Write-through cache with in-memory HashMap |
|
|
| Deduplication | ✅ `find_similar()` with Levenshtein similarity threshold 0.8 |
|
|
| Pruning | ✅ `prune_stale()` removes patterns not seen in N days |
|
|
| Tests | ✅ 8 unit tests for store operations |
|
|
|
|
### 7.6.3 Pattern Normalization ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `normalize_pattern()` | ✅ `learning/normalizer.rs` — replaces literals with placeholders |
|
|
| Version detection | ✅ `"1.0"`, `"TLSv1.2"` → `<string:version>` |
|
|
| Boolean detection | ✅ `true`/`false` → `<boolean>` |
|
|
| Number detection | ✅ Standalone numbers → `<number>` |
|
|
| String detection | ✅ Remaining quoted strings → `<string>` |
|
|
| `pattern_similarity()` | ✅ Levenshtein distance normalized to 0.0-1.0 |
|
|
| Tests | ✅ 17 unit tests for normalization |
|
|
|
|
### 7.6.4 Configuration ✅
|
|
|
|
```toml
|
|
# aphoria.toml
|
|
[learning]
|
|
enabled = true # Enable pattern learning (default: false)
|
|
store = "local" # "local" | "hosted"
|
|
min_confidence = 0.7 # Minimum LLM confidence to learn
|
|
prune_after_days = 90 # Remove patterns not seen in N days
|
|
|
|
[learning.promotion]
|
|
min_projects = 5 # Projects needed before promotion
|
|
min_confidence = 0.8 # Average confidence needed
|
|
auto_promote = false # Require human approval (Phase 7.7)
|
|
```
|
|
|
|
### 7.6.5 Scan Integration ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| Initialize pattern store | ✅ `scan.rs` — only in persistent mode with learning enabled |
|
|
| Project hash computation | ✅ BLAKE3 hash for privacy-preserving project identification |
|
|
| Record LLM-extracted claims | ✅ After LLM extraction, record patterns meeting min_confidence |
|
|
| Update existing patterns | ✅ Merge observations when similar pattern found |
|
|
| Logging | ✅ Reports patterns_recorded count on scan completion |
|
|
|
|
### 7.6.6 Error Handling ✅
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `LearningStore` error variant | ✅ `error.rs` — for storage/cache failures |
|
|
| Graceful degradation | ✅ Store failures logged, don't block scan |
|
|
|
|
**Files:** `learning/mod.rs`, `learning/types.rs`, `learning/normalizer.rs`, `learning/store.rs`, `config/mod.rs`, `scan.rs`, `error.rs`, `lib.rs`
|
|
|
|
**Tests:** 30 tests covering types, normalization, and store operations.
|
|
|
|
---
|
|
|
|
## Phase 7.6 (Legacy Documentation)
|
|
|
|
> **Note:** The following is the original spec for reference. See above for implemented status.
|
|
|
|
### Original Schema (Reference)
|
|
|
|
```rust
|
|
/// A pattern learned from LLM extraction that could become a declarative extractor.
|
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
|
pub struct LearnedPattern {
|
|
/// Unique identifier
|
|
pub id: Uuid,
|
|
|
|
/// Example code that triggered this pattern
|
|
pub example_code: String,
|
|
|
|
/// Normalized pattern (variables replaced with placeholders)
|
|
/// e.g., "const TLS_MIN_VERSION = \"1.0\"" → "const TLS_MIN_VERSION = <version>"
|
|
pub normalized_pattern: String,
|
|
|
|
/// The claim this pattern produces
|
|
pub claim_template: ClaimTemplate,
|
|
|
|
/// Language this pattern applies to
|
|
pub language: Language,
|
|
|
|
/// When first seen
|
|
pub first_seen: DateTime<Utc>,
|
|
|
|
/// When last seen
|
|
pub last_seen: DateTime<Utc>,
|
|
|
|
/// Projects that have this pattern (hashed for privacy)
|
|
pub project_hashes: HashSet<String>,
|
|
|
|
/// Total occurrences across all projects
|
|
pub occurrences: u32,
|
|
|
|
/// Average LLM confidence when extracting this
|
|
pub avg_confidence: f32,
|
|
|
|
/// Has this been promoted to a declarative extractor?
|
|
pub promoted: bool,
|
|
|
|
/// If promoted, the extractor ID
|
|
pub promoted_to: Option<String>,
|
|
}
|
|
|
|
/// Template for generating claims from a learned pattern.
|
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
|
pub struct ClaimTemplate {
|
|
pub subject_template: String, // "tls/min_version"
|
|
pub predicate: String, // "version"
|
|
pub value_type: ValueType, // String, Boolean, Number
|
|
pub description_template: String,
|
|
}
|
|
```
|
|
|
|
### Original PatternStore Trait (Reference)
|
|
|
|
```rust
|
|
pub trait PatternStore: Send + Sync {
|
|
/// Record a pattern learned from LLM extraction
|
|
fn record_pattern(&self, pattern: &LearnedPattern) -> Result<()>;
|
|
|
|
/// Find existing pattern matching this example
|
|
fn find_similar(&self, normalized: &str, language: Language, threshold: f32) -> Option<LearnedPattern>;
|
|
|
|
/// Get patterns ready for promotion (threshold met)
|
|
fn get_promotion_candidates(&self, min_projects: usize, min_confidence: f32) -> Vec<LearnedPattern>;
|
|
|
|
/// Mark pattern as promoted
|
|
fn mark_promoted(&self, id: &Uuid, extractor_name: &str) -> Result<()>;
|
|
|
|
/// Prune old patterns
|
|
async fn prune_stale(&self, max_age_days: u32) -> Result<usize>;
|
|
}
|
|
```
|
|
|
|
### 7.6.3 Pattern Normalization ⬜
|
|
|
|
| Task | Description |
|
|
|------|-------------|
|
|
| Variable extraction | Identify literals that vary (versions, names, values) |
|
|
| Placeholder insertion | Replace literals with typed placeholders |
|
|
| Similarity scoring | Compare normalized patterns for dedup |
|
|
|
|
```rust
|
|
fn normalize_pattern(code: &str, claim: &ExtractedClaim) -> String {
|
|
// "const TLS_MIN = \"1.0\"" → "const TLS_MIN = <string:version>"
|
|
// "pool_size: 25" → "pool_size: <number>"
|
|
// "verify_ssl: false" → "verify_ssl: <boolean>"
|
|
}
|
|
|
|
fn similarity_score(a: &str, b: &str) -> f32 {
|
|
// Levenshtein distance normalized to 0.0-1.0
|
|
// Patterns with score > 0.8 are considered duplicates
|
|
}
|
|
```
|
|
|
|
### 7.6.4 Integration with Scan ⬜
|
|
|
|
```rust
|
|
// In scan.rs, after LLM extraction
|
|
for claim in llm_claims {
|
|
// Check if this is a new pattern
|
|
if let Some(existing) = pattern_store.find_similar(&claim.matched_text, language).await {
|
|
// Update existing pattern
|
|
pattern_store.increment_occurrence(&existing.id, project_hash).await?;
|
|
} else {
|
|
// Record new pattern
|
|
let pattern = LearnedPattern::from_claim(&claim, &code_context, project_hash);
|
|
pattern_store.record_pattern(&pattern).await?;
|
|
}
|
|
}
|
|
```
|
|
|
|
### 7.6.5 Configuration ⬜
|
|
|
|
```toml
|
|
# aphoria.toml
|
|
[learning]
|
|
enabled = true # Enable pattern learning
|
|
store = "local" # "local" | "hosted"
|
|
min_confidence = 0.7 # Minimum LLM confidence to learn
|
|
prune_after_days = 90 # Remove patterns not seen in N days
|
|
|
|
[learning.promotion]
|
|
min_projects = 5 # Projects needed before promotion
|
|
min_confidence = 0.8 # Average confidence needed
|
|
auto_promote = false # Require human approval (Phase 7.7)
|
|
```
|
|
|
|
**Files:** `learning/mod.rs`, `learning/pattern.rs`, `learning/store.rs`, `learning/normalize.rs`
|
|
|
|
---
|
|
|
|
## Phase 7.7: Pattern → Extractor Promotion ⬜
|
|
|
|
> High-frequency learned patterns get promoted to declarative extractors. This closes the learning loop: patterns discovered by LLM become permanent, fast regex extractors.
|
|
|
|
### Vision
|
|
|
|
```
|
|
LearnedPattern (5+ projects, >0.8 confidence)
|
|
↓
|
|
Claude: "Generate regex for this pattern"
|
|
↓
|
|
Candidate declarative extractor
|
|
↓
|
|
Validate against stored examples
|
|
↓
|
|
Human review (optional) → Approve/Reject
|
|
↓
|
|
Merge to project's .aphoria/extractors/
|
|
```
|
|
|
|
### 7.7.1 Promotion Pipeline ⬜
|
|
|
|
| Task | Description |
|
|
|------|-------------|
|
|
| Candidate selection | Query patterns meeting threshold |
|
|
| Regex generation | LLM generates regex from examples |
|
|
| YAML generation | Convert to declarative extractor format |
|
|
| Validation | Test against all stored examples |
|
|
| Review queue | Present candidates for human approval |
|
|
|
|
```rust
|
|
pub struct PromotionPipeline {
|
|
pattern_store: Arc<dyn PatternStore>,
|
|
llm_client: ClaudeClient,
|
|
validator: ExtractorValidator,
|
|
}
|
|
|
|
impl PromotionPipeline {
|
|
/// Get patterns ready for promotion
|
|
pub async fn get_candidates(&self) -> Vec<PromotionCandidate> {
|
|
let patterns = self.pattern_store
|
|
.get_promotion_candidates(5, 0.8)
|
|
.await?;
|
|
|
|
patterns.into_iter()
|
|
.map(|p| self.generate_candidate(p))
|
|
.collect()
|
|
}
|
|
|
|
/// Generate declarative extractor from pattern
|
|
async fn generate_candidate(&self, pattern: LearnedPattern) -> PromotionCandidate {
|
|
// Ask Claude to generate regex
|
|
let regex = self.llm_client.generate_regex(&pattern).await?;
|
|
|
|
// Build declarative extractor
|
|
let extractor = DeclarativeExtractor {
|
|
name: pattern.id.to_string(),
|
|
language: pattern.language,
|
|
pattern: regex,
|
|
claim: pattern.claim_template.clone(),
|
|
source: ExtractorSource::Learned {
|
|
pattern_id: pattern.id,
|
|
projects: pattern.project_hashes.len(),
|
|
},
|
|
};
|
|
|
|
// Validate against examples
|
|
let validation = self.validator.validate(&extractor, &pattern).await;
|
|
|
|
PromotionCandidate { pattern, extractor, validation }
|
|
}
|
|
}
|
|
```
|
|
|
|
### 7.7.2 Regex Generation ⬜
|
|
|
|
| Task | Description |
|
|
|------|-------------|
|
|
| Multi-example prompt | Include all examples in generation prompt |
|
|
| Regex safety | Prevent catastrophic backtracking |
|
|
| Test coverage | Generate test cases alongside regex |
|
|
|
|
```rust
|
|
async fn generate_regex(examples: &[String], claim: &ClaimTemplate) -> Result<String> {
|
|
let prompt = format!(
|
|
"Generate a regex pattern that matches all these code examples:\n\n{}\n\n\
|
|
The regex should extract the value for claim: {}\n\
|
|
Requirements:\n\
|
|
- Must match ALL examples\n\
|
|
- Use named capture groups for extracted values\n\
|
|
- Avoid catastrophic backtracking (no nested quantifiers)\n\
|
|
- Return ONLY the regex, no explanation",
|
|
examples.join("\n---\n"),
|
|
claim.subject_template
|
|
);
|
|
|
|
let response = claude.message(&prompt).await?;
|
|
validate_regex_safety(&response)?;
|
|
Ok(response)
|
|
}
|
|
```
|
|
|
|
### 7.7.3 Validation Suite ⬜
|
|
|
|
| Task | Description |
|
|
|------|-------------|
|
|
| Positive tests | Must match all stored examples |
|
|
| Negative tests | Must NOT match known-safe code |
|
|
| Performance test | Must complete in < 100ms |
|
|
| False positive check | Run against sample codebase |
|
|
|
|
```rust
|
|
pub struct ExtractorValidator {
|
|
sample_codebases: Vec<PathBuf>, // Known-good projects for FP testing
|
|
}
|
|
|
|
impl ExtractorValidator {
|
|
pub async fn validate(
|
|
&self,
|
|
extractor: &DeclarativeExtractor,
|
|
pattern: &LearnedPattern
|
|
) -> ValidationResult {
|
|
let mut result = ValidationResult::default();
|
|
|
|
// Must match all positive examples
|
|
for example in &pattern.examples {
|
|
if !extractor.matches(example) {
|
|
result.positive_failures.push(example.clone());
|
|
}
|
|
}
|
|
|
|
// Must not have excessive false positives
|
|
for codebase in &self.sample_codebases {
|
|
let fps = self.count_false_positives(extractor, codebase).await;
|
|
if fps > 10 {
|
|
result.false_positive_warning = true;
|
|
}
|
|
}
|
|
|
|
// Must be fast
|
|
let duration = self.benchmark(extractor);
|
|
if duration > Duration::from_millis(100) {
|
|
result.performance_warning = true;
|
|
}
|
|
|
|
result
|
|
}
|
|
}
|
|
```
|
|
|
|
### 7.7.4 Human Review Gate ⬜
|
|
|
|
| Task | Description |
|
|
|------|-------------|
|
|
| `aphoria extractors review` | CLI to review pending promotions |
|
|
| Approval workflow | Approve, reject, or request changes |
|
|
| Rejection tracking | Record why patterns were rejected |
|
|
| Auto-approve mode | Skip review for >0.95 confidence (Phase 9) |
|
|
|
|
```bash
|
|
$ aphoria extractors review
|
|
|
|
Pending promotions: 3
|
|
|
|
[1/3] Pattern: tls_min_version_const
|
|
Examples: 47 (across 8 projects)
|
|
Confidence: 0.91
|
|
|
|
Generated regex: (?i)(tls|ssl)_?(min|minimum)_?version\s*[:=]\s*["']?(1\.[01])["']?
|
|
|
|
Sample matches:
|
|
const TLS_MIN_VERSION = "1.0" ✓ matches
|
|
TLS_MINIMUM_VERSION: "1.1" ✓ matches
|
|
ssl_min_version = "1.2" ✓ matches (TLS 1.2 is safe, false positive?)
|
|
|
|
[a]pprove [r]eject [e]dit [s]kip [q]uit: _
|
|
```
|
|
|
|
### 7.7.5 Extractor Output ⬜
|
|
|
|
Promoted patterns become declarative extractors in `.aphoria/extractors/`:
|
|
|
|
```yaml
|
|
# .aphoria/extractors/learned/tls_min_version_const.yaml
|
|
# Auto-generated from learned pattern. DO NOT EDIT.
|
|
# Pattern ID: 550e8400-e29b-41d4-a716-446655440000
|
|
# Learned from: 8 projects, 47 occurrences
|
|
# Confidence: 0.91
|
|
# Promoted: 2026-02-10
|
|
|
|
name: "tls_min_version_const"
|
|
language: ["rust", "go", "python", "javascript", "typescript"]
|
|
pattern: '(?i)(tls|ssl)_?(min|minimum)_?version\s*[:=]\s*["\']?(1\.[01])["\']?'
|
|
claim:
|
|
subject: "tls/min_version"
|
|
predicate: "version"
|
|
value_capture: 1 # Capture group for version
|
|
description: "TLS minimum version set to deprecated {value}"
|
|
metadata:
|
|
source: "learned"
|
|
pattern_id: "550e8400-e29b-41d4-a716-446655440000"
|
|
projects: 8
|
|
occurrences: 47
|
|
confidence: 0.91
|
|
```
|
|
|
|
### 7.7.6 Configuration ⬜
|
|
|
|
```toml
|
|
# aphoria.toml
|
|
[promotion]
|
|
enabled = true # Enable promotion pipeline
|
|
auto_promote = false # Require human approval
|
|
output_dir = ".aphoria/extractors/learned"
|
|
min_confidence = 0.8 # Minimum to consider
|
|
require_validation = true # Must pass validation suite
|
|
|
|
[promotion.review]
|
|
notify = "slack://webhook/..." # Notify when candidates ready
|
|
batch_size = 10 # Max candidates per review session
|
|
```
|
|
|
|
**Files:** `promotion/mod.rs`, `promotion/pipeline.rs`, `promotion/regex_gen.rs`, `promotion/validator.rs`, `promotion/review.rs`
|
|
|
|
---
|
|
|
|
## Phase 9: Autonomous Extractor Generation ⬜
|
|
|
|
> The system generates, tests, and deploys extractors without human approval for high-confidence patterns. This is the endgame: a fully self-improving extraction system.
|
|
|
|
### Vision
|
|
|
|
```
|
|
Learned pattern exceeds autonomous threshold (>0.95 confidence, >10 projects)
|
|
↓
|
|
Auto-generate extractor
|
|
↓
|
|
Validate against comprehensive test suite
|
|
↓
|
|
A/B test: run new extractor in shadow mode
|
|
↓
|
|
If FP rate < 5%: auto-deploy
|
|
↓
|
|
If FP rate spikes: auto-rollback
|
|
```
|
|
|
|
### 9.1 Autonomous Promotion ⬜
|
|
|
|
| Task | Description |
|
|
|------|-------------|
|
|
| High-confidence threshold | Skip human review for >0.95 confidence |
|
|
| Project threshold | Require >10 projects for autonomous |
|
|
| Validation strictness | Stricter validation for autonomous |
|
|
|
|
```rust
|
|
fn should_auto_promote(pattern: &LearnedPattern, validation: &ValidationResult) -> bool {
|
|
pattern.avg_confidence > 0.95 &&
|
|
pattern.project_hashes.len() > 10 &&
|
|
validation.positive_failures.is_empty() &&
|
|
!validation.false_positive_warning &&
|
|
!validation.performance_warning
|
|
}
|
|
```
|
|
|
|
### 9.2 Shadow Mode Testing ⬜
|
|
|
|
| Task | Description |
|
|
|------|-------------|
|
|
| Shadow execution | Run new extractor alongside existing |
|
|
| Metrics collection | Track matches, FP rate, performance |
|
|
| Comparison report | Compare shadow vs production results |
|
|
| Promotion criteria | Promote if metrics meet threshold |
|
|
|
|
```rust
|
|
pub struct ShadowTest {
|
|
extractor: DeclarativeExtractor,
|
|
start_time: DateTime<Utc>,
|
|
scans_completed: u32,
|
|
matches: u32,
|
|
confirmed_true_positives: u32,
|
|
confirmed_false_positives: u32,
|
|
}
|
|
|
|
impl ShadowTest {
|
|
fn false_positive_rate(&self) -> f32 {
|
|
self.confirmed_false_positives as f32 / self.matches as f32
|
|
}
|
|
|
|
fn should_promote(&self) -> bool {
|
|
self.scans_completed >= 100 &&
|
|
self.false_positive_rate() < 0.05
|
|
}
|
|
}
|
|
```
|
|
|
|
### 9.3 Auto-Rollback ⬜
|
|
|
|
| Task | Description |
|
|
|------|-------------|
|
|
| Anomaly detection | Detect FP rate spikes |
|
|
| Rollback trigger | Auto-disable if FP > 10% |
|
|
| Notification | Alert on rollback |
|
|
| Quarantine | Move extractor to review queue |
|
|
|
|
```rust
|
|
async fn check_extractor_health(extractor_id: &str, metrics: &Metrics) -> Action {
|
|
let recent_fp_rate = metrics.false_positive_rate_last_24h(extractor_id);
|
|
let baseline_fp_rate = metrics.false_positive_rate_baseline(extractor_id);
|
|
|
|
if recent_fp_rate > 0.10 {
|
|
Action::Rollback { reason: "FP rate exceeded 10%" }
|
|
} else if recent_fp_rate > baseline_fp_rate * 2.0 {
|
|
Action::Rollback { reason: "FP rate doubled from baseline" }
|
|
} else {
|
|
Action::Continue
|
|
}
|
|
}
|
|
```
|
|
|
|
### 9.4 Cross-Project Learning ⬜
|
|
|
|
| Task | Description |
|
|
|------|-------------|
|
|
| Hosted pattern sync | Patterns from all projects aggregate on server |
|
|
| Global promotion | Promote patterns seen across many orgs |
|
|
| Privacy preservation | Only normalized patterns shared, no code |
|
|
| Opt-in distribution | Orgs can opt-in to receive community extractors |
|
|
|
|
```
|
|
Org A: Pattern seen in 3 projects → shared to hosted
|
|
Org B: Same pattern in 5 projects → shared to hosted
|
|
Org C: Same pattern in 4 projects → shared to hosted
|
|
↓
|
|
Hosted aggregates: 12 projects total
|
|
↓
|
|
Promotes to community extractor
|
|
↓
|
|
All orgs receive new extractor (if opted in)
|
|
```
|
|
|
|
### 9.5 Extractor Versioning ⬜
|
|
|
|
| Task | Description |
|
|
|------|-------------|
|
|
| Version tracking | Track which version caught which issues |
|
|
| Changelog | Record changes between versions |
|
|
| Rollback support | Revert to previous version |
|
|
| A/B metrics | Compare versions side-by-side |
|
|
|
|
```yaml
|
|
# .aphoria/extractors/learned/tls_min_version_const.yaml
|
|
version: 2
|
|
previous_version: 1
|
|
changelog:
|
|
- version: 2
|
|
date: 2026-03-15
|
|
changes: "Added support for YAML configs"
|
|
metrics:
|
|
matches: +15%
|
|
false_positives: -3%
|
|
- version: 1
|
|
date: 2026-02-10
|
|
changes: "Initial auto-generated version"
|
|
```
|
|
|
|
### 9.6 Configuration ⬜
|
|
|
|
```toml
|
|
# aphoria.toml
|
|
[autonomous]
|
|
enabled = false # Opt-in to autonomous mode
|
|
min_confidence = 0.95 # Higher threshold for auto
|
|
min_projects = 10 # More evidence required
|
|
shadow_scans = 100 # Scans before promotion
|
|
max_fp_rate = 0.05 # Auto-rollback threshold
|
|
|
|
[autonomous.distribution]
|
|
receive_community = true # Receive community extractors
|
|
contribute_patterns = true # Share patterns to community
|
|
```
|
|
|
|
**Files:** `autonomous/mod.rs`, `autonomous/shadow.rs`, `autonomous/rollback.rs`, `autonomous/distribution.rs`
|
|
|
|
---
|
|
|
|
## Milestone Summary
|
|
|
|
| Phase | Deliverable | Depends On | Status |
|
|
|-------|-------------|------------|--------|
|
|
| 0 | ConceptPath in StemeDB | concept-hierarchy spec | ✅ |
|
|
| 2 | Aphoria CLI (scan, report, ack) | Phase 0 | ✅ |
|
|
| 2A | Concept matching (leaf, alias, auto-alias) | Phase 2 | ✅ |
|
|
| 1 | Authoritative corpus expansion | Phase 0 | ✅ |
|
|
| 3 | Claude Code skill + hooks | Phase 2A | ✅ |
|
|
| 4.5 | Ephemeral scan mode (40x faster) | Phase 2 | ✅ |
|
|
| 5 | Research agent loop | Phase 3 | ✅ |
|
|
| 6 | Federated Policy & Trust Packs | Phase 4.5 | ✅ |
|
|
| **6.5** | **Trust Pack Extensions (Predicate Aliases, Key Rotation)** | Phase 6 | ⬜ |
|
|
| 4A | Observational claims (Tier 4 write-back) | Phase 6 | ✅ |
|
|
| 4B | Self-conflict detection (drift) | Phase 4A | ✅ |
|
|
| 4C | Diff-only scanning (--staged) | Phase 4B | ✅ |
|
|
| 4E | Hosted mode (team aggregation) | Phase 4C | ✅ |
|
|
| 4D | Enhanced ack (--reason, policy updates) | Phase 4C | ✅ |
|
|
| 5.6 | Community Corpus Contributions | Phase 4E | ✅ |
|
|
| 7 | Declarative Extractors | Phase 6 | ✅ |
|
|
| **7.5** | **LLM-in-the-Loop Extraction (Gemini)** | Phase 7 | ✅ |
|
|
| **7.6** | **Pattern Learning Store** | Phase 7.5 | ✅ |
|
|
| **7.7** | **Pattern → Extractor Promotion** | Phase 7.6 | ⬜ |
|
|
| 8 | Enterprise Extractors (MVP: 8.1, 8.6, 8.11) | Phase 7.5 | ✅ |
|
|
| **9** | **Autonomous Extractor Generation** | Phase 8 | ⬜ |
|
|
|
|
**Current state:**
|
|
- Phases 0-3, 4.5, 4A-4E, 5, 5.6, 6, 7, 7.5, 7.6, 8 (MVP) complete (clippy clean)
|
|
- Full corpus: RFC, OWASP, Vendor sources
|
|
- 17 extractors including security (weak_crypto, command_injection, sql_injection, high_entropy_secrets, auth_bypass, insecure_cookies)
|
|
- Trust Packs: signed policy bundles with import/export
|
|
- Ephemeral mode: 40x faster for CI
|
|
- Observation write-back: `--sync` records novel claims as Tier 4 project memory
|
|
- Drift detection: Detects changes from prior observations
|
|
- Staged scanning: `--staged` flag for fast pre-commit hooks
|
|
- Hosted mode: Team aggregation via central StemeDB server
|
|
- Enhanced ack: `--reason` flag, `aphoria update` for policy changes
|
|
- Community Corpus: Opt-in anonymous pattern sharing with privacy-preserving anonymization
|
|
- Declarative Extractors: TOML-defined custom extractors without Rust code
|
|
- LLM Extraction: Gemini-powered semantic claim extraction for high-value files
|
|
- Enterprise Extractors MVP: High-entropy secrets (Shannon entropy), auth bypass patterns, insecure cookie flags
|
|
- Pattern Learning: LLM-extracted claims recorded for promotion to declarative extractors
|
|
|
|
**Next:** Phase 7.7 → 8 (full) → 9 (Self-Learning Extraction System)
|
|
|
|
### The Self-Learning Vision
|
|
|
|
```
|
|
Phase 7: Declarative Extractors (foundation) ✅ COMPLETE
|
|
↓
|
|
Phase 7.5: LLM-in-the-Loop (Gemini semantic extraction) ✅ COMPLETE
|
|
↓
|
|
Phase 7.6: Pattern Learning (remember what LLM finds) ✅ COMPLETE
|
|
↓
|
|
Phase 7.7: Pattern Promotion (patterns → extractors) ⬜ NEXT
|
|
↓
|
|
Phase 8: Enterprise Extractors (generated + curated) ✅ MVP (8.1, 8.6, 8.11)
|
|
↓
|
|
Phase 9: Autonomous Generation (fully self-improving) ⬜
|
|
```
|
|
|
|
**The endgame:** Every PR teaches Aphoria. After a month, it knows your security patterns better than your team does.
|
|
|
|
### Bidirectional Knowledge Sync (Complete)
|
|
|
|
The pre-commit hook is now a bidirectional knowledge sync:
|
|
1. **4A** ✅: Record code claims as Tier 4 observations (project memory)
|
|
2. **4B** ✅: Detect drift from prior observations (self-conflict)
|
|
3. **4C** ✅: Fast diff-only scanning for pre-commit hooks (`--staged`)
|
|
4. **4E** ✅: Team aggregation via hosted StemeDB server
|
|
5. **4D** ✅: Enhanced ack with rationale and policy updates
|
|
|
|
This transforms Aphoria from a linter into a learning system that builds institutional memory per-project and collective intelligence across teams via hosted mode.
|
|
|
|
---
|
|
|
|
## Phase 8: Enterprise Extractor Improvements
|
|
|
|
> **Goal:** Transform extractors from "toy examples" to enterprise-grade detection that catches real violations in production codebases.
|
|
|
|
### Current State Audit
|
|
|
|
| Extractor | Languages | Strengths | Weaknesses |
|
|
|-----------|-----------|-----------|------------|
|
|
| `tls_verify` | 8 | Multi-lang, configs | Misses custom wrappers |
|
|
| `tls_version` | 8 | API patterns | Misses semantic (const = "1.0") |
|
|
| `hardcoded_secrets` | 8 | Placeholders, test files | No entropy detection |
|
|
| `weak_crypto` | 5 | MD5/SHA1/DES/RC4 | SHA1 false positives, misses bcrypt cost |
|
|
| `sql_injection` | 5 | Interpolation patterns | Misses ORM unsafe methods |
|
|
| `jwt_config` | 8 | alg:none, skip sig | Library-specific gaps |
|
|
| `cors_config` | 8 | Wildcard + credentials | Misses dynamic origin reflection |
|
|
| `rate_limit` | 8 | Basic patterns | Limited depth |
|
|
| `timeout_config` | 8 | Basic patterns | Limited depth |
|
|
| `command_injection` | 5 | exec/system calls | Indirect injection |
|
|
| `dep_versions` | 3 | Version parsing | No CVE correlation |
|
|
|
|
**Enterprise Reality:** Current extractors catch ~30% of real-world security misconfigurations. Config files are highest value (patterns consistent), code is lowest (semantic understanding required).
|
|
|
|
---
|
|
|
|
### 8.1 High-Entropy Secret Detection ✅
|
|
|
|
**Impact:** HIGH | **Effort:** MEDIUM | **Status:** Complete
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `HighEntropySecretsExtractor` | ✅ `extractors/high_entropy_secrets.rs` |
|
|
| Shannon entropy algorithm | ✅ `shannon_entropy()` with 4.5 threshold |
|
|
| Charset variety check | ✅ 0.4 minimum variety ratio |
|
|
| Known secret prefixes | ✅ AWS (AKIA), Stripe (sk_live_, sk_test_), GitHub (ghp_, gho_), GitLab (glpat-), Slack (xox[baprs]-) |
|
|
| High-entropy context patterns | ✅ api_key, secret, token, credential, auth_key contexts |
|
|
| False positive exclusions | ✅ UUIDs, git SHAs (40-char hex), file hashes (64-char hex) |
|
|
| Test file confidence reduction | ✅ 0.6 confidence for test files |
|
|
| Tests | ✅ 10+ tests covering all patterns |
|
|
|
|
**Configuration:**
|
|
```toml
|
|
# aphoria.toml
|
|
[extractors.entropy]
|
|
min_entropy = 4.5 # Shannon entropy threshold
|
|
min_charset_variety = 0.4 # Unique chars / length ratio
|
|
min_length = 20 # Minimum string length
|
|
max_length = 200 # Maximum string length
|
|
```
|
|
|
|
**Languages:** Rust, Go, Python, JavaScript, TypeScript, YAML, TOML, JSON, Dotenv
|
|
|
|
---
|
|
|
|
### 8.2 Framework-Specific Extractors ⬜
|
|
|
|
**Impact:** HIGH | **Effort:** HIGH
|
|
|
|
Generic patterns miss framework-specific misconfigurations. Enterprise codebases use frameworks.
|
|
|
|
#### 8.2.1 Spring Boot Security
|
|
```yaml
|
|
# application.yml misconfigs
|
|
security:
|
|
basic:
|
|
enabled: false # Auth disabled
|
|
csrf:
|
|
enabled: false # CSRF disabled
|
|
headers:
|
|
frame-options: DISABLE # Clickjacking
|
|
```
|
|
|
|
```java
|
|
// Java code patterns
|
|
@EnableWebSecurity
|
|
public class Config extends WebSecurityConfigurerAdapter {
|
|
http.csrf().disable(); // CSRF disabled
|
|
http.authorizeRequests().antMatchers("/**").permitAll(); // Auth bypass
|
|
}
|
|
```
|
|
|
|
#### 8.2.2 Django Security
|
|
```python
|
|
# settings.py misconfigs
|
|
DEBUG = True # Debug in production
|
|
ALLOWED_HOSTS = ['*'] # All hosts
|
|
CSRF_COOKIE_SECURE = False # Insecure cookies
|
|
SESSION_COOKIE_SECURE = False
|
|
```
|
|
|
|
#### 8.2.3 Express.js Security
|
|
```javascript
|
|
// Missing security middleware
|
|
app.use(helmet()); // helmet() should exist
|
|
app.use(cors({ origin: '*', credentials: true })); // CORS + creds
|
|
app.disable('x-powered-by'); // Should be disabled
|
|
```
|
|
|
|
#### 8.2.4 Rails Security
|
|
```ruby
|
|
# config/environments/production.rb
|
|
config.force_ssl = false # Should be true
|
|
config.action_dispatch.cookies_same_site_protection = :none
|
|
```
|
|
|
|
---
|
|
|
|
### 8.3 Config File Deep Parsing ⬜
|
|
|
|
**Impact:** HIGH | **Effort:** MEDIUM
|
|
|
|
Current extractors use regex on config files. This misses:
|
|
- Nested structures
|
|
- Environment-specific overrides
|
|
- Comments that disable security
|
|
|
|
**Implementation:**
|
|
```rust
|
|
// Parse YAML/JSON/TOML into structured form
|
|
enum ConfigValue {
|
|
String(String),
|
|
Number(f64),
|
|
Bool(bool),
|
|
Array(Vec<ConfigValue>),
|
|
Object(HashMap<String, ConfigValue>),
|
|
}
|
|
|
|
// Then extract with path awareness
|
|
fn extract_config_claims(config: &ConfigValue, path: &[String]) -> Vec<ExtractedClaim> {
|
|
// Recursively walk structure
|
|
// Track full path: "server.tls.min_version"
|
|
// Apply semantic rules based on path
|
|
}
|
|
```
|
|
|
|
**Patterns to catch:**
|
|
- `tls.verify: false` anywhere in hierarchy
|
|
- `security.enabled: false` in production configs
|
|
- `debug: true` or `DEBUG: true` in non-dev files
|
|
|
|
---
|
|
|
|
### 8.4 Semantic TLS Version Detection ⬜
|
|
|
|
**Impact:** MEDIUM | **Effort:** MEDIUM
|
|
|
|
Current `tls_version` misses:
|
|
```rust
|
|
const TLS_MIN_VERSION: &str = "1.0"; // Not caught!
|
|
const MIN_TLS: &str = "TLSv1"; // Not caught!
|
|
```
|
|
|
|
**Implementation:**
|
|
```rust
|
|
// Semantic pattern: variable name suggests TLS + value is deprecated
|
|
let semantic_tls = Regex::new(
|
|
r#"(?i)(tls|ssl)_?(min|minimum|version)[^=]*[:=]\s*["']?(1\.[01]|TLSv?1(?:\.[01])?|SSL)"#
|
|
).unwrap();
|
|
```
|
|
|
|
**Also catch:**
|
|
- Environment variables: `TLS_MIN_VERSION=1.0`
|
|
- Terraform: `min_tls_version = "TLS1_0"`
|
|
- Kubernetes: `minTLSVersion: VersionTLS10`
|
|
|
|
---
|
|
|
|
### 8.5 ORM SQL Injection Detection ⬜
|
|
|
|
**Impact:** MEDIUM | **Effort:** MEDIUM
|
|
|
|
Current `sql_injection` catches raw string interpolation but misses ORM escape hatches:
|
|
|
|
```python
|
|
# SQLAlchemy
|
|
db.execute(text(f"SELECT * FROM users WHERE id = {user_id}"))
|
|
User.query.filter(text("name = '" + name + "'"))
|
|
|
|
# Django
|
|
User.objects.raw("SELECT * FROM users WHERE id = %s" % user_id)
|
|
User.objects.extra(where=["name = '%s'" % name])
|
|
```
|
|
|
|
```javascript
|
|
// Sequelize
|
|
sequelize.query(`SELECT * FROM users WHERE id = ${userId}`);
|
|
Model.findAll({ where: sequelize.literal(`id = ${id}`) });
|
|
|
|
// Prisma
|
|
prisma.$queryRawUnsafe(`SELECT * FROM users WHERE id = ${id}`);
|
|
```
|
|
|
|
```ruby
|
|
# ActiveRecord
|
|
User.where("name = '#{name}'")
|
|
User.find_by_sql("SELECT * FROM users WHERE id = #{id}")
|
|
```
|
|
|
|
---
|
|
|
|
### 8.6 Authentication Bypass Patterns ✅
|
|
|
|
**Impact:** HIGH | **Effort:** MEDIUM | **Status:** Complete
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `AuthBypassExtractor` | ✅ `extractors/auth_bypass.rs` |
|
|
| Hardcoded admin credentials | ✅ `username == "admin" && password == "..."` patterns |
|
|
| Debug auth headers | ✅ X-Debug-Auth, X-Internal-Auth, X-Admin-Auth |
|
|
| Skip auth env vars | ✅ SKIP_AUTH, BYPASS_AUTH, NO_AUTH, DEBUG_AUTH |
|
|
| Backdoor patterns | ✅ `if username == "backdoor"`, `if user == "test"` |
|
|
| Default credentials | ✅ admin/admin, root/root, test/test, guest/guest |
|
|
| Test file confidence reduction | ✅ 0.5 confidence for test files |
|
|
| Tests | ✅ 11+ tests covering all patterns |
|
|
|
|
**Detected patterns:**
|
|
```python
|
|
# Hardcoded credentials
|
|
if username == "admin" and password == "admin":
|
|
|
|
# Debug auth headers
|
|
if request.headers.get("X-Debug-Auth") == "secret":
|
|
|
|
# Skip auth env vars
|
|
if os.environ.get("SKIP_AUTH") == "true":
|
|
```
|
|
|
|
**Languages:** Python, JavaScript, TypeScript, Go, Rust
|
|
|
|
---
|
|
|
|
### 8.7 Insecure Deserialization ⬜
|
|
|
|
**Impact:** HIGH | **Effort:** MEDIUM
|
|
|
|
Unsafe deserialization of untrusted data:
|
|
|
|
```python
|
|
# Python
|
|
pickle.loads(user_input)
|
|
yaml.load(user_input) # Without Loader=SafeLoader
|
|
eval(user_input)
|
|
exec(user_input)
|
|
```
|
|
|
|
```java
|
|
// Java
|
|
ObjectInputStream ois = new ObjectInputStream(userInput);
|
|
ois.readObject(); // Dangerous!
|
|
```
|
|
|
|
```ruby
|
|
# Ruby
|
|
Marshal.load(user_input)
|
|
YAML.load(user_input) # Should use safe_load
|
|
```
|
|
|
|
---
|
|
|
|
### 8.8 Path Traversal Patterns ⬜
|
|
|
|
**Impact:** MEDIUM | **Effort:** LOW
|
|
|
|
File operations with user input:
|
|
|
|
```python
|
|
# Python
|
|
open(user_input)
|
|
os.path.join(base, user_input) # Doesn't prevent ../
|
|
shutil.copy(user_input, dest)
|
|
```
|
|
|
|
```javascript
|
|
// JavaScript
|
|
fs.readFile(userInput)
|
|
path.join(base, userInput) // Doesn't prevent ../
|
|
res.sendFile(userInput)
|
|
```
|
|
|
|
---
|
|
|
|
### 8.9 SSRF Patterns ⬜
|
|
|
|
**Impact:** HIGH | **Effort:** MEDIUM
|
|
|
|
HTTP requests with user-controlled URLs:
|
|
|
|
```python
|
|
# Python
|
|
requests.get(user_url)
|
|
urllib.request.urlopen(user_input)
|
|
```
|
|
|
|
```javascript
|
|
// JavaScript
|
|
fetch(userUrl)
|
|
axios.get(userUrl)
|
|
http.get(userUrl)
|
|
```
|
|
|
|
```go
|
|
// Go
|
|
http.Get(userURL)
|
|
client.Do(req) // Where req.URL is user-controlled
|
|
```
|
|
|
|
---
|
|
|
|
### 8.10 Missing Security Headers ⬜
|
|
|
|
**Impact:** MEDIUM | **Effort:** LOW
|
|
|
|
Detect when security headers are explicitly removed or not set:
|
|
|
|
```python
|
|
# Response headers missing
|
|
response.headers.pop('X-Content-Type-Options')
|
|
response.headers['X-Frame-Options'] = 'ALLOWALL'
|
|
```
|
|
|
|
```javascript
|
|
// Express without helmet
|
|
app.use(cors()); // CORS without other security
|
|
// No app.use(helmet()) found
|
|
```
|
|
|
|
---
|
|
|
|
### 8.11 Insecure Cookie Flags ✅
|
|
|
|
**Impact:** MEDIUM | **Effort:** LOW | **Status:** Complete
|
|
|
|
| Task | Status |
|
|
|------|--------|
|
|
| `InsecureCookiesExtractor` | ✅ `extractors/insecure_cookies.rs` |
|
|
| Missing Secure flag | ✅ `secure=False`, `secure: false` |
|
|
| Missing HttpOnly flag | ✅ `httponly=False`, `httpOnly: false` |
|
|
| SameSite=None without Secure | ✅ `sameSite: 'none'`, `SameSite=None` |
|
|
| Django settings | ✅ SESSION_COOKIE_SECURE, CSRF_COOKIE_SECURE = False |
|
|
| Go cookie patterns | ✅ `Secure: false`, `HttpOnly: false` |
|
|
| Rust actix-web patterns | ✅ `.secure(false)`, `.http_only(false)` |
|
|
| Test file confidence reduction | ✅ 0.5 confidence for test files |
|
|
| Tests | ✅ 8+ tests covering all patterns |
|
|
|
|
**Detected patterns:**
|
|
```python
|
|
# Python/Flask/Django
|
|
response.set_cookie('session', value, secure=False)
|
|
SESSION_COOKIE_SECURE = False
|
|
```
|
|
|
|
```javascript
|
|
// JavaScript/Express
|
|
res.cookie('session', value, { httpOnly: false });
|
|
res.cookie('auth', value, { sameSite: 'none' });
|
|
```
|
|
|
|
**Languages:** Python, JavaScript, TypeScript, Go, Rust, Ruby, YAML
|
|
|
|
---
|
|
|
|
### 8.12 Unvalidated Redirects ⬜
|
|
|
|
**Impact:** MEDIUM | **Effort:** LOW
|
|
|
|
Open redirect vulnerabilities:
|
|
|
|
```python
|
|
# Python
|
|
return redirect(request.args.get('next'))
|
|
return redirect(request.GET['url'])
|
|
```
|
|
|
|
```javascript
|
|
// JavaScript
|
|
res.redirect(req.query.redirect);
|
|
window.location = userInput;
|
|
window.location.href = params.url;
|
|
```
|
|
|
|
---
|
|
|
|
### 8.13 XXE (XML External Entity) ⬜
|
|
|
|
**Impact:** HIGH | **Effort:** MEDIUM
|
|
|
|
Unsafe XML parsing:
|
|
|
|
```python
|
|
# Python
|
|
etree.parse(user_input) # Without disabling entities
|
|
xml.etree.ElementTree.parse(user_input)
|
|
```
|
|
|
|
```java
|
|
// Java
|
|
DocumentBuilderFactory.newInstance() // Without setFeature to disable XXE
|
|
SAXParserFactory.newInstance() // Without secure processing
|
|
```
|
|
|
|
---
|
|
|
|
### 8.14 Weak Password Requirements ⬜
|
|
|
|
**Impact:** MEDIUM | **Effort:** LOW
|
|
|
|
Password validation that's too weak:
|
|
|
|
```python
|
|
# Python
|
|
if len(password) >= 4: # Too short
|
|
if len(password) >= 6: # Still weak
|
|
MIN_PASSWORD_LENGTH = 6 # Config too low
|
|
```
|
|
|
|
```javascript
|
|
// JavaScript
|
|
if (password.length >= 4)
|
|
const MIN_LENGTH = 6;
|
|
/^.{4,}$/ // Regex allows 4+ chars
|
|
```
|
|
|
|
---
|
|
|
|
### 8.15 LLM-Assisted Extraction (Future) ⬜
|
|
|
|
**Impact:** VERY HIGH | **Effort:** VERY HIGH
|
|
|
|
Use Claude to understand code semantically:
|
|
|
|
```rust
|
|
// Pseudo-implementation
|
|
async fn extract_with_llm(code: &str, file: &str) -> Vec<ExtractedClaim> {
|
|
let prompt = format!(
|
|
"Analyze this code for security issues. Return JSON with:\n\
|
|
- concept_path: security concept (e.g., 'tls/cert_verification')\n\
|
|
- predicate: what aspect (e.g., 'enabled')\n\
|
|
- value: the value found\n\
|
|
- confidence: 0.0-1.0\n\
|
|
- description: why this is an issue\n\n\
|
|
Code:\n```\n{}\n```",
|
|
code
|
|
);
|
|
|
|
let response = claude_api.message(&prompt).await?;
|
|
parse_claims_from_llm_response(&response)
|
|
}
|
|
```
|
|
|
|
**When to use:**
|
|
- High-value files (auth, crypto, config)
|
|
- After regex extractors find nothing
|
|
- For code review mode (not CI)
|
|
|
|
**Considerations:**
|
|
- Cost per scan
|
|
- Latency
|
|
- Rate limits
|
|
- Privacy (code leaves machine)
|
|
|
|
---
|
|
|
|
### Implementation Priority
|
|
|
|
| Phase | Extractors | Impact | Effort | Enterprise Value | Status |
|
|
|-------|------------|--------|--------|------------------|--------|
|
|
| **8.1** | High-entropy secrets | HIGH | MEDIUM | Catches real leaked secrets | ✅ |
|
|
| **8.2** | Framework-specific | HIGH | HIGH | Spring/Django/Express coverage | ⬜ |
|
|
| **8.3** | Config deep parsing | HIGH | MEDIUM | Nested YAML/JSON understanding | ⬜ |
|
|
| **8.4** | Semantic TLS | MEDIUM | MEDIUM | Catches const TLS_MIN = "1.0" | ⬜ |
|
|
| **8.5** | ORM SQL injection | MEDIUM | MEDIUM | SQLAlchemy, Django, Sequelize | ⬜ |
|
|
| **8.6** | Auth bypass | HIGH | MEDIUM | Backdoors, hardcoded creds | ✅ |
|
|
| **8.7** | Deserialization | HIGH | MEDIUM | pickle, Marshal, eval | ⬜ |
|
|
| **8.8** | Path traversal | MEDIUM | LOW | ../../../etc/passwd | ⬜ |
|
|
| **8.9** | SSRF | HIGH | MEDIUM | Internal network access | ⬜ |
|
|
| **8.10** | Security headers | MEDIUM | LOW | Missing helmet(), CSP | ⬜ |
|
|
| **8.11** | Cookie flags | MEDIUM | LOW | httpOnly, secure, sameSite | ✅ |
|
|
| **8.12** | Open redirects | MEDIUM | LOW | Phishing via redirect | ⬜ |
|
|
| **8.13** | XXE | HIGH | MEDIUM | XML entity injection | ⬜ |
|
|
| **8.14** | Weak passwords | MEDIUM | LOW | MIN_LENGTH = 4 | ⬜ |
|
|
| **8.15** | LLM extraction | VERY HIGH | VERY HIGH | Semantic understanding | ✅ (Phase 7.5) |
|
|
|
|
**MVP Complete (8.1, 8.6, 8.11):** High-impact extractors for enterprise pilots.
|
|
|
|
**Recommended order for remaining extractors:**
|
|
1. **8.3** Config deep parsing (foundational for 8.2)
|
|
2. **8.2** Framework-specific (customer-driven)
|
|
3. **8.5** ORM SQL injection (common in enterprise apps)
|
|
4. **8.7** Deserialization (critical vulnerabilities)
|
|
|
|
---
|
|
|
|
### Success Metrics
|
|
|
|
| Metric | Current | Target | How to Measure |
|
|
|--------|---------|--------|----------------|
|
|
| Detection rate (known vulns) | ~30% | >70% | Run against OWASP benchmark |
|
|
| False positive rate | Unknown | <10% | Manual review of 100 findings |
|
|
| Config file coverage | Regex only | Full parse | Structure-aware extraction |
|
|
| Framework coverage | 0 | 4 major | Spring, Django, Express, Rails |
|
|
| Enterprise pilot feedback | N/A | >4/5 | Post-pilot survey |
|
|
|