jordan/stemedb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
42 Commits 1 Branch 0 Tags 18 MiB
6430ff0fd6
Commit Graph

14 Commits

Author SHA1 Message Date
jml
3b5f88b4f0 feat(aphoria): implement claims architecture (A1-A5) with verify engine, corpus, coverage, and explain 
Complete Aphoria claims system overhaul:
- A1: Rename ExtractedClaim to Observation (extractors produce observations, not claims)
- A2: Add AuthoredClaim with full provenance, invariants, and authority tiers
- A3: Verify engine comparing observations against authored claims, CLI + formatters
- A4: Corpus as first-class assertions with predicate indexing, authority lens, trust packs
- A5: Coverage analysis, explain/docs generation, self-audit extractor, claim suggester skill

Also includes: 42 extractors updated for Observation type, verifiable_predicates trait,
conflict detection with comparison modes, claims TOML persistence, Grafana dashboard,
backup/restore scripts, and comprehensive test coverage.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-08 09:11:47 +00:00
jordan
c65066fd1c feat(aphoria): implement ignore & exclusion system (Phase 16) 
Reduces scan noise by 96% through proper exclusion of test fixtures,
demo apps, and intentional vulnerabilities.

Phase 16.1 - Glob Pattern Matching:
- Replace starts_with() with globset for ** and * patterns
- Backwards compatible with legacy prefix patterns
- Add walker/mod.rs tests for glob exclusions

Phase 16.2 - .aphoriaignore File:
- Create walker/ignore_file.rs for gitignore-style parsing
- Merge with aphoria.toml excludes
- Support # comments and whitespace trimming

Phase 16.3 - Inline Ignore Comments:
- Create extractors/ignore_comments.rs parser
- Support // aphoria:ignore, // aphoria:ignore-next-line
- Support // aphoria:ignore-block / // aphoria:end-ignore
- Multiple comment styles: //, #, /*, --, <!--
- Integrate with ExtractorRegistry.extract_all()

Phase 16.4 - Ack Export/Import:
- Create ack_file.rs for TOML serialization
- Add 'aphoria ack add' subcommand
- Add 'aphoria ack export' to .aphoria/acks.toml
- Add 'aphoria ack import' from .aphoria/acks.toml
- Preserve expiry and reason fields

Also configures stemedb with:
- aphoria.toml with glob excludes for vulnbank, extractors, fixtures
- .aphoriaignore for dashboard, community, latent, SDK examples

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-07 17:28:50 -07:00
jordan
8af9b48ac7 feat: Complete Aphoria Phase 14 - Governance Workflows 
Implement structured approval workflows for pattern promotion with full
audit trails for SOC 2 compliance.

Core Components:
- governance/types.rs: ApprovalRequest, ApprovalStatus, ApprovalDecision
- governance/workflow.rs: ApprovalWorkflow, ApprovalStage with escalation
- governance/store.rs: JSONL persistence for requests and decisions
- governance/state_machine.rs: Approval state transitions with auto-advance
- governance/audit.rs: AuditTrail with JSON/CSV/Markdown export

CLI Commands:
- aphoria governance pending/approve/reject/escalate/status/create
- aphoria audit trail/export/summary

Integration:
- Pipeline gate blocks promotion until governance approval
- Auto-creates approval requests when governance enabled
- Evidence-based auto-approval for high-confidence patterns

Also includes:
- Phase 11-13: Evidence, Lifecycle, Scope modules
- 62+ governance-specific tests (946 total passing)
- Clippy clean with -D warnings
- Refactored cli.rs into submodules (governance, lifecycle, scope, etc.)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-07 05:16:26 -07:00
jordan
bbeee18b68 feat: Institutional knowledge vision + roadmap phases 11-15 
## Vision Update
- Shift from "code-level truth linter" to "self-learning institutional knowledge"
- Evidence-based authority model: merit over titles
  - ProductSpec → 0.95 authority, 1 usage to graduate
  - Standard (RFC) → 0.85 authority, 3 usages
  - Research (ADR) → 0.70 authority, 5 usages
  - Commit only → 0.40 authority, 10 usages
- Three-tier knowledge: Policies → Conventions → Observations
- Knowledge compounds with every commit

## Gap Analysis
- Documented missing features for enterprise pilot
- Phases 11-15 spec with implementation details
- Evidence detection, scope hierarchy, lifecycle management

## Roadmap Additions
- Phase 11: Evidence-Based Authority (🎯 current)
- Phase 12: Knowledge Scope Hierarchy
- Phase 13: Knowledge Lifecycle Management
- Phase 14: Governance Workflows
- Phase 15: Evidence Source Integration

## Enterprise Simulation UAT
- 6-month simulation: 3 teams, 19 contributors
- Month-by-month scenarios with expected outcomes
- Success metrics for 90-day and 180-day milestones

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-06 23:35:41 -07:00
jordan
157dbbb9eb feat: Complete Aphoria Phase 8-9 + UAT suite (90/90 tests passing) 
## Phase 8: Enterprise Extractor Improvements ✅
- 14 security extractors (TLS, JWT, SQL injection, XSS, etc.)
- 10 framework-specific extractors (Spring, Django, Rails, etc.)
- Config file security detection (YAML, TOML)

## Phase 9: Autonomous Extractor Generation ✅
- Shadow mode executor with TP/FP tracking
- Graduation pipeline with confidence thresholds
- Auto-rollback on regression detection
- Cross-project pattern syncing

## UAT Suite Complete (14 scripts, 90 tests)
- test-core-detection.sh (6 tests)
- test-declarative-extractors.sh (5 tests)
- test-domain-frameworks.sh (5 tests)
- test-domain-unreal.sh (3 tests)
- test-llm-extraction.sh (6 tests)
- test-eval-harness.sh (5 tests)
- test-cross-language.sh (3 tests)
- test-precommit-performance.sh (4 tests)
- test-output-formats.sh (8 tests)
- test-drift-detection.sh (6 tests)
- test-exit-codes.sh (12 tests)
+ 3 more scripts

## Other Changes
- Updated roadmap to mark Phase 8-9 complete
- Added .gitignore entries for build artifacts
- Updated pre-commit: 800 line limit, exclude tests/data/cmd

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-06 22:50:55 -07:00
jordan
9698e63702 docs: fix Aphoria pitch materials based on skeptical buyer review 
Demo script & slides:
- Update speed claims from "0.25s" to "<100ms staged, <1s full"
- Fix CLI output mockups to match actual Aphoria table.rs format
- Remove fake --approver and --expires flags from ack examples
- Remove non-existent "Contact: #security-policy" field
- Update ACK output to describe summary table behavior accurately

Roadmap additions (Phase 10):
- 10.1 Acknowledgment Expiry: --expires flag with duration/ISO date
- 10.2 Human-Readable Signer Names: signer_name + contact in PackHeader
- 10.3 Speed Benchmarks: aphoria scan --benchmark self-test

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-06 16:56:19 -07:00
jordan
bbe6aedc40 feat: Aphoria security extractors + LLM evaluation architecture + ontology docs 
New security extractors:
- insecure_deserialization, orm_injection, path_traversal, security_headers
- ssrf, unvalidated_redirects, weak_password, xxe
- Enhanced tls_version extractor with comprehensive cipher/protocol checks

Architecture docs:
- Scout-judge extraction pattern for LLM-based code analysis
- LLM prompt evaluation framework
- LLM eval implementation guide

Core improvements:
- stemedb-ontology README and client enhancements
- WAL journal/segment instrumentation
- Signing and ingestion refinements
- Consumer health demo script

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-05 15:22:55 -07:00
jordan
41c676a78e feat: Aphoria enterprise features + ontology SDK + file length compliance 
Enterprise Features:
- Hosted mode with remote sync for team pattern aggregation
- Community sharing with privacy-preserving anonymization
- LLM-based semantic claim extraction with Gemini integration
- Pattern learning with promotion to declarative extractors
- High-entropy secrets extractor with configurable thresholds
- Auth bypass and insecure cookies extractors

Module Refactoring:
- Split oversized files to comply with 500-line limit
- Config split: types/core.rs, types/extractors.rs, types/hosted.rs, etc.
- Handlers split: scan.rs, policy.rs, report.rs modules
- Extractors split: declarative/, high_entropy_secrets/, insecure_cookies/
- Learning split: store modules with metrics and persistence

SDK & Ontology:
- stemedb-ontology SDK with fluent builders and StemeDB client
- Pharma domain extractors for FDA Orange Book data
- Consumer health UAT test infrastructure

Code Quality:
- Fixed clippy warnings (needless_borrows_for_generic_args)
- Added KVStore trait imports where needed
- Fixed utoipa path re-exports for OpenAPI docs

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-05 12:55:29 -07:00
jordan
8f6506b70a feat: Aphoria scan modes + stemedb-ontology crate + consumer health UAT 
Major additions:
- Staged scanning modes (working tree, staged, committed) with git integration
- Drift detection for baseline vs current state comparisons
- Hosted API handlers for policy CRUD operations via StemeDB API
- stemedb-ontology crate with domain definitions and medical extractors
- Consumer health vertical UAT scenarios (GLP-1, gastroparesis, etc.)
- Aphoria development skill documentation

Code organization:
- Split large files into focused modules to stay under 500-line limit
- Extracted config tests, episteme helpers/drift/aliases, API helpers

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-04 21:57:33 -07:00
jordan
b3e8a9a058 feat: Multi-application expansion with chaos testing and community UI 
Major additions:
- Community Next.js app (port 18187) for browsing claims with API docs
- stemedb-chaos crate: Fault injection, chaos testing, CRDT properties
- Latent ingestion system: Reddit/FDA ingesters with ADK-Go agents
- Disputed claims handling: Manual review workflows and validation
- Aphoria security scanner: New extractors (SQL injection, command
  injection, weak crypto, TLS version), policy-based ignores, UAT reports
- Docker infrastructure: Dockerfile, docker-compose.yml for full stack
- VulnBank demo: Intentionally vulnerable multi-language test corpus

SDK & API enhancements:
- Source registry handlers for tracking data provenance
- Metrics endpoint
- Skeptic filtering improvements

Code quality:
- Split 14 large files (>500 lines) into focused modules
- All files now under 500-line limit per project guidelines

Documentation:
- Chaos testing guide, circuit breakers, observability docs
- Phase 7 UAT documentation updates
- Martin Kleppmann technical writer agent

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-04 01:24:14 -07:00
jordan
a734be3a0d feat: Phase 7 Content Defense + code structure refactoring 
Content Defense (Phase 7):
- Add SimilarityIndex with MinHash/LSH for near-duplicate detection
- Add QuarantineStore for flagged assertions awaiting admin review
- Add CircuitBreakerStore for per-agent circuit breaker state
- Add ContentDefenseLayer for ingestion pipeline integration
- Add API endpoints for quarantine and circuit breaker management
- Add research module with gap detection and documentation fetching

Code Structure Improvements:
- Extract research CLI commands to research_commands.rs
- Extract API routers to routers.rs module
- Extract key_codec extraction functions to separate module
- Extract test modules to separate files across multiple crates
- All files now under 500 line limit per pre-commit hook

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-03 12:44:05 -07:00
jordan
d3a88585fe feat: Phase 6 UAT - Admission control, HLC recency, cluster coordination 
This commit includes comprehensive work on Phase 6 features:

## Admission Control (Phase 6 admission middleware)
- AdmissionStore implementation backed by TrustRankStore
- PoW verification with tier-based difficulty computation
- Trust tier progression (Newcomer → Established → Trusted → Authority)
- API integration with admission status endpoints

## HLC Recency Lens (Phase 6C)
- HlcRecencyLens for distributed system ordering
- Hybrid logical clock integration with causality preservation

## Cluster Coordination (Phase 6C)
- Multi-node cluster tests (availability, partition tolerance)
- CRDT convergence tests for anti-entropy sync
- Gateway handler improvements

## Aphoria Code Linter (Phase 2A)
- RFC/OWASP corpus builders with network fetching and caching
- Concept hierarchy with auto-alias creation on conflict detection
- Multiple security extractors (TLS, JWT, CORS, secrets, rate limiting)

## Code Organization
- Split large files into modules to comply with 500-line limit
- Improved test organization with separate test modules
- Fixed rkyv serialization for EigenTrustState (AgentScore struct)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-03 00:43:37 -07:00
jordan
42d4e09508 feat: Index persistence (Phase 5C) - vector hot/cold, visual checkpoint 
Phase 5C (Index Persistence) implementation:
- PersistentVectorIndex with hot/cold architecture
  - Hot: in-memory HNSW for recent vectors
  - Cold: memory-mapped HNSW loaded from disk
  - Background builder for WAL replay and atomic swap
  - BLAKE3 integrity verification
- PersistentVisualIndex with checkpoint persistence
  - BkTreeSnapshot with rkyv serialization
  - CRC32C corruption detection
  - Atomic write pattern (temp → fsync → rename)
- Key codec additions for vector index metadata
- Split large files into modules (<500 lines each)
  - battery_pre_sentinel.rs → battery/ directory
  - visual_index.rs → visual_index/ directory
  - persistent.rs → persistent/ directory
- Refactored ingest worker tests for clarity
- Updated roadmap to mark Phase 5 complete

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-02 15:43:18 -07:00
jordan
3320c24afa feat: WAL hardening (Phase 5B) - CRC32C, crash recovery, group commit, log rotation 
Add CRC32C checksums to WAL record format (v2), implement crash recovery
with automatic truncation of corrupt records, add feature-gated group commit
buffer for batched fsync under concurrent load, and implement log rotation
via segment files with global offset addressing.

Key changes:
- Record format v2: [len:u32][crc32c:u32][blake3:32][payload:N]
- recover_file() scans and truncates corrupt tail records
- GroupCommitBuffer batches fsync via MPSC channel (tokio feature gate)
- SegmentManager with binary search resolution and cursor-based cleanup
- Journal::read() auto-refreshes segments on miss for writer/reader split
- Split recovery.rs and key_codec.rs into directory modules for 500-line max

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-02 12:36:35 -07:00