157dbbb9eb
## Phase 8: Enterprise Extractor Improvements ✅ - 14 security extractors (TLS, JWT, SQL injection, XSS, etc.) - 10 framework-specific extractors (Spring, Django, Rails, etc.) - Config file security detection (YAML, TOML) ## Phase 9: Autonomous Extractor Generation ✅ - Shadow mode executor with TP/FP tracking - Graduation pipeline with confidence thresholds - Auto-rollback on regression detection - Cross-project pattern syncing ## UAT Suite Complete (14 scripts, 90 tests) - test-core-detection.sh (6 tests) - test-declarative-extractors.sh (5 tests) - test-domain-frameworks.sh (5 tests) - test-domain-unreal.sh (3 tests) - test-llm-extraction.sh (6 tests) - test-eval-harness.sh (5 tests) - test-cross-language.sh (3 tests) - test-precommit-performance.sh (4 tests) - test-output-formats.sh (8 tests) - test-drift-detection.sh (6 tests) - test-exit-codes.sh (12 tests) + 3 more scripts ## Other Changes - Updated roadmap to mark Phase 8-9 complete - Added .gitignore entries for build artifacts - Updated pre-commit: 800 line limit, exclude tests/data/cmd Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
89 lines
3.3 KiB
Markdown
89 lines
3.3 KiB
Markdown
# Aphoria User Acceptance Testing
|
|
|
|
End-to-end validation of Aphoria workflows.
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
# Run the enterprise workflow UAT
|
|
./scripts/test-enterprise-workflow.sh
|
|
```
|
|
|
|
## Comprehensive Vision UAT
|
|
|
|
**[Comprehensive Vision UAT Plan](./comprehensive-vision-uat.md)** - The master plan for validating Aphoria's complete vision:
|
|
|
|
| Category | Tests | Priority | Status |
|
|
|----------|-------|----------|--------|
|
|
| 1. Core Detection | 10 tests | P0 | In Progress |
|
|
| 2. Enterprise Policy | 13 tests | P0-P2 | PASS (existing scripts) |
|
|
| 3. Pre-Commit Integration | 11 tests | P0-P1 | Partial |
|
|
| 4. LLM Extraction | 8 tests | P1-P2 | Planned |
|
|
| 5. Declarative Extractors | 7 tests | P0-P2 | Planned |
|
|
| 6. Output Formats | 8 tests | P0 | Partial |
|
|
| 7. Domain-Specific Audits | 6 tests | P1-P2 | PASS (Masq) |
|
|
| 8. Protocol Vision | 3 tests | P3 | Future |
|
|
|
|
## UAT Reports
|
|
|
|
| Report | Status | Description |
|
|
|--------|--------|-------------|
|
|
| **[Comprehensive Vision UAT](./comprehensive-vision-uat.md)** | Draft | Master plan for full vision validation |
|
|
| **[Gap Analysis](./gap-analysis-2026-02-06.md)** | Complete | Code vs UAT gap analysis |
|
|
| [Policy Source Tracking](./2026-02-04-uat-real-world-policy-source.md) | PASS | Trust Pack workflow validation |
|
|
| [Masq Unreal Audit](./2026-02-04-masq-unreal-audit.md) | PASS | Domain-specific (Unreal Engine) |
|
|
| [Future Scenarios](./future-scenarios.md) | Planned | Deferred scenarios awaiting enterprise feedback |
|
|
|
|
## Scripts
|
|
|
|
### Existing (Passing)
|
|
|
|
| Script | Purpose | Status |
|
|
|--------|---------|--------|
|
|
| [test-enterprise-workflow.sh](./scripts/test-enterprise-workflow.sh) | Full Trust Pack round-trip test | PASS (12/12) |
|
|
| [test-multi-pack-conflict.sh](./scripts/test-multi-pack-conflict.sh) | Multiple packs, same concept | PASS (7/7) |
|
|
| [test-pack-version-update.sh](./scripts/test-pack-version-update.sh) | Pack version supersession | PASS (6/6) |
|
|
|
|
### New (Comprehensive Vision)
|
|
|
|
| Script | Purpose | Category | Priority |
|
|
|--------|---------|----------|----------|
|
|
| [test-core-detection.sh](./scripts/test-core-detection.sh) | Cross-language detection tests | Cat 1 | P0 |
|
|
| [test-exit-codes.sh](./scripts/test-exit-codes.sh) | Exit code validation | Cat 3 | P0 |
|
|
|
|
## CI Integration
|
|
|
|
The UAT is integrated into CI via `.github/workflows/ci.yml`:
|
|
|
|
```yaml
|
|
aphoria-uat:
|
|
name: Aphoria Enterprise UAT
|
|
runs-on: ubuntu-latest
|
|
needs: [check, test]
|
|
steps:
|
|
- name: Build Aphoria
|
|
run: cargo build --release --package aphoria
|
|
- name: Run Enterprise Workflow UAT
|
|
run: ./applications/aphoria/uat/scripts/test-enterprise-workflow.sh
|
|
```
|
|
|
|
## Adding New UAT Scenarios
|
|
|
|
1. Create `YYYY-MM-DD-uat-{scenario}.md` with test plan
|
|
2. Add automated script in `scripts/`
|
|
3. Update this README
|
|
4. Add to CI workflow if needed
|
|
|
|
## Structure
|
|
|
|
```
|
|
uat/
|
|
├── README.md # This file
|
|
├── 2026-02-04-uat-real-world-policy-source.md # Policy source tracking UAT
|
|
├── future-scenarios.md # Tested & deferred scenarios
|
|
└── scripts/
|
|
├── test-enterprise-workflow.sh # Basic Trust Pack workflow
|
|
├── test-multi-pack-conflict.sh # Multi-pack behavior
|
|
└── test-pack-version-update.sh # Version supersession
|
|
```
|