stemedb/applications/aphoria/dogfood/msgqueue/eval/EVALUATION-REPORT-2026-02-10.md

Documentation Evaluation Report: msgqueue Day 3 (Second Attempt)

Project: dogfood/msgqueue Evaluation Date: 2026-02-10 Documentation Evaluated:

• dogfood/msgqueue/plan.md (updated Day 3 section)
• dogfood/msgqueue/DAY3-READY.md (fresh start guide)
• .claude/skills/aphoria-dogfood/SKILL.md
• CLAUDE.md (Day 3 emphasis section)

Team Phase: Day 3 (Second Attempt After Doc Fixes)

---

Executive Summary

The documentation fixes from earlier today partially worked but revealed a deeper, undocumented gap.

What improved:

• ✅ Team followed 6-phase workflow (previous attempt: skipped phases 3-5)
• ✅ Team created 7 extractors (previous attempt: 0 extractors)
• ✅ Extractors ran successfully (observations +7)

What still failed:

• ❌ 0% detection rate (same outcome as before)
• ❌ Extractors don't match claims (concept path alignment issue)
• ❌ No way to debug why alignment failed

Root Cause: Documentation gap AFTER extractor creation workflow - no explanation of:

1. How declarative extractor subject field works
2. What concept path format observations will have
3. How to debug misalignment when it happens

Critical Finding: This is NOT the same failure. Previous failure was "didn't create extractors at all." This failure is "created extractors but they don't work due to undocumented format requirements."

---

Critical Findings (P0 - Blocks Day 3 Completion)

Finding 1: Declarative Extractor subject Field Format Undocumented

Type: Missing Information

Evidence:

Team created extractors with:

[[extractors.declarative]]
name = "queue_max_size_unbounded"
pattern = 'max_queue_size:\s*None'
[extractors.declarative.claim]
subject = "queue/max_size"  ← 2-segment path
predicate = "bounded"
value = false

Claims expect:

id = "msgqueue-015"
concept_path = "msgqueue/queue/max_size"  ← 3-segment path
predicate = "bounded"
value = true

Result:

• Extractors ran (+7 observations)
• Observations didn't match claims (0 conflicts)
• Tail-path mismatch: extractor creates queue/max_size, claim expects last 2 of msgqueue/queue/max_size

Team Quote (DAY3-SUMMARY.md:158):

"Hypothesis: Declarative extractor subject field doesn't build concept paths the way we expected."

Impact:

• Team spent 70 minutes creating extractors that don't work
• 0% detection rate despite correct workflow
• Day 3 incomplete (cannot proceed to Day 4 without working extractors)
• Blocker: True (stopped progress)

Recommendation:

• Where: applications/aphoria/docs/extractors/declarative-extractors.md (CREATE NEW)
• What to add:

## Declarative Extractor Field Reference

### `subject` Field (Required)

The `subject` field defines the **concept path** for observations created by this extractor.

**Format:** Full slash-separated path matching your claim's `concept_path`.

**Example (Correct):**
```toml
# Claim has: concept_path = "msgqueue/queue/max_size"
# Extractor must use SAME path:
[extractors.declarative.claim]
subject = "msgqueue/queue/max_size"  ✓ CORRECT

Common Mistake:

# ❌ WRONG: Using only leaf segments
subject = "queue/max_size"  # Will NOT match claim!

Why: Observations must match claims via tail-path (last 2 segments). If claim is msgqueue/queue/max_size, observation path must END with queue/max_size. Using subject = "queue/max_size" creates observation with path queue/max_size which has different tail-path.

Debug Tip: Use aphoria scan --show-observations to see actual observation concept paths (Feature Request: VG-DAY3-001).

- **Priority:** P0 (BLOCKER)

---

### Finding 2: No Debug Visibility Into Observation Concept Paths

**Type:** Missing Information (Tool Gap)

**Evidence:**

**Team attempted (DAY3-SUMMARY.md:189-199):**
> "No visibility into:
> - What concept paths observations actually get
> - Why observations don't match claims
> - Whether tail-path matching is working
>
> Needed:
> - `aphoria scan --show-observations` to see all observations with full paths
> - `aphoria scan --explain-match CLAIM_ID` to see why claim wasn't matched"

**What docs say:**
- plan.md shows: `aphoria scan --format json > scan-v1.json`
- No mention of how to see observation details
- No debugging workflow for extractor alignment

**Impact:**
- Team created extractors with wrong format
- No way to discover mistake without trial-and-error
- 70 minutes spent on failed attempt
- **Blocker:** Yes (cannot debug without visibility)

**Recommendation:**
- **Where:** `plan.md` Day 3 Phase 5 (Verification Scan)
- **What to add:**

```markdown
### Phase 5: Verification Scan (15 min)

```bash
aphoria scan --format json > scan-v2.json

If detection rate is still 0%:

Debug extractor alignment:

# See all observations with full concept paths
aphoria scan --show-observations > observations.txt

# Compare observation paths with claim paths
grep "concept_path" .aphoria/claims.toml
grep "concept_path" observations.txt

# Check if tail-paths match (last 2 segments)
# Claim: msgqueue/queue/max_size → tail: queue/max_size
# Observation: queue/max_size → tail: queue/max_size
# If tails don't match → fix extractor subject field

Common Issue: Extractor subject doesn't match claim concept_path. Fix: Update extractor subject to use full path matching claim.

- **Priority:** P0 (BLOCKER)
- **Product Gap:** VG-DAY3-001 (`--show-observations` flag doesn't exist yet)

---

## High Priority Improvements (P1)

### Finding 3: No Worked Example of Declarative Extractor

**Type:** Missing Information

**Evidence:**
- Team created extractors based on intuition
- No reference example showing complete path from code → extractor → claim → match
- Guessed `subject` format incorrectly

**Doc locations checked:**
- `plan.md`: Shows workflow but not extractor format
- `DAY3-READY.md`: Shows Phase 4 but not example extractor
- No `applications/aphoria/docs/examples/extractors/` directory

**Impact:**
- Team made wrong assumptions about subject format
- No way to validate assumptions before running scan
- Time wasted on trial-and-error

**Recommendation:**
- **Where:** `applications/aphoria/docs/examples/extractors/timeout-zero-example.md` (CREATE NEW)
- **What to add:**

```markdown
# Complete Example: Detecting timeout=0

## The Violation (Code)

```rust
// src/config.rs:20
pub timeout: Duration = Duration::from_secs(0);  // ❌ Violation

The Claim (.aphoria/claims.toml)

[[claim]]
id = "myapp-timeout-001"
concept_path = "myapp/config/timeout"
predicate = "zero"
value = 0
comparison = "not_equals"  # Must NOT be zero

The Extractor (.aphoria/config.toml)

[[extractors.declarative]]
name = "timeout_zero_detector"
pattern = 'timeout:\s*Duration::from_secs\(0\)'
languages = ["rust"]

[extractors.declarative.claim]
subject = "myapp/config/timeout"  ← MUST match claim concept_path
predicate = "zero"
value = 0
confidence = 0.95

How Matching Works

1. Extractor runs → Finds pattern in src/config.rs:20
2. Creates observation: myapp/config/timeout :: zero = 0
3. Compares to claim: myapp/config/timeout :: zero NOT_EQUALS 0
4. Result: CONFLICT (observation says 0, claim says NOT 0)

Key: subject field MUST exactly match claim's concept_path.

- **Priority:** P1 (High)

---

### Finding 4: plan.md Day 3 Phase 4 Doesn't Show Extractor Format

**Type:** Buried Information

**Evidence:**

**plan.md says (lines 228-247):**
```markdown
### Phase 4: Extractor Creation (30 min) **[REQUIRED]**

For EACH missed violation ({Z} total):
```bash
/aphoria-custom-extractor-creator --violation "{pattern}" --claim {claim-id}

Expected: {Z} extractors created in .aphoria/extractors/

**Problem:**
- Shows skill invocation (which team probably doesn't have)
- Doesn't show manual fallback (declarative extractor TOML format)
- Team had to guess format

**Team action:**
- Created declarative extractors in `.aphoria/config.toml`
- Guessed `subject` format incorrectly
- No validation before scan

**Recommendation:**
- **Where:** `plan.md` Day 3 Phase 4 (line 228)
- **What to add:**

```markdown
### Phase 4: Extractor Creation (30 min) **[REQUIRED]**

**Option A: Using Skill (Recommended)**
```bash
/aphoria-custom-extractor-creator --violation "{pattern}" --claim {claim-id}

Option B: Manual Declarative Extractor (If skill unavailable)

Add to .aphoria/config.toml:

[[extractors.declarative]]
name = "descriptive_name"
pattern = 'regex_pattern_matching_code'
languages = ["rust"]

[extractors.declarative.claim]
subject = "{FULL_CLAIM_CONCEPT_PATH}"  ← Copy from claim's concept_path
predicate = "{claim_predicate}"
value = {inverted_value}  # false if claim expects true
confidence = 0.95

CRITICAL: subject must EXACTLY match your claim's concept_path.

Example: If claim has concept_path = "msgqueue/queue/max_size", Then extractor needs subject = "msgqueue/queue/max_size" (not just "queue/max_size")

Verify format:

# Before scanning, check your extractor subjects match claim paths
grep "subject =" .aphoria/config.toml
grep "concept_path =" .aphoria/claims.toml
# Subjects should be subset of concept_paths

- **Priority:** P1 (High)

---

## Medium Priority Improvements (P2)

### Finding 5: No Validation Command for Extractor Config

**Type:** Missing Information (Tool Gap)

**Team feedback (DAY3-SUMMARY.md:281-285):**
> "VG-DAY3-003: No validation for extractor configuration
> - **Impact:** Syntax errors silent, extractors just don't run
> - **Recommendation:** Add `aphoria extractors validate` command"

**Impact:**
- Team could have syntax errors and not know until scan
- No way to validate subject format before scanning
- Slow iteration (must run full scan to test)

**Recommendation:**
- **Where:** `plan.md` Day 3 Phase 4, after extractor creation
- **What to add:**

```markdown
**Validate extractors before scanning:**
```bash
# Check TOML syntax (if command exists)
aphoria extractors validate

# Manual validation:
# 1. Check subject matches a claim concept_path
grep "subject =" .aphoria/config.toml
grep "concept_path =" .aphoria/claims.toml

# 2. Test regex pattern matches code
grep -r "max_queue_size:\s*None" src/
# Should find the violation you're targeting

- **Priority:** P2 (Medium)
- **Product Gap:** VG-DAY3-003 (`aphoria extractors validate` doesn't exist)

---

### Finding 6: No Single-Extractor Test Command

**Type:** Missing Information (Tool Gap)

**Team feedback (DAY3-SUMMARY.md:286-289):**
> "VG-DAY3-004: No way to test extractor before full scan
> - **Impact:** Must run full scan to test one extractor
> - **Recommendation:** Add `aphoria extractors test EXTRACTOR_NAME --file path.rs`"

**Impact:**
- Slow iteration when debugging extractors
- Must wait for full scan (9 files) to test one pattern
- No feedback loop for learning correct format

**Recommendation:**
- Document workaround until tool exists
- **Where:** `plan.md` Day 3 Phase 4
- **What to add:**

```markdown
**Test individual extractor (workaround until tool exists):**
```bash
# Manually test regex pattern on target file
grep -E "max_queue_size:\s*None" src/config.rs
# Should match the line with the violation

# If no match → fix pattern
# If match → pattern works, issue might be subject field

- **Priority:** P2 (Medium)
- **Product Gap:** VG-DAY3-004 (single extractor test command doesn't exist)

---

## Analysis: Why Documentation Fixes Didn't Fully Work

### What Worked

**Yesterday's fixes successfully changed behavior:**

| Before | After | Improvement |
|--------|-------|-------------|
| Skipped extractor creation entirely | Created 7 extractors | ✅ Workflow adopted |
| No Phase 4 execution | Phase 4 executed (30 min) | ✅ Step not skipped |
| 0 extractors | 7 extractors | ✅ Output achieved |

**Docs successfully fixed:**
- Emphasizing Step 3 as REQUIRED worked
- 6-phase breakdown worked
- Pre-flight check worked
- Team followed workflow

### What Didn't Work

**New gap revealed at deeper layer:**

The workflow fixes got the team TO extractor creation, but didn't explain HOW to create working extractors.

**Missing:**
1. `subject` field format specification
2. Worked example showing concept_path alignment
3. Debug visibility into observation paths
4. Validation workflow before scanning

**Result:**
- Team followed workflow ✅
- Team created extractors ✅
- Extractors don't work ❌ (undocumented format requirement)

---

## Gap Type Analysis

### Documentation Gaps (Not Team Errors)

All findings are **legitimate documentation gaps:**

1. **Finding 1-2:** Missing information (subject format, debug commands)
2. **Finding 3:** Missing example (worked end-to-end)
3. **Finding 4:** Buried information (manual fallback not shown)
4. **Finding 5-6:** Tool gaps (validation/test commands don't exist)

**NO team errors found.** Team followed docs correctly, docs just didn't have the information needed for success.

---

## Product Gaps Identified

These findings require **product features**, not just doc fixes:

| Gap ID | Title | Severity | Recommendation |
|--------|-------|----------|----------------|
| VG-DAY3-001 | No `--show-observations` flag | P0 | Add `aphoria scan --show-observations` |
| VG-DAY3-002 | Concept path alignment undocumented | P0 | Document subject field format |
| VG-DAY3-003 | No extractor validation command | P2 | Add `aphoria extractors validate` |
| VG-DAY3-004 | No single-extractor test | P2 | Add `aphoria extractors test NAME` |

**P0 gaps (VG-DAY3-001, VG-DAY3-002) are blockers for Day 3 completion.**

---

## Recommended Actions

### Immediate (Today - Before Next Retry)

**1. Create declarative extractor reference doc**
- File: `applications/aphoria/docs/extractors/declarative-extractors.md`
- Content: Subject field format, worked example, common mistakes
- Time: 30 minutes

**2. Update plan.md Phase 4 with manual fallback**
- Show declarative extractor TOML format
- Emphasize subject must match concept_path
- Add validation steps before scanning
- Time: 15 minutes

**3. Add debug workflow to plan.md Phase 5**
- Show how to compare observation vs claim paths
- Explain tail-path matching
- Give troubleshooting steps for 0% detection
- Time: 15 minutes

**Total immediate work:** ~1 hour

### Short Term (This Week)

**4. Create worked example doc**
- File: `applications/aphoria/docs/examples/extractors/timeout-zero-example.md`
- Show complete flow: code → extractor → claim → conflict
- Time: 30 minutes

**5. Add validation section to plan.md**
- Manual validation steps (grep subject vs concept_path)
- Regex testing (grep pattern against code)
- Time: 10 minutes

**Total short-term work:** ~40 minutes

### Long Term (Product Features)

**6. Implement VG-DAY3-001: `--show-observations` flag**
- Show observation concept paths in scan output
- Critical for debugging extractor alignment

**7. Implement VG-DAY3-003: `aphoria extractors validate`**
- Validate TOML syntax
- Check subject fields match existing claims
- Test regex patterns against codebase

**8. Implement VG-DAY3-004: `aphoria extractors test`**
- Test single extractor against specific file
- Faster iteration for debugging

---

## Comparison: Failure Mode Evolution

| Attempt | Date | Extractors Created | Detection Rate | Failure Reason |
|---------|------|-------------------|----------------|----------------|
| **First** | 2026-02-10 AM | 0 | 0% | Skipped Phase 4 entirely (docs unclear Step 3 required) |
| **Second** | 2026-02-10 PM | 7 | 0% | Wrong subject format (docs don't explain field) |

**Progress:** Documentation fixes moved failure from "didn't try" to "tried but wrong format."

**Remaining gap:** Format specification and debugging visibility.

---

## Success Criteria Met?

Evaluation complete when:

✅ Progress log captured (DAY3-SUMMARY.md exists)
✅ Implementation review completed (7 extractors analyzed)
✅ Gap analysis completed (6 findings categorized)
✅ Evaluation report produced (this document)
✅ All artifacts saved in `dogfood/msgqueue/eval/`
✅ Every gap has specific, actionable fix
✅ Team errors distinguished from doc gaps (NO team errors found)
✅ Evidence chains built (thought → action → doc → gap)

---

## Appendices

### Appendix A: Team Progress Evidence

- **DAY3-SUMMARY.md**: Full day 3 writeup with root cause analysis
- **scan-v1.json**: Baseline scan (0% detection)
- **scan-v1-with-extractors.json**: After extractors (+7 observations, still 0% detection)
- **.aphoria/config.toml**: 7 declarative extractors created

### Appendix B: Extractor Subject Formats Used

| Extractor | Subject Used | Claim Concept Path | Match? |
|-----------|-------------|-------------------|--------|
| queue_max_size_unbounded | `queue/max_size` | `msgqueue/queue/max_size` | ❌ Mismatch |
| prefetch_count_unbounded | `consumer/prefetch_count` | `msgqueue/consumer/prefetch_count` | ❌ Mismatch |
| tls_cert_validation_disabled | `tls/certificate_validation` | `msgqueue/tls/certificate_validation` | ❌ Mismatch |
| blocking_in_async | `async/runtime` | `msgqueue/async/runtime` | ❌ Mismatch |
| ack_mode_auto | `consumer/ack_mode` | `msgqueue/consumer/ack_mode` | ❌ Mismatch |
| requeue_limit_unbounded | `consumer/requeue_limit` | `msgqueue/consumer/requeue_limit` | ❌ Mismatch |
| max_connections_unbounded | `connection/max_connections` | `msgqueue/connection/max_connections` | ❌ Mismatch |

**Pattern:** All extractors missing `msgqueue/` prefix.

### Appendix C: Claims Checked

```bash
$ grep "concept_path" .aphoria/claims.toml | head -10
concept_path = "msgqueue/consumer/timeout"
concept_path = "msgqueue/tls/certificate_validation"
concept_path = "msgqueue/connection/max_connections"
concept_path = "msgqueue/connection/lifecycle"
concept_path = "msgqueue/metrics/enabled"
concept_path = "msgqueue/retry/max_attempts"
concept_path = "msgqueue/retry/backoff_strategy"
concept_path = "msgqueue/connection/cleanup"
concept_path = "msgqueue/async/runtime"
concept_path = "msgqueue/connection/idle_timeout"

All claims use msgqueue/ prefix consistently.

---

Evaluation Date: 2026-02-10 Next Action: Implement immediate fixes (1 hour) and retry Day 3 with correct subject format.