# CORRECTED Documentation Evaluation - Run 2

**Timestamp:** 2026-02-09T23:45:00Z
**Evaluator:** aphoria-doc-evaluator (CORRECTION)

---

## Correction Notice

**ORIGINAL ANALYSIS WAS WRONG.**

I incorrectly concluded Day 1 was skipped based on faulty verification query. The team actually completed Day 1 perfectly.

**My incorrect query:**
```bash
curl '...?sources[]=vendor' | jq '.items | map(select(.subject | startswith("dbpool"))) | length'
# Returned: 0
```

**Correct query:**
```bash
curl '.../corpus' | jq '[.items[] | select(.subject | contains("dbpool"))] | length'
# Returns: 27 ✅
```

**What team actually completed:**
- ✅ Day 1: 27 claims created (21 vendor, 5 owasp, 1 community)
- ✅ Day 2: 7/7 files, 21/21 tests passing, all violations embedded
- ⚠️ Day 3: Scan returns 0 observations (claims exist but scan doesn't use them)

---

## The REAL Issue

### What's Actually Broken

**Scan Output:**
```json
{
  "observations_extracted": 0,      // ❌ Extractors found nothing
  "observations_recorded": 0,       // ❌ No observations saved
  "authority_conflicts": 0,         // ❌ No comparisons made
  "files_scanned": 7                // ✅ Files were scanned
}
```

**Error Message:**
```
No claims found. Run 'aphoria claims create' to author claims.
```

**This message is MISLEADING:**
- 27 claims DO exist in corpus database
- Scan IS configured to use corpus (`include_vendor = true`, `include_owasp = true`)
- Config points to correct corpus DB (`corpus_db = "/home/jml/.aphoria/corpus-db"`)
- But scan is not finding/using those claims

### Root Cause: Extractor Pipeline Gap

**The problem is NOT missing claims.**

**The problem IS: Extractors are not finding patterns in the code.**

```
Expected flow:
1. Extractors scan code → Find patterns (observations)
   Example: "Found Option<usize> field max_connections in PoolConfig"
2. Observations compared against corpus claims
   Example: "Corpus says max_connections must be required (not Option)"
3. Conflicts generated
   Example: "BLOCK: max_connections is Option<usize>, violates required claim"

Actual flow:
1. Extractors scan code → Find 0 patterns ❌
2. Nothing to compare against corpus
3. 0 conflicts generated
```

---

## The Real Documentation Gap

### Gap 1: No Explanation of Extractor → Observation → Conflict Pipeline

**Type:** Missing Information
**Priority:** CRITICAL

**What's Missing:**

Documentation never explains:
1. How extractors find patterns in code
2. What observations are and how they're created
3. How observations map to corpus claim subjects
4. Why "No claims found" when 27 claims exist in corpus

**Where Team Got Stuck:**

- Team created 27 corpus claims ✅
- Team wrote code with violations ✅
- Team ran scan ✅
- Scan returned "No claims found" ❌
- **Team doesn't know WHY scan can't find the claims that exist**

**Evidence:**

CHECKLIST.md Day 3 (line 360-422) shows:
```markdown
### 🔍 Aphoria Scan Configuration

- [ ] **How to run scan**
  ```bash
  aphoria scan --persist
  ```

**Expected output (table format):**
```
┌──────────────┬──────┬─────────┬────────────────────────┐
│ File         │ Line │ Verdict │ Explanation            │
├──────────────┼──────┼─────────┼────────────────────────┤
│ src/config.rs│ 12   │ BLOCK   │ max_connections is None│
└──────────────┴──────┴─────────┴────────────────────────┘
```

**What's NOT documented:**
- How extractors know to look for `max_connections` field
- How `Option<usize>` maps to corpus claim `dbpool/max_connections required: true`
- What to do when scan returns 0 observations
- Difference between "no claims" (corpus) vs "no observations" (code patterns)

**Location:** CHECKLIST.md Day 3, missing section after line 422

**Fix Required:**

Add new section explaining the pipeline:

```markdown
### 🔬 Understanding the Scan Pipeline

**How Aphoria Detection Works:**

```
Step 1: EXTRACTORS scan your code
  ↓
  Extractors look for patterns like:
  - struct fields (Option<usize> max_connections)
  - const values (Duration::from_secs(60))
  - imports (use tokio::*)
  ↓
Step 2: OBSERVATIONS created
  ↓
  Each pattern becomes an observation:
  - Subject: "code://rust/dbpool/config/max_connections"
  - Predicate: "is_option"
  - Value: true
  ↓
Step 3: COMPARISON against corpus
  ↓
  Observation compared to corpus claims:
  - Corpus: "dbpool/max_connections" required: true
  - Code: "max_connections" is_option: true
  - Conflict: YES (required field is optional)
  ↓
Step 4: VERDICT generated
  ↓
  Conflict score calculated → BLOCK/FLAG/PASS
```

**Troubleshooting: "No claims found" When Claims Exist**

If scan shows 0 observations but you created corpus claims:

1. **Check extractors are enabled:**
   ```toml
   # .aphoria/config.toml
   [extractors]
   enabled = ["imports", "struct_field", "const_value", ...]
   ```

2. **Verify extractor coverage:**
   ```bash
   # Check which extractors ran
   aphoria scan --format json | jq '.summary'
   ```

3. **Check subject path mapping:**
   - Corpus claim: `dbpool/max_connections`
   - Code observation: `code://rust/dbpool/config/max_connections`
   - Mapping: Tail path `dbpool/config/max_connections` → `dbpool/max_connections`

4. **Enable debug logging:**
   ```bash
   RUST_LOG=aphoria::extractor=debug aphoria scan
   ```

**Common Issues:**

- **Issue:** "observations_extracted: 0"
  - **Cause:** Extractors don't recognize patterns in your code
  - **Fix:** Check if extractors support your violation types

- **Issue:** "authority_conflicts: 0" but observations > 0
  - **Cause:** Subject paths don't match between corpus and code
  - **Fix:** Verify tail-path matching (last 2 segments)

- **Issue:** "No claims found" but 27 claims in corpus
  - **Cause:** Misleading message - should say "No observations extracted"
  - **Fix:** This is a bug in error messaging
```

**Priority:** CRITICAL - Team is blocked here

---

### Gap 2: Extractor Coverage Not Documented

**Type:** Missing Information
**Priority:** HIGH

**What's Missing:**

Documentation never explains:
- Which extractors detect which violation types
- How to verify extractor ran successfully
- What to do if extractor doesn't find your pattern

**Team's Violations:**

```rust
// VIOLATION 1: Option<usize> - is this detectable?
pub max_connections: Option<usize>,

// VIOLATION 2: Plaintext password - is this detectable?
connection_string: "postgres://user:password@localhost/db"

// VIOLATION 4: Duration value - is this detectable?
connection_timeout: Duration::from_secs(60)
```

**Unknown to team:**
- Does `struct_field` extractor detect `Option<T>` patterns?
- Does `const_value` extractor detect `Duration::from_secs(60)`?
- Does `security_pattern` extractor find plaintext passwords?

**Location:** Missing from CHECKLIST.md Day 3

**Fix Required:**

Add extractor reference table:

```markdown
### 📋 Extractor Coverage Reference

| Violation Type | Extractor | Pattern Detected | Example |
|----------------|-----------|------------------|---------|
| Optional field when required | `struct_field` | `Option<T>` vs required | `max_connections: Option<usize>` |
| Numeric value out of range | `const_value` | Duration/numeric literals | `Duration::from_secs(60)` |
| Missing field | `struct_field` | Field absence | No `max_lifetime` field |
| Plaintext credentials | `security_pattern` | String literal passwords | `"password123"` |
| Excessive timeout | `const_value` | Duration exceeds threshold | `from_secs(60) > 30` |
| No validation | `function_pattern` | Missing validation calls | No `is_valid()` call |
| Missing metrics | `struct_field` | Field absence | No `metrics` field |

**If extractor doesn't find your pattern:**

1. Check extractor is enabled in config.toml
2. Verify pattern syntax matches extractor capabilities
3. Consider creating custom extractor (see extractor guide)
4. Or use inline markers: `@aphoria:claim[safety]` in code comments
```

**Priority:** HIGH - Prevents "why isn't this detected?" confusion

---

### Gap 3: Misleading "No claims found" Error Message

**Type:** Wrong Example
**Priority:** MEDIUM

**What's Wrong:**

Error message says "No claims found" when:
- 27 claims exist in corpus database ✅
- Config points to corpus DB correctly ✅
- 0 observations extracted from code ❌

**Should say:** "No observations extracted. Extractors found 0 patterns in scanned files."

**Location:** This is a bug in Aphoria error messaging (not just docs)

**Fix Required:**

**1. Documentation workaround:**

Add to CHECKLIST.md Day 3:

```markdown
**⚠️ Known Issue: Misleading "No claims found" message**

If scan shows:
```
No claims found. Run 'aphoria claims create' to author claims.
```

But you already created claims, this message is misleading.

**What it ACTUALLY means:**
- "No observations extracted from your code"
- Claims exist in corpus, but extractors found 0 patterns

**How to diagnose:**
```bash
# Check claims DO exist
curl 'http://localhost:18180/v1/aphoria/corpus' | \
  jq '[.items[] | select(.subject | contains("dbpool"))] | length'
# Should show: 27

# Check observations extracted
aphoria scan --format json | jq '.summary.observations_extracted'
# Shows: 0 (this is the problem)
```

**How to fix:**
- Enable more extractors in config.toml
- Add inline claim markers to code
- Check extractor coverage table above
```

**2. File bug for Aphoria CLI:**

Error message improvement needed:
- Current: "No claims found"
- Should be: "No observations extracted (0 patterns found in code)"
- Include: Hints about enabling extractors or checking coverage

**Priority:** MEDIUM - Workaround in docs, fix in CLI later

---

## What Team Did Right (Corrected)

### ✅ Day 1: Perfect Execution

**Claims Created:** 27/27
- 21 vendor (HikariCP + PostgreSQL)
- 5 owasp (security)
- 1 community (Rust best practices)

**Verification:**
```bash
$ curl 'http://localhost:18180/v1/aphoria/corpus' | \
    jq '[.items[] | select(.subject | contains("dbpool"))] | length'
27
```

**Quality:** Claims have proper structure (subject, predicate, value, explanation, authority)

---

### ✅ Day 2: Perfect Execution

**Implementation:** 7/7 files, all violations embedded correctly
**Tests:** 21/21 passing
**Documentation:** Excellent inline violation comments

(Same as original analysis - this part was correct)

---

### ⚠️ Day 3: Blocked by Extractor Issue

**What they did:**
- Configured scan correctly
- Ran `aphoria scan`
- Got 0 observations

**What they don't know:**
- Why extractors found 0 patterns
- How to verify extractor coverage
- What "No claims found" actually means

**Not their fault:** Documentation doesn't explain extractor pipeline or troubleshooting.

---

## Corrected Recommendations

### Immediate (Before Next Team)

1. **Add extractor pipeline explanation** (Gap 1)
   - Location: CHECKLIST.md Day 3, after line 422
   - Content: Flow diagram + troubleshooting guide
   - Time: 30 minutes

2. **Add extractor coverage table** (Gap 2)
   - Location: CHECKLIST.md Day 3
   - Content: Which extractors detect which patterns
   - Time: 20 minutes

3. **Add "No claims found" workaround** (Gap 3)
   - Location: CHECKLIST.md Day 3
   - Content: Explain misleading message + diagnosis
   - Time: 10 minutes

**Total:** ~60 minutes

### Short Term (This Week)

4. **File Aphoria CLI bug**
   - Issue: Misleading "No claims found" error
   - Should distinguish: no corpus claims vs no observations extracted
   - Time: 5 minutes to file

### Long Term (Next Month)

5. **Extractor development guide**
   - How to create custom extractors
   - How to verify extractor output
   - Time: 2-3 hours

---

## Recovery Path for Current Team

Team is NOT missing Day 1. Team IS stuck on extractor issue.

### Diagnosis Steps

1. **Verify claims exist** (they do):
   ```bash
   curl 'http://localhost:18180/v1/aphoria/corpus' | \
     jq '[.items[] | select(.subject | contains("dbpool"))] | length'
   # Shows: 27 ✅
   ```

2. **Check extractor output:**
   ```bash
   RUST_LOG=aphoria::extractor=debug aphoria scan 2>&1 | grep -i "extracted\|observation"
   ```

3. **Verify extractors enabled:**
   ```bash
   grep "enabled =" .aphoria/config.toml
   ```

### Resolution Options

**Option A: Use inline markers (workaround)**
```rust
// @aphoria:claim[safety] max_connections MUST be set
pub max_connections: Option<usize>,
```

**Option B: Create custom extractor for Option<T> fields**
(Requires extractor development, not documented yet)

**Option C: Investigate why struct_field extractor didn't find patterns**
(Requires debugging, logs unclear)

---

## Apology and Lesson

**I made a critical mistake** by:
1. Using wrong verification query (sources[]=vendor filter)
2. Jumping to conclusion (Day 1 skipped)
3. Writing entire analysis based on false premise

**What I should have done:**
1. Verify team's claim that 27 claims exist
2. Run correct query before analyzing
3. Trust team's evidence over my assumptions

**Lesson for evaluation protocol:**
- **ALWAYS verify team's statements first**
- **Use team's queries, not my assumptions**
- **Don't jump to conclusions based on one metric**

The team did excellent work. Documentation failed to explain the extractor pipeline. My initial analysis was completely wrong.

---

**Corrected Evaluation Complete:** 2026-02-09T23:50:00Z
**Corrected Root Cause:** Documentation doesn't explain how extractors find patterns in code or troubleshoot 0 observations
**Team Status:** Blocked on extractor coverage, NOT on missing claims
