stemedb/applications/aphoria/docs/dogfooding-common-mistakes.md

# Common Dogfooding Mistakes

This document catalogs common mistakes made during Aphoria dogfooding exercises, with evidence from real failures and how to avoid them.

---

## Mistake #1: Skipping Day 3 Extractor Creation (CRITICAL)

**Severity:** 🚨 CRITICAL - Breaks the entire flywheel

### What People Do Wrong

```bash
# Day 3 (incorrect execution):
aphoria scan --format json > scan-results-v1.json
# Looks at results (0/8 violations detected)
# Moves on to Day 4 without creating extractors
```

**Result:**
- 0 extractors created (should be 8)
- No `.aphoria/extractors/` directory
- No `scan-v2.json` file
- No `DAY3-SUMMARY.md`
- Detection rate: 0% (no improvement)
- **Flywheel completely broken**

### Why It's Wrong

1. **No knowledge captured** - The 8 patterns that should have been learned are lost
2. **No corpus growth** - Next msgqueue dogfood will ALSO have 0% detection
3. **Flywheel doesn't compound** - No benefit from previous work
4. **Misses the point** - Day 3 IS the autonomous learning validation
5. **Product not validated** - Can't prove Aphoria creates extractors dynamically

### Evidence from msgqueue Dogfood (2026-02-10)

**What was done:**
- ✅ Day 1: 22 claims authored (50% reused)
- ✅ Day 2: 8 violations embedded in code
- ❌ Day 3: Scan ran once, showed 0/8 violations, **stopped there**
  - No extractors created
  - No gap analysis
  - No re-scan
  - No DAY3-SUMMARY.md
- ⚠️ Day 4-5: Can't proceed without working extractors

**Scan results:**
- v1: 0/8 violations detected (0%)
- v2: **Not run** (should have been 8/8 = 100%)
- Missing: 20/22 claims had no observations
- **Detection rate improvement: 0%** (should have been +100%)

**Files that should exist but don't:**
```bash
$ ls .aphoria/extractors/
# No such directory

$ ls scan-v2.json
# No such file

$ ls DAY3-SUMMARY.md
# No such file
```

### What To Do Instead

**Day 3 (correct execution - 5 phases):**

#### Phase 1: Pre-Flight Check (5 min)
```bash
/help | grep aphoria-custom-extractor-creator  # Verify skill available
grep -r "@aphoria:claim" src/ | wc -l          # Verify markers (should be 8)
cargo check                                    # Verify code compiles
```

#### Phase 2: Baseline Scan (15 min)
```bash
aphoria scan --format json > scan-v1.json
# Result: 0/8 violations detected (expected for new domain)
```

#### Phase 3: Gap Analysis (15 min)
Analyze why 0/8 detected:
- No extractors exist for msgqueue patterns
- Need to create 8 extractors (one per violation)

#### Phase 4: Extractor Creation (30 min) **[CRITICAL]**
```bash
/aphoria-custom-extractor-creator --violation "timeout=0" --claim msgqueue-001
/aphoria-custom-extractor-creator --violation "prefetch_count=u16::MAX" --claim msgqueue-012
/aphoria-custom-extractor-creator --violation "verify_certificates=false" --claim msgqueue-002
/aphoria-custom-extractor-creator --violation "blocking in async fn" --claim msgqueue-009
/aphoria-custom-extractor-creator --violation "max_queue_size=None" --claim msgqueue-015
/aphoria-custom-extractor-creator --violation "ack_mode=AutoAck" --claim msgqueue-013
/aphoria-custom-extractor-creator --violation "max_requeue_count=None" --claim msgqueue-018
/aphoria-custom-extractor-creator --violation "max_connections=None" --claim msgqueue-003

# Verify:
ls .aphoria/extractors/*.toml | wc -l  # Should be: 8
```

#### Phase 5: Verification Scan (15 min)
```bash
aphoria scan --format json > scan-v2.json
# Result: 8/8 violations detected (100% improvement!)
```

#### Phase 6: Documentation (15 min)
Create `DAY3-SUMMARY.md` with:
- Detection rate v1 vs v2 (0% → 100%)
- Extractors created (8 total)
- Time breakdown
- Learning captured

### How to Verify Correct Execution

After Day 3, these MUST exist:

```bash
# 1. Extractor directory with 8 files
$ ls .aphoria/extractors/*.toml | wc -l
8

# 2. Verification scan
$ ls scan-v2.json
scan-v2.json

# 3. Daily summary
$ ls DAY3-SUMMARY.md
DAY3-SUMMARY.md

# 4. Detection rate improvement
$ jq '.summary.claims_conflict' scan-v1.json
0
$ jq '.summary.claims_conflict' scan-v2.json
8
# Improvement: +8 (0 → 8)
```

**If ANY of these checks fail, Day 3 was not completed correctly. Redo from Phase 4.**

### Why This Mistake Happens

**Root cause: Mental model mismatch**

People think:
- ❌ "Aphoria is a CLI tool you run manually"
- ❌ "Scan shows results, that's the end"
- ❌ "Low detection rate means Aphoria doesn't work"

Reality:
- ✅ "Aphoria is an autonomous learning system"
- ✅ "Low initial detection is EXPECTED, creation phase fixes it"
- ✅ "The flywheel requires LLM to create extractors dynamically"

**Contributing factors:**
1. plan.md Step 3 could be read as optional
2. CLI worked without errors (reinforced wrong model)
3. No pre-flight check to verify skill availability
4. Scan output doesn't suggest next action

### How We're Fixing This

**Documentation updates:**
- ✅ plan.md now emphasizes Step 3 as **REQUIRED**
- ✅ SKILL.md rewritten with 6 explicit phases
- ✅ Pre-flight checks added to verify skill availability
- ✅ Success criteria now includes "8 extractors created"
- ✅ Evidence checklist added (ls commands to verify)

**Product improvements (planned):**
- Scan output will suggest: "Run /aphoria-custom-extractor-creator"
- New CLI command: `aphoria extractors coverage` (show gaps)
- New CLI command: `aphoria dogfood metrics` (track Day 3 progress)
- Pre-flight check command: `aphoria dogfood preflight --day 3`

### Related Issues

- VG-025: No default extractors ship for common patterns
- VG-027: No skill availability check
- VG-028: No example extractor TOML files
- VG-031: No visual diff between scan-v1 and scan-v2

---

## Mistake #2: Creating Extractors with Wrong Subject Format (CRITICAL)

**Severity:** 🚨 CRITICAL - Breaks extractor matching

### What People Do Wrong

Create extractors that run successfully but don't match claims due to incorrect `subject` field:

```toml
# Claim has:
concept_path = "msgqueue/queue/max_size"

# Extractor uses (WRONG):
[extractors.declarative.claim]
subject = "queue/max_size"  # ❌ Missing "msgqueue/" prefix
```

**Result:**
- ✅ Extractors run (no errors)
- ✅ Observations created (+7 observations)
- ❌ 0% detection rate (observations don't match claims)
- ❌ Day 3 still incomplete (can't proceed without working extractors)

---

### Why It's Wrong

**Subject field MUST exactly match claim's concept_path.**

Aphoria uses tail-path matching (last 2 segments), but if the observation path is `queue/max_size` and claim path is `msgqueue/queue/max_size`, the alignment fails because:
- Observation tail: `queue/max_size`
- Claim tail: `queue/max_size`
- But observation is missing the namespace prefix, causing match failures

**Rule:** Copy claim's `concept_path` EXACTLY into extractor's `subject`.

---

### Evidence from msgqueue Dogfood (2026-02-10, Second Attempt)

**What was done:**
- ✅ Day 1-2: Claims authored, code written
- ✅ Day 3 Step 3: Created 7 extractors (fixed from first attempt)
- ✅ Day 3 Step 4: Verification scan ran
- ❌ **Result: 0% detection rate** (same as before creating extractors)

**Extractor subjects used (all WRONG):**
```toml
subject = "queue/max_size"                    # ❌ Should be: msgqueue/queue/max_size
subject = "consumer/prefetch_count"           # ❌ Should be: msgqueue/consumer/prefetch_count
subject = "tls/certificate_validation"        # ❌ Should be: msgqueue/tls/certificate_validation
subject = "async/runtime"                     # ❌ Should be: msgqueue/async/runtime
subject = "consumer/ack_mode"                 # ❌ Should be: msgqueue/consumer/ack_mode
subject = "consumer/requeue_limit"            # ❌ Should be: msgqueue/consumer/requeue_limit
subject = "connection/max_connections"        # ❌ Should be: msgqueue/connection/max_connections
```

**Pattern:** All missing `msgqueue/` prefix.

**Scan results:**
- scan-v1.json: 0/8 violations (0%) - before extractors
- scan-v2.json: 0/8 violations (0%) - after extractors (7 observations, no conflicts)
- Improvement: **0%** (no change despite creating extractors)

**Files that exist but don't work:**
```bash
$ ls .aphoria/extractors/
# (No directory - wrong location, should be in config.toml)

$ grep "subject =" .aphoria/config.toml
subject = "queue/max_size"
subject = "consumer/prefetch_count"
# ... (all missing prefix)

$ grep "concept_path =" .aphoria/claims.toml
concept_path = "msgqueue/queue/max_size"
concept_path = "msgqueue/consumer/prefetch_count"
# ... (all have msgqueue/ prefix)

# Mismatch!
```

---

### What To Do Instead

**Step 1: Copy concept_path from claim EXACTLY**

```bash
# Find your claim's concept_path:
grep "id = \"msgqueue-015\"" -A 1 .aphoria/claims.toml
# Output: concept_path = "msgqueue/queue/max_size"

# Copy this EXACTLY into extractor subject:
subject = "msgqueue/queue/max_size"  # ✅ CORRECT (exact copy)
```

**Step 2: Validate BEFORE scanning**

```bash
# Compare subjects vs concept_paths
grep "subject =" .aphoria/config.toml | sort
grep "concept_path =" .aphoria/claims.toml | sort

# Verify: Every subject should appear in a concept_path
# If subject = "queue/max_size" and no concept_path = "queue/max_size" → WRONG
# Must use full path: "msgqueue/queue/max_size"
```

**Step 3: Test pattern matches code**

```bash
# For each extractor pattern, verify it matches code:
grep -rE 'max_queue_size:\s*None' src/
# Should find: src/config.rs:45: max_queue_size: None

# If no match → pattern is wrong, fix regex
# If matches → pattern is correct, issue is subject field
```

**Step 4: Create extractor with correct format**

```toml
[[extractors.declarative]]
name = "queue_max_size_unbounded"
pattern = 'max_queue_size:\s*None'
languages = ["rust"]

[extractors.declarative.claim]
subject = "msgqueue/queue/max_size"  # ✅ Copied from claim concept_path
predicate = "bounded"
value = false
confidence = 0.95
```

---

### How to Verify Correct Format

After creating extractors, before scanning:

```bash
# 1. Check all subjects
grep "subject =" .aphoria/config.toml

# 2. Check all concept_paths
grep "concept_path =" .aphoria/claims.toml

# 3. Verify alignment
# For each subject, there MUST be a claim with matching concept_path
# "msgqueue/queue/max_size" → MUST exist in claims.toml

# Example check:
for subject in $(grep "subject =" .aphoria/config.toml | cut -d'"' -f2); do
  if ! grep -q "concept_path = \"$subject\"" .aphoria/claims.toml; then
    echo "❌ MISMATCH: $subject not found in claims"
  else
    echo "✅ OK: $subject"
  fi
done
```

Expected output: All subjects show `✅ OK`

---

### Debug 0% Detection After Creating Extractors

If you created extractors and detection rate is still 0%:

**Step 1: Were observations created?**
```bash
jq '.observations | length' scan-results-v2.json
# Expected: > 0
```

- If **0 observations** → Pattern doesn't match code (test with `grep -rE "pattern" src/`)
- If **>0 observations** → Observations don't match claims (subject mismatch, proceed to Step 2)

**Step 2: Compare observation paths vs claim paths**
```bash
# Observation paths (what extractors created):
jq '.observations[].concept_path' scan-results-v2.json | sort -u

# Claim paths (what exists in claims.toml):
grep "concept_path =" .aphoria/claims.toml | cut -d'"' -f2 | sort -u

# Compare: Do observation paths END with same tail as claim paths?
```

**Example mismatch:**
- Observation: `queue/max_size`
- Claim: `msgqueue/queue/max_size`
- Tail: Both have `queue/max_size` (last 2 segments)
- **Problem:** Observation missing `msgqueue/` prefix

**Fix:** Update extractor subject to match claim's full path.

---

### How We're Fixing This

**Documentation updates (2026-02-10):**
- ✅ Created `docs/extractors/declarative-extractors.md` with subject field reference
- ✅ Created `docs/examples/extractors/timeout-zero-example.md` with worked example
- ✅ Updated plan.md Day 3 Step 3 to show manual extractor format
- ✅ Updated plan.md Day 3 Step 4 with debug workflow for 0% detection
- ✅ Added validation steps (grep subject vs concept_path)

**Product improvements (planned):**
- VG-DAY3-001: `aphoria scan --show-observations` to see observation concept paths
- VG-DAY3-002: Better error messages when subject doesn't match any claim
- VG-DAY3-003: `aphoria extractors validate` to check subject alignment
- VG-DAY3-004: `aphoria extractors test NAME --file path.rs` for single-extractor testing

---

### Comparison: Two Failure Modes

| Attempt | Extractors Created | Detection Rate | Failure Reason |
|---------|-------------------|----------------|----------------|
| **First** | 0 | 0% | Skipped Phase 4 entirely (docs unclear) |
| **Second** | 7 | 0% | Wrong subject format (undocumented requirement) |
| **Correct** | 7 | 100% | Subject matches concept_path exactly |

**Progress:** First fix got team to CREATE extractors. Second fix ensures extractors WORK.

---

## Mistake #3: Treating Aphoria as Static Scanner

**Severity:** 🚨 CRITICAL - Fundamental misunderstanding

### What People Think

"Aphoria is a CLI tool you run to check code, like a linter."

### What It Actually Is

"Aphoria is an autonomous learning system where LLM skills drive the workflow, and CLI is a debug interface."

### How This Manifests

**Wrong workflow:**
1. Run `aphoria scan`
2. Look at output
3. Done

**Correct workflow:**
1. Use `/aphoria-suggest` to discover patterns (Day 1)
2. Use `/aphoria-claims` to author claims (Day 1)
3. Write code with violations (Day 2)
4. Run `aphoria scan` to get baseline (Day 3)
5. Use `/aphoria-custom-extractor-creator` to close gaps (Day 3)
6. Re-scan to verify (Day 3)
7. Fix violations progressively (Day 4)

**Key difference:** LLM skills (`/aphoria-*`) are PRIMARY, CLI is FALLBACK.

### How to Avoid

Before starting dogfood:
1. Verify skills are available: `/help | grep aphoria`
2. Understand: Skills drive the process, not manual CLI
3. Reference: Read `applications/aphoria/vision.md` sections on autonomous workflows

---

## Mistake #4: Not Verifying Prerequisites

**Severity:** ⚠️ MAJOR - Wastes time mid-execution

### What People Do Wrong

Start Day 3 without checking:
- Is `/aphoria-custom-extractor-creator` skill available?
- Are inline markers present in code?
- Does code compile?

Result: Workflow fails mid-execution, must backtrack.

### What To Do Instead

**Pre-flight check at start of EACH day:**

**Day 1:**
```bash
/help | grep aphoria-suggest           # Skill available?
/help | grep aphoria-claims            # Skill available?
ls .aphoria/config.toml                # Config exists?
```

**Day 2:**
```bash
ls src/                                # Project structure exists?
cargo check                            # Dependencies resolve?
```

**Day 3:**
```bash
/help | grep aphoria-custom-extractor-creator  # Skill available?
grep -r "@aphoria:claim" src/ | wc -l          # Markers present?
cargo check                                    # Code compiles?
```

**Day 4:**
```bash
ls scan-v2.json                        # Verification scan exists?
jq '.summary.claims_conflict' scan-v2.json  # Violations detected?
```

**If any check fails, STOP and fix before proceeding.**

---

## Mistake #5: Skipping Gap Analysis

**Severity:** ⚠️ MAJOR - Can't prioritize what to fix

### What People Do Wrong

See "20/22 claims MISSING" in scan output, don't investigate why.

### What To Do Instead

Create gap analysis table after scan-v1:

```markdown
## Gap Analysis

| Violation | Location | Marker Present? | Observation Found? | Extractor Exists? | Action |
|-----------|----------|----------------|-------------------|------------------|--------|
| timeout=0 | config.rs:20 | ✅ | ❌ | ❌ | Create extractor |
| prefetch=MAX | config.rs:33 | ✅ | ❌ | ❌ | Create extractor |
| verify_tls=false | config.rs:68 | ✅ | ❌ | ❌ | Create extractor |
... (8 total)

**Summary:**
- Total violations: 8
- Markers present: 8/8
- Observations found: 0/8
- Extractors needed: 8

**Root cause:** Zero extractors exist for msgqueue domain patterns.
```

This makes it clear WHAT to create in Phase 4.

---

## Mistake #6: No Time Tracking

**Severity:** ℹ️ MINOR - Can't optimize workflow

### What People Do Wrong

Don't track time per phase, can't calculate efficiency.

### What To Do Instead

Track time in daily summary:

```markdown
## Time Breakdown

| Phase | Target | Actual | Delta |
|-------|--------|--------|-------|
| Pre-flight check | 5 min | 3 min | -2 min ✅ |
| Baseline scan | 15 min | 12 min | -3 min ✅ |
| Gap analysis | 15 min | 18 min | +3 min |
| Extractor creation | 30 min | 35 min | +5 min |
| Verification scan | 15 min | 10 min | -5 min ✅ |
| Documentation | 15 min | 12 min | -3 min ✅ |
| **Total** | **95 min** | **90 min** | **-5 min ✅** |
```

This shows where time is spent and where to optimize.

---

## Mistake #7: No Detection Rate Calculation

**Severity:** ℹ️ MINOR - Can't prove success

### What People Do Wrong

Scan results exist but no explicit detection rate calculated.

### What To Do Instead

```markdown
## Detection Rate

| Scan | Violations Detected | Total Violations | Detection Rate | Target | Pass? |
|------|---------------------|-----------------|----------------|--------|-------|
| v1 (baseline) | 0 | 8 | 0% | N/A | Baseline |
| v2 (after extractors) | 8 | 8 | 100% | ≥90% | ✅ PASS |

**Improvement:** +100 percentage points (0% → 100%)

**Root Cause of Initial 0%:** Zero extractors existed for msgqueue patterns. After creating 8 extractors, 100% detection achieved.
```

---

## Mistake #8: Not Comparing to httpclient

**Severity:** ℹ️ MINOR - Misses learning opportunity

### What People Do Wrong

Don't reference why httpclient succeeded (100% detection on first scan) where msgqueue failed (0% detection).

### What To Do Instead

```markdown
## Comparison: httpclient vs msgqueue

| Metric | httpclient | msgqueue | Why Different? |
|--------|-----------|----------|----------------|
| Initial detection | 7/7 (100%) | 0/8 (0%) | httpclient had extractors from corpus |
| Extractors created | 0 (existed) | 8 (new) | msgqueue required new extractors |
| Final detection | 7/7 (100%) | 8/8 (100%) | After creation, both 100% |

**Lesson:** First dogfood in new domain requires extractor creation (Day 3 Phase 4). Subsequent dogfoods reuse extractors (corpus compounding).

**Corpus growth:** These 8 msgqueue extractors will benefit:
- Next msgqueue project (100% detection on first scan)
- Any async Rust project (timeout, TLS, blocking-in-async patterns reusable)
```

---

## Checklist: "Did I Do Day 3 Correctly?"

Use this checklist after completing Day 3:

### ✅ Pre-Flight (5 min)
- [ ] Verified skill availability (`/help | grep aphoria-custom-extractor-creator`)
- [ ] Verified inline markers present (`grep -r "@aphoria:claim" src/`)
- [ ] Verified code compiles (`cargo check`)

### ✅ Baseline Scan (15 min)
- [ ] Ran `aphoria scan > scan-v1.json`
- [ ] Reviewed results (expected: low detection rate for new domain)

### ✅ Gap Analysis (15 min)
- [ ] Created gap table (violations vs observations)
- [ ] Identified which extractors are needed (8 total)

### ✅ Extractor Creation (30 min) **[CRITICAL]**
- [ ] Invoked `/aphoria-custom-extractor-creator` 8 times (one per violation)
- [ ] Created `.aphoria/extractors/` directory
- [ ] 8 .toml files exist in extractors/ directory
- [ ] Each extractor file has: name, pattern, concept_path, predicate, value

### ✅ Verification Scan (15 min)
- [ ] Ran `aphoria scan > scan-v2.json`
- [ ] Compared v1 vs v2 (detection rate improved from 0% to ≥90%)
- [ ] Zero false positives

### ✅ Documentation (15 min)
- [ ] Created `DAY3-SUMMARY.md`
- [ ] Included metrics table (v1 vs v2 detection rate)
- [ ] Listed all 8 extractors created
- [ ] Documented time per phase
- [ ] Described learning captured (patterns identified)

### Evidence Check

Run these commands to verify:

```bash
# 1. Extractor files exist
ls .aphoria/extractors/*.toml | wc -l
# Expected: 8

# 2. Verification scan exists
ls scan-v2.json
# Expected: file exists

# 3. Daily summary exists
ls DAY3-SUMMARY.md
# Expected: file exists

# 4. Detection improved
jq '.summary.claims_conflict' scan-v1.json  # Should be: 0
jq '.summary.claims_conflict' scan-v2.json  # Should be: 8
# Improvement: +8 violations detected
```

**If ANY check fails, Day 3 is incomplete. Redo from Phase 4 (extractor creation).**

---

## How to Recover from Mistakes

### If You Skipped Day 3 Extractor Creation

**Symptoms:**
- No `.aphoria/extractors/` directory
- Only `scan-v1.json` exists (no v2)
- No `DAY3-SUMMARY.md`
- Detection rate still 0%

**Recovery:**
1. Load skill: `/aphoria-custom-extractor-creator`
2. Create extractors (Phase 4 of Day 3)
3. Run verification scan (Phase 5)
4. Write summary (Phase 6)
5. Mark Day 3 as complete

**Time:** ~1 hour

### If You Forgot Pre-Flight Check

**Symptoms:**
- Workflow failed mid-execution
- Skill not found errors
- Code doesn't compile

**Recovery:**
1. Run pre-flight check now
2. Fix blockers (load skills, fix compilation)
3. Resume from where you stopped

**Time:** ~15 minutes

### If You Have No Gap Analysis

**Symptoms:**
- Can't explain why violations were missed
- Don't know which extractors to create

**Recovery:**
1. Review `scan-v1.json`
2. Create gap table (template above)
3. Proceed with extractor creation

**Time:** ~15 minutes

---

## Prevention: What We Fixed

### Documentation Updates (2026-02-10)

✅ **plan.md:**
- Day 3 Step 3 now says **[REQUIRED - DO NOT SKIP]**
- Added pre-flight check section
- Broke Day 3 into 6 explicit phases
- Added evidence checklist (ls commands)

✅ **SKILL.md (aphoria-dogfood):**
- Rewrote Day 3 section with emphasis on extractor creation
- Added Phase 1-6 breakdown
- Added warning: "THIS IS THE CORE FLYWHEEL STEP"

✅ **This document (dogfooding-common-mistakes.md):**
- Documents msgqueue failure as cautionary example
- Provides recovery procedures
- Includes verification checklists

### Product Improvements (Planned)

🔜 **Scan output enhancement:**
- Show "Run /aphoria-custom-extractor-creator" suggestion when claims are MISSING

🔜 **New CLI commands:**
- `aphoria extractors coverage` - Show which extractors exist vs needed
- `aphoria dogfood metrics --day 3` - Calculate detection rate improvement
- `aphoria scan diff scan-v1.json scan-v2.json` - Visual diff

🔜 **Pre-flight validation:**
- `aphoria dogfood preflight --day 3` - Verify prerequisites before starting

---

## Summary

**Most Critical Mistake:** Skipping Day 3 extractor creation (breaks flywheel completely)

**How to Avoid:**
1. Understand Aphoria is autonomous learning system (not static scanner)
2. Follow plan.md Day 3 phases 1-6 WITHOUT skipping any
3. Verify evidence after Day 3 (8 extractors, scan-v2.json, DAY3-SUMMARY.md)
4. Run pre-flight check before each day

**How to Verify Success:**
```bash
ls .aphoria/extractors/*.toml | wc -l  # Must be: 8
ls scan-v2.json                        # Must exist
ls DAY3-SUMMARY.md                     # Must exist
```

**If ANY check fails, Day 3 is incomplete.**

---

**Last Updated:** 2026-02-10 (after msgqueue dogfood Day 3 failure)