# Enriched Corpus Patterns - Making Community Patterns Actionable

## Problem Statement

**Current State:** Community corpus shows raw statistics without context.

Example:
```
code://rust/*/core/auction/imports/std
imported: true
1 project, 1 observation
```

**User Confusion:**
- ❓ What does this mean?
- ❓ Is this good or bad?
- ❓ Should I do anything?

**Expected State:** Users assumed corpus would provide best practices and actionable guidance, like a security scanner or linter.

## User Experience Goal

Transform patterns from "confusing statistics" to "actionable insights":

### Example 1: Security Best Practice
```
┌─────────────────────────────────────────────────────────────┐
│ 🔒 TLS Certificate Verification                             │
├─────────────────────────────────────────────────────────────┤
│ Pattern: TLS cert verification is enabled                   │
│ Prevalence: 847 of 892 projects (95%)                      │
│ Verdict: ✅ RECOMMENDED                                     │
│                                                             │
│ Why it matters:                                             │
│ Certificate verification prevents man-in-the-middle attacks │
│                                                             │
│ Authority: RFC 5246, OWASP A02:2021                        │
│ Learn more: https://owasp.org/tls-guide                    │
└─────────────────────────────────────────────────────────────┘
```

### Example 2: Anti-Pattern
```
┌─────────────────────────────────────────────────────────────┐
│ 🚨 MD5 Hash Usage                                           │
├─────────────────────────────────────────────────────────────┤
│ Pattern: MD5 used for cryptographic hashing                │
│ Prevalence: 47 of 892 projects (5%)                        │
│ Verdict: ❌ DEPRECATED (trend: ↓ -2% this month)           │
│                                                             │
│ Why this is dangerous:                                      │
│ MD5 is cryptographically broken. Collisions can be         │
│ generated in seconds, allowing attackers to forge          │
│ signatures or bypass integrity checks.                      │
│                                                             │
│ Authority: NIST deprecated 2010                            │
│ Replace with: SHA-256, SHA-3, or BLAKE3                    │
│ Migration guide: https://...                                │
└─────────────────────────────────────────────────────────────┘
```

### Example 3: Emerging Pattern
```
┌─────────────────────────────────────────────────────────────┐
│ 📈 BLAKE3 Adoption                                          │
├─────────────────────────────────────────────────────────────┤
│ Pattern: BLAKE3 used for hashing                            │
│ Prevalence: 34 of 892 projects (4%)                        │
│ Verdict: ℹ️  EMERGING (trend: ↑ +3% this quarter)          │
│                                                             │
│ Why this is interesting:                                    │
│ BLAKE3 is faster than SHA-256 while maintaining security.  │
│ Growing adoption in performance-critical applications.      │
│                                                             │
│ Trade-offs:                                                 │
│ ✅ 10x faster than SHA-256                                  │
│ ✅ Parallel computation support                             │
│ ⚠️  Less mature ecosystem than SHA-2 family                │
└─────────────────────────────────────────────────────────────┘
```

### Example 4: Noise (Hide by Default)
```
┌─────────────────────────────────────────────────────────────┐
│ ℹ️  Standard Library Import                                 │
├─────────────────────────────────────────────────────────────┤
│ Pattern: std library imported                               │
│ Prevalence: 891 of 892 projects (99.9%)                    │
│ Verdict: ⚪ COMMON (not actionable)                         │
│                                                             │
│ This is a standard pattern with no security or              │
│ architectural implications.                                  │
└─────────────────────────────────────────────────────────────┘
```

## Data Model Requirements

### Current PatternAggregate (Minimal)
```rust
pub struct PatternAggregate {
    pub subject: String,              // "code://rust/*/crypto/hash/algorithm"
    pub predicate: String,             // "value"
    pub value_hash: String,            // BLAKE3 hash
    pub value_display: String,         // "md5"
    pub project_count: u64,            // 47
    pub observation_count: u64,        // 89
    pub first_seen: u64,               // Unix timestamp
    pub last_seen: u64,                // Unix timestamp
}
```

### Required Enrichment Fields
```rust
pub struct EnrichedPattern {
    // Existing fields
    pub subject: String,
    pub predicate: String,
    pub value_display: String,
    pub project_count: u64,
    pub observation_count: u64,
    pub first_seen: u64,
    pub last_seen: u64,

    // NEW: Enrichment metadata
    pub title: Option<String>,                    // "MD5 Hash Usage"
    pub category: Option<String>,                 // "security" | "architecture" | "performance"
    pub verdict: Option<String>,                  // "recommended" | "deprecated" | "emerging" | "common"
    pub severity: Option<String>,                 // "critical" | "high" | "medium" | "low" | "info"
    pub explanation: Option<String>,              // "MD5 is cryptographically broken..."
    pub why_dangerous: Option<String>,            // "Collisions can be generated..."
    pub authority_sources: Vec<String>,           // ["NIST", "RFC-5246"]
    pub recommendations: Vec<String>,             // ["Use SHA-256", "Use BLAKE3"]
    pub learn_more_url: Option<String>,           // Documentation link
    pub related_patterns: Vec<String>,            // Similar patterns
    pub interestingness_score: f32,               // 0.0-1.0 for sorting

    // NEW: Trend data (Phase 4)
    pub trend: Option<TrendData>,
}

pub struct TrendData {
    pub direction: String,           // "up" | "down" | "stable"
    pub percentage_change: f32,      // -2.0 = "down 2%"
    pub time_period: String,         // "month" | "quarter" | "year"
    pub velocity: f32,               // Rate of change
}
```

## Implementation Phases

### Phase 1: Minimum Viable Enrichment (1-2 days)

**Goal:** Add basic enrichment so patterns are understandable.

**Changes:**

#### 1. StemeDB Storage
File: `crates/stemedb-storage/src/pattern_aggregate_store/mod.rs`

```rust
// Add optional enrichment fields to PatternAggregate
pub struct PatternAggregate {
    // ... existing fields ...

    // Enrichment metadata (backwards compatible)
    pub category: Option<String>,
    pub verdict: Option<String>,
    pub explanation: Option<String>,
    pub authority_source: Option<String>,
}
```

**Storage:** Serialize as JSON blob in existing KV store. No migration needed (all fields are `Option<T>`).

#### 2. Aphoria Extractors
File: `applications/aphoria/src/extractors/trait.rs`

Add method to Extractor trait:
```rust
pub trait Extractor {
    // ... existing methods ...

    /// Provide metadata about patterns this extractor recognizes.
    ///
    /// Used to enrich community corpus patterns with explanations
    /// and verdicts.
    fn pattern_metadata(&self) -> HashMap<String, PatternMetadata> {
        HashMap::new() // Default: no metadata
    }
}

pub struct PatternMetadata {
    pub category: String,        // "security" | "architecture" | "performance"
    pub verdict: String,         // "recommended" | "deprecated" | "emerging"
    pub severity: String,        // "critical" | "high" | "medium" | "low" | "info"
    pub explanation: String,     // Human-readable explanation
    pub authority: Option<String>, // "RFC-5246" | "OWASP-A02" | "NIST"
}
```

#### 3. Example: Crypto Hash Extractor
File: `applications/aphoria/src/extractors/crypto_hash.rs`

```rust
impl Extractor for CryptoHashExtractor {
    fn pattern_metadata(&self) -> HashMap<String, PatternMetadata> {
        let mut map = HashMap::new();

        // MD5 is deprecated
        map.insert(
            "crypto/hash/algorithm::md5".to_string(),
            PatternMetadata {
                category: "security".to_string(),
                verdict: "deprecated".to_string(),
                severity: "high".to_string(),
                explanation: "MD5 is cryptographically broken. Collisions can be generated in seconds.".to_string(),
                authority: Some("NIST deprecated 2010".to_string()),
            }
        );

        // SHA1 is deprecated
        map.insert(
            "crypto/hash/algorithm::sha1".to_string(),
            PatternMetadata {
                category: "security".to_string(),
                verdict: "deprecated".to_string(),
                severity: "high".to_string(),
                explanation: "SHA1 is cryptographically broken. Use SHA-256 or better.".to_string(),
                authority: Some("NIST deprecated 2015".to_string()),
            }
        );

        // SHA256 is recommended
        map.insert(
            "crypto/hash/algorithm::sha256".to_string(),
            PatternMetadata {
                category: "security".to_string(),
                verdict: "recommended".to_string(),
                severity: "info".to_string(),
                explanation: "SHA-256 is secure and widely supported.".to_string(),
                authority: Some("NIST FIPS 180-4".to_string()),
            }
        );

        // BLAKE3 is emerging
        map.insert(
            "crypto/hash/algorithm::blake3".to_string(),
            PatternMetadata {
                category: "performance".to_string(),
                verdict: "emerging".to_string(),
                severity: "info".to_string(),
                explanation: "BLAKE3 is faster than SHA-256 with equivalent security.".to_string(),
                authority: None,
            }
        );

        map
    }
}
```

#### 4. Pattern Enricher Service
File: `applications/aphoria/src/corpus/enricher.rs` (NEW)

```rust
use std::collections::HashMap;
use crate::extractors::{Extractor, PatternMetadata};

/// Enriches raw patterns with metadata from extractors.
pub struct PatternEnricher {
    /// Metadata from all registered extractors.
    metadata_registry: HashMap<String, PatternMetadata>,
}

impl PatternEnricher {
    /// Create enricher from registered extractors.
    pub fn from_extractors(extractors: &[Box<dyn Extractor>]) -> Self {
        let mut registry = HashMap::new();

        for extractor in extractors {
            for (pattern_key, metadata) in extractor.pattern_metadata() {
                registry.insert(pattern_key, metadata);
            }
        }

        Self { metadata_registry: registry }
    }

    /// Enrich a pattern with metadata if available.
    pub fn enrich(&self, subject: &str, predicate: &str, value: &str) -> Option<PatternMetadata> {
        // Try exact match first: "crypto/hash/algorithm::md5"
        let exact_key = format!("{}::{}::{}", subject, predicate, value);
        if let Some(meta) = self.metadata_registry.get(&exact_key) {
            return Some(meta.clone());
        }

        // Try predicate + value: "crypto/hash/algorithm::md5"
        let predicate_key = format!("{}::{}", predicate, value);
        if let Some(meta) = self.metadata_registry.get(&predicate_key) {
            return Some(meta.clone());
        }

        // No metadata found
        None
    }

    /// Compute interestingness score for sorting.
    ///
    /// High scores = more interesting/actionable patterns.
    /// Low scores = common/noise patterns to hide.
    pub fn compute_interestingness(
        &self,
        pattern: &PatternAggregate,
        metadata: Option<&PatternMetadata>,
    ) -> f32 {
        let mut score = 0.5; // Default: neutral

        // Deprecated/critical patterns are highly interesting
        if let Some(meta) = metadata {
            match meta.verdict.as_str() {
                "deprecated" => score += 0.4,
                "emerging" => score += 0.2,
                "recommended" => score += 0.1,
                _ => {}
            }

            match meta.severity.as_str() {
                "critical" => score += 0.3,
                "high" => score += 0.2,
                "medium" => score += 0.1,
                _ => {}
            }
        }

        // Very common patterns (>90% adoption) are less interesting
        // unless they're deprecated
        let adoption_rate = pattern.project_count as f32 / 1000.0; // Assuming ~1000 projects
        if adoption_rate > 0.9 {
            if metadata.map_or(true, |m| m.verdict != "deprecated") {
                score -= 0.3; // Common + not-deprecated = noise
            }
        }

        // Very rare patterns (<3 projects) are less interesting
        if pattern.project_count < 3 {
            score -= 0.2;
        }

        score.clamp(0.0, 1.0)
    }
}
```

#### 5. Send Enriched Metadata with Observations
File: `applications/aphoria/src/hosted.rs`

```rust
// Update CommunityObservationDto to include enrichment
pub struct CommunityObservationDto {
    pub subject: String,
    pub predicate: String,
    pub object: CommunityValueDto,
    pub confidence: f32,
    pub anon_hash: String,
    pub timestamp_hour: u64,

    // NEW: Enrichment metadata
    pub category: Option<String>,
    pub verdict: Option<String>,
    pub explanation: Option<String>,
    pub authority_source: Option<String>,
}

// In assertion_to_community_dto(), look up metadata
fn assertion_to_community_dto(
    assertion: &Assertion,
    project_id: &str,
    enricher: &PatternEnricher,
) -> CommunityObservationDto {
    // ... existing code ...

    // Look up enrichment metadata
    let metadata = enricher.enrich(&subject, &assertion.predicate, &value_str);

    CommunityObservationDto {
        // ... existing fields ...
        category: metadata.as_ref().map(|m| m.category.clone()),
        verdict: metadata.as_ref().map(|m| m.verdict.clone()),
        explanation: metadata.as_ref().map(|m| m.explanation.clone()),
        authority_source: metadata.as_ref().and_then(|m| m.authority.clone()),
    }
}
```

#### 6. Dashboard UI Updates
File: `applications/aphoria-dashboard/src/app/corpus/page.tsx`

Changes:
- Group patterns by category (Security, Architecture, Performance)
- Sort by interestingness score (hide noise)
- Show verdict badges (✅ ❌ ℹ️ 📈)
- Display explanation in expandable cards
- Add filter: "Show only actionable patterns"
- Parse concept paths into breadcrumbs for readability

**Result:** Users see "MD5 is deprecated (NIST 2010)" instead of just "md5: true"

---

### Phase 2: Pattern Rules Engine (2-3 days)

**Goal:** Admins can define custom pattern interpretations for domain-specific needs.

**Use Case:** A company has internal standards (e.g., "All services MUST use gRPC on port 50051"). They want to define this as a pattern rule and check compliance.

#### 1. StemeDB: Pattern Rules Table
File: `crates/stemedb-storage/src/pattern_rules_store.rs` (NEW)

```rust
pub struct PatternRule {
    pub rule_id: String,                    // "internal-grpc-port"
    pub pattern_matcher: PatternMatcher,    // Regex for matching patterns
    pub metadata: PatternMetadata,          // Enrichment to apply
    pub source: String,                     // "extractor" | "admin" | "llm"
    pub created_at: u64,
}

pub struct PatternMatcher {
    pub subject_pattern: Option<Regex>,    // Match subject path
    pub predicate_pattern: Option<Regex>,  // Match predicate
    pub value_pattern: Option<Regex>,      // Match value
}
```

#### 2. Aphoria: Pattern Rules CLI
File: `applications/aphoria/src/cli/pattern_rules.rs` (NEW)

```bash
# Add a rule
aphoria pattern-rules add \
  --subject "code://*/grpc/port" \
  --value "50051" \
  --verdict "recommended" \
  --explanation "Company standard: gRPC services MUST use port 50051"

# List rules
aphoria pattern-rules list

# Import from TOML
aphoria pattern-rules import rules.toml
```

#### 3. Query-Time Enrichment
File: `crates/stemedb-api/src/handlers/aphoria/report.rs`

```rust
pub async fn get_patterns(
    State(state): State<AppState>,
    Query(params): Query<GetPatternsQuery>,
) -> Result<Json<GetPatternsResponse>> {
    // 1. Fetch raw patterns from storage
    let patterns = pattern_store.get_patterns(params.min_projects).await?;

    // 2. Enrich each pattern by matching against rules
    let enricher = PatternEnricher::new(&state.pattern_rules);
    let enriched = patterns.into_iter()
        .map(|p| enricher.enrich(p))
        .collect();

    // 3. Compute interestingness scores
    let scored = compute_scores(enriched);

    // 4. Filter out noise (score < threshold)
    let actionable = scored.into_iter()
        .filter(|p| p.interestingness_score >= params.min_score.unwrap_or(0.3))
        .collect();

    // 5. Return enriched patterns
    Ok(Json(GetPatternsResponse { patterns: actionable }))
}
```

**Result:** Admins can teach the system about domain-specific patterns without writing code.

---

### Phase 3: Authoritative Corpus Linking (3-4 days)

**Goal:** Automatically connect community patterns to RFC/OWASP authoritative assertions.

**Example:**
- Community pattern: `code://rust/*/tls/cert_verification = true`
- Matches: `rfc://5246/tls/cert_verification = true`
- Inherit: RFC explanation, authority, recommendations automatically

#### 1. Pattern Matching Engine
File: `crates/stemedb-query/src/pattern_matcher.rs` (NEW)

```rust
pub struct AuthorityMatcher {
    authority_corpus: Vec<Assertion>,
}

impl AuthorityMatcher {
    /// Fuzzy match a community pattern to authoritative assertions.
    pub fn match_to_authority(
        &self,
        community_pattern: &Pattern,
    ) -> Option<AuthorityMatch> {
        // Normalize both patterns for comparison
        let normalized = normalize_pattern(community_pattern);

        for authority in &self.authority_corpus {
            if patterns_match(&normalized, authority) {
                return Some(AuthorityMatch {
                    authority_assertion: authority.clone(),
                    confidence: compute_match_confidence(&normalized, authority),
                });
            }
        }

        None
    }
}

fn patterns_match(community: &Pattern, authority: &Assertion) -> bool {
    // Extract tail paths for comparison
    let comm_tail = extract_tail_path(&community.subject);
    let auth_tail = extract_tail_path(&authority.subject);

    // Match if:
    // 1. Tail paths are similar (fuzzy match)
    // 2. Predicates are the same
    // 3. Values are equivalent

    tail_paths_similar(comm_tail, auth_tail)
        && community.predicate == authority.predicate
        && values_equivalent(&community.value, &authority.object)
}
```

**Result:** Patterns show "Authority: RFC 5246" without manual tagging.

---

### Phase 4: Trend Analysis (2-3 days)

**Goal:** Show adoption/abandonment trends over time.

#### 1. Time-Series Aggregation
File: `crates/stemedb-storage/src/pattern_time_series.rs` (NEW)

```rust
pub struct PatternTimeSeries {
    pub pattern_id: String,
    pub week: u64,              // Week number since epoch
    pub project_count: u64,
    pub observation_count: u64,
}
```

#### 2. Trend Computation
```rust
pub fn compute_trend(
    current: &PatternAggregate,
    history: &[PatternTimeSeries],
) -> TrendData {
    // Compare current week to previous week/month
    let last_week = history.last();
    let last_month = history.iter().rev().nth(4); // 4 weeks ago

    let direction = if current.project_count > last_week.project_count {
        "up"
    } else if current.project_count < last_week.project_count {
        "down"
    } else {
        "stable"
    };

    let percentage_change = compute_percentage_change(
        current.project_count,
        last_week.project_count,
    );

    TrendData {
        direction,
        percentage_change,
        time_period: "week",
        velocity: compute_velocity(history),
    }
}
```

**Result:** Users see "↑ +15% adoption this quarter" or "↓ -8% abandonment this month"

---

## Success Metrics

### Phase 1 Success
- Users can understand what patterns mean without external context
- "Actionable patterns" filter shows only interesting patterns
- Dashboard displays category badges and explanations

### Phase 2 Success
- Admins can add custom pattern rules via CLI
- Domain-specific patterns are enriched automatically
- Rules can be imported from TOML files

### Phase 3 Success
- Community patterns automatically link to RFC/OWASP rules
- Authority sources are displayed without manual tagging
- Pattern coverage increases (more patterns have explanations)

### Phase 4 Success
- Trends show emerging vs. dying patterns
- Users can see "what's growing in adoption"
- Historical data informs decision-making

---

## Rollout Plan

1. **Phase 1:** 1-2 days
   - Extend data model
   - Add metadata to 10 key extractors
   - Update dashboard UI
   - Ship to users for feedback

2. **Phase 2:** 2-3 days (based on Phase 1 feedback)
   - Build pattern rules engine
   - Add CLI for rule management
   - Enable admin customization

3. **Phase 3:** 3-4 days
   - Build authority matcher
   - Integrate with authoritative corpus
   - Automatically enrich patterns

4. **Phase 4:** 2-3 days
   - Add time-series storage
   - Compute trends
   - Display in dashboard

**Total Timeline:** ~10-14 days for complete implementation

---

## Open Questions

1. **Which patterns should we enrich first?**
   - Security (crypto, TLS, secrets)?
   - Architecture (async, dependencies)?
   - Performance (algorithms, data structures)?

2. **Should we start with dashboard mockups or data model?**
   - Validate UX first vs. build infrastructure first?

3. **How do we handle pattern ambiguity?**
   - What if a pattern matches multiple rules?
   - Prioritization: Extractor > Admin Rules > Authority > Default?

4. **Should enrichment be at write-time or query-time?**
   - Write-time: Faster queries, but can't update old patterns
   - Query-time: Slower, but always up-to-date with latest rules

5. **Privacy implications of enriched patterns?**
   - Does adding explanations leak information?
   - Should enrichment be client-side only?
