stemedb/docs/app-concepts/consumer-health.md

Consumer Health Intelligence: Application Layer Guide

Vertical: Consumer-facing health information Use Case: consumer-health-intelligence.md Status: Design spec — implementation not started

This guide describes the application layer components needed to build a Consumer Health Intelligence platform on Episteme. It covers what you build, not what Episteme provides.

---

Architecture Overview

┌─────────────────────────────────────────────────────────────────────────────┐
│                           CONSUMER HEALTH APP                                │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐     │
│  │   PubMed     │  │   Reddit     │  │    FAERS     │  │    FDA       │     │
│  │   Crawler    │  │   Crawler    │  │   Crawler    │  │   Crawler    │     │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘     │
│         │                 │                 │                 │              │
│         └────────────┬────┴────────┬────────┴─────────┬───────┘              │
│                      │             │                  │                      │
│                      ▼             ▼                  ▼                      │
│              ┌───────────────────────────────────────────────┐               │
│              │           NLP Extraction Pipeline              │               │
│              │  (claim identification, confidence scoring)    │               │
│              └───────────────────────┬───────────────────────┘               │
│                                      │                                       │
│                                      ▼                                       │
│              ┌───────────────────────────────────────────────┐               │
│              │          Source-Class Classifier               │               │
│              │  (tier assignment: 0=regulatory → 6=media)     │               │
│              └───────────────────────┬───────────────────────┘               │
│                                      │                                       │
│                                      ▼                                       │
│              ┌───────────────────────────────────────────────┐               │
│              │          Metadata Enrichment Service           │               │
│              │  (DOI lookup, journal info, engagement stats)  │               │
│              └───────────────────────┬───────────────────────┘               │
│                                      │                                       │
│                                      ▼                                       │
│              ┌───────────────────────────────────────────────┐               │
│              │              Agent Wallet                      │               │
│              │  (key management, Ed25519 signing)             │               │
│              └───────────────────────┬───────────────────────┘               │
│                                      │                                       │
└──────────────────────────────────────┼───────────────────────────────────────┘
                                       │ POST /assert
                                       ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                              EPISTEME DATABASE                               │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐           │
│  │   WAL   │  │   KV    │  │ Indexes │  │ Lenses  │  │   MV    │           │
│  └─────────┘  └─────────┘  └─────────┘  └─────────┘  └─────────┘           │
└─────────────────────────────────────────────────────────────────────────────┘
                                       │ GET /query
                                       ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                           CONSUMER HEALTH APP                                │
│                                                                              │
│              ┌───────────────────────────────────────────────┐               │
│              │           Background Gardener                  │               │
│              │  (cluster detection, escalation assertions)    │               │
│              └───────────────────────────────────────────────┘               │
│                                                                              │
│              ┌───────────────────────────────────────────────┐               │
│              │           LLM Summary Generator                │               │
│              │  (plain-language synthesis of query results)   │               │
│              └───────────────────────────────────────────────┘               │
│                                                                              │
│              ┌───────────────────────────────────────────────┐               │
│              │           Disagreement Dashboard               │               │
│              │  (web UI for consumers)                        │               │
│              └───────────────────────────────────────────────┘               │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

---

Component 1: Ingestion Pipeline

1.1 Data Sources

Source	API/Method	Volume Estimate	Source Class
PubMed/biorxiv	NCBI E-utilities API	~5,000 papers for GLP-1	Tier 1-2
ClinicalTrials.gov	ClinicalTrials.gov API	~800 registered trials	Tier 1-2
FDA Labels	DailyMed API / EDGAR scraping	~50 documents	Tier 0
FAERS	openFDA API	~50,000 reports for semaglutide	Tier 3
Reddit	Reddit API (r/Ozempic, r/loseit)	~500,000 posts/comments	Tier 5
Patient Forums	Web scraping	~100,000 posts	Tier 5
News	News API	~10,000 articles	Tier 6
Social Media	Platform APIs / Firehose	~1,000,000+ mentions	Tier 6

1.2 Crawler Implementation

Each source needs a dedicated crawler. Here's the pattern:

// crawler/pubmed/crawler.go

type PubMedCrawler struct {
    apiKey    string
    baseURL   string
    rateLimit time.Duration
}

func (c *PubMedCrawler) FetchArticles(query string, since time.Time) ([]RawArticle, error) {
    // Call NCBI E-utilities API
    // Respect rate limits (3 req/sec with API key)
    // Return raw article data
}

type RawArticle struct {
    PMID        string
    Title       string
    Abstract    string
    Authors     []string
    Journal     string
    DOI         string
    PubDate     time.Time
    MeSHTerms   []string
    StudyDesign string // RCT, observational, meta-analysis, etc.
}

1.3 NLP Extraction Pipeline

Transforms raw documents into structured claims.

// extraction/pipeline.go

type ClaimExtractor struct {
    llm         LLMClient   // Claude, GPT-4, or local model
    promptStore PromptStore // Domain-specific extraction prompts
}

type ExtractedClaim struct {
    Subject    string  // e.g., "semaglutide/adverse-effects/gastroparesis"
    Predicate  string  // e.g., "risk_level"
    Object     string  // e.g., "No statistically significant increase"
    Confidence float32 // Extraction confidence, not source authority
    Evidence   string  // Quote from source supporting the claim
}

func (e *ClaimExtractor) Extract(doc RawDocument) ([]ExtractedClaim, error) {
    // Use LLM to identify claims in the document
    // Map claims to subject/predicate ontology
    // Assign extraction confidence based on clarity
    // Return structured claims
}

Prompt Engineering: The extraction prompt is critical. It must:

• Define the claim ontology (what subjects/predicates are valid)
• Distinguish claims from context
• Extract supporting evidence for provenance
• Rate extraction confidence (not source authority)

Example prompt structure:

You are extracting health claims from a medical document.

Ontology:
- Subjects: {drug}/adverse-effects/{condition}, {drug}/efficacy/{outcome}
- Predicates: risk_level, incidence_rate, relative_risk, clinical_significance

For each claim found, return:
- subject: The specific topic (e.g., "semaglutide/adverse-effects/gastroparesis")
- predicate: What aspect is being claimed
- object: The claim value (use exact quotes where possible)
- confidence: Your confidence in the extraction accuracy (0.0-1.0)
- evidence: The exact text supporting this claim

Document:
{document_text}

1.4 Source-Class Classifier

Assigns tier based on source metadata.

// classification/source_class.go

type SourceClassifier struct {
    rules []ClassificationRule
}

type ClassificationRule struct {
    Name      string
    Predicate func(SourceMetadata) bool
    Tier      uint8
}

func (c *SourceClassifier) Classify(meta SourceMetadata) uint8 {
    for _, rule := range c.rules {
        if rule.Predicate(meta) {
            return rule.Tier
        }
    }
    return 6 // Default to lowest tier
}

// Default pharma rules
var PharmaRules = []ClassificationRule{
    {
        Name: "FDA Label",
        Predicate: func(m SourceMetadata) bool {
            return m.Source == "dailymed" || m.Source == "fda.gov"
        },
        Tier: 0,
    },
    {
        Name: "Peer-Reviewed RCT",
        Predicate: func(m SourceMetadata) bool {
            return m.DOI != "" && m.StudyDesign == "RCT" && m.PeerReviewed
        },
        Tier: 1,
    },
    {
        Name: "Meta-Analysis",
        Predicate: func(m SourceMetadata) bool {
            return m.StudyDesign == "meta-analysis" && m.PeerReviewed
        },
        Tier: 1,
    },
    {
        Name: "Observational Study",
        Predicate: func(m SourceMetadata) bool {
            return m.DOI != "" && (m.StudyDesign == "cohort" || m.StudyDesign == "case-control")
        },
        Tier: 2,
    },
    {
        Name: "FAERS Report",
        Predicate: func(m SourceMetadata) bool {
            return m.Source == "openfda" || m.Source == "faers"
        },
        Tier: 3,
    },
    {
        Name: "Case Report",
        Predicate: func(m SourceMetadata) bool {
            return m.StudyDesign == "case-report"
        },
        Tier: 4,
    },
    {
        Name: "Reddit",
        Predicate: func(m SourceMetadata) bool {
            return strings.Contains(m.URL, "reddit.com")
        },
        Tier: 5,
    },
    {
        Name: "Patient Forum",
        Predicate: func(m SourceMetadata) bool {
            return m.Source == "patient-forum"
        },
        Tier: 5,
    },
    {
        Name: "News",
        Predicate: func(m SourceMetadata) bool {
            return m.Source == "news-api"
        },
        Tier: 6,
    },
    {
        Name: "Social Media",
        Predicate: func(m SourceMetadata) bool {
            return m.Source == "tiktok" || m.Source == "instagram" || m.Source == "twitter"
        },
        Tier: 6,
    },
}

1.5 Metadata Enrichment

Adds structured metadata before submission.

// enrichment/service.go

type EnrichmentService struct {
    crossref CrossRefClient
    pubmed   PubMedClient
    reddit   RedditClient
}

type EnrichedMetadata struct {
    // Academic sources
    Journal     string `json:"journal,omitempty"`
    DOI         string `json:"doi,omitempty"`
    SampleSize  int    `json:"sample_size,omitempty"`
    StudyDesign string `json:"study_design,omitempty"`

    // Social sources
    Platform    string `json:"platform,omitempty"`
    Subreddit   string `json:"subreddit,omitempty"`
    Upvotes     int    `json:"upvotes,omitempty"`
    Replies     int    `json:"replies,omitempty"`

    // Sentiment (computed during extraction)
    Sentiment   string  `json:"sentiment,omitempty"`
    SentimentPolarity float32 `json:"sentiment_polarity,omitempty"`
}

func (e *EnrichmentService) Enrich(source SourceMetadata) (*EnrichedMetadata, error) {
    meta := &EnrichedMetadata{}

    if source.DOI != "" {
        // CrossRef lookup for journal, citations
        crossrefData, _ := e.crossref.GetByDOI(source.DOI)
        meta.Journal = crossrefData.ContainerTitle
    }

    if strings.Contains(source.URL, "reddit.com") {
        // Reddit API for engagement metrics
        redditData, _ := e.reddit.GetPost(source.URL)
        meta.Upvotes = redditData.Score
        meta.Replies = redditData.NumComments
        meta.Subreddit = redditData.Subreddit
    }

    return meta, nil
}

1.6 Assembly & Submission

Combines all components into a signed assertion.

// ingestion/submit.go

type AssertionSubmitter struct {
    episteme    EpistemeClient
    wallet      AgentWallet
    classifier  SourceClassifier
    enricher    EnrichmentService
}

func (s *AssertionSubmitter) Submit(claim ExtractedClaim, source SourceMetadata) error {
    // 1. Classify source tier
    tier := s.classifier.Classify(source)

    // 2. Enrich metadata
    meta, _ := s.enricher.Enrich(source)
    metaJSON, _ := json.Marshal(meta)

    // 3. Compute source hash
    sourceHash := blake3.Sum256([]byte(source.URL + source.Content))

    // 4. Build assertion
    assertion := &CreateAssertionRequest{
        Subject:        claim.Subject,
        Predicate:      claim.Predicate,
        Object:         ObjectText(claim.Object),
        Confidence:     claim.Confidence,
        SourceClass:    &tier,
        SourceHash:     hex.EncodeToString(sourceHash[:]),
        SourceMetadata: string(metaJSON),
        Lifecycle:      "Proposed",
    }

    // 5. Sign with agent key
    signature, _ := s.wallet.Sign(assertion)
    assertion.Signatures = []SignatureDTO{signature}

    // 6. Submit to Episteme
    return s.episteme.Assert(assertion)
}

---

Component 2: Background Gardener

Monitors the knowledge graph for emerging signals.

2.1 Cluster Detection

// gardener/cluster_detector.go

type ClusterDetector struct {
    episteme   EpistemeClient
    thresholds ClusterThresholds
}

type ClusterThresholds struct {
    MinAssertions   int           // Minimum assertions to be considered a cluster
    TimeWindow      time.Duration // Window for growth rate calculation
    GrowthRateHigh  float64       // Assertions per month considered "high"
    TierGapTrigger  bool          // Trigger if Tier 5+ exists but Tier 1-2 doesn't
}

type DetectedCluster struct {
    Subject        string
    Predicate      string
    Tier5Count     int
    Tier1Count     int
    GrowthRate     float64 // Assertions per month
    HasClinicalGap bool    // Tier 5+ exists, no Tier 1-2
    EarliestReport time.Time
}

func (d *ClusterDetector) Scan() ([]DetectedCluster, error) {
    // Query Episteme for all subject+predicate pairs
    // For each pair, count assertions by tier
    // Calculate growth rate over time window
    // Flag clusters that meet thresholds
}

2.2 Escalation Assertion Generation

// gardener/escalation.go

func (g *Gardener) GenerateEscalation(cluster DetectedCluster) *CreateAssertionRequest {
    meta := map[string]interface{}{
        "trigger":          "cluster_threshold",
        "tier_5_count":     cluster.Tier5Count,
        "tier_1_count":     cluster.Tier1Count,
        "growth_rate":      fmt.Sprintf("%.0f/month", cluster.GrowthRate),
        "clinical_gap":     cluster.HasClinicalGap,
        "earliest_report":  cluster.EarliestReport.Format(time.RFC3339),
    }
    metaJSON, _ := json.Marshal(meta)

    return &CreateAssertionRequest{
        Subject:        cluster.Subject,
        Predicate:      "escalation_signal",
        Object:         ObjectText("Anecdotal cluster detected"),
        Confidence:     0.6, // Moderate confidence in the signal
        SourceClass:    ptr(uint8(255)), // Meta-tier for system-generated
        SourceMetadata: string(metaJSON),
        Lifecycle:      "UnderReview",
    }
}

2.3 Scheduled Tasks

// gardener/scheduler.go

func (g *Gardener) RunSchedule(ctx context.Context) {
    // Every hour: Scan for new clusters
    go g.runEvery(ctx, 1*time.Hour, g.ScanClusters)

    // Every day: Decay trust ranks
    go g.runEvery(ctx, 24*time.Hour, g.DecayTrustRanks)

    // Every week: Generate summary reports
    go g.runEvery(ctx, 7*24*time.Hour, g.GenerateReports)
}

func (g *Gardener) DecayTrustRanks(ctx context.Context) error {
    return g.episteme.Post("/v1/admin/decay-trust-ranks", nil)
}

---

Component 3: Disagreement Dashboard

3.1 Query Patterns

The dashboard needs these Episteme queries:

// dashboard/queries.go

// Get layered consensus for a topic
func (d *Dashboard) GetLayeredConsensus(subject string) (*LayeredResponse, error) {
    return d.episteme.Query(&QueryParams{
        Subject: subject,
        Lens:    "LayeredConsensus",
    })
}

// Get conflict map using Skeptic lens
func (d *Dashboard) GetConflictMap(subject string) (*SkepticResponse, error) {
    return d.episteme.Query(&QueryParams{
        Subject: subject,
        Lens:    "Skeptic",
    })
}

// Get historical state at a point in time
func (d *Dashboard) GetHistoricalState(subject string, asOf time.Time) (*QueryResponse, error) {
    return d.episteme.Query(&QueryParams{
        Subject: subject,
        Lens:    "LayeredConsensus",
        AsOf:    asOf.Unix(),
    })
}

// Get changes since last visit
func (d *Dashboard) GetChangesSince(subject string, since time.Time) (*QueryResponse, error) {
    return d.episteme.Query(&QueryParams{
        Subject: subject,
        Lens:    "LayeredConsensus",
        Since:   since.Unix(),
    })
}

3.2 Response Transformation

// dashboard/transform.go

type ConsumerView struct {
    Topic           string              `json:"topic"`
    Summary         string              `json:"summary"` // LLM-generated
    ConflictScore   float32             `json:"conflict_score"`
    TierPositions   []TierPosition      `json:"tier_positions"`
    ResolvedTopics  []ResolvedTopic     `json:"resolved"`
    ActiveDisputes  []ActiveDispute     `json:"active_disputes"`
    EmergingSignals []EmergingSignal    `json:"emerging_signals"`
    LastUpdated     time.Time           `json:"last_updated"`
}

type TierPosition struct {
    TierName      string  `json:"tier_name"` // "Clinical Evidence", "Patient Community"
    TierNumber    uint8   `json:"tier_number"`
    Position      string  `json:"position"`
    Confidence    float32 `json:"confidence"`
    AssertionCount int    `json:"assertion_count"`
}

func TransformForConsumer(layered *LayeredResponse, skeptic *SkepticResponse) *ConsumerView {
    view := &ConsumerView{
        ConflictScore: layered.OverallConflictScore,
    }

    for _, tier := range layered.Tiers {
        view.TierPositions = append(view.TierPositions, TierPosition{
            TierName:       tierToName(tier.Tier),
            TierNumber:     tier.Tier,
            Position:       summarizePosition(tier.Winner),
            Confidence:     tier.Winner.Confidence,
            AssertionCount: tier.CandidatesCount,
        })
    }

    // Categorize by conflict level
    for _, topic := range skeptic.Topics {
        if topic.ConflictScore < 0.2 {
            view.ResolvedTopics = append(view.ResolvedTopics, ...)
        } else if topic.ConflictScore < 0.7 {
            view.ActiveDisputes = append(view.ActiveDisputes, ...)
        } else {
            view.EmergingSignals = append(view.EmergingSignals, ...)
        }
    }

    return view
}

func tierToName(tier uint8) string {
    names := map[uint8]string{
        0: "Regulatory",
        1: "Clinical Evidence",
        2: "Real-World Evidence",
        3: "Pharmacovigilance",
        4: "Clinical Anecdote",
        5: "Patient Community",
        6: "Media",
    }
    return names[tier]
}

3.3 LLM Summary Generation

// dashboard/summary.go

type SummaryGenerator struct {
    llm LLMClient
}

func (g *SummaryGenerator) GenerateSummary(view *ConsumerView) (string, error) {
    prompt := fmt.Sprintf(`
Summarize the following health topic for a consumer in 2-3 sentences.
Be factual. Acknowledge uncertainty where it exists. Do not give medical advice.

Topic: %s
Conflict Score: %.2f (0 = agreement, 1 = max disagreement)

Tier Positions:
%s

Output a concise, balanced summary.
`, view.Topic, view.ConflictScore, formatTierPositions(view.TierPositions))

    return g.llm.Complete(prompt)
}

Example output:

"Clinical trials show low incidence of gastroparesis with semaglutide; post-marketing reports and patient communities report higher rates. The FDA added gastroparesis to the label in January 2024. There is moderate disagreement between clinical evidence and real-world reports."

---

Component 4: Agent Wallet

Manages signing keys for the ingestion pipeline.

4.1 Key Storage

// wallet/wallet.go

type AgentWallet struct {
    keyPath    string
    privateKey ed25519.PrivateKey
    publicKey  ed25519.PublicKey
}

func NewAgentWallet(keyPath string) (*AgentWallet, error) {
    // Load or generate Ed25519 keypair
    // Store private key securely (file permissions, encryption at rest)
}

func (w *AgentWallet) Sign(assertion *CreateAssertionRequest) (*SignatureDTO, error) {
    // Serialize assertion for signing
    message := fmt.Sprintf("%s:%s", assertion.Subject, assertion.Predicate)
    sig := ed25519.Sign(w.privateKey, []byte(message))

    return &SignatureDTO{
        AgentID:   hex.EncodeToString(w.publicKey),
        Signature: hex.EncodeToString(sig),
        Timestamp: time.Now().Unix(),
    }, nil
}

4.2 Multi-Agent Setup

For different source types, use different agent identities:

// wallet/multi.go

type MultiAgentWallet struct {
    agents map[string]*AgentWallet
}

func (m *MultiAgentWallet) SignAs(agentName string, assertion *CreateAssertionRequest) (*SignatureDTO, error) {
    agent, ok := m.agents[agentName]
    if !ok {
        return nil, fmt.Errorf("unknown agent: %s", agentName)
    }
    return agent.Sign(assertion)
}

// Usage:
// - "pubmed-crawler" agent for academic sources
// - "reddit-crawler" agent for social sources
// - "gardener" agent for escalation assertions

---

Deployment Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        Kubernetes Cluster                        │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐              │
│  │   PubMed    │  │   Reddit    │  │   FAERS     │   Crawlers   │
│  │   Crawler   │  │   Crawler   │  │   Crawler   │   (CronJob)  │
│  │   Pod       │  │   Pod       │  │   Pod       │              │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘              │
│         │                │                │                      │
│         └────────────────┼────────────────┘                      │
│                          │                                       │
│                          ▼                                       │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │                    Extraction Service                      │  │
│  │                    (Deployment, 3 replicas)                │  │
│  └───────────────────────────────────────────────────────────┘  │
│                          │                                       │
│                          ▼                                       │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │                    Episteme API                            │  │
│  │                    (Deployment, 3 replicas)                │  │
│  └───────────────────────────────────────────────────────────┘  │
│                          │                                       │
│                          ▼                                       │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │                    Dashboard API                           │  │
│  │                    (Deployment, 2 replicas)                │  │
│  └───────────────────────────────────────────────────────────┘  │
│                          │                                       │
│                          ▼                                       │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │                    Dashboard UI                            │  │
│  │                    (Static, CDN)                           │  │
│  └───────────────────────────────────────────────────────────┘  │
│                                                                  │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │                    Background Gardener                     │  │
│  │                    (CronJob, hourly)                       │  │
│  └───────────────────────────────────────────────────────────┘  │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

---

Checklist: What You Need to Build

Crawlers (one per source)

• PubMed/biorxiv crawler (NCBI E-utilities)
• ClinicalTrials.gov crawler
• FDA DailyMed scraper
• openFDA FAERS consumer
• Reddit API consumer
• Patient forum scrapers
• News API consumer
• Social media sampler

Extraction Pipeline

• LLM-based claim extractor
• Subject/predicate ontology for pharma domain
• Extraction prompt library
• Confidence calibration

Classification & Enrichment

• Source-class classifier with pharma rules
• CrossRef integration for DOI lookup
• PubMed integration for study metadata
• Reddit API for engagement metrics
• Sentiment analysis model

Infrastructure

• Agent wallet with secure key storage
• Multi-agent identity management
• Rate limiting for external APIs
• Retry logic and error handling

Gardener

• Cluster detection algorithm
• Escalation assertion generator
• TrustRank decay scheduler
• Alert/notification system

Dashboard

• Query orchestration layer
• Response transformation
• LLM summary generator
• React/Vue frontend
• Time-travel UI
• Change notification system

---

See Also

• Use Case: Consumer Health Intelligence — Full scenario walkthrough
• App Concepts Index — General application layer patterns
• Roadmap: Phase 3 — Database features needed
• ADK-Go Integration — Agent framework integration