8f6506b70a
Major additions: - Staged scanning modes (working tree, staged, committed) with git integration - Drift detection for baseline vs current state comparisons - Hosted API handlers for policy CRUD operations via StemeDB API - stemedb-ontology crate with domain definitions and medical extractors - Consumer health vertical UAT scenarios (GLP-1, gastroparesis, etc.) - Aphoria development skill documentation Code organization: - Split large files into focused modules to stay under 500-line limit - Extracted config tests, episteme helpers/drift/aliases, API helpers Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
177 lines
4.6 KiB
Markdown
177 lines
4.6 KiB
Markdown
# UAT: Semantic Decay (Knowledge Half-Life)
|
|
|
|
**Date:** YYYY-MM-DD
|
|
**Feature:** Source-Aware Decay
|
|
**Status:** [ ] PASS / [ ] FAIL / [ ] BLOCKED
|
|
|
|
## Scenario
|
|
|
|
Medical knowledge decays at different rates based on source class:
|
|
- **Regulatory (Tier 0):** Never decays
|
|
- **Clinical (Tier 1):** 730-day half-life (2 years)
|
|
- **Anecdotal (Tier 5):** 30-day half-life
|
|
|
|
A 200-day-old Clinical study should have effective confidence ~0.45x original.
|
|
A 200-day-old Reddit post should have effective confidence ~0.01x original (essentially expired).
|
|
|
|
## Acceptance Criteria
|
|
|
|
| Criterion | Expected | Met? |
|
|
|-----------|----------|------|
|
|
| Regulatory never decays | conf unchanged at 200 days | [ ] |
|
|
| Clinical decay correct | ~0.45x at 200 days | [ ] |
|
|
| Anecdotal decay correct | ~0.01x at 200 days | [ ] |
|
|
| Decay applied at query | Not at write time | [ ] |
|
|
|
|
## Test Matrix
|
|
|
|
| Step | Action | Expected | Actual | Status |
|
|
|------|--------|----------|--------|--------|
|
|
| 1 | Ingest Regulatory assertion | Hash returned | | [ ] |
|
|
| 2 | Ingest Clinical assertion | Hash returned | | [ ] |
|
|
| 3 | Ingest Anecdotal assertion | Hash returned | | [ ] |
|
|
| 4 | Query with decay (200 days) | Decayed confidences | | [ ] |
|
|
| 5 | Verify Regulatory unchanged | 0.95 | | [ ] |
|
|
| 6 | Verify Clinical decayed | ~0.43 | | [ ] |
|
|
| 7 | Verify Anecdotal expired | ~0.01 | | [ ] |
|
|
|
|
## Decay Formula
|
|
|
|
Confidence decay follows exponential half-life:
|
|
```
|
|
effective_conf = original_conf * (0.5 ^ (age_days / half_life_days))
|
|
```
|
|
|
|
| Source Class | Half-Life (days) | 200-day decay factor |
|
|
|--------------|------------------|----------------------|
|
|
| Regulatory | Infinite | 1.0 |
|
|
| Clinical | 730 | 0.826 |
|
|
| Observational | 365 | 0.682 |
|
|
| Expert | 180 | 0.464 |
|
|
| Community | 90 | 0.214 |
|
|
| Anecdotal | 30 | 0.010 |
|
|
|
|
## Setup Commands
|
|
|
|
```bash
|
|
# Start StemeDB
|
|
cargo run --bin stemedb-api &
|
|
sleep 2
|
|
```
|
|
|
|
## Test Commands
|
|
|
|
### Step 1: Ingest Regulatory Assertion (FDA)
|
|
|
|
```bash
|
|
# Use a timestamp 200 days ago
|
|
TIMESTAMP_200_DAYS_AGO=$(($(date +%s) - 17280000))
|
|
|
|
curl -X POST http://localhost:18180/v1/assertions \
|
|
-H "Content-Type: application/json" \
|
|
-d "{
|
|
\"subject\": \"Semaglutide\",
|
|
\"predicate\": \"approved_for\",
|
|
\"object\": {\"Text\": \"Type2Diabetes\"},
|
|
\"confidence\": 0.95,
|
|
\"source_class\": \"Regulatory\",
|
|
\"timestamp\": $TIMESTAMP_200_DAYS_AGO
|
|
}"
|
|
```
|
|
|
|
**Expected:** Hash returned
|
|
**Actual:**
|
|
**Status:** [ ]
|
|
|
|
### Step 2: Ingest Clinical Assertion
|
|
|
|
```bash
|
|
curl -X POST http://localhost:18180/v1/assertions \
|
|
-H "Content-Type: application/json" \
|
|
-d "{
|
|
\"subject\": \"Semaglutide\",
|
|
\"predicate\": \"hba1c_reduction_percent\",
|
|
\"object\": {\"Number\": 1.5},
|
|
\"confidence\": 0.95,
|
|
\"source_class\": \"Clinical\",
|
|
\"timestamp\": $TIMESTAMP_200_DAYS_AGO
|
|
}"
|
|
```
|
|
|
|
**Expected:** Hash returned
|
|
**Actual:**
|
|
**Status:** [ ]
|
|
|
|
### Step 3: Ingest Anecdotal Assertion (Reddit)
|
|
|
|
```bash
|
|
curl -X POST http://localhost:18180/v1/assertions \
|
|
-H "Content-Type: application/json" \
|
|
-d "{
|
|
\"subject\": \"Semaglutide\",
|
|
\"predicate\": \"user_experience\",
|
|
\"object\": {\"Text\": \"Lost 20 lbs in 3 months\"},
|
|
\"confidence\": 0.95,
|
|
\"source_class\": \"Anecdotal\",
|
|
\"timestamp\": $TIMESTAMP_200_DAYS_AGO
|
|
}"
|
|
```
|
|
|
|
**Expected:** Hash returned
|
|
**Actual:**
|
|
**Status:** [ ]
|
|
|
|
### Step 4: Query with Decay Lens
|
|
|
|
```bash
|
|
# Query all Semaglutide assertions with decay applied
|
|
curl "http://localhost:18180/v1/query?subject=Semaglutide&lens=decay"
|
|
```
|
|
|
|
**Expected:** Three assertions with decayed effective_confidence values
|
|
**Actual:**
|
|
**Status:** [ ]
|
|
|
|
### Step 5: Verify Regulatory Unchanged
|
|
|
|
From Step 4 response, find the `approved_for` assertion.
|
|
|
|
**Expected:** effective_confidence = 0.95 (unchanged)
|
|
**Actual:**
|
|
**Status:** [ ]
|
|
|
|
### Step 6: Verify Clinical Decayed
|
|
|
|
From Step 4 response, find the `hba1c_reduction_percent` assertion.
|
|
|
|
**Expected:** effective_confidence ~ 0.79 (0.95 * 0.826)
|
|
**Actual:**
|
|
**Status:** [ ]
|
|
|
|
### Step 7: Verify Anecdotal Expired
|
|
|
|
From Step 4 response, find the `user_experience` assertion.
|
|
|
|
**Expected:** effective_confidence ~ 0.01 (essentially expired)
|
|
**Actual:**
|
|
**Status:** [ ]
|
|
|
|
## Sign-Off Checklist
|
|
|
|
- [ ] All three source classes ingested
|
|
- [ ] Decay applied at query time (not write time)
|
|
- [ ] Regulatory sources never decay
|
|
- [ ] Clinical sources decay correctly
|
|
- [ ] Anecdotal sources expire quickly
|
|
- [ ] Decay factors match documented half-lives
|
|
|
|
## Notes
|
|
|
|
*The "73-day half-life" mentioned in the GLP-1 Living Review use case refers to the overall effective knowledge decay for mixed-source queries. Individual source classes have different rates.*
|
|
|
|
---
|
|
|
|
**Tester:**
|
|
**Date:**
|
|
**Result:**
|