# Production Readiness Verification

Systematic verification checklist for deploying StemeDB in production environments.

## Quick Reference

| Category | Status | Last Verified |
|----------|--------|---------------|
| Crash Recovery | ✅ Pass | 2026-02-05 |
| Signature Verification | ✅ Pass | 2026-02-05 |
| End-to-End Pipeline | ✅ Pass | 2026-02-05 |
| Load Testing | ✅ Tooling ready | Run `./scripts/run-load-test.sh` |
| API Security | ❌ Not done | - |
| Backup/Restore | ❌ Not done | - |
| Observability | ⚠️ Partial | - |

## Verification Areas

### 1. Data Integrity & Durability

| Check | Command | Expected |
|-------|---------|----------|
| WAL crash recovery | `cargo test -p stemedb-ingest test_crash_recovery` | All pass |
| No duplicate assertions | `cargo test -p stemedb-ingest test_p0_crash_recovery` | All pass |
| Cursor checkpoint | `cargo test -p stemedb-ingest test_cursor` | All pass |

### 2. Signature Verification

| Check | Command | Expected |
|-------|---------|----------|
| v1 signatures (legacy) | `cargo test -p stemedb-ingest test_ingest_assertion` | Pass |
| v2 signatures (enterprise) | Pharma-ingest with real keys | All assertions accepted |
| Invalid signature rejection | `cargo test -p stemedb-ingest test_rejects_invalid` | Pass |
| Unsigned assertion rejection | `cargo test -p stemedb-ingest test_rejects_unsigned` | Pass |

### 3. End-to-End Pipeline

| Check | Command | Expected |
|-------|---------|----------|
| API server starts | `cargo run --bin stemedb-api` | Binds to :18180 |
| Assertion ingestion | `POST /v1/assert` | Returns hash |
| Query returns data | `GET /v1/query?subject=X` | Returns assertions |
| Skeptic conflict analysis | `GET /v1/skeptic?subject=X&predicate=Y` | Returns conflict_score |
| Health check | `GET /v1/health` | assertions_count > 0 |

### 4. Load Testing

**Tool:** `cmd/load-test` (Go-based with native Ed25519 signing)

| Scenario | Command | Target | Status |
|----------|---------|--------|--------|
| Baseline latency | `--scenario baseline` | 10K assertions, p99 < 200ms | ✅ Ready |
| Sustained writes | `--scenario sustained` | 1K/sec for 1 hour, p99 < 200ms | ✅ Ready |
| Concurrent readers | `--scenario concurrent` | 100 readers, <2x degradation | ✅ Ready |

**Quick Start:**
```bash
# Run all scenarios (5 min sustained by default)
./scripts/run-load-test.sh

# Run full 1-hour sustained test
LOAD_TEST_DURATION=1h ./scripts/run-load-test.sh

# Run specific scenario
./scripts/run-load-test.sh --scenario baseline
go run ./cmd/load-test --api-url http://localhost:18180 --scenario sustained --duration 10m
```

**Prerequisites:**
- Set `STEMEDB_METER_ENABLED=false` for accurate sustained test results
- Ensure ~10-20GB disk space for 1-hour tests (~3.6M assertions)
- Results saved to `uat/production-readiness/results/`

### 5. API Security (TODO)

| Check | Implementation | Status |
|-------|----------------|--------|
| Authentication | JWT or API keys | Not implemented |
| Rate limiting | Per-client limits | Not implemented |
| Input validation | Oversized payloads rejected | Partial |
| TLS in transit | HTTPS termination | External (nginx/LB) |

### 6. Backup & Restore (TODO)

| Check | Procedure | Status |
|-------|-----------|--------|
| Point-in-time recovery | WAL replay | Not documented |
| WAL archival | S3/GCS upload | Not implemented |
| Restore test | Full recovery | Not tested |

### 7. Observability (Partial)

| Check | Implementation | Status |
|-------|----------------|--------|
| Structured logs | `tracing` crate | ✅ Implemented |
| Metrics endpoint | `/metrics` Prometheus | Not implemented |
| Distributed tracing | OpenTelemetry | Not implemented |
| Alerting | WAL lag, errors | Not implemented |

## Running Full Verification

```bash
# 1. Run all unit tests
cargo test --workspace --lib

# 2. Start fresh API server
rm -rf /tmp/stemedb-prod-test && mkdir -p /tmp/stemedb-prod-test
STEMEDB_DATA_DIR=/tmp/stemedb-prod-test cargo run --bin stemedb-api &
sleep 5

# 3. Run pharma-ingest (tests v2 signatures)
cargo run -p stemedb-ontology --bin pharma-ingest -- --with-conflicts

# 4. Verify endpoints
curl http://localhost:18180/v1/health
curl "http://localhost:18180/v1/query?subject=Semaglutide"
curl "http://localhost:18180/v1/skeptic?subject=Semaglutide&predicate=nausea_rate"

# 5. Kill and restart (crash recovery test)
pkill -9 -f stemedb-api
STEMEDB_DATA_DIR=/tmp/stemedb-prod-test cargo run --bin stemedb-api &
sleep 3
curl http://localhost:18180/v1/health  # Should show same assertion count
```

## Results Archive

Date-stamped verification results:

| Date | Report | Summary |
|------|--------|---------|
| 2026-02-05 | [wal-sync-fix.md](./results/2026-02-05-wal-sync-fix.md) | WAL segment cache fix, all tests pass |

## Related

- [UAT Report Template](../how-to.md)
- [Consumer Health UAT](../consumer-health/README.md)
- [Production Readiness Facts](../../ai-lookup/features/production-readiness.md)