Production Readiness Verification
Systematic verification checklist for deploying StemeDB in production environments.
Quick Reference
| Category |
Status |
Last Verified |
| Crash Recovery |
✅ Pass |
2026-02-05 |
| Signature Verification |
✅ Pass |
2026-02-05 |
| End-to-End Pipeline |
✅ Pass |
2026-02-05 |
| Load Testing |
✅ Tooling ready |
Run ./scripts/run-load-test.sh |
| API Security |
❌ Not done |
- |
| Backup/Restore |
❌ Not done |
- |
| Observability |
⚠️ Partial |
- |
Verification Areas
1. Data Integrity & Durability
| Check |
Command |
Expected |
| WAL crash recovery |
cargo test -p stemedb-ingest test_crash_recovery |
All pass |
| No duplicate assertions |
cargo test -p stemedb-ingest test_p0_crash_recovery |
All pass |
| Cursor checkpoint |
cargo test -p stemedb-ingest test_cursor |
All pass |
2. Signature Verification
| Check |
Command |
Expected |
| v1 signatures (legacy) |
cargo test -p stemedb-ingest test_ingest_assertion |
Pass |
| v2 signatures (enterprise) |
Pharma-ingest with real keys |
All assertions accepted |
| Invalid signature rejection |
cargo test -p stemedb-ingest test_rejects_invalid |
Pass |
| Unsigned assertion rejection |
cargo test -p stemedb-ingest test_rejects_unsigned |
Pass |
3. End-to-End Pipeline
| Check |
Command |
Expected |
| API server starts |
cargo run --bin stemedb-api |
Binds to :18180 |
| Assertion ingestion |
POST /v1/assert |
Returns hash |
| Query returns data |
GET /v1/query?subject=X |
Returns assertions |
| Skeptic conflict analysis |
GET /v1/skeptic?subject=X&predicate=Y |
Returns conflict_score |
| Health check |
GET /v1/health |
assertions_count > 0 |
4. Load Testing
Tool: cmd/load-test (Go-based with native Ed25519 signing)
| Scenario |
Command |
Target |
Status |
| Baseline latency |
--scenario baseline |
10K assertions, p99 < 200ms |
✅ Ready |
| Sustained writes |
--scenario sustained |
1K/sec for 1 hour, p99 < 200ms |
✅ Ready |
| Concurrent readers |
--scenario concurrent |
100 readers, <2x degradation |
✅ Ready |
Quick Start:
# Run all scenarios (5 min sustained by default)
./scripts/run-load-test.sh
# Run full 1-hour sustained test
LOAD_TEST_DURATION=1h ./scripts/run-load-test.sh
# Run specific scenario
./scripts/run-load-test.sh --scenario baseline
go run ./cmd/load-test --api-url http://localhost:18180 --scenario sustained --duration 10m
Prerequisites:
- Set
STEMEDB_METER_ENABLED=false for accurate sustained test results
- Ensure ~10-20GB disk space for 1-hour tests (~3.6M assertions)
- Results saved to
uat/production-readiness/results/
5. API Security (TODO)
| Check |
Implementation |
Status |
| Authentication |
JWT or API keys |
Not implemented |
| Rate limiting |
Per-client limits |
Not implemented |
| Input validation |
Oversized payloads rejected |
Partial |
| TLS in transit |
HTTPS termination |
External (nginx/LB) |
6. Backup & Restore (TODO)
| Check |
Procedure |
Status |
| Point-in-time recovery |
WAL replay |
Not documented |
| WAL archival |
S3/GCS upload |
Not implemented |
| Restore test |
Full recovery |
Not tested |
7. Observability (Partial)
| Check |
Implementation |
Status |
| Structured logs |
tracing crate |
✅ Implemented |
| Metrics endpoint |
/metrics Prometheus |
Not implemented |
| Distributed tracing |
OpenTelemetry |
Not implemented |
| Alerting |
WAL lag, errors |
Not implemented |
Running Full Verification
# 1. Run all unit tests
cargo test --workspace --lib
# 2. Start fresh API server
rm -rf /tmp/stemedb-prod-test && mkdir -p /tmp/stemedb-prod-test
STEMEDB_DATA_DIR=/tmp/stemedb-prod-test cargo run --bin stemedb-api &
sleep 5
# 3. Run pharma-ingest (tests v2 signatures)
cargo run -p stemedb-ontology --bin pharma-ingest -- --with-conflicts
# 4. Verify endpoints
curl http://localhost:18180/v1/health
curl "http://localhost:18180/v1/query?subject=Semaglutide"
curl "http://localhost:18180/v1/skeptic?subject=Semaglutide&predicate=nausea_rate"
# 5. Kill and restart (crash recovery test)
pkill -9 -f stemedb-api
STEMEDB_DATA_DIR=/tmp/stemedb-prod-test cargo run --bin stemedb-api &
sleep 3
curl http://localhost:18180/v1/health # Should show same assertion count
Results Archive
Date-stamped verification results:
| Date |
Report |
Summary |
| 2026-02-05 |
wal-sync-fix.md |
WAL segment cache fix, all tests pass |
Related
157dbbb9eb