jordan b3e8a9a058 feat: Multi-application expansion with chaos testing and community UI

Major additions:
- Community Next.js app (port 18187) for browsing claims with API docs
- stemedb-chaos crate: Fault injection, chaos testing, CRDT properties
- Latent ingestion system: Reddit/FDA ingesters with ADK-Go agents
- Disputed claims handling: Manual review workflows and validation
- Aphoria security scanner: New extractors (SQL injection, command
  injection, weak crypto, TLS version), policy-based ignores, UAT reports
- Docker infrastructure: Dockerfile, docker-compose.yml for full stack
- VulnBank demo: Intentionally vulnerable multi-language test corpus

SDK & API enhancements:
- Source registry handlers for tracking data provenance
- Metrics endpoint
- Skeptic filtering improvements

Code quality:
- Split 14 large files (>500 lines) into focused modules
- All files now under 500-line limit per project guidelines

Documentation:
- Chaos testing guide, circuit breakers, observability docs
- Phase 7 UAT documentation updates
- Martin Kleppmann technical writer agent

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

2026-02-04 01:24:14 -07:00

3.5 KiB

Raw Blame History

The Meter (Economic Throttling)

Overview

The Meter implements per-agent per-hour quota enforcement using a Token Bucket algorithm. This prevents runaway agents from exhausting system resources while allowing bursty behavior.

Cost Model

Operation	Base Cost	Notes
Assert	10 tokens	Writing truth is expensive
Vote	1 token	Voting is cheap
Query	5 tokens	+ 1 per Lens applied
Payload	+1/KB	Prevents spamming massive blobs

Default quota: 10,000 tokens per agent per hour.

Components

QuotaStore (`stemedb-storage/src/quota_store.rs`)

Storage abstraction for quota tracking.

Key Types:

QuotaRecord - Per-agent per-hour usage record
CostConfig - Operation cost configuration
QuotaCheckResult - Result of quota check (allowed, remaining, reset_at)
OperationType - Assert, Vote, or Query

Storage Layout:

QT:{agent_id}:{hour} -> QuotaRecord (serialized)
QL:{agent_id} -> u64 (custom limit override)

Key Methods:

async fn check_and_record(&self, agent_id, operation, payload_bytes, timestamp) -> QuotaCheckResult;
async fn get_quota_status(&self, agent_id, timestamp) -> QuotaCheckResult;
async fn set_quota_limit(&self, agent_id, limit) -> Result<()>;

MeterLayer (`stemedb-api/src/middleware/meter.rs`)

Tower middleware that intercepts requests and enforces quotas.

Request Flow:

Extract X-Agent-Id header (hex-encoded 32-byte public key)
Determine operation type from path
Calculate cost based on operation + payload size
Check quota and reject with 429 if exceeded
Record cost and forward request
Add quota headers to response

Headers:

Header	Direction	Description
`X-Agent-Id`	Request	Agent's Ed25519 public key (hex, 64 chars)
`X-Quota-Remaining`	Response	Remaining tokens in current window
`X-Quota-Limit`	Response	Total tokens allowed per hour
`X-Quota-Reset`	Response	Unix timestamp when window resets

API Endpoints

GET /v1/meter/quota Check quota status for an agent.

Query params:

agent_id - Hex-encoded agent public key

Response:

{
  "agent_id": "...",
  "remaining": 9500,
  "limit": 10000,
  "reset_at": 1705317600,
  "used": 500,
  "window_start": 1705314000
}

POST /v1/meter/quota/limit Set custom quota limit (admin operation).

Request:

{
  "agent_id": "...",
  "limit": 50000
}

Configuration

Environment variable: STEMEDB_METER_ENABLED

true (default): Enable economic throttling
false or 0: Disable (all requests pass through)

Bypass Paths

These paths bypass metering:

/v1/health
/swagger-ui/*
/api-docs/*

Usage Example

# Check quota before operations
curl "http://localhost:18180/v1/meter/quota?agent_id=$(xxd -p -l 32 /dev/urandom)"

# Make request with agent ID
curl -X POST http://localhost:18180/v1/assert \
  -H "Content-Type: application/json" \
  -H "X-Agent-Id: 0102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20" \
  -d '{"subject": "test", ...}'

# Response headers include quota info
# X-Quota-Remaining: 9989
# X-Quota-Limit: 10000
# X-Quota-Reset: 1705317600

Implementation Notes

Quota windows are hour-aligned (truncated to the start of the hour)
Quotas reset automatically at the start of each hour
On storage errors, requests are allowed (fail open for availability)
Anonymous requests (no X-Agent-Id) bypass metering
Token bucket allows burst behavior while enforcing long-term limits

3.5 KiB Raw Blame History