b3e8a9a058
Major additions: - Community Next.js app (port 18187) for browsing claims with API docs - stemedb-chaos crate: Fault injection, chaos testing, CRDT properties - Latent ingestion system: Reddit/FDA ingesters with ADK-Go agents - Disputed claims handling: Manual review workflows and validation - Aphoria security scanner: New extractors (SQL injection, command injection, weak crypto, TLS version), policy-based ignores, UAT reports - Docker infrastructure: Dockerfile, docker-compose.yml for full stack - VulnBank demo: Intentionally vulnerable multi-language test corpus SDK & API enhancements: - Source registry handlers for tracking data provenance - Metrics endpoint - Skeptic filtering improvements Code quality: - Split 14 large files (>500 lines) into focused modules - All files now under 500-line limit per project guidelines Documentation: - Chaos testing guide, circuit breakers, observability docs - Phase 7 UAT documentation updates - Martin Kleppmann technical writer agent Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
132 lines
3.5 KiB
Markdown
132 lines
3.5 KiB
Markdown
# The Meter (Economic Throttling)
|
|
|
|
## Overview
|
|
|
|
The Meter implements per-agent per-hour quota enforcement using a Token Bucket algorithm. This prevents runaway agents from exhausting system resources while allowing bursty behavior.
|
|
|
|
## Cost Model
|
|
|
|
| Operation | Base Cost | Notes |
|
|
|-----------|-----------|-------|
|
|
| Assert | 10 tokens | Writing truth is expensive |
|
|
| Vote | 1 token | Voting is cheap |
|
|
| Query | 5 tokens | + 1 per Lens applied |
|
|
| Payload | +1/KB | Prevents spamming massive blobs |
|
|
|
|
Default quota: **10,000 tokens per agent per hour**.
|
|
|
|
## Components
|
|
|
|
### QuotaStore (`stemedb-storage/src/quota_store.rs`)
|
|
|
|
Storage abstraction for quota tracking.
|
|
|
|
**Key Types:**
|
|
- `QuotaRecord` - Per-agent per-hour usage record
|
|
- `CostConfig` - Operation cost configuration
|
|
- `QuotaCheckResult` - Result of quota check (allowed, remaining, reset_at)
|
|
- `OperationType` - Assert, Vote, or Query
|
|
|
|
**Storage Layout:**
|
|
```
|
|
QT:{agent_id}:{hour} -> QuotaRecord (serialized)
|
|
QL:{agent_id} -> u64 (custom limit override)
|
|
```
|
|
|
|
**Key Methods:**
|
|
```rust
|
|
async fn check_and_record(&self, agent_id, operation, payload_bytes, timestamp) -> QuotaCheckResult;
|
|
async fn get_quota_status(&self, agent_id, timestamp) -> QuotaCheckResult;
|
|
async fn set_quota_limit(&self, agent_id, limit) -> Result<()>;
|
|
```
|
|
|
|
### MeterLayer (`stemedb-api/src/middleware/meter.rs`)
|
|
|
|
Tower middleware that intercepts requests and enforces quotas.
|
|
|
|
**Request Flow:**
|
|
1. Extract `X-Agent-Id` header (hex-encoded 32-byte public key)
|
|
2. Determine operation type from path
|
|
3. Calculate cost based on operation + payload size
|
|
4. Check quota and reject with 429 if exceeded
|
|
5. Record cost and forward request
|
|
6. Add quota headers to response
|
|
|
|
**Headers:**
|
|
| Header | Direction | Description |
|
|
|--------|-----------|-------------|
|
|
| `X-Agent-Id` | Request | Agent's Ed25519 public key (hex, 64 chars) |
|
|
| `X-Quota-Remaining` | Response | Remaining tokens in current window |
|
|
| `X-Quota-Limit` | Response | Total tokens allowed per hour |
|
|
| `X-Quota-Reset` | Response | Unix timestamp when window resets |
|
|
|
|
### API Endpoints
|
|
|
|
**GET /v1/meter/quota**
|
|
Check quota status for an agent.
|
|
|
|
Query params:
|
|
- `agent_id` - Hex-encoded agent public key
|
|
|
|
Response:
|
|
```json
|
|
{
|
|
"agent_id": "...",
|
|
"remaining": 9500,
|
|
"limit": 10000,
|
|
"reset_at": 1705317600,
|
|
"used": 500,
|
|
"window_start": 1705314000
|
|
}
|
|
```
|
|
|
|
**POST /v1/meter/quota/limit**
|
|
Set custom quota limit (admin operation).
|
|
|
|
Request:
|
|
```json
|
|
{
|
|
"agent_id": "...",
|
|
"limit": 50000
|
|
}
|
|
```
|
|
|
|
## Configuration
|
|
|
|
Environment variable: `STEMEDB_METER_ENABLED`
|
|
- `true` (default): Enable economic throttling
|
|
- `false` or `0`: Disable (all requests pass through)
|
|
|
|
## Bypass Paths
|
|
|
|
These paths bypass metering:
|
|
- `/v1/health`
|
|
- `/swagger-ui/*`
|
|
- `/api-docs/*`
|
|
|
|
## Usage Example
|
|
|
|
```bash
|
|
# Check quota before operations
|
|
curl "http://localhost:18180/v1/meter/quota?agent_id=$(xxd -p -l 32 /dev/urandom)"
|
|
|
|
# Make request with agent ID
|
|
curl -X POST http://localhost:18180/v1/assert \
|
|
-H "Content-Type: application/json" \
|
|
-H "X-Agent-Id: 0102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20" \
|
|
-d '{"subject": "test", ...}'
|
|
|
|
# Response headers include quota info
|
|
# X-Quota-Remaining: 9989
|
|
# X-Quota-Limit: 10000
|
|
# X-Quota-Reset: 1705317600
|
|
```
|
|
|
|
## Implementation Notes
|
|
|
|
- Quota windows are hour-aligned (truncated to the start of the hour)
|
|
- Quotas reset automatically at the start of each hour
|
|
- On storage errors, requests are allowed (fail open for availability)
|
|
- Anonymous requests (no X-Agent-Id) bypass metering
|
|
- Token bucket allows burst behavior while enforcing long-term limits
|