120 lines
4.7 KiB
Markdown
120 lines
4.7 KiB
Markdown
# pg1-instrumented-metrics: Design
|
|
|
|
## Architecture Overview
|
|
|
|
```
|
|
signal() ──→ per-type counter ──→ MetricsState.signal_writes_by_type
|
|
──→ user timestamp ──→ MetricsState.user_last_signal
|
|
|
|
signal_with_context() ──→ records user_id → (ts, entity_id)
|
|
|
|
retrieve()/search() ──→ reads user_last_signal ──→ staleness histogram
|
|
──→ after results ──→ feedback-loop histogram
|
|
|
|
/metrics ──→ render_prometheus() ──→ per-type counters + percentile gauges
|
|
/diagnostics ──→ render_diagnostics() ──→ JSON summary
|
|
```
|
|
|
|
## Component Design
|
|
|
|
### 1. LatencyHistogram Extensions (histogram.rs)
|
|
|
|
Add three methods to the existing `LatencyHistogram`:
|
|
|
|
- `total_count() -> u64`: Returns the total observation count (reads `self.count`).
|
|
- `percentile(p: f64) -> Option<u64>`: Computes approximate percentile via linear interpolation within cumulative buckets. Returns `None` if no observations. Clamps `p` to `[0.0, 1.0]`.
|
|
- `render_percentile_gauges(name: &str) -> String`: Renders p50, p95, p99 as Prometheus gauge lines.
|
|
|
|
**Percentile algorithm**: Given target = ceil(p * total_count), scan cumulative buckets. When a bucket's cumulative count >= target, interpolate linearly between the previous bucket's bound and this bucket's bound based on position within the bucket.
|
|
|
|
### 2. Per-Signal-Type Counters (metrics/mod.rs)
|
|
|
|
New field on `MetricsState`:
|
|
```rust
|
|
#[cfg(feature = "metrics")]
|
|
pub(crate) signal_writes_by_type: DashMap<String, AtomicU64>,
|
|
```
|
|
|
|
Capped at 256 entries. In `signal()`, after the existing metrics block, increment `signal_writes_by_type[signal_type]`.
|
|
|
|
Rendered in Prometheus as:
|
|
```
|
|
tidaldb_signal_writes_by_type{signal_type="view"} 1234
|
|
tidaldb_signal_writes_by_type{signal_type="like"} 567
|
|
```
|
|
|
|
### 3. UserSignalTimestampMap (metrics/mod.rs)
|
|
|
|
```rust
|
|
#[cfg(feature = "metrics")]
|
|
pub(crate) struct UserSignalTimestampMap {
|
|
map: DashMap<u64, (u64, u64)>, // user_id -> (timestamp_ns, entity_id)
|
|
cap: usize,
|
|
}
|
|
```
|
|
|
|
Methods: `new(cap)`, `record(user_id, ts_ns, entity_id)`, `get(user_id)`, `evict_oldest()`.
|
|
|
|
Eviction: When `map.len() >= cap`, scan 64 random entries, evict the one with the oldest timestamp. This is O(1) amortized.
|
|
|
|
### 4. Personalization Staleness (query_ops.rs)
|
|
|
|
At the top of both `retrieve()` and `search()`:
|
|
```rust
|
|
#[cfg(feature = "metrics")]
|
|
if let Some(uid) = query.for_user {
|
|
if let Some((last_signal_ns, _)) = self.metrics.user_last_signal.get(uid) {
|
|
let staleness_us = (now_ns - last_signal_ns) / 1_000;
|
|
self.metrics.personalization_staleness.observe(staleness_us);
|
|
}
|
|
}
|
|
```
|
|
|
|
New histogram field on `MetricsState`:
|
|
```rust
|
|
#[cfg(feature = "metrics")]
|
|
pub(crate) personalization_staleness: LatencyHistogram,
|
|
```
|
|
|
|
Bounds: `[1_000, 10_000, 100_000, 500_000, 1_000_000, 5_000_000, 10_000_000, 30_000_000, 60_000_000]` (1ms to 60s in microseconds).
|
|
|
|
### 5. Feedback-Loop Latency (query_ops.rs)
|
|
|
|
After query result assembly, if the user's last signal was within 60 seconds and the signaled entity appears in results:
|
|
```rust
|
|
#[cfg(feature = "metrics")]
|
|
if let Some((last_signal_ns, entity_id)) = user_signal_ctx {
|
|
let age_secs = (now_ns - last_signal_ns) / 1_000_000_000;
|
|
if age_secs <= 60 && result_contains(entity_id) {
|
|
let loop_us = (now_ns - last_signal_ns) / 1_000;
|
|
self.metrics.feedback_loop_latency.observe(loop_us);
|
|
}
|
|
}
|
|
```
|
|
|
|
### 6. /diagnostics Endpoint (http.rs)
|
|
|
|
Add `/diagnostics` route to `handle_connection`:
|
|
```rust
|
|
#[cfg(feature = "metrics")]
|
|
"/diagnostics" => ("200 OK", "application/json", state.render_diagnostics()),
|
|
```
|
|
|
|
`render_diagnostics()` produces JSON with: per-type signal counts, latency percentiles, staleness percentiles, feedback-loop percentiles.
|
|
|
|
## File Change Matrix
|
|
|
|
| File | Change |
|
|
|------|--------|
|
|
| `tidal/src/db/metrics/histogram.rs` | Add `total_count()`, `percentile()`, `render_percentile_gauges()`, 7 new tests |
|
|
| `tidal/src/db/metrics/mod.rs` | Add `UserSignalTimestampMap`, 4 new fields, `render_diagnostics()`, update `render_prometheus()` |
|
|
| `tidal/src/db/signals.rs` | Per-type counter increment, user timestamp recording |
|
|
| `tidal/src/db/query_ops.rs` | Staleness + feedback-loop recording in retrieve() and search() |
|
|
| `tidal/src/db/http.rs` | /diagnostics route |
|
|
|
|
## Risk Analysis
|
|
|
|
- **Performance**: All new instrumentation is behind `#[cfg(feature = "metrics")]`. DashMap operations are lock-free shard reads. No measurable impact on hot path.
|
|
- **Memory**: UserSignalTimestampMap bounded at 10K entries (~240KB). Per-type counter map bounded at 256 entries (~8KB).
|
|
- **Correctness**: Percentile interpolation is approximate (standard for Prometheus-style histograms). Staleness/feedback-loop are best-effort -- missing entries are silently skipped.
|