# pg1-instrumented-metrics: Design

## Architecture Overview

```
signal() ──→ per-type counter ──→ MetricsState.signal_writes_by_type
         ──→ user timestamp    ──→ MetricsState.user_last_signal

signal_with_context() ──→ records user_id → (ts, entity_id)

retrieve()/search() ──→ reads user_last_signal ──→ staleness histogram
                    ──→ after results           ──→ feedback-loop histogram

/metrics  ──→ render_prometheus() ──→ per-type counters + percentile gauges
/diagnostics ──→ render_diagnostics() ──→ JSON summary
```

## Component Design

### 1. LatencyHistogram Extensions (histogram.rs)

Add three methods to the existing `LatencyHistogram`:

- `total_count() -> u64`: Returns the total observation count (reads `self.count`).
- `percentile(p: f64) -> Option<u64>`: Computes approximate percentile via linear interpolation within cumulative buckets. Returns `None` if no observations. Clamps `p` to `[0.0, 1.0]`.
- `render_percentile_gauges(name: &str) -> String`: Renders p50, p95, p99 as Prometheus gauge lines.

**Percentile algorithm**: Given target = ceil(p * total_count), scan cumulative buckets. When a bucket's cumulative count >= target, interpolate linearly between the previous bucket's bound and this bucket's bound based on position within the bucket.

### 2. Per-Signal-Type Counters (metrics/mod.rs)

New field on `MetricsState`:
```rust
#[cfg(feature = "metrics")]
pub(crate) signal_writes_by_type: DashMap<String, AtomicU64>,
```

Capped at 256 entries. In `signal()`, after the existing metrics block, increment `signal_writes_by_type[signal_type]`.

Rendered in Prometheus as:
```
tidaldb_signal_writes_by_type{signal_type="view"} 1234
tidaldb_signal_writes_by_type{signal_type="like"} 567
```

### 3. UserSignalTimestampMap (metrics/mod.rs)

```rust
#[cfg(feature = "metrics")]
pub(crate) struct UserSignalTimestampMap {
    map: DashMap<u64, (u64, u64)>,  // user_id -> (timestamp_ns, entity_id)
    cap: usize,
}
```

Methods: `new(cap)`, `record(user_id, ts_ns, entity_id)`, `get(user_id)`, `evict_oldest()`.

Eviction: When `map.len() >= cap`, scan 64 random entries, evict the one with the oldest timestamp. This is O(1) amortized.

### 4. Personalization Staleness (query_ops.rs)

At the top of both `retrieve()` and `search()`:
```rust
#[cfg(feature = "metrics")]
if let Some(uid) = query.for_user {
    if let Some((last_signal_ns, _)) = self.metrics.user_last_signal.get(uid) {
        let staleness_us = (now_ns - last_signal_ns) / 1_000;
        self.metrics.personalization_staleness.observe(staleness_us);
    }
}
```

New histogram field on `MetricsState`:
```rust
#[cfg(feature = "metrics")]
pub(crate) personalization_staleness: LatencyHistogram,
```

Bounds: `[1_000, 10_000, 100_000, 500_000, 1_000_000, 5_000_000, 10_000_000, 30_000_000, 60_000_000]` (1ms to 60s in microseconds).

### 5. Feedback-Loop Latency (query_ops.rs)

After query result assembly, if the user's last signal was within 60 seconds and the signaled entity appears in results:
```rust
#[cfg(feature = "metrics")]
if let Some((last_signal_ns, entity_id)) = user_signal_ctx {
    let age_secs = (now_ns - last_signal_ns) / 1_000_000_000;
    if age_secs <= 60 && result_contains(entity_id) {
        let loop_us = (now_ns - last_signal_ns) / 1_000;
        self.metrics.feedback_loop_latency.observe(loop_us);
    }
}
```

### 6. /diagnostics Endpoint (http.rs)

Add `/diagnostics` route to `handle_connection`:
```rust
#[cfg(feature = "metrics")]
"/diagnostics" => ("200 OK", "application/json", state.render_diagnostics()),
```

`render_diagnostics()` produces JSON with: per-type signal counts, latency percentiles, staleness percentiles, feedback-loop percentiles.

## File Change Matrix

| File | Change |
|------|--------|
| `tidal/src/db/metrics/histogram.rs` | Add `total_count()`, `percentile()`, `render_percentile_gauges()`, 7 new tests |
| `tidal/src/db/metrics/mod.rs` | Add `UserSignalTimestampMap`, 4 new fields, `render_diagnostics()`, update `render_prometheus()` |
| `tidal/src/db/signals.rs` | Per-type counter increment, user timestamp recording |
| `tidal/src/db/query_ops.rs` | Staleness + feedback-loop recording in retrieve() and search() |
| `tidal/src/db/http.rs` | /diagnostics route |

## Risk Analysis

- **Performance**: All new instrumentation is behind `#[cfg(feature = "metrics")]`. DashMap operations are lock-free shard reads. No measurable impact on hot path.
- **Memory**: UserSignalTimestampMap bounded at 10K entries (~240KB). Per-type counter map bounded at 256 entries (~8KB).
- **Correctness**: Percentile interpolation is approximate (standard for Prometheus-style histograms). Staleness/feedback-loop are best-effort -- missing entries are silently skipped.