pg1-instrumented-metrics: Design

Architecture Overview

signal() ──→ per-type counter ──→ MetricsState.signal_writes_by_type
         ──→ user timestamp    ──→ MetricsState.user_last_signal

signal_with_context() ──→ records user_id → (ts, entity_id)

retrieve()/search() ──→ reads user_last_signal ──→ staleness histogram
                    ──→ after results           ──→ feedback-loop histogram

/metrics  ──→ render_prometheus() ──→ per-type counters + percentile gauges
/diagnostics ──→ render_diagnostics() ──→ JSON summary

Component Design

1. LatencyHistogram Extensions (histogram.rs)

Add three methods to the existing LatencyHistogram:

total_count() -> u64: Returns the total observation count (reads self.count).
percentile(p: f64) -> Option<u64>: Computes approximate percentile via linear interpolation within cumulative buckets. Returns None if no observations. Clamps p to [0.0, 1.0].
render_percentile_gauges(name: &str) -> String: Renders p50, p95, p99 as Prometheus gauge lines.

Percentile algorithm: Given target = ceil(p * total_count), scan cumulative buckets. When a bucket's cumulative count >= target, interpolate linearly between the previous bucket's bound and this bucket's bound based on position within the bucket.

2. Per-Signal-Type Counters (metrics/mod.rs)

New field on MetricsState:

#[cfg(feature = "metrics")]
pub(crate) signal_writes_by_type: DashMap<String, AtomicU64>,

Capped at 256 entries. In signal(), after the existing metrics block, increment signal_writes_by_type[signal_type].

Rendered in Prometheus as:

tidaldb_signal_writes_by_type{signal_type="view"} 1234
tidaldb_signal_writes_by_type{signal_type="like"} 567

3. UserSignalTimestampMap (metrics/mod.rs)

#[cfg(feature = "metrics")]
pub(crate) struct UserSignalTimestampMap {
    map: DashMap<u64, (u64, u64)>,  // user_id -> (timestamp_ns, entity_id)
    cap: usize,
}

Methods: new(cap), record(user_id, ts_ns, entity_id), get(user_id), evict_oldest().

Eviction: When map.len() >= cap, scan 64 random entries, evict the one with the oldest timestamp. This is O(1) amortized.

4. Personalization Staleness (query_ops.rs)

At the top of both retrieve() and search():

#[cfg(feature = "metrics")]
if let Some(uid) = query.for_user {
    if let Some((last_signal_ns, _)) = self.metrics.user_last_signal.get(uid) {
        let staleness_us = (now_ns - last_signal_ns) / 1_000;
        self.metrics.personalization_staleness.observe(staleness_us);
    }
}

New histogram field on MetricsState:

#[cfg(feature = "metrics")]
pub(crate) personalization_staleness: LatencyHistogram,

Bounds: [1_000, 10_000, 100_000, 500_000, 1_000_000, 5_000_000, 10_000_000, 30_000_000, 60_000_000] (1ms to 60s in microseconds).

5. Feedback-Loop Latency (query_ops.rs)

After query result assembly, if the user's last signal was within 60 seconds and the signaled entity appears in results:

#[cfg(feature = "metrics")]
if let Some((last_signal_ns, entity_id)) = user_signal_ctx {
    let age_secs = (now_ns - last_signal_ns) / 1_000_000_000;
    if age_secs <= 60 && result_contains(entity_id) {
        let loop_us = (now_ns - last_signal_ns) / 1_000;
        self.metrics.feedback_loop_latency.observe(loop_us);
    }
}

6. /diagnostics Endpoint (http.rs)

Add /diagnostics route to handle_connection:

#[cfg(feature = "metrics")]
"/diagnostics" => ("200 OK", "application/json", state.render_diagnostics()),

render_diagnostics() produces JSON with: per-type signal counts, latency percentiles, staleness percentiles, feedback-loop percentiles.

File Change Matrix

File	Change
`tidal/src/db/metrics/histogram.rs`	Add `total_count()`, `percentile()`, `render_percentile_gauges()`, 7 new tests
`tidal/src/db/metrics/mod.rs`	Add `UserSignalTimestampMap`, 4 new fields, `render_diagnostics()`, update `render_prometheus()`
`tidal/src/db/signals.rs`	Per-type counter increment, user timestamp recording
`tidal/src/db/query_ops.rs`	Staleness + feedback-loop recording in retrieve() and search()
`tidal/src/db/http.rs`	/diagnostics route

Risk Analysis

Performance: All new instrumentation is behind #[cfg(feature = "metrics")]. DashMap operations are lock-free shard reads. No measurable impact on hot path.
Memory: UserSignalTimestampMap bounded at 10K entries (~240KB). Per-type counter map bounded at 256 entries (~8KB).
Correctness: Percentile interpolation is approximate (standard for Prometheus-style histograms). Staleness/feedback-loop are best-effort -- missing entries are silently skipped.

4.7 KiB Raw Blame History