tidaldb/.sdlc/features/pg1-instrumented-metrics/spec.md

pg1-instrumented-metrics: Instrumented Metrics Pipeline

Problem Statement

tidalDB has basic operational metrics (uptime, health, WAL lag, cumulative histograms) but lacks the personalization-specific instrumentation needed to verify the feedback loop is working. Operators cannot currently answer:

1. What are the p50/p95/p99 latencies for signal writes, retrieves, and searches?
2. Which signal types dominate write traffic?
3. How stale is a user's personalization context when they issue a query?
4. How quickly does the system close the feedback loop (signal -> query reflecting that signal)?
5. Is there a single JSON endpoint that summarizes all these for debugging?

Without these metrics, the pg1 milestone ("Personalization Core Done Gate") cannot validate that the personalization loop is correct, immediate, and measurably better than baseline.

Goals

1. Percentile summaries -- Expose p50, p95, p99 latency gauges derived from existing cumulative histograms for signal writes, retrieves, and searches.
2. Per-signal-type write counters -- Break down tidaldb_signal_writes_total by signal type so operators can see which signals dominate traffic.
3. Personalization staleness -- Track the time delta between a user's last signal and their next query (how "fresh" is the user context at query time).
4. Feedback-loop latency -- Track end-to-end time from a user's signal write to a query whose results include the signaled entity.
5. Diagnostics endpoint -- A single /diagnostics JSON endpoint summarizing all instrumented metrics for debugging.

Non-Goals

• Distributed tracing (spans, trace IDs) -- out of scope.
• Metrics push to external collectors (Prometheus scrape is sufficient).
• Dashboard or alerting rules -- infrastructure concern, not engine concern.
• Changing existing histogram bucket boundaries.

Functional Requirements

FR-1: Percentile Extraction from Existing Histograms

Add percentile(p: f64) -> Option<u64> to LatencyHistogram that computes an approximate percentile via linear interpolation within cumulative histogram buckets. Expose p50, p95, p99 as Prometheus gauges (e.g., tidaldb_signal_write_latency_us_p50).

FR-2: Per-Signal-Type Write Counters

On each signal() call, increment a per-signal-type counter (DashMap<String, AtomicU64>). Render as tidaldb_signal_writes_by_type{signal_type="view"} 1234 in Prometheus output.

FR-3: Personalization Staleness Histogram

At query entry (both retrieve() and search()), if the query has a for_user, look up that user's most recent signal timestamp and compute staleness_us = now - last_signal_ts. Record into a new LatencyHistogram with appropriate bounds (ms to seconds). Expose p50/p95/p99 gauges.

FR-4: Feedback-Loop Latency Histogram

After query execution, if the query's user had a signal within the last 60 seconds and the signaled entity appears in the result set, record feedback_loop_us = now - last_signal_ts. This measures how quickly the system "closes the loop."

FR-5: /diagnostics JSON Endpoint

Add a /diagnostics route to the metrics HTTP server (feature-gated behind metrics). Returns a JSON object with:

• Per-signal-type write counts
• Latency percentiles (signal write, retrieve, search)
• Staleness percentiles
• Feedback-loop latency percentiles

FR-6: User Signal Timestamp Map

Maintain a bounded DashMap<u64, (u64, u64)> mapping user_id -> (last_signal_timestamp_ns, entity_id) for FR-3 and FR-4 lookups. Bounded to 10,000 entries with approximate-LRU eviction.

Non-Functional Requirements

• NFR-1: All new metrics are behind #[cfg(feature = "metrics")] -- zero cost when disabled.
• NFR-2: Per-signal-type counter map is bounded at 256 entries to prevent memory growth from adversarial signal type names.
• NFR-3: User signal timestamp map eviction uses sampling (scan 64 entries, evict oldest) to stay O(1) amortized.
• NFR-4: No new dependencies. Uses existing DashMap, AtomicU64, and LatencyHistogram.

Test Strategy

• Unit tests for percentile() (empty, single, known distribution, beyond bounds).
• Unit tests for UserSignalTimestampMap (record, get, eviction, cap).
• Unit tests for render_percentile_gauges() and render_diagnostics().
• Integration test: write signals, run queries, verify Prometheus output contains new metrics and /diagnostics returns valid JSON.

Dependencies

• Existing LatencyHistogram (histogram.rs)
• Existing MetricsState (metrics/mod.rs)
• Existing metrics HTTP server (http.rs)