79 lines
4.5 KiB
Markdown
79 lines
4.5 KiB
Markdown
# pg1-instrumented-metrics: Instrumented Metrics Pipeline
|
|
|
|
## Problem Statement
|
|
|
|
tidalDB has basic operational metrics (uptime, health, WAL lag, cumulative histograms) but lacks the **personalization-specific** instrumentation needed to verify the feedback loop is working. Operators cannot currently answer:
|
|
|
|
1. What are the p50/p95/p99 latencies for signal writes, retrieves, and searches?
|
|
2. Which signal types dominate write traffic?
|
|
3. How stale is a user's personalization context when they issue a query?
|
|
4. How quickly does the system close the feedback loop (signal -> query reflecting that signal)?
|
|
5. Is there a single JSON endpoint that summarizes all these for debugging?
|
|
|
|
Without these metrics, the pg1 milestone ("Personalization Core Done Gate") cannot validate that the personalization loop is correct, immediate, and measurably better than baseline.
|
|
|
|
## Goals
|
|
|
|
1. **Percentile summaries** -- Expose p50, p95, p99 latency gauges derived from existing cumulative histograms for signal writes, retrieves, and searches.
|
|
2. **Per-signal-type write counters** -- Break down `tidaldb_signal_writes_total` by signal type so operators can see which signals dominate traffic.
|
|
3. **Personalization staleness** -- Track the time delta between a user's last signal and their next query (how "fresh" is the user context at query time).
|
|
4. **Feedback-loop latency** -- Track end-to-end time from a user's signal write to a query whose results include the signaled entity.
|
|
5. **Diagnostics endpoint** -- A single `/diagnostics` JSON endpoint summarizing all instrumented metrics for debugging.
|
|
|
|
## Non-Goals
|
|
|
|
- Distributed tracing (spans, trace IDs) -- out of scope.
|
|
- Metrics push to external collectors (Prometheus scrape is sufficient).
|
|
- Dashboard or alerting rules -- infrastructure concern, not engine concern.
|
|
- Changing existing histogram bucket boundaries.
|
|
|
|
## Functional Requirements
|
|
|
|
### FR-1: Percentile Extraction from Existing Histograms
|
|
|
|
Add `percentile(p: f64) -> Option<u64>` to `LatencyHistogram` that computes an approximate percentile via linear interpolation within cumulative histogram buckets. Expose p50, p95, p99 as Prometheus gauges (e.g., `tidaldb_signal_write_latency_us_p50`).
|
|
|
|
### FR-2: Per-Signal-Type Write Counters
|
|
|
|
On each `signal()` call, increment a per-signal-type counter (`DashMap<String, AtomicU64>`). Render as `tidaldb_signal_writes_by_type{signal_type="view"} 1234` in Prometheus output.
|
|
|
|
### FR-3: Personalization Staleness Histogram
|
|
|
|
At query entry (both `retrieve()` and `search()`), if the query has a `for_user`, look up that user's most recent signal timestamp and compute `staleness_us = now - last_signal_ts`. Record into a new `LatencyHistogram` with appropriate bounds (ms to seconds). Expose p50/p95/p99 gauges.
|
|
|
|
### FR-4: Feedback-Loop Latency Histogram
|
|
|
|
After query execution, if the query's user had a signal within the last 60 seconds and the signaled entity appears in the result set, record `feedback_loop_us = now - last_signal_ts`. This measures how quickly the system "closes the loop."
|
|
|
|
### FR-5: `/diagnostics` JSON Endpoint
|
|
|
|
Add a `/diagnostics` route to the metrics HTTP server (feature-gated behind `metrics`). Returns a JSON object with:
|
|
- Per-signal-type write counts
|
|
- Latency percentiles (signal write, retrieve, search)
|
|
- Staleness percentiles
|
|
- Feedback-loop latency percentiles
|
|
|
|
### FR-6: User Signal Timestamp Map
|
|
|
|
Maintain a bounded `DashMap<u64, (u64, u64)>` mapping `user_id -> (last_signal_timestamp_ns, entity_id)` for FR-3 and FR-4 lookups. Bounded to 10,000 entries with approximate-LRU eviction.
|
|
|
|
## Non-Functional Requirements
|
|
|
|
- **NFR-1**: All new metrics are behind `#[cfg(feature = "metrics")]` -- zero cost when disabled.
|
|
- **NFR-2**: Per-signal-type counter map is bounded at 256 entries to prevent memory growth from adversarial signal type names.
|
|
- **NFR-3**: User signal timestamp map eviction uses sampling (scan 64 entries, evict oldest) to stay O(1) amortized.
|
|
- **NFR-4**: No new dependencies. Uses existing `DashMap`, `AtomicU64`, and `LatencyHistogram`.
|
|
|
|
## Test Strategy
|
|
|
|
- Unit tests for `percentile()` (empty, single, known distribution, beyond bounds).
|
|
- Unit tests for `UserSignalTimestampMap` (record, get, eviction, cap).
|
|
- Unit tests for `render_percentile_gauges()` and `render_diagnostics()`.
|
|
- Integration test: write signals, run queries, verify Prometheus output contains new metrics and `/diagnostics` returns valid JSON.
|
|
|
|
## Dependencies
|
|
|
|
- Existing `LatencyHistogram` (histogram.rs)
|
|
- Existing `MetricsState` (metrics/mod.rs)
|
|
- Existing metrics HTTP server (http.rs)
|