tidaldb/docs/ops/observability.md

Observability: usage, errors, and live debugging

Four surfaces, in the order you reach for them during an incident.

Question	Surface
How much traffic, and how much of it is failing?	Grafana → tidalDB — usage, errors, and cluster health
What exactly failed, for which request?	VictoriaLogs, level:error, correlate on request_id
Is the cluster converged right now?	tidalctl cluster-status / tidalctl watch
What does a query actually return?	tidalctl search / tidalctl feed

---

1. Metrics

Every node serves Prometheus text on its --metrics address (:9091 in the shipped manifests). Never expose that port externally — it is unauthenticated. In-cluster it is reachable only from the observability namespace (k8s/cluster/networkpolicy.yaml).

Two families answer the usage/error questions:

tidaldb_http_requests_total{route,method,status}   counter
tidaldb_http_request_duration_us{route}            histogram

route is the matched route template (/cluster/shards/{id}/transfer), not the concrete path, so cardinality is bounded by the router rather than by traffic. Unmatched requests collapse into a single <unmatched> bucket — a 404 flood cannot mint series.

These are published by tidal-server onto the engine's existing listener (via MetricsState::set_extra_renderer), so a node remains one scrape target.

Useful queries:

# request rate by route
sum by (route) (rate(tidaldb_http_requests_total[5m]))

# server-fault ratio
  sum(rate(tidaldb_http_requests_total{status=~"5.."}[5m]))
/ sum(rate(tidaldb_http_requests_total[5m]))

# who is being refused, and where
sum by (status, route) (rate(tidaldb_http_requests_total{status=~"401|403"}[5m]))

# client-observed p99, in ms
histogram_quantile(0.99,
  sum by (route, le) (rate(tidaldb_http_request_duration_us_bucket[5m]))) / 1000

Read 403 precisely: it means a valid data bearer was used against an operator verb (/cluster/*). A sustained 403 rate is a client holding the wrong key, or someone probing — not a server fault.

Dashboard

docs/ops/grafana-tidaldb.json is the source of truth, mirrored into the fleet as the tidaldb-overview.json key of the grafana-database-dashboards ConfigMap — the same convention docs/ops/prometheus-alerts.yaml follows. Edit here, then mirror. It lands in Grafana's Databases folder as tidaldb-overview.

---

2. Logs

Set JSON_LOGS=1 (or TIDAL_LOG_FORMAT=json) to emit one JSON object per line in the collector's wire format:

{"ts":"2026-08-23T15:57:57.090256Z","level":"info","service":"tidal-server",
 "env":"prod","msg":"committing 5","target":"tantivy::indexer","request_id":"418"}

Why it matters: the fleet's Vector collector parses each line as JSON and, on success, replaces the event with that object; on failure it keeps the text and stamps level = "info". Without JSON, every tidalDB line was recorded as info and level:error matched nothing — errors were collected but invisible.

• level is lowercase, matching the collector's enum (critical|error|warn|info|debug).
• request_id from the per-request span appears on every line emitted inside it, so one request's work is correlatable rather than grep-adjacent.
• target is the real module even for dependencies logging through the log crate (their bridge metadata is pruned rather than indexed forever).
• ANSI colour is off in both formats. It used to leak escape codes into stored logs.

Tunables: TIDAL_SERVER_LOG (env-filter, e.g. tidal_server=debug,tantivy=warn,info to quiet a noisy dependency), TIDAL_SERVICE_NAME, TIDAL_ENV.

# errors, last 15 minutes
_time:15m AND kubernetes.pod_namespace:tidaldb-cluster AND level:error

# one request end to end
_time:1h AND request_id:"418"

---

3. Live debugging with tidalctl

Every other tidalctl command reads a data directory at rest — some require a stopped node. These take --url and talk to a running server.

export TIDAL_API_KEY=...          # or pass --key

# Is it converged? Exits 2 when not, so it gates a deploy.
tidalctl cluster-status --url http://127.0.0.1:9500

# Watch convergence (Ctrl-C to stop, or bound it with --count)
tidalctl watch --url http://127.0.0.1:9500 --interval 5 --count 12

# What does a query actually return?
tidalctl search --url http://127.0.0.1:9500 --query "cold brew" --limit 5
tidalctl feed   --url http://127.0.0.1:9500 --profile trending --user-id 42

TLS. A cluster node's client port is served with the internal cluster CA, issued for in-cluster DNS names, so a port-forward fails both CA and hostname verification. Pin the CA with --ca ca.pem, or use --insecure for a local forward:

kubectl -n tidaldb-cluster port-forward svc/tidaldb 9500:9500 &
tidalctl cluster-status --url https://127.0.0.1:9500 --insecure

Exit codes follow the crate contract: 0 ok, 1 usage error, 2 degraded — unreachable, non-2xx, or not converged. A 403 response prints the hint to use the admin key.

---

4. Known caveat: the aggregated /cluster/status under-reports peers

The aggregated view can report a peer it holds no frontier report for as applied_events: 0, and derive lag_events against that zero — so a fully converged peer appears to be the leader's entire history behind, sometimes with reachable: false / partitioned: true alongside.

Observed 2026-08-23: two healthy nodes shown as UNREACHABLE PARTITIONED at 13.3M lag, while every node's own /cluster/status/local reported lag=0, reseed_required=false, and identical per-group frontiers. Pod-to-pod connectivity was open and no error was logged.

tidalctl cluster-status detects that signature (applied == 0 with non-zero lag) and prints NO REPORT (aggregated view; query the node directly) rather than repeating it as replication lag. A genuine lag report — non-zero applied with non-zero lag — still reports BEHIND.

When you see NO REPORT, confirm against the nodes themselves before treating it as an incident:

for i in 0 1 2; do
  kubectl -n tidaldb-cluster port-forward tidaldb-$i 1970$i:9500 >/dev/null 2>&1 &
  sleep 8
  curl -sk -H "Authorization: Bearer $TIDAL_API_KEY" \
    "https://127.0.0.1:1970$i/cluster/status/local" | jq '{region, reseed_required, shards}'
  kill %1
done

The underlying reporting gap in the aggregated surface is open work; it is a reporting defect, not a replication one.