There was no metric anywhere that could answer "how much traffic are we serving"
or "what is our error rate". The engine published a rich DOMAIN surface (search
latency, WAL fsync, quorum timeouts, replication lag) and nothing about HTTP, so
a cluster could serve 401s or 503s indefinitely with every existing gauge looking
healthy. Logs were collected but unusable. There was no way to ask a RUNNING node
anything.
1. HTTP metrics. tidaldb_http_requests_total{route,method,status} plus a
per-route duration histogram, recorded by one layer placed OUTSIDE the auth,
timeout and rate-limit layers so it sees the status actually returned to the
client. Cardinality is the whole design: the route label is axum's MatchedPath
TEMPLATE, not the path, and unmatched requests collapse into one <unmatched>
bucket so a 404 flood cannot mint series. A hard cap folds anything past it
into an overflow bucket while established series keep counting.
The engine owns the /metrics listener but must not learn what a route or a
status code is, so it gained one registration hook
(MetricsState::set_extra_renderer) and tidal-server publishes through it. One
scrape target per node, not two.
2. Structured logs. The previous init was a bare tracing_subscriber::fmt(), which
produced two real defects: ANSI escapes leaked into collected logs, and every
line failed the collector's JSON parse and was stamped level=info — so
`level:error` matched NOTHING and errors were invisible to the log platform
while being collected. JSON_LOGS=1 emits the collector's exact wire format
(ts/level/service/env/msg), span fields are lifted so request_id lands on every
line of a request, and ANSI is off unconditionally in both formats.
Verified against the running binary, which caught a defect no unit test would
have: dependencies logging through the `log` crate arrived with target="log"
and four log.* metadata fields (absolute cargo registry paths, indexed
forever). The real module is now lifted into target and the bridge metadata
pruned.
3. Dashboard. docs/ops/grafana-tidaldb.json, 13 panels, mirrored into the fleet
as a grafana-database-dashboards key. Every metric name was checked against a
live endpoint and all 26 PromQL expressions were executed against the live
TSDB before commit, because a dashboard full of "No data" is worse than none.
Confirmed loaded in Grafana (uid tidaldb-overview, Databases folder).
4. tidalctl live mode. Every other subcommand reads a data dir AT REST, some
requiring a stopped node. `search`, `feed`, `cluster-status` and `watch` take
--url and talk to a running server, with --ca/--insecure because a cluster's
client port is served with the INTERNAL cluster CA. Exit codes follow the crate
contract, so `tidalctl cluster-status && deploy` gates on convergence.
Its first real run immediately found a reporting defect: the aggregated
/cluster/status reported two HEALTHY peers as UNREACHABLE PARTITIONED at 13.3M
lag, having derived lag against an uninitialised applied=0, while every node's
own status reported lag=0, reseed=false and identical frontiers, with
pod-to-pod connectivity open and nothing logged. cluster-status now names that
signature "NO REPORT (aggregated view; query the node directly)" instead of
repeating it as replication lag; a genuine non-zero-applied lag still reports
BEHIND. The underlying gap is documented as open work in
docs/ops/observability.md.
Verified: 2101 + 175 engine/server unit tests, 8 standalone integration (3 new,
including the cardinality proof and the cross-crate metrics seam), 23 tidalctl
(10 new), reseed + catchup + admin-gate e2e green, clippy clean, and both the
metrics and the log format exercised against a real running binary.
4766f566de