4766f566de
There was no metric anywhere that could answer "how much traffic are we serving"
or "what is our error rate". The engine published a rich DOMAIN surface (search
latency, WAL fsync, quorum timeouts, replication lag) and nothing about HTTP, so
a cluster could serve 401s or 503s indefinitely with every existing gauge looking
healthy. Logs were collected but unusable. There was no way to ask a RUNNING node
anything.
1. HTTP metrics. tidaldb_http_requests_total{route,method,status} plus a
per-route duration histogram, recorded by one layer placed OUTSIDE the auth,
timeout and rate-limit layers so it sees the status actually returned to the
client. Cardinality is the whole design: the route label is axum's MatchedPath
TEMPLATE, not the path, and unmatched requests collapse into one <unmatched>
bucket so a 404 flood cannot mint series. A hard cap folds anything past it
into an overflow bucket while established series keep counting.
The engine owns the /metrics listener but must not learn what a route or a
status code is, so it gained one registration hook
(MetricsState::set_extra_renderer) and tidal-server publishes through it. One
scrape target per node, not two.
2. Structured logs. The previous init was a bare tracing_subscriber::fmt(), which
produced two real defects: ANSI escapes leaked into collected logs, and every
line failed the collector's JSON parse and was stamped level=info — so
`level:error` matched NOTHING and errors were invisible to the log platform
while being collected. JSON_LOGS=1 emits the collector's exact wire format
(ts/level/service/env/msg), span fields are lifted so request_id lands on every
line of a request, and ANSI is off unconditionally in both formats.
Verified against the running binary, which caught a defect no unit test would
have: dependencies logging through the `log` crate arrived with target="log"
and four log.* metadata fields (absolute cargo registry paths, indexed
forever). The real module is now lifted into target and the bridge metadata
pruned.
3. Dashboard. docs/ops/grafana-tidaldb.json, 13 panels, mirrored into the fleet
as a grafana-database-dashboards key. Every metric name was checked against a
live endpoint and all 26 PromQL expressions were executed against the live
TSDB before commit, because a dashboard full of "No data" is worse than none.
Confirmed loaded in Grafana (uid tidaldb-overview, Databases folder).
4. tidalctl live mode. Every other subcommand reads a data dir AT REST, some
requiring a stopped node. `search`, `feed`, `cluster-status` and `watch` take
--url and talk to a running server, with --ca/--insecure because a cluster's
client port is served with the INTERNAL cluster CA. Exit codes follow the crate
contract, so `tidalctl cluster-status && deploy` gates on convergence.
Its first real run immediately found a reporting defect: the aggregated
/cluster/status reported two HEALTHY peers as UNREACHABLE PARTITIONED at 13.3M
lag, having derived lag against an uninitialised applied=0, while every node's
own status reported lag=0, reseed=false and identical frontiers, with
pod-to-pod connectivity open and nothing logged. cluster-status now names that
signature "NO REPORT (aggregated view; query the node directly)" instead of
repeating it as replication lag; a genuine non-zero-applied lag still reports
BEHIND. The underlying gap is documented as open work in
docs/ops/observability.md.
Verified: 2101 + 175 engine/server unit tests, 8 standalone integration (3 new,
including the cardinality proof and the cross-crate metrics seam), 23 tidalctl
(10 new), reseed + catchup + admin-gate e2e green, clippy clean, and both the
metrics and the log format exercised against a real running binary.
75 lines
3.9 KiB
TOML
75 lines
3.9 KiB
TOML
[package]
|
|
name = "tidalctl"
|
|
version = "0.1.0"
|
|
edition.workspace = true
|
|
# Higher than the workspace floor: aws-config -> aws-types 1.3.16 declares
|
|
# rustc 1.91.1, and resolution fails workspace-wide below it. Declared here so
|
|
# the requirement is visible where it originates rather than only in a lockfile
|
|
# error.
|
|
rust-version = "1.91.1"
|
|
description = "Command-line inspector for embedded tidalDB instances"
|
|
license.workspace = true
|
|
|
|
# ── tidal-crate lint posture (single source of truth) ──────────────────────
|
|
# IDENTICAL block across tidaldb / tidal-net / tidal-server / tidalctl. These
|
|
# crates deliberately DO NOT inherit `[workspace.lints]`; they hold the embedded
|
|
# recommendation DB + its transport/server/CLI to a stricter correctness bar
|
|
# (`unsafe_code = forbid`, `clippy::all = deny`, `unwrap_used = deny`).
|
|
# `unwrap_used = "deny"` is kept per-crate rather than in `[workspace.lints]`
|
|
# because the workspace also hosts the example/consumer crates under
|
|
# `applications/` (not held to the engine's bar). Keep these four blocks BYTE-IDENTICAL.
|
|
[lints.rust]
|
|
unsafe_code = "forbid"
|
|
|
|
[lints.clippy]
|
|
all = { level = "deny", priority = -1 }
|
|
pedantic = { level = "warn", priority = -1 }
|
|
nursery = { level = "warn", priority = -1 }
|
|
# Justified allows (lossy numeric casts are pervasive + intentional in the
|
|
# ranking/scoring math; module_name_repetitions is idiomatic for the flat
|
|
# module layout documented in CLAUDE.md):
|
|
cast_possible_truncation = "allow"
|
|
module_name_repetitions = "allow"
|
|
unwrap_used = "deny"
|
|
|
|
[dependencies]
|
|
tidaldb = { path = "../tidal" }
|
|
serde = { version = "1", features = ["derive"] }
|
|
serde_json = "1"
|
|
# Backup/restore manifest integrity (m11p8) — the same hash the engine verifies
|
|
# WAL segments and snapshot artifacts with, so a tidalctl-written manifest and an
|
|
# engine-verified one agree.
|
|
blake3 = "1"
|
|
# Object-store (S3-compatible) export/import for the DR gate (m11p8 R2). Chosen
|
|
# over `rust-s3` because the AWS SDK shares the workspace's existing
|
|
# hyper/rustls/tokio tree (no native-tls / second TLS stack), and R2 is handled
|
|
# with `.endpoint_url(<account>.r2.cloudflarestorage.com)` + `.force_path_style`.
|
|
# `default-features = false` + `rustls` keeps the TLS stack aligned with the rest
|
|
# of the workspace (reqwest is already `rustls-tls`); `behavior-version-latest`
|
|
# pins the SDK behavior contract so an SDK minor bump can't silently change it.
|
|
aws-config = { version = "1.8", default-features = false, features = ["behavior-version-latest", "rustls"] }
|
|
# `rt-tokio` exposes `ByteStream::from_path` (stream a file body straight off
|
|
# disk on `put_object` — no whole-file read into memory).
|
|
aws-sdk-s3 = { version = "1.137", default-features = false, features = ["behavior-version-latest", "rustls", "rt-tokio"] }
|
|
# `hardcoded-credentials` exposes `Credentials::from_keys` so the env-var creds
|
|
# (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY) become a static SigV4 provider.
|
|
aws-credential-types = { version = "1.2", features = ["hardcoded-credentials"] }
|
|
# S3 transfers run on a locally-built current-thread runtime ONLY when an --s3-*
|
|
# flag is given; the local-dir backup/restore path stays fully synchronous.
|
|
tokio = { version = "1", default-features = false, features = ["rt", "macros"] }
|
|
# Fresh staging dir for an S3 import (downloaded prefix -> temp dir -> the
|
|
# UNCHANGED verified restore reads it). Reaped when the restore returns.
|
|
tempfile = "3"
|
|
# Live-server commands (`search`, `feed`, `cluster-status`, `watch`) talk HTTP to
|
|
# a RUNNING node. Blocking client: this is a CLI, so a runtime would buy nothing.
|
|
# `rustls-tls` keeps the TLS stack aligned with the rest of the workspace, and
|
|
# matters here because a cluster's client port is served with the INTERNAL
|
|
# cluster CA — hence `--ca` / `--insecure`.
|
|
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls", "blocking"] }
|
|
|
|
[dev-dependencies]
|
|
tidaldb = { path = "../tidal", features = ["test-utils"] }
|
|
|
|
[[test]]
|
|
name = "cli"
|