4766f566de
There was no metric anywhere that could answer "how much traffic are we serving"
or "what is our error rate". The engine published a rich DOMAIN surface (search
latency, WAL fsync, quorum timeouts, replication lag) and nothing about HTTP, so
a cluster could serve 401s or 503s indefinitely with every existing gauge looking
healthy. Logs were collected but unusable. There was no way to ask a RUNNING node
anything.
1. HTTP metrics. tidaldb_http_requests_total{route,method,status} plus a
per-route duration histogram, recorded by one layer placed OUTSIDE the auth,
timeout and rate-limit layers so it sees the status actually returned to the
client. Cardinality is the whole design: the route label is axum's MatchedPath
TEMPLATE, not the path, and unmatched requests collapse into one <unmatched>
bucket so a 404 flood cannot mint series. A hard cap folds anything past it
into an overflow bucket while established series keep counting.
The engine owns the /metrics listener but must not learn what a route or a
status code is, so it gained one registration hook
(MetricsState::set_extra_renderer) and tidal-server publishes through it. One
scrape target per node, not two.
2. Structured logs. The previous init was a bare tracing_subscriber::fmt(), which
produced two real defects: ANSI escapes leaked into collected logs, and every
line failed the collector's JSON parse and was stamped level=info — so
`level:error` matched NOTHING and errors were invisible to the log platform
while being collected. JSON_LOGS=1 emits the collector's exact wire format
(ts/level/service/env/msg), span fields are lifted so request_id lands on every
line of a request, and ANSI is off unconditionally in both formats.
Verified against the running binary, which caught a defect no unit test would
have: dependencies logging through the `log` crate arrived with target="log"
and four log.* metadata fields (absolute cargo registry paths, indexed
forever). The real module is now lifted into target and the bridge metadata
pruned.
3. Dashboard. docs/ops/grafana-tidaldb.json, 13 panels, mirrored into the fleet
as a grafana-database-dashboards key. Every metric name was checked against a
live endpoint and all 26 PromQL expressions were executed against the live
TSDB before commit, because a dashboard full of "No data" is worse than none.
Confirmed loaded in Grafana (uid tidaldb-overview, Databases folder).
4. tidalctl live mode. Every other subcommand reads a data dir AT REST, some
requiring a stopped node. `search`, `feed`, `cluster-status` and `watch` take
--url and talk to a running server, with --ca/--insecure because a cluster's
client port is served with the INTERNAL cluster CA. Exit codes follow the crate
contract, so `tidalctl cluster-status && deploy` gates on convergence.
Its first real run immediately found a reporting defect: the aggregated
/cluster/status reported two HEALTHY peers as UNREACHABLE PARTITIONED at 13.3M
lag, having derived lag against an uninitialised applied=0, while every node's
own status reported lag=0, reseed=false and identical frontiers, with
pod-to-pod connectivity open and nothing logged. cluster-status now names that
signature "NO REPORT (aggregated view; query the node directly)" instead of
repeating it as replication lag; a genuine non-zero-applied lag still reports
BEHIND. The underlying gap is documented as open work in
docs/ops/observability.md.
Verified: 2101 + 175 engine/server unit tests, 8 standalone integration (3 new,
including the cardinality proof and the cross-crate metrics seam), 23 tidalctl
(10 new), reseed + catchup + admin-gate e2e green, clippy clean, and both the
metrics and the log format exercised against a real running binary.
123 lines
6.2 KiB
TOML
123 lines
6.2 KiB
TOML
[package]
|
|
name = "tidal-server"
|
|
version = "0.1.0"
|
|
edition.workspace = true
|
|
rust-version.workspace = true
|
|
license.workspace = true
|
|
|
|
[lib]
|
|
name = "tidal_server"
|
|
path = "src/lib.rs"
|
|
|
|
# ── tidal-crate lint posture (single source of truth) ──────────────────────
|
|
# IDENTICAL block across tidaldb / tidal-net / tidal-server / tidalctl. These
|
|
# crates deliberately DO NOT inherit `[workspace.lints]`; they hold the embedded
|
|
# recommendation DB + its transport/server/CLI to a stricter correctness bar
|
|
# (`unsafe_code = forbid`, `clippy::all = deny`, `unwrap_used = deny`).
|
|
# `unwrap_used = "deny"` is kept per-crate rather than in `[workspace.lints]`
|
|
# because the workspace also hosts the example/consumer crates under
|
|
# `applications/` (not held to the engine's bar). Keep these four blocks BYTE-IDENTICAL.
|
|
[lints.rust]
|
|
unsafe_code = "forbid"
|
|
|
|
[lints.clippy]
|
|
all = { level = "deny", priority = -1 }
|
|
pedantic = { level = "warn", priority = -1 }
|
|
nursery = { level = "warn", priority = -1 }
|
|
# Justified allows (lossy numeric casts are pervasive + intentional in the
|
|
# ranking/scoring math; module_name_repetitions is idiomatic for the flat
|
|
# module layout documented in CLAUDE.md):
|
|
cast_possible_truncation = "allow"
|
|
module_name_repetitions = "allow"
|
|
unwrap_used = "deny"
|
|
|
|
[dependencies]
|
|
# m11p7 hot rotation: the reloadable credential holder (bearer + cluster key)
|
|
# and the inter-node HTTP TLS cert resolver swap their material lock-free under
|
|
# load via `arc-swap` (already in the lock transitively; promoted to a direct
|
|
# dep here). Same primitive tidal-net uses for the gRPC cert resolver.
|
|
arc-swap = "1"
|
|
axum = "0.8"
|
|
# Snapshot-artifact manifest hashing (m11p5 §2): the leader-side
|
|
# NodeSnapshotSource BLAKE3-hashes every staged file once; the puller verifies
|
|
# against the manifest. Same crate + version tidaldb already uses, so no new
|
|
# transitive surface. m11p7 also uses BLAKE3's keyed-hash MODE as the MAC for
|
|
# per-node signed internal tokens (a foreign pod without the cluster key cannot
|
|
# forge one) — no new crypto dependency.
|
|
blake3 = "1"
|
|
# m11p7: base64url for the signed node-token wire form; tokio-rustls serves the
|
|
# inter-node HTTP listener over TLS reusing tidal-net's hot-swappable cert
|
|
# resolver. Both already in the lock (base64 transitively, tokio-rustls via
|
|
# tidal-net); promoted to direct deps here.
|
|
base64 = "0.22"
|
|
tokio-rustls = "0.26"
|
|
clap = { version = "4.5", features = ["derive", "env"] }
|
|
crossbeam = "0.8"
|
|
# Concurrent peer fan-out for cluster broadcast / promote / status aggregation
|
|
# (m8p10 task 03): `join_all` drives every peer request on the handler task,
|
|
# in input order, with no per-peer detached task. `default-features = false`
|
|
# keeps it to the `std` future combinators — no executor, no extra runtime.
|
|
futures-util = { version = "0.3", default-features = false, features = ["std", "async-await"] }
|
|
subtle = "2"
|
|
tower = { version = "0.5", features = ["limit"] }
|
|
tower-http = { version = "0.6", features = ["timeout", "trace", "request-id"] }
|
|
serde = { version = "1", features = ["derive"] }
|
|
serde_json = "1"
|
|
# serde_yml is the maintained fork of the deprecated/unmaintained serde_yaml
|
|
# 0.9 (RUSTSEC-2024-0320). It keeps the same `Value`/`Mapping` API, so the
|
|
# schema/topology/profile parsing below is byte-for-byte behaviour-identical.
|
|
serde_yml = "0.0.12"
|
|
thiserror = "2"
|
|
tokio = { version = "1", features = ["macros", "rt-multi-thread", "signal", "sync"] }
|
|
tracing = "0.1"
|
|
# `json` backs the structured log format (JSON_LOGS=1). Without it span fields
|
|
# are only stored as a human string, so `request_id` could not be lifted into a
|
|
# machine-readable field and per-request log correlation would stay grep-only.
|
|
tracing-subscriber = { version = "0.3", features = ["env-filter", "json"] }
|
|
# RFC3339 timestamps for the structured log wire format. Already compiled in the
|
|
# workspace as a transitive dependency, so this is a promotion to a direct dep
|
|
# rather than new build cost; hand-rolling civil-date arithmetic from
|
|
# SystemTime would be a defect waiting to happen.
|
|
chrono = { version = "0.4", default-features = false, features = ["clock", "std"] }
|
|
# utoipa 5.x derives the OpenAPI 3.1 document (ApiDoc) and per-handler path
|
|
# attributes that back the unauthenticated GET /openapi.json route. JSON-only:
|
|
# no swagger-ui asset bundle (that crate's vendored JS does not pass our
|
|
# -D warnings posture). The `axum_extras` feature is the axum 0.8-compatible
|
|
# integration surface.
|
|
utoipa = { version = "5", features = ["axum_extras"] }
|
|
# `metrics` is load-bearing here (not just a default): cluster mode wires the
|
|
# tidaldb_cluster_* series + the /metrics listener through cfg(metrics)-gated
|
|
# engine APIs, so a no-default-features consumer must still get them.
|
|
tidaldb = { path = "../tidal", features = ["test-utils", "metrics"] }
|
|
tidal-net = { path = "../tidal-net" }
|
|
# Cross-process cluster forwarding/broadcast/reconcile + status aggregation
|
|
# (m8p10 task 03) use the async reqwest client directly on the axum reactor.
|
|
# `default-features = false` + rustls drops the OpenSSL/native-tls system
|
|
# dependency (cleaner static build, matches the rustls posture tonic already
|
|
# uses in tidal-net); `json` for the typed bodies. `blocking` is required in
|
|
# production too: the `/sharded/*` scatter-gather workers are detached OS
|
|
# threads (no tokio runtime) and fetch remote shards over a blocking client —
|
|
# converting that pipeline to async would change the in-process behavior the
|
|
# seam must preserve.
|
|
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls", "blocking"] }
|
|
|
|
[features]
|
|
cluster-e2e = []
|
|
# m11p9 chaos testing: passes through to tidaldb's WAL fault injection. The
|
|
# tier-3 fault suite spawns a binary built with this; production never sets it.
|
|
fault-injection = ["tidaldb/fault-injection"]
|
|
|
|
[dev-dependencies]
|
|
tempfile = "3"
|
|
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls", "blocking"] }
|
|
serde_json = "1"
|
|
criterion = { version = "0.5", features = ["html_reports"] }
|
|
# m11p7 tier-3 security tests: generate a cluster CA + per-node leaf certs (with
|
|
# loopback SANs) at test time so the mTLS cluster boots over real TLS. Same crate
|
|
# tidal-net's mtls.rs uses.
|
|
rcgen = "0.13"
|
|
|
|
[[bench]]
|
|
name = "scatter"
|
|
harness = false
|