tidaldb/tidal-server
jordan 67a175e19a
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
fix(cluster): /cluster/status reported a total partition on a healthy fleet
The status aggregator probed every peer's /cluster/status/local with NO
credential. That route is token-gated, so on any cluster with TIDAL_API_KEY set
each peer answered 401, and every peer row collapsed to the honest-unknown
placeholder: reachable false, partitioned true, applied_events null,
lag_events null, version "".

Only the OWN region survived, because it is served in-process with no HTTP hop.
The result: the one surface an operator reads to clear the N/N+1 version skew
before a rolling upgrade - and the deploy runbook's own step 4 - reported the
whole cluster partitioned while it was perfectly healthy, with every peer's
version blank so the skew check was blind.

Observed on the GKE cluster: all three pods answered /health 200 under leader
tidaldb-1, replication applied, and a curl between the exact same pod FQDNs
returned 200, while /cluster/status insisted both siblings were unreachable.

security::bearer_from_env documents this precise trap - a node that "dials an
authenticated peer with NO credential" - and count_alive_other_voters already
attaches the bearer. This forwards the CALLER's Authorization header instead of
reaching for creds.bearer(), matching the relayed-operator-hop convention that
/cluster/promote already uses, so a weakly-authenticated caller cannot borrow
the node's own credential to read peers it could not read directly.

Why it escaped: every other multi-process test runs with no TIDAL_API_KEY, where
a credential-less probe succeeds - including cluster_multiproc's all-reachable
assertion. The new test carries the key. Verified differential: it fails on the
reverted code with exactly the observed shape (us-east reachable, both peers
null/false/empty) and passes with the fix. cluster_multiproc still 5/5.
2026-09-16 00:51:57 -06:00
..
benches feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
config feat: kubernetes deployment, OpenAPI spec, guides, and docker consolidation 2026-06-09 17:06:34 -06:00
src fix(cluster): /cluster/status reported a total partition on a healthy fleet 2026-09-16 00:51:57 -06:00
tests fix(cluster): /cluster/status reported a total partition on a healthy fleet 2026-09-16 00:51:57 -06:00
BUILD.bazel feat: Bazel build, crate docs/ai-lookup, docker images, and engine hardening 2026-06-07 18:29:38 -06:00
Cargo.toml feat(observability): HTTP metrics, structured logs, dashboard, live tidalctl 2026-08-23 10:31:57 -06:00