Every destructive /cluster/* verb sat behind the SAME bearer as /items and
/search, so any application key could remove a member, force a partition, or
transfer a shard. There was no way to hand out a client credential without also
handing out the ability to destroy the cluster.
Adds TIDAL_ADMIN_KEY (and TIDAL_ADMIN_KEY_FILE, rotatable without restart like
the others). /cluster/promote, /cluster/partition, /cluster/heal,
/cluster/members/remove, /cluster/reseed and /cluster/shards/{id}/{replicas,
transfer} move into their own router subtree behind an admin gate; the data
bearer now gets 403 there - authenticated but not authorized, distinct from the
401 for a bad token.
Three things this had to get right:
* The admin key must ALSO authenticate. A request carries one Authorization
header, so if the admin key did not satisfy the bearer gate, an operator
presenting it would be 401'd before the admin gate ran and the verbs would be
reachable by nobody. Caught while writing the test, not after.
* A verified sibling node token clears the gate too. Nodes relay operator verbs
to the leader/target carrying whatever credential the caller sent, and the
legacy fan-out promote uses the internal marker, so requiring the admin key on
that hop would partition the control plane.
* The peer-callable verbs stay on the plain bearer. /cluster/catchup (self-heal
nudge), /cluster/join + /cluster/members (seed-join) and the
/cluster/reconcile* pair are dialled node-to-node, so gating them would break
replication and joining.
Absent admin key = previous behavior exactly, plus a startup WARN naming the
exposure, so this is safe to upgrade into. The k8s secret mount is optional:true
because without that a deployment lacking the key would fail to MOUNT and never
start.
Also closes the /cluster/status hole this exposed: it and /cluster/status/local
reported leader identity, membership, term and per-shard applied/lag/commit
seqnos from the UNAUTHENTICATED probe group. They are protected now, which is
what k8s/cluster/networkpolicy.yaml deferred to rather than working around at the
network layer.
And fixes a latent bug found on the way: seed-join discovery, reseed discovery
and the self-heal catch-up nudge read std::env::var("TIDAL_API_KEY") directly,
which yields nothing on a *_FILE-only deployment - the node would dial an
authenticated peer with no credential. They use security::bearer_from_env() now,
which honours both shapes.
Verified: 5 new unit tests; two multi-process runbook tests on real 3-process
clusters (data bearer 403 on promote / 204 on signals, admin key 200 on status
and through the gate on heal; bare /cluster/status 401, 200 with the bearer).
That the authenticated cluster converges at all is the load-bearing assertion -
if moving status behind auth had broken leader discovery, startup would hang.
Full unit suites green (2101 + 162), reseed e2e green, clippy clean.
Before this, ANY pod in the k3s cluster could read tidalDB's :9091 metrics -
corpus size, seqnos, leader identity, all unauthenticated - and reach the peer
gRPC plane. Measured, not assumed: scraping tidaldb-0:9091 from gitea-0 in
namespace threesix returned metrics, and returns "connection refused" after this.
NetworkPolicy enforcement on this k3s is therefore confirmed, not presumed.
Ingress only. Egress is left unrestricted deliberately - the WAL-archival/S3 and
peer dial-out surfaces are not fully enumerated, and a wrong egress rule
partitions the cluster instead of merely blocking a scrape.
:9500 stays open, and the file records why. All three probes (startup, readiness,
liveness) target it, probes come from the NODE rather than a pod, and node-to-pod
handling is CNI-specific - a wrong rule there fails liveness at 6x10s and restarts
every pod. The exposure that would have closed is /cluster/status, which is
unauthenticated by design. That is an engine defect and gets fixed in
tidal-server, not worked around at the network layer.
Verified after apply: scraper still collects 353 tidaldb_ series, a quorum-acked
write through the public ingress returns 201, and all three regions report
applied=13322229 lag=0 reachable partitioned=false with zero new restarts.
tidaldb.threesix.ai now serves the cluster's data surface over a Let's Encrypt
cert, verified end-to-end from the internet: 401 without a bearer, 401 with a
wrong one, 200 with the real key, and a quorum-acked write returning 201 on all
three node IPs.
Three things this had to get right, each of which failed first:
* The backend is HTTPS, not HTTP. Pods serve :9500 over TLS with the internal
cluster CA whenever grpc_tls is configured, so a plaintext backend dial answers
500. Added a ServersTransport that VERIFIES that hop - every pod mounts the same
tidaldb-cluster-tls leaf and its SANs include the client-Service DNS name, so
serverName pinning validates it without insecureSkipVerify.
* `service.*` annotations are read from the Service, not the Ingress. Putting
serversscheme/serverstransport on the Ingress is silently ignored and presents
exactly as a broken backend.
* http01 cannot be used behind any gateway gate that rejects unknown callers,
because it rejects the ACME challenge too. Uses the Cloudflare dns01 solver.
Deliberately unpublished: /cluster/* (every mutating admin verb shares the SAME
single bearer as the data routes, so a client key could remove members or transfer
shards), /cluster/status (unauthenticated - leaks leader, membership, seqnos),
/openapi.json (unauthenticated, enumerates the admin routes), and /metrics (only
on the headless peers Service, unreachable here).
Documents two controls that are NOT available and why: an IP allowlist cannot work
while the shared Traefik Service runs externalTrafficPolicy=Cluster (svclb SNATs
the client address), and Traefik basicAuth cannot stack in front of the bearer
because both occupy the Authorization header.
