388e445a38
Every destructive /cluster/* verb sat behind the SAME bearer as /items and
/search, so any application key could remove a member, force a partition, or
transfer a shard. There was no way to hand out a client credential without also
handing out the ability to destroy the cluster.
Adds TIDAL_ADMIN_KEY (and TIDAL_ADMIN_KEY_FILE, rotatable without restart like
the others). /cluster/promote, /cluster/partition, /cluster/heal,
/cluster/members/remove, /cluster/reseed and /cluster/shards/{id}/{replicas,
transfer} move into their own router subtree behind an admin gate; the data
bearer now gets 403 there - authenticated but not authorized, distinct from the
401 for a bad token.
Three things this had to get right:
* The admin key must ALSO authenticate. A request carries one Authorization
header, so if the admin key did not satisfy the bearer gate, an operator
presenting it would be 401'd before the admin gate ran and the verbs would be
reachable by nobody. Caught while writing the test, not after.
* A verified sibling node token clears the gate too. Nodes relay operator verbs
to the leader/target carrying whatever credential the caller sent, and the
legacy fan-out promote uses the internal marker, so requiring the admin key on
that hop would partition the control plane.
* The peer-callable verbs stay on the plain bearer. /cluster/catchup (self-heal
nudge), /cluster/join + /cluster/members (seed-join) and the
/cluster/reconcile* pair are dialled node-to-node, so gating them would break
replication and joining.
Absent admin key = previous behavior exactly, plus a startup WARN naming the
exposure, so this is safe to upgrade into. The k8s secret mount is optional:true
because without that a deployment lacking the key would fail to MOUNT and never
start.
Also closes the /cluster/status hole this exposed: it and /cluster/status/local
reported leader identity, membership, term and per-shard applied/lag/commit
seqnos from the UNAUTHENTICATED probe group. They are protected now, which is
what k8s/cluster/networkpolicy.yaml deferred to rather than working around at the
network layer.
And fixes a latent bug found on the way: seed-join discovery, reseed discovery
and the self-heal catch-up nudge read std::env::var("TIDAL_API_KEY") directly,
which yields nothing on a *_FILE-only deployment - the node would dial an
authenticated peer with no credential. They use security::bearer_from_env() now,
which honours both shapes.
Verified: 5 new unit tests; two multi-process runbook tests on real 3-process
clusters (data bearer 403 on promote / 204 on signals, admin key 200 on status
and through the gate on heal; bare /cluster/status 401, 200 with the bearer).
That the authenticated cluster converges at all is the load-bearing assertion -
if moving status behind auth had broken leader discovery, startup would hang.
Full unit suites green (2101 + 162), reseed e2e green, clippy clean.
50 lines
2.6 KiB
YAML
50 lines
2.6 KiB
YAML
# TEMPLATE ONLY — do NOT commit a real key. The real secret is created
|
|
# out-of-band and is deliberately excluded from kustomization.yaml so no key
|
|
# lands in git.
|
|
#
|
|
# Secret shape: `tidaldb-credentials` with up to THREE keys:
|
|
# - TIDAL_API_KEY — the DATA-PLANE bearer. Every pod and every external
|
|
# client uses this one (forwarded requests pass the caller's Authorization
|
|
# verbatim). Hand this to applications.
|
|
# - TIDAL_ADMIN_KEY — OPTIONAL but strongly recommended: the OPERATOR
|
|
# credential. Without it the destructive verbs (/cluster/promote,
|
|
# /cluster/partition, /cluster/heal, /cluster/members/remove,
|
|
# /cluster/reseed, /cluster/shards/*) accept the DATA bearer, so any
|
|
# application key can remove a member or move a shard. With it they require
|
|
# this key (or a verified sibling node token) and the data bearer gets 403.
|
|
# It is a SUPERSET credential - it also authenticates the data routes - so
|
|
# keep it off application hosts. Mounted as a FILE
|
|
# (TIDAL_ADMIN_KEY_FILE) so rotation needs no restart.
|
|
# - TIDAL_CLUSTER_KEY — m11p7: the SHARED CLUSTER KEY. Mints/verifies per-node
|
|
# signed internal tokens so inter-node HTTP carries verifiable node identity
|
|
# and the x-tidal-internal marker is honored ONLY from a verified sibling.
|
|
# Mounted as a FILE (TIDAL_CLUSTER_KEY_FILE) so a rotation is hot (no restart).
|
|
# Known ONLY to cluster pods — never hand it to external clients.
|
|
#
|
|
# Create the real one (do not apply this file):
|
|
# kubectl -n tidaldb-cluster create secret generic tidaldb-credentials \
|
|
# --from-literal=TIDAL_API_KEY="$(openssl rand -hex 32)" \
|
|
# --from-literal=TIDAL_ADMIN_KEY="$(openssl rand -hex 32)" \
|
|
# --from-literal=TIDAL_CLUSTER_KEY="$(openssl rand -hex 32)"
|
|
#
|
|
# Inter-node TLS material is a SEPARATE Secret (`tidaldb-cluster-tls`), issued by
|
|
# cert-manager (certs.yaml) or provisioned with scripts/gen-cluster-certs.sh.
|
|
#
|
|
# In production manage these with External Secrets Operator / Sealed Secrets /
|
|
# Vault. An empty TIDAL_API_KEY runs UNAUTHENTICATED (WARN); an absent
|
|
# TIDAL_CLUSTER_KEY disables per-node tokens (the marker keeps hint-only
|
|
# behavior, WARN) — never do either on a shared network.
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: tidaldb-credentials
|
|
namespace: tidaldb-cluster
|
|
labels:
|
|
app.kubernetes.io/name: tidaldb
|
|
app.kubernetes.io/part-of: tidaldb
|
|
type: Opaque
|
|
stringData:
|
|
TIDAL_API_KEY: "replace-me-do-not-commit"
|
|
TIDAL_ADMIN_KEY: "replace-me-do-not-commit-distinct-from-api-key"
|
|
TIDAL_CLUSTER_KEY: "replace-me-do-not-commit-distinct-from-api-key"
|