388e445a38
Every destructive /cluster/* verb sat behind the SAME bearer as /items and
/search, so any application key could remove a member, force a partition, or
transfer a shard. There was no way to hand out a client credential without also
handing out the ability to destroy the cluster.
Adds TIDAL_ADMIN_KEY (and TIDAL_ADMIN_KEY_FILE, rotatable without restart like
the others). /cluster/promote, /cluster/partition, /cluster/heal,
/cluster/members/remove, /cluster/reseed and /cluster/shards/{id}/{replicas,
transfer} move into their own router subtree behind an admin gate; the data
bearer now gets 403 there - authenticated but not authorized, distinct from the
401 for a bad token.
Three things this had to get right:
* The admin key must ALSO authenticate. A request carries one Authorization
header, so if the admin key did not satisfy the bearer gate, an operator
presenting it would be 401'd before the admin gate ran and the verbs would be
reachable by nobody. Caught while writing the test, not after.
* A verified sibling node token clears the gate too. Nodes relay operator verbs
to the leader/target carrying whatever credential the caller sent, and the
legacy fan-out promote uses the internal marker, so requiring the admin key on
that hop would partition the control plane.
* The peer-callable verbs stay on the plain bearer. /cluster/catchup (self-heal
nudge), /cluster/join + /cluster/members (seed-join) and the
/cluster/reconcile* pair are dialled node-to-node, so gating them would break
replication and joining.
Absent admin key = previous behavior exactly, plus a startup WARN naming the
exposure, so this is safe to upgrade into. The k8s secret mount is optional:true
because without that a deployment lacking the key would fail to MOUNT and never
start.
Also closes the /cluster/status hole this exposed: it and /cluster/status/local
reported leader identity, membership, term and per-shard applied/lag/commit
seqnos from the UNAUTHENTICATED probe group. They are protected now, which is
what k8s/cluster/networkpolicy.yaml deferred to rather than working around at the
network layer.
And fixes a latent bug found on the way: seed-join discovery, reseed discovery
and the self-heal catch-up nudge read std::env::var("TIDAL_API_KEY") directly,
which yields nothing on a *_FILE-only deployment - the node would dial an
authenticated peer with no credential. They use security::bearer_from_env() now,
which honours both shapes.
Verified: 5 new unit tests; two multi-process runbook tests on real 3-process
clusters (data bearer 403 on promote / 204 on signals, admin key 200 on status
and through the gate on heal; bare /cluster/status 401, 200 with the bearer).
That the authenticated cluster converges at all is the load-bearing assertion -
if moving status behind auth had broken leader discovery, startup would hang.
Full unit suites green (2101 + 162), reseed e2e green, clippy clean.
45 lines
1.7 KiB
YAML
45 lines
1.7 KiB
YAML
# tidalDB CLUSTER deployment (m11p5 §4). Apply with:
|
|
# kubectl apply -k k8s/cluster/
|
|
#
|
|
# MUTUALLY EXCLUSIVE with the standalone set (k8s/, namespace `tidaldb`). This
|
|
# set lives in namespace `tidaldb-cluster`. Deploy ONE or the OTHER per
|
|
# namespace — never both (they share the StatefulSet name `tidaldb` and the
|
|
# standalone set's replicas:1 is load-bearing).
|
|
#
|
|
# Create the credentials secret FIRST (deliberately excluded so no key is
|
|
# committed - see secret.example.yaml). Three keys: the data-plane bearer, the
|
|
# OPERATOR key that gates the destructive /cluster/* verbs, and the cluster key:
|
|
# kubectl -n tidaldb-cluster create secret generic tidaldb-credentials \
|
|
# --from-literal=TIDAL_API_KEY="$(openssl rand -hex 32)" \
|
|
# --from-literal=TIDAL_ADMIN_KEY="$(openssl rand -hex 32)" \
|
|
# --from-literal=TIDAL_CLUSTER_KEY="$(openssl rand -hex 32)"
|
|
#
|
|
# Omitting TIDAL_ADMIN_KEY is supported (the mount is optional) but leaves every
|
|
# admin verb reachable with the data bearer; the server WARNs at startup.
|
|
#
|
|
# certs.yaml (m11p7 inter-node TLS) requires cert-manager. If you do NOT run
|
|
# cert-manager, comment certs.yaml out and provision the `tidaldb-cluster-tls`
|
|
# Secret with scripts/gen-cluster-certs.sh instead.
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
|
|
namespace: tidaldb-cluster
|
|
|
|
resources:
|
|
- namespace.yaml
|
|
- schema-configmap.yaml
|
|
- topology-configmap.yaml
|
|
- certs.yaml
|
|
- service-peers.yaml
|
|
- service-client.yaml
|
|
- statefulset.yaml
|
|
- poddisruptionbudget.yaml
|
|
- networkpolicy.yaml
|
|
# Public exposure (Traefik + Let's Encrypt). Remove for internal-only.
|
|
- ingress.yaml
|
|
|
|
labels:
|
|
- pairs:
|
|
app.kubernetes.io/part-of: tidaldb
|
|
includeSelectors: false
|