388e445a38
Every destructive /cluster/* verb sat behind the SAME bearer as /items and
/search, so any application key could remove a member, force a partition, or
transfer a shard. There was no way to hand out a client credential without also
handing out the ability to destroy the cluster.
Adds TIDAL_ADMIN_KEY (and TIDAL_ADMIN_KEY_FILE, rotatable without restart like
the others). /cluster/promote, /cluster/partition, /cluster/heal,
/cluster/members/remove, /cluster/reseed and /cluster/shards/{id}/{replicas,
transfer} move into their own router subtree behind an admin gate; the data
bearer now gets 403 there - authenticated but not authorized, distinct from the
401 for a bad token.
Three things this had to get right:
* The admin key must ALSO authenticate. A request carries one Authorization
header, so if the admin key did not satisfy the bearer gate, an operator
presenting it would be 401'd before the admin gate ran and the verbs would be
reachable by nobody. Caught while writing the test, not after.
* A verified sibling node token clears the gate too. Nodes relay operator verbs
to the leader/target carrying whatever credential the caller sent, and the
legacy fan-out promote uses the internal marker, so requiring the admin key on
that hop would partition the control plane.
* The peer-callable verbs stay on the plain bearer. /cluster/catchup (self-heal
nudge), /cluster/join + /cluster/members (seed-join) and the
/cluster/reconcile* pair are dialled node-to-node, so gating them would break
replication and joining.
Absent admin key = previous behavior exactly, plus a startup WARN naming the
exposure, so this is safe to upgrade into. The k8s secret mount is optional:true
because without that a deployment lacking the key would fail to MOUNT and never
start.
Also closes the /cluster/status hole this exposed: it and /cluster/status/local
reported leader identity, membership, term and per-shard applied/lag/commit
seqnos from the UNAUTHENTICATED probe group. They are protected now, which is
what k8s/cluster/networkpolicy.yaml deferred to rather than working around at the
network layer.
And fixes a latent bug found on the way: seed-join discovery, reseed discovery
and the self-heal catch-up nudge read std::env::var("TIDAL_API_KEY") directly,
which yields nothing on a *_FILE-only deployment - the node would dial an
authenticated peer with no credential. They use security::bearer_from_env() now,
which honours both shapes.
Verified: 5 new unit tests; two multi-process runbook tests on real 3-process
clusters (data bearer 403 on promote / 204 on signals, admin key 200 on status
and through the gate on heal; bare /cluster/status 401, 200 with the bearer).
That the authenticated cluster converges at all is the load-bearing assertion -
if moving status behind auth had broken leader discovery, startup would hang.
Full unit suites green (2101 + 162), reseed e2e green, clippy clean.
124 lines
4.9 KiB
YAML
124 lines
4.9 KiB
YAML
# Public client exposure for the tidalDB cluster (Traefik + Let's Encrypt).
|
|
#
|
|
# OPTIONAL: this file is what publishes the cluster on the open internet. Comment
|
|
# it out of kustomization.yaml for an internal-only deployment.
|
|
#
|
|
# WHAT THIS DELIBERATELY DOES NOT PUBLISH
|
|
# * `/cluster/*` - the destructive admin verbs (`promote`, `partition`, `heal`,
|
|
# `members/remove`, `reseed`, `shards/{id}/transfer`) now require the separate
|
|
# OPERATOR credential (`TIDAL_ADMIN_KEY`), so a data-plane key gets 403. They
|
|
# stay unpublished anyway: operator authority has no business being reachable
|
|
# from the internet, and this is the layer that makes that unconditional.
|
|
# * `/cluster/status` + `/cluster/status/local` - now authenticated (they moved
|
|
# out of the open probe group), but still not published: they report leader
|
|
# identity, membership, term, and per-shard applied/lag/commit seqnos, which
|
|
# no external client needs.
|
|
# * `/openapi.json` - unauthenticated, and enumerates the admin routes above.
|
|
# * `/metrics` - never reachable here: :9091 is published only on the headless
|
|
# `tidaldb-peers` Service, not on the client Service this Ingress targets.
|
|
#
|
|
# CREDENTIALS: the ONLY external credential is tidalDB's bearer token
|
|
# (`TIDAL_API_KEY`). A Traefik `basicAuth` middleware CANNOT be stacked in front
|
|
# of it - both occupy the `Authorization` header, so a client can only ever send
|
|
# one of them. If a second factor is required, use gateway mTLS (a `TLSOption`
|
|
# with `clientAuth`), which occupies a different layer.
|
|
#
|
|
# IP ALLOWLISTING IS NOT AVAILABLE on this fleet: the shared Traefik Service runs
|
|
# `externalTrafficPolicy: Cluster`, so k3s svclb SNATs the client address and an
|
|
# `ipAllowList` sees an internal IP and rejects everyone. Preserving the source IP
|
|
# would require flipping that Service to `Local`, which is fleet-wide and unsafe
|
|
# here (Traefik runs 2 replicas across 3 nodes, so one node IP would blackhole,
|
|
# and 35 Ingresses share it).
|
|
---
|
|
# Traefik must dial the backend over TLS with the internal CA, and can VERIFY it:
|
|
# every pod mounts the same `tidaldb-cluster-tls` leaf, whose SANs include the
|
|
# client-Service DNS name, so pinning `serverName` to that validates the hop while
|
|
# connecting to a pod IP. No `insecureSkipVerify`.
|
|
apiVersion: traefik.io/v1alpha1
|
|
kind: ServersTransport
|
|
metadata:
|
|
name: tidaldb-internal
|
|
namespace: tidaldb-cluster
|
|
labels:
|
|
app.kubernetes.io/name: tidaldb
|
|
app.kubernetes.io/part-of: tidaldb
|
|
spec:
|
|
serverName: tidaldb.tidaldb-cluster.svc.cluster.local
|
|
rootCAsSecrets:
|
|
- tidaldb-cluster-ca
|
|
---
|
|
# Gateway-level cap. tidalDB's OWN limiter is unlimited unless
|
|
# `TIDAL_RATE_LIMIT_RPS` is set, and it keys per PRINCIPAL - with one shared
|
|
# bearer every caller is the same principal, so it is a single global bucket
|
|
# either way. Because svclb SNATs the source, this Traefik limit is likewise
|
|
# effectively global rather than per-client.
|
|
apiVersion: traefik.io/v1alpha1
|
|
kind: Middleware
|
|
metadata:
|
|
name: tidaldb-ratelimit
|
|
namespace: tidaldb-cluster
|
|
labels:
|
|
app.kubernetes.io/name: tidaldb
|
|
app.kubernetes.io/part-of: tidaldb
|
|
spec:
|
|
rateLimit:
|
|
average: 200
|
|
burst: 400
|
|
---
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: Ingress
|
|
metadata:
|
|
name: tidaldb
|
|
namespace: tidaldb-cluster
|
|
labels:
|
|
app.kubernetes.io/name: tidaldb
|
|
app.kubernetes.io/part-of: tidaldb
|
|
annotations:
|
|
# dns01, NOT http01: any gateway middleware that rejects unknown callers also
|
|
# rejects Let's Encrypt's http01 challenge. The Cloudflare DNS solver has no
|
|
# such dependency. `letsencrypt-prod` carries the threesix.ai dns01 solver.
|
|
cert-manager.io/cluster-issuer: letsencrypt-prod
|
|
traefik.ingress.kubernetes.io/router.entrypoints: websecure
|
|
traefik.ingress.kubernetes.io/router.tls: "true"
|
|
traefik.ingress.kubernetes.io/router.middlewares: tidaldb-cluster-tidaldb-ratelimit@kubernetescrd
|
|
spec:
|
|
ingressClassName: traefik
|
|
tls:
|
|
- hosts:
|
|
- tidaldb.threesix.ai
|
|
secretName: tidaldb-public-tls
|
|
rules:
|
|
- host: tidaldb.threesix.ai
|
|
http:
|
|
paths:
|
|
- path: /items
|
|
pathType: Prefix
|
|
backend: &client
|
|
service:
|
|
name: tidaldb
|
|
port:
|
|
number: 9500
|
|
- path: /embeddings
|
|
pathType: Prefix
|
|
backend: *client
|
|
- path: /signals
|
|
pathType: Prefix
|
|
backend: *client
|
|
- path: /hardnegs
|
|
pathType: Prefix
|
|
backend: *client
|
|
- path: /feed
|
|
pathType: Prefix
|
|
backend: *client
|
|
- path: /search
|
|
pathType: Prefix
|
|
backend: *client
|
|
- path: /vector_search
|
|
pathType: Prefix
|
|
backend: *client
|
|
# Unauthenticated by design (probe contract). It does report leader and
|
|
# region names; drop this path if that is unacceptable externally.
|
|
- path: /health
|
|
pathType: Exact
|
|
backend: *client
|