087b83154a
tidaldb.threesix.ai now serves the cluster's data surface over a Let's Encrypt cert, verified end-to-end from the internet: 401 without a bearer, 401 with a wrong one, 200 with the real key, and a quorum-acked write returning 201 on all three node IPs. Three things this had to get right, each of which failed first: * The backend is HTTPS, not HTTP. Pods serve :9500 over TLS with the internal cluster CA whenever grpc_tls is configured, so a plaintext backend dial answers 500. Added a ServersTransport that VERIFIES that hop - every pod mounts the same tidaldb-cluster-tls leaf and its SANs include the client-Service DNS name, so serverName pinning validates it without insecureSkipVerify. * `service.*` annotations are read from the Service, not the Ingress. Putting serversscheme/serverstransport on the Ingress is silently ignored and presents exactly as a broken backend. * http01 cannot be used behind any gateway gate that rejects unknown callers, because it rejects the ACME challenge too. Uses the Cloudflare dns01 solver. Deliberately unpublished: /cluster/* (every mutating admin verb shares the SAME single bearer as the data routes, so a client key could remove members or transfer shards), /cluster/status (unauthenticated - leaks leader, membership, seqnos), /openapi.json (unauthenticated, enumerates the admin routes), and /metrics (only on the headless peers Service, unreachable here). Documents two controls that are NOT available and why: an IP allowlist cannot work while the shared Traefik Service runs externalTrafficPolicy=Cluster (svclb SNATs the client address), and Traefik basicAuth cannot stack in front of the bearer because both occupy the Authorization header.
123 lines
4.9 KiB
YAML
123 lines
4.9 KiB
YAML
# Public client exposure for the tidalDB cluster (Traefik + Let's Encrypt).
|
|
#
|
|
# OPTIONAL: this file is what publishes the cluster on the open internet. Comment
|
|
# it out of kustomization.yaml for an internal-only deployment.
|
|
#
|
|
# WHAT THIS DELIBERATELY DOES NOT PUBLISH
|
|
# * `/cluster/*` - every mutating admin verb (`promote`, `partition`, `heal`,
|
|
# `members/remove`, `join`, `reseed`, `shards/{id}/transfer`) sits behind the
|
|
# SAME single bearer token as the data routes (tidal-server cluster/node.rs
|
|
# `protected`). There is no operator/data credential split, so publishing
|
|
# these would let any client key destroy the cluster.
|
|
# * `/cluster/status` + `/cluster/status/local` - these are UNAUTHENTICATED
|
|
# (they live in the `public` router next to the health probes) and report
|
|
# leader identity, membership, term, and per-shard applied/lag/commit seqnos.
|
|
# * `/openapi.json` - unauthenticated, and enumerates the admin routes above.
|
|
# * `/metrics` - never reachable here: :9091 is published only on the headless
|
|
# `tidaldb-peers` Service, not on the client Service this Ingress targets.
|
|
#
|
|
# CREDENTIALS: the ONLY external credential is tidalDB's bearer token
|
|
# (`TIDAL_API_KEY`). A Traefik `basicAuth` middleware CANNOT be stacked in front
|
|
# of it - both occupy the `Authorization` header, so a client can only ever send
|
|
# one of them. If a second factor is required, use gateway mTLS (a `TLSOption`
|
|
# with `clientAuth`), which occupies a different layer.
|
|
#
|
|
# IP ALLOWLISTING IS NOT AVAILABLE on this fleet: the shared Traefik Service runs
|
|
# `externalTrafficPolicy: Cluster`, so k3s svclb SNATs the client address and an
|
|
# `ipAllowList` sees an internal IP and rejects everyone. Preserving the source IP
|
|
# would require flipping that Service to `Local`, which is fleet-wide and unsafe
|
|
# here (Traefik runs 2 replicas across 3 nodes, so one node IP would blackhole,
|
|
# and 35 Ingresses share it).
|
|
---
|
|
# Traefik must dial the backend over TLS with the internal CA, and can VERIFY it:
|
|
# every pod mounts the same `tidaldb-cluster-tls` leaf, whose SANs include the
|
|
# client-Service DNS name, so pinning `serverName` to that validates the hop while
|
|
# connecting to a pod IP. No `insecureSkipVerify`.
|
|
apiVersion: traefik.io/v1alpha1
|
|
kind: ServersTransport
|
|
metadata:
|
|
name: tidaldb-internal
|
|
namespace: tidaldb-cluster
|
|
labels:
|
|
app.kubernetes.io/name: tidaldb
|
|
app.kubernetes.io/part-of: tidaldb
|
|
spec:
|
|
serverName: tidaldb.tidaldb-cluster.svc.cluster.local
|
|
rootCAsSecrets:
|
|
- tidaldb-cluster-ca
|
|
---
|
|
# Gateway-level cap. tidalDB's OWN limiter is unlimited unless
|
|
# `TIDAL_RATE_LIMIT_RPS` is set, and it keys per PRINCIPAL - with one shared
|
|
# bearer every caller is the same principal, so it is a single global bucket
|
|
# either way. Because svclb SNATs the source, this Traefik limit is likewise
|
|
# effectively global rather than per-client.
|
|
apiVersion: traefik.io/v1alpha1
|
|
kind: Middleware
|
|
metadata:
|
|
name: tidaldb-ratelimit
|
|
namespace: tidaldb-cluster
|
|
labels:
|
|
app.kubernetes.io/name: tidaldb
|
|
app.kubernetes.io/part-of: tidaldb
|
|
spec:
|
|
rateLimit:
|
|
average: 200
|
|
burst: 400
|
|
---
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: Ingress
|
|
metadata:
|
|
name: tidaldb
|
|
namespace: tidaldb-cluster
|
|
labels:
|
|
app.kubernetes.io/name: tidaldb
|
|
app.kubernetes.io/part-of: tidaldb
|
|
annotations:
|
|
# dns01, NOT http01: any gateway middleware that rejects unknown callers also
|
|
# rejects Let's Encrypt's http01 challenge. The Cloudflare DNS solver has no
|
|
# such dependency. `letsencrypt-prod` carries the threesix.ai dns01 solver.
|
|
cert-manager.io/cluster-issuer: letsencrypt-prod
|
|
traefik.ingress.kubernetes.io/router.entrypoints: websecure
|
|
traefik.ingress.kubernetes.io/router.tls: "true"
|
|
traefik.ingress.kubernetes.io/router.middlewares: tidaldb-cluster-tidaldb-ratelimit@kubernetescrd
|
|
spec:
|
|
ingressClassName: traefik
|
|
tls:
|
|
- hosts:
|
|
- tidaldb.threesix.ai
|
|
secretName: tidaldb-public-tls
|
|
rules:
|
|
- host: tidaldb.threesix.ai
|
|
http:
|
|
paths:
|
|
- path: /items
|
|
pathType: Prefix
|
|
backend: &client
|
|
service:
|
|
name: tidaldb
|
|
port:
|
|
number: 9500
|
|
- path: /embeddings
|
|
pathType: Prefix
|
|
backend: *client
|
|
- path: /signals
|
|
pathType: Prefix
|
|
backend: *client
|
|
- path: /hardnegs
|
|
pathType: Prefix
|
|
backend: *client
|
|
- path: /feed
|
|
pathType: Prefix
|
|
backend: *client
|
|
- path: /search
|
|
pathType: Prefix
|
|
backend: *client
|
|
- path: /vector_search
|
|
pathType: Prefix
|
|
backend: *client
|
|
# Unauthenticated by design (probe contract). It does report leader and
|
|
# region names; drop this path if that is unacceptable externally.
|
|
- path: /health
|
|
pathType: Exact
|
|
backend: *client
|