087b83154a
tidaldb.threesix.ai now serves the cluster's data surface over a Let's Encrypt cert, verified end-to-end from the internet: 401 without a bearer, 401 with a wrong one, 200 with the real key, and a quorum-acked write returning 201 on all three node IPs. Three things this had to get right, each of which failed first: * The backend is HTTPS, not HTTP. Pods serve :9500 over TLS with the internal cluster CA whenever grpc_tls is configured, so a plaintext backend dial answers 500. Added a ServersTransport that VERIFIES that hop - every pod mounts the same tidaldb-cluster-tls leaf and its SANs include the client-Service DNS name, so serverName pinning validates it without insecureSkipVerify. * `service.*` annotations are read from the Service, not the Ingress. Putting serversscheme/serverstransport on the Ingress is silently ignored and presents exactly as a broken backend. * http01 cannot be used behind any gateway gate that rejects unknown callers, because it rejects the ACME challenge too. Uses the Cloudflare dns01 solver. Deliberately unpublished: /cluster/* (every mutating admin verb shares the SAME single bearer as the data routes, so a client key could remove members or transfer shards), /cluster/status (unauthenticated - leaks leader, membership, seqnos), /openapi.json (unauthenticated, enumerates the admin routes), and /metrics (only on the headless peers Service, unreachable here). Documents two controls that are NOT available and why: an IP allowlist cannot work while the shared Traefik Service runs externalTrafficPolicy=Cluster (svclb SNATs the client address), and Traefik basicAuth cannot stack in front of the bearer because both occupy the Authorization header.
43 lines
1.9 KiB
YAML
43 lines
1.9 KiB
YAML
# Client-facing Service (m11p5 §4) — readiness-gated load balancing.
|
|
#
|
|
# This is the address in-cluster clients hit for HTTP reads/writes. It is a
|
|
# normal (VIP) Service, so the default readiness gating applies: a pod that the
|
|
# m11p5 readiness predicate marks 503 (a joiner before first-convergence, a
|
|
# quarantined node, a removed/reseeding node, a draining node on SIGTERM) is
|
|
# REMOVED from this Service's endpoints and receives no client traffic. Only
|
|
# converged, serving voters answer clients here.
|
|
#
|
|
# Contrast with tidaldb-peers (headless, publishNotReadyAddresses: true): peers
|
|
# must still resolve a not-ready joiner to feed it a snapshot + stream, so that
|
|
# Service keeps not-ready pods. This Service must NOT — a client must never be
|
|
# routed to a node still catching up.
|
|
#
|
|
# Forwarding works regardless of which pod a client lands on: any region forwards
|
|
# writes to the current leader and reads serve locally. Front this with an
|
|
# Ingress/Gateway for external traffic; never expose :9091 metrics externally
|
|
# (unauthenticated).
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: tidaldb
|
|
namespace: tidaldb-cluster
|
|
labels:
|
|
app.kubernetes.io/name: tidaldb
|
|
app.kubernetes.io/part-of: tidaldb
|
|
annotations:
|
|
# Traefik dials this backend over TLS. The pods serve :9500 as HTTPS whenever
|
|
# grpc_tls is configured (tidal-server `http_tls`), so a plaintext backend
|
|
# dial answers 500. These MUST live on the Service - `service.*` annotations
|
|
# are read from the Service, NOT the Ingress (putting them on the Ingress is
|
|
# silently ignored and looks exactly like a broken backend).
|
|
traefik.ingress.kubernetes.io/service.serversscheme: https
|
|
traefik.ingress.kubernetes.io/service.serverstransport: tidaldb-cluster-tidaldb-internal@kubernetescrd
|
|
spec:
|
|
selector:
|
|
app.kubernetes.io/name: tidaldb
|
|
app.kubernetes.io/component: cluster-node
|
|
ports:
|
|
- name: http
|
|
port: 9500
|
|
targetPort: http
|