tidaldb.threesix.ai now serves the cluster's data surface over a Let's Encrypt
cert, verified end-to-end from the internet: 401 without a bearer, 401 with a
wrong one, 200 with the real key, and a quorum-acked write returning 201 on all
three node IPs.
Three things this had to get right, each of which failed first:
* The backend is HTTPS, not HTTP. Pods serve :9500 over TLS with the internal
cluster CA whenever grpc_tls is configured, so a plaintext backend dial answers
500. Added a ServersTransport that VERIFIES that hop - every pod mounts the same
tidaldb-cluster-tls leaf and its SANs include the client-Service DNS name, so
serverName pinning validates it without insecureSkipVerify.
* `service.*` annotations are read from the Service, not the Ingress. Putting
serversscheme/serverstransport on the Ingress is silently ignored and presents
exactly as a broken backend.
* http01 cannot be used behind any gateway gate that rejects unknown callers,
because it rejects the ACME challenge too. Uses the Cloudflare dns01 solver.
Deliberately unpublished: /cluster/* (every mutating admin verb shares the SAME
single bearer as the data routes, so a client key could remove members or transfer
shards), /cluster/status (unauthenticated - leaks leader, membership, seqnos),
/openapi.json (unauthenticated, enumerates the admin routes), and /metrics (only
on the headless peers Service, unreachable here).
Documents two controls that are NOT available and why: an IP allowlist cannot work
while the shared Traefik Service runs externalTrafficPolicy=Cluster (svclb SNATs
the client address), and Traefik basicAuth cannot stack in front of the bearer
because both occupy the Authorization header.
