Before this, ANY pod in the k3s cluster could read tidalDB's :9091 metrics -
corpus size, seqnos, leader identity, all unauthenticated - and reach the peer
gRPC plane. Measured, not assumed: scraping tidaldb-0:9091 from gitea-0 in
namespace threesix returned metrics, and returns "connection refused" after this.
NetworkPolicy enforcement on this k3s is therefore confirmed, not presumed.
Ingress only. Egress is left unrestricted deliberately - the WAL-archival/S3 and
peer dial-out surfaces are not fully enumerated, and a wrong egress rule
partitions the cluster instead of merely blocking a scrape.
:9500 stays open, and the file records why. All three probes (startup, readiness,
liveness) target it, probes come from the NODE rather than a pod, and node-to-pod
handling is CNI-specific - a wrong rule there fails liveness at 6x10s and restarts
every pod. The exposure that would have closed is /cluster/status, which is
unauthenticated by design. That is an engine defect and gets fixed in
tidal-server, not worked around at the network layer.
Verified after apply: scraper still collects 353 tidaldb_ series, a quorum-acked
write through the public ingress returns 201, and all three regions report
applied=13322229 lag=0 reachable partitioned=false with zero new restarts.
tidaldb.threesix.ai now serves the cluster's data surface over a Let's Encrypt
cert, verified end-to-end from the internet: 401 without a bearer, 401 with a
wrong one, 200 with the real key, and a quorum-acked write returning 201 on all
three node IPs.
Three things this had to get right, each of which failed first:
* The backend is HTTPS, not HTTP. Pods serve :9500 over TLS with the internal
cluster CA whenever grpc_tls is configured, so a plaintext backend dial answers
500. Added a ServersTransport that VERIFIES that hop - every pod mounts the same
tidaldb-cluster-tls leaf and its SANs include the client-Service DNS name, so
serverName pinning validates it without insecureSkipVerify.
* `service.*` annotations are read from the Service, not the Ingress. Putting
serversscheme/serverstransport on the Ingress is silently ignored and presents
exactly as a broken backend.
* http01 cannot be used behind any gateway gate that rejects unknown callers,
because it rejects the ACME challenge too. Uses the Cloudflare dns01 solver.
Deliberately unpublished: /cluster/* (every mutating admin verb shares the SAME
single bearer as the data routes, so a client key could remove members or transfer
shards), /cluster/status (unauthenticated - leaks leader, membership, seqnos),
/openapi.json (unauthenticated, enumerates the admin routes), and /metrics (only
on the headless peers Service, unreachable here).
Documents two controls that are NOT available and why: an IP allowlist cannot work
while the shared Traefik Service runs externalTrafficPolicy=Cluster (svclb SNATs
the client address), and Traefik basicAuth cannot stack in front of the bearer
because both occupy the Authorization header.
