0a861d9144
Before this, ANY pod in the k3s cluster could read tidalDB's :9091 metrics - corpus size, seqnos, leader identity, all unauthenticated - and reach the peer gRPC plane. Measured, not assumed: scraping tidaldb-0:9091 from gitea-0 in namespace threesix returned metrics, and returns "connection refused" after this. NetworkPolicy enforcement on this k3s is therefore confirmed, not presumed. Ingress only. Egress is left unrestricted deliberately - the WAL-archival/S3 and peer dial-out surfaces are not fully enumerated, and a wrong egress rule partitions the cluster instead of merely blocking a scrape. :9500 stays open, and the file records why. All three probes (startup, readiness, liveness) target it, probes come from the NODE rather than a pod, and node-to-pod handling is CNI-specific - a wrong rule there fails liveness at 6x10s and restarts every pod. The exposure that would have closed is /cluster/status, which is unauthenticated by design. That is an engine defect and gets fixed in tidal-server, not worked around at the network layer. Verified after apply: scraper still collects 353 tidaldb_ series, a quorum-acked write through the public ingress returns 201, and all three regions report applied=13322229 lag=0 reachable partitioned=false with zero new restarts.
63 lines
2.7 KiB
YAML
63 lines
2.7 KiB
YAML
# Ingress isolation for the tidalDB cluster.
|
|
#
|
|
# WHY: before this existed, ANY pod in the k3s cluster could read :9091 metrics
|
|
# (unauthenticated - corpus size, seqnos, leader identity) and reach the peer gRPC
|
|
# plane. Verified by scraping tidaldb-0:9091 from an unrelated pod in another
|
|
# namespace.
|
|
#
|
|
# INGRESS ONLY, deliberately. Egress is left unrestricted: the WAL-archival / S3
|
|
# paths and the peer dial-out surface are not fully enumerated here, and a wrong
|
|
# egress rule partitions the cluster rather than merely blocking a scrape.
|
|
#
|
|
# :9500 IS INTENTIONALLY LEFT OPEN, and stays that way. It is the only port the
|
|
# kubelet probes - startup /health/startup, readiness /health, AND liveness
|
|
# /health/live all target it - and probes originate from the NODE, not a pod.
|
|
# Node-to-pod handling differs between CNIs, so a wrong rule here fails liveness
|
|
# (threshold 6 x 10s) and RESTARTS every pod.
|
|
#
|
|
# The exposure this would have closed is `/cluster/status`, which is
|
|
# unauthenticated by design and leaks leader identity, membership and seqnos to
|
|
# any in-cluster pod. That is an ENGINE defect, not a network one, and network
|
|
# rules are the wrong layer to fix it: the route is moved behind the bearer in
|
|
# tidal-server instead. Every data and admin route on this port is already
|
|
# bearer-gated.
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: NetworkPolicy
|
|
metadata:
|
|
name: tidaldb
|
|
namespace: tidaldb-cluster
|
|
labels:
|
|
app.kubernetes.io/name: tidaldb
|
|
app.kubernetes.io/part-of: tidaldb
|
|
spec:
|
|
podSelector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: tidaldb
|
|
app.kubernetes.io/component: cluster-node
|
|
policyTypes:
|
|
- Ingress
|
|
ingress:
|
|
# 1. Same namespace, every port: the sibling replication plane (gRPC 9601-9603
|
|
# plus the inter-node HTTP forwards on 9500) and the soak monitor.
|
|
- from:
|
|
- podSelector: {}
|
|
# 2. The client + health port from anywhere. See the header for why this is
|
|
# not narrowed to kube-system: kubelet probes come from the node.
|
|
# To tighten later, add an ipBlock for the node InternalIPs
|
|
# (208.122.204.172/173/174) alongside a kube-system namespaceSelector, then
|
|
# confirm readiness holds on all three pods before trusting it.
|
|
- ports:
|
|
- protocol: TCP
|
|
port: 9500
|
|
# 3. Metrics: the scraper only. tidalDB's :9091 has no authentication, so this
|
|
# is the rule that actually closes the leak. Pods carry
|
|
# prometheus.io/scrape, so discovery is annotation-based from vmagent in
|
|
# ns observability - there is no ServiceMonitor.
|
|
- from:
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: observability
|
|
ports:
|
|
- protocol: TCP
|
|
port: 9091
|