k8s(cluster): close the unauthenticated metrics and peer plane to foreign pods

Before this, ANY pod in the k3s cluster could read tidalDB's :9091 metrics -
corpus size, seqnos, leader identity, all unauthenticated - and reach the peer
gRPC plane. Measured, not assumed: scraping tidaldb-0:9091 from gitea-0 in
namespace threesix returned metrics, and returns "connection refused" after this.
NetworkPolicy enforcement on this k3s is therefore confirmed, not presumed.

Ingress only. Egress is left unrestricted deliberately - the WAL-archival/S3 and
peer dial-out surfaces are not fully enumerated, and a wrong egress rule
partitions the cluster instead of merely blocking a scrape.

:9500 stays open, and the file records why. All three probes (startup, readiness,
liveness) target it, probes come from the NODE rather than a pod, and node-to-pod
handling is CNI-specific - a wrong rule there fails liveness at 6x10s and restarts
every pod. The exposure that would have closed is /cluster/status, which is
unauthenticated by design. That is an engine defect and gets fixed in
tidal-server, not worked around at the network layer.

Verified after apply: scraper still collects 353 tidaldb_ series, a quorum-acked
write through the public ingress returns 201, and all three regions report
applied=13322229 lag=0 reachable partitioned=false with zero new restarts.

k8s/cluster/kustomization.yaml

@@ -30,6 +30,7 @@ resources:
   - service-client.yaml
   - statefulset.yaml
   - poddisruptionbudget.yaml
+  - networkpolicy.yaml
   # Public exposure (Traefik + Let's Encrypt). Remove for internal-only.
   - ingress.yaml
 

k8s/cluster/networkpolicy.yaml

@@ -0,0 +1,62 @@
+# Ingress isolation for the tidalDB cluster.
+#
+# WHY: before this existed, ANY pod in the k3s cluster could read :9091 metrics
+# (unauthenticated - corpus size, seqnos, leader identity) and reach the peer gRPC
+# plane. Verified by scraping tidaldb-0:9091 from an unrelated pod in another
+# namespace.
+#
+# INGRESS ONLY, deliberately. Egress is left unrestricted: the WAL-archival / S3
+# paths and the peer dial-out surface are not fully enumerated here, and a wrong
+# egress rule partitions the cluster rather than merely blocking a scrape.
+#
+# :9500 IS INTENTIONALLY LEFT OPEN, and stays that way. It is the only port the
+# kubelet probes - startup /health/startup, readiness /health, AND liveness
+# /health/live all target it - and probes originate from the NODE, not a pod.
+# Node-to-pod handling differs between CNIs, so a wrong rule here fails liveness
+# (threshold 6 x 10s) and RESTARTS every pod.
+#
+# The exposure this would have closed is `/cluster/status`, which is
+# unauthenticated by design and leaks leader identity, membership and seqnos to
+# any in-cluster pod. That is an ENGINE defect, not a network one, and network
+# rules are the wrong layer to fix it: the route is moved behind the bearer in
+# tidal-server instead. Every data and admin route on this port is already
+# bearer-gated.
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: tidaldb
+  namespace: tidaldb-cluster
+  labels:
+    app.kubernetes.io/name: tidaldb
+    app.kubernetes.io/part-of: tidaldb
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/name: tidaldb
+      app.kubernetes.io/component: cluster-node
+  policyTypes:
+    - Ingress
+  ingress:
+    # 1. Same namespace, every port: the sibling replication plane (gRPC 9601-9603
+    #    plus the inter-node HTTP forwards on 9500) and the soak monitor.
+    - from:
+        - podSelector: {}
+    # 2. The client + health port from anywhere. See the header for why this is
+    #    not narrowed to kube-system: kubelet probes come from the node.
+    #    To tighten later, add an ipBlock for the node InternalIPs
+    #    (208.122.204.172/173/174) alongside a kube-system namespaceSelector, then
+    #    confirm readiness holds on all three pods before trusting it.
+    - ports:
+        - protocol: TCP
+          port: 9500
+    # 3. Metrics: the scraper only. tidalDB's :9091 has no authentication, so this
+    #    is the rule that actually closes the leak. Pods carry
+    #    prometheus.io/scrape, so discovery is annotation-based from vmagent in
+    #    ns observability - there is no ServiceMonitor.
+    - from:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: observability
+      ports:
+        - protocol: TCP
+          port: 9091