release: stop building under the server's egress policy
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful

The build Job's pods carried `app: hush`, which is the podSelector on hush's
NetworkPolicy — a default-deny policy that permits egress to DNS and Redis and
nothing else. That is a true statement about the server and a false one about a
build, which needs Gitea, the registry, Docker Hub and gcr.io.

kube-router REJECTS rather than drops, so the symptom was `connection refused`
from whichever host the build reached for next: four pushes to
registry.threesix.ai, then a blob fetch from production.cloudfront.docker.com.
Intermittently, because policy sync leaves windows where the rules are briefly
absent — which is why an earlier build pushed successfully and read as normal.

Measured 2026-09-05 from pods in this namespace: `app=hush` reached Docker Hub
10 of 12 times, `app=hush-build` 12 of 12, and in-cluster destinations 16 of 16
under both. The Job is now `app: hush-build`, so hush's policy no longer
selects it, and the Service selector no longer matches it either.
This commit is contained in:
jx12n 2026-09-05 14:30:54 -06:00
parent 138ab7306e
commit 37cafb2b75

View File

@ -72,18 +72,26 @@ echo " build resolves git+registry to traefik at $TRAEFIK_IP"
# re-created; replacing it is the idempotent thing to do. # re-created; replacing it is the idempotent thing to do.
kubectl -n "$NS" delete job "$JOB" --ignore-not-found >/dev/null kubectl -n "$NS" delete job "$JOB" --ignore-not-found >/dev/null
# NOTE the label: `app: hush-build`, NOT `app: hush`. hush's NetworkPolicy
# selects `app: hush` and permits egress to DNS and Redis only — a true
# statement about the SERVER, and a build pod that inherits it cannot reach
# Docker Hub, gcr.io, Gitea or the registry. kube-router REJECTS, so that
# arrives as `connection refused` from whichever host the build happened to
# need next, intermittently, because policy sync leaves windows where the rules
# are briefly absent. Measured 2026-09-05: five consecutive kaniko builds
# failed on three different external hosts under `app: hush`.
kubectl -n "$NS" apply -f - >/dev/null <<EOF kubectl -n "$NS" apply -f - >/dev/null <<EOF
apiVersion: batch/v1 apiVersion: batch/v1
kind: Job kind: Job
metadata: metadata:
name: $JOB name: $JOB
labels: { app: hush, component: build } labels: { app: hush-build, component: build }
spec: spec:
backoffLimit: 1 backoffLimit: 1
ttlSecondsAfterFinished: 3600 ttlSecondsAfterFinished: 3600
template: template:
metadata: metadata:
labels: { app: hush, component: build } labels: { app: hush-build, component: build }
spec: spec:
restartPolicy: Never restartPolicy: Never
# See the note above: the public address is not reliably reachable from a # See the note above: the public address is not reliably reachable from a