release: build through the cluster, not through its public address
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed

Four consecutive Kaniko pushes failed with
`dial tcp 208.122.204.172:443: connect: connection refused` while zot and
Traefik were both healthy and my laptop reached the same URL fine.

Measured from a pod in the same namespace, with the same labels the build Job
carries: 14 of 24 requests to https://registry.threesix.ai/v2/ succeeded, and
8 of 8 succeeded against Traefik's ClusterIP with the same SNI. So the flaky
leg is pod -> the cluster's own public address, and the build was taking it
for both the clone and the push.

The Job now carries hostAliases pinning git.threesix.ai and
registry.threesix.ai to Traefik's ClusterIP, read from the Service at release
time. The image is still named registry.threesix.ai/hush/api:SHA — the route
changes, the reference does not, and the kubelet pull path is untouched.

No retry loop: a retry would have made a 40% failure rate invisible instead of
fixed.

docs/DEPLOY.md no longer duplicates the Job YAML. That duplicate is why the
push guard drifted onto the wrong remote, and a second copy of a spec nobody
runs is worth less than a pointer to the one that runs.
This commit is contained in:
jx12n 2026-09-05 14:28:42 -06:00
parent bc33747568
commit 138ab7306e
2 changed files with 34 additions and 27 deletions

View File

@ -129,34 +129,21 @@ it 404s at the edge while working fine in `make dev` — is
directory to publish a path would roll the workload back onto the unpullable directory to publish a path would roll the workload back onto the unpullable
bootstrap image. bootstrap image.
The build Job, which is what Woodpecker's Kaniko step does by hand: The build Job itself is not reproduced here. `scripts/release.sh` emits it, and
a copy in this file drifts from the copy that runs — which is how the script's
own push guard ended up checking the wrong remote. Read it there; `make
release` is the same path, and it needs no CI credential because the Gitea repo
is public.
```yaml One thing about that Job is worth stating outside the script, because it is a
apiVersion: batch/v1 property of the cluster rather than of hush: `git.threesix.ai` and
kind: Job `registry.threesix.ai` resolve to the cluster's public address, and reaching
metadata: { name: hush-build, namespace: projects } that address from inside a pod hairpins unreliably. Measured 2026-09-05: 14 of
spec: 24 requests from a pod succeeded, four consecutive kaniko pushes were refused
backoffLimit: 1 with `connection refused`, and Traefik's ClusterIP answered 8 of 8. The Job
ttlSecondsAfterFinished: 3600 therefore carries `hostAliases` pinning both names to that ClusterIP. Anything
template: else that builds in-cluster — Woodpecker's Kaniko step included — is exposed to
spec: the same hairpin and will fail the same way, intermittently.
restartPolicy: Never
containers:
- name: kaniko
image: gcr.io/kaniko-project/executor:v1.23.2
args:
- --context=git://git.threesix.ai/jordan/hush.git#refs/heads/main
- --dockerfile=Dockerfile
- --destination=registry.threesix.ai/hush/api:SHA
- --skip-tls-verify
- --skip-tls-verify-pull
- --single-snapshot
resources:
requests: { cpu: 500m, memory: 1Gi }
limits: { cpu: "2", memory: 3Gi }
```
The git context needs no credential because the Gitea repo is public.
## Woodpecker: activated ## Woodpecker: activated

View File

@ -53,6 +53,21 @@ IMAGE="registry.threesix.ai/hush/api:$SHA"
JOB="hush-build-$SHA" JOB="hush-build-$SHA"
echo "releasing $SHA" echo "releasing $SHA"
# Kaniko clones git.threesix.ai and pushes registry.threesix.ai. Both names
# resolve to the cluster's PUBLIC address, and reaching that from inside a pod
# takes a hairpin path that drops connections: measured 2026-09-05, 14 of 24
# requests from a pod succeeded, four consecutive kaniko pushes were refused,
# and Traefik's own ClusterIP answered 8 of 8. One Traefik serves both names, so
# the build resolves them to that ClusterIP and never leaves the cluster. The
# pushed image is still named registry.threesix.ai/hush/api:SHA, which is what
# the kubelet pulls — this changes the route, not the reference.
TRAEFIK_IP="$(kubectl -n kube-system get svc traefik -o jsonpath='{.spec.clusterIP}')"
if [ -z "$TRAEFIK_IP" ]; then
echo "refusing: kube-system/traefik has no ClusterIP, so the build has no in-cluster route" >&2
exit 1
fi
echo " build resolves git+registry to traefik at $TRAEFIK_IP"
# A previous attempt at the same SHA leaves a completed Job that cannot be # A previous attempt at the same SHA leaves a completed Job that cannot be
# re-created; replacing it is the idempotent thing to do. # re-created; replacing it is the idempotent thing to do.
kubectl -n "$NS" delete job "$JOB" --ignore-not-found >/dev/null kubectl -n "$NS" delete job "$JOB" --ignore-not-found >/dev/null
@ -71,6 +86,11 @@ spec:
labels: { app: hush, component: build } labels: { app: hush, component: build }
spec: spec:
restartPolicy: Never restartPolicy: Never
# See the note above: the public address is not reliably reachable from a
# pod, and the ClusterIP is.
hostAliases:
- ip: $TRAEFIK_IP
hostnames: [git.threesix.ai, registry.threesix.ai]
containers: containers:
- name: kaniko - name: kaniko
image: gcr.io/kaniko-project/executor:v1.23.2 image: gcr.io/kaniko-project/executor:v1.23.2