From 138ab7306e9b6b50669ef27a1f5d28471b87c1cb Mon Sep 17 00:00:00 2001 From: jx12n Date: Sat, 5 Sep 2026 14:28:42 -0600 Subject: [PATCH] release: build through the cluster, not through its public address MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four consecutive Kaniko pushes failed with `dial tcp 208.122.204.172:443: connect: connection refused` while zot and Traefik were both healthy and my laptop reached the same URL fine. Measured from a pod in the same namespace, with the same labels the build Job carries: 14 of 24 requests to https://registry.threesix.ai/v2/ succeeded, and 8 of 8 succeeded against Traefik's ClusterIP with the same SNI. So the flaky leg is pod -> the cluster's own public address, and the build was taking it for both the clone and the push. The Job now carries hostAliases pinning git.threesix.ai and registry.threesix.ai to Traefik's ClusterIP, read from the Service at release time. The image is still named registry.threesix.ai/hush/api:SHA — the route changes, the reference does not, and the kubelet pull path is untouched. No retry loop: a retry would have made a 40% failure rate invisible instead of fixed. docs/DEPLOY.md no longer duplicates the Job YAML. That duplicate is why the push guard drifted onto the wrong remote, and a second copy of a spec nobody runs is worth less than a pointer to the one that runs. --- docs/DEPLOY.md | 41 ++++++++++++++--------------------------- scripts/release.sh | 20 ++++++++++++++++++++ 2 files changed, 34 insertions(+), 27 deletions(-) diff --git a/docs/DEPLOY.md b/docs/DEPLOY.md index 60c533d..de5892f 100644 --- a/docs/DEPLOY.md +++ b/docs/DEPLOY.md @@ -129,34 +129,21 @@ it 404s at the edge while working fine in `make dev` — is directory to publish a path would roll the workload back onto the unpullable bootstrap image. -The build Job, which is what Woodpecker's Kaniko step does by hand: +The build Job itself is not reproduced here. `scripts/release.sh` emits it, and +a copy in this file drifts from the copy that runs — which is how the script's +own push guard ended up checking the wrong remote. Read it there; `make +release` is the same path, and it needs no CI credential because the Gitea repo +is public. -```yaml -apiVersion: batch/v1 -kind: Job -metadata: { name: hush-build, namespace: projects } -spec: - backoffLimit: 1 - ttlSecondsAfterFinished: 3600 - template: - spec: - restartPolicy: Never - containers: - - name: kaniko - image: gcr.io/kaniko-project/executor:v1.23.2 - args: - - --context=git://git.threesix.ai/jordan/hush.git#refs/heads/main - - --dockerfile=Dockerfile - - --destination=registry.threesix.ai/hush/api:SHA - - --skip-tls-verify - - --skip-tls-verify-pull - - --single-snapshot - resources: - requests: { cpu: 500m, memory: 1Gi } - limits: { cpu: "2", memory: 3Gi } -``` - -The git context needs no credential because the Gitea repo is public. +One thing about that Job is worth stating outside the script, because it is a +property of the cluster rather than of hush: `git.threesix.ai` and +`registry.threesix.ai` resolve to the cluster's public address, and reaching +that address from inside a pod hairpins unreliably. Measured 2026-09-05: 14 of +24 requests from a pod succeeded, four consecutive kaniko pushes were refused +with `connection refused`, and Traefik's ClusterIP answered 8 of 8. The Job +therefore carries `hostAliases` pinning both names to that ClusterIP. Anything +else that builds in-cluster — Woodpecker's Kaniko step included — is exposed to +the same hairpin and will fail the same way, intermittently. ## Woodpecker: activated diff --git a/scripts/release.sh b/scripts/release.sh index 6a6cc0b..b9106a7 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -53,6 +53,21 @@ IMAGE="registry.threesix.ai/hush/api:$SHA" JOB="hush-build-$SHA" echo "releasing $SHA" +# Kaniko clones git.threesix.ai and pushes registry.threesix.ai. Both names +# resolve to the cluster's PUBLIC address, and reaching that from inside a pod +# takes a hairpin path that drops connections: measured 2026-09-05, 14 of 24 +# requests from a pod succeeded, four consecutive kaniko pushes were refused, +# and Traefik's own ClusterIP answered 8 of 8. One Traefik serves both names, so +# the build resolves them to that ClusterIP and never leaves the cluster. The +# pushed image is still named registry.threesix.ai/hush/api:SHA, which is what +# the kubelet pulls — this changes the route, not the reference. +TRAEFIK_IP="$(kubectl -n kube-system get svc traefik -o jsonpath='{.spec.clusterIP}')" +if [ -z "$TRAEFIK_IP" ]; then + echo "refusing: kube-system/traefik has no ClusterIP, so the build has no in-cluster route" >&2 + exit 1 +fi +echo " build resolves git+registry to traefik at $TRAEFIK_IP" + # A previous attempt at the same SHA leaves a completed Job that cannot be # re-created; replacing it is the idempotent thing to do. kubectl -n "$NS" delete job "$JOB" --ignore-not-found >/dev/null @@ -71,6 +86,11 @@ spec: labels: { app: hush, component: build } spec: restartPolicy: Never + # See the note above: the public address is not reliably reachable from a + # pod, and the ClusterIP is. + hostAliases: + - ip: $TRAEFIK_IP + hostnames: [git.threesix.ai, registry.threesix.ai] containers: - name: kaniko image: gcr.io/kaniko-project/executor:v1.23.2