diff --git a/docs/DEPLOY.md b/docs/DEPLOY.md index 60c533d..de5892f 100644 --- a/docs/DEPLOY.md +++ b/docs/DEPLOY.md @@ -129,34 +129,21 @@ it 404s at the edge while working fine in `make dev` — is directory to publish a path would roll the workload back onto the unpullable bootstrap image. -The build Job, which is what Woodpecker's Kaniko step does by hand: +The build Job itself is not reproduced here. `scripts/release.sh` emits it, and +a copy in this file drifts from the copy that runs — which is how the script's +own push guard ended up checking the wrong remote. Read it there; `make +release` is the same path, and it needs no CI credential because the Gitea repo +is public. -```yaml -apiVersion: batch/v1 -kind: Job -metadata: { name: hush-build, namespace: projects } -spec: - backoffLimit: 1 - ttlSecondsAfterFinished: 3600 - template: - spec: - restartPolicy: Never - containers: - - name: kaniko - image: gcr.io/kaniko-project/executor:v1.23.2 - args: - - --context=git://git.threesix.ai/jordan/hush.git#refs/heads/main - - --dockerfile=Dockerfile - - --destination=registry.threesix.ai/hush/api:SHA - - --skip-tls-verify - - --skip-tls-verify-pull - - --single-snapshot - resources: - requests: { cpu: 500m, memory: 1Gi } - limits: { cpu: "2", memory: 3Gi } -``` - -The git context needs no credential because the Gitea repo is public. +One thing about that Job is worth stating outside the script, because it is a +property of the cluster rather than of hush: `git.threesix.ai` and +`registry.threesix.ai` resolve to the cluster's public address, and reaching +that address from inside a pod hairpins unreliably. Measured 2026-09-05: 14 of +24 requests from a pod succeeded, four consecutive kaniko pushes were refused +with `connection refused`, and Traefik's ClusterIP answered 8 of 8. The Job +therefore carries `hostAliases` pinning both names to that ClusterIP. Anything +else that builds in-cluster — Woodpecker's Kaniko step included — is exposed to +the same hairpin and will fail the same way, intermittently. ## Woodpecker: activated diff --git a/scripts/release.sh b/scripts/release.sh index 6a6cc0b..b9106a7 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -53,6 +53,21 @@ IMAGE="registry.threesix.ai/hush/api:$SHA" JOB="hush-build-$SHA" echo "releasing $SHA" +# Kaniko clones git.threesix.ai and pushes registry.threesix.ai. Both names +# resolve to the cluster's PUBLIC address, and reaching that from inside a pod +# takes a hairpin path that drops connections: measured 2026-09-05, 14 of 24 +# requests from a pod succeeded, four consecutive kaniko pushes were refused, +# and Traefik's own ClusterIP answered 8 of 8. One Traefik serves both names, so +# the build resolves them to that ClusterIP and never leaves the cluster. The +# pushed image is still named registry.threesix.ai/hush/api:SHA, which is what +# the kubelet pulls — this changes the route, not the reference. +TRAEFIK_IP="$(kubectl -n kube-system get svc traefik -o jsonpath='{.spec.clusterIP}')" +if [ -z "$TRAEFIK_IP" ]; then + echo "refusing: kube-system/traefik has no ClusterIP, so the build has no in-cluster route" >&2 + exit 1 +fi +echo " build resolves git+registry to traefik at $TRAEFIK_IP" + # A previous attempt at the same SHA leaves a completed Job that cannot be # re-created; replacing it is the idempotent thing to do. kubectl -n "$NS" delete job "$JOB" --ignore-not-found >/dev/null @@ -71,6 +86,11 @@ spec: labels: { app: hush, component: build } spec: restartPolicy: Never + # See the note above: the public address is not reliably reachable from a + # pod, and the ClusterIP is. + hostAliases: + - ip: $TRAEFIK_IP + hostnames: [git.threesix.ai, registry.threesix.ai] containers: - name: kaniko image: gcr.io/kaniko-project/executor:v1.23.2