release: build through the cluster, not through its public address
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
Four consecutive Kaniko pushes failed with `dial tcp 208.122.204.172:443: connect: connection refused` while zot and Traefik were both healthy and my laptop reached the same URL fine. Measured from a pod in the same namespace, with the same labels the build Job carries: 14 of 24 requests to https://registry.threesix.ai/v2/ succeeded, and 8 of 8 succeeded against Traefik's ClusterIP with the same SNI. So the flaky leg is pod -> the cluster's own public address, and the build was taking it for both the clone and the push. The Job now carries hostAliases pinning git.threesix.ai and registry.threesix.ai to Traefik's ClusterIP, read from the Service at release time. The image is still named registry.threesix.ai/hush/api:SHA — the route changes, the reference does not, and the kubelet pull path is untouched. No retry loop: a retry would have made a 40% failure rate invisible instead of fixed. docs/DEPLOY.md no longer duplicates the Job YAML. That duplicate is why the push guard drifted onto the wrong remote, and a second copy of a spec nobody runs is worth less than a pointer to the one that runs.
This commit is contained in:
parent
bc33747568
commit
138ab7306e
@ -129,34 +129,21 @@ it 404s at the edge while working fine in `make dev` — is
|
||||
directory to publish a path would roll the workload back onto the unpullable
|
||||
bootstrap image.
|
||||
|
||||
The build Job, which is what Woodpecker's Kaniko step does by hand:
|
||||
The build Job itself is not reproduced here. `scripts/release.sh` emits it, and
|
||||
a copy in this file drifts from the copy that runs — which is how the script's
|
||||
own push guard ended up checking the wrong remote. Read it there; `make
|
||||
release` is the same path, and it needs no CI credential because the Gitea repo
|
||||
is public.
|
||||
|
||||
```yaml
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata: { name: hush-build, namespace: projects }
|
||||
spec:
|
||||
backoffLimit: 1
|
||||
ttlSecondsAfterFinished: 3600
|
||||
template:
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: kaniko
|
||||
image: gcr.io/kaniko-project/executor:v1.23.2
|
||||
args:
|
||||
- --context=git://git.threesix.ai/jordan/hush.git#refs/heads/main
|
||||
- --dockerfile=Dockerfile
|
||||
- --destination=registry.threesix.ai/hush/api:SHA
|
||||
- --skip-tls-verify
|
||||
- --skip-tls-verify-pull
|
||||
- --single-snapshot
|
||||
resources:
|
||||
requests: { cpu: 500m, memory: 1Gi }
|
||||
limits: { cpu: "2", memory: 3Gi }
|
||||
```
|
||||
|
||||
The git context needs no credential because the Gitea repo is public.
|
||||
One thing about that Job is worth stating outside the script, because it is a
|
||||
property of the cluster rather than of hush: `git.threesix.ai` and
|
||||
`registry.threesix.ai` resolve to the cluster's public address, and reaching
|
||||
that address from inside a pod hairpins unreliably. Measured 2026-09-05: 14 of
|
||||
24 requests from a pod succeeded, four consecutive kaniko pushes were refused
|
||||
with `connection refused`, and Traefik's ClusterIP answered 8 of 8. The Job
|
||||
therefore carries `hostAliases` pinning both names to that ClusterIP. Anything
|
||||
else that builds in-cluster — Woodpecker's Kaniko step included — is exposed to
|
||||
the same hairpin and will fail the same way, intermittently.
|
||||
|
||||
## Woodpecker: activated
|
||||
|
||||
|
||||
@ -53,6 +53,21 @@ IMAGE="registry.threesix.ai/hush/api:$SHA"
|
||||
JOB="hush-build-$SHA"
|
||||
echo "releasing $SHA"
|
||||
|
||||
# Kaniko clones git.threesix.ai and pushes registry.threesix.ai. Both names
|
||||
# resolve to the cluster's PUBLIC address, and reaching that from inside a pod
|
||||
# takes a hairpin path that drops connections: measured 2026-09-05, 14 of 24
|
||||
# requests from a pod succeeded, four consecutive kaniko pushes were refused,
|
||||
# and Traefik's own ClusterIP answered 8 of 8. One Traefik serves both names, so
|
||||
# the build resolves them to that ClusterIP and never leaves the cluster. The
|
||||
# pushed image is still named registry.threesix.ai/hush/api:SHA, which is what
|
||||
# the kubelet pulls — this changes the route, not the reference.
|
||||
TRAEFIK_IP="$(kubectl -n kube-system get svc traefik -o jsonpath='{.spec.clusterIP}')"
|
||||
if [ -z "$TRAEFIK_IP" ]; then
|
||||
echo "refusing: kube-system/traefik has no ClusterIP, so the build has no in-cluster route" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo " build resolves git+registry to traefik at $TRAEFIK_IP"
|
||||
|
||||
# A previous attempt at the same SHA leaves a completed Job that cannot be
|
||||
# re-created; replacing it is the idempotent thing to do.
|
||||
kubectl -n "$NS" delete job "$JOB" --ignore-not-found >/dev/null
|
||||
@ -71,6 +86,11 @@ spec:
|
||||
labels: { app: hush, component: build }
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
# See the note above: the public address is not reliably reachable from a
|
||||
# pod, and the ClusterIP is.
|
||||
hostAliases:
|
||||
- ip: $TRAEFIK_IP
|
||||
hostnames: [git.threesix.ai, registry.threesix.ai]
|
||||
containers:
|
||||
- name: kaniko
|
||||
image: gcr.io/kaniko-project/executor:v1.23.2
|
||||
|
||||
Loading…
Reference in New Issue
Block a user