37cafb2b75
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
The build Job's pods carried `app: hush`, which is the podSelector on hush's NetworkPolicy — a default-deny policy that permits egress to DNS and Redis and nothing else. That is a true statement about the server and a false one about a build, which needs Gitea, the registry, Docker Hub and gcr.io. kube-router REJECTS rather than drops, so the symptom was `connection refused` from whichever host the build reached for next: four pushes to registry.threesix.ai, then a blob fetch from production.cloudfront.docker.com. Intermittently, because policy sync leaves windows where the rules are briefly absent — which is why an earlier build pushed successfully and read as normal. Measured 2026-09-05 from pods in this namespace: `app=hush` reached Docker Hub 10 of 12 times, `app=hush-build` 12 of 12, and in-cluster destinations 16 of 16 under both. The Job is now `app: hush-build`, so hush's policy no longer selects it, and the Service selector no longer matches it either.
162 lines
7.3 KiB
Bash
Executable File
162 lines
7.3 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Build the current commit in-cluster and roll it out. No CI credential needed.
|
|
#
|
|
# Woodpecker IS activated for this repo, so a push to main builds and deploys.
|
|
# This is the path for when you do not want to wait for CI, when CI is down, or
|
|
# when you are rolling back — and it is how the first deploy happened, before
|
|
# activation. It does exactly what the pipeline's build and deploy steps do: a
|
|
# Kaniko Job for an amd64 image from the pushed git ref, then
|
|
# `kubectl set image`, then a real end-to-end check.
|
|
#
|
|
# Credentials: none. The Gitea repo is public so the Kaniko git context needs no
|
|
# token, and the rollout uses your kubeconfig.
|
|
#
|
|
# ./scripts/release.sh
|
|
set -euo pipefail
|
|
|
|
export KUBECONFIG="${KUBECONFIG:-$HOME/.kube/orchard9-k3sf.yaml}"
|
|
NS="${NS:-projects}"
|
|
HOST="${HOST:-hush.threesix.ai}"
|
|
# The Gitea repo Kaniko clones, and the remote that points at it. Both are
|
|
# named once: the guard below has to check the ref that gets BUILT, and a
|
|
# guard that checks a different remote is worse than no guard.
|
|
GIT_CONTEXT="${GIT_CONTEXT:-git://git.threesix.ai/jordan/hush.git#refs/heads/main}"
|
|
GIT_REMOTE="${GIT_REMOTE:-origin}"
|
|
GIT_BRANCH="${GIT_BRANCH:-main}"
|
|
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
|
cd "$ROOT"
|
|
|
|
# Kaniko builds from the GIT CONTEXT, not from this working tree. So a dirty or
|
|
# unpushed tree would silently build something other than what you are looking
|
|
# at — the single most confusing failure this script can have. Refuse instead.
|
|
if [ -n "$(git status --porcelain)" ]; then
|
|
echo "refusing: working tree is dirty. Kaniko builds from the pushed git ref," >&2
|
|
echo "so uncommitted changes would NOT be in the image." >&2
|
|
git status --short >&2
|
|
exit 1
|
|
fi
|
|
# `@{upstream}` is NOT the right comparison: this checkout tracks a mirror, so
|
|
# HEAD can be pushed there while Gitea — the repo Kaniko clones — is behind,
|
|
# and the build would silently produce the previous commit. Compare against the
|
|
# branch that actually gets built.
|
|
git fetch --quiet "$GIT_REMOTE" "$GIT_BRANCH"
|
|
if [ "$(git rev-parse HEAD)" != "$(git rev-parse FETCH_HEAD)" ]; then
|
|
echo "refusing: HEAD is not what $GIT_REMOTE/$GIT_BRANCH points at, and Kaniko clones from there." >&2
|
|
echo " HEAD $(git rev-parse --short=8 HEAD) $(git log -1 --format=%s HEAD)" >&2
|
|
echo " $GIT_REMOTE/$GIT_BRANCH $(git rev-parse --short=8 FETCH_HEAD) $(git log -1 --format=%s FETCH_HEAD)" >&2
|
|
echo "Push to $GIT_REMOTE first: git push $GIT_REMOTE $GIT_BRANCH" >&2
|
|
exit 1
|
|
fi
|
|
|
|
SHA="$(git rev-parse --short=8 HEAD)"
|
|
IMAGE="registry.threesix.ai/hush/api:$SHA"
|
|
JOB="hush-build-$SHA"
|
|
echo "releasing $SHA"
|
|
|
|
# Kaniko clones git.threesix.ai and pushes registry.threesix.ai. Both names
|
|
# resolve to the cluster's PUBLIC address, and reaching that from inside a pod
|
|
# takes a hairpin path that drops connections: measured 2026-09-05, 14 of 24
|
|
# requests from a pod succeeded, four consecutive kaniko pushes were refused,
|
|
# and Traefik's own ClusterIP answered 8 of 8. One Traefik serves both names, so
|
|
# the build resolves them to that ClusterIP and never leaves the cluster. The
|
|
# pushed image is still named registry.threesix.ai/hush/api:SHA, which is what
|
|
# the kubelet pulls — this changes the route, not the reference.
|
|
TRAEFIK_IP="$(kubectl -n kube-system get svc traefik -o jsonpath='{.spec.clusterIP}')"
|
|
if [ -z "$TRAEFIK_IP" ]; then
|
|
echo "refusing: kube-system/traefik has no ClusterIP, so the build has no in-cluster route" >&2
|
|
exit 1
|
|
fi
|
|
echo " build resolves git+registry to traefik at $TRAEFIK_IP"
|
|
|
|
# A previous attempt at the same SHA leaves a completed Job that cannot be
|
|
# re-created; replacing it is the idempotent thing to do.
|
|
kubectl -n "$NS" delete job "$JOB" --ignore-not-found >/dev/null
|
|
|
|
# NOTE the label: `app: hush-build`, NOT `app: hush`. hush's NetworkPolicy
|
|
# selects `app: hush` and permits egress to DNS and Redis only — a true
|
|
# statement about the SERVER, and a build pod that inherits it cannot reach
|
|
# Docker Hub, gcr.io, Gitea or the registry. kube-router REJECTS, so that
|
|
# arrives as `connection refused` from whichever host the build happened to
|
|
# need next, intermittently, because policy sync leaves windows where the rules
|
|
# are briefly absent. Measured 2026-09-05: five consecutive kaniko builds
|
|
# failed on three different external hosts under `app: hush`.
|
|
kubectl -n "$NS" apply -f - >/dev/null <<EOF
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: $JOB
|
|
labels: { app: hush-build, component: build }
|
|
spec:
|
|
backoffLimit: 1
|
|
ttlSecondsAfterFinished: 3600
|
|
template:
|
|
metadata:
|
|
labels: { app: hush-build, component: build }
|
|
spec:
|
|
restartPolicy: Never
|
|
# See the note above: the public address is not reliably reachable from a
|
|
# pod, and the ClusterIP is.
|
|
hostAliases:
|
|
- ip: $TRAEFIK_IP
|
|
hostnames: [git.threesix.ai, registry.threesix.ai]
|
|
containers:
|
|
- name: kaniko
|
|
image: gcr.io/kaniko-project/executor:v1.23.2
|
|
args:
|
|
# The Gitea repo is public, so the git context needs no credential.
|
|
- --context=$GIT_CONTEXT
|
|
- --dockerfile=Dockerfile
|
|
- --destination=$IMAGE
|
|
# The internal Zot registry serves a self-signed cert.
|
|
- --skip-tls-verify
|
|
- --skip-tls-verify-pull
|
|
- --single-snapshot
|
|
resources:
|
|
requests: { cpu: 500m, memory: 1Gi }
|
|
limits: { cpu: "2", memory: 3Gi }
|
|
EOF
|
|
|
|
# Poll rather than `kubectl wait --for=condition=complete`. That is one long
|
|
# WATCH against a cluster on the other side of a WAN link: measured 2026-09-05,
|
|
# the Job reached Complete in 156s and the watch still sat there until its
|
|
# 900s timeout, then reported "build FAILED" for an image that had already been
|
|
# pushed. Each poll below is a fresh short request, so a dropped connection
|
|
# costs one poll — and a deadline here means "still building", never "failed".
|
|
echo " building (amd64, in-cluster)…"
|
|
DEADLINE=$((SECONDS + 900))
|
|
while :; do
|
|
# A missing field prints nothing, so default to 0 and keep the comparison
|
|
# numeric. `|| true` covers a poll that loses the connection outright.
|
|
SUCCEEDED=$(kubectl -n "$NS" get "job/$JOB" -o jsonpath='{.status.succeeded}' 2>/dev/null || true)
|
|
FAILED=$(kubectl -n "$NS" get "job/$JOB" -o jsonpath='{.status.failed}' 2>/dev/null || true)
|
|
if [ "${SUCCEEDED:-0}" -ge 1 ]; then
|
|
break
|
|
fi
|
|
if [ "${FAILED:-0}" -ge 1 ]; then
|
|
echo "build FAILED — last lines:" >&2
|
|
kubectl -n "$NS" logs "job/$JOB" --tail=30 >&2
|
|
exit 1
|
|
fi
|
|
if [ "$SECONDS" -ge "$DEADLINE" ]; then
|
|
echo "the build has not finished after 900s. It may still be running:" >&2
|
|
echo " kubectl -n $NS get job/$JOB" >&2
|
|
echo " kubectl -n $NS logs job/$JOB --tail=30" >&2
|
|
exit 1
|
|
fi
|
|
sleep 5
|
|
done
|
|
echo " built $IMAGE"
|
|
|
|
kubectl -n "$NS" set image deployment/hush "hushd=$IMAGE" >/dev/null
|
|
kubectl -n "$NS" rollout status deployment/hush --timeout=180s | sed 's/^/ /'
|
|
|
|
# Prove the rolled pod is the image we just built. `set image` matching nothing
|
|
# is silent, and the rollout would "succeed" on the old pod.
|
|
LIVE="$(kubectl -n "$NS" get deployment hush -o jsonpath='{.spec.template.spec.containers[0].image}')"
|
|
[ "$LIVE" = "$IMAGE" ] || { echo "live image is $LIVE, expected $IMAGE" >&2; exit 1; }
|
|
echo " live image: $LIVE"
|
|
|
|
echo
|
|
echo "verifying end to end against https://$HOST"
|
|
BASE="https://$HOST" "$ROOT/scripts/smoke.sh"
|