release: poll the build Job instead of watching it
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
Measured on the previous commit's release: the Kaniko Job reached Complete in
156s and pushed registry.threesix.ai/hush/api:d7cd57f3, and
`kubectl wait --for=condition=complete` sat on its watch until the 900s
timeout, then reported "build FAILED" and exited before the rollout. The
image existed; the deploy did not happen; the message said the opposite of
what had happened.
One long watch against a cluster across a WAN link is the wrong instrument. A
fresh short GET every 5s costs one poll when a connection drops, distinguishes
`.status.failed` from "not finished yet", and reports the deadline as "may
still be running" with the two commands to check — never as a failure.
Both branches exercised against the cluster with the loop as committed: a Job
that exits 1 is reported as FAILED with its last lines, and the already
complete d7cd57f3 Job breaks the loop immediately.
This commit is contained in:
parent
d7cd57f330
commit
bc33747568
@ -88,12 +88,35 @@ spec:
|
|||||||
limits: { cpu: "2", memory: 3Gi }
|
limits: { cpu: "2", memory: 3Gi }
|
||||||
EOF
|
EOF
|
||||||
|
|
||||||
|
# Poll rather than `kubectl wait --for=condition=complete`. That is one long
|
||||||
|
# WATCH against a cluster on the other side of a WAN link: measured 2026-09-05,
|
||||||
|
# the Job reached Complete in 156s and the watch still sat there until its
|
||||||
|
# 900s timeout, then reported "build FAILED" for an image that had already been
|
||||||
|
# pushed. Each poll below is a fresh short request, so a dropped connection
|
||||||
|
# costs one poll — and a deadline here means "still building", never "failed".
|
||||||
echo " building (amd64, in-cluster)…"
|
echo " building (amd64, in-cluster)…"
|
||||||
if ! kubectl -n "$NS" wait --for=condition=complete "job/$JOB" --timeout=900s >/dev/null 2>&1; then
|
DEADLINE=$((SECONDS + 900))
|
||||||
|
while :; do
|
||||||
|
# A missing field prints nothing, so default to 0 and keep the comparison
|
||||||
|
# numeric. `|| true` covers a poll that loses the connection outright.
|
||||||
|
SUCCEEDED=$(kubectl -n "$NS" get "job/$JOB" -o jsonpath='{.status.succeeded}' 2>/dev/null || true)
|
||||||
|
FAILED=$(kubectl -n "$NS" get "job/$JOB" -o jsonpath='{.status.failed}' 2>/dev/null || true)
|
||||||
|
if [ "${SUCCEEDED:-0}" -ge 1 ]; then
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
if [ "${FAILED:-0}" -ge 1 ]; then
|
||||||
echo "build FAILED — last lines:" >&2
|
echo "build FAILED — last lines:" >&2
|
||||||
kubectl -n "$NS" logs "job/$JOB" --tail=30 >&2
|
kubectl -n "$NS" logs "job/$JOB" --tail=30 >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
if [ "$SECONDS" -ge "$DEADLINE" ]; then
|
||||||
|
echo "the build has not finished after 900s. It may still be running:" >&2
|
||||||
|
echo " kubectl -n $NS get job/$JOB" >&2
|
||||||
|
echo " kubectl -n $NS logs job/$JOB --tail=30" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
echo " built $IMAGE"
|
echo " built $IMAGE"
|
||||||
|
|
||||||
kubectl -n "$NS" set image deployment/hush "hushd=$IMAGE" >/dev/null
|
kubectl -n "$NS" set image deployment/hush "hushd=$IMAGE" >/dev/null
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user