Woodpecker is not activated for this repo — the WOODPECKER_API_TOKEN in
rdev-credentials returns 401 on /api/user, so it is the token and not the
request shape, and minting a new one needs a browser login I cannot do. That
left `git push` silently not deploying, which is a trap for whoever pushes next.
So `make release` does what the pipeline's build and deploy steps do: a Kaniko
Job for an amd64 image from the pushed git ref, `kubectl set image`, rollout,
then the production smoke. When Woodpecker is activated this becomes redundant,
and stays useful as the manual path for a hotfix or a rollback when CI is down.
Two refusals in it are the interesting part, both for failures that are
otherwise silent:
* A dirty or unpushed tree is refused. Kaniko builds from the GIT CONTEXT, not
the working tree, so uncommitted work would produce an image that does not
contain it while every log line says success.
* After the rollout it asserts the live image equals the one just built.
`kubectl set image` matching no container is silent, and the rollout then
"succeeds" against the old pod.
Written after the service was live, so every command and every number here was
run against the real deployment rather than assumed:
* DEPLOY.md records the things a rebuild needs and git does not hold — the
Redis ACL user (and why +getdel is the one to notice), the GCP Secret
Manager entry, the DNS record, and the three coordinated edits vmalert
needs because it has no ConfigMap auto-discovery.
* It also records two blockers rather than hiding them: Woodpecker is NOT
activated (the token in rdev-credentials returns 401), so pushes do not
deploy yet and the Kaniko Job is the interim path; and the host is
hush.threesix.ai rather than hush.orchard9.ai because orchard9.ai is on
GoDaddy and no GoDaddy credential exists anywhere I can reach.
* OPERATIONS.md is one section per alert, plus the failure modes that are not
alerts — chiefly that "gone" cannot distinguish already-revealed from
expired from LRU-evicted, on purpose, so the operator's default reading of
an unexpected "gone" is that the secret is compromised and should be
rotated.
* scripts/logs.sh and alerts-check.sh verify rather than assert:
alerts-check asks vmalert what it actually loaded AND checks each rule's
series exists, because a rule reading a metric nothing exports can never
fire and looks exactly like a healthy service.
* scripts/smoke.sh is a real client — it generates a key, encrypts, posts only
ciphertext, reveals, decrypts, then asserts the second reveal is 410, that
three GETs did not consume the secret, that missing and malformed ids are
indistinguishable, and that a plaintext field is refused.
install-mcp.sh proves the MCP handshake before writing any config, backs up
mcp.json, and rewrites only hush's entry — a config pointing at a broken server
surfaces as an opaque host-side connect failure, which is worth one extra check
to avoid.
