hush/docs/DEPLOY.md

# Deploying hush

Live at <https://hush.threesix.ai>, one replica in the `projects` namespace on
the orchard9 k3s cluster.

```
git push origin main → Gitea webhook → Woodpecker → Kaniko (amd64, in-cluster)
  → Zot registry → kubectl set image → projects/hush
```

`origin` **must** be Gitea (`git.threesix.ai`); that remote carries the webhook.
The GitHub mirror is a backup and pushing there deploys nothing.

## Never build the image locally

Two reasons, both of which cost real time to discover:

1. A laptop build on Apple Silicon produces **arm64**; the cluster runs amd64.
2. `registry.threesix.ai` accepts **only OCI image manifests** — not Docker
   schema2, and not an OCI *index*. `docker push` and `crane push` both fail
   `MANIFEST_INVALID`, and buildx wraps even a single-platform build in an index.

Kaniko sidesteps both. If you must build outside the pipeline, use a Job — see
"Bootstrap" below, which is exactly what the first deploy did.

## Dependencies are vendored, deliberately

`github.com/orchard9/go-chassis` is a **private** module. Neither the Woodpecker
test container nor the Kaniko build holds a git credential, so a build that
resolved dependencies from the network would fail:

```
$ curl https://proxy.golang.org/github.com/orchard9/go-chassis/@v/list
404  ... could not read Username for 'https://github.com'
```

So `vendor/` is committed and both CI and the Dockerfile run `-mod=vendor` with
`GOPROXY=off`. `GOPROXY=off` is the important half: it turns "silently fetched
from somewhere" into a hard failure. `make vendor` is the only way versions
move, and `make verify` proves the tree still builds with no network.

## One-time setup, already done

Recorded because it is what a rebuild would need, and none of it is in git.

### 1. Redis ACL user

hush connects as its own ACL user, scoped to `~hush:*`, on **db 5** (0 and 3 and
4 are taken by pantheon/rdev, reel, and jit):

```bash
redis-cli ACL SETUSER hush on '>PASSWORD' '~hush:*' resetchannels \
  -@all +ping +set +getdel +incr +pexpire +pttl +select
redis-cli ACL SAVE     # persists to /data/users.acl
```

**`+getdel` is the one to notice.** No other service's ACL user has it, because
no other service needs an atomic read-and-destroy. Omit it and creates keep
working while every reveal fails `NOPERM` — a service that accepts secrets and
cannot deliver them. `+incr +pexpire +pttl` are the rate limiter.

### 2. The credential

A JSON object in GCP Secret Manager, pulled into the namespace by ESO:

```bash
gcloud secrets create k3sf-hush-credentials --project orchard9 \
  --replication-policy=automatic --data-file=- <<< \
  '{"REDIS_URL":"redis://hush:PASSWORD@redis.databases.svc.cluster.local:6379/5"}'
```

`ExternalSecret/hush-credentials` (in `deployments/k8s/hush.yaml`) syncs it to a
Secret of the same name. Confirm with:

```bash
kubectl -n projects get externalsecret hush-credentials \
  -o jsonpath='{.status.conditions[0].reason}'   # want SecretSynced
```

### 3. DNS

`hush.threesix.ai` → `208.122.204.172`, A record, **DNS-only** (not proxied),
TTL 120 — matching every other `*.threesix.ai` service. There is no wildcard on
the zone, so each host needs its own record. cert-manager then issues TLS over
HTTP-01 with no DNS credential needed.

> `hush.orchard9.ai` was the original intent and is **not** what shipped.
> `orchard9.ai` is on GoDaddy and no GoDaddy credential exists in rdev, the
> cluster, or `~/.squiddy-dns`. Moving the host there needs that credential;
> everything else is a one-line Ingress change plus a new record.

### 4. Alert rules

vmalert has **no ConfigMap auto-discovery**. Three coordinated edits in
`orchard9-k3sf`, and missing any one leaves the rules silently absent:

1. `observability/hush-alerting-rules.yaml` — the ConfigMap
2. `observability/kustomization.yaml` — list it under `resources:`
3. `observability/victoria-metrics.yaml` — vmalert needs a matching
   `-rule=/etc/rules-hush/*.yaml`, `volumeMount` and `volume`

`make alerts-check` asks vmalert what it actually loaded, and separately checks
that every series the rules reference exists — a rule reading a metric nothing
exports can never fire and looks exactly like a healthy service.

## Bootstrap (what the first deploy did)

The pipeline's deploy step runs `kubectl set image`, so a Deployment must exist
first. But the committed image tag cannot be `:latest`: the cluster's
`stable-controller-images.orchard9.ai` admission policy refuses
`latest|main|master|dev|edge|canary|nightly|snapshot`, because a floating tag
cannot pin a rollback.

So the manifest carries `:bootstrap`, which is policy-legal and does not exist.
Apply everything, then build once by hand:

```bash
make deploy-manifests        # pod sits in ImagePullBackOff — expected

SHA=$(git rev-parse --short=8 HEAD)
kubectl -n projects create job hush-build-$SHA --dry-run=client -o yaml ... # see below
kubectl -n projects set image deployment/hush hushd=registry.threesix.ai/hush/api:$SHA
```

That `:bootstrap` tag is also why the public route lives in its own file,
`deployments/k8s/ingress.yaml`. A new public path — every handler needs one, or
it 404s at the edge while working fine in `make dev` — is
`make deploy-ingress`, which applies that object alone. Applying the whole
directory to publish a path would roll the workload back onto the unpullable
bootstrap image.

The build Job itself is not reproduced here. `scripts/release.sh` emits it, and
a copy in this file drifts from the copy that runs — which is how the script's
own push guard ended up checking the wrong remote. Read it there; `make
release` is the same path, and it needs no CI credential because the Gitea repo
is public.

One thing about that Job is worth stating outside the script, because it is a
property of the cluster rather than of hush: `git.threesix.ai` and
`registry.threesix.ai` resolve to the cluster's public address, and reaching
that address from inside a pod hairpins unreliably. Measured 2026-09-05: 14 of
24 requests from a pod succeeded, four consecutive kaniko pushes were refused
with `connection refused`, and Traefik's ClusterIP answered 8 of 8. The Job
therefore carries `hostAliases` pinning both names to that ClusterIP. Anything
else that builds in-cluster — Woodpecker's Kaniko step included — is exposed to
the same hairpin and will fail the same way, intermittently.

## Woodpecker: activated

`jordan/hush` is repo 139 in Woodpecker, active, with the webhook installed on
the Gitea side. A push to `main` builds and deploys.

Activation is `POST /api/repos?forge_remote_id=<numeric gitea repo id>` — the
**numeric** id (183 here), not `owner/name`:

```bash
GID=$(curl -s -H "Authorization: token $THREE_SIX_GITEA" \
        https://git.threesix.ai/api/v1/repos/jordan/hush | jq -r .id)
curl -X POST "https://ci.threesix.ai/api/repos?forge_remote_id=$GID" \
  -H "Authorization: Bearer $THREE_SIX_WOODPECKER"
```

### The credential

Use **`$THREE_SIX_WOODPECKER`** (and `$THREE_SIX_GITEA` for Gitea). Both are in
the operator's environment.

The copy in `rdev/rdev-credentials` was stale and returned
`401 User not authorized` on `/api/user` — which is worth knowing how to
diagnose, because a 401 on `POST /api/repos` looks exactly like a malformed
`forge_remote_id`. `GET /api/user` separates the two: it is auth-only, so a 401
there is the token and a 200 there means the request shape is what is wrong.

That stale copy is **fixed at the source**: `k3sf-rdev-admin-key` in GCP Secret
Manager (property `WOODPECKER_API_TOKEN`) now carries the working token, ESO
resynced it, and the token read out of `rdev/rdev-credentials` returns 200. The
other properties in that secret were preserved. Do not patch the k8s Secret
directly — it is ESO-owned and a direct edit is reverted on the next refresh.

```bash
# force a resync rather than waiting out refreshInterval: 1h
kubectl -n rdev annotate externalsecret rdev-credentials force-sync="$(date +%s)" --overwrite
```

### `make release` — the path that needs no CI credential

Still useful with CI working: it is the hotfix and rollback path when the
pipeline is down, and it was how the first deploy happened.

```bash
make release
```

It refuses on a dirty or unpushed tree, because Kaniko builds from the pushed
git ref and would otherwise silently build something other than what you are
looking at. It also asserts the live image equals the one just built, since
`set image` matching nothing is silent and the rollout would "succeed" on the
old pod.

## Rollback

Every build is SHA-tagged, so rollback is naming the previous one:

```bash
kubectl -n projects rollout undo deployment/hush
# or explicitly
kubectl -n projects set image deployment/hush hushd=registry.threesix.ai/hush/api:<older-sha>
```

Nothing to migrate and no schema: Redis holds only TTL'd ciphertext, and a
rollback cannot invalidate an outstanding link because the id and the wire
format are stable.

## Verifying a deploy

```bash
make deploy-status                       # rollout, pods, ingress, certificate
BASE=https://hush.threesix.ai make smoke  # real crypto, create → reveal → gone
make logs                                # the lifecycle in VictoriaLogs
make alerts-check                        # rules loaded, series present
```

`make smoke` is the one that matters. It encrypts with a real AES-256-GCM key,
posts only ciphertext, reveals once, decrypts, and then asserts the second
reveal is `410`, that three `GET`s did not consume the secret, that missing and
malformed ids are indistinguishable, and that a plaintext field is refused.