d7cd57f330
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
Using hush from an agent needed a clone and docs/MCP.md. It now needs one
command, and the instructions are served by the deployment itself.
`go install github.com/orchard9/hush/cmd/hush-mcp@latest` is the whole
install: cmd/hush-mcp imports only the standard library, so module graph
pruning never reaches the private go-chassis dependency cmd/hushd needs.
Verified against an empty module cache and the public proxy, then create ->
reveal end to end against production with the resulting binary.
The page carries the per-client configuration for Claude Code, Codex CLI,
Gemini CLI, VS Code, Claude Desktop, Cursor and omp. Each command was run
against the installed client rather than copied from documentation, which is
how the differences on it are there at all: VS Code's wrapper key is
`servers`, not `mcpServers`; gemini defaults to project scope, not user;
Claude Code rejects `--env` immediately before the server name.
The shared browser crypto moves from base.html into templates/crypto.html,
which the two pages that encrypt parse and this one does not. An empty
`{{define}}` cannot replace a non-empty one — text/template reads an empty
body as no definition — so the shell holds the call and the partial holds the
code, and the docs page ships no script at all.
Three things this exposed, fixed here:
- The public Ingress enumerates paths, so a handler without one 404s at the
edge while working in `make dev`. The Ingress is now its own manifest:
hush.yaml pins a `:bootstrap` image that does not exist, so re-applying it
to publish a path would roll the workload onto an unpullable image.
`make deploy-ingress` applies the route alone.
- release.sh guarded HEAD against `@{upstream}`, which is the GitHub mirror
here, while Kaniko clones Gitea. A commit pushed to one and not the other
would have built the previous commit silently. It now fetches and compares
the branch that actually gets built.
- smoke.sh checks that /mcp serves the install command, so a stale rollout or
an unexecutable template fails the release instead of being found later.
Confirmed it fails: against production before this deploy it reported 404.
240 lines
9.3 KiB
Markdown
240 lines
9.3 KiB
Markdown
# Deploying hush
|
|
|
|
Live at <https://hush.threesix.ai>, one replica in the `projects` namespace on
|
|
the orchard9 k3s cluster.
|
|
|
|
```
|
|
git push origin main → Gitea webhook → Woodpecker → Kaniko (amd64, in-cluster)
|
|
→ Zot registry → kubectl set image → projects/hush
|
|
```
|
|
|
|
`origin` **must** be Gitea (`git.threesix.ai`); that remote carries the webhook.
|
|
The GitHub mirror is a backup and pushing there deploys nothing.
|
|
|
|
## Never build the image locally
|
|
|
|
Two reasons, both of which cost real time to discover:
|
|
|
|
1. A laptop build on Apple Silicon produces **arm64**; the cluster runs amd64.
|
|
2. `registry.threesix.ai` accepts **only OCI image manifests** — not Docker
|
|
schema2, and not an OCI *index*. `docker push` and `crane push` both fail
|
|
`MANIFEST_INVALID`, and buildx wraps even a single-platform build in an index.
|
|
|
|
Kaniko sidesteps both. If you must build outside the pipeline, use a Job — see
|
|
"Bootstrap" below, which is exactly what the first deploy did.
|
|
|
|
## Dependencies are vendored, deliberately
|
|
|
|
`github.com/orchard9/go-chassis` is a **private** module. Neither the Woodpecker
|
|
test container nor the Kaniko build holds a git credential, so a build that
|
|
resolved dependencies from the network would fail:
|
|
|
|
```
|
|
$ curl https://proxy.golang.org/github.com/orchard9/go-chassis/@v/list
|
|
404 ... could not read Username for 'https://github.com'
|
|
```
|
|
|
|
So `vendor/` is committed and both CI and the Dockerfile run `-mod=vendor` with
|
|
`GOPROXY=off`. `GOPROXY=off` is the important half: it turns "silently fetched
|
|
from somewhere" into a hard failure. `make vendor` is the only way versions
|
|
move, and `make verify` proves the tree still builds with no network.
|
|
|
|
## One-time setup, already done
|
|
|
|
Recorded because it is what a rebuild would need, and none of it is in git.
|
|
|
|
### 1. Redis ACL user
|
|
|
|
hush connects as its own ACL user, scoped to `~hush:*`, on **db 5** (0 and 3 and
|
|
4 are taken by pantheon/rdev, reel, and jit):
|
|
|
|
```bash
|
|
redis-cli ACL SETUSER hush on '>PASSWORD' '~hush:*' resetchannels \
|
|
-@all +ping +set +getdel +incr +pexpire +pttl +select
|
|
redis-cli ACL SAVE # persists to /data/users.acl
|
|
```
|
|
|
|
**`+getdel` is the one to notice.** No other service's ACL user has it, because
|
|
no other service needs an atomic read-and-destroy. Omit it and creates keep
|
|
working while every reveal fails `NOPERM` — a service that accepts secrets and
|
|
cannot deliver them. `+incr +pexpire +pttl` are the rate limiter.
|
|
|
|
### 2. The credential
|
|
|
|
A JSON object in GCP Secret Manager, pulled into the namespace by ESO:
|
|
|
|
```bash
|
|
gcloud secrets create k3sf-hush-credentials --project orchard9 \
|
|
--replication-policy=automatic --data-file=- <<< \
|
|
'{"REDIS_URL":"redis://hush:PASSWORD@redis.databases.svc.cluster.local:6379/5"}'
|
|
```
|
|
|
|
`ExternalSecret/hush-credentials` (in `deployments/k8s/hush.yaml`) syncs it to a
|
|
Secret of the same name. Confirm with:
|
|
|
|
```bash
|
|
kubectl -n projects get externalsecret hush-credentials \
|
|
-o jsonpath='{.status.conditions[0].reason}' # want SecretSynced
|
|
```
|
|
|
|
### 3. DNS
|
|
|
|
`hush.threesix.ai` → `208.122.204.172`, A record, **DNS-only** (not proxied),
|
|
TTL 120 — matching every other `*.threesix.ai` service. There is no wildcard on
|
|
the zone, so each host needs its own record. cert-manager then issues TLS over
|
|
HTTP-01 with no DNS credential needed.
|
|
|
|
> `hush.orchard9.ai` was the original intent and is **not** what shipped.
|
|
> `orchard9.ai` is on GoDaddy and no GoDaddy credential exists in rdev, the
|
|
> cluster, or `~/.squiddy-dns`. Moving the host there needs that credential;
|
|
> everything else is a one-line Ingress change plus a new record.
|
|
|
|
### 4. Alert rules
|
|
|
|
vmalert has **no ConfigMap auto-discovery**. Three coordinated edits in
|
|
`orchard9-k3sf`, and missing any one leaves the rules silently absent:
|
|
|
|
1. `observability/hush-alerting-rules.yaml` — the ConfigMap
|
|
2. `observability/kustomization.yaml` — list it under `resources:`
|
|
3. `observability/victoria-metrics.yaml` — vmalert needs a matching
|
|
`-rule=/etc/rules-hush/*.yaml`, `volumeMount` and `volume`
|
|
|
|
`make alerts-check` asks vmalert what it actually loaded, and separately checks
|
|
that every series the rules reference exists — a rule reading a metric nothing
|
|
exports can never fire and looks exactly like a healthy service.
|
|
|
|
## Bootstrap (what the first deploy did)
|
|
|
|
The pipeline's deploy step runs `kubectl set image`, so a Deployment must exist
|
|
first. But the committed image tag cannot be `:latest`: the cluster's
|
|
`stable-controller-images.orchard9.ai` admission policy refuses
|
|
`latest|main|master|dev|edge|canary|nightly|snapshot`, because a floating tag
|
|
cannot pin a rollback.
|
|
|
|
So the manifest carries `:bootstrap`, which is policy-legal and does not exist.
|
|
Apply everything, then build once by hand:
|
|
|
|
```bash
|
|
make deploy-manifests # pod sits in ImagePullBackOff — expected
|
|
|
|
SHA=$(git rev-parse --short=8 HEAD)
|
|
kubectl -n projects create job hush-build-$SHA --dry-run=client -o yaml ... # see below
|
|
kubectl -n projects set image deployment/hush hushd=registry.threesix.ai/hush/api:$SHA
|
|
```
|
|
|
|
That `:bootstrap` tag is also why the public route lives in its own file,
|
|
`deployments/k8s/ingress.yaml`. A new public path — every handler needs one, or
|
|
it 404s at the edge while working fine in `make dev` — is
|
|
`make deploy-ingress`, which applies that object alone. Applying the whole
|
|
directory to publish a path would roll the workload back onto the unpullable
|
|
bootstrap image.
|
|
|
|
The build Job, which is what Woodpecker's Kaniko step does by hand:
|
|
|
|
```yaml
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata: { name: hush-build, namespace: projects }
|
|
spec:
|
|
backoffLimit: 1
|
|
ttlSecondsAfterFinished: 3600
|
|
template:
|
|
spec:
|
|
restartPolicy: Never
|
|
containers:
|
|
- name: kaniko
|
|
image: gcr.io/kaniko-project/executor:v1.23.2
|
|
args:
|
|
- --context=git://git.threesix.ai/jordan/hush.git#refs/heads/main
|
|
- --dockerfile=Dockerfile
|
|
- --destination=registry.threesix.ai/hush/api:SHA
|
|
- --skip-tls-verify
|
|
- --skip-tls-verify-pull
|
|
- --single-snapshot
|
|
resources:
|
|
requests: { cpu: 500m, memory: 1Gi }
|
|
limits: { cpu: "2", memory: 3Gi }
|
|
```
|
|
|
|
The git context needs no credential because the Gitea repo is public.
|
|
|
|
## Woodpecker: activated
|
|
|
|
`jordan/hush` is repo 139 in Woodpecker, active, with the webhook installed on
|
|
the Gitea side. A push to `main` builds and deploys.
|
|
|
|
Activation is `POST /api/repos?forge_remote_id=<numeric gitea repo id>` — the
|
|
**numeric** id (183 here), not `owner/name`:
|
|
|
|
```bash
|
|
GID=$(curl -s -H "Authorization: token $THREE_SIX_GITEA" \
|
|
https://git.threesix.ai/api/v1/repos/jordan/hush | jq -r .id)
|
|
curl -X POST "https://ci.threesix.ai/api/repos?forge_remote_id=$GID" \
|
|
-H "Authorization: Bearer $THREE_SIX_WOODPECKER"
|
|
```
|
|
|
|
### The credential
|
|
|
|
Use **`$THREE_SIX_WOODPECKER`** (and `$THREE_SIX_GITEA` for Gitea). Both are in
|
|
the operator's environment.
|
|
|
|
The copy in `rdev/rdev-credentials` was stale and returned
|
|
`401 User not authorized` on `/api/user` — which is worth knowing how to
|
|
diagnose, because a 401 on `POST /api/repos` looks exactly like a malformed
|
|
`forge_remote_id`. `GET /api/user` separates the two: it is auth-only, so a 401
|
|
there is the token and a 200 there means the request shape is what is wrong.
|
|
|
|
That stale copy is **fixed at the source**: `k3sf-rdev-admin-key` in GCP Secret
|
|
Manager (property `WOODPECKER_API_TOKEN`) now carries the working token, ESO
|
|
resynced it, and the token read out of `rdev/rdev-credentials` returns 200. The
|
|
other properties in that secret were preserved. Do not patch the k8s Secret
|
|
directly — it is ESO-owned and a direct edit is reverted on the next refresh.
|
|
|
|
```bash
|
|
# force a resync rather than waiting out refreshInterval: 1h
|
|
kubectl -n rdev annotate externalsecret rdev-credentials force-sync="$(date +%s)" --overwrite
|
|
```
|
|
|
|
### `make release` — the path that needs no CI credential
|
|
|
|
Still useful with CI working: it is the hotfix and rollback path when the
|
|
pipeline is down, and it was how the first deploy happened.
|
|
|
|
```bash
|
|
make release
|
|
```
|
|
|
|
It refuses on a dirty or unpushed tree, because Kaniko builds from the pushed
|
|
git ref and would otherwise silently build something other than what you are
|
|
looking at. It also asserts the live image equals the one just built, since
|
|
`set image` matching nothing is silent and the rollout would "succeed" on the
|
|
old pod.
|
|
|
|
## Rollback
|
|
|
|
Every build is SHA-tagged, so rollback is naming the previous one:
|
|
|
|
```bash
|
|
kubectl -n projects rollout undo deployment/hush
|
|
# or explicitly
|
|
kubectl -n projects set image deployment/hush hushd=registry.threesix.ai/hush/api:<older-sha>
|
|
```
|
|
|
|
Nothing to migrate and no schema: Redis holds only TTL'd ciphertext, and a
|
|
rollback cannot invalidate an outstanding link because the id and the wire
|
|
format are stable.
|
|
|
|
## Verifying a deploy
|
|
|
|
```bash
|
|
make deploy-status # rollout, pods, ingress, certificate
|
|
BASE=https://hush.threesix.ai make smoke # real crypto, create → reveal → gone
|
|
make logs # the lifecycle in VictoriaLogs
|
|
make alerts-check # rules loaded, series present
|
|
```
|
|
|
|
`make smoke` is the one that matters. It encrypts with a real AES-256-GCM key,
|
|
posts only ciphertext, reveals once, decrypts, and then asserts the second
|
|
reveal is `410`, that three `GET`s did not consume the secret, that missing and
|
|
malformed ids are indistinguishable, and that a plaintext field is refused.
|