docs, ops scripts, and the MCP install
Written after the service was live, so every command and every number here was
run against the real deployment rather than assumed:
* DEPLOY.md records the things a rebuild needs and git does not hold — the
Redis ACL user (and why +getdel is the one to notice), the GCP Secret
Manager entry, the DNS record, and the three coordinated edits vmalert
needs because it has no ConfigMap auto-discovery.
* It also records two blockers rather than hiding them: Woodpecker is NOT
activated (the token in rdev-credentials returns 401), so pushes do not
deploy yet and the Kaniko Job is the interim path; and the host is
hush.threesix.ai rather than hush.orchard9.ai because orchard9.ai is on
GoDaddy and no GoDaddy credential exists anywhere I can reach.
* OPERATIONS.md is one section per alert, plus the failure modes that are not
alerts — chiefly that "gone" cannot distinguish already-revealed from
expired from LRU-evicted, on purpose, so the operator's default reading of
an unexpected "gone" is that the secret is compromised and should be
rotated.
* scripts/logs.sh and alerts-check.sh verify rather than assert:
alerts-check asks vmalert what it actually loaded AND checks each rule's
series exists, because a rule reading a metric nothing exports can never
fire and looks exactly like a healthy service.
* scripts/smoke.sh is a real client — it generates a key, encrypts, posts only
ciphertext, reveals, decrypts, then asserts the second reveal is 410, that
three GETs did not consume the secret, that missing and malformed ids are
indistinguishable, and that a plaintext field is refused.
install-mcp.sh proves the MCP handshake before writing any config, backs up
mcp.json, and rewrites only hush's entry — a config pointing at a broken server
surfaces as an opaque host-side connect failure, which is worth one extra check
to avoid.
This commit is contained in:
parent
4d9a26498e
commit
62c95f8c2f
197
docs/DEPLOY.md
Normal file
197
docs/DEPLOY.md
Normal file
@ -0,0 +1,197 @@
|
||||
# Deploying hush
|
||||
|
||||
Live at <https://hush.threesix.ai>, one replica in the `projects` namespace on
|
||||
the orchard9 k3s cluster.
|
||||
|
||||
```
|
||||
git push origin main → Gitea webhook → Woodpecker → Kaniko (amd64, in-cluster)
|
||||
→ Zot registry → kubectl set image → projects/hush
|
||||
```
|
||||
|
||||
`origin` **must** be Gitea (`git.threesix.ai`); that remote carries the webhook.
|
||||
The GitHub mirror is a backup and pushing there deploys nothing.
|
||||
|
||||
## Never build the image locally
|
||||
|
||||
Two reasons, both of which cost real time to discover:
|
||||
|
||||
1. A laptop build on Apple Silicon produces **arm64**; the cluster runs amd64.
|
||||
2. `registry.threesix.ai` accepts **only OCI image manifests** — not Docker
|
||||
schema2, and not an OCI *index*. `docker push` and `crane push` both fail
|
||||
`MANIFEST_INVALID`, and buildx wraps even a single-platform build in an index.
|
||||
|
||||
Kaniko sidesteps both. If you must build outside the pipeline, use a Job — see
|
||||
"Bootstrap" below, which is exactly what the first deploy did.
|
||||
|
||||
## Dependencies are vendored, deliberately
|
||||
|
||||
`github.com/orchard9/go-chassis` is a **private** module. Neither the Woodpecker
|
||||
test container nor the Kaniko build holds a git credential, so a build that
|
||||
resolved dependencies from the network would fail:
|
||||
|
||||
```
|
||||
$ curl https://proxy.golang.org/github.com/orchard9/go-chassis/@v/list
|
||||
404 ... could not read Username for 'https://github.com'
|
||||
```
|
||||
|
||||
So `vendor/` is committed and both CI and the Dockerfile run `-mod=vendor` with
|
||||
`GOPROXY=off`. `GOPROXY=off` is the important half: it turns "silently fetched
|
||||
from somewhere" into a hard failure. `make vendor` is the only way versions
|
||||
move, and `make verify` proves the tree still builds with no network.
|
||||
|
||||
## One-time setup, already done
|
||||
|
||||
Recorded because it is what a rebuild would need, and none of it is in git.
|
||||
|
||||
### 1. Redis ACL user
|
||||
|
||||
hush connects as its own ACL user, scoped to `~hush:*`, on **db 5** (0 and 3 and
|
||||
4 are taken by pantheon/rdev, reel, and jit):
|
||||
|
||||
```bash
|
||||
redis-cli ACL SETUSER hush on '>PASSWORD' '~hush:*' resetchannels \
|
||||
-@all +ping +set +getdel +incr +pexpire +pttl +select
|
||||
redis-cli ACL SAVE # persists to /data/users.acl
|
||||
```
|
||||
|
||||
**`+getdel` is the one to notice.** No other service's ACL user has it, because
|
||||
no other service needs an atomic read-and-destroy. Omit it and creates keep
|
||||
working while every reveal fails `NOPERM` — a service that accepts secrets and
|
||||
cannot deliver them. `+incr +pexpire +pttl` are the rate limiter.
|
||||
|
||||
### 2. The credential
|
||||
|
||||
A JSON object in GCP Secret Manager, pulled into the namespace by ESO:
|
||||
|
||||
```bash
|
||||
gcloud secrets create k3sf-hush-credentials --project orchard9 \
|
||||
--replication-policy=automatic --data-file=- <<< \
|
||||
'{"REDIS_URL":"redis://hush:PASSWORD@redis.databases.svc.cluster.local:6379/5"}'
|
||||
```
|
||||
|
||||
`ExternalSecret/hush-credentials` (in `deployments/k8s/hush.yaml`) syncs it to a
|
||||
Secret of the same name. Confirm with:
|
||||
|
||||
```bash
|
||||
kubectl -n projects get externalsecret hush-credentials \
|
||||
-o jsonpath='{.status.conditions[0].reason}' # want SecretSynced
|
||||
```
|
||||
|
||||
### 3. DNS
|
||||
|
||||
`hush.threesix.ai` → `208.122.204.172`, A record, **DNS-only** (not proxied),
|
||||
TTL 120 — matching every other `*.threesix.ai` service. There is no wildcard on
|
||||
the zone, so each host needs its own record. cert-manager then issues TLS over
|
||||
HTTP-01 with no DNS credential needed.
|
||||
|
||||
> `hush.orchard9.ai` was the original intent and is **not** what shipped.
|
||||
> `orchard9.ai` is on GoDaddy and no GoDaddy credential exists in rdev, the
|
||||
> cluster, or `~/.squiddy-dns`. Moving the host there needs that credential;
|
||||
> everything else is a one-line Ingress change plus a new record.
|
||||
|
||||
### 4. Alert rules
|
||||
|
||||
vmalert has **no ConfigMap auto-discovery**. Three coordinated edits in
|
||||
`orchard9-k3sf`, and missing any one leaves the rules silently absent:
|
||||
|
||||
1. `observability/hush-alerting-rules.yaml` — the ConfigMap
|
||||
2. `observability/kustomization.yaml` — list it under `resources:`
|
||||
3. `observability/victoria-metrics.yaml` — vmalert needs a matching
|
||||
`-rule=/etc/rules-hush/*.yaml`, `volumeMount` and `volume`
|
||||
|
||||
`make alerts-check` asks vmalert what it actually loaded, and separately checks
|
||||
that every series the rules reference exists — a rule reading a metric nothing
|
||||
exports can never fire and looks exactly like a healthy service.
|
||||
|
||||
## Bootstrap (what the first deploy did)
|
||||
|
||||
The pipeline's deploy step runs `kubectl set image`, so a Deployment must exist
|
||||
first. But the committed image tag cannot be `:latest`: the cluster's
|
||||
`stable-controller-images.orchard9.ai` admission policy refuses
|
||||
`latest|main|master|dev|edge|canary|nightly|snapshot`, because a floating tag
|
||||
cannot pin a rollback.
|
||||
|
||||
So the manifest carries `:bootstrap`, which is policy-legal and does not exist.
|
||||
Apply it, then build once by hand:
|
||||
|
||||
```bash
|
||||
make deploy-manifests # pod sits in ImagePullBackOff — expected
|
||||
|
||||
SHA=$(git rev-parse --short=8 HEAD)
|
||||
kubectl -n projects create job hush-build-$SHA --dry-run=client -o yaml ... # see below
|
||||
kubectl -n projects set image deployment/hush hushd=registry.threesix.ai/hush/api:$SHA
|
||||
```
|
||||
|
||||
The build Job, which is what Woodpecker's Kaniko step does by hand:
|
||||
|
||||
```yaml
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata: { name: hush-build, namespace: projects }
|
||||
spec:
|
||||
backoffLimit: 1
|
||||
ttlSecondsAfterFinished: 3600
|
||||
template:
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: kaniko
|
||||
image: gcr.io/kaniko-project/executor:v1.23.2
|
||||
args:
|
||||
- --context=git://git.threesix.ai/jordan/hush.git#refs/heads/main
|
||||
- --dockerfile=Dockerfile
|
||||
- --destination=registry.threesix.ai/hush/api:SHA
|
||||
- --skip-tls-verify
|
||||
- --skip-tls-verify-pull
|
||||
- --single-snapshot
|
||||
resources:
|
||||
requests: { cpu: 500m, memory: 1Gi }
|
||||
limits: { cpu: "2", memory: 3Gi }
|
||||
```
|
||||
|
||||
The git context needs no credential because the Gitea repo is public.
|
||||
|
||||
## Woodpecker is NOT yet activated
|
||||
|
||||
The repo exists on Gitea and `.woodpecker.yml` is committed, but activation
|
||||
failed: the `WOODPECKER_API_TOKEN` in `rdev/rdev-credentials` returns
|
||||
`401 User not authorized`.
|
||||
|
||||
Until a valid token replaces it, **pushes do not deploy** — use the Kaniko Job
|
||||
above and `kubectl set image`. To finish it:
|
||||
|
||||
```bash
|
||||
WP=$(curl -s -H "X-API-Key: $RDEV_API_KEY" "$RDEV_API_URL/credentials/WOODPECKER_API_TOKEN" | jq -r '.data.value')
|
||||
curl -X POST "https://ci.threesix.ai/api/repos?forge_remote_id=jordan/hush" -H "Authorization: Bearer $WP"
|
||||
```
|
||||
|
||||
A fresh token comes from Woodpecker → User Settings → Token, and belongs back in
|
||||
rdev rather than anywhere else.
|
||||
|
||||
## Rollback
|
||||
|
||||
Every build is SHA-tagged, so rollback is naming the previous one:
|
||||
|
||||
```bash
|
||||
kubectl -n projects rollout undo deployment/hush
|
||||
# or explicitly
|
||||
kubectl -n projects set image deployment/hush hushd=registry.threesix.ai/hush/api:<older-sha>
|
||||
```
|
||||
|
||||
Nothing to migrate and no schema: Redis holds only TTL'd ciphertext, and a
|
||||
rollback cannot invalidate an outstanding link because the id and the wire
|
||||
format are stable.
|
||||
|
||||
## Verifying a deploy
|
||||
|
||||
```bash
|
||||
make deploy-status # rollout, pods, ingress, certificate
|
||||
BASE=https://hush.threesix.ai make smoke # real crypto, create → reveal → gone
|
||||
make logs # the lifecycle in VictoriaLogs
|
||||
make alerts-check # rules loaded, series present
|
||||
```
|
||||
|
||||
`make smoke` is the one that matters. It encrypts with a real AES-256-GCM key,
|
||||
posts only ciphertext, reveals once, decrypts, and then asserts the second
|
||||
reveal is `410`, that three `GET`s did not consume the secret, that missing and
|
||||
malformed ids are indistinguishable, and that a plaintext field is refused.
|
||||
111
docs/MCP.md
Normal file
111
docs/MCP.md
Normal file
@ -0,0 +1,111 @@
|
||||
# The MCP server
|
||||
|
||||
`cmd/hush-mcp` gives an MCP host (omp, Claude Code, any client) two tools:
|
||||
|
||||
| Tool | Does |
|
||||
|---|---|
|
||||
| `hush_create` | Encrypts a secret locally, stores the ciphertext, returns a one-time link |
|
||||
| `hush_reveal` | Fetches and decrypts a link, **destroying it** |
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
make mcp
|
||||
```
|
||||
|
||||
That builds the binary to `~/.local/bin/hush-mcp`, proves the MCP handshake
|
||||
works before wiring anything, and adds a `hush` entry to
|
||||
`~/.omp/agent/mcp.json` — backing the file up first and leaving every other
|
||||
server alone. Restart omp to pick it up.
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"hush": {
|
||||
"type": "stdio",
|
||||
"command": "/Users/you/.local/bin/hush-mcp",
|
||||
"env": { "HUSH_BASE_URL": "https://hush.threesix.ai" },
|
||||
"timeout": 20000
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The same file shape works for Claude Code (`~/.claude.json`), Cursor and VS
|
||||
Code — the stdio transport is the portable part.
|
||||
|
||||
## Why it runs locally instead of being an endpoint on hushd
|
||||
|
||||
hushd could serve `/mcp` and encrypt on the server. It deliberately does not.
|
||||
|
||||
If the server did the encrypting, the server would see every plaintext created
|
||||
through MCP. hush's guarantee — *we cannot read your secrets* — would then hold
|
||||
for browser users and quietly not hold for agent users, and no one could tell
|
||||
which they had by looking at a link. Two guarantees behind one URL is worse
|
||||
than one honest guarantee.
|
||||
|
||||
So `hush-mcp` is a peer of the browser, not of the server: it mints the AES-256
|
||||
key, encrypts, posts only ciphertext, and assembles the `#fragment` link
|
||||
itself. hushd sees exactly what it sees from a browser and no more.
|
||||
|
||||
The cost is that this is a local binary to install rather than a URL to
|
||||
configure. That is the right trade for a service whose entire value is where
|
||||
the key sits.
|
||||
|
||||
## Wire compatibility
|
||||
|
||||
Three implementations produce and consume one format — the browser
|
||||
(`internal/web/templates/base.html`), this server, and `scripts/smoke.sh`:
|
||||
|
||||
```
|
||||
AES-256-GCM, 96-bit nonce PREPENDED to the ciphertext,
|
||||
both ciphertext and key base64url-encoded WITHOUT padding
|
||||
```
|
||||
|
||||
A link minted by any of the three opens in the other two.
|
||||
`TestWireFormatMatchesAnIndependentImplementation` pins that by round-tripping
|
||||
Go↔Python in both directions, so a change to one implementation's encoding
|
||||
fails the build rather than producing links that only work in the client that
|
||||
made them.
|
||||
|
||||
## Configuration
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `HUSH_BASE_URL` | `https://hush.threesix.ai` | Which deployment `hush_create` posts to |
|
||||
| `HUSH_CREATE_TOKEN` | unset | Only needed if that deployment has `HUSH_REQUIRE_AUTH=true` |
|
||||
|
||||
`hush_reveal` ignores `HUSH_BASE_URL` and reveals against **the link's own
|
||||
origin**. A link from another hush deployment must not be posted to this one,
|
||||
where its id would be meaningless — and silently revealing against the wrong
|
||||
host would report `gone` for a secret that was never touched.
|
||||
|
||||
## Behaviour worth knowing before you call it
|
||||
|
||||
- **`hush_reveal` is destructive and irreversible.** After it returns, the link
|
||||
is dead and the intended recipient cannot open it. The tool description says
|
||||
so, because a model that calls it to "check" a link has burned it.
|
||||
- **`hush_create` returns the link once.** It cannot be recovered: the key was
|
||||
never sent to the server, so nothing can rebuild it.
|
||||
- **A link with no `#fragment` is reported as an error without touching the
|
||||
secret.** Chat and email clients truncate fragments, and this is the commonest
|
||||
real failure. The secret is intact and the fix is to ask for the full link —
|
||||
which the error says, rather than reporting a generic failure.
|
||||
- **Errors come back as tool errors** (`isError: true`), not protocol errors, so
|
||||
the model reads the message and can act on it instead of seeing an opaque
|
||||
transport failure.
|
||||
|
||||
## Protocol notes
|
||||
|
||||
Implemented directly against the JSON-RPC 2.0 stdio transport rather than via an
|
||||
SDK: the surface needed is `initialize`, `notifications/initialized`,
|
||||
`tools/list`, `tools/call` and `ping`, which is less code than an SDK
|
||||
dependency's API churn would cost.
|
||||
|
||||
Two rules the implementation is careful about, both of which produce
|
||||
hard-to-diagnose host-side failures when broken:
|
||||
|
||||
- **stdout carries protocol frames only.** Every diagnostic goes to stderr. A
|
||||
stray `Println` corrupts the stream and the host reports an opaque parse error.
|
||||
- **A notification (no `id`) is never answered.** Replying to one desyncs the
|
||||
host, which then attributes the unsolicited response to the next request.
|
||||
178
docs/OPERATIONS.md
Normal file
178
docs/OPERATIONS.md
Normal file
@ -0,0 +1,178 @@
|
||||
# Operating hush
|
||||
|
||||
hush is a stateless Go process in front of TTL'd Redis keys. There is no
|
||||
schema, no queue, no background worker and no durable state of its own, so
|
||||
almost every incident is one of: Redis is unreachable, the pod is not being
|
||||
scraped, or someone is abusing anonymous create.
|
||||
|
||||
## Reading the logs
|
||||
|
||||
```bash
|
||||
make logs # hush, last hour
|
||||
./scripts/logs.sh 'service:hush level:error'
|
||||
./scripts/logs.sh 'service:hush category:secret' # the create/reveal/gone lifecycle
|
||||
./scripts/logs.sh 'service:hush sid:fb26b024452a' # one secret, end to end
|
||||
```
|
||||
|
||||
Vector collects pod stdout cluster-wide with no annotation, so hush's JSON lands
|
||||
in VictoriaLogs automatically. Indexed stream fields are `service`, `level`,
|
||||
`host`, `unit` — everything else (`request_id`, `sid`, `category`, `error_type`)
|
||||
is exact-match queryable and deliberately not indexed.
|
||||
|
||||
`level` is lowercase in the corpus. `{level="ERROR"}` matches nothing.
|
||||
|
||||
### `sid`, and why no id is ever logged
|
||||
|
||||
The secret id is the capability: anyone holding it can reveal the secret. It is
|
||||
never logged. The correlation handle is `sid = sha256(id)[:12]`, which follows
|
||||
one secret across `secret.created` → `secret.revealed` → `secret.gone` and is
|
||||
useless for opening it.
|
||||
|
||||
Verified rather than asserted: creating a secret and searching the whole corpus
|
||||
for its raw id returns zero hits, while its `sid` returns the lifecycle. If you
|
||||
ever see a 43-character base64url string in a hush log line, that is a **P1
|
||||
capability leak** — the id type is built so it cannot happen (see
|
||||
`internal/secret/id.go`) and a regression means someone added a `Value()` call
|
||||
at a log site.
|
||||
|
||||
## Alerts
|
||||
|
||||
Routing (`alertmanager.yaml`): `critical` and `high` reach Discord **and** open a
|
||||
Pantheon incident; `warning` is Discord only.
|
||||
|
||||
### HushRedisUnreachable — critical
|
||||
|
||||
`hush_store_up == 0` for 2m. hush can neither store nor deliver a secret;
|
||||
readiness fails and the pod has left the Service.
|
||||
|
||||
Nothing is lost — Redis owns the secrets and hush fails closed — but the URL is
|
||||
down. In order:
|
||||
|
||||
```bash
|
||||
kubectl -n databases get pod -l app=redis # is Redis up?
|
||||
kubectl -n projects logs -l app=hush --tail=50 # what does hushd say?
|
||||
kubectl -n databases exec redis-0 -- redis-cli --no-auth-warning -a "$PW" ACL LIST | grep hush
|
||||
```
|
||||
|
||||
That last check matters more than it looks: the Redis pod's init container
|
||||
reconciles the `default` ACL user on every start. If a change ever dropped the
|
||||
`hush` user, or dropped `+getdel` from it, the symptom is identical to an
|
||||
outage — and a missing `+getdel` specifically breaks *only* reveal while create
|
||||
keeps succeeding, so the service looks half-healthy.
|
||||
|
||||
### HushMetricsAbsent — high
|
||||
|
||||
`absent(hush_store_up)` for 10m. Every other rule reads a hush metric, so
|
||||
absence silently disables the whole group.
|
||||
|
||||
Most likely cause is not a dead pod but a **dropped scrape target**. vmagent
|
||||
gates on `prometheus.io/scrape=true` AND a `prometheus.io/port` that *equals* a
|
||||
declared `containerPort` — via `keepequal`, which drops a mismatch **silently**:
|
||||
no error, no `up=0`, the target simply never appears.
|
||||
|
||||
```bash
|
||||
kubectl -n projects get pod -l app=hush -o jsonpath='{.items[0].metadata.annotations}'
|
||||
kubectl -n projects get pod -l app=hush -o jsonpath='{.items[0].spec.containers[0].ports}'
|
||||
# the annotation value and the containerPort number must be identical strings
|
||||
```
|
||||
|
||||
Then confirm the NetworkPolicy still admits `observability` on 18500; without
|
||||
it vmagent discovers the target and every scrape is connection-refused.
|
||||
|
||||
### Hush5xxRateHigh — warning
|
||||
|
||||
>5% 5xx over 15m on non-trivial traffic. A 5xx means a secret was
|
||||
accepted-but-not-stored, or a reveal failed **without** destroying the secret.
|
||||
Neither loses data, but a caller who saw a 500 on create does not know whether
|
||||
their link exists. Check `error_type` in the logs.
|
||||
|
||||
### HushRateLimitSustained — warning
|
||||
|
||||
Steady create refusals for 30m. hush is anonymous-create by design, so this is
|
||||
how bulk automation shows up; a single user retrying cannot sustain it, because
|
||||
the limit is per IP.
|
||||
|
||||
If it is abuse rather than a busy NAT:
|
||||
|
||||
```bash
|
||||
kubectl -n projects set env deployment/hush HUSH_REQUIRE_AUTH=true HUSH_CREATE_TOKEN=<token>
|
||||
```
|
||||
|
||||
Reveal stays anonymous either way — the recipient is external and holds no
|
||||
credential. That asymmetry is the design, not an oversight.
|
||||
|
||||
### HushCreateRejectionsHigh — warning
|
||||
|
||||
More than half of creates failing validation. Break down by reason:
|
||||
|
||||
```bash
|
||||
./scripts/logs.sh 'service:hush level:warn'
|
||||
# or in Grafana: hush_secrets_rejected_total by (reason)
|
||||
```
|
||||
|
||||
`ciphertext_invalid` in bulk means a client is posting something that is not
|
||||
base64url — either a broken page deploy or someone treating hush as a plaintext
|
||||
API. `ciphertext_too_large` means someone is trying to use it as a file host.
|
||||
|
||||
## Failure modes that are not alerts
|
||||
|
||||
### A user says "the link says gone" and swears they never opened it
|
||||
|
||||
Three possible causes and hush deliberately cannot tell them apart, because
|
||||
distinguishing them would leak whether a given link was real:
|
||||
|
||||
1. Someone else opened it — **treat the secret as compromised and rotate it.**
|
||||
2. It expired.
|
||||
3. Redis evicted it early (below).
|
||||
|
||||
Assume (1) unless the TTL clearly elapsed. That is the conservative reading and
|
||||
it is cheap: rotating a credential costs less than a leaked one.
|
||||
|
||||
### Redis evicted a secret before its TTL
|
||||
|
||||
The shared Redis runs `maxmemory-policy allkeys-lru` at 256 MiB, so under memory
|
||||
pressure it can drop a hush key **before** its TTL fires.
|
||||
|
||||
This is an **availability** risk and never a confidentiality one: eviction only
|
||||
deletes. A secret can become unavailable early; it can never outlive its TTL and
|
||||
can never be read twice. For a secret courier that is the correct direction to
|
||||
fail, which is why `410 gone` does not distinguish it — the user-visible
|
||||
contract is identical.
|
||||
|
||||
If it starts happening, the fix is upstream (Redis memory, or `volatile-lru`,
|
||||
which is a cluster-wide change affecting every tenant) rather than anything in
|
||||
hush.
|
||||
|
||||
### The pod restarts
|
||||
|
||||
Nothing is lost. All state is in Redis. In-flight requests get the two-phase
|
||||
drain: readiness flips to 503, the load balancer stops sending traffic, then the
|
||||
process shuts down. `terminationGracePeriodSeconds: 45` exceeds the chassis's
|
||||
5s drain + 25s shutdown, so the kubelet does not SIGKILL mid-drain.
|
||||
|
||||
### Someone reports a link that "lost its #"
|
||||
|
||||
Chat and email clients truncate URL fragments. The secret is **intact and
|
||||
unopened** — the fragment never reaches the server, so nothing was consumed. The
|
||||
reveal page and the MCP tool both say this explicitly rather than reporting a
|
||||
generic failure. Ask the sender to re-send the whole link.
|
||||
|
||||
## Routine checks
|
||||
|
||||
```bash
|
||||
make deploy-status # rollout, pods, ingress, certificate
|
||||
BASE=https://hush.threesix.ai make smoke # end-to-end with real crypto
|
||||
make alerts-check # rules loaded AND their series exist
|
||||
```
|
||||
|
||||
`make smoke` creates and burns a real secret against production. It is safe to
|
||||
run any time; it touches nothing but its own secret.
|
||||
|
||||
## What has no runbook because it cannot happen
|
||||
|
||||
- **Reading a stored secret as an operator.** There is no key. `kubectl exec` into
|
||||
Redis and you get ciphertext.
|
||||
- **Restoring a revealed secret.** `GETDEL` is atomic and there is no backup of
|
||||
a value that existed for one read.
|
||||
- **Listing outstanding secrets.** The store contract has no `List` and no
|
||||
`Exists`. `--scan` in Redis yields opaque keys and opaque values.
|
||||
50
scripts/alerts-check.sh
Executable file
50
scripts/alerts-check.sh
Executable file
@ -0,0 +1,50 @@
|
||||
#!/usr/bin/env bash
|
||||
# Confirm vmalert has LOADED hush's rules and can evaluate them.
|
||||
#
|
||||
# Reading the ConfigMap is not verification: vmalert has no auto-discovery, so a
|
||||
# rules file can be applied and committed and still be absent because
|
||||
# victoria-metrics.yaml lacks the matching -rule=, volumeMount and volume. This
|
||||
# asks vmalert what it actually has.
|
||||
#
|
||||
# It also checks each rule's series EXISTS. A rule whose expression references a
|
||||
# metric nothing exports is a rule that can never fire, which reads identically
|
||||
# to a healthy service.
|
||||
set -euo pipefail
|
||||
|
||||
export KUBECONFIG="${KUBECONFIG:-$HOME/.kube/orchard9-k3sf.yaml}"
|
||||
|
||||
kubectl -n observability port-forward svc/vmalert 8880:8880 >/dev/null 2>&1 &
|
||||
PFA=$!
|
||||
kubectl -n observability port-forward svc/vmsingle 8428:8428 >/dev/null 2>&1 &
|
||||
PFS=$!
|
||||
trap 'kill $PFA $PFS 2>/dev/null || true' EXIT
|
||||
sleep 3
|
||||
|
||||
echo "=== rules vmalert has loaded for hush ==="
|
||||
curl -sS http://localhost:8880/api/v1/rules \
|
||||
| python3 -c '
|
||||
import json, sys
|
||||
d = json.load(sys.stdin)
|
||||
groups = [g for g in d["data"]["groups"] if g["name"] == "hush"]
|
||||
if not groups:
|
||||
print(" NO hush group loaded — check the -rule=, volumeMount and volume in victoria-metrics.yaml")
|
||||
sys.exit(1)
|
||||
for g in groups:
|
||||
print(" group {} (file {})".format(g["name"], g.get("file", "?")))
|
||||
for r in g["rules"]:
|
||||
print(" {:32} state={:8} health={} for={}".format(
|
||||
r.get("name", "?"), r.get("state", "?"), r.get("health", "?"), r.get("duration", "?")))
|
||||
if r.get("lastError"):
|
||||
print(" lastError:", r["lastError"])
|
||||
'
|
||||
|
||||
echo
|
||||
echo "=== do the series each rule reads actually exist? ==="
|
||||
for metric in hush_store_up hush_secrets_created_total hush_secrets_revealed_total \
|
||||
hush_secrets_rejected_total hush_rate_limited_total http_requests_total; do
|
||||
n=$(curl -sS -G http://localhost:8428/prometheus/api/v1/query \
|
||||
--data-urlencode "query=count(${metric}{service=\"hush\"}) or count(${metric})" \
|
||||
| python3 -c 'import json,sys; r=json.load(sys.stdin)["data"]["result"]; print(int(float(r[0]["value"][1])) if r else 0)')
|
||||
if [ "$n" -gt 0 ]; then printf ' ok %-32s %s series\n' "$metric" "$n"
|
||||
else printf ' MISSING %-29s 0 series — a rule on this can never fire\n' "$metric"; fi
|
||||
done
|
||||
40
scripts/format-logs.py
Executable file
40
scripts/format-logs.py
Executable file
@ -0,0 +1,40 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Render VictoriaLogs' JSON-lines output as one readable line per entry.
|
||||
|
||||
Kept as a file rather than inlined in logs.sh: quoting a python f-string inside
|
||||
a shell heredoc inside a pipeline is how you get a SyntaxError that only shows
|
||||
up against the live cluster.
|
||||
"""
|
||||
import json
|
||||
import sys
|
||||
|
||||
# Fields Vector or the wire format always sets. They are shown in fixed columns
|
||||
# or are pod plumbing, so they are not repeated in the trailing key=value list.
|
||||
FIXED = {
|
||||
"_time", "_stream", "_stream_id", "_msg", "msg",
|
||||
"level", "service", "env", "host", "unit", "k8s_pod", "k8s_container",
|
||||
}
|
||||
|
||||
rows = []
|
||||
for line in sys.stdin:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
rows.append(json.loads(line))
|
||||
except ValueError:
|
||||
print("unparseable line:", line[:200], file=sys.stderr)
|
||||
|
||||
if not rows:
|
||||
print("no lines matched — has hush served a request in this window?")
|
||||
sys.exit(0)
|
||||
|
||||
# Oldest first, so reading top-to-bottom follows the sequence of events.
|
||||
for r in sorted(rows, key=lambda x: x.get("_time", "")):
|
||||
ts = r.get("_time", "")[:23]
|
||||
level = r.get("level", "")
|
||||
msg = r.get("_msg") or r.get("msg", "")
|
||||
extra = " ".join(f"{k}={v}" for k, v in sorted(r.items()) if k not in FIXED)
|
||||
print("{:24} {:8} {:34} {}".format(ts, level, msg, extra))
|
||||
|
||||
print("\n{} lines".format(len(rows)), file=sys.stderr)
|
||||
79
scripts/install-mcp.sh
Executable file
79
scripts/install-mcp.sh
Executable file
@ -0,0 +1,79 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build hush-mcp and register it with omp.
|
||||
#
|
||||
# The server runs LOCALLY and does the AES-256-GCM itself. That is the point: if
|
||||
# hushd served an MCP endpoint and encrypted server-side, the server could read
|
||||
# every secret created through MCP, and hush's guarantee would hold for browser
|
||||
# users while quietly not holding for agent users. This binary is a peer of the
|
||||
# browser, not of the server.
|
||||
#
|
||||
# Idempotent: re-running rebuilds the binary and rewrites only hush's entry in
|
||||
# ~/.omp/agent/mcp.json, leaving every other server alone.
|
||||
set -euo pipefail
|
||||
|
||||
BIN_DIR="${BIN_DIR:-$HOME/.local/bin}"
|
||||
MCP_JSON="${MCP_JSON:-$HOME/.omp/agent/mcp.json}"
|
||||
BASE_URL="${HUSH_BASE_URL:-https://hush.threesix.ai}"
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
|
||||
echo "building hush-mcp"
|
||||
mkdir -p "$BIN_DIR"
|
||||
(cd "$ROOT" && go build -trimpath -ldflags="-s -w" -o "$BIN_DIR/hush-mcp" ./cmd/hush-mcp)
|
||||
echo " installed $BIN_DIR/hush-mcp"
|
||||
|
||||
# Prove the binary speaks MCP before wiring it in. A config pointing at a broken
|
||||
# server surfaces as an opaque host-side connect failure, so the handshake is
|
||||
# checked here where the error is legible.
|
||||
echo "verifying the MCP handshake"
|
||||
HANDSHAKE=$(printf '%s\n' \
|
||||
'{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{}}}' \
|
||||
'{"jsonrpc":"2.0","id":2,"method":"tools/list"}' \
|
||||
| HUSH_BASE_URL="$BASE_URL" "$BIN_DIR/hush-mcp" 2>/dev/null)
|
||||
echo "$HANDSHAKE" | python3 -c '
|
||||
import json, sys
|
||||
tools = None
|
||||
for line in sys.stdin:
|
||||
m = json.loads(line)
|
||||
if m.get("id") == 1:
|
||||
print(" protocol", m["result"]["protocolVersion"], "server", m["result"]["serverInfo"]["name"])
|
||||
if m.get("id") == 2:
|
||||
tools = [t["name"] for t in m["result"]["tools"]]
|
||||
if not tools:
|
||||
sys.exit(" the server did not answer tools/list")
|
||||
print(" tools:", ", ".join(tools))
|
||||
'
|
||||
|
||||
echo "registering with omp at $MCP_JSON"
|
||||
mkdir -p "$(dirname "$MCP_JSON")"
|
||||
[ -f "$MCP_JSON" ] || printf '{"mcpServers":{}}\n' > "$MCP_JSON"
|
||||
# Back up before touching a config that may hold other servers.
|
||||
cp "$MCP_JSON" "$MCP_JSON.bak.$(date +%Y%m%d%H%M%S)"
|
||||
|
||||
BIN="$BIN_DIR/hush-mcp" BASE="$BASE_URL" TARGET="$MCP_JSON" python3 - <<'PY'
|
||||
import json, os
|
||||
|
||||
target = os.environ["TARGET"]
|
||||
with open(target) as f:
|
||||
cfg = json.load(f)
|
||||
|
||||
cfg.setdefault("$schema",
|
||||
"https://raw.githubusercontent.com/can1357/oh-my-pi/main/packages/coding-agent/src/config/mcp-schema.json")
|
||||
servers = cfg.setdefault("mcpServers", {})
|
||||
|
||||
# stdio, not http: the encryption has to happen on this machine.
|
||||
servers["hush"] = {
|
||||
"type": "stdio",
|
||||
"command": os.environ["BIN"],
|
||||
"env": {"HUSH_BASE_URL": os.environ["BASE"]},
|
||||
"timeout": 20000,
|
||||
}
|
||||
|
||||
with open(target, "w") as f:
|
||||
json.dump(cfg, f, indent=2)
|
||||
f.write("\n")
|
||||
|
||||
print(" servers now configured:", ", ".join(sorted(servers)))
|
||||
PY
|
||||
|
||||
echo
|
||||
echo "done. Restart omp to pick up the new server, then: hush_create / hush_reveal"
|
||||
34
scripts/logs.sh
Executable file
34
scripts/logs.sh
Executable file
@ -0,0 +1,34 @@
|
||||
#!/usr/bin/env bash
|
||||
# hush's structured logs out of VictoriaLogs.
|
||||
#
|
||||
# VictoriaLogs has no read-path ingress (the only ingress, telemetry.threesix.ai,
|
||||
# fronts vmauth-WRITE and is bearer-gated), and its NetworkPolicy admits only
|
||||
# vector, vmauth-write, vmagent and grafana. So a laptop reads it through a
|
||||
# port-forward, which goes node→apiserver→pod and bypasses the pod-to-pod policy.
|
||||
#
|
||||
# Usage:
|
||||
# ./scripts/logs.sh # hush, last hour
|
||||
# ./scripts/logs.sh 'service:hush level:error' # any LogsQL
|
||||
# LIMIT=200 ./scripts/logs.sh 'service:hush category:secret'
|
||||
set -euo pipefail
|
||||
|
||||
export KUBECONFIG="${KUBECONFIG:-$HOME/.kube/orchard9-k3sf.yaml}"
|
||||
PORT="${PORT:-9428}"
|
||||
QUERY="${1:-service:hush _time:1h}"
|
||||
LIMIT="${LIMIT:-50}"
|
||||
|
||||
kubectl -n observability port-forward svc/victoria-logs "$PORT:9428" >/dev/null 2>&1 &
|
||||
PF=$!
|
||||
trap 'kill $PF 2>/dev/null || true' EXIT
|
||||
for _ in $(seq 1 40); do
|
||||
curl -sf -o /dev/null -G "http://localhost:$PORT/select/logsql/query" \
|
||||
--data-urlencode 'query=*' --data-urlencode 'limit=1' && break
|
||||
sleep 0.25
|
||||
done
|
||||
|
||||
curl -sS -G "http://localhost:$PORT/select/logsql/query" \
|
||||
--data-urlencode "query=$QUERY" --data-urlencode "limit=$LIMIT" \
|
||||
> /tmp/hush-logs.$$
|
||||
|
||||
python3 "$(dirname "$0")/format-logs.py" < /tmp/hush-logs.$$
|
||||
rm -f /tmp/hush-logs.$$
|
||||
Loading…
Reference in New Issue
Block a user