62c95f8c2f
Written after the service was live, so every command and every number here was
run against the real deployment rather than assumed:
* DEPLOY.md records the things a rebuild needs and git does not hold — the
Redis ACL user (and why +getdel is the one to notice), the GCP Secret
Manager entry, the DNS record, and the three coordinated edits vmalert
needs because it has no ConfigMap auto-discovery.
* It also records two blockers rather than hiding them: Woodpecker is NOT
activated (the token in rdev-credentials returns 401), so pushes do not
deploy yet and the Kaniko Job is the interim path; and the host is
hush.threesix.ai rather than hush.orchard9.ai because orchard9.ai is on
GoDaddy and no GoDaddy credential exists anywhere I can reach.
* OPERATIONS.md is one section per alert, plus the failure modes that are not
alerts — chiefly that "gone" cannot distinguish already-revealed from
expired from LRU-evicted, on purpose, so the operator's default reading of
an unexpected "gone" is that the secret is compromised and should be
rotated.
* scripts/logs.sh and alerts-check.sh verify rather than assert:
alerts-check asks vmalert what it actually loaded AND checks each rule's
series exists, because a rule reading a metric nothing exports can never
fire and looks exactly like a healthy service.
* scripts/smoke.sh is a real client — it generates a key, encrypts, posts only
ciphertext, reveals, decrypts, then asserts the second reveal is 410, that
three GETs did not consume the secret, that missing and malformed ids are
indistinguishable, and that a plaintext field is refused.
install-mcp.sh proves the MCP handshake before writing any config, backs up
mcp.json, and rewrites only hush's entry — a config pointing at a broken server
surfaces as an opaque host-side connect failure, which is worth one extra check
to avoid.
35 lines
1.3 KiB
Bash
Executable File
35 lines
1.3 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# hush's structured logs out of VictoriaLogs.
|
|
#
|
|
# VictoriaLogs has no read-path ingress (the only ingress, telemetry.threesix.ai,
|
|
# fronts vmauth-WRITE and is bearer-gated), and its NetworkPolicy admits only
|
|
# vector, vmauth-write, vmagent and grafana. So a laptop reads it through a
|
|
# port-forward, which goes node→apiserver→pod and bypasses the pod-to-pod policy.
|
|
#
|
|
# Usage:
|
|
# ./scripts/logs.sh # hush, last hour
|
|
# ./scripts/logs.sh 'service:hush level:error' # any LogsQL
|
|
# LIMIT=200 ./scripts/logs.sh 'service:hush category:secret'
|
|
set -euo pipefail
|
|
|
|
export KUBECONFIG="${KUBECONFIG:-$HOME/.kube/orchard9-k3sf.yaml}"
|
|
PORT="${PORT:-9428}"
|
|
QUERY="${1:-service:hush _time:1h}"
|
|
LIMIT="${LIMIT:-50}"
|
|
|
|
kubectl -n observability port-forward svc/victoria-logs "$PORT:9428" >/dev/null 2>&1 &
|
|
PF=$!
|
|
trap 'kill $PF 2>/dev/null || true' EXIT
|
|
for _ in $(seq 1 40); do
|
|
curl -sf -o /dev/null -G "http://localhost:$PORT/select/logsql/query" \
|
|
--data-urlencode 'query=*' --data-urlencode 'limit=1' && break
|
|
sleep 0.25
|
|
done
|
|
|
|
curl -sS -G "http://localhost:$PORT/select/logsql/query" \
|
|
--data-urlencode "query=$QUERY" --data-urlencode "limit=$LIMIT" \
|
|
> /tmp/hush-logs.$$
|
|
|
|
python3 "$(dirname "$0")/format-logs.py" < /tmp/hush-logs.$$
|
|
rm -f /tmp/hush-logs.$$
|