67a175e19a
3 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
jordan
|
e117333fec |
ops(k8s): pin the fixed image, then park the GKE cluster at 0 with both blockers recorded
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Pipeline #19 finally produced an image once the 60-min repo timeout was raised to 180 (fast-suites passed at 60m26s — it would have died at the old wall for a third time). Pinned by digest in the OVERLAY, never the base, because the k3s fleet applies k8s/cluster/ directly and editing the base pin would silently repoint the fleet's production cluster as a side effect of a GKE change. Applied, and it does not reach Ready. Parked at replicas: 0 rather than deleted, because deleting the namespace destroys the self-signed root and stales every copy of it — the coupling documented one section above. The credentials, both certs and all three PVCs are retained, so resuming is one kubectl scale. Two blockers recorded in dependency order, because fixing them out of order wastes a 75-minute build: 1. The image came from the wrong build path. The runbook is explicit that scripts/build-release.sh must be used and the runtime must be trixie, because the binary needs libmvec.so.1 which is ABSENT on bookworm — and the CI Dockerfile's runtime stage is FROM debian:bookworm-slim. The pods boot on it anyway (WAL recovery, elections, HTTPS listener all fine), so nothing fails loudly; libmvec is vectorized math reached on the vector paths, not at startup. That is exactly why it is written down instead of trusting that it started. 2. A fresh cluster did not converge: all three pods Running with /health/live and /health/startup at 200, but /health 503 for 15 minutes on "shard group N has not yet first-converged against a KNOWN leader frontier". tidaldb-0 saw its own groups at leader_seqno 1 but group 1 at leader_seqno 0 and reseeding true, while both siblings had applied that term marker as followers. Readiness needs the converged flag, which note_lag_for_readiness will not set while leader_seqno == 0 ("absence of lag is not convergence", cluster/node.rs:3840). Blocker 2 is NOT called a bug here and must be re-tested on a correctly built image first — diagnosing a convergence stall against a runtime the runbook rejects is how a wrong root cause gets written down as fact. |
||
|
jordan
|
2c25b6d375 |
docs(k8s): record the CA-copy coupling for the GKE overlay
The consumer needs the cluster's CA and Secrets do not cross namespaces, so ca.crt has to be copied into peach's namespace. A copied certificate deserves suspicion — k3s-fleet's own history records an incident where the expired central SDLC wildcard was copied manually and never renewed into its consumer namespaces — so the numbers are written down rather than assumed: tidaldb-cluster-ca 87600h (10y), renewBefore 8760h (1y), isCA tidaldb-cluster-tls 2160h (90d), renewBefore 720h (30d) The 90-day LEAF is what rotates, roughly every 60 days, and it rotates under this same root, so the copied CA keeps verifying it with no attention. The fleet incident was an expiring leaf; this is a decade-lived root. The coupling that actually bites is recreation, not expiry: the root is self-signed and minted fresh on first apply, so destroying and re-applying the namespace invalidates every copy at once and every consumer request then fails TLS with nothing visibly wrong on either side. Recorded as a mandatory step of any recreate. |
||
|
jordan
|
19316227c5 |
feat(k8s): GKE overlay for the RF3 cluster, internal-only
thepeach's discover corpus should run the CLUSTER, not the standalone, and it
should run on GKE. Both halves are now evidenced rather than assumed.
Cluster, because the reason for the standalone is gone. The standalone was
chosen to work around cluster-mode `POST /signals` silently discarding
user_id/creator_id behind a 204 — fixed at the source in
|