Compare commits

..

2 Commits

Author SHA1 Message Date
jordan
e117333fec ops(k8s): pin the fixed image, then park the GKE cluster at 0 with both blockers recorded
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Pipeline #19 finally produced an image once the 60-min repo timeout was raised
to 180 (fast-suites passed at 60m26s — it would have died at the old wall for
a third time). Pinned by digest in the OVERLAY, never the base, because the
k3s fleet applies k8s/cluster/ directly and editing the base pin would
silently repoint the fleet's production cluster as a side effect of a GKE
change.

Applied, and it does not reach Ready. Parked at replicas: 0 rather than
deleted, because deleting the namespace destroys the self-signed root and
stales every copy of it — the coupling documented one section above. The
credentials, both certs and all three PVCs are retained, so resuming is one
kubectl scale.

Two blockers recorded in dependency order, because fixing them out of order
wastes a 75-minute build:

1. The image came from the wrong build path. The runbook is explicit that
   scripts/build-release.sh must be used and the runtime must be trixie,
   because the binary needs libmvec.so.1 which is ABSENT on bookworm — and the
   CI Dockerfile's runtime stage is FROM debian:bookworm-slim. The pods boot on
   it anyway (WAL recovery, elections, HTTPS listener all fine), so nothing
   fails loudly; libmvec is vectorized math reached on the vector paths, not at
   startup. That is exactly why it is written down instead of trusting that it
   started.

2. A fresh cluster did not converge: all three pods Running with /health/live
   and /health/startup at 200, but /health 503 for 15 minutes on "shard group N
   has not yet first-converged against a KNOWN leader frontier". tidaldb-0 saw
   its own groups at leader_seqno 1 but group 1 at leader_seqno 0 and
   reseeding true, while both siblings had applied that term marker as
   followers. Readiness needs the converged flag, which
   note_lag_for_readiness will not set while leader_seqno == 0 ("absence of
   lag is not convergence", cluster/node.rs:3840).

Blocker 2 is NOT called a bug here and must be re-tested on a correctly built
image first — diagnosing a convergence stall against a runtime the runbook
rejects is how a wrong root cause gets written down as fact.
2026-09-15 22:43:25 -06:00
jordan
2c25b6d375 docs(k8s): record the CA-copy coupling for the GKE overlay
The consumer needs the cluster's CA and Secrets do not cross namespaces, so
ca.crt has to be copied into peach's namespace. A copied certificate deserves
suspicion — k3s-fleet's own history records an incident where the expired
central SDLC wildcard was copied manually and never renewed into its consumer
namespaces — so the numbers are written down rather than assumed:

  tidaldb-cluster-ca   87600h (10y), renewBefore 8760h (1y), isCA
  tidaldb-cluster-tls   2160h (90d), renewBefore  720h (30d)

The 90-day LEAF is what rotates, roughly every 60 days, and it rotates under
this same root, so the copied CA keeps verifying it with no attention. The
fleet incident was an expiring leaf; this is a decade-lived root.

The coupling that actually bites is recreation, not expiry: the root is
self-signed and minted fresh on first apply, so destroying and re-applying the
namespace invalidates every copy at once and every consumer request then fails
TLS with nothing visibly wrong on either side. Recorded as a mandatory step of
any recreate.
2026-09-15 22:13:30 -06:00

View File

@ -84,6 +84,67 @@
# reports a mode other than `standalone`. That guard existed BECAUSE of the
# context-dropping bug; relax it in the same change that repoints the URL, not
# before, so a stale image cannot be adopted silently.
#
# The consumer also needs the CA, because `:9500` serves TLS from the cluster's
# OWN cert-manager root and no public root can verify it. Kubernetes Secrets do
# not cross namespaces, so `ca.crt` from `tidaldb-cluster-ca` has to be copied
# into the consumer's namespace (peach reads it via
# `TidalClient::from_env_with_ca`).
#
# A copied certificate deserves suspicion — k3s-fleet's own history records an
# incident where "the expired central SDLC wildcard had been copied manually
# and never renewed into its consumer namespaces". This copy is materially
# different and the numbers are why:
#
# tidaldb-cluster-ca duration 87600h (10y), renewBefore 8760h (1y) isCA
# tidaldb-cluster-tls duration 2160h (90d), renewBefore 720h (30d)
#
# The 90-day LEAF is what rotates, roughly every 60 days, and it rotates UNDER
# this same root — so the copied CA keeps verifying it and needs no attention.
# The fleet incident was an expiring leaf; this is a decade-lived root.
#
# The coupling that DOES bite is recreation, not expiry: `tidaldb-cluster-ca` is
# self-signed and generated fresh on first apply, so destroying and re-applying
# this namespace mints a NEW root and every copy goes stale at once — every
# consumer request then fails TLS verification with nothing wrong on either
# side. Treat "refresh the CA copy" as a mandatory step of any recreate, and
# prefer `kubectl rollout restart` over deleting the namespace.
#
# ── STATUS 2026-09-16: applied, then PARKED at replicas: 0 ───────────────────
# Two blockers, in the order they must be fixed. The namespace, credentials,
# both certs and all three PVCs are retained, so resuming is `kubectl scale
# sts/tidaldb --replicas=3`. It was NOT deleted, precisely because deleting the
# namespace destroys the self-signed root and staleness every copy of it (see
# above).
#
# 1. THE IMAGE CAME FROM THE WRONG BUILD PATH — fix this first.
# `docs/runbooks/kubernetes.md` §"Deploy the cluster" is explicit: use
# `./scripts/build-release.sh <tag> server`, NOT a bare docker build, because
# the binary needs `libmvec.so.1` which is ABSENT on bookworm, so the runtime
# must be `debian:trixie-slim`. The Woodpecker/Kaniko build uses
# `docker/standalone/Dockerfile`, whose runtime stage is
# `FROM debian:bookworm-slim`. The digest pinned above is that CI build.
# The pods DO boot on it (WAL recovery, elections and the HTTPS listener all
# work), so nothing fails loudly — libmvec is vectorized math, reached on the
# vector paths rather than at startup, which is exactly why this is worth
# writing down instead of trusting "it started".
# The runbook also requires pinning the linux/amd64 PLATFORM manifest digest,
# never the OCI index or the `unknown/unknown` attestation manifest; verify
# with `docker buildx imagetools inspect ... --raw` before pinning.
#
# 2. A FRESH CLUSTER DID NOT REACH READY, and this must be re-tested on a
# correctly built image before being treated as a bug. Measured: all three
# pods Running and `/health/live` + `/health/startup` 200, but `/health` 503
# for 15 minutes with
# "shard group N has not yet first-converged against a KNOWN leader frontier"
# `/cluster/status` from tidaldb-0 showed its own groups at `leader_seqno: 1`
# (the term marker) but group 1 at `leader_seqno: 0, reseeding: true`, while
# tidaldb-1 and tidaldb-2 both logged "leadership view updated (follower)"
# under leader tidaldb-0 term 1 and applied that term marker. Readiness needs
# `converged`, which `note_lag_for_readiness` refuses to set while
# `leader_seqno == 0` ("absence of lag is not convergence",
# cluster/node.rs:3840).
# Do not debug this against a bookworm image. Blocker 1 first.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
@ -92,6 +153,21 @@ namespace: tidaldb-cluster
resources:
- ../cluster
# GKE fact 3: the image. Pinned HERE and not in the base, because the k3s fleet
# applies `k8s/cluster/` directly — editing the base pin would silently repoint
# the fleet's production cluster as a side effect of a GKE change. Same
# isolation reason `cluster-local-kind` retargets here.
#
# By DIGEST, not tag: this is the first build carrying the signal-context fix
# (tidalDB 6ad8c51, image created 2026-09-16T04:28:15Z). The base's pin is
# m12-agesort-20260901 / sha256:9234aacb…, created 2026-09-01 — two weeks
# BEFORE the fix — so deploying the base unmodified would run the binary that
# silently discards user_id/creator_id behind a 204, which is the whole reason
# this deployment exists.
images:
- name: registry.threesix.ai/tidal/server
digest: sha256:96696a662c3a3c1bcf2a5c9bdcd0455cd77ca8d876f2d5fa7cacccee137d780f
patches:
# GKE fact 1 — see header.
- target: