tidaldb/k8s
jordan e117333fec
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
ops(k8s): pin the fixed image, then park the GKE cluster at 0 with both blockers recorded
Pipeline #19 finally produced an image once the 60-min repo timeout was raised
to 180 (fast-suites passed at 60m26s — it would have died at the old wall for
a third time). Pinned by digest in the OVERLAY, never the base, because the
k3s fleet applies k8s/cluster/ directly and editing the base pin would
silently repoint the fleet's production cluster as a side effect of a GKE
change.

Applied, and it does not reach Ready. Parked at replicas: 0 rather than
deleted, because deleting the namespace destroys the self-signed root and
stales every copy of it — the coupling documented one section above. The
credentials, both certs and all three PVCs are retained, so resuming is one
kubectl scale.

Two blockers recorded in dependency order, because fixing them out of order
wastes a 75-minute build:

1. The image came from the wrong build path. The runbook is explicit that
   scripts/build-release.sh must be used and the runtime must be trixie,
   because the binary needs libmvec.so.1 which is ABSENT on bookworm — and the
   CI Dockerfile's runtime stage is FROM debian:bookworm-slim. The pods boot on
   it anyway (WAL recovery, elections, HTTPS listener all fine), so nothing
   fails loudly; libmvec is vectorized math reached on the vector paths, not at
   startup. That is exactly why it is written down instead of trusting that it
   started.

2. A fresh cluster did not converge: all three pods Running with /health/live
   and /health/startup at 200, but /health 503 for 15 minutes on "shard group N
   has not yet first-converged against a KNOWN leader frontier". tidaldb-0 saw
   its own groups at leader_seqno 1 but group 1 at leader_seqno 0 and
   reseeding true, while both siblings had applied that term marker as
   followers. Readiness needs the converged flag, which
   note_lag_for_readiness will not set while leader_seqno == 0 ("absence of
   lag is not convergence", cluster/node.rs:3840).

Blocker 2 is NOT called a bug here and must be re-tested on a correctly built
image first — diagnosing a convergence stall against a runtime the runbook
rejects is how a wrong root cause gets written down as fact.
2026-09-15 22:43:25 -06:00
..
cluster fix(k8s): reconcile tidaldb StatefulSet resources with live cluster (300m/4Gi requests, 3/7Gi limits) 2026-09-03 05:16:27 -06:00
cluster-gke-peach ops(k8s): pin the fixed image, then park the GKE cluster at 0 with both blockers recorded 2026-09-15 22:43:25 -06:00
cluster-local-kind feat(m12p4): sharded ingestion — scatter-gather pool + cross-shard unified reads (L4) 2026-06-14 15:17:35 -06:00
cluster-t4-kind fix(m12p6): complete T4 TLS scale-up — two-tier PKI + join_boot grpc_tls fallback 2026-06-14 22:41:59 -06:00
discover discover: repin to m12-agesort-20260901 2026-08-31 23:12:08 -06:00
kustomization.yaml feat: kubernetes deployment, OpenAPI spec, guides, and docker consolidation 2026-06-09 17:06:34 -06:00
namespace.yaml feat: kubernetes deployment, OpenAPI spec, guides, and docker consolidation 2026-06-09 17:06:34 -06:00
poddisruptionbudget.yaml feat: kubernetes deployment, OpenAPI spec, guides, and docker consolidation 2026-06-09 17:06:34 -06:00
schema-configmap.yaml feat: kubernetes deployment, OpenAPI spec, guides, and docker consolidation 2026-06-09 17:06:34 -06:00
secret.example.yaml feat: kubernetes deployment, OpenAPI spec, guides, and docker consolidation 2026-06-09 17:06:34 -06:00
service.yaml feat: kubernetes deployment, OpenAPI spec, guides, and docker consolidation 2026-06-09 17:06:34 -06:00
servicemonitor.yaml feat: kubernetes deployment, OpenAPI spec, guides, and docker consolidation 2026-06-09 17:06:34 -06:00
statefulset.yaml fleet remediation: make the workspace gate runnable, then fix what it caught 2026-08-16 12:38:14 -06:00