e117333fec
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Pipeline #19 finally produced an image once the 60-min repo timeout was raised to 180 (fast-suites passed at 60m26s — it would have died at the old wall for a third time). Pinned by digest in the OVERLAY, never the base, because the k3s fleet applies k8s/cluster/ directly and editing the base pin would silently repoint the fleet's production cluster as a side effect of a GKE change. Applied, and it does not reach Ready. Parked at replicas: 0 rather than deleted, because deleting the namespace destroys the self-signed root and stales every copy of it — the coupling documented one section above. The credentials, both certs and all three PVCs are retained, so resuming is one kubectl scale. Two blockers recorded in dependency order, because fixing them out of order wastes a 75-minute build: 1. The image came from the wrong build path. The runbook is explicit that scripts/build-release.sh must be used and the runtime must be trixie, because the binary needs libmvec.so.1 which is ABSENT on bookworm — and the CI Dockerfile's runtime stage is FROM debian:bookworm-slim. The pods boot on it anyway (WAL recovery, elections, HTTPS listener all fine), so nothing fails loudly; libmvec is vectorized math reached on the vector paths, not at startup. That is exactly why it is written down instead of trusting that it started. 2. A fresh cluster did not converge: all three pods Running with /health/live and /health/startup at 200, but /health 503 for 15 minutes on "shard group N has not yet first-converged against a KNOWN leader frontier". tidaldb-0 saw its own groups at leader_seqno 1 but group 1 at leader_seqno 0 and reseeding true, while both siblings had applied that term marker as followers. Readiness needs the converged flag, which note_lag_for_readiness will not set while leader_seqno == 0 ("absence of lag is not convergence", cluster/node.rs:3840). Blocker 2 is NOT called a bug here and must be re-tested on a correctly built image first — diagnosing a convergence stall against a runtime the runbook rejects is how a wrong root cause gets written down as fact.
223 lines
12 KiB
YAML
223 lines
12 KiB
YAML
# The tidalDB RF3 CLUSTER on GKE, for thepeach's discover post corpus.
|
|
#
|
|
# Same shape as the k3s fleet deployment — ONE StatefulSet, 3 replicas, every
|
|
# pod a region, quorum-acked writes, inter-node mTLS from the cluster's own
|
|
# cert-manager CA. This is an overlay and not a fork: the base owns the
|
|
# topology, and the two things below are the only GKE facts the base cannot
|
|
# know. Precedent is `../cluster-local-kind`, which patches the same field.
|
|
#
|
|
# WHY A CLUSTER AND NOT THE STANDALONE
|
|
# ------------------------------------
|
|
# The standalone was chosen for staging ONLY to work around a real defect:
|
|
# cluster-mode `POST /signals` applied (signal, entity, weight) and silently
|
|
# discarded `user_id`/`creator_id` while still answering 204, so a clustered
|
|
# deployment accepted every behavioural signal and learned nothing. That is
|
|
# fixed at the source in 6ad8c51 (`signal_with_context_staged`), so the
|
|
# workaround is retired rather than inherited.
|
|
#
|
|
# WHY GKE AND NOT THE FLEET
|
|
# -------------------------
|
|
# The consumer is here. `k3s-fleet/deployments/history/tidaldb.md` measured the
|
|
# fleet instance holding 1,720m of CPU requests — 17% of fleet allocatable —
|
|
# while serving NOTHING, with `tidaldb-staging` reporting `items: 0` after 53
|
|
# days, because "the only in-source consumers ... run on GKE". thepeach's api
|
|
# and companion-worker are those consumers, and cluster-local DNS does not
|
|
# cross clusters.
|
|
#
|
|
# ── GKE fact 1: storage class ────────────────────────────────────────────────
|
|
# The base asks for `local-path`, the k3s local-path provisioner. GKE has no
|
|
# such class, so the volumeClaimTemplate would leave every pod Pending
|
|
# indefinitely with no error on the StatefulSet itself. `standard-rwo` is the
|
|
# GKE default (pd.csi.storage.gke.io, WaitForFirstConsumer), which is also the
|
|
# binding mode a StatefulSet wants.
|
|
#
|
|
# ── GKE fact 2: no public ingress ────────────────────────────────────────────
|
|
# The base's `ingress.yaml` publishes `tidaldb.threesix.ai` via Traefik + a
|
|
# Let's Encrypt cert. That hostname is a FLEET name; it has no DNS record here
|
|
# and must not get one. Its own comment says "Remove for internal-only", and
|
|
# internal-only is exactly right: peach reaches this over cluster-local DNS at
|
|
# `tidaldb.tidaldb-cluster.svc.cluster.local:9500`, so nothing needs to be
|
|
# exposed. Deleting the three objects rather than leaving them inert keeps DNS,
|
|
# certificates and load-balancer topology entirely out of this change.
|
|
#
|
|
# The NetworkPolicy is kept verbatim and needs no widening: its `:9500` rule
|
|
# carries no `from` selector, so any namespace may reach the client port. That
|
|
# is deliberate in the base ("`:9500` IS INTENTIONALLY LEFT OPEN") because
|
|
# kubelet probes originate from the node, and it is what lets `thepeach-staging`
|
|
# pods reach this namespace with no cross-namespace rule.
|
|
#
|
|
# ── Preflight, in order ──────────────────────────────────────────────────────
|
|
# 1. Prove nothing public is rendered. This is the check that caught the
|
|
# surviving Ingress; it is cheap and it is the difference between an
|
|
# internal service and a published hostname:
|
|
#
|
|
# kubectl kustomize k8s/cluster-gke-peach | grep -E '^kind:' | sort -u
|
|
#
|
|
# Expect exactly: Certificate, ConfigMap, Issuer, Namespace, NetworkPolicy,
|
|
# PodDisruptionBudget, Service, StatefulSet. Any Ingress, IngressRoute,
|
|
# ServersTransport or Middleware means a delete patch stopped matching —
|
|
# STOP and fix the patch, never apply past it.
|
|
#
|
|
# 2. Create `tidaldb-credentials`. It is NOT in this overlay and never should
|
|
# be — no credential belongs in a manifest. `TIDAL_API_KEY` must be
|
|
# byte-identical to the consumer's `TIDALDB_POSTS_API_KEY` (GSM
|
|
# `thepeach-staging-tidaldb-discover-api-key`) or the bearer check refuses
|
|
# every request; also needs `TIDAL_CLUSTER_KEY`. See
|
|
# `../cluster/secret.example.yaml`.
|
|
#
|
|
# 3. Pin the image digest that carries the signal-context fix. The base's pin
|
|
# predates it, and anything older reintroduces the silent drop this
|
|
# deployment exists to avoid.
|
|
#
|
|
# 4. Server dry-run, which validates admission without creating anything:
|
|
#
|
|
# kubectl apply -k k8s/cluster-gke-peach --dry-run=server
|
|
#
|
|
# Namespaced objects report `NotFound` until the namespace exists; create it
|
|
# first and re-run. Validated clean against GKE 2026-09-16 — all 12 objects,
|
|
# warnings only (the `kubectl create`-vs-`apply` annotation, and
|
|
# cert-manager's rotationPolicy default change).
|
|
#
|
|
# ── After applying ───────────────────────────────────────────────────────────
|
|
# The consumer refuses a cluster instance until its own guard is relaxed:
|
|
# `thepeach/services/api/src/state.rs` disables the client when `/health`
|
|
# reports a mode other than `standalone`. That guard existed BECAUSE of the
|
|
# context-dropping bug; relax it in the same change that repoints the URL, not
|
|
# before, so a stale image cannot be adopted silently.
|
|
#
|
|
# The consumer also needs the CA, because `:9500` serves TLS from the cluster's
|
|
# OWN cert-manager root and no public root can verify it. Kubernetes Secrets do
|
|
# not cross namespaces, so `ca.crt` from `tidaldb-cluster-ca` has to be copied
|
|
# into the consumer's namespace (peach reads it via
|
|
# `TidalClient::from_env_with_ca`).
|
|
#
|
|
# A copied certificate deserves suspicion — k3s-fleet's own history records an
|
|
# incident where "the expired central SDLC wildcard had been copied manually
|
|
# and never renewed into its consumer namespaces". This copy is materially
|
|
# different and the numbers are why:
|
|
#
|
|
# tidaldb-cluster-ca duration 87600h (10y), renewBefore 8760h (1y) isCA
|
|
# tidaldb-cluster-tls duration 2160h (90d), renewBefore 720h (30d)
|
|
#
|
|
# The 90-day LEAF is what rotates, roughly every 60 days, and it rotates UNDER
|
|
# this same root — so the copied CA keeps verifying it and needs no attention.
|
|
# The fleet incident was an expiring leaf; this is a decade-lived root.
|
|
#
|
|
# The coupling that DOES bite is recreation, not expiry: `tidaldb-cluster-ca` is
|
|
# self-signed and generated fresh on first apply, so destroying and re-applying
|
|
# this namespace mints a NEW root and every copy goes stale at once — every
|
|
# consumer request then fails TLS verification with nothing wrong on either
|
|
# side. Treat "refresh the CA copy" as a mandatory step of any recreate, and
|
|
# prefer `kubectl rollout restart` over deleting the namespace.
|
|
#
|
|
# ── STATUS 2026-09-16: applied, then PARKED at replicas: 0 ───────────────────
|
|
# Two blockers, in the order they must be fixed. The namespace, credentials,
|
|
# both certs and all three PVCs are retained, so resuming is `kubectl scale
|
|
# sts/tidaldb --replicas=3`. It was NOT deleted, precisely because deleting the
|
|
# namespace destroys the self-signed root and staleness every copy of it (see
|
|
# above).
|
|
#
|
|
# 1. THE IMAGE CAME FROM THE WRONG BUILD PATH — fix this first.
|
|
# `docs/runbooks/kubernetes.md` §"Deploy the cluster" is explicit: use
|
|
# `./scripts/build-release.sh <tag> server`, NOT a bare docker build, because
|
|
# the binary needs `libmvec.so.1` which is ABSENT on bookworm, so the runtime
|
|
# must be `debian:trixie-slim`. The Woodpecker/Kaniko build uses
|
|
# `docker/standalone/Dockerfile`, whose runtime stage is
|
|
# `FROM debian:bookworm-slim`. The digest pinned above is that CI build.
|
|
# The pods DO boot on it (WAL recovery, elections and the HTTPS listener all
|
|
# work), so nothing fails loudly — libmvec is vectorized math, reached on the
|
|
# vector paths rather than at startup, which is exactly why this is worth
|
|
# writing down instead of trusting "it started".
|
|
# The runbook also requires pinning the linux/amd64 PLATFORM manifest digest,
|
|
# never the OCI index or the `unknown/unknown` attestation manifest; verify
|
|
# with `docker buildx imagetools inspect ... --raw` before pinning.
|
|
#
|
|
# 2. A FRESH CLUSTER DID NOT REACH READY, and this must be re-tested on a
|
|
# correctly built image before being treated as a bug. Measured: all three
|
|
# pods Running and `/health/live` + `/health/startup` 200, but `/health` 503
|
|
# for 15 minutes with
|
|
# "shard group N has not yet first-converged against a KNOWN leader frontier"
|
|
# `/cluster/status` from tidaldb-0 showed its own groups at `leader_seqno: 1`
|
|
# (the term marker) but group 1 at `leader_seqno: 0, reseeding: true`, while
|
|
# tidaldb-1 and tidaldb-2 both logged "leadership view updated (follower)"
|
|
# under leader tidaldb-0 term 1 and applied that term marker. Readiness needs
|
|
# `converged`, which `note_lag_for_readiness` refuses to set while
|
|
# `leader_seqno == 0` ("absence of lag is not convergence",
|
|
# cluster/node.rs:3840).
|
|
# Do not debug this against a bookworm image. Blocker 1 first.
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
|
|
namespace: tidaldb-cluster
|
|
|
|
resources:
|
|
- ../cluster
|
|
|
|
# GKE fact 3: the image. Pinned HERE and not in the base, because the k3s fleet
|
|
# applies `k8s/cluster/` directly — editing the base pin would silently repoint
|
|
# the fleet's production cluster as a side effect of a GKE change. Same
|
|
# isolation reason `cluster-local-kind` retargets here.
|
|
#
|
|
# By DIGEST, not tag: this is the first build carrying the signal-context fix
|
|
# (tidalDB 6ad8c51, image created 2026-09-16T04:28:15Z). The base's pin is
|
|
# m12-agesort-20260901 / sha256:9234aacb…, created 2026-09-01 — two weeks
|
|
# BEFORE the fix — so deploying the base unmodified would run the binary that
|
|
# silently discards user_id/creator_id behind a 204, which is the whole reason
|
|
# this deployment exists.
|
|
images:
|
|
- name: registry.threesix.ai/tidal/server
|
|
digest: sha256:96696a662c3a3c1bcf2a5c9bdcd0455cd77ca8d876f2d5fa7cacccee137d780f
|
|
|
|
patches:
|
|
# GKE fact 1 — see header.
|
|
- target:
|
|
kind: StatefulSet
|
|
name: tidaldb
|
|
patch: |-
|
|
- op: replace
|
|
path: /spec/volumeClaimTemplates/0/spec/storageClassName
|
|
value: standard-rwo
|
|
|
|
# GKE fact 2 — drop the fleet-only public exposure. Deleted by exact kind and
|
|
# name so a base change that adds a fourth object fails loudly here instead
|
|
# of quietly publishing it.
|
|
#
|
|
# The public object is a plain `networking.k8s.io/v1` Ingress carrying Traefik
|
|
# ANNOTATIONS — NOT a `traefik.io` IngressRoute. An earlier version of this
|
|
# file targeted IngressRoute, which matched nothing: kustomize rendered
|
|
# happily and the Ingress survived, so `tidaldb.threesix.ai` would have been
|
|
# published from GKE. Rendering the overlay and reading the object inventory
|
|
# is what caught it, which is why that check is a runbook step below and not
|
|
# left to someone's memory.
|
|
- target:
|
|
group: networking.k8s.io
|
|
version: v1
|
|
kind: Ingress
|
|
name: tidaldb
|
|
patch: |-
|
|
$patch: delete
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: Ingress
|
|
metadata:
|
|
name: tidaldb
|
|
- target:
|
|
group: traefik.io
|
|
kind: ServersTransport
|
|
name: tidaldb-internal
|
|
patch: |-
|
|
$patch: delete
|
|
apiVersion: traefik.io/v1alpha1
|
|
kind: ServersTransport
|
|
metadata:
|
|
name: tidaldb-internal
|
|
- target:
|
|
group: traefik.io
|
|
kind: Middleware
|
|
name: tidaldb-ratelimit
|
|
patch: |-
|
|
$patch: delete
|
|
apiVersion: traefik.io/v1alpha1
|
|
kind: Middleware
|
|
metadata:
|
|
name: tidaldb-ratelimit
|