# The tidalDB RF3 CLUSTER on GKE, for thepeach's discover post corpus. # # Same shape as the k3s fleet deployment — ONE StatefulSet, 3 replicas, every # pod a region, quorum-acked writes, inter-node mTLS from the cluster's own # cert-manager CA. This is an overlay and not a fork: the base owns the # topology, and the two things below are the only GKE facts the base cannot # know. Precedent is `../cluster-local-kind`, which patches the same field. # # WHY A CLUSTER AND NOT THE STANDALONE # ------------------------------------ # The standalone was chosen for staging ONLY to work around a real defect: # cluster-mode `POST /signals` applied (signal, entity, weight) and silently # discarded `user_id`/`creator_id` while still answering 204, so a clustered # deployment accepted every behavioural signal and learned nothing. That is # fixed at the source in 6ad8c51 (`signal_with_context_staged`), so the # workaround is retired rather than inherited. # # WHY GKE AND NOT THE FLEET # ------------------------- # The consumer is here. `k3s-fleet/deployments/history/tidaldb.md` measured the # fleet instance holding 1,720m of CPU requests — 17% of fleet allocatable — # while serving NOTHING, with `tidaldb-staging` reporting `items: 0` after 53 # days, because "the only in-source consumers ... run on GKE". thepeach's api # and companion-worker are those consumers, and cluster-local DNS does not # cross clusters. # # ── GKE fact 1: storage class ──────────────────────────────────────────────── # The base asks for `local-path`, the k3s local-path provisioner. GKE has no # such class, so the volumeClaimTemplate would leave every pod Pending # indefinitely with no error on the StatefulSet itself. `standard-rwo` is the # GKE default (pd.csi.storage.gke.io, WaitForFirstConsumer), which is also the # binding mode a StatefulSet wants. # # ── GKE fact 2: no public ingress ──────────────────────────────────────────── # The base's `ingress.yaml` publishes `tidaldb.threesix.ai` via Traefik + a # Let's Encrypt cert. That hostname is a FLEET name; it has no DNS record here # and must not get one. Its own comment says "Remove for internal-only", and # internal-only is exactly right: peach reaches this over cluster-local DNS at # `tidaldb.tidaldb-cluster.svc.cluster.local:9500`, so nothing needs to be # exposed. Deleting the three objects rather than leaving them inert keeps DNS, # certificates and load-balancer topology entirely out of this change. # # The NetworkPolicy is kept verbatim and needs no widening: its `:9500` rule # carries no `from` selector, so any namespace may reach the client port. That # is deliberate in the base ("`:9500` IS INTENTIONALLY LEFT OPEN") because # kubelet probes originate from the node, and it is what lets `thepeach-staging` # pods reach this namespace with no cross-namespace rule. # # ── Preflight, in order ────────────────────────────────────────────────────── # 1. Prove nothing public is rendered. This is the check that caught the # surviving Ingress; it is cheap and it is the difference between an # internal service and a published hostname: # # kubectl kustomize k8s/cluster-gke-peach | grep -E '^kind:' | sort -u # # Expect exactly: Certificate, ConfigMap, Issuer, Namespace, NetworkPolicy, # PodDisruptionBudget, Service, StatefulSet. Any Ingress, IngressRoute, # ServersTransport or Middleware means a delete patch stopped matching — # STOP and fix the patch, never apply past it. # # 2. Create `tidaldb-credentials`. It is NOT in this overlay and never should # be — no credential belongs in a manifest. `TIDAL_API_KEY` must be # byte-identical to the consumer's `TIDALDB_POSTS_API_KEY` (GSM # `thepeach-staging-tidaldb-discover-api-key`) or the bearer check refuses # every request; also needs `TIDAL_CLUSTER_KEY`. See # `../cluster/secret.example.yaml`. # # 3. Pin the image digest that carries the signal-context fix. The base's pin # predates it, and anything older reintroduces the silent drop this # deployment exists to avoid. # # 4. Server dry-run, which validates admission without creating anything: # # kubectl apply -k k8s/cluster-gke-peach --dry-run=server # # Namespaced objects report `NotFound` until the namespace exists; create it # first and re-run. Validated clean against GKE 2026-09-16 — all 12 objects, # warnings only (the `kubectl create`-vs-`apply` annotation, and # cert-manager's rotationPolicy default change). # # ── After applying ─────────────────────────────────────────────────────────── # The consumer refuses a cluster instance until its own guard is relaxed: # `thepeach/services/api/src/state.rs` disables the client when `/health` # reports a mode other than `standalone`. That guard existed BECAUSE of the # context-dropping bug; relax it in the same change that repoints the URL, not # before, so a stale image cannot be adopted silently. # # The consumer also needs the CA, because `:9500` serves TLS from the cluster's # OWN cert-manager root and no public root can verify it. Kubernetes Secrets do # not cross namespaces, so `ca.crt` from `tidaldb-cluster-ca` has to be copied # into the consumer's namespace (peach reads it via # `TidalClient::from_env_with_ca`). # # A copied certificate deserves suspicion — k3s-fleet's own history records an # incident where "the expired central SDLC wildcard had been copied manually # and never renewed into its consumer namespaces". This copy is materially # different and the numbers are why: # # tidaldb-cluster-ca duration 87600h (10y), renewBefore 8760h (1y) isCA # tidaldb-cluster-tls duration 2160h (90d), renewBefore 720h (30d) # # The 90-day LEAF is what rotates, roughly every 60 days, and it rotates UNDER # this same root — so the copied CA keeps verifying it and needs no attention. # The fleet incident was an expiring leaf; this is a decade-lived root. # # The coupling that DOES bite is recreation, not expiry: `tidaldb-cluster-ca` is # self-signed and generated fresh on first apply, so destroying and re-applying # this namespace mints a NEW root and every copy goes stale at once — every # consumer request then fails TLS verification with nothing wrong on either # side. Treat "refresh the CA copy" as a mandatory step of any recreate, and # prefer `kubectl rollout restart` over deleting the namespace. # # ── STATUS 2026-09-16: applied, then PARKED at replicas: 0 ─────────────────── # Two blockers, in the order they must be fixed. The namespace, credentials, # both certs and all three PVCs are retained, so resuming is `kubectl scale # sts/tidaldb --replicas=3`. It was NOT deleted, precisely because deleting the # namespace destroys the self-signed root and staleness every copy of it (see # above). # # 1. THE IMAGE CAME FROM THE WRONG BUILD PATH — fix this first. # `docs/runbooks/kubernetes.md` §"Deploy the cluster" is explicit: use # `./scripts/build-release.sh server`, NOT a bare docker build, because # the binary needs `libmvec.so.1` which is ABSENT on bookworm, so the runtime # must be `debian:trixie-slim`. The Woodpecker/Kaniko build uses # `docker/standalone/Dockerfile`, whose runtime stage is # `FROM debian:bookworm-slim`. The digest pinned above is that CI build. # The pods DO boot on it (WAL recovery, elections and the HTTPS listener all # work), so nothing fails loudly — libmvec is vectorized math, reached on the # vector paths rather than at startup, which is exactly why this is worth # writing down instead of trusting "it started". # The runbook also requires pinning the linux/amd64 PLATFORM manifest digest, # never the OCI index or the `unknown/unknown` attestation manifest; verify # with `docker buildx imagetools inspect ... --raw` before pinning. # # 2. A FRESH CLUSTER DID NOT REACH READY, and this must be re-tested on a # correctly built image before being treated as a bug. Measured: all three # pods Running and `/health/live` + `/health/startup` 200, but `/health` 503 # for 15 minutes with # "shard group N has not yet first-converged against a KNOWN leader frontier" # `/cluster/status` from tidaldb-0 showed its own groups at `leader_seqno: 1` # (the term marker) but group 1 at `leader_seqno: 0, reseeding: true`, while # tidaldb-1 and tidaldb-2 both logged "leadership view updated (follower)" # under leader tidaldb-0 term 1 and applied that term marker. Readiness needs # `converged`, which `note_lag_for_readiness` refuses to set while # `leader_seqno == 0` ("absence of lag is not convergence", # cluster/node.rs:3840). # Do not debug this against a bookworm image. Blocker 1 first. apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization namespace: tidaldb-cluster resources: - ../cluster # GKE fact 3: the image. Pinned HERE and not in the base, because the k3s fleet # applies `k8s/cluster/` directly — editing the base pin would silently repoint # the fleet's production cluster as a side effect of a GKE change. Same # isolation reason `cluster-local-kind` retargets here. # # By DIGEST, not tag: this is the first build carrying the signal-context fix # (tidalDB 6ad8c51, image created 2026-09-16T04:28:15Z). The base's pin is # m12-agesort-20260901 / sha256:9234aacb…, created 2026-09-01 — two weeks # BEFORE the fix — so deploying the base unmodified would run the binary that # silently discards user_id/creator_id behind a 204, which is the whole reason # this deployment exists. images: - name: registry.threesix.ai/tidal/server digest: sha256:96696a662c3a3c1bcf2a5c9bdcd0455cd77ca8d876f2d5fa7cacccee137d780f patches: # GKE fact 1 — see header. - target: kind: StatefulSet name: tidaldb patch: |- - op: replace path: /spec/volumeClaimTemplates/0/spec/storageClassName value: standard-rwo # GKE fact 2 — drop the fleet-only public exposure. Deleted by exact kind and # name so a base change that adds a fourth object fails loudly here instead # of quietly publishing it. # # The public object is a plain `networking.k8s.io/v1` Ingress carrying Traefik # ANNOTATIONS — NOT a `traefik.io` IngressRoute. An earlier version of this # file targeted IngressRoute, which matched nothing: kustomize rendered # happily and the Ingress survived, so `tidaldb.threesix.ai` would have been # published from GKE. Rendering the overlay and reading the object inventory # is what caught it, which is why that check is a runbook step below and not # left to someone's memory. - target: group: networking.k8s.io version: v1 kind: Ingress name: tidaldb patch: |- $patch: delete apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: tidaldb - target: group: traefik.io kind: ServersTransport name: tidaldb-internal patch: |- $patch: delete apiVersion: traefik.io/v1alpha1 kind: ServersTransport metadata: name: tidaldb-internal - target: group: traefik.io kind: Middleware name: tidaldb-ratelimit patch: |- $patch: delete apiVersion: traefik.io/v1alpha1 kind: Middleware metadata: name: tidaldb-ratelimit