# The tidalDB RF3 CLUSTER on GKE, for thepeach's discover post corpus. # # Same shape as the k3s fleet deployment — ONE StatefulSet, 3 replicas, every # pod a region, quorum-acked writes, inter-node mTLS from the cluster's own # cert-manager CA. This is an overlay and not a fork: the base owns the # topology, and the two things below are the only GKE facts the base cannot # know. Precedent is `../cluster-local-kind`, which patches the same field. # # WHY A CLUSTER AND NOT THE STANDALONE # ------------------------------------ # The standalone was chosen for staging ONLY to work around a real defect: # cluster-mode `POST /signals` applied (signal, entity, weight) and silently # discarded `user_id`/`creator_id` while still answering 204, so a clustered # deployment accepted every behavioural signal and learned nothing. That is # fixed at the source in 6ad8c51 (`signal_with_context_staged`), so the # workaround is retired rather than inherited. # # WHY GKE AND NOT THE FLEET # ------------------------- # The consumer is here. `k3s-fleet/deployments/history/tidaldb.md` measured the # fleet instance holding 1,720m of CPU requests — 17% of fleet allocatable — # while serving NOTHING, with `tidaldb-staging` reporting `items: 0` after 53 # days, because "the only in-source consumers ... run on GKE". thepeach's api # and companion-worker are those consumers, and cluster-local DNS does not # cross clusters. # # ── GKE fact 1: storage class ──────────────────────────────────────────────── # The base asks for `local-path`, the k3s local-path provisioner. GKE has no # such class, so the volumeClaimTemplate would leave every pod Pending # indefinitely with no error on the StatefulSet itself. `standard-rwo` is the # GKE default (pd.csi.storage.gke.io, WaitForFirstConsumer), which is also the # binding mode a StatefulSet wants. # # ── GKE fact 2: no public ingress ──────────────────────────────────────────── # The base's `ingress.yaml` publishes `tidaldb.threesix.ai` via Traefik + a # Let's Encrypt cert. That hostname is a FLEET name; it has no DNS record here # and must not get one. Its own comment says "Remove for internal-only", and # internal-only is exactly right: peach reaches this over cluster-local DNS at # `tidaldb.tidaldb-cluster.svc.cluster.local:9500`, so nothing needs to be # exposed. Deleting the three objects rather than leaving them inert keeps DNS, # certificates and load-balancer topology entirely out of this change. # # The NetworkPolicy is kept verbatim and needs no widening: its `:9500` rule # carries no `from` selector, so any namespace may reach the client port. That # is deliberate in the base ("`:9500` IS INTENTIONALLY LEFT OPEN") because # kubelet probes originate from the node, and it is what lets `thepeach-staging` # pods reach this namespace with no cross-namespace rule. # # ── Preflight, in order ────────────────────────────────────────────────────── # 1. Prove nothing public is rendered. This is the check that caught the # surviving Ingress; it is cheap and it is the difference between an # internal service and a published hostname: # # kubectl kustomize k8s/cluster-gke-peach | grep -E '^kind:' | sort -u # # Expect exactly: Certificate, ConfigMap, Issuer, Namespace, NetworkPolicy, # PodDisruptionBudget, Service, StatefulSet. Any Ingress, IngressRoute, # ServersTransport or Middleware means a delete patch stopped matching — # STOP and fix the patch, never apply past it. # # 2. Create `tidaldb-credentials`. It is NOT in this overlay and never should # be — no credential belongs in a manifest. `TIDAL_API_KEY` must be # byte-identical to the consumer's `TIDALDB_POSTS_API_KEY` (GSM # `thepeach-staging-tidaldb-discover-api-key`) or the bearer check refuses # every request; also needs `TIDAL_CLUSTER_KEY`. See # `../cluster/secret.example.yaml`. # # 3. Pin the image digest that carries the signal-context fix. The base's pin # predates it, and anything older reintroduces the silent drop this # deployment exists to avoid. # # 4. Server dry-run, which validates admission without creating anything: # # kubectl apply -k k8s/cluster-gke-peach --dry-run=server # # Namespaced objects report `NotFound` until the namespace exists; create it # first and re-run. Validated clean against GKE 2026-09-16 — all 12 objects, # warnings only (the `kubectl create`-vs-`apply` annotation, and # cert-manager's rotationPolicy default change). # # ── After applying ─────────────────────────────────────────────────────────── # The consumer refuses a cluster instance until its own guard is relaxed: # `thepeach/services/api/src/state.rs` disables the client when `/health` # reports a mode other than `standalone`. That guard existed BECAUSE of the # context-dropping bug; relax it in the same change that repoints the URL, not # before, so a stale image cannot be adopted silently. # # The consumer also needs the CA, because `:9500` serves TLS from the cluster's # OWN cert-manager root and no public root can verify it. Kubernetes Secrets do # not cross namespaces, so `ca.crt` from `tidaldb-cluster-ca` has to be copied # into the consumer's namespace (peach reads it via # `TidalClient::from_env_with_ca`). # # A copied certificate deserves suspicion — k3s-fleet's own history records an # incident where "the expired central SDLC wildcard had been copied manually # and never renewed into its consumer namespaces". This copy is materially # different and the numbers are why: # # tidaldb-cluster-ca duration 87600h (10y), renewBefore 8760h (1y) isCA # tidaldb-cluster-tls duration 2160h (90d), renewBefore 720h (30d) # # The 90-day LEAF is what rotates, roughly every 60 days, and it rotates UNDER # this same root — so the copied CA keeps verifying it and needs no attention. # The fleet incident was an expiring leaf; this is a decade-lived root. # # The coupling that DOES bite is recreation, not expiry: `tidaldb-cluster-ca` is # self-signed and generated fresh on first apply, so destroying and re-applying # this namespace mints a NEW root and every copy goes stale at once — every # consumer request then fails TLS verification with nothing wrong on either # side. Treat "refresh the CA copy" as a mandatory step of any recreate, and # prefer `kubectl rollout restart` over deleting the namespace. apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization namespace: tidaldb-cluster resources: - ../cluster patches: # GKE fact 1 — see header. - target: kind: StatefulSet name: tidaldb patch: |- - op: replace path: /spec/volumeClaimTemplates/0/spec/storageClassName value: standard-rwo # GKE fact 2 — drop the fleet-only public exposure. Deleted by exact kind and # name so a base change that adds a fourth object fails loudly here instead # of quietly publishing it. # # The public object is a plain `networking.k8s.io/v1` Ingress carrying Traefik # ANNOTATIONS — NOT a `traefik.io` IngressRoute. An earlier version of this # file targeted IngressRoute, which matched nothing: kustomize rendered # happily and the Ingress survived, so `tidaldb.threesix.ai` would have been # published from GKE. Rendering the overlay and reading the object inventory # is what caught it, which is why that check is a runbook step below and not # left to someone's memory. - target: group: networking.k8s.io version: v1 kind: Ingress name: tidaldb patch: |- $patch: delete apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: tidaldb - target: group: traefik.io kind: ServersTransport name: tidaldb-internal patch: |- $patch: delete apiVersion: traefik.io/v1alpha1 kind: ServersTransport metadata: name: tidaldb-internal - target: group: traefik.io kind: Middleware name: tidaldb-ratelimit patch: |- $patch: delete apiVersion: traefik.io/v1alpha1 kind: Middleware metadata: name: tidaldb-ratelimit