2c25b6d375
The consumer needs the cluster's CA and Secrets do not cross namespaces, so ca.crt has to be copied into peach's namespace. A copied certificate deserves suspicion — k3s-fleet's own history records an incident where the expired central SDLC wildcard was copied manually and never renewed into its consumer namespaces — so the numbers are written down rather than assumed: tidaldb-cluster-ca 87600h (10y), renewBefore 8760h (1y), isCA tidaldb-cluster-tls 2160h (90d), renewBefore 720h (30d) The 90-day LEAF is what rotates, roughly every 60 days, and it rotates under this same root, so the copied CA keeps verifying it with no attention. The fleet incident was an expiring leaf; this is a decade-lived root. The coupling that actually bites is recreation, not expiry: the root is self-signed and minted fresh on first apply, so destroying and re-applying the namespace invalidates every copy at once and every consumer request then fails TLS with nothing visibly wrong on either side. Recorded as a mandatory step of any recreate.
172 lines
8.3 KiB
YAML
172 lines
8.3 KiB
YAML
# The tidalDB RF3 CLUSTER on GKE, for thepeach's discover post corpus.
|
|
#
|
|
# Same shape as the k3s fleet deployment — ONE StatefulSet, 3 replicas, every
|
|
# pod a region, quorum-acked writes, inter-node mTLS from the cluster's own
|
|
# cert-manager CA. This is an overlay and not a fork: the base owns the
|
|
# topology, and the two things below are the only GKE facts the base cannot
|
|
# know. Precedent is `../cluster-local-kind`, which patches the same field.
|
|
#
|
|
# WHY A CLUSTER AND NOT THE STANDALONE
|
|
# ------------------------------------
|
|
# The standalone was chosen for staging ONLY to work around a real defect:
|
|
# cluster-mode `POST /signals` applied (signal, entity, weight) and silently
|
|
# discarded `user_id`/`creator_id` while still answering 204, so a clustered
|
|
# deployment accepted every behavioural signal and learned nothing. That is
|
|
# fixed at the source in 6ad8c51 (`signal_with_context_staged`), so the
|
|
# workaround is retired rather than inherited.
|
|
#
|
|
# WHY GKE AND NOT THE FLEET
|
|
# -------------------------
|
|
# The consumer is here. `k3s-fleet/deployments/history/tidaldb.md` measured the
|
|
# fleet instance holding 1,720m of CPU requests — 17% of fleet allocatable —
|
|
# while serving NOTHING, with `tidaldb-staging` reporting `items: 0` after 53
|
|
# days, because "the only in-source consumers ... run on GKE". thepeach's api
|
|
# and companion-worker are those consumers, and cluster-local DNS does not
|
|
# cross clusters.
|
|
#
|
|
# ── GKE fact 1: storage class ────────────────────────────────────────────────
|
|
# The base asks for `local-path`, the k3s local-path provisioner. GKE has no
|
|
# such class, so the volumeClaimTemplate would leave every pod Pending
|
|
# indefinitely with no error on the StatefulSet itself. `standard-rwo` is the
|
|
# GKE default (pd.csi.storage.gke.io, WaitForFirstConsumer), which is also the
|
|
# binding mode a StatefulSet wants.
|
|
#
|
|
# ── GKE fact 2: no public ingress ────────────────────────────────────────────
|
|
# The base's `ingress.yaml` publishes `tidaldb.threesix.ai` via Traefik + a
|
|
# Let's Encrypt cert. That hostname is a FLEET name; it has no DNS record here
|
|
# and must not get one. Its own comment says "Remove for internal-only", and
|
|
# internal-only is exactly right: peach reaches this over cluster-local DNS at
|
|
# `tidaldb.tidaldb-cluster.svc.cluster.local:9500`, so nothing needs to be
|
|
# exposed. Deleting the three objects rather than leaving them inert keeps DNS,
|
|
# certificates and load-balancer topology entirely out of this change.
|
|
#
|
|
# The NetworkPolicy is kept verbatim and needs no widening: its `:9500` rule
|
|
# carries no `from` selector, so any namespace may reach the client port. That
|
|
# is deliberate in the base ("`:9500` IS INTENTIONALLY LEFT OPEN") because
|
|
# kubelet probes originate from the node, and it is what lets `thepeach-staging`
|
|
# pods reach this namespace with no cross-namespace rule.
|
|
#
|
|
# ── Preflight, in order ──────────────────────────────────────────────────────
|
|
# 1. Prove nothing public is rendered. This is the check that caught the
|
|
# surviving Ingress; it is cheap and it is the difference between an
|
|
# internal service and a published hostname:
|
|
#
|
|
# kubectl kustomize k8s/cluster-gke-peach | grep -E '^kind:' | sort -u
|
|
#
|
|
# Expect exactly: Certificate, ConfigMap, Issuer, Namespace, NetworkPolicy,
|
|
# PodDisruptionBudget, Service, StatefulSet. Any Ingress, IngressRoute,
|
|
# ServersTransport or Middleware means a delete patch stopped matching —
|
|
# STOP and fix the patch, never apply past it.
|
|
#
|
|
# 2. Create `tidaldb-credentials`. It is NOT in this overlay and never should
|
|
# be — no credential belongs in a manifest. `TIDAL_API_KEY` must be
|
|
# byte-identical to the consumer's `TIDALDB_POSTS_API_KEY` (GSM
|
|
# `thepeach-staging-tidaldb-discover-api-key`) or the bearer check refuses
|
|
# every request; also needs `TIDAL_CLUSTER_KEY`. See
|
|
# `../cluster/secret.example.yaml`.
|
|
#
|
|
# 3. Pin the image digest that carries the signal-context fix. The base's pin
|
|
# predates it, and anything older reintroduces the silent drop this
|
|
# deployment exists to avoid.
|
|
#
|
|
# 4. Server dry-run, which validates admission without creating anything:
|
|
#
|
|
# kubectl apply -k k8s/cluster-gke-peach --dry-run=server
|
|
#
|
|
# Namespaced objects report `NotFound` until the namespace exists; create it
|
|
# first and re-run. Validated clean against GKE 2026-09-16 — all 12 objects,
|
|
# warnings only (the `kubectl create`-vs-`apply` annotation, and
|
|
# cert-manager's rotationPolicy default change).
|
|
#
|
|
# ── After applying ───────────────────────────────────────────────────────────
|
|
# The consumer refuses a cluster instance until its own guard is relaxed:
|
|
# `thepeach/services/api/src/state.rs` disables the client when `/health`
|
|
# reports a mode other than `standalone`. That guard existed BECAUSE of the
|
|
# context-dropping bug; relax it in the same change that repoints the URL, not
|
|
# before, so a stale image cannot be adopted silently.
|
|
#
|
|
# The consumer also needs the CA, because `:9500` serves TLS from the cluster's
|
|
# OWN cert-manager root and no public root can verify it. Kubernetes Secrets do
|
|
# not cross namespaces, so `ca.crt` from `tidaldb-cluster-ca` has to be copied
|
|
# into the consumer's namespace (peach reads it via
|
|
# `TidalClient::from_env_with_ca`).
|
|
#
|
|
# A copied certificate deserves suspicion — k3s-fleet's own history records an
|
|
# incident where "the expired central SDLC wildcard had been copied manually
|
|
# and never renewed into its consumer namespaces". This copy is materially
|
|
# different and the numbers are why:
|
|
#
|
|
# tidaldb-cluster-ca duration 87600h (10y), renewBefore 8760h (1y) isCA
|
|
# tidaldb-cluster-tls duration 2160h (90d), renewBefore 720h (30d)
|
|
#
|
|
# The 90-day LEAF is what rotates, roughly every 60 days, and it rotates UNDER
|
|
# this same root — so the copied CA keeps verifying it and needs no attention.
|
|
# The fleet incident was an expiring leaf; this is a decade-lived root.
|
|
#
|
|
# The coupling that DOES bite is recreation, not expiry: `tidaldb-cluster-ca` is
|
|
# self-signed and generated fresh on first apply, so destroying and re-applying
|
|
# this namespace mints a NEW root and every copy goes stale at once — every
|
|
# consumer request then fails TLS verification with nothing wrong on either
|
|
# side. Treat "refresh the CA copy" as a mandatory step of any recreate, and
|
|
# prefer `kubectl rollout restart` over deleting the namespace.
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
|
|
namespace: tidaldb-cluster
|
|
|
|
resources:
|
|
- ../cluster
|
|
|
|
patches:
|
|
# GKE fact 1 — see header.
|
|
- target:
|
|
kind: StatefulSet
|
|
name: tidaldb
|
|
patch: |-
|
|
- op: replace
|
|
path: /spec/volumeClaimTemplates/0/spec/storageClassName
|
|
value: standard-rwo
|
|
|
|
# GKE fact 2 — drop the fleet-only public exposure. Deleted by exact kind and
|
|
# name so a base change that adds a fourth object fails loudly here instead
|
|
# of quietly publishing it.
|
|
#
|
|
# The public object is a plain `networking.k8s.io/v1` Ingress carrying Traefik
|
|
# ANNOTATIONS — NOT a `traefik.io` IngressRoute. An earlier version of this
|
|
# file targeted IngressRoute, which matched nothing: kustomize rendered
|
|
# happily and the Ingress survived, so `tidaldb.threesix.ai` would have been
|
|
# published from GKE. Rendering the overlay and reading the object inventory
|
|
# is what caught it, which is why that check is a runbook step below and not
|
|
# left to someone's memory.
|
|
- target:
|
|
group: networking.k8s.io
|
|
version: v1
|
|
kind: Ingress
|
|
name: tidaldb
|
|
patch: |-
|
|
$patch: delete
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: Ingress
|
|
metadata:
|
|
name: tidaldb
|
|
- target:
|
|
group: traefik.io
|
|
kind: ServersTransport
|
|
name: tidaldb-internal
|
|
patch: |-
|
|
$patch: delete
|
|
apiVersion: traefik.io/v1alpha1
|
|
kind: ServersTransport
|
|
metadata:
|
|
name: tidaldb-internal
|
|
- target:
|
|
group: traefik.io
|
|
kind: Middleware
|
|
name: tidaldb-ratelimit
|
|
patch: |-
|
|
$patch: delete
|
|
apiVersion: traefik.io/v1alpha1
|
|
kind: Middleware
|
|
metadata:
|
|
name: tidaldb-ratelimit
|