8e39ee1078
The m12p5 idle-readiness work converged on an idle cluster, but the real T4 1M/1536 scale-up over mTLS still failed to admit new pods. Three real blockers, all invisible to the plaintext in-process tests: - CryptoProvider crash-loop: the seed-join/reseed boot path builds a blocking reqwest (rustls) HTTPS client on a dedicated boot thread BEFORE GrpcTransport::new installs the process-wide provider, so every TLS joiner panicked. Install it at the top of main(); ensure_crypto_provider() is now pub, idempotent, harmless on the plaintext standalone path. - Wrong seed scheme + target: peer_url honors an explicit URL scheme verbatim, so http:// dialed plaintext at the TLS :9500 port. Seed is now https:// AND points at the ready-only client Service (ClusterIP VIP), not the headless peers Service — so a joiner never round-robins onto a not-ready pod (incl. itself) and burns the 120s discovery window. - Too-tight poll budget: a cold status poll pays a full rustls handshake on top of DNS+TCP; under CPU contention that alone blew the 500ms budget, so the joiner timed out every poll for the whole window despite the peer being reachable. Status-poll timeout is now 5s (env: TIDAL_SEED_STATUS_TIMEOUT_MS) with a separate 2s connect timeout (dead seeds still fail fast) and debug-level logging on every discovery failure mode. Refactors riding along: - on_heartbeat takes a HeartbeatContext struct (additive fields, no silent u64 transposition) across tidal-net, election_driver, and both test hooks. - ShardReplica::applied_for_leader_shard centralizes per-source-shard keying (BUG 1) shared by the readiness drive and local_status. - idle-readiness test now asserts convergence within ½ budget — a slow-path regression (periodic self-heal / status-poll dependency) the binary budget check would otherwise wave through. New k8s T4 manifests: cluster-t4-kind kustomization + single-group topology patch; tidal-stress t4 seed/load Jobs.
86 lines
3.9 KiB
YAML
86 lines
3.9 KiB
YAML
# m12p5 T4 (idle-readiness + scale 3→5→3 elasticity) overlay for local `kind`.
|
|
#
|
|
# Reuses the canonical k8s/cluster/ base but (1) collapses to a SINGLE
|
|
# replication group (topology-singlegroup-patch.yaml — T4 is replica elasticity,
|
|
# not sharding), (2) swaps the registry image for the locally-built m12p5 tag,
|
|
# (3) repoints the PVC at kind's `standard` StorageClass, and (4) gives each pod
|
|
# room for a 1536-dim corpus + a catch-up joiner.
|
|
#
|
|
# Run (the build tags MUST match the `images:` newTag below and the Job specs in
|
|
# tidal-stress/k8s/t4-*.yaml — kind serves only locally-loaded images, so a tag
|
|
# mismatch is an ImagePullBackOff):
|
|
# # server image — tag MUST equal the `images: newTag` below (m12p5-fix2)
|
|
# docker build -f docker/deploy/Dockerfile -t tidaldb-server:m12p5-fix2 .
|
|
# kind load docker-image tidaldb-server:m12p5-fix2 --name canopy
|
|
# # stress generator — the m12p4 binary is reused unchanged for m12p5; tag MUST
|
|
# # equal the `image:` in tidal-stress/k8s/t4-{seed,load}-job.yaml (m12p4-local)
|
|
# docker build -f docker/stress/Dockerfile -t tidaldb-stress:m12p4-local .
|
|
# kind load docker-image tidaldb-stress:m12p4-local --name canopy
|
|
# kubectl create namespace tidaldb-cluster
|
|
# kubectl -n tidaldb-cluster create secret generic tidaldb-credentials \
|
|
# --from-literal=TIDAL_API_KEY="$(openssl rand -hex 32)" \
|
|
# --from-literal=TIDAL_CLUSTER_KEY="$(openssl rand -hex 32)"
|
|
# kubectl apply -k k8s/cluster-t4-kind/
|
|
# kubectl apply -f tidal-stress/k8s/t4-seed-job.yaml # seed the 1536-dim corpus
|
|
# kubectl apply -f tidal-stress/k8s/t4-load-job.yaml # steady load across the scale
|
|
# # scale 3→5 (idle-readiness fix + wildcard-SAN mTLS), then 5→3.
|
|
# kubectl -n tidaldb-cluster scale statefulset tidaldb --replicas=5
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
|
|
namespace: tidaldb-cluster
|
|
|
|
resources:
|
|
- ../cluster
|
|
|
|
images:
|
|
- name: registry.threesix.ai/tidal/server
|
|
newName: tidaldb-server
|
|
newTag: m12p5-fix2
|
|
|
|
patches:
|
|
# Single replication group (drop the base's 3-group `shards:` block).
|
|
- path: topology-singlegroup-patch.yaml
|
|
target:
|
|
kind: ConfigMap
|
|
name: tidaldb-cluster-topology
|
|
|
|
# kind ships `standard` (rancher.io/local-path), not the `local-path` the base
|
|
# names; repoint the PVC so claims bind. Bump the per-pod ceiling so a 1536-dim
|
|
# corpus + a joining replica's catch-up index fit (the 2Gi base is sized for a
|
|
# smoke corpus, not a real read-recall shape).
|
|
- target:
|
|
kind: StatefulSet
|
|
name: tidaldb
|
|
patch: |-
|
|
- op: replace
|
|
path: /spec/volumeClaimTemplates/0/spec/storageClassName
|
|
value: standard
|
|
- op: replace
|
|
path: /spec/template/spec/containers/0/resources/limits/memory
|
|
value: 6Gi
|
|
- op: replace
|
|
path: /spec/template/spec/containers/0/resources/limits/cpu
|
|
value: "3"
|
|
- op: replace
|
|
path: /spec/template/spec/containers/0/resources/requests/memory
|
|
value: 512Mi
|
|
# The 1536-dim HNSW index is rebuilt from durable storage on every open;
|
|
# under CPU contention on a single shared kind node that can exceed the
|
|
# base 5-min startup budget (a restarted/joining pod 503s its startup probe
|
|
# → SIGKILL → re-rebuild → cascade). Widen the budget to 10 min so a real
|
|
# catch-up + index rebuild finishes before the probe gives up.
|
|
- op: replace
|
|
path: /spec/template/spec/containers/0/startupProbe/failureThreshold
|
|
value: 120
|
|
# Local-run convenience: only roll pods with ordinal >= 3 (the joiners) on a
|
|
# template change, so iterating on the scale-up command never restarts the
|
|
# seeded 0/1/2 (and never triggers a simultaneous index-rebuild cascade on
|
|
# the shared kind node). NOT for the canonical base — production rolls all.
|
|
- op: add
|
|
path: /spec/updateStrategy
|
|
value:
|
|
type: RollingUpdate
|
|
rollingUpdate:
|
|
partition: 3
|