a0399550d6
Boot now LOADS the per-slot HNSW graph instead of rebuilding it. Clean
shutdown writes {data_dir}/vector/<kind>__<slot>.usearch; the next open loads
it when it matches the durable corpus (seconds), falling back to a full rebuild
only when the graph is missing/stale/corrupt. Eliminates the multi-minute boot
rebuild (~50-70 min at 1M/1536-D) that let the WAL compact past a restarting
node and triggered the reseed cascade.
Graceful SIGTERM now actually runs the close: bounded_drain caps the post-signal
HTTP drain (TIDAL_SHUTDOWN_DRAIN_MS, default 15s) then runs the deterministic
close regardless — sibling keep-alive connections no longer block the drain past
the k8s 60s grace into a SIGKILL (which cannot run Drop). ClusterNode and
ShardReplica::shutdown are now &self (db handle is an ArcSwapOption) so the close
fires even when a stuck connection task holds an Arc.
Fix USearch insert to be a true upsert (remove+add): it was unconditional add,
which a multi:false index rejects on a reseeding follower's post-snapshot WAL
replay -> applied_events stalls -> catch-up deadlock -> unrecoverable cluster.
Also: circuit-breaker peer last-contact tracking; real k3s 1536-dim deploy +
recall findings (recall@10 0.9869, read p99 8.71ms @ 200rps @ 100k) in
docs/profiling/m12-cluster-deploy-findings.md; new tidal-stress k8s jobs and
m12p6 graph-persistence + SIGTERM tier-3 regression tests.
98 lines
3.1 KiB
YAML
98 lines
3.1 KiB
YAML
# Rollout readiness trickle — keeps the WAL ship path active through the M12
|
|
# rolling restart so each rejoining pod observes convergence and flips Ready.
|
|
#
|
|
# WHY: during the m11p6 -> m12 rollout the leaders still run m11p6 (no heartbeat
|
|
# live-frontier field), so a rejoining M12 pod on an OTHERWISE-IDLE cluster can
|
|
# sit Ready=false forever (the WORKLOG idle-readiness bug). A steady low-rate
|
|
# quorum write stream gives every rejoiner real catch-up traffic to observe.
|
|
#
|
|
# Targets the READY-ONLY client Service VIP so writes route around the pod that
|
|
# is currently restarting. NO --max-error-pct gate: transient quorum failures
|
|
# while a pod cycles are expected and must not kill the trickle. Long ramp so it
|
|
# spans the whole 3-pod rollout; delete when the rollout is green.
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: tidal-rollout-trickle
|
|
namespace: tidaldb-cluster
|
|
labels:
|
|
app.kubernetes.io/name: tidal-stress
|
|
app.kubernetes.io/part-of: tidaldb
|
|
spec:
|
|
backoffLimit: 0
|
|
ttlSecondsAfterFinished: 600
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: tidal-stress
|
|
app.kubernetes.io/part-of: tidaldb
|
|
spec:
|
|
restartPolicy: Never
|
|
automountServiceAccountToken: false
|
|
securityContext:
|
|
runAsNonRoot: true
|
|
runAsUser: 1000
|
|
runAsGroup: 1000
|
|
seccompProfile:
|
|
type: RuntimeDefault
|
|
containers:
|
|
- name: stress
|
|
image: registry.threesix.ai/tidal/stress@sha256:4b21c1b89790f7a995f4d9e754fe0faf530079d7ad67a5ad74c15377dcc18ac2
|
|
imagePullPolicy: IfNotPresent
|
|
args:
|
|
- --target
|
|
- https://tidaldb.tidaldb-cluster.svc.cluster.local:9500
|
|
- --ca-cert
|
|
- /etc/tidaldb/tls/ca.crt
|
|
- --ack
|
|
- quorum
|
|
- --skip-seed
|
|
- --corpus
|
|
- "100000"
|
|
- --embedding-dim
|
|
- "1536"
|
|
- --users
|
|
- "100000"
|
|
# 30 rps quorum write-heavy mix for ~20 min — pure ship-path keepalive.
|
|
- --ramp
|
|
- "30:1200"
|
|
- --mix
|
|
- "view=3,like=1,item=1,embed=1"
|
|
- --max-inflight
|
|
- "256"
|
|
env:
|
|
- name: TIDAL_API_KEY
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: tidaldb-credentials
|
|
key: TIDAL_API_KEY
|
|
- name: TIDAL_STRESS_LOG
|
|
value: warn
|
|
resources:
|
|
requests:
|
|
cpu: 250m
|
|
memory: 256Mi
|
|
limits:
|
|
cpu: "1"
|
|
memory: 512Mi
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
capabilities:
|
|
drop: ["ALL"]
|
|
volumeMounts:
|
|
- name: cluster-tls
|
|
mountPath: /etc/tidaldb/tls
|
|
readOnly: true
|
|
- name: tmp
|
|
mountPath: /tmp
|
|
volumes:
|
|
- name: cluster-tls
|
|
secret:
|
|
secretName: tidaldb-cluster-tls
|
|
items:
|
|
- key: ca.crt
|
|
path: ca.crt
|
|
- name: tmp
|
|
emptyDir: {}
|