a0399550d6
Boot now LOADS the per-slot HNSW graph instead of rebuilding it. Clean
shutdown writes {data_dir}/vector/<kind>__<slot>.usearch; the next open loads
it when it matches the durable corpus (seconds), falling back to a full rebuild
only when the graph is missing/stale/corrupt. Eliminates the multi-minute boot
rebuild (~50-70 min at 1M/1536-D) that let the WAL compact past a restarting
node and triggered the reseed cascade.
Graceful SIGTERM now actually runs the close: bounded_drain caps the post-signal
HTTP drain (TIDAL_SHUTDOWN_DRAIN_MS, default 15s) then runs the deterministic
close regardless — sibling keep-alive connections no longer block the drain past
the k8s 60s grace into a SIGKILL (which cannot run Drop). ClusterNode and
ShardReplica::shutdown are now &self (db handle is an ArcSwapOption) so the close
fires even when a stuck connection task holds an Arc.
Fix USearch insert to be a true upsert (remove+add): it was unconditional add,
which a multi:false index rejects on a reseeding follower's post-snapshot WAL
replay -> applied_events stalls -> catch-up deadlock -> unrecoverable cluster.
Also: circuit-breaker peer last-contact tracking; real k3s 1536-dim deploy +
recall findings (recall@10 0.9869, read p99 8.71ms @ 200rps @ 100k) in
docs/profiling/m12-cluster-deploy-findings.md; new tidal-stress k8s jobs and
m12p6 graph-persistence + SIGTERM tier-3 regression tests.
93 lines
2.8 KiB
YAML
93 lines
2.8 KiB
YAML
# T-read recall gate — true p99 + recall@10 vs a brute-force cosine oracle.
|
|
#
|
|
# Seeds a 100k / 1536-dim corpus (regenerates the deterministic base vectors so
|
|
# it can build the ground-truth oracle locally), then ramps pure k-NN
|
|
# /vector_search probes against tidaldb-0 and reports recall@10 + read p99 per
|
|
# stage. Requires the M12 server (the /vector_search endpoint did not exist
|
|
# pre-m12; the m11p6 image 404s).
|
|
#
|
|
# Apply: kubectl apply -f tidal-stress/k8s/recall-verify-job.yaml
|
|
# Watch: kubectl logs -f job/tidal-recall-verify -n tidaldb-cluster
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: tidal-recall-verify
|
|
namespace: tidaldb-cluster
|
|
labels:
|
|
app.kubernetes.io/name: tidal-stress
|
|
app.kubernetes.io/part-of: tidaldb
|
|
spec:
|
|
backoffLimit: 0
|
|
ttlSecondsAfterFinished: 7200
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: tidal-stress
|
|
app.kubernetes.io/part-of: tidaldb
|
|
spec:
|
|
restartPolicy: Never
|
|
automountServiceAccountToken: false
|
|
securityContext:
|
|
runAsNonRoot: true
|
|
runAsUser: 1000
|
|
runAsGroup: 1000
|
|
seccompProfile:
|
|
type: RuntimeDefault
|
|
containers:
|
|
- name: stress
|
|
image: registry.threesix.ai/tidal/stress@sha256:4b21c1b89790f7a995f4d9e754fe0faf530079d7ad67a5ad74c15377dcc18ac2
|
|
imagePullPolicy: IfNotPresent
|
|
args:
|
|
- --target
|
|
- https://tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
|
|
- --ca-cert
|
|
- /etc/tidaldb/tls/ca.crt
|
|
- --verify-recall
|
|
- --corpus
|
|
- "100000"
|
|
- --embedding-dim
|
|
- "1536"
|
|
- --recall-k
|
|
- "10"
|
|
- --recall-queries
|
|
- "1000"
|
|
- --read-p99-target-ms
|
|
- "10"
|
|
- --recall-target
|
|
- "0.95"
|
|
- --recall-ef-search
|
|
- "64"
|
|
- --ramp
|
|
- "200:30,500:30,1000:30,2000:30"
|
|
env:
|
|
- name: TIDAL_API_KEY
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: tidaldb-credentials
|
|
key: TIDAL_API_KEY
|
|
- name: TIDAL_STRESS_LOG
|
|
value: warn
|
|
resources:
|
|
requests:
|
|
cpu: 500m
|
|
memory: 512Mi
|
|
limits:
|
|
cpu: "3"
|
|
memory: 2Gi
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
capabilities:
|
|
drop: ["ALL"]
|
|
volumeMounts:
|
|
- name: cluster-tls
|
|
mountPath: /etc/tidaldb/tls
|
|
readOnly: true
|
|
volumes:
|
|
- name: cluster-tls
|
|
secret:
|
|
secretName: tidaldb-cluster-tls
|
|
items:
|
|
- key: ca.crt
|
|
path: ca.crt
|