a946c6128c
Read-SLA fix (rc12→rc13 — cpu-cgroup starvation → multi-second p99 + churning elections): - offload.rs: add SEARCH_GATE semaphore (core_count+1 permits, 50ms shed to 429) so per-shard searches gate on CPU, not reactor threads; concurrent scatter_merge fan-out (join_all) replaces the serial blocking offload_region_read loop - node.rs: scatter_merge → async; per-shard futures run via offload_search (each acquires one SEARCH_GATE permit, moves it into spawn_blocking so the permit is held for the search's full CPU lifetime) - main.rs: explicit tokio runtime with worker_threads floored at 4, independent of the cgroup quota — keeps the control plane (heartbeat/election/apply) on its own workers even when quota < 4 - k8s statefulset: CPU limit 2→3 (was: available_parallelism()=2 → only 2 async workers; search burst starved the reactor) - tidal/wal/compaction.rs: WAL_RETENTION_SEGMENTS 4→16 (64 MiB→256 MiB per-shard catch-up window; a briefly-down follower across a rolling restart streams up instead of forcing snapshot reseed; disk floor 768 MiB/pod, self-trimming) - cluster_reseed.rs: OFFLINE_ITEMS 1800→5600 to exceed the new 16-segment retention window (19 segs > 17); fix sequential quarantine/reseed race via await_status_bool tidalctl S3/R2 backup DR: - tidalctl/Cargo.toml: aws-config, aws-sdk-s3, aws-credential-types, tokio, tempfile - commands/s3.rs: S3Target + export_dir (upload every file, manifest last as atomicity marker) + import_to_dir (download prefix into temp staging dir) - commands/backup.rs: run_backup/run_restore accept Option<&S3Target>; S3 export is additive after local fsync barrier; S3 import stages into TempDir then runs the unchanged verified restore on it - main.rs: --s3-endpoint / --s3-bucket / --s3-prefix flags; all-or-nothing endpoint+bucket validation; usage updated tidal-stress/k8s: recall-rc12-spread-job, soak-nightly-cronjob, soak-monitor, soak-results-pvc, t5-readtput-job manifests
103 lines
3.3 KiB
YAML
103 lines
3.3 KiB
YAML
# rc12 read-SLA gate — true p99 + recall@10 vs a brute-force cosine oracle,
|
|
# with read load SPREAD across all 3 nodes (full placement => any node serves any
|
|
# read). recall.rs round-robins each probe across the --target list, so three
|
|
# per-pod DNS targets give an even 1/3-per-node split (deterministic, unlike a
|
|
# single ClusterIP Service which leans on kube-proxy connection balancing).
|
|
# All three per-pod DNS names are cert SANs (wildcard *.tidaldb-peers + -0/-1/-2).
|
|
#
|
|
# Skip-seed: the 100k/1536 corpus already lives on the PVCs (a rolling restart
|
|
# does not wipe them); the harness regenerates the SAME deterministic base
|
|
# vectors locally to build the brute-force ground-truth oracle.
|
|
#
|
|
# Apply: kubectl apply -f tidal-stress/k8s/recall-rc12-spread-job.yaml
|
|
# Watch: kubectl logs -f job/tidal-recall-rc12 -n tidaldb-cluster
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: tidal-recall-rc12
|
|
namespace: tidaldb-cluster
|
|
labels:
|
|
app.kubernetes.io/name: tidal-stress
|
|
app.kubernetes.io/part-of: tidaldb
|
|
spec:
|
|
backoffLimit: 0
|
|
ttlSecondsAfterFinished: 7200
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: tidal-stress
|
|
app.kubernetes.io/part-of: tidaldb
|
|
spec:
|
|
restartPolicy: Never
|
|
automountServiceAccountToken: false
|
|
securityContext:
|
|
runAsNonRoot: true
|
|
runAsUser: 1000
|
|
runAsGroup: 1000
|
|
seccompProfile:
|
|
type: RuntimeDefault
|
|
containers:
|
|
- name: stress
|
|
image: registry.threesix.ai/tidal/stress:m12-rc7-seedretry
|
|
imagePullPolicy: IfNotPresent
|
|
args:
|
|
- --target
|
|
- https://tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
|
|
- --target
|
|
- https://tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
|
|
- --target
|
|
- https://tidaldb-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
|
|
- --ca-cert
|
|
- /etc/tidaldb/tls/ca.crt
|
|
- --verify-recall
|
|
- --skip-seed
|
|
- --seed-concurrency
|
|
- "32"
|
|
- --corpus
|
|
- "100000"
|
|
- --embedding-dim
|
|
- "1536"
|
|
- --recall-k
|
|
- "10"
|
|
- --recall-queries
|
|
- "1000"
|
|
- --read-p99-target-ms
|
|
- "10"
|
|
- --recall-target
|
|
- "0.95"
|
|
- --recall-ef-search
|
|
- "64"
|
|
- --ramp
|
|
- "100:30,200:30,300:30,500:30"
|
|
env:
|
|
- name: TIDAL_API_KEY
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: tidaldb-credentials
|
|
key: TIDAL_API_KEY
|
|
- name: TIDAL_STRESS_LOG
|
|
value: warn
|
|
resources:
|
|
requests:
|
|
cpu: 500m
|
|
memory: 512Mi
|
|
limits:
|
|
cpu: "3"
|
|
memory: 2Gi
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
capabilities:
|
|
drop: ["ALL"]
|
|
volumeMounts:
|
|
- name: cluster-tls
|
|
mountPath: /etc/tidaldb/tls
|
|
readOnly: true
|
|
volumes:
|
|
- name: cluster-tls
|
|
secret:
|
|
secretName: tidaldb-cluster-tls
|
|
items:
|
|
- key: ca.crt
|
|
path: ca.crt
|