1b5bcbacd7
A client-side ship DEADLINE means the RPC did not round-trip within
request_timeout — which a slow-but-ALIVE follower produces under a sustained
1536-D ack=quorum apply burst (transport runtime momentarily starved by the
CPU-heavy HNSW apply on its single segment-receiver thread) exactly as a
genuinely blackholed peer does. Counting that as record_failure was the
write-burst false-partition: 5 such opened both followers' breakers, the commit
index stalled, ack=quorum 503'd, and retries re-burst the same starved peers
with no self-heal.
- CircuitBreaker::record_timeout: opens ONLY when the peer shows no recent proof
of life (no round-tripped success/backpressure within reset_duration); neutral
no-op when liveness is fresh; re-opens (never wedges) HalfOpen; never refreshes
the liveness stamp (no reply arrived).
- PeerPool::send_to routes tonic DeadlineExceeded/Cancelled -> record_timeout;
genuine severance still surfaces as connect-level Unavailable/transport reset
-> record_failure and still opens the breaker.
- ship_timeout_breaker.rs: end-to-end proof over a REAL tonic WalShipping server
(handler succeeds once then hangs past the client deadline) + 6 unit tests.
Also: re-scope G-S Scalability guarantee to read-throughput with the Ref-A
tidal-t5-readtput owner-test (write 2.5x is structurally impossible on 3-node
full-placement RF3); bump k8s image to m12-rc6 (live, commit 0919b0a); rustfmt
soak_eval / soak-eval / s3 / tidalctl.
74 lines
3.0 KiB
YAML
74 lines
3.0 KiB
YAML
# T5 read-throughput (the measurable half): how many /vector_search ops/s the
|
|
# 3-node cluster sustains within SLA, spread round-robin across all 3 pods.
|
|
# (T5-as-written's 2.5x WRITE-scaling is structurally impossible on 3-node RF3
|
|
# full placement — proven in docs/profiling/m12p4-t5-sharded-throughput.md.)
|
|
#
|
|
# THIS JOB IS THE Ref-A OWNER-TEST for the re-scoped G-S Scalability guarantee
|
|
# (read-throughput) — see docs/planning/milestone-11/guarantee-traceability.md
|
|
# (G-S row, re-scoped 2026-06-19). The structural basis (a node serves reads from
|
|
# its LOCAL placement, no cross-node forward → per-node capacity adds) is proven by
|
|
# the cargo e2e `mp_partial_placement_feed_spans_all_groups` (cluster_cross_shard_reads).
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: tidal-t5-readtput
|
|
namespace: tidaldb-cluster
|
|
labels: { app.kubernetes.io/name: tidal-stress, app.kubernetes.io/part-of: tidaldb }
|
|
spec:
|
|
backoffLimit: 0
|
|
ttlSecondsAfterFinished: 7200
|
|
template:
|
|
metadata:
|
|
labels: { app.kubernetes.io/name: tidal-stress, app.kubernetes.io/part-of: tidaldb }
|
|
spec:
|
|
restartPolicy: Never
|
|
automountServiceAccountToken: false
|
|
securityContext: { runAsNonRoot: true, runAsUser: 1000, runAsGroup: 1000, seccompProfile: { type: RuntimeDefault } }
|
|
containers:
|
|
- name: stress
|
|
image: registry.threesix.ai/tidal/stress:m12-rc7-seedretry
|
|
imagePullPolicy: IfNotPresent
|
|
args:
|
|
- --target
|
|
- https://tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
|
|
- --target
|
|
- https://tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
|
|
- --target
|
|
- https://tidaldb-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
|
|
- --ca-cert
|
|
- /etc/tidaldb/tls/ca.crt
|
|
- --verify-recall
|
|
- --skip-seed
|
|
- --corpus
|
|
- "100000"
|
|
- --embedding-dim
|
|
- "1536"
|
|
- --recall-k
|
|
- "10"
|
|
- --recall-queries
|
|
- "1000"
|
|
- --read-p99-target-ms
|
|
- "10"
|
|
- --recall-target
|
|
- "0.95"
|
|
- --recall-ef-search
|
|
- "64"
|
|
- --ramp
|
|
- "1000:20,2000:20,3000:20,3800:20"
|
|
env:
|
|
- name: TIDAL_API_KEY
|
|
valueFrom: { secretKeyRef: { name: tidaldb-credentials, key: TIDAL_API_KEY } }
|
|
- name: TIDAL_STRESS_LOG
|
|
value: warn
|
|
resources:
|
|
requests: { cpu: "1", memory: 512Mi }
|
|
limits: { cpu: "3", memory: 2Gi }
|
|
securityContext: { allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities: { drop: ["ALL"] } }
|
|
volumeMounts:
|
|
- { name: cluster-tls, mountPath: /etc/tidaldb/tls, readOnly: true }
|
|
volumes:
|
|
- name: cluster-tls
|
|
secret:
|
|
secretName: tidaldb-cluster-tls
|
|
items: [ { key: ca.crt, path: ca.crt } ]
|