f640764d89
New workspace crate: an open-loop, coordinated-omission-corrected HTTP load generator + capacity ramp for the standalone and multi-process cluster surfaces, modeling a thepeach feed session (feed reads + view/like/skip signals + search, signal-dominated per their user-graph spec). Throttleable target rate, ramp presets (smoke/quick/peach-100k/max) or rps:secs specs, peach/reads/writes/custom mixes, leader vs sharded write paths, per-op p50/p90/p99/p999/max latency, a backpressure-aware status breakdown (429/408/503/4xx/5xx/transport), and a verdict translated to supported DAU. Runs in-cluster as a k8s Job (tidal-stress/k8s/). Open-loop scheduler (scheduler.rs) fires at a fixed arrival rate and measures latency from each request's intended send time, so a server stall inflates the percentiles a closed-loop test hides; it shed-and-counts rather than blocking when the in-flight cap is reached. Pure-Rust (tokio + reqwest/rustls), no engine deps. Findings on the live 3-region k3s cluster (docs/ops/stress-test-thepeach.md): reads scale to thousands/s at <15ms p99; the replicated /signals path saturates at ~90 signals/s (single-leader funnel + 2-worker write pool + synchronous gRPC ship); the sharded path sustains 3,669 signals/s at 0 errors and ~27% cluster CPU (≈ the 100k-DAU peak, knee not reached). Overload degrades gracefully (429; 0 pod restarts). thepeach's planned in-process embedding sidesteps all of it (write ≈82ns).
99 lines
3.5 KiB
YAML
99 lines
3.5 KiB
YAML
# In-cluster capacity ramp for the tidalDB multi-process cluster.
|
|
#
|
|
# Runs the load generator AS A POD so requests take the real cluster network path
|
|
# (a `kubectl port-forward` serializes everything through the API server and adds
|
|
# tens of ms — useless for capacity numbers). Targets the three region ClusterIPs
|
|
# directly; reads round-robin across them (each serves locally), leader-path
|
|
# writes are pinned to us-east (10.43.99.11) to measure the single-leader funnel.
|
|
#
|
|
# Apply: kubectl apply -f tidal-stress/k8s/stress-job.yaml
|
|
# Watch: kubectl logs -f job/tidal-stress -n tidaldb
|
|
# Re-run: kubectl delete job tidal-stress -n tidaldb; kubectl apply -f ...
|
|
#
|
|
# To compare the horizontally-scaled path, change `--write-path leader` to
|
|
# `--write-path sharded` (hash-partitions writes across all 3 regions, no funnel).
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: tidal-stress
|
|
namespace: tidaldb
|
|
labels:
|
|
app.kubernetes.io/name: tidal-stress
|
|
app.kubernetes.io/part-of: tidaldb
|
|
spec:
|
|
backoffLimit: 0 # a load run is not retried — read the logs
|
|
ttlSecondsAfterFinished: 7200
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: tidal-stress
|
|
app.kubernetes.io/part-of: tidaldb
|
|
spec:
|
|
restartPolicy: Never
|
|
automountServiceAccountToken: false
|
|
affinity:
|
|
podAntiAffinity:
|
|
# Keep the generator OFF the leader's node so it never steals CPU from
|
|
# the write bottleneck we are measuring.
|
|
requiredDuringSchedulingIgnoredDuringExecution:
|
|
- labelSelector:
|
|
matchLabels:
|
|
tidaldb.region: us-east
|
|
topologyKey: kubernetes.io/hostname
|
|
securityContext:
|
|
runAsNonRoot: true
|
|
runAsUser: 1000
|
|
runAsGroup: 1000
|
|
seccompProfile:
|
|
type: RuntimeDefault
|
|
containers:
|
|
- name: stress
|
|
image: registry.threesix.ai/tidal/stress@sha256:77395cc2857e2adabaa37607b198316d6f1cd0ec04f749caf2de959e6bade55b # :v1
|
|
imagePullPolicy: IfNotPresent
|
|
args:
|
|
- --target
|
|
- http://10.43.99.11:9500 # us-east (leader)
|
|
- --target
|
|
- http://10.43.99.12:9500 # eu-west
|
|
- --target
|
|
- http://10.43.99.13:9500 # ap-south
|
|
- --leader-url
|
|
- http://10.43.99.11:9500
|
|
- --ramp
|
|
- peach-100k
|
|
- --stage-secs
|
|
- "45"
|
|
- --mix
|
|
- peach
|
|
- --write-path
|
|
- leader
|
|
- --corpus
|
|
- "20000"
|
|
- --users
|
|
- "100000"
|
|
- --poll-status
|
|
env:
|
|
- name: TIDAL_API_KEY
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: tidaldb-credentials
|
|
key: TIDAL_API_KEY
|
|
- name: TIDAL_STRESS_LOG
|
|
value: warn
|
|
resources:
|
|
# Small REQUEST (the cluster is request-saturated though ~15% utilised)
|
|
# with a high LIMIT: the generator bursts to the cycles it needs on the
|
|
# idle node. If it ever CPU-saturates, the report's schedule-lag /
|
|
# client-shed will say so — then split the load across multiple Jobs.
|
|
requests:
|
|
cpu: 250m
|
|
memory: 256Mi
|
|
limits:
|
|
cpu: "3"
|
|
memory: 1Gi
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
capabilities:
|
|
drop: ["ALL"]
|