a946c6128c
Read-SLA fix (rc12→rc13 — cpu-cgroup starvation → multi-second p99 + churning elections): - offload.rs: add SEARCH_GATE semaphore (core_count+1 permits, 50ms shed to 429) so per-shard searches gate on CPU, not reactor threads; concurrent scatter_merge fan-out (join_all) replaces the serial blocking offload_region_read loop - node.rs: scatter_merge → async; per-shard futures run via offload_search (each acquires one SEARCH_GATE permit, moves it into spawn_blocking so the permit is held for the search's full CPU lifetime) - main.rs: explicit tokio runtime with worker_threads floored at 4, independent of the cgroup quota — keeps the control plane (heartbeat/election/apply) on its own workers even when quota < 4 - k8s statefulset: CPU limit 2→3 (was: available_parallelism()=2 → only 2 async workers; search burst starved the reactor) - tidal/wal/compaction.rs: WAL_RETENTION_SEGMENTS 4→16 (64 MiB→256 MiB per-shard catch-up window; a briefly-down follower across a rolling restart streams up instead of forcing snapshot reseed; disk floor 768 MiB/pod, self-trimming) - cluster_reseed.rs: OFFLINE_ITEMS 1800→5600 to exceed the new 16-segment retention window (19 segs > 17); fix sequential quarantine/reseed race via await_status_bool tidalctl S3/R2 backup DR: - tidalctl/Cargo.toml: aws-config, aws-sdk-s3, aws-credential-types, tokio, tempfile - commands/s3.rs: S3Target + export_dir (upload every file, manifest last as atomicity marker) + import_to_dir (download prefix into temp staging dir) - commands/backup.rs: run_backup/run_restore accept Option<&S3Target>; S3 export is additive after local fsync barrier; S3 import stages into TempDir then runs the unchanged verified restore on it - main.rs: --s3-endpoint / --s3-bucket / --s3-prefix flags; all-or-nothing endpoint+bucket validation; usage updated tidal-stress/k8s: recall-rc12-spread-job, soak-nightly-cronjob, soak-monitor, soak-results-pvc, t5-readtput-job manifests
26 lines
934 B
YAML
26 lines
934 B
YAML
# Durable result sink for the 30-night soak (GA bar: nightly suite green 30
|
|
# consecutive days — docs/planning/milestone-11/phase-9.md "Exit gate").
|
|
#
|
|
# longhorn-rwx: RWX so the nightly soak CronJob pods (writers) and the always-on
|
|
# soak-monitor (reader) can mount it concurrently, and Retain reclaim policy so
|
|
# the 30 nights of JSON summaries survive a PVC delete / job churn / operator
|
|
# session ending. Each night writes /results/soak-YYYY-MM-DD.json + appends one
|
|
# line to /results/ledger.tsv (date<TAB>verdict<TAB>p99<TAB>err% <TAB>image).
|
|
#
|
|
# Apply: kubectl apply -f tidal-stress/k8s/soak-results-pvc.yaml
|
|
apiVersion: v1
|
|
kind: PersistentVolumeClaim
|
|
metadata:
|
|
name: tidal-soak-results
|
|
namespace: tidaldb-cluster
|
|
labels:
|
|
app.kubernetes.io/name: tidal-soak
|
|
app.kubernetes.io/part-of: tidaldb
|
|
spec:
|
|
accessModes:
|
|
- ReadWriteMany
|
|
storageClassName: longhorn-rwx
|
|
resources:
|
|
requests:
|
|
storage: 2Gi
|