docs(tidal-stress): README + worklog + continue-the-process flow
README.md — tool reference pinned to the m11p5 cluster (tidaldb-cluster ns, pod-DNS targets, local-path storage); retires the stale 10.43.99.11-13 ClusterIPs. WORKLOG.md — m11p5 migration + T3 gate (10/10 kills, max 6157ms), the three run-costing traps (Longhorn storage, exec leader-detection, kubectl wait hang), reseed behavior, and the vector-DB positioning note. Records a m11p5 finding: a reseeded node stays NotReady forever on an idle cluster because the first-converged readiness check needs observed replication traffic — flagged as an upstream fix, not a manifest bandaid. PROCESS.md — the repeatable five-beat checkpoint loop, the operational-traps table, the T4 elasticity flow (now unblocked on m11p5, with the idle-readiness watch-item), and the T-read vector-search gate that still needs a recall oracle built before any read-path/vector-DB claim.
This commit is contained in:
jx12n
parent
d5d1e7d81a
commit
46741a3a8c
145
tidal-stress/PROCESS.md
Normal file
145
tidal-stress/PROCESS.md
Normal file
@ -0,0 +1,145 @@
|
||||
# tidal-stress process — running the next checkpoint
|
||||
|
||||
How to take the cluster from "deployed" to "gated" without re-learning the
|
||||
operational traps. Read [README.md](README.md) for the tool, [WORKLOG.md](WORKLOG.md)
|
||||
for what's been run. This doc is the **repeatable flow**.
|
||||
|
||||
The legacy plan in `../tmp/stress-test-plan.md` is correct on *gates and intent*
|
||||
but **stale on addressing** — it names the old 3-StatefulSet ClusterIPs
|
||||
(`10.43.99.11-13`) and namespace `tidaldb`. Use the m11p5 DNS targets from the
|
||||
README instead. Migrating that plan in place is step 0 below.
|
||||
|
||||
---
|
||||
|
||||
## The standard checkpoint loop
|
||||
|
||||
Every checkpoint, old or new, follows the same five beats:
|
||||
|
||||
1. **Pre-flight.** Context + cluster healthy before anything else.
|
||||
```bash
|
||||
export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
|
||||
kubectl config current-context # must be the k3s one
|
||||
kubectl get pods -n tidaldb-cluster # all 3 → 1/1 Running, 0 restarts
|
||||
kubectl port-forward -n tidaldb-cluster svc/tidaldb 19500:9500 &
|
||||
curl -sf localhost:19500/cluster/status | jq . # leader set, every region lag=0
|
||||
```
|
||||
Do **not** start a run with any pod restarting or mid-reseed (`0/1 Running`).
|
||||
A reseed cycle from a prior run takes ~120–150s to settle — wait it out.
|
||||
|
||||
2. **Arm the Job.** Copy the nearest `k8s/stress-job-*.yaml`, set the m11p5 DNS
|
||||
targets (README), pin `--ack quorum`, `--corpus 20000`, `--poll-status`.
|
||||
For a fresh PVC the first run seeds; add `--skip-seed` on subsequent runs in
|
||||
the same session.
|
||||
|
||||
3. **Run + chaos.** Apply the Job; if the checkpoint has a chaos arm (kill /
|
||||
scale), drive it from a script that talks to the **VIP over a port-forward**,
|
||||
never `kubectl exec` into a tidaldb pod (see traps below).
|
||||
|
||||
4. **Capture.** Paste the `CAPACITY VERDICT` block + `[pre/post-stage]` lag lines
|
||||
into `docs/ops/stress-test-<phase>.md`. Record the one number the next
|
||||
checkpoint must not regress.
|
||||
|
||||
5. **Record + re-arm.** Update `WORKLOG.md` (newest first) and the checkpoint
|
||||
table in `README.md`. `kubectl delete job <name> -n tidaldb-cluster` to re-arm.
|
||||
|
||||
---
|
||||
|
||||
## Operational traps (each cost a run — do not relearn)
|
||||
|
||||
| Trap | Symptom | Rule |
|
||||
|------|---------|------|
|
||||
| **Longhorn storage** | 74–85% quorum-write error rate | PVCs MUST be `local-path` (5Gi). Networked storage on the fsync commit path is fatal to quorum throughput. |
|
||||
| **`kubectl exec` for leader detection** | `container not found`, bogus 17–19s "elections" | Read leader via `curl <VIP>/cluster/status`, never exec into a tidaldb pod (it may be the one dying). |
|
||||
| **`"none (election in progress)"`** | loop tries to delete a pod by that literal string | Filter leader name with `^tidaldb-[0-9]+$`; retry until a real name. |
|
||||
| **`kubectl wait` across force-delete** | hangs past its own `--timeout` | Don't gate the loop on `kubectl wait`. Poll `kubectl get pod -o jsonpath` for `Ready=True`, or bound the wait and move on. |
|
||||
| **Port-forward dies on VIP kill** | curl to `:19500` refuses mid-run | The VIP routed to the killed pod. Re-establish the port-forward before resuming; for failover timing, fall back to the Raft log timestamp. |
|
||||
|
||||
---
|
||||
|
||||
## Next: T4 — elasticity under load (unblocked on m11p5)
|
||||
|
||||
m11p5 added the membership + snapshot-install path T4 needs. **Gate:** scale
|
||||
3→5→3 online under steady load; each joiner reaches `lag=0` within 5 min on a
|
||||
100k-item corpus; p99 impact < 2× baseline for < 60s; zero acked loss.
|
||||
|
||||
m11p5 makes scale-up a plain `kubectl scale` — pod ordinal ≥3 boots with
|
||||
`--seed` (the headless Service resolves to a live peer), joins as a **learner**,
|
||||
and **auto-promotes to a voter** once caught up (statefulset.yaml §ORD≥3 branch).
|
||||
|
||||
```bash
|
||||
# 1. Seed a large corpus first (100k items so catch-up is non-trivial):
|
||||
# one run with --corpus 100000 --ramp smoke, let it seed, then stop.
|
||||
|
||||
# 2. Start steady-state load (leader path, quorum), let it reach a stable stage.
|
||||
kubectl apply -f k8s/stress-job.yaml # peach-100k or a pinned rps:secs
|
||||
|
||||
# 3. Scale up under load and watch the joiners catch up:
|
||||
kubectl scale statefulset tidaldb -n tidaldb-cluster --replicas=5
|
||||
watch 'curl -s localhost:19500/cluster/status | jq ".regions[] | {name, lag_events, partitioned}"'
|
||||
|
||||
# 4. Pass when tidaldb-3 and tidaldb-4 show lag_events=0 within 5 min,
|
||||
# and the stress verdict shows p99 < 2× baseline for < 60s during the join.
|
||||
|
||||
# 5. Scale back down, then run a full ramp to confirm no residual impact:
|
||||
kubectl scale statefulset tidaldb -n tidaldb-cluster --replicas=3
|
||||
```
|
||||
|
||||
Watch for: a joiner on a fresh PVC will snapshot-install (not stream) for a 100k
|
||||
corpus — that's the path under test. Confirm it promotes learner→voter (the
|
||||
region appears in `/cluster/status` as reachable, lag=0) before declaring pass.
|
||||
|
||||
**T4 watch-item — keep load running through the whole join window.** A reseeded
|
||||
node's K8s readiness only flips when it *observes* replication traffic
|
||||
(WORKLOG 2026-06-13 finding: an idle cluster left a fully-caught-up node `0/1`
|
||||
for 11.5h). So T4 must hold write load continuously from before the scale-up
|
||||
until after both joiners are Ready — never let the cluster go idle mid-join, or
|
||||
the readiness gate stalls on a node that is actually caught up. If you see a node
|
||||
with `/cluster/status` lag=0 but pod `0/1`, that's this bug, not a catch-up
|
||||
failure — and it's an **upstream fix**, not a manifest workaround.
|
||||
|
||||
---
|
||||
|
||||
## Then: T-read — the vector-search gate (NOT YET BUILT)
|
||||
|
||||
This is the gap that blocks any honest "competes with vector DBs" claim. Every
|
||||
number to date is write-path. The tool can already drive reads (`--mix reads`,
|
||||
ops `feed`/`search`) and report **latency/throughput/error**, but it has **no
|
||||
recall oracle** — it cannot tell you whether the results are *correct*.
|
||||
|
||||
**What exists today** (latency/throughput only):
|
||||
```bash
|
||||
# Read-latency under load — runnable now, but only answers "how fast", not "how right":
|
||||
# --mix reads --ramp peach-100k --skip-seed --poll-status
|
||||
kubectl apply -f k8s/stress-job.yaml # with mix=reads
|
||||
```
|
||||
|
||||
**What must be built** for a real gate (new work in `src/`):
|
||||
1. **Ground-truth oracle.** For a sample of query vectors, compute exact
|
||||
top-k by brute-force cosine over the seeded corpus (offline or a `--verify`
|
||||
subcommand). Store as `(query_id → [item_id…])`.
|
||||
2. **Recall@k metric.** During the read ramp, issue the same queries against the
|
||||
cluster, compare returned ids to the oracle, emit **recall@10** alongside the
|
||||
existing latency histogram.
|
||||
3. **Query-QPS verdict.** Extend the capacity verdict with a read knee: QPS at
|
||||
which query p99 breaches SLO *while recall@10 stays ≥ target*.
|
||||
|
||||
**Proposed gate:** recall@10 ≥ 0.95 at ≥ X query QPS with query p99 ≤ 50ms, on a
|
||||
128-dim / 20k corpus — then re-run at **1536-dim** (thepeach's real width;
|
||||
~2.3× RAM/item per capacity-planning.md) since that's the production shape.
|
||||
|
||||
Until T-read exists, scope every performance claim to the **write path** and say
|
||||
so explicitly.
|
||||
|
||||
---
|
||||
|
||||
## Checkpoint order from here
|
||||
|
||||
```
|
||||
T3 (done) ─→ T4 elasticity ─→ T-read recall ─┐
|
||||
├─→ T5 sharded throughput (needs p6)
|
||||
build recall oracle first ───────┘
|
||||
```
|
||||
|
||||
T4 is pure ops (no tool change) — do it next. T-read needs `src/` work (the
|
||||
oracle + recall metric) — schedule that build before claiming any read number.
|
||||
T5 stays blocked until p6 lands the shard groups.
|
||||
106
tidal-stress/README.md
Normal file
106
tidal-stress/README.md
Normal file
@ -0,0 +1,106 @@
|
||||
# tidal-stress
|
||||
|
||||
Open-loop capacity ramp + chaos harness for the tidalDB cluster. Drives the
|
||||
`thepeach` feed workload (signals + vector embeddings) against a live cluster and
|
||||
reports a per-stage capacity verdict.
|
||||
|
||||
- **Worklog** (what's been run, what we learned): [WORKLOG.md](WORKLOG.md)
|
||||
- **Process** (how to run the next checkpoint): [PROCESS.md](PROCESS.md)
|
||||
|
||||
---
|
||||
|
||||
## Current target cluster (as of 2026-06-13)
|
||||
|
||||
The cluster moved to the **m11p5 single-StatefulSet architecture**. The old
|
||||
3-StatefulSet / static-ClusterIP model (namespace `tidaldb`, IPs `10.43.99.11-13`)
|
||||
is **retired** — any manifest or doc still naming those IPs is stale.
|
||||
|
||||
| Fact | Value |
|
||||
|------|-------|
|
||||
| Namespace | `tidaldb-cluster` |
|
||||
| Pods | `tidaldb-{0,1,2}` (one StatefulSet, 3 replicas) |
|
||||
| Peer DNS | `tidaldb-N.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500` (HTTP), `:9601` (gRPC) |
|
||||
| Client VIP | `tidaldb.tidaldb-cluster.svc.cluster.local:9500` (readiness-gated) |
|
||||
| Server image | `registry.threesix.ai/tidal/server@sha256:173e803…` (`:m11p5`) |
|
||||
| Stress image | `registry.threesix.ai/tidal/stress@sha256:3a75c311…` (`:m11p3`) |
|
||||
| Storage | local-path 5Gi/pod (on-node NVMe) — **NOT** Longhorn (see WORKLOG) |
|
||||
| CPU/pod | limit `2` (the write pool is ~2 workers on the leader) |
|
||||
|
||||
Targets for any new Job manifest — use pod DNS for `--target` (so status-polling
|
||||
reaches survivors during a kill window) and the VIP for `--leader-url`:
|
||||
|
||||
```yaml
|
||||
args:
|
||||
- --target
|
||||
- http://tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
|
||||
- --target
|
||||
- http://tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
|
||||
- --target
|
||||
- http://tidaldb-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
|
||||
- --leader-url
|
||||
- http://tidaldb.tidaldb-cluster.svc.cluster.local:9500
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Run pattern
|
||||
|
||||
Every run deploys the generator as an in-cluster Job (port-forward adds
|
||||
API-server serialization latency — never use it for capacity numbers; only the
|
||||
kill loop port-forwards, and only to read `/cluster/status`).
|
||||
|
||||
```bash
|
||||
export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
|
||||
kubectl apply -f k8s/<job>.yaml
|
||||
kubectl logs -f job/<job-name> -n tidaldb-cluster
|
||||
kubectl delete job <job-name> -n tidaldb-cluster # re-arm before re-running
|
||||
```
|
||||
|
||||
## CLI flags (authoritative — from `src/main.rs`)
|
||||
|
||||
| Flag | Default | Notes |
|
||||
|------|---------|-------|
|
||||
| `--target <url>` | (required, repeatable) | Region gateway; reads round-robin across all |
|
||||
| `--leader-url <url>` | none | Pin leader-path writes here to skip the forward hop |
|
||||
| `--api-key` | `$TIDAL_API_KEY` | Bearer; the cluster requires it |
|
||||
| `--ack <leader\|quorum>` | topology default | Sent as `x-tidal-ack` per write |
|
||||
| `--ramp <preset\|rps:secs,…>` | `peach-100k` | Presets: `smoke`, `quick`, `peach-100k`, `max` |
|
||||
| `--stage-secs <n>` | 45 | Hold per preset stage; 300–600 for soak |
|
||||
| `--mix <preset\|op=w,…>` | `peach` | Presets: `peach`, `reads`, `writes`. Ops: `feed,search,view,like,skip,item,embed` |
|
||||
| `--write-path <leader\|sharded>` | `leader` | `sharded` removes the single-leader funnel (not replicated) |
|
||||
| `--corpus <n>` | 10000 | Items+embeddings to seed; 20k for gate runs |
|
||||
| `--users <n>` | 50000 | Virtual user id space |
|
||||
| `--skip-seed` | false | Set after the first run of a session (corpus persists on PVC) |
|
||||
| `--embedding-dim <n>` | 128 | Deployed schema = 128; thepeach real = 1536 |
|
||||
| `--hot-skew <f>` | 1.3 | Power-law concentration onto hot items |
|
||||
| `--poll-status` | false | Poll `/cluster/status` between stages for lag — **always set when measuring lag** |
|
||||
| `--stop-on-knee` | false | Stop at first SLO-breaching stage |
|
||||
| `--dau <n>` | 100000 | DAU the verdict translates the ceiling against |
|
||||
|
||||
SLO: feed p99 ≤ 150ms (network-hop allowance over the in-process 50ms SLA);
|
||||
error rate ≥ 1% (429/408/503/5xx/transport) = the knee.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
src/ generator (scheduler, workload model, client, metrics)
|
||||
k8s/ Job manifests — one per checkpoint
|
||||
stress-job.yaml generic ramp
|
||||
stress-job-t2a.yaml T2-A quorum throughput
|
||||
stress-job-t2b.yaml T2-B acked-loss under kills
|
||||
stress-job-t3.yaml T3 automatic-failover gate
|
||||
scripts/
|
||||
t3-kill-loop-v3.sh HTTP-polling leader-kill loop (no exec into pods)
|
||||
```
|
||||
|
||||
## Checkpoint status
|
||||
|
||||
| ID | Gate | Status |
|
||||
|----|------|--------|
|
||||
| T0 | baseline (~90/s replicated, 3669/s sharded) | ✓ done |
|
||||
| T2-A | ≥1000 quorum writes/s | ✓ 2980/s |
|
||||
| T2-B | 0 acked loss across kills | ✓ |
|
||||
| T3 | leader-kill failover <10s p99 ×10 | ✓ max 6157ms (m11p5) |
|
||||
| **T4** | scale 3→5→3 under load, joiner ≤5min | **next** |
|
||||
| T-read | vector-search recall@k + query QPS/p99 | **not built** (see PROCESS) |
|
||||
| T5 | sharded ≥5000 quorum writes/s | blocked on p6 |
|
||||
136
tidal-stress/WORKLOG.md
Normal file
136
tidal-stress/WORKLOG.md
Normal file
@ -0,0 +1,136 @@
|
||||
# tidal-stress worklog
|
||||
|
||||
Newest first. Chronicle of stress-test runs against the live k3s cluster, the
|
||||
gates they cleared, and the operational lessons each one cost.
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-12/13 — m11p5 migration + T3 automatic-failover gate
|
||||
|
||||
### What shipped
|
||||
|
||||
Rolled the cluster from m11p3/p4 (3 StatefulSets, static ClusterIPs) to **m11p5**
|
||||
(1 StatefulSet × 3 replicas, headless-Service DNS, elastic membership). New
|
||||
namespace `tidaldb-cluster`. Then gated **T3** (automatic leader-failover).
|
||||
|
||||
**Result: T3 PASS — 10/10 leader kills, max election 6157ms (gate <10 000ms),
|
||||
zero acked loss, post-ramp lag=0.**
|
||||
|
||||
| Kill | Victim → leader | ms | Kill | Victim → leader | ms |
|
||||
|------|-----------------|----|----|-----------------|----|
|
||||
| 1 | tidaldb-0 → tidaldb-2 | 2752 | 6 | tidaldb-2 → tidaldb-0 | 2656 |
|
||||
| 2 | tidaldb-2 → tidaldb-0 | 4312 | 7 | tidaldb-0 → tidaldb-1 | 3517 |
|
||||
| 3 | tidaldb-0 → tidaldb-2 | 2180 | 8 | tidaldb-1 → tidaldb-0 | 2758¹ |
|
||||
| 4 | tidaldb-2 → tidaldb-0 | 3675 | 9 | tidaldb-0 → tidaldb-1 | 2235 |
|
||||
| 5 | tidaldb-0 → tidaldb-2 | 2168 | 10 | tidaldb-1 → tidaldb-2 | 6157 |
|
||||
|
||||
¹ Kill 8 measured from Raft logs (election won 03:55:42.758, kill 03:55:40) —
|
||||
the port-forward died when the VIP routed to the killed pod, so the loop's own
|
||||
timer recorded a false TIMEOUT. The election itself was clean.
|
||||
|
||||
Stress envelope during the gate: 1500 rps quorum writes, 900s, 1.35M ops,
|
||||
1.51% error (all 503s = quorum rejections inside the ~2–4s kill windows, not
|
||||
acked losses), post-ramp lag=0 on all three nodes.
|
||||
|
||||
### Three things that cost us a run each
|
||||
|
||||
1. **Longhorn storage → 74–85% error rate.** The new cluster first provisioned
|
||||
Longhorn 10Gi PVCs (networked storage). Every quorum write needs follower
|
||||
fsync; over Longhorn that's a network hop on the commit path, which collapsed
|
||||
throughput. Two full T3 runs were invalid before we spotted it. **Fix:**
|
||||
`storageClassName: local-path`, 5Gi — matches the on-node NVMe that T2-A was
|
||||
measured on. The first valid run came straight after the switch.
|
||||
|
||||
2. **`kubectl exec` leader-detection died on the killed pod.** The v1/v2 kill
|
||||
loops exec'd into a pod to read cluster status; when that pod was the one
|
||||
being killed (or mid-initContainer on reboot), exec returned
|
||||
`container not found` and the loop logged bogus 17–19s "election" times. The
|
||||
status API also returns `"none (election in progress)"` as the leader field
|
||||
mid-election, which the loop tried to `kubectl delete`. **Fix:** v3 uses a
|
||||
port-forward to the VIP + `curl /cluster/status` with a strict
|
||||
`^tidaldb-[0-9]+$` leader regex — never execs into a tidaldb pod.
|
||||
|
||||
3. **`kubectl wait --for=condition=Ready` hangs across a force-delete+recreate.**
|
||||
When a pod is force-deleted and the StatefulSet recreates it with the same
|
||||
name, `kubectl wait` latches the old object and blocks past its own
|
||||
`--timeout`. Had to `kill` the hung wait twice (kills 7 and 9) to let the loop
|
||||
advance. Kills 9–10 were driven by hand with a fresh port-forward.
|
||||
|
||||
### The reseed observation (expected, not a bug)
|
||||
|
||||
At 1500 rps a ~50s pod outage opens a WAL gap large enough that the returning
|
||||
pod can't catch up by streaming segments — it latches `reseed_required`
|
||||
(m11p5 §2.4), self-exits, and installs a full snapshot on its next boot.
|
||||
Snapshot install takes ~120–150s, during which the pod is `0/1 Running`. Quorum
|
||||
holds throughout (2/3 voters always up), and **a reseeding pod still
|
||||
participates in elections** — kill 4 elected tidaldb-0 in 3675ms while it was
|
||||
mid-reseed. So the slow K8s readiness is orthogonal to election health; the gate
|
||||
measures election time from the Raft log, not pod readiness.
|
||||
|
||||
### FINDING — reseeded node stays NotReady forever on an idle cluster (m11p5 bug)
|
||||
|
||||
Surfaced 11h after the gate, verifying cluster state. tidaldb-1 (kill 10's
|
||||
victim) installed its snapshot cleanly at **03:59:42** (`snapshot_seq=1441537`)
|
||||
and was a healthy follower from that moment — `/cluster/status` showed it
|
||||
reachable, `lag_events=0`; `/health` returned `ok:true`. But K8s kept it
|
||||
**`0/1 Running` for 11.5 hours**. The readiness flag did not flip to ready until
|
||||
**15:21:25**, when it finally logged `joiner first-converged … readiness is now
|
||||
sticky-ready`.
|
||||
|
||||
Root cause: the install-boot readiness predicate (§4 hysteresis) flips ready only
|
||||
when it *observes* `lag <= learner_promote_lag` through the catch-up
|
||||
receiver/ship path. The stress job ended right after kill 10, so the cluster went
|
||||
idle — **no new events were shipped, so the convergence-observation path was
|
||||
never exercised**, and readiness never flipped. A fully-caught-up node sat out of
|
||||
the client VIP for 11h on a technicality.
|
||||
|
||||
Impact: in production this bites a **scale-up during a quiet period** — the new
|
||||
node installs, catches up, but never joins the client Service until write traffic
|
||||
resumes. It also means T4's joiner-Ready timing is only valid *under sustained
|
||||
load*; an idle-window join would hang the readiness gate.
|
||||
|
||||
This is an **upstream tidaldb fix**, not a stress-harness workaround: the
|
||||
first-converged check should fire on a caught-up node even with zero inbound
|
||||
replication (e.g. evaluate against a leader heartbeat / committed-index compare,
|
||||
not only against observed ship traffic). Do not bandaid it in the manifest.
|
||||
Filed as a T4 watch-item in PROCESS.md.
|
||||
|
||||
### Strategic note (vector-DB positioning)
|
||||
|
||||
Discussed where tidaldb sits vs other databases. CockroachDB is a different
|
||||
animal (distributed SQL) — not the comp. The real comp is **vector DBs**
|
||||
(Qdrant/Milvus/Weaviate/pgvector), and the honest read is:
|
||||
|
||||
- tidaldb's differentiator is **synchronous quorum durability + <6s auto-failover
|
||||
on the write path** — which almost no vector DB has (they treat vectors as a
|
||||
rebuildable derived index and skip write durability). That's a real niche
|
||||
*when the signals are source-of-truth* (thepeach view/like/skip are).
|
||||
- We should **not** race them on ANN QPS/recall/ingest — a dedicated engine wins
|
||||
that and should.
|
||||
- **Gap:** every number we have is write-path only (`--mix writes`). We have
|
||||
zero read-path data — no recall@k, no query QPS/p99. That's exactly the axis
|
||||
vector DBs are judged on, and the tool can measure read latency/throughput but
|
||||
**cannot measure recall** (no ground-truth oracle). Building that is the
|
||||
T-read checkpoint in PROCESS.md.
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-11 — m11p4 regression + m11p3 T2 gate
|
||||
|
||||
m11p3 (quorum acks, one replicated log) cleared **T2-A** (2980 quorum writes/s,
|
||||
p99=49ms, knee ~5k rps) and **T2-B** (3 leader kills, lag=0 after each, zero
|
||||
acked loss). m11p4 (catch-up timer retry + WAL TSEG segment versioning) held the
|
||||
gate with no regression (2975 writes/s, −0.2%). Full results:
|
||||
`docs/ops/stress-test-p3-t2.md`, `docs/ops/stress-test-p4-regression.md`.
|
||||
|
||||
This was still on the **old 3-StatefulSet / static-ClusterIP** architecture —
|
||||
the `10.43.99.11-13` IPs in `tmp/stress-test-plan.md` date from here and are now
|
||||
stale (see PROCESS.md for the migration).
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-10 — T0 baseline
|
||||
|
||||
Replicated `/signals` ~90/s (WAL fsync + synchronous gRPC ship per write, ~2
|
||||
workers on a cpu-2 pod); sharded `/sharded/signals` 3669/s at 0 errors, knee not
|
||||
reached. Full results: `docs/ops/stress-test-thepeach.md`.
|
||||
Loading…
Reference in New Issue
Block a user