95461d3cf8
Kind-3 term markers in the WAL stream, STREAM-relative vote frontiers, heartbeat-only divergence detection + quarantine, and fenced promote. Elections converge in 0.6–1.0s; zero acked-write loss across all kill points. Closes G5 (leaderless recovery) from the v0.9 wave.
65 lines
2.4 KiB
Markdown
65 lines
2.4 KiB
Markdown
# Stress test regression — p4 (m11p4) (2026-06-11)
|
||
|
||
Live 3-StatefulSet k3s cluster, m11p4 image (`sha256:86dd59c4...`). Fresh
|
||
deployment over existing PVCs (no reset needed — m11p4 WAL reader accepts m11p3
|
||
segments as implicit v0). All 3 pods Running, lag=0 at test start.
|
||
|
||
---
|
||
|
||
## Catch-up self-healing validation
|
||
|
||
Validates the m11p4 fix: failed/missed catch-up pulls now retry on a 30s timer
|
||
rather than waiting for the next push event.
|
||
|
||
**Procedure:**
|
||
1. Confirmed baseline lag=0 on all 3 nodes.
|
||
2. Killed tidaldb-eu-west-0; wrote 500 events to leader (ack=leader) while pod
|
||
was down. Pod restarted via StatefulSet (fast restart).
|
||
3. Stopped all writes with eu-west at lag=2,035,769.
|
||
4. Polled `/cluster/status` every 10s — zero writes issued.
|
||
|
||
**Result:**
|
||
|
||
| t+ | eu-west lag |
|
||
|---|---|
|
||
| 0s (write-stop) | 2,035,769 |
|
||
| 10s | **0** |
|
||
|
||
eu-west caught up to lag=0 within 10s of pod start with no new writes to the
|
||
cluster. Prior to m11p4, this scenario left the follower stuck indefinitely
|
||
(the 2026-06-11 p3 rollout incident: both followers stuck at lag=136507 until
|
||
the cluster received new writes).
|
||
|
||
**Verdict: timer-triggered catch-up CONFIRMED.**
|
||
|
||
---
|
||
|
||
## Throughput regression (p3 gate re-check)
|
||
|
||
`tidal-stress` open-loop, `--ack quorum`, `--mix writes`, `peach-100k` ramp,
|
||
60s/stage, 20k corpus, 100k users, `--stop-on-knee`.
|
||
|
||
| Stage | Target rps | Signal ok/s | Error rate | p99 write | Max lag | Pass |
|
||
|---|---|---|---|---|---|---|
|
||
| 1 | 50 | 50 | 0.00% | 35 ms | 0 | ✓ |
|
||
| 2 | 150 | 150 | 0.00% | 35 ms | 0 | ✓ |
|
||
| 3 | 400 | 400 | 0.00% | 37 ms | 0 | ✓ |
|
||
| 4 | 800 | 800 | 0.00% | 43 ms | 8 | ✓ |
|
||
| 5 | 1,500 | 1,499 | 0.04% | 41 ms | 0 | ✓ |
|
||
| **6** | **3,000** | **2,975** | **0.82%** | **49 ms** | **27** | **✓ highest** |
|
||
| 7 ⚠ | 5,000 | 4,411 | 11.76% | 164 ms | 24 | ✗ knee |
|
||
|
||
No 503 quorum timeouts at any stage.
|
||
|
||
### Regression verdict
|
||
|
||
| Metric | T2-A (m11p3) | Regression (m11p4) | Delta | Result |
|
||
|---|---|---|---|---|
|
||
| Quorum signal-writes/s | 2,980/s | **2,975/s** | −0.2% | **✓ no regression** |
|
||
| Write p99 at stage 6 | 49 ms | **49 ms** | 0 | **✓ no regression** |
|
||
| Error rate at stage 6 | 0.67% | **0.82%** | +0.15% | **✓ within gate (<1%)** |
|
||
| Capacity knee | stage 7 (5k) | **stage 7 (5k)** | none | **✓ no regression** |
|
||
| Lag at stage 6 | 23 events | **27 events** | +4 | **✓ within gate (≤2s)** |
|
||
|
||
**p3 gate (≥1,000/s quorum, p99 ≤50ms, error <1%) holds through m11p4.**
|