95461d3cf8
Kind-3 term markers in the WAL stream, STREAM-relative vote frontiers, heartbeat-only divergence detection + quarantine, and fenced promote. Elections converge in 0.6–1.0s; zero acked-write loss across all kill points. Closes G5 (leaderless recovery) from the v0.9 wave.
2.4 KiB
Stress test regression — p4 (m11p4) (2026-06-11)
Live 3-StatefulSet k3s cluster, m11p4 image (sha256:86dd59c4...). Fresh
deployment over existing PVCs (no reset needed — m11p4 WAL reader accepts m11p3
segments as implicit v0). All 3 pods Running, lag=0 at test start.
Catch-up self-healing validation
Validates the m11p4 fix: failed/missed catch-up pulls now retry on a 30s timer rather than waiting for the next push event.
Procedure:
- Confirmed baseline lag=0 on all 3 nodes.
- Killed tidaldb-eu-west-0; wrote 500 events to leader (ack=leader) while pod was down. Pod restarted via StatefulSet (fast restart).
- Stopped all writes with eu-west at lag=2,035,769.
- Polled
/cluster/statusevery 10s — zero writes issued.
Result:
| t+ | eu-west lag |
|---|---|
| 0s (write-stop) | 2,035,769 |
| 10s | 0 |
eu-west caught up to lag=0 within 10s of pod start with no new writes to the cluster. Prior to m11p4, this scenario left the follower stuck indefinitely (the 2026-06-11 p3 rollout incident: both followers stuck at lag=136507 until the cluster received new writes).
Verdict: timer-triggered catch-up CONFIRMED.
Throughput regression (p3 gate re-check)
tidal-stress open-loop, --ack quorum, --mix writes, peach-100k ramp,
60s/stage, 20k corpus, 100k users, --stop-on-knee.
| Stage | Target rps | Signal ok/s | Error rate | p99 write | Max lag | Pass |
|---|---|---|---|---|---|---|
| 1 | 50 | 50 | 0.00% | 35 ms | 0 | ✓ |
| 2 | 150 | 150 | 0.00% | 35 ms | 0 | ✓ |
| 3 | 400 | 400 | 0.00% | 37 ms | 0 | ✓ |
| 4 | 800 | 800 | 0.00% | 43 ms | 8 | ✓ |
| 5 | 1,500 | 1,499 | 0.04% | 41 ms | 0 | ✓ |
| 6 | 3,000 | 2,975 | 0.82% | 49 ms | 27 | ✓ highest |
| 7 ⚠ | 5,000 | 4,411 | 11.76% | 164 ms | 24 | ✗ knee |
No 503 quorum timeouts at any stage.
Regression verdict
| Metric | T2-A (m11p3) | Regression (m11p4) | Delta | Result |
|---|---|---|---|---|
| Quorum signal-writes/s | 2,980/s | 2,975/s | −0.2% | ✓ no regression |
| Write p99 at stage 6 | 49 ms | 49 ms | 0 | ✓ no regression |
| Error rate at stage 6 | 0.67% | 0.82% | +0.15% | ✓ within gate (<1%) |
| Capacity knee | stage 7 (5k) | stage 7 (5k) | none | ✓ no regression |
| Lag at stage 6 | 23 events | 27 events | +4 | ✓ within gate (≤2s) |
p3 gate (≥1,000/s quorum, p99 ≤50ms, error <1%) holds through m11p4.