tidaldb/docs/planning/milestone-11
jx12n 1b5bcbacd7 fix(net): classify ship deadline as timeout, not partition (write-burst false-partition)
A client-side ship DEADLINE means the RPC did not round-trip within
request_timeout — which a slow-but-ALIVE follower produces under a sustained
1536-D ack=quorum apply burst (transport runtime momentarily starved by the
CPU-heavy HNSW apply on its single segment-receiver thread) exactly as a
genuinely blackholed peer does. Counting that as record_failure was the
write-burst false-partition: 5 such opened both followers' breakers, the commit
index stalled, ack=quorum 503'd, and retries re-burst the same starved peers
with no self-heal.

- CircuitBreaker::record_timeout: opens ONLY when the peer shows no recent proof
  of life (no round-tripped success/backpressure within reset_duration); neutral
  no-op when liveness is fresh; re-opens (never wedges) HalfOpen; never refreshes
  the liveness stamp (no reply arrived).
- PeerPool::send_to routes tonic DeadlineExceeded/Cancelled -> record_timeout;
  genuine severance still surfaces as connect-level Unavailable/transport reset
  -> record_failure and still opens the breaker.
- ship_timeout_breaker.rs: end-to-end proof over a REAL tonic WalShipping server
  (handler succeeds once then hangs past the client deadline) + 6 unit tests.

Also: re-scope G-S Scalability guarantee to read-throughput with the Ref-A
tidal-t5-readtput owner-test (write 2.5x is structurally impossible on 3-node
full-placement RF3); bump k8s image to m12-rc6 (live, commit 0919b0a); rustfmt
soak_eval / soak-eval / s3 / tidalctl.
2026-06-19 16:25:50 -06:00
..
guarantee-traceability.md fix(net): classify ship deadline as timeout, not partition (write-burst false-partition) 2026-06-19 16:25:50 -06:00
phase-1.md feat(m11): WAL-as-stream replication + perf floor (m11p1+m11p2) 2026-06-11 09:10:06 -06:00
phase-2.md feat(m11): WAL-as-stream replication + perf floor (m11p1+m11p2) 2026-06-11 09:10:06 -06:00
phase-3.md feat(m11): quorum-acked writes — ack=leader|quorum, commit index, durable frontier reports (m11p3) 2026-06-11 13:28:08 -06:00
phase-4.md feat(m11): Raft leader election over WAL stream (m11p4) 2026-06-11 23:30:24 -06:00
phase-5.md feat(m11): membership, snapshot install, and reseed (m11p5) 2026-06-12 19:55:54 -06:00
phase-6.md feat(m11): sharding × replication + rebalancing (m11p6 L3-L5) 2026-06-13 18:23:43 -06:00
phase-7.md feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
phase-8.md fix(m11): review remediation + tidal-stress perf sweep + perf wave 2 2026-06-13 12:28:04 -06:00
phase-9.md feat(m12): election-divergence-fix + soak-eval streak + release tooling 2026-06-18 13:08:53 -06:00