1b5bcbacd7
A client-side ship DEADLINE means the RPC did not round-trip within
request_timeout — which a slow-but-ALIVE follower produces under a sustained
1536-D ack=quorum apply burst (transport runtime momentarily starved by the
CPU-heavy HNSW apply on its single segment-receiver thread) exactly as a
genuinely blackholed peer does. Counting that as record_failure was the
write-burst false-partition: 5 such opened both followers' breakers, the commit
index stalled, ack=quorum 503'd, and retries re-burst the same starved peers
with no self-heal.
- CircuitBreaker::record_timeout: opens ONLY when the peer shows no recent proof
of life (no round-tripped success/backpressure within reset_duration); neutral
no-op when liveness is fresh; re-opens (never wedges) HalfOpen; never refreshes
the liveness stamp (no reply arrived).
- PeerPool::send_to routes tonic DeadlineExceeded/Cancelled -> record_timeout;
genuine severance still surfaces as connect-level Unavailable/transport reset
-> record_failure and still opens the breaker.
- ship_timeout_breaker.rs: end-to-end proof over a REAL tonic WalShipping server
(handler succeeds once then hangs past the client deadline) + 6 unit tests.
Also: re-scope G-S Scalability guarantee to read-throughput with the Ref-A
tidal-t5-readtput owner-test (write 2.5x is structurally impossible on 3-node
full-placement RF3); bump k8s image to m12-rc6 (live, commit
|
||
|---|---|---|
| .. | ||
| catchup_retry.rs | ||
| election_rpc.rs | ||
| large_payload.rs | ||
| mtls.rs | ||
| multi_node_uat.rs | ||
| reconnection.rs | ||
| ship_timeout_breaker.rs | ||
| snapshot_fetch.rs | ||
| transport_contract.rs | ||