Root cause: SyncStorageBridge returned bind addresses (0.0.0.0) in
PingResponse, causing peers to overwrite each other's addresses with
loopback. Subsequent probes connected to self, registering self as a
member (node_count: 4 instead of 3).
Two fixes:
- alive_node() now skips self (node_id == local_id guard)
- Advertise POD_IP (k8s downward API) in PingResponse instead of bind addr
- handle_health: use all_members() filtered to Alive state instead of
members() which returned 0 when nodes were in Suspect state
- handle_cluster_status: add +1 for self since members map excludes local node
Merged 10 upstream commits (MemTable, read-your-writes tests, feed endpoint,
security hardening, signed assertions, source registry, dashboard enhancements)
and fixed all test failures across the full workspace (2656/2656 passing).
Key fixes:
- fix(cluster): DashMap deadlock in swim.rs suspect_node/fail_node/alive_node
- DashMap::get_mut RefMut + iter() on same map = non-reentrant write lock deadlock
- Fix: extract clone in scoped block to drop RefMut before calling update_node_gauges()
- 6 previously-hanging SWIM tests now pass in <2s
- fix(sim): replace background-task+polling ingestion with synchronous process_pending()
- smoke_high_volume_simulation was CPU-starved under 2656 parallel tests
- Removed ingestor.start() + wait_until_ingested() pattern throughout sim
- All arena functions now call ingestor.process_pending() directly (deterministic)
- fix(test): v2 signature helper used wrong hash (rkyv vs canonical compute_content_hash_v2)
- fix(test): quota test signed "test" but v1 requires "subject:predicate" format
- fix(test): http_validation now accepts 400 for valid-format-but-invalid-crypto hex
- fix(test): scale_adaptive micro tier assertions updated (auto_promote upstream change)
- config: add nextest.toml with slow-timeout for background-task-tests group
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add stemedb-cluster crate implementing horizontal scaling:
- SWIM-based membership protocol for node discovery and failure detection
- Consistent hashing (jump hash) for subject-to-shard routing
- Range management with dynamic split (>64MB) and merge (<20MB) operations
- Stateless HTTP gateway for client request routing via axum
- Meta-range gossip merge for cluster-wide metadata propagation
Includes restrictive CORS policy, proper error propagation from routing,
replica cache invalidation on node failure, and 84 tests (57 unit + 27
integration). Raft MV coordination deferred per design decision.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
