End the "replicated XOR sharded" split: S shard groups, each a
replication group at RF with its own elected leader, leaders balanced
across nodes; any gateway hash-routes.
- One unified write surface: /items,/embeddings,/signals hash-route to
the owning shard group's leader (ShardRouter FNV-1a) AND replicate at
RF. x-tidal-ack/x-tidal-seq, quorum await, NotLeader/QuorumTimeout are
per-group; NotLeader names the group.
- Rebalance verbs (L3): POST /cluster/shards/{id}/transfer (fenced
leadership move) + /cluster/shards/{id}/replicas (add/remove replica).
A ?shard= selector threads through every per-shard admin verb and is
propagated on intra-group forwards (ShardReplica::admin_path). S=1 is
byte-for-byte (no selector, no shard in NotLeader body).
- Tier-3 exit gate (cluster_sharding.rs): 3 nodes × 3 shards × RF=3 over
real OS processes — SIGKILL a node under ack=quorum load → only its
shard-leaderships re-elect, reads never stop, zero acked loss across
random kill points; plus a rebalance-verb test. Harness:
MultiProcCluster::start_sharded.
- tidal-stress drives the single path (WritePath::Leader|Sharded gone),
spreading writes round-robin across gateways or pinning --leader-url.
- Throughput: local 3×3 sustains 3,000 quorum signal-writes/s @ 0% err,
~30% CPU, lag ~0 (generator-bound). ≥5,000/s + ≥2.5× scaling is Ref-A.
Known follow-up (tracked): per-group-aware node readiness and cross-node
read fan-out under PARTIAL placement.
ClusterNode hosts a BTreeMap<ShardId, Arc<ShardReplica>>: writes hash-route
to the owning shard leader, reads scatter over shard groups. In-group
shard==region preserved so the engine and tidal-net are untouched; S=1 stays
byte-for-byte (today's cluster is a 1-shard × RF=N group). Topology grows
shard-group awareness; membership, election, forward, reseed, and join_boot
thread ShardId through.
Proven by an in-process 2×2 RF=2 gRPC test plus S=1 parity, incl. tier-3
real-OS-process failover. clippy/fmt clean.
