44b768b8c6
End the "replicated XOR sharded" split: S shard groups, each a
replication group at RF with its own elected leader, leaders balanced
across nodes; any gateway hash-routes.
- One unified write surface: /items,/embeddings,/signals hash-route to
the owning shard group's leader (ShardRouter FNV-1a) AND replicate at
RF. x-tidal-ack/x-tidal-seq, quorum await, NotLeader/QuorumTimeout are
per-group; NotLeader names the group.
- Rebalance verbs (L3): POST /cluster/shards/{id}/transfer (fenced
leadership move) + /cluster/shards/{id}/replicas (add/remove replica).
A ?shard= selector threads through every per-shard admin verb and is
propagated on intra-group forwards (ShardReplica::admin_path). S=1 is
byte-for-byte (no selector, no shard in NotLeader body).
- Tier-3 exit gate (cluster_sharding.rs): 3 nodes × 3 shards × RF=3 over
real OS processes — SIGKILL a node under ack=quorum load → only its
shard-leaderships re-elect, reads never stop, zero acked loss across
random kill points; plus a rebalance-verb test. Harness:
MultiProcCluster::start_sharded.
- tidal-stress drives the single path (WritePath::Leader|Sharded gone),
spreading writes round-robin across gateways or pinning --leader-url.
- Throughput: local 3×3 sustains 3,000 quorum signal-writes/s @ 0% err,
~30% CPU, lag ~0 (generator-bound). ≥5,000/s + ≥2.5× scaling is Ref-A.
Known follow-up (tracked): per-group-aware node readiness and cross-node
read fan-out under PARTIAL placement.
104 lines
4.5 KiB
Rust
104 lines
4.5 KiB
Rust
use std::path::PathBuf;
|
|
|
|
use thiserror::Error;
|
|
|
|
pub type Result<T, E = ServerError> = std::result::Result<T, E>;
|
|
|
|
#[derive(Debug, Error)]
|
|
pub enum ServerError {
|
|
/// A filesystem read that knows WHICH path failed — config / topology /
|
|
/// schema file loads carry the path so the operator sees the offending file.
|
|
/// Constructed via [`ServerError::io`]; never `?`-converted (that path lacks
|
|
/// a path to attach).
|
|
#[error("failed to read {path}: {source}")]
|
|
Io {
|
|
path: PathBuf,
|
|
source: std::io::Error,
|
|
},
|
|
#[error("invalid schema config: {0}")]
|
|
SchemaConfig(String),
|
|
#[error("schema build failed: {0}")]
|
|
SchemaBuild(#[from] tidaldb::schema::SchemaError),
|
|
#[error("tidalDB error: {0}")]
|
|
Tidal(#[from] tidaldb::TidalError),
|
|
/// A pathless `std::io::Error` from socket setup / serving (TCP bind,
|
|
/// `local_addr`, `axum::serve`). This is the `?`-conversion target for
|
|
/// `std::io::Error`; the path-tagged file-read case uses [`ServerError::Io`]
|
|
/// instead. Named for its source (the network/serve stack) rather than the
|
|
/// old misleading "Http", which read as an HTTP-protocol error.
|
|
#[error("network/serve I/O error: {0}")]
|
|
Network(#[from] std::io::Error),
|
|
#[error("bad request: {0}")]
|
|
BadRequest(String),
|
|
#[error("cluster mode is experimental and disabled: {0}")]
|
|
ExperimentalDisabled(String),
|
|
#[error("cluster error: {0}")]
|
|
Cluster(String),
|
|
/// The server is shutting down and the cluster fabric has been taken; map to
|
|
/// 503 so a post-shutdown access degrades cleanly instead of panicking.
|
|
#[error("service unavailable: {0}")]
|
|
Unavailable(String),
|
|
/// A write reached a non-leader region node in multi-process cluster mode.
|
|
/// Maps to 503 with a JSON body naming the leader (and its HTTP address when
|
|
/// known) so the node is honest standalone; task 03 upgrades this to
|
|
/// transparent leader forwarding.
|
|
#[error("not the leader; current leader is '{leader}' (term {term})")]
|
|
NotLeader {
|
|
leader: String,
|
|
http_addr: Option<String>,
|
|
/// The responder's election term (m11p4): lets a forwarder tell a
|
|
/// stale answer from a fresh one during election churn.
|
|
term: u64,
|
|
/// The shard group this not-leader answer is for (m11p6). `None` for the
|
|
/// legacy single group (S=1, byte-for-byte). Names the group in the body
|
|
/// so a client retrying a per-shard write learns which group rejected it
|
|
/// and which `?shard=` to re-target.
|
|
shard: Option<u16>,
|
|
},
|
|
/// A region-pinned read named a region this process does not own (multi-
|
|
/// process cluster mode). Maps to 400 naming the region; task 03 upgrades
|
|
/// this to forwarding to the owning process.
|
|
#[error("region '{region}' is not served by this node")]
|
|
NotLocal { region: String },
|
|
/// The current leader could not be reached while forwarding a write
|
|
/// (multi-process cluster mode). Maps to 503 with a body naming the leader,
|
|
/// its HTTP address, and the transport cause, so the client can retry or
|
|
/// re-target the leader directly.
|
|
#[error("leader '{leader}' unreachable while forwarding: {cause}")]
|
|
LeaderUnreachable {
|
|
leader: String,
|
|
http_addr: String,
|
|
cause: String,
|
|
},
|
|
/// A sibling region could not be reached (region-pinned read forward,
|
|
/// sharded write to its owner, or the reconcile snapshot exchange). Maps to
|
|
/// 503 naming the region and the transport cause.
|
|
#[error("region '{region}' unreachable: {cause}")]
|
|
RegionUnreachable { region: String, cause: String },
|
|
/// An `ack=quorum` write was durable on the leader but a majority of the
|
|
/// replica set did not confirm durability within the quorum budget
|
|
/// (m11p3). Maps to a retryable 503 naming the laggards. The write is in
|
|
/// the leader's log and MAY still commit — retries are at-least-once
|
|
/// (see runbook §8 for the dedup guidance).
|
|
#[error(
|
|
"quorum not reached for seqno {seq} within budget: {confirmed} of {needed} required \
|
|
follower confirmations (commit index {committed}); laggards: {laggards:?}"
|
|
)]
|
|
QuorumTimeout {
|
|
seq: u64,
|
|
needed: usize,
|
|
confirmed: usize,
|
|
committed: u64,
|
|
laggards: Vec<String>,
|
|
},
|
|
}
|
|
|
|
impl ServerError {
|
|
pub fn io(path: impl Into<PathBuf>, source: std::io::Error) -> Self {
|
|
Self::Io {
|
|
path: path.into(),
|
|
source,
|
|
}
|
|
}
|
|
}
|