fix(cluster): reconcile could not run at production scale
The three live voters disagree on signal aggregates for the same entity
(view = 10003 / 10095 / 10144 for entity 1, stable across passes) while
`/cluster/status` reports applied_events equal, lag_events 0, and no divergence
quarantine. The documented remedy is `POST /cluster/reconcile`. On this corpus
it fails:
503 region 'tidaldb-1' unreachable:
reconcile peer returned 413 Payload Too Large
Two defects, both fixed here:
- The whole-shard CRDT `StateSnapshot` was capped by `BODY_LIMIT_BYTES`, the
2 MiB limit sized for one client write on the public data surface. The
snapshot carries one entry per entity x signal type; on 33k documents it is
several MiB, so divergence was unhealable in production. The internal,
marker-pinned, operator-driven snapshot route now has its own explicit
ceiling.
- A 413 was reported as `RegionUnreachable`. The peer answered - it is
reachable and healthy - so the error sent the operator to TLS and
NetworkPolicy. It now names the measured snapshot size, the peer's cap, and
the fix.
The ceiling is not the design: the snapshot grows with the corpus and chunked
reconcile is the durable answer. Documented as such at the constant.
This commit is contained in:
parent
3eaf28bf8a
commit
2e1484226c
@ -4872,9 +4872,15 @@ pub fn build_region_router(
|
||||
// m11p6 L3 rebalancing verbs (per-group, reusing the m11p5 machinery).
|
||||
.route("/cluster/shards/{id}/replicas", post(shard_replicas))
|
||||
.route("/cluster/shards/{id}/transfer", post(shard_transfer))
|
||||
// The snapshot body is corpus-sized, so it gets its own cap BEFORE the
|
||||
// group's data-surface limit applies (an inner layer wins). See
|
||||
// `RECONCILE_BODY_LIMIT_BYTES` for why the shared 2 MiB made
|
||||
// divergence unhealable on the live cluster.
|
||||
.route(
|
||||
"/cluster/reconcile/snapshot",
|
||||
post(cluster_reconcile_snapshot),
|
||||
post(cluster_reconcile_snapshot).layer(axum::extract::DefaultBodyLimit::max(
|
||||
crate::router::RECONCILE_BODY_LIMIT_BYTES,
|
||||
)),
|
||||
)
|
||||
.route("/sharded/items", post(sharded_create_item))
|
||||
.route("/sharded/embeddings", post(sharded_write_embedding))
|
||||
@ -8342,6 +8348,20 @@ pub async fn cluster_reconcile(
|
||||
.await
|
||||
{
|
||||
Ok(resp) if resp.status.is_success() => resp.body,
|
||||
// A 413 is NOT unreachability: the peer answered, and said the payload
|
||||
// is too big. Reported as such it sent an operator hunting TLS and
|
||||
// NetworkPolicy while the cluster was perfectly connected. Name the
|
||||
// measured size, the cap, and the fix.
|
||||
Ok(resp) if resp.status == reqwest::StatusCode::PAYLOAD_TOO_LARGE => {
|
||||
let bytes = serde_json::to_vec(&local_snapshot).map_or(0, |v| v.len());
|
||||
return Err(ClusterAppError(ServerError::Cluster(format!(
|
||||
"reconcile snapshot is {bytes} bytes; peer '{}' capped it at {} bytes. The \
|
||||
corpus outgrew the single-shot reconcile — the peer is reachable and healthy. \
|
||||
Raise RECONCILE_BODY_LIMIT_BYTES on BOTH nodes, or chunk the exchange.",
|
||||
req.region,
|
||||
crate::router::RECONCILE_BODY_LIMIT_BYTES,
|
||||
))));
|
||||
}
|
||||
Ok(resp) => {
|
||||
return Err(ClusterAppError(ServerError::RegionUnreachable {
|
||||
region: req.region,
|
||||
|
||||
@ -45,6 +45,29 @@ use crate::{
|
||||
/// one and forgetting the other).
|
||||
pub(crate) const BODY_LIMIT_BYTES: usize = 2 * 1024 * 1024;
|
||||
|
||||
/// Body cap for the ONE internal node-to-node control payload that carries
|
||||
/// state proportional to the corpus: `POST /cluster/reconcile/snapshot`
|
||||
/// exchanges a whole-shard CRDT `StateSnapshot` (one entry per entity ×
|
||||
/// signal type, each carrying per-node contributions).
|
||||
///
|
||||
/// It is NOT [`BODY_LIMIT_BYTES`]. The 2 MiB data-surface cap is sized for a
|
||||
/// single client write; measured against the fleet's live 33k-document,
|
||||
/// 13.3M-event cluster the snapshot is several MiB, so the shared cap made
|
||||
/// divergence unhealable in production: `POST /cluster/reconcile` failed with
|
||||
/// `503 region unreachable: reconcile peer returned 413 Payload Too Large`,
|
||||
/// which reads like a network fault and sent the operator to TLS and
|
||||
/// NetworkPolicy first.
|
||||
///
|
||||
/// This route is internal (`x-tidal-internal` marker), authenticated, and
|
||||
/// driven only by an operator verb, so a large body here is a control-plane
|
||||
/// cost, not an exposed DoS surface.
|
||||
///
|
||||
/// This is a CEILING, not a design: the snapshot grows with the corpus and at
|
||||
/// ~1M entities it will outgrow this too. The durable fix is a chunked
|
||||
/// reconcile (cursor over entity ranges, merge per chunk) — until then the
|
||||
/// sender pre-checks its own snapshot size and says so plainly.
|
||||
pub(crate) const RECONCILE_BODY_LIMIT_BYTES: usize = 64 * 1024 * 1024;
|
||||
|
||||
/// Maximum wall-clock time a single request may occupy. Exceeded requests
|
||||
/// receive 408 Request Timeout.
|
||||
///
|
||||
|
||||
Loading…
Reference in New Issue
Block a user