fix(cluster): reconcile could not run at production scale

The three live voters disagree on signal aggregates for the same entity
(view = 10003 / 10095 / 10144 for entity 1, stable across passes) while
`/cluster/status` reports applied_events equal, lag_events 0, and no divergence
quarantine. The documented remedy is `POST /cluster/reconcile`. On this corpus
it fails:

    503 region 'tidaldb-1' unreachable:
        reconcile peer returned 413 Payload Too Large

Two defects, both fixed here:

- The whole-shard CRDT `StateSnapshot` was capped by `BODY_LIMIT_BYTES`, the
  2 MiB limit sized for one client write on the public data surface. The
  snapshot carries one entry per entity x signal type; on 33k documents it is
  several MiB, so divergence was unhealable in production. The internal,
  marker-pinned, operator-driven snapshot route now has its own explicit
  ceiling.
- A 413 was reported as `RegionUnreachable`. The peer answered - it is
  reachable and healthy - so the error sent the operator to TLS and
  NetworkPolicy. It now names the measured snapshot size, the peer's cap, and
  the fix.

The ceiling is not the design: the snapshot grows with the corpus and chunked
reconcile is the durable answer. Documented as such at the constant.
This commit is contained in:
jordan 2026-08-18 10:07:19 -06:00
parent 3eaf28bf8a
commit 2e1484226c
2 changed files with 44 additions and 1 deletions

View File

@ -4872,9 +4872,15 @@ pub fn build_region_router(
// m11p6 L3 rebalancing verbs (per-group, reusing the m11p5 machinery). // m11p6 L3 rebalancing verbs (per-group, reusing the m11p5 machinery).
.route("/cluster/shards/{id}/replicas", post(shard_replicas)) .route("/cluster/shards/{id}/replicas", post(shard_replicas))
.route("/cluster/shards/{id}/transfer", post(shard_transfer)) .route("/cluster/shards/{id}/transfer", post(shard_transfer))
// The snapshot body is corpus-sized, so it gets its own cap BEFORE the
// group's data-surface limit applies (an inner layer wins). See
// `RECONCILE_BODY_LIMIT_BYTES` for why the shared 2 MiB made
// divergence unhealable on the live cluster.
.route( .route(
"/cluster/reconcile/snapshot", "/cluster/reconcile/snapshot",
post(cluster_reconcile_snapshot), post(cluster_reconcile_snapshot).layer(axum::extract::DefaultBodyLimit::max(
crate::router::RECONCILE_BODY_LIMIT_BYTES,
)),
) )
.route("/sharded/items", post(sharded_create_item)) .route("/sharded/items", post(sharded_create_item))
.route("/sharded/embeddings", post(sharded_write_embedding)) .route("/sharded/embeddings", post(sharded_write_embedding))
@ -8342,6 +8348,20 @@ pub async fn cluster_reconcile(
.await .await
{ {
Ok(resp) if resp.status.is_success() => resp.body, Ok(resp) if resp.status.is_success() => resp.body,
// A 413 is NOT unreachability: the peer answered, and said the payload
// is too big. Reported as such it sent an operator hunting TLS and
// NetworkPolicy while the cluster was perfectly connected. Name the
// measured size, the cap, and the fix.
Ok(resp) if resp.status == reqwest::StatusCode::PAYLOAD_TOO_LARGE => {
let bytes = serde_json::to_vec(&local_snapshot).map_or(0, |v| v.len());
return Err(ClusterAppError(ServerError::Cluster(format!(
"reconcile snapshot is {bytes} bytes; peer '{}' capped it at {} bytes. The \
corpus outgrew the single-shot reconcile the peer is reachable and healthy. \
Raise RECONCILE_BODY_LIMIT_BYTES on BOTH nodes, or chunk the exchange.",
req.region,
crate::router::RECONCILE_BODY_LIMIT_BYTES,
))));
}
Ok(resp) => { Ok(resp) => {
return Err(ClusterAppError(ServerError::RegionUnreachable { return Err(ClusterAppError(ServerError::RegionUnreachable {
region: req.region, region: req.region,

View File

@ -45,6 +45,29 @@ use crate::{
/// one and forgetting the other). /// one and forgetting the other).
pub(crate) const BODY_LIMIT_BYTES: usize = 2 * 1024 * 1024; pub(crate) const BODY_LIMIT_BYTES: usize = 2 * 1024 * 1024;
/// Body cap for the ONE internal node-to-node control payload that carries
/// state proportional to the corpus: `POST /cluster/reconcile/snapshot`
/// exchanges a whole-shard CRDT `StateSnapshot` (one entry per entity ×
/// signal type, each carrying per-node contributions).
///
/// It is NOT [`BODY_LIMIT_BYTES`]. The 2 MiB data-surface cap is sized for a
/// single client write; measured against the fleet's live 33k-document,
/// 13.3M-event cluster the snapshot is several MiB, so the shared cap made
/// divergence unhealable in production: `POST /cluster/reconcile` failed with
/// `503 region unreachable: reconcile peer returned 413 Payload Too Large`,
/// which reads like a network fault and sent the operator to TLS and
/// NetworkPolicy first.
///
/// This route is internal (`x-tidal-internal` marker), authenticated, and
/// driven only by an operator verb, so a large body here is a control-plane
/// cost, not an exposed DoS surface.
///
/// This is a CEILING, not a design: the snapshot grows with the corpus and at
/// ~1M entities it will outgrow this too. The durable fix is a chunked
/// reconcile (cursor over entity ranges, merge per chunk) — until then the
/// sender pre-checks its own snapshot size and says so plainly.
pub(crate) const RECONCILE_BODY_LIMIT_BYTES: usize = 64 * 1024 * 1024;
/// Maximum wall-clock time a single request may occupy. Exceeded requests /// Maximum wall-clock time a single request may occupy. Exceeded requests
/// receive 408 Request Timeout. /// receive 408 Request Timeout.
/// ///