tidaldb/tidal-server
jordan 2e1484226c fix(cluster): reconcile could not run at production scale
The three live voters disagree on signal aggregates for the same entity
(view = 10003 / 10095 / 10144 for entity 1, stable across passes) while
`/cluster/status` reports applied_events equal, lag_events 0, and no divergence
quarantine. The documented remedy is `POST /cluster/reconcile`. On this corpus
it fails:

    503 region 'tidaldb-1' unreachable:
        reconcile peer returned 413 Payload Too Large

Two defects, both fixed here:

- The whole-shard CRDT `StateSnapshot` was capped by `BODY_LIMIT_BYTES`, the
  2 MiB limit sized for one client write on the public data surface. The
  snapshot carries one entry per entity x signal type; on 33k documents it is
  several MiB, so divergence was unhealable in production. The internal,
  marker-pinned, operator-driven snapshot route now has its own explicit
  ceiling.
- A 413 was reported as `RegionUnreachable`. The peer answered - it is
  reachable and healthy - so the error sent the operator to TLS and
  NetworkPolicy. It now names the measured snapshot size, the peer's cap, and
  the fix.

The ceiling is not the design: the snapshot grows with the corpus and chunked
reconcile is the durable answer. Documented as such at the constant.
2026-08-18 10:07:19 -06:00
..
benches feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
config feat: kubernetes deployment, OpenAPI spec, guides, and docker consolidation 2026-06-09 17:06:34 -06:00
src fix(cluster): reconcile could not run at production scale 2026-08-18 10:07:19 -06:00
tests fleet remediation: make the workspace gate runnable, then fix what it caught 2026-08-16 12:38:14 -06:00
BUILD.bazel feat: Bazel build, crate docs/ai-lookup, docker images, and engine hardening 2026-06-07 18:29:38 -06:00
Cargo.toml feat(m11): continuous correctness (m11p9) — fault classes, invariant checkers, soak gates, nightly pipeline 2026-06-13 15:23:59 -06:00