tidaldb/tidal-server
jx12n 6f17409f40 fix(cluster): forwarded item/embedding writes never replicated; heal backfill 401'd under auth
Two correctness bugs in multi-process cluster mode (m8p10), both found live on a
real 3-pod k3s cluster while validating the deployment, both invisible to the
existing in-process / unauthenticated test suites:

1. Forwarded item/embedding writes dropped on the floor. create_item /
   write_embedding gated the leader's peer broadcast on `if internal { return }`.
   A write to a NON-leader gateway is forwarded to the leader with the internal
   marker set (loop-prevention), so it hit that branch and terminated WITHOUT
   broadcasting — the item landed only on the leader. Signals were unaffected
   (the WAL relay ships regardless of the marker), which masked it. Fix: gate on
   leadership, not the marker — a follower applying a marked broadcast/heal
   terminates; the leader (external OR forwarded) always fans out. Forwarded
   writes now return the {replicated_to,failed} report instead of a bodyless null.

2. Heal item/embedding backfill 401'd whenever TIDAL_API_KEY is set.
   post_marked_blocking sent the internal marker but no Authorization header. The
   marker is a trust signal, not an auth bypass (the bearer middleware runs
   first), so every backfill POST was rejected 401 — a region that missed an item
   while down stayed permanently inconsistent at lag 0. Unauthenticated tests
   never caught it. Fix: thread TIDAL_API_KEY into RegionClusterState and attach
   it (same key on every region) to the backfill POSTs.

Verified live: forwarded writes via follower gateways converge to all 3 regions;
a region scaled to 0 during an item write backfills on heal (item_failures=0).
Follow-up: add multiproc regression tests with auth for both paths.
2026-06-10 20:37:46 -06:00
..
config feat: kubernetes deployment, OpenAPI spec, guides, and docker consolidation 2026-06-09 17:06:34 -06:00
src fix(cluster): forwarded item/embedding writes never replicated; heal backfill 401'd under auth 2026-06-10 20:37:46 -06:00
tests fix(cluster): forwarded item/embedding writes never replicated; heal backfill 401'd under auth 2026-06-10 20:37:46 -06:00
BUILD.bazel feat: Bazel build, crate docs/ai-lookup, docker images, and engine hardening 2026-06-07 18:29:38 -06:00
Cargo.toml feat(m8p10): multi-process cluster mode — scatter-gather, reconcile relay, chaos/UAT suites 2026-06-10 14:07:33 -06:00