8a0950260f
Splits monolithic cluster.rs into tidal-server/src/cluster/ modules. Adds redeliver-missed relay, bounded HLC drift, lag tracking, and reconcile idempotence. Five new tier-3 test suites (chaos, lifecycle, multiproc, region, routes, runbook) all green. Docs, CHANGELOG, and ROADMAP updated with G4/G5/G6 known gaps.
161 lines
8.1 KiB
Markdown
161 lines
8.1 KiB
Markdown
# Task 03: Cross-process routes — forwarding, promote fan-out, status aggregation, reconcile, sharded
|
|
|
|
## Delivers
|
|
|
|
The multi-process cluster becomes a single coherent HTTP surface: any node accepts any
|
|
runbook operation and the cluster behaves as one system. Concretely: leader forwarding for
|
|
writes, region forwarding for reads, promote propagation, aggregated `/cluster/status`,
|
|
cross-process CRDT reconciliation (`/cluster/reconcile`), item/embedding broadcast, and
|
|
multi-process `/sharded/*` scatter-gather with honest degraded semantics.
|
|
|
|
## Complexity: XL
|
|
|
|
## Dependencies
|
|
|
|
Task 02 (`RegionClusterState`, typed `NotLeader`/`NotLocal` errors, peer_http map).
|
|
|
|
## Technical Design
|
|
|
|
### Forwarding client (`cluster/forward.rs`)
|
|
|
|
Promote `reqwest` from dev-dependency to dependency (workspace-consistent version 0.12,
|
|
`features = ["json"]`, `default-features = false` + rustls if the workspace builds clean
|
|
that way; otherwise default features — implementer verifies). One shared
|
|
`reqwest::Client` on `RegionClusterState` (connection pooling), short timeouts:
|
|
connect 1s, request 5s (status aggregation uses 500ms overall per peer).
|
|
|
|
Internal-propagation marker: header `x-tidal-internal: 1`. A request carrying it is
|
|
NEVER re-forwarded/re-broadcast (loop prevention). Forwarded requests pass through the
|
|
`Authorization` header verbatim (all nodes share one `TIDAL_API_KEY`).
|
|
|
|
### Write forwarding
|
|
|
|
`POST /signals|/items|/embeddings|/hardnegs` on a non-leader (without the internal
|
|
marker): forward body to `http://{leader_http}{path}`, relay status + body back. With the
|
|
marker: apply locally only (this is how broadcasts and forwarded writes terminate).
|
|
Forward happens in the async handler (reqwest is async; no write-pool involvement —
|
|
the pool is only for local gRPC ship work). Leader unreachable → 503 with a JSON body
|
|
naming the leader and the connect error.
|
|
|
|
### Item/embedding broadcast (m8p8 parity)
|
|
|
|
On the leader, `POST /items|/embeddings` (external request): apply locally, then
|
|
best-effort fan out the same body to every peer `http_addr` with the internal marker,
|
|
concurrently, 2s per-peer timeout. Per-peer failures are WARN-logged and reported in the
|
|
response body (`{"replicated_to": n, "failed": ["ap-south"]}` with 201/204 preserved as
|
|
status); a partitioned peer catches up via operator re-broadcast or the heal-era
|
|
re-write — document the contract in the handler rustdoc and runbook (items are broadcast
|
|
config-data, signals are the replicated stream; same split as m8p8).
|
|
|
|
### Promote fan-out
|
|
|
|
`POST /cluster/promote {region}` external: validate, apply locally, fan out with internal
|
|
marker to ALL peers concurrently (2s timeout each), then 200 with
|
|
`{ok, leader, acked: [..], failed: [..]}`. Internal: apply locally only. The promote is
|
|
correct even if some peers miss it (their next forwarded write returns NotLeader→ the
|
|
forwarding node knows the leader; stale views self-correct on the next status poll —
|
|
document this as eventual leadership propagation; the UAT asserts all-3 convergence).
|
|
|
|
`/cluster/partition` + `/cluster/heal` external on non-leader: forward to current leader
|
|
(they mutate leader-side ship-skip state).
|
|
|
|
### Status aggregation
|
|
|
|
`GET /cluster/status` on ANY node: query every region's `/cluster/status/local`
|
|
(including own, in-process) with 500ms per-peer budget, concurrently. Response keeps the
|
|
documented shape and adds `reachable`:
|
|
|
|
```json
|
|
{
|
|
"leader": "us-east",
|
|
"relay_log_len": 125,
|
|
"regions": [
|
|
{"name":"us-east","applied_events":125,"lag_events":0,"partitioned":false,"reachable":true},
|
|
{"name":"ap-south","applied_events":0,"lag_events":125,"partitioned":true,"reachable":false}
|
|
]
|
|
}
|
|
```
|
|
|
|
- `leader` = this node's view; `relay_log_len` = leader's `last_seq` (from the leader's
|
|
local status; if THIS node leads, read it directly).
|
|
- Per region: `applied_events` from its local status; `lag_events =
|
|
leader_last_seq.saturating_sub(applied)`; leader row mirrors today's semantics
|
|
(applied = its replication-from-others counter, lag 0).
|
|
- Unreachable peer: `reachable: false`, `partitioned: true`, `applied_events: 0`,
|
|
`lag_events = leader_last_seq` (worst-case honest).
|
|
|
|
### Cross-process reconciliation
|
|
|
|
Two routes (both utoipa-annotated, bearer-protected):
|
|
|
|
- `POST /cluster/reconcile/snapshot` (internal, marker required): body = wire
|
|
`StateSnapshot`. Node takes its OWN snapshot first, then `reconcile_with(remote)`,
|
|
responds with its pre-merge snapshot + `elapsed_ms` (merge+apply time only).
|
|
- `POST /cluster/reconcile {region}` (operator-facing): this node snapshots itself, POSTs
|
|
to the target's `/cluster/reconcile/snapshot`, applies the returned pre-merge snapshot
|
|
via its own `reconcile_with`, responds
|
|
`{ok, region, local_elapsed_ms, remote_elapsed_ms, ops_applied}`. CRDT merge
|
|
determinism guarantees both sides converge to the same state.
|
|
|
|
### Multi-process `/sharded/*`
|
|
|
|
Introduce a shard-executor seam in `scatter_gather.rs`: the merge/dedup/diversity logic
|
|
already operates on per-shard results; factor the per-shard fetch into
|
|
|
|
```rust
|
|
trait ShardExec: Send + Sync {
|
|
fn retrieve(&self, shard: RegionId, q: &Retrieve, deadline: Duration) -> Result<Results>;
|
|
fn search(&self, shard: RegionId, q: &Search, deadline: Duration) -> Result<SearchResults>;
|
|
}
|
|
```
|
|
|
|
with the existing in-process impl for `ClusterState` (zero behavior change — existing
|
|
scatter-gather tests must pass untouched) and an HTTP impl for `RegionClusterState`:
|
|
local region executes locally; remote regions via
|
|
`GET {peer_http}/feed|/search?...&limit=...` with the per-shard deadline as the request
|
|
timeout, marker header set. Timeout/unreachable/error ⇒ that shard lands in
|
|
`unavailable_shards`, `degraded: true` — never an error, never silently dropped
|
|
(preserve the documented semantics verbatim).
|
|
|
|
Sharded writes (`/sharded/items|embeddings|signals`): owning shard via the engine
|
|
`ShardRouter` (same hash the in-process path uses); owner == self → local apply; else
|
|
forward to the owner's `http_addr` `/sharded/*` with marker (owner applies locally on
|
|
marker receipt). Note the data-model split (documented in runbook rewrite): `/sharded/*`
|
|
hash-partitions, the non-sharded surface replicates.
|
|
|
|
## Test Strategy
|
|
|
|
In-process integration tests with 3 `RegionClusterState`s + real axum servers on loopback
|
|
(tokio runtimes per node, pattern from `cluster_grpc.rs::http_signal_replicates…`), no OS
|
|
processes (tier-3 OS-process coverage is tasks 04/05):
|
|
|
|
- `forwarded_write_lands_on_leader_and_replicates`: POST /signals to follower → 204 →
|
|
converges on all nodes.
|
|
- `promote_fans_out`: promote via follower B → all 3 status views agree; writes to old
|
|
leader forward to new.
|
|
- `status_aggregates_all_regions` + unreachable peer (stop one node) → `reachable: false`,
|
|
worst-case lag.
|
|
- `reconcile_converges_and_times`: divergent hardnegs (hide on A during simulated
|
|
divergence) → `/cluster/reconcile` → both sides converged (hide present on both),
|
|
`local_elapsed_ms`/`remote_elapsed_ms` reported; second reconcile is a no-op
|
|
(idempotent).
|
|
- `sharded_feed_degrades_honestly`: 3 nodes, one stopped → `degraded: true`,
|
|
`unavailable_shards` names it, merged results from the live shards.
|
|
- `sharded_write_routes_to_owner`: entity whose `ShardRouter` owner is a remote region →
|
|
forwarded, readable via that region's local feed.
|
|
- Existing single-process scatter-gather + cluster tests green (executor seam neutral).
|
|
|
|
## Acceptance Criteria
|
|
|
|
- [ ] Writes to ANY node succeed via leader forwarding; internal marker prevents loops
|
|
- [ ] Items/embeddings broadcast leader→peers with per-peer failure reporting
|
|
- [ ] Promote propagates to all peers; partition/heal forward to the leader
|
|
- [ ] `/cluster/status` on any node aggregates all regions with `reachable` honesty
|
|
- [ ] `/cluster/reconcile` exchanges snapshots, converges both sides, reports elapsed_ms,
|
|
idempotent on repeat
|
|
- [ ] `/sharded/feed|search` fan out across processes with preserved degraded semantics;
|
|
sharded writes route to the owning region
|
|
- [ ] Existing in-process scatter-gather behavior unchanged (seam is refactor-neutral)
|
|
- [ ] All new routes utoipa-annotated and present in the cluster OpenAPI document
|
|
- [ ] `cargo clippy -p tidal-server -- -D warnings` clean; full workspace tests pass
|