tidaldb/docs/planning/milestone-8/phase-10/task-02-single-region-cluster-mode.md

# Task 02: Single-region cluster mode (`--region`, `RegionClusterState`)

## Delivers

`tidal-server cluster --region <name>` runs ONE region in this process: one `TidalDb`, one
`GrpcTransport` (its gRPC server bound to this region's `grpc_addr`, peer table pointing
at every sibling region's real `grpc_addr`), a segment-receiver thread applying inbound
WAL batches, and a `SignalRelay`-backed leader write path. Without `--region`, the
existing single-process mode runs byte-for-byte unchanged.

Also executes the tracked Maintainability-S follow-up: `cluster.rs` (1379 lines) splits
into `tidal-server/src/cluster/{mod,topology,transport,state,node,routes}.rs` before the
new mode is added, so the new code lands in a clean home.

## Complexity: XL

## Dependencies

Task 01 (`replication::relay::SignalRelay`, moved primitives).

## Technical Design

### Module split (mechanical, zero behavior change, separate commit-sized step)

- `cluster/topology.rs` — `TopologySpec`, `RegionSpec`, `load_topology`, validation
- `cluster/transport.rs` — `build_grpc_transports`, `build_ready_follower_transport`,
  `grpc_server_ready`, `resolve_grpc_addr`, `free_loopback_addr`, the GRPC_* consts
- `cluster/state.rs` — `ClusterState` (single-process, unchanged)
- `cluster/routes.rs` — router build + all existing handlers + `ClusterAppError`
- `cluster/node.rs` — NEW: `RegionClusterState`
- `cluster/mod.rs` — re-exports preserving the current public paths
  (`tidal_server::cluster::{ClusterState, build_cluster_router, ensure_experimental_enabled, load_topology}`)

### Topology extension

```rust
pub struct RegionSpec {
    pub name: String,
    pub grpc_addr: Option<String>,
    /// This region's public HTTP address (host:port), required in
    /// multi-process mode for write/read forwarding and status aggregation.
    #[serde(default)]
    pub http_addr: Option<String>,
}
```

Multi-process validation (`validate_multiproc(topology, my_region)`): every region
declares both `grpc_addr` and `http_addr`; `my_region` names a declared region; names
unique; leader declared. Single-process validation unchanged.

### CLI

`ClusterArgs` gains:

```rust
/// Run ONLY this region in this process (multi-process cluster mode).
/// Peers are reached via the topology's per-region grpc_addr/http_addr.
#[arg(long, env = "TIDAL_REGION")]
region: Option<String>,
/// Data directory for this region's TidalDb (multi-process mode).
/// Omitted ⇒ ephemeral.
#[arg(long)]
data_dir: Option<PathBuf>,
```

`run_cluster` dispatches: `Some(region)` → build `RegionClusterState` on the dedicated
non-async thread (same `GrpcTransport::new` blocking constraint), serve with the
multi-process router (task 03 fills in the cross-process handlers; this task wires
local-only behavior). `None` → existing path untouched. HLC skew env: read
`TIDAL_HLC_SKEW_MS` (i64, default 0) in multi-process mode and pass to
`TidalDbBuilder::with_hlc_offset_ms` — the env var is the operator/test surface, the
builder is the mechanism.

### `RegionClusterState` (cluster/node.rs)

```rust
pub struct RegionClusterState {
    region: RegionId,                       // this process
    region_name: String,
    db: Option<Arc<TidalDb>>,               // taken on shutdown, like ClusterState
    transport: Arc<GrpcTransport>,          // server on my grpc_addr; peers = siblings
    relay: SignalRelay,                     // leader stream when this node leads
    leader: RwLock<RegionId>,               // current leadership view
    partitioned: RwLock<HashSet<RegionId>>, // leader-side ship-skip set
    name_to_id / id_to_name: HashMap<…>,    // from topology order (index = RegionId)
    peer_http: HashMap<RegionId, String>,   // for task-03 forwarding/aggregation
    signal_type_ids: HashMap<String, u8>,   // same u8-guarded map SimulatedCluster builds
    write_pool: ClusterWritePool,
    shutting_down: AtomicBool,
}
```

Construction:

1. RegionIds by topology declaration order (identical rule to `ClusterState::new` so the
   same topology file yields the same ids in every process).
2. `TidalDb::builder()` with schema/profiles, `NodeRole::Single`,
   `shard_id = ShardId(region.0)`, `peer_shards` = all sibling shards; `with_data_dir`
   when `--data-dir` given else `ephemeral()`; `with_hlc_offset_ms` from env.
3. One `GrpcTransport`: `local_shard = ShardId(my_region)`, `listen_addr = my grpc_addr`
   (explicit — tried once, no port reallocation), `peers = {ShardId(r) → r.grpc_addr}`
   for every OTHER region, `insecure: true` (TLS config wiring stays available via
   `GrpcTransportConfig` defaults; not exercised in this phase).
4. `db.start_replication(transport.clone())` — EVERY node runs a receiver (leadership can
   move; an inbound segment on the current leader is legal after a promote elsewhere).
   Reuse the readiness probe (`grpc_server_ready`) before accepting HTTP.

Write path (`write_signal_local`, called on the write pool):

- If `self.leader.read() != self.region` → return a typed `NotLeader { leader, http }`
  error (task 03 turns this into forwarding; this task maps it to 503 + JSON body naming
  the leader so the node is honest standalone).
- Else `relay.write_and_ship(db, my_shard, type_id, …, transport, peer_shards, partitioned)`.

Items/embeddings writes: leader-applied locally and… **replication of item metadata and
embeddings is NOT in the WAL relay** (the relay ships signal events only — same as
m8p8/SimulatedCluster, whose `write_item_with_metadata` broadcast to all in-process
nodes). Multi-process equivalent: leader applies locally AND fans the same JSON write out
to every peer's HTTP `/items` / `/embeddings` with the internal-propagation marker
(task 03 implements the fan-out client; this task structures the handler so the fan-out
slots in). This preserves m8p8's "items are broadcast, signals are replicated" semantics.

Local management surface (this task):

- `GET /cluster/status/local` → `{ region, is_leader, leader, last_seq, applied_events,
  lag_events, partitioned: [names], reachable: true }` where `last_seq` =
  `relay.last_seq()` when leading, `applied_events` = `db.replication_state()
  .applied_seqno(current_leader_shard)`, `lag_events` from the lag gauge. utoipa-annotated.
- `POST /cluster/promote` (local effect only in this task): update `leader` view; 400 on
  unknown region.
- `POST /cluster/partition` / `POST /cluster/heal`: only meaningful while this node
  leads — partition inserts into ship-skip set; heal removes + `relay` redelivers to that
  peer over gRPC on the write pool. Non-leader: typed `NotLeader` (task 03 forwards).
- `POST /hardnegs { user_id, item_id }` → records a hide hard-negative on THIS node
  (node-local by design; convergence is the CRDT reconcile path, task 03). Use the
  engine's hard-negative API (`entities/hard_neg.rs` / the same call
  `take_crdt_snapshot` reads from). utoipa-annotated, bearer-protected.
- `/health` reports `mode: "cluster"`, `region`, `leader`, `process: "single-region"`.
- Data reads (`/feed`, `/search`): serve from the LOCAL db (default read region = local);
  `?region=` naming a different region returns typed `NotLocal` (task 03 forwards).
  Same limit clamping and offload as `ClusterState` handlers.

Shutdown mirrors `ClusterState`: flip readiness, drop db (checkpoint + WAL fsync +
thread join), `GrpcTransport` drop uses `shutdown_background`.

### Router

`build_region_router(state: Arc<RegionClusterState>, api_key)` — same public/protected
split, same load-shedding stack and shared constants as `build_cluster_router`. The
`/sharded/*` routes are wired in task 03 (multi-process scatter-gather); this task mounts
the non-sharded surface.

### Experimental gate

`ensure_experimental_enabled` stays required for both modes. The WARN text branches: the
multi-process WARN states process isolation is real but quorum-ack writes and automatic
failure detection are not yet provided.

## Test Strategy

In-process integration tests (NOT feature-gated — they use real `GrpcTransport` on
loopback like `cluster_grpc.rs`, no OS processes):

- `region_node_replicates_over_grpc`: two `RegionClusterState`s in one test process
  (distinct topologies pointing at each other's real grpc addrs), write signals on the
  leader's state, poll the follower's `applied_seqno` until convergence, decay parity 1e-6.
- `region_node_rejects_writes_when_not_leader` (typed NotLeader surfaces as 503 + leader
  identity in body).
- `region_node_partition_heal`: partition peer → ships skipped (lag grows on follower) →
  heal → redeliver converges; heal is idempotent (second heal no-op, scores unchanged).
- `region_node_promote_local`: promote changes leadership view; old leader's
  `write_signal_local` now returns NotLeader; new leader accepts and ships (receiver on
  the demoted node applies — proves the always-on receiver).
- Topology validation unit tests (missing grpc_addr/http_addr, unknown --region, dup names).
- Module split: full existing `tidal-server` test suite green (split is behavior-neutral).

## Acceptance Criteria

- [ ] `cluster.rs` split into `cluster/` submodules; existing public API paths preserved;
      all pre-existing tests pass unchanged
- [ ] `--region`/`TIDAL_REGION` selects multi-process mode; absent ⇒ existing mode
      byte-identical (existing `cluster_grpc.rs` + `cluster_e2e.rs` suites green)
- [ ] `RegionClusterState` boots one region: own gRPC server, peers at real sibling
      addrs, always-on receiver, `SignalRelay` write path, data-dir support
- [ ] Two in-process region nodes converge over real loopback gRPC with decay parity 1e-6
- [ ] Partition/heal and local promote behave per design, heal idempotent
- [ ] `POST /hardnegs` records a hide on the local node
- [ ] All new handlers/DTOs carry utoipa annotations
- [ ] `cargo clippy -p tidal-server -- -D warnings` clean; fmt clean; full workspace tests pass