tidaldb/docs/planning/milestone-8/phase-10/task-02-single-region-cluster-mode.md

Task 02: Single-region cluster mode (--region, RegionClusterState)

Delivers

tidal-server cluster --region <name> runs ONE region in this process: one TidalDb, one GrpcTransport (its gRPC server bound to this region's grpc_addr, peer table pointing at every sibling region's real grpc_addr), a segment-receiver thread applying inbound WAL batches, and a SignalRelay-backed leader write path. Without --region, the existing single-process mode runs byte-for-byte unchanged.

Also executes the tracked Maintainability-S follow-up: cluster.rs (1379 lines) splits into tidal-server/src/cluster/{mod,topology,transport,state,node,routes}.rs before the new mode is added, so the new code lands in a clean home.

Complexity: XL

Dependencies

Task 01 (replication::relay::SignalRelay, moved primitives).

Technical Design

Module split (mechanical, zero behavior change, separate commit-sized step)

• cluster/topology.rs — TopologySpec, RegionSpec, load_topology, validation
• cluster/transport.rs — build_grpc_transports, build_ready_follower_transport, grpc_server_ready, resolve_grpc_addr, free_loopback_addr, the GRPC_* consts
• cluster/state.rs — ClusterState (single-process, unchanged)
• cluster/routes.rs — router build + all existing handlers + ClusterAppError
• cluster/node.rs — NEW: RegionClusterState
• cluster/mod.rs — re-exports preserving the current public paths (tidal_server::cluster::{ClusterState, build_cluster_router, ensure_experimental_enabled, load_topology})

Topology extension

pub struct RegionSpec {
    pub name: String,
    pub grpc_addr: Option<String>,
    /// This region's public HTTP address (host:port), required in
    /// multi-process mode for write/read forwarding and status aggregation.
    #[serde(default)]
    pub http_addr: Option<String>,
}

Multi-process validation (validate_multiproc(topology, my_region)): every region declares both grpc_addr and http_addr; my_region names a declared region; names unique; leader declared. Single-process validation unchanged.

CLI

ClusterArgs gains:

/// Run ONLY this region in this process (multi-process cluster mode).
/// Peers are reached via the topology's per-region grpc_addr/http_addr.
#[arg(long, env = "TIDAL_REGION")]
region: Option<String>,
/// Data directory for this region's TidalDb (multi-process mode).
/// Omitted ⇒ ephemeral.
#[arg(long)]
data_dir: Option<PathBuf>,

run_cluster dispatches: Some(region) → build RegionClusterState on the dedicated non-async thread (same GrpcTransport::new blocking constraint), serve with the multi-process router (task 03 fills in the cross-process handlers; this task wires local-only behavior). None → existing path untouched. HLC skew env: read TIDAL_HLC_SKEW_MS (i64, default 0) in multi-process mode and pass to TidalDbBuilder::with_hlc_offset_ms — the env var is the operator/test surface, the builder is the mechanism.

RegionClusterState (cluster/node.rs)

pub struct RegionClusterState {
    region: RegionId,                       // this process
    region_name: String,
    db: Option<Arc<TidalDb>>,               // taken on shutdown, like ClusterState
    transport: Arc<GrpcTransport>,          // server on my grpc_addr; peers = siblings
    relay: SignalRelay,                     // leader stream when this node leads
    leader: RwLock<RegionId>,               // current leadership view
    partitioned: RwLock<HashSet<RegionId>>, // leader-side ship-skip set
    name_to_id / id_to_name: HashMap<…>,    // from topology order (index = RegionId)
    peer_http: HashMap<RegionId, String>,   // for task-03 forwarding/aggregation
    signal_type_ids: HashMap<String, u8>,   // same u8-guarded map SimulatedCluster builds
    write_pool: ClusterWritePool,
    shutting_down: AtomicBool,
}

Construction:

1. RegionIds by topology declaration order (identical rule to ClusterState::new so the same topology file yields the same ids in every process).
2. TidalDb::builder() with schema/profiles, NodeRole::Single, shard_id = ShardId(region.0), peer_shards = all sibling shards; with_data_dir when --data-dir given else ephemeral(); with_hlc_offset_ms from env.
3. One GrpcTransport: local_shard = ShardId(my_region), listen_addr = my grpc_addr (explicit — tried once, no port reallocation), peers = {ShardId(r) → r.grpc_addr} for every OTHER region, insecure: true (TLS config wiring stays available via GrpcTransportConfig defaults; not exercised in this phase).
4. db.start_replication(transport.clone()) — EVERY node runs a receiver (leadership can move; an inbound segment on the current leader is legal after a promote elsewhere). Reuse the readiness probe (grpc_server_ready) before accepting HTTP.

Write path (write_signal_local, called on the write pool):

• If self.leader.read() != self.region → return a typed NotLeader { leader, http } error (task 03 turns this into forwarding; this task maps it to 503 + JSON body naming the leader so the node is honest standalone).
• Else relay.write_and_ship(db, my_shard, type_id, …, transport, peer_shards, partitioned).

Items/embeddings writes: leader-applied locally and… replication of item metadata and embeddings is NOT in the WAL relay (the relay ships signal events only — same as m8p8/SimulatedCluster, whose write_item_with_metadata broadcast to all in-process nodes). Multi-process equivalent: leader applies locally AND fans the same JSON write out to every peer's HTTP /items / /embeddings with the internal-propagation marker (task 03 implements the fan-out client; this task structures the handler so the fan-out slots in). This preserves m8p8's "items are broadcast, signals are replicated" semantics.

Local management surface (this task):

• GET /cluster/status/local → { region, is_leader, leader, last_seq, applied_events, lag_events, partitioned: [names], reachable: true } where last_seq = relay.last_seq() when leading, applied_events = db.replication_state() .applied_seqno(current_leader_shard), lag_events from the lag gauge. utoipa-annotated.
• POST /cluster/promote (local effect only in this task): update leader view; 400 on unknown region.
• POST /cluster/partition / POST /cluster/heal: only meaningful while this node leads — partition inserts into ship-skip set; heal removes + relay redelivers to that peer over gRPC on the write pool. Non-leader: typed NotLeader (task 03 forwards).
• POST /hardnegs { user_id, item_id } → records a hide hard-negative on THIS node (node-local by design; convergence is the CRDT reconcile path, task 03). Use the engine's hard-negative API (entities/hard_neg.rs / the same call take_crdt_snapshot reads from). utoipa-annotated, bearer-protected.
• /health reports mode: "cluster", region, leader, process: "single-region".
• Data reads (/feed, /search): serve from the LOCAL db (default read region = local); ?region= naming a different region returns typed NotLocal (task 03 forwards). Same limit clamping and offload as ClusterState handlers.

Shutdown mirrors ClusterState: flip readiness, drop db (checkpoint + WAL fsync + thread join), GrpcTransport drop uses shutdown_background.

Router

build_region_router(state: Arc<RegionClusterState>, api_key) — same public/protected split, same load-shedding stack and shared constants as build_cluster_router. The /sharded/* routes are wired in task 03 (multi-process scatter-gather); this task mounts the non-sharded surface.

Experimental gate

ensure_experimental_enabled stays required for both modes. The WARN text branches: the multi-process WARN states process isolation is real but quorum-ack writes and automatic failure detection are not yet provided.

Test Strategy

In-process integration tests (NOT feature-gated — they use real GrpcTransport on loopback like cluster_grpc.rs, no OS processes):

• region_node_replicates_over_grpc: two RegionClusterStates in one test process (distinct topologies pointing at each other's real grpc addrs), write signals on the leader's state, poll the follower's applied_seqno until convergence, decay parity 1e-6.
• region_node_rejects_writes_when_not_leader (typed NotLeader surfaces as 503 + leader identity in body).
• region_node_partition_heal: partition peer → ships skipped (lag grows on follower) → heal → redeliver converges; heal is idempotent (second heal no-op, scores unchanged).
• region_node_promote_local: promote changes leadership view; old leader's write_signal_local now returns NotLeader; new leader accepts and ships (receiver on the demoted node applies — proves the always-on receiver).
• Topology validation unit tests (missing grpc_addr/http_addr, unknown --region, dup names).
• Module split: full existing tidal-server test suite green (split is behavior-neutral).

Acceptance Criteria

• cluster.rs split into cluster/ submodules; existing public API paths preserved; all pre-existing tests pass unchanged
• --region/TIDAL_REGION selects multi-process mode; absent ⇒ existing mode byte-identical (existing cluster_grpc.rs + cluster_e2e.rs suites green)
• RegionClusterState boots one region: own gRPC server, peers at real sibling addrs, always-on receiver, SignalRelay write path, data-dir support
• Two in-process region nodes converge over real loopback gRPC with decay parity 1e-6
• Partition/heal and local promote behave per design, heal idempotent
• POST /hardnegs records a hide on the local node
• All new handlers/DTOs carry utoipa annotations
• cargo clippy -p tidal-server -- -D warnings clean; fmt clean; full workspace tests pass