# Task 04: TenantMigration ## Delivers `TenantMigration` in `tidal/src/replication/migration.rs`. Moves a tenant from one shard/region to another with zero downtime via a dual-write window. During migration, writes go to both the old shard and the new shard. After the new shard's seqno matches the old shard's, reads are atomically switched to the new shard, and the old shard's tenant data is garbage-collected. ## Complexity: L ## Dependencies - Task 01 (TenantId, TenantConfig) - Task 02 (TenantRouter) - Task 03 (ControlPlane) - Phase 8.2 (WAL shipping -- used to bootstrap the new shard from existing WAL segments) ## Technical Design ```rust // tidal/src/replication/migration.rs /// State machine for tenant migration. /// /// States: /// Idle -> PreparingTarget -> DualWrite -> Finalizing -> Complete /// /// The migration progresses monotonically. If it fails at any stage, /// it can be retried from the same state (idempotent by design). #[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)] pub enum MigrationState { Idle, /// Source shard is shipping WAL segments to target shard. PreparingTarget { last_shipped_seqno: u64, }, /// Both shards receive writes. Source seqno is the cut-over gate. DualWrite { cutover_seqno: u64, }, /// Writes routed to target only. Waiting for read switchover. Finalizing { switched_at_ns: u64, }, /// Migration complete. Old shard data can be GC'd. Complete, } /// Migrates a tenant from one shard to another with zero downtime. pub struct TenantMigration { tenant_id: TenantId, source_shard: ShardId, target_shard: ShardId, state: Mutex, control_plane: Arc, tenant_router: Arc, transport: Arc, } impl TenantMigration { pub fn new( tenant_id: TenantId, source_shard: ShardId, target_shard: ShardId, control_plane: Arc, tenant_router: Arc, transport: Arc, ) -> Self { Self { tenant_id, source_shard, target_shard, state: Mutex::new(MigrationState::Idle), control_plane, tenant_router, transport, } } /// Phase 1: Ship all existing WAL segments to the target shard. /// /// The target shard replays these segments to build up state. /// Returns the seqno of the last shipped segment. pub async fn prepare_target(&self) -> Result { let mut state = self.state.lock().unwrap(); assert_eq!(*state, MigrationState::Idle); // Ship all sealed WAL segments for this tenant to the target. let segments = self.list_tenant_segments()?; let mut last_seqno = 0u64; for seg in segments { let payload = self.read_segment_payload(&seg)?; self.transport.send_segment( self.target_shard_region()?, payload, ).await?; last_seqno = seg.seqno; } *state = MigrationState::PreparingTarget { last_shipped_seqno: last_seqno }; Ok(last_seqno) } /// Phase 2: Enter dual-write mode. /// /// All subsequent writes for this tenant go to BOTH source and target shards. /// The `cutover_seqno` is the source shard's current seqno when dual-write starts. /// Once target reaches `cutover_seqno`, we can finalize. pub async fn enter_dual_write(&self) -> Result { let mut state = self.state.lock().unwrap(); assert!(matches!(*state, MigrationState::PreparingTarget { .. })); // Get current seqno from source shard (the cut-over gate). let cutover_seqno = self.current_source_seqno()?; // Update routing: writes now go to both source and target. self.tenant_router.set_dual_write(self.tenant_id, self.source_shard, self.target_shard); *state = MigrationState::DualWrite { cutover_seqno }; Ok(cutover_seqno) } /// Phase 3: Finalize -- switch reads to target, stop writing to source. /// /// Only called after target shard has caught up to `cutover_seqno`. /// Reads are atomically switched to the target shard. pub async fn finalize(&self) -> Result<()> { let mut state = self.state.lock().unwrap(); let cutover_seqno = match *state { MigrationState::DualWrite { cutover_seqno } => cutover_seqno, _ => return Err(TidalError::InvalidState("finalize called outside DualWrite state".into())), }; // Verify target has caught up. let target_seqno = self.current_target_seqno()?; if target_seqno < cutover_seqno { return Err(TidalError::NotReady( format!("target shard at seqno {}, cutover requires {}", target_seqno, cutover_seqno) )); } // Atomically switch routing: reads now go to target only, no more writes to source. self.tenant_router.finalize_migration(self.tenant_id, self.target_shard); self.control_plane.update_topology(ShardAssignment { shard_id: self.source_shard, region_id: self.source_shard_region()?, }); *state = MigrationState::Finalizing { switched_at_ns: crate::util::now_ns() }; Ok(()) } /// Phase 4: Garbage-collect source shard tenant data. /// /// Called after a GC window (e.g., 5 minutes) to ensure no in-flight /// reads are still served from the source shard. pub fn gc_source(&self, gc_window_ns: u64) -> Result<()> { let mut state = self.state.lock().unwrap(); let switched_at = match *state { MigrationState::Finalizing { switched_at_ns } => switched_at_ns, _ => return Err(TidalError::InvalidState("gc called outside Finalizing state".into())), }; let now = crate::util::now_ns(); if now.saturating_sub(switched_at) < gc_window_ns { return Err(TidalError::NotReady("GC window not elapsed".into())); } self.delete_tenant_data_on_source()?; *state = MigrationState::Complete; Ok(()) } fn list_tenant_segments(&self) -> Result> { todo!() } fn read_segment_payload(&self, meta: &SegmentMeta) -> Result { todo!() } fn current_source_seqno(&self) -> Result { todo!() } fn current_target_seqno(&self) -> Result { todo!() } fn target_shard_region(&self) -> Result { todo!() } fn source_shard_region(&self) -> Result { todo!() } fn delete_tenant_data_on_source(&self) -> Result<()> { todo!() } } ``` ## Acceptance Criteria - [ ] Migration state machine progresses `Idle -> PreparingTarget -> DualWrite -> Finalizing -> Complete`; state transitions are validated (panic/error on invalid transitions) - [ ] During `DualWrite` state: writes to `signal_for_tenant` go to BOTH source and target shards (verified by reading from both after 10 writes) - [ ] `finalize()` fails with `NotReady` if target seqno < cutover seqno; succeeds once target catches up - [ ] After `finalize()`: queries are served from the target shard; source shard data is not queried - [ ] `gc_source()` fails if < GC window elapsed; deletes tenant WAL segments and signal state from source shard after window - [ ] Zero downtime test: start migration, write 1000 signals during `DualWrite`, finalize, verify all 1000 signals present on target - [ ] `cargo clippy -D warnings` and `cargo fmt` pass