tidaldb/docs/planning/milestone-8/phase-2/task-04-segment-receiver.md

Task 04: SegmentReceiver

Delivers

SegmentReceiver background task in tidal/src/replication/receiver.rs. Receives WalSegmentPayload from transport, validates BLAKE3 checksum, decodes batches, and replays events through SignalLedger::apply_wal_event(). Idempotent via seqno high-water-mark.

Complexity: M

Dependencies

• Task 01 (Transport trait)
• Task 02 (InProcessTransport)
• Phase 8.1 (ReplicationState for high-water-mark)

Technical Design

// tidal/src/replication/receiver.rs

/// Receives WAL segments from a leader and replays them locally.
///
/// Runs as a background tokio task. The receiver maintains strict
/// idempotency: segments with seqno <= `applied_seqno` are skipped.
pub struct SegmentReceiver {
    transport: Arc<dyn Transport>,
    signal_ledger: Arc<SignalLedger>,
    replication_state: Arc<ReplicationState>,
    leader_shard: ShardId,
}

impl SegmentReceiver {
    pub fn new(
        transport: Arc<dyn Transport>,
        signal_ledger: Arc<SignalLedger>,
        replication_state: Arc<ReplicationState>,
        leader_shard: ShardId,
    ) -> Self {
        Self { transport, signal_ledger, replication_state, leader_shard }
    }

    pub fn start(self: Arc<Self>, shutdown_rx: tokio::sync::watch::Receiver<bool>)
        -> tokio::task::JoinHandle<()>
    {
        tokio::spawn(async move {
            self.run(shutdown_rx).await;
        })
    }

    async fn run(&self, mut shutdown: tokio::sync::watch::Receiver<bool>) {
        loop {
            tokio::select! {
                segment = self.transport.recv_segment() => {
                    match segment {
                        Some(payload) => {
                            if let Err(e) = self.apply_segment(payload).await {
                                tracing::error!("SegmentReceiver: apply error: {e}");
                            }
                        }
                        None => {
                            tracing::info!("SegmentReceiver: transport closed, stopping");
                            break;
                        }
                    }
                }
                Ok(_) = shutdown.changed() => {
                    if *shutdown.borrow() { break; }
                }
            }
        }
    }

    async fn apply_segment(&self, payload: WalSegmentPayload) -> Result<(), WalError> {
        let seqno = payload.id.seqno;
        let shard = payload.id.shard_id;

        // Idempotency check: skip segments already applied.
        let applied = self.replication_state
            .applied_seqno(shard)
            .unwrap_or(0);
        if seqno <= applied {
            tracing::trace!(seqno, applied, "SegmentReceiver: skipping duplicate segment");
            return Ok(());
        }

        // BLAKE3 checksum validation.
        let expected_checksum = blake3::hash(&payload.bytes);
        // (Extract checksum from BatchHeader and compare)

        // Decode and replay each event.
        let batches = decode_wal_segment(&payload.bytes)?;
        for batch in batches {
            for event in batch.events {
                self.signal_ledger.apply_wal_event(
                    event.entity_id,
                    &event.signal_type,
                    event.weight,
                    event.timestamp,
                )?;
            }
        }

        // Advance high-water-mark.
        self.replication_state.advance(shard, seqno);
        tracing::debug!(seqno, "SegmentReceiver: applied segment");
        Ok(())
    }
}

Acceptance Criteria

• SegmentReceiver::start() spawns a background tokio task that reads from transport.recv_segment()
• BLAKE3 checksum validation: corrupted segments return WalError::Corruption and are NOT applied
• Idempotency: segments with seqno <= replication_state.applied_seqno(shard) are skipped (no double-counting)
• All events in a received segment are replayed through SignalLedger::apply_wal_event()
• replication_state.advance(shard, seqno) is called after successful replay
• Transport close (recv_segment returns None) causes the receiver to stop gracefully
• Integration test: ship 100 segments -> receiver applies all -> decay scores match
• cargo clippy -D warnings and cargo fmt pass