tidaldb/docs/planning/milestone-8/phase-2/task-04-segment-receiver.md
jordan f4cfd6c81f feat: complete M8 replication primitives + forage enhancements + docs
Milestone 8 (phases 1-4):
- Shard-aware WAL segment naming, BatchHeader v2, ShardRouter
- Transport trait, InProcessTransport, WalShipper, FollowerDb
- HLC, PNCounter, LWWRegister, CrdtSignalState, ReconciliationEngine
- Session replication bridge with SeqNo/HWM, idempotency store

Forage application:
- Multi-source discovery engine with MAB exploration
- Embedding-based label system, server handlers, UI refresh

Other:
- QUICKSTART.md, README.md, milestone-8 planning docs
- Hard negative union semantics, RLHF export enhancements
- Recovery benchmark and visibility test expansions
- Split 8 oversized source files per CODING_GUIDELINES §9

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 13:17:19 -07:00

120 lines
4.2 KiB
Markdown

# Task 04: SegmentReceiver
## Delivers
`SegmentReceiver` background task in `tidal/src/replication/receiver.rs`. Receives `WalSegmentPayload` from transport, validates BLAKE3 checksum, decodes batches, and replays events through `SignalLedger::apply_wal_event()`. Idempotent via seqno high-water-mark.
## Complexity: M
## Dependencies
- Task 01 (Transport trait)
- Task 02 (InProcessTransport)
- Phase 8.1 (ReplicationState for high-water-mark)
## Technical Design
```rust
// tidal/src/replication/receiver.rs
/// Receives WAL segments from a leader and replays them locally.
///
/// Runs as a background tokio task. The receiver maintains strict
/// idempotency: segments with seqno <= `applied_seqno` are skipped.
pub struct SegmentReceiver {
transport: Arc<dyn Transport>,
signal_ledger: Arc<SignalLedger>,
replication_state: Arc<ReplicationState>,
leader_shard: ShardId,
}
impl SegmentReceiver {
pub fn new(
transport: Arc<dyn Transport>,
signal_ledger: Arc<SignalLedger>,
replication_state: Arc<ReplicationState>,
leader_shard: ShardId,
) -> Self {
Self { transport, signal_ledger, replication_state, leader_shard }
}
pub fn start(self: Arc<Self>, shutdown_rx: tokio::sync::watch::Receiver<bool>)
-> tokio::task::JoinHandle<()>
{
tokio::spawn(async move {
self.run(shutdown_rx).await;
})
}
async fn run(&self, mut shutdown: tokio::sync::watch::Receiver<bool>) {
loop {
tokio::select! {
segment = self.transport.recv_segment() => {
match segment {
Some(payload) => {
if let Err(e) = self.apply_segment(payload).await {
tracing::error!("SegmentReceiver: apply error: {e}");
}
}
None => {
tracing::info!("SegmentReceiver: transport closed, stopping");
break;
}
}
}
Ok(_) = shutdown.changed() => {
if *shutdown.borrow() { break; }
}
}
}
}
async fn apply_segment(&self, payload: WalSegmentPayload) -> Result<(), WalError> {
let seqno = payload.id.seqno;
let shard = payload.id.shard_id;
// Idempotency check: skip segments already applied.
let applied = self.replication_state
.applied_seqno(shard)
.unwrap_or(0);
if seqno <= applied {
tracing::trace!(seqno, applied, "SegmentReceiver: skipping duplicate segment");
return Ok(());
}
// BLAKE3 checksum validation.
let expected_checksum = blake3::hash(&payload.bytes);
// (Extract checksum from BatchHeader and compare)
// Decode and replay each event.
let batches = decode_wal_segment(&payload.bytes)?;
for batch in batches {
for event in batch.events {
self.signal_ledger.apply_wal_event(
event.entity_id,
&event.signal_type,
event.weight,
event.timestamp,
)?;
}
}
// Advance high-water-mark.
self.replication_state.advance(shard, seqno);
tracing::debug!(seqno, "SegmentReceiver: applied segment");
Ok(())
}
}
```
## Acceptance Criteria
- [ ] `SegmentReceiver::start()` spawns a background tokio task that reads from `transport.recv_segment()`
- [ ] BLAKE3 checksum validation: corrupted segments return `WalError::Corruption` and are NOT applied
- [ ] Idempotency: segments with `seqno <= replication_state.applied_seqno(shard)` are skipped (no double-counting)
- [ ] All events in a received segment are replayed through `SignalLedger::apply_wal_event()`
- [ ] `replication_state.advance(shard, seqno)` is called after successful replay
- [ ] Transport close (`recv_segment` returns `None`) causes the receiver to stop gracefully
- [ ] Integration test: ship 100 segments -> receiver applies all -> decay scores match
- [ ] `cargo clippy -D warnings` and `cargo fmt` pass