Milestone 8 (phases 1-4): - Shard-aware WAL segment naming, BatchHeader v2, ShardRouter - Transport trait, InProcessTransport, WalShipper, FollowerDb - HLC, PNCounter, LWWRegister, CrdtSignalState, ReconciliationEngine - Session replication bridge with SeqNo/HWM, idempotency store Forage application: - Multi-source discovery engine with MAB exploration - Embedding-based label system, server handlers, UI refresh Other: - QUICKSTART.md, README.md, milestone-8 planning docs - Hard negative union semantics, RLHF export enhancements - Recovery benchmark and visibility test expansions - Split 8 oversized source files per CODING_GUIDELINES §9 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
120 lines
4.2 KiB
Markdown
120 lines
4.2 KiB
Markdown
# Task 04: SegmentReceiver
|
|
|
|
## Delivers
|
|
|
|
`SegmentReceiver` background task in `tidal/src/replication/receiver.rs`. Receives `WalSegmentPayload` from transport, validates BLAKE3 checksum, decodes batches, and replays events through `SignalLedger::apply_wal_event()`. Idempotent via seqno high-water-mark.
|
|
|
|
## Complexity: M
|
|
|
|
## Dependencies
|
|
|
|
- Task 01 (Transport trait)
|
|
- Task 02 (InProcessTransport)
|
|
- Phase 8.1 (ReplicationState for high-water-mark)
|
|
|
|
## Technical Design
|
|
|
|
```rust
|
|
// tidal/src/replication/receiver.rs
|
|
|
|
/// Receives WAL segments from a leader and replays them locally.
|
|
///
|
|
/// Runs as a background tokio task. The receiver maintains strict
|
|
/// idempotency: segments with seqno <= `applied_seqno` are skipped.
|
|
pub struct SegmentReceiver {
|
|
transport: Arc<dyn Transport>,
|
|
signal_ledger: Arc<SignalLedger>,
|
|
replication_state: Arc<ReplicationState>,
|
|
leader_shard: ShardId,
|
|
}
|
|
|
|
impl SegmentReceiver {
|
|
pub fn new(
|
|
transport: Arc<dyn Transport>,
|
|
signal_ledger: Arc<SignalLedger>,
|
|
replication_state: Arc<ReplicationState>,
|
|
leader_shard: ShardId,
|
|
) -> Self {
|
|
Self { transport, signal_ledger, replication_state, leader_shard }
|
|
}
|
|
|
|
pub fn start(self: Arc<Self>, shutdown_rx: tokio::sync::watch::Receiver<bool>)
|
|
-> tokio::task::JoinHandle<()>
|
|
{
|
|
tokio::spawn(async move {
|
|
self.run(shutdown_rx).await;
|
|
})
|
|
}
|
|
|
|
async fn run(&self, mut shutdown: tokio::sync::watch::Receiver<bool>) {
|
|
loop {
|
|
tokio::select! {
|
|
segment = self.transport.recv_segment() => {
|
|
match segment {
|
|
Some(payload) => {
|
|
if let Err(e) = self.apply_segment(payload).await {
|
|
tracing::error!("SegmentReceiver: apply error: {e}");
|
|
}
|
|
}
|
|
None => {
|
|
tracing::info!("SegmentReceiver: transport closed, stopping");
|
|
break;
|
|
}
|
|
}
|
|
}
|
|
Ok(_) = shutdown.changed() => {
|
|
if *shutdown.borrow() { break; }
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
async fn apply_segment(&self, payload: WalSegmentPayload) -> Result<(), WalError> {
|
|
let seqno = payload.id.seqno;
|
|
let shard = payload.id.shard_id;
|
|
|
|
// Idempotency check: skip segments already applied.
|
|
let applied = self.replication_state
|
|
.applied_seqno(shard)
|
|
.unwrap_or(0);
|
|
if seqno <= applied {
|
|
tracing::trace!(seqno, applied, "SegmentReceiver: skipping duplicate segment");
|
|
return Ok(());
|
|
}
|
|
|
|
// BLAKE3 checksum validation.
|
|
let expected_checksum = blake3::hash(&payload.bytes);
|
|
// (Extract checksum from BatchHeader and compare)
|
|
|
|
// Decode and replay each event.
|
|
let batches = decode_wal_segment(&payload.bytes)?;
|
|
for batch in batches {
|
|
for event in batch.events {
|
|
self.signal_ledger.apply_wal_event(
|
|
event.entity_id,
|
|
&event.signal_type,
|
|
event.weight,
|
|
event.timestamp,
|
|
)?;
|
|
}
|
|
}
|
|
|
|
// Advance high-water-mark.
|
|
self.replication_state.advance(shard, seqno);
|
|
tracing::debug!(seqno, "SegmentReceiver: applied segment");
|
|
Ok(())
|
|
}
|
|
}
|
|
```
|
|
|
|
## Acceptance Criteria
|
|
|
|
- [ ] `SegmentReceiver::start()` spawns a background tokio task that reads from `transport.recv_segment()`
|
|
- [ ] BLAKE3 checksum validation: corrupted segments return `WalError::Corruption` and are NOT applied
|
|
- [ ] Idempotency: segments with `seqno <= replication_state.applied_seqno(shard)` are skipped (no double-counting)
|
|
- [ ] All events in a received segment are replayed through `SignalLedger::apply_wal_event()`
|
|
- [ ] `replication_state.advance(shard, seqno)` is called after successful replay
|
|
- [ ] Transport close (`recv_segment` returns `None`) causes the receiver to stop gracefully
|
|
- [ ] Integration test: ship 100 segments -> receiver applies all -> decay scores match
|
|
- [ ] `cargo clippy -D warnings` and `cargo fmt` pass
|