tidaldb/tidal/docs/planning/milestone-8/phase-2/task-04-segment-receiver.md
jx12n 3bcfb3c576 feat: Bazel build, crate docs/ai-lookup, docker images, and engine hardening
- Add BUILD.bazel across tidal, tidal-net, tidal-server, tidalctl for bzlmod build
- Add tidal/ crate docs (README, CHANGELOG, CONTRIBUTING, AGENTS, CLAUDE, API, ARCHITECTURE) and ai-lookup reference
- Add docker standalone/cluster/deploy images, compose, and prometheus config
- Harden WAL (batch format, writer, dedup, diagnostics), text syncer/collectors, and vector registry
- Expand tidalctl CLI and tests; restructure WAL/visibility integration test suites
- Refine tidal-net transport/client/server and tidal-server cluster/scatter-gather
2026-06-07 18:29:38 -06:00

120 lines
4.2 KiB
Markdown

# Task 04: SegmentReceiver
## Delivers
`SegmentReceiver` background task in `tidal/src/replication/receiver.rs`. Receives `WalSegmentPayload` from transport, validates BLAKE3 checksum, decodes batches, and replays events through `SignalLedger::apply_wal_event()`. Idempotent via seqno high-water-mark.
## Complexity: M
## Dependencies
- Task 01 (Transport trait)
- Task 02 (InProcessTransport)
- Phase 8.1 (ReplicationState for high-water-mark)
## Technical Design
```rust
// tidal/src/replication/receiver.rs
/// Receives WAL segments from a leader and replays them locally.
///
/// Runs as a background tokio task. The receiver maintains strict
/// idempotency: segments with seqno <= `applied_seqno` are skipped.
pub struct SegmentReceiver {
transport: Arc<dyn Transport>,
signal_ledger: Arc<SignalLedger>,
replication_state: Arc<ReplicationState>,
leader_shard: ShardId,
}
impl SegmentReceiver {
pub fn new(
transport: Arc<dyn Transport>,
signal_ledger: Arc<SignalLedger>,
replication_state: Arc<ReplicationState>,
leader_shard: ShardId,
) -> Self {
Self { transport, signal_ledger, replication_state, leader_shard }
}
pub fn start(self: Arc<Self>, shutdown_rx: tokio::sync::watch::Receiver<bool>)
-> tokio::task::JoinHandle<()>
{
tokio::spawn(async move {
self.run(shutdown_rx).await;
})
}
async fn run(&self, mut shutdown: tokio::sync::watch::Receiver<bool>) {
loop {
tokio::select! {
segment = self.transport.recv_segment() => {
match segment {
Some(payload) => {
if let Err(e) = self.apply_segment(payload).await {
tracing::error!("SegmentReceiver: apply error: {e}");
}
}
None => {
tracing::info!("SegmentReceiver: transport closed, stopping");
break;
}
}
}
Ok(_) = shutdown.changed() => {
if *shutdown.borrow() { break; }
}
}
}
}
async fn apply_segment(&self, payload: WalSegmentPayload) -> Result<(), WalError> {
let seqno = payload.id.seqno;
let shard = payload.id.shard_id;
// Idempotency check: skip segments already applied.
let applied = self.replication_state
.applied_seqno(shard)
.unwrap_or(0);
if seqno <= applied {
tracing::trace!(seqno, applied, "SegmentReceiver: skipping duplicate segment");
return Ok(());
}
// BLAKE3 checksum validation.
let expected_checksum = blake3::hash(&payload.bytes);
// (Extract checksum from BatchHeader and compare)
// Decode and replay each event.
let batches = decode_wal_segment(&payload.bytes)?;
for batch in batches {
for event in batch.events {
self.signal_ledger.apply_wal_event(
event.entity_id,
&event.signal_type,
event.weight,
event.timestamp,
)?;
}
}
// Advance high-water-mark.
self.replication_state.advance(shard, seqno);
tracing::debug!(seqno, "SegmentReceiver: applied segment");
Ok(())
}
}
```
## Acceptance Criteria
- [ ] `SegmentReceiver::start()` spawns a background tokio task that reads from `transport.recv_segment()`
- [ ] BLAKE3 checksum validation: corrupted segments return `WalError::Corruption` and are NOT applied
- [ ] Idempotency: segments with `seqno <= replication_state.applied_seqno(shard)` are skipped (no double-counting)
- [ ] All events in a received segment are replayed through `SignalLedger::apply_wal_event()`
- [ ] `replication_state.advance(shard, seqno)` is called after successful replay
- [ ] Transport close (`recv_segment` returns `None`) causes the receiver to stop gracefully
- [ ] Integration test: ship 100 segments -> receiver applies all -> decay scores match
- [ ] `cargo clippy -D warnings` and `cargo fmt` pass