jordan f4cfd6c81f feat: complete M8 replication primitives + forage enhancements + docs

Milestone 8 (phases 1-4):
- Shard-aware WAL segment naming, BatchHeader v2, ShardRouter
- Transport trait, InProcessTransport, WalShipper, FollowerDb
- HLC, PNCounter, LWWRegister, CrdtSignalState, ReconciliationEngine
- Session replication bridge with SeqNo/HWM, idempotency store

Forage application:
- Multi-source discovery engine with MAB exploration
- Embedding-based label system, server handlers, UI refresh

Other:
- QUICKSTART.md, README.md, milestone-8 planning docs
- Hard negative union semantics, RLHF export enhancements
- Recovery benchmark and visibility test expansions
- Split 8 oversized source files per CODING_GUIDELINES §9

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-02-24 13:17:19 -07:00

2.7 KiB

Raw Permalink Blame History

Task 01: Transport Trait

Delivers

Define Transport trait with send_segment / recv_segment async methods and WalSegmentPayload (segment bytes + WalSegmentId header) in tidal/src/replication/transport.rs. The trait is the abstraction boundary between replication logic (phase-independent correctness) and network I/O (deployment-specific).

Complexity: S

Dependencies

Phase 8.1 complete (WalSegmentId, ShardId)

Technical Design

// tidal/src/replication/transport.rs

use crate::replication::{ShardId, WalSegmentId};
use async_trait::async_trait;

/// A WAL segment payload ready for transport.
///
/// Contains the segment's globally unique ID, the raw segment bytes
/// (already BLAKE3-checksummed by the WAL writer), and the count of
/// events for quick validation on the receiver side.
#[derive(Debug, Clone)]
pub struct WalSegmentPayload {
    pub id: WalSegmentId,
    pub bytes: bytes::Bytes,
    pub event_count: u64,
}

/// Transport abstraction for WAL segment shipping.
///
/// Implementations include:
/// - `InProcessTransport` (for testing, via tokio mpsc channels)
/// - Future: gRPC transport for production deployments
///
/// The trait is async to support both in-memory and network transports
/// without blocking the Tokio runtime.
#[async_trait]
pub trait Transport: Send + Sync + 'static {
    /// Send a WAL segment to a follower shard.
    ///
    /// Returns `Ok(())` when the segment is durably queued for delivery.
    /// Does NOT wait for the follower to apply the segment.
    async fn send_segment(
        &self,
        to_shard: ShardId,
        payload: WalSegmentPayload,
    ) -> Result<(), TransportError>;

    /// Receive the next WAL segment from a leader.
    ///
    /// Blocks until a segment is available. Returns `None` when the
    /// transport is closed (leader has shut down).
    async fn recv_segment(&self) -> Option<WalSegmentPayload>;

    /// Returns the ShardId this transport endpoint represents.
    fn local_shard(&self) -> ShardId;
}

#[derive(Debug, thiserror::Error)]
pub enum TransportError {
    #[error("peer shard {0} not registered")]
    UnknownPeer(ShardId),
    #[error("transport channel closed")]
    Closed,
    #[error("payload too large: {size} bytes > max {max}")]
    PayloadTooLarge { size: usize, max: usize },
}

Acceptance Criteria

WalSegmentPayload has id: WalSegmentId, bytes: bytes::Bytes, event_count: u64
Transport trait has send_segment and recv_segment async methods
Transport: Send + Sync + 'static (object-safe, can be used in Arc<dyn Transport>)
TransportError covers UnknownPeer, Closed, PayloadTooLarge
cargo clippy -D warnings and cargo fmt pass

2.7 KiB Raw Permalink Blame History