jx12n 3bcfb3c576 feat: Bazel build, crate docs/ai-lookup, docker images, and engine hardening

- Add BUILD.bazel across tidal, tidal-net, tidal-server, tidalctl for bzlmod build
- Add tidal/ crate docs (README, CHANGELOG, CONTRIBUTING, AGENTS, CLAUDE, API, ARCHITECTURE) and ai-lookup reference
- Add docker standalone/cluster/deploy images, compose, and prometheus config
- Harden WAL (batch format, writer, dedup, diagnostics), text syncer/collectors, and vector registry
- Expand tidalctl CLI and tests; restructure WAL/visibility integration test suites
- Refine tidal-net transport/client/server and tidal-server cluster/scatter-gather

2026-06-07 18:29:38 -06:00

2.7 KiB

Raw Blame History

Task 01: Transport Trait

Delivers

Define Transport trait with send_segment / recv_segment async methods and WalSegmentPayload (segment bytes + WalSegmentId header) in tidal/src/replication/transport.rs. The trait is the abstraction boundary between replication logic (phase-independent correctness) and network I/O (deployment-specific).

Complexity: S

Dependencies

Phase 8.1 complete (WalSegmentId, ShardId)

Technical Design

// tidal/src/replication/transport.rs

use crate::replication::{ShardId, WalSegmentId};
use async_trait::async_trait;

/// A WAL segment payload ready for transport.
///
/// Contains the segment's globally unique ID, the raw segment bytes
/// (already BLAKE3-checksummed by the WAL writer), and the count of
/// events for quick validation on the receiver side.
#[derive(Debug, Clone)]
pub struct WalSegmentPayload {
    pub id: WalSegmentId,
    pub bytes: bytes::Bytes,
    pub event_count: u64,
}

/// Transport abstraction for WAL segment shipping.
///
/// Implementations include:
/// - `InProcessTransport` (for testing, via tokio mpsc channels)
/// - Future: gRPC transport for production deployments
///
/// The trait is async to support both in-memory and network transports
/// without blocking the Tokio runtime.
#[async_trait]
pub trait Transport: Send + Sync + 'static {
    /// Send a WAL segment to a follower shard.
    ///
    /// Returns `Ok(())` when the segment is durably queued for delivery.
    /// Does NOT wait for the follower to apply the segment.
    async fn send_segment(
        &self,
        to_shard: ShardId,
        payload: WalSegmentPayload,
    ) -> Result<(), TransportError>;

    /// Receive the next WAL segment from a leader.
    ///
    /// Blocks until a segment is available. Returns `None` when the
    /// transport is closed (leader has shut down).
    async fn recv_segment(&self) -> Option<WalSegmentPayload>;

    /// Returns the ShardId this transport endpoint represents.
    fn local_shard(&self) -> ShardId;
}

#[derive(Debug, thiserror::Error)]
pub enum TransportError {
    #[error("peer shard {0} not registered")]
    UnknownPeer(ShardId),
    #[error("transport channel closed")]
    Closed,
    #[error("payload too large: {size} bytes > max {max}")]
    PayloadTooLarge { size: usize, max: usize },
}

Acceptance Criteria

WalSegmentPayload has id: WalSegmentId, bytes: bytes::Bytes, event_count: u64
Transport trait has send_segment and recv_segment async methods
Transport: Send + Sync + 'static (object-safe, can be used in Arc<dyn Transport>)
TransportError covers UnknownPeer, Closed, PayloadTooLarge
cargo clippy -D warnings and cargo fmt pass

2.7 KiB Raw Blame History