tidaldb/docs/planning/milestone-8/phase-2/task-07-replication-integration-tests.md

Task 07: Replication Integration Tests

Delivers

Integration tests in tidal/tests/m8p2_replication.rs covering the full replication stack: leader->follower segment delivery, decay score equivalence to 6 decimal places, follower read-only enforcement, lag convergence, and segment corruption rejection.

Complexity: M

Dependencies

• Tasks 01-06 complete

Technical Design

// tidal/tests/m8p2_replication.rs

use tidaldb::{TidalDb, TidalDbBuilder, NodeRole, ShardId, RegionId, NodeConfig};
use tidaldb::replication::{InProcessTransportFactory, ReplicationLagGauge};

fn leader_config(data_dir: &Path) -> Config {
    Config {
        cluster: NodeConfig {
            role: NodeRole::Leader,
            shard_id: ShardId(0),
            ..Default::default()
        },
        ..Config::with_data_dir(data_dir)
    }
}

fn follower_config(data_dir: &Path) -> Config {
    Config {
        cluster: NodeConfig {
            role: NodeRole::Follower,
            shard_id: ShardId(0),
            ..Default::default()
        },
        ..Config::with_data_dir(data_dir)
    }
}

#[tokio::test]
async fn replication_decay_scores_match() {
    // Leader writes 1,000 signals.
    // Follower replays all segments.
    // Verify: read_decay_score on follower matches leader to 6 decimal places.
}

#[tokio::test]
async fn follower_rejects_writes() {
    // Open follower. Attempt signal() write.
    // Verify: returns TidalError::ReadOnly.
}

#[tokio::test]
async fn follower_serves_retrieve_queries() {
    // Leader writes items + signals.
    // Follower applies.
    // Follower.retrieve() returns ranked results.
}

#[tokio::test]
async fn replication_lag_converges_to_zero() {
    // Leader writes 500 segments.
    // Wait for follower to apply all.
    // Assert: lag_seqno(follower) == 0 within 5 seconds.
}

#[tokio::test]
async fn corrupted_segment_is_rejected() {
    // Manually corrupt BLAKE3 checksum in segment bytes.
    // Send to follower via transport.
    // Verify: segment is not applied (decay scores unchanged).
}

#[tokio::test]
async fn leader_restart_follower_continues() {
    // Leader writes 100 signals.
    // Leader shuts down.
    // Follower serves read queries from replayed state.
    // Leader restarts; ships remaining segments.
    // Follower catches up.
}

#[tokio::test]
async fn idempotent_segment_replay() {
    // Ship same segment twice to follower.
    // Verify: signal counts NOT doubled (seqno idempotency).
}

Acceptance Criteria

• All 7 integration tests pass under cargo test --test m8p2_replication
• Test replication_decay_scores_match: leader 1K signals -> follower matches to 6 decimal places
• Test follower_rejects_writes: TidalError::ReadOnly on all write methods
• Test follower_serves_retrieve_queries: follower returns correct ranked results
• Test replication_lag_converges_to_zero: lag = 0 within 5 seconds of leader quiesce
• Test corrupted_segment_is_rejected: corrupt checksums rejected, no state change
• Test leader_restart_follower_continues: follower serves reads after leader crash
• Test idempotent_segment_replay: no double-counting on duplicate segments
• cargo clippy -D warnings and cargo fmt pass