tidaldb/tidal-net/proto/wal_shipping.proto

syntax = "proto3";
package tidal.replication.v1;

// Globally unique identifier for a WAL segment.
message WalSegmentId {
  uint32 region_id = 1;
  uint32 shard_id = 2;
  uint64 seqno = 3;
}

// A WAL segment ready for shipping to a peer shard.
message ShipSegmentRequest {
  WalSegmentId id = 1;
  bytes payload = 2;
  uint64 event_count = 3;
  // The segment's authoritative last WAL sequence number, computed by the
  // leader from the ORIGINAL (pre-community-overlay-filter) bytes. Lets the
  // receiver advance its replication-lag leader high-water-mark even for an
  // all-local segment that filters to an empty payload (obs-REPL-1). A 0 value
  // (e.g. from an older sender that omits this field) means "unknown" and the
  // receiver falls back to the per-batch boundaries it decodes.
  uint64 leader_last_seq = 4;
  // The shipping stream's baseline (m11p2): the WAL seqno at which this
  // leader's stream STARTED. Non-zero only on catch-up stream chunks from a
  // promoted leader; seqnos at or below it are pre-stream history the
  // receiver jumps its frontier past instead of treating as a gap. 0 (the
  // default, and what every live unary ship carries) means "stream from the
  // beginning".
  uint64 stream_baseline = 5;
  // The sender's leadership term (m11p4 fencing). 0 = the topology era or a
  // pre-m11p4 sender; a receiver at term >= 1 rejects term-0 traffic, and any
  // receiver rejects a term below its own (FAILED_PRECONDITION) — a deposed
  // leader's ships can never apply.
  uint64 term = 6;
  // The region claiming leadership of `term` (m11p4).
  uint32 leader_region = 7;
}

// Response to a segment shipment.
message ShipSegmentResponse {
  bool accepted = 1;
  // The receiver's contiguous applied seqno for the request's source shard at
  // acceptance time (m11p2): a monotonic hint the sender folds into its acked
  // frontier so retries of already-applied data prune and heal needs no
  // separate status fetch. 0 = unknown (older peer / no applied source wired).
  uint64 applied_seqno = 2;
  // The receiver's current term (m11p4): a value above the sender's term is
  // the sender's step-down signal.
  uint64 term = 3;
}

// Request to stream segments from a given sequence number.
message StreamRequest {
  uint32 shard_id = 1;
  uint64 from_seqno = 2;
  // The puller's current term (m11p4). The source serves only when the terms
  // match: a stale puller must rejoin first, and a stale SOURCE must step
  // down rather than serve its possibly-divergent tail.
  uint64 term = 3;
}

// Heartbeat request matching ControlPlane's ShardStats.
message HeartbeatRequest {
  uint32 shard_id = 1;
  uint32 region_id = 2;
  uint64 entity_count = 3;
  double signal_throughput_eps = 4;
  uint64 disk_bytes = 5;
  // Replication lag per peer region (region_id -> lag in events).
  map<uint32, uint64> replication_lag = 6;
  uint64 last_heartbeat_ns = 7;
  // The sender's leadership term (m11p4): a leader heartbeat is the lease
  // assertion + failure-detector input. 0 = the topology era / a non-leader
  // health probe.
  uint64 term = 8;
  // The region asserting leadership of `term` (m11p4).
  uint32 leader_region = 9;
  // The asserted term's activation stream baseline (m11p4): immutable for
  // the term; a joining follower jumps its applied frontier for the leader's
  // stream to it (seqnos at or below are pre-stream history).
  uint64 stream_baseline = 10;
  // The leader's ELECTION-TIME log position (m11p4): the term and frontier
  // of its log in the PREVIOUS stream's numbering — the same pair the vote
  // restriction compares. A joining node is DIVERGENT iff its own position
  // exceeds this lexicographically (it holds entries the new leadership's
  // history does not subsume).
  uint64 prev_log_term = 11;
  uint64 prev_log_seq = 12;
  // The sender binary's build version (m11p8 rolling-upgrade handshake), e.g.
  // "0.1.0" (the Cargo package version, stamped at the transport boundary). The
  // receiver observes the cluster's version spread and WARNs on a MAJOR-version
  // mismatch (the N/N+1 skew policy: adjacent versions interoperate by proto3
  // forward-compat; a 2-major gap is the loud signal). Never a rejection —
  // rolling upgrade REQUIRES N/N+1 to interoperate. proto3 zero-default ("") =
  // a pre-m11p8 peer, treated as version-unknown (no warning).
  string build_version = 13;
  // The leader's CURRENT flushed WAL frontier at heartbeat time (m12p5): the
  // live high-water-mark every follower should converge to (its
  // `ship_feed.flushed_seq()`, same numbering as the segment `leader_last_seq`
  // and a follower's per-shard `applied_seqno`). Unlike `stream_baseline` (the
  // term's immutable activation point), this advances as the leader writes — so
  // a follower can compute lag and converge readiness from the HEARTBEAT, which
  // flows even on an IDLE cluster, instead of waiting for observed ship traffic
  // (the idle-readiness stall, m12p5). proto3 zero-default (0) = a pre-m12p5
  // leader, treated as "unknown" — the follower falls back to the status-poll
  // readiness path.
  uint64 leader_last_seq = 14;
}

// Heartbeat acknowledgement.
message HeartbeatResponse {
  bool acknowledged = 1;
  // The responder's current term (m11p4): above the sender's term = the
  // sender's step-down signal.
  uint64 term = 2;
  // Whether the responder accepted the sender's leadership assertion
  // (false = the sender's term is stale).
  bool accepted = 3;
  // The responder binary's capability bit-field (m11p5). See the BIT REGISTRY
  // below `WalShipping`. proto3 zero-default = a pre-m11p5 follower (incapable
  // of folding kind-4 membership records): a `0` here gates the leader off
  // appending any conf-change until every voter reports the kind-4 bit.
  uint64 capabilities = 4;
  // The TYPED REMOVED SIGNAL (m11p5 §3.3): true iff the responder's APPLIED
  // roster lists the heartbeat SENDER (`region_id`) as a `Removed` member.
  // proto3 zero-default = false = "not removed" = backward-compatible with a
  // pre-m11p5 responder. A sender that sees `true` was decommissioned: it
  // flips its readiness to 503 and suppresses campaigning WITHOUT latching a
  // reseed marker (a remove is not a reseed). This is the delivery path for a
  // node that MISSED the `Removed` record in the stream (it was down during
  // the removal-delivery grace) — it learns of its removal from the leader's
  // heartbeat refusal rather than the log.
  bool removed = 5;
}

// A follower's self-report of its durable frontier (m11p3).
//
// Pushed by the receiver once per apply round — fully decoupled from ship
// acks, so the leader's quorum commit index stays fresh even when its
// outbound ships stall (gap-parked follower, quiet leader, pull catch-up).
message AppliedReport {
  // The reporting node's shard id.
  uint32 reporter_shard = 1;
  // The stream's source shard (the leader being reported to).
  uint32 source_shard = 2;
  // The reporter's contiguous durably-applied seqno for that stream.
  uint64 applied_seqno = 3;
  // The reporter's current term (m11p4): the leader folds a report into its
  // quorum commit index ONLY when this matches the index's activation term —
  // a stale or cross-leadership report can never advance commitment.
  uint64 reporter_term = 4;
  // The reporter binary's capability bit-field (m11p5). See the BIT REGISTRY
  // below `WalShipping`. proto3 zero-default = a pre-m11p5 reporter. Reported
  // on the frontier push (decoupled from heartbeats) so the leader learns a
  // follower's capabilities even when it is the one being pulled from — the
  // join/conf-change gate (§3.1) reads the leader's last-seen-per-peer map.
  uint64 capabilities = 5;
}

// Applied-report acknowledgement.
message AppliedReportAck {
  bool acknowledged = 1;
}

// A pre-vote or vote request (m11p4 leader election).
message VoteRequest {
  // The term votes are requested for. For a pre-vote this is the PROPOSED
  // term (candidate's current + 1) — nothing has been bumped.
  uint64 term = 1;
  uint32 candidate_region = 2;
  // The candidate's log position for the up-to-date restriction,
  // compared lexicographically: (last_log_term, last_log_seq).
  uint64 last_log_term = 3;
  uint64 last_log_seq = 4;
  // Pre-vote probe: changes no voter state, never inflates terms.
  bool prevote = 5;
  // Leadership-transfer election (`TimeoutNow`): voters skip the
  // leader-freshness refusal — the current leader sanctioned this.
  bool transfer = 6;
}

// A vote (or pre-vote) reply.
message VoteResponse {
  // The voter's current term (above the candidate's = step-down signal).
  uint64 term = 1;
  bool granted = 2;
  // The TYPED REMOVED SIGNAL (m11p5 §3.3): true iff the voter's APPLIED roster
  // lists the CANDIDATE (`candidate_region`) as a `Removed` member. proto3
  // zero-default = false = backward-compatible. A candidate that sees `true`
  // was decommissioned: it flips readiness to 503 and suppresses campaigning,
  // exempt from the reseed marker. Vote refusals are a second delivery channel
  // (alongside heartbeats) for a removed node that missed the `Removed` record.
  bool removed = 3;
}

// The current leader tells `target` to start an immediate transfer election
// (m11p4 fenced promote).
message TimeoutNowRequest {
  // The sanctioning leader's current term.
  uint64 term = 1;
  uint32 leader_region = 2;
}

message TimeoutNowResponse {
  // Whether the target started an election.
  bool accepted = 1;
}

// ── Snapshot transfer (m11p5 §2): the reseed and joiner catch-up path ──────
//
// A joiner (or a node behind a compacted leader) cannot stream from the WAL
// when the leader has compacted past its `from_seqno` (the structural hole
// `StreamSegments` answers with FAILED_PRECONDITION + an `x-tidal-catchup`
// trailer). `FetchSnapshot` ships a staged `create_backup` artifact as a
// chunked server-stream, term-fenced exactly like `StreamSegments`, after
// which the joiner resumes `StreamSegments` from the snapshot's seqno.

// A puller's request for a snapshot to install before resuming the stream.
message SnapshotRequest {
  uint32 shard_id = 1;
  // The puller's frontier+1: the first seqno it still needs. The source
  // answers `needed=false` (just stream) when the live WAL can still serve
  // this, else stages an artifact whose recovered tail is >= from_seqno.
  uint64 from_seqno = 2;
  // The puller's current term (m11p5 fencing): a deposed source must step
  // down rather than serve a snapshot of its possibly-divergent state; a
  // stale puller must rejoin the current term first.
  uint64 term = 3;
}

// One file entry in a snapshot manifest: a relative path under the artifact
// root plus its size and content hash. The BLAKE3 is the END-TO-END integrity
// contract — computed once when the artifact is staged (node-side), forwarded
// verbatim here, and verified by the installing puller. The server NEVER
// recomputes it per chunk (the manifest hash is authoritative).
message SnapshotFileEntry {
  string path = 1;
  uint64 size = 2;
  bytes blake3 = 3;
}

// The leading chunk of a `FetchSnapshot` stream: either "no snapshot needed,
// just stream" or the manifest of files that follow.
message SnapshotHeader {
  // false = the live WAL still serves `from_seqno`; the puller resumes
  // `StreamSegments` directly and NO file chunks follow this header. true =
  // the file chunks below install before the stream resumes.
  bool needed = 1;
  // The snapshot's recovered WAL tail: the seqno the installed artifact is
  // valid at (the puller advances its frontier to this, then resumes from
  // `snapshot_seq + 1`). 0 when `needed=false`.
  uint64 snapshot_seq = 2;
  // The term and region of the staging leader (m11p5 fencing visibility).
  uint64 term = 3;
  uint32 leader_region = 4;
  // The complete manifest: every file the puller must receive and verify.
  // Empty when `needed=false`.
  repeated SnapshotFileEntry files = 5;
}

// One chunk of one file's bytes. Files stream in manifest order; a file is
// complete when a chunk with `last=true` arrives (offset + data.len() then
// equals the manifest size).
message SnapshotFileChunk {
  string path = 1;
  uint64 offset = 2;
  bytes data = 3;
  bool last = 4;
}

// A message in the `FetchSnapshot` stream: the header (always first) then file
// chunks.
message SnapshotChunk {
  oneof chunk {
    SnapshotHeader header = 1;
    SnapshotFileChunk file = 2;
  }
}

// ── Membership conf-changes (m11p5 §3.3): the join verb ────────────────────
//
// A joining node POSTs/dials any seed; a non-leader seed refuses with a leader
// hint and the joiner re-targets. The leader assigns a PERMANENT id, appends a
// Learner kind-4 record, waits for same-term quorum commit, and answers with
// the full roster so the joiner can build its peer tables before catching up.

// One member in a `JoinResponse` roster (the wire shape of a `MemberEntry`).
message MemberInfo {
  uint32 id = 1;
  string name = 2;
  string grpc_addr = 3;
  string http_addr = 4;
  // 0 = Voter, 1 = Learner, 2 = Removed (matches MemberRole's on-disk byte).
  uint32 role = 5;
}

// A node's request to join the cluster (m11p5 §3.3). Idempotent by `name`:
// a re-join from a known member returns its existing id/role and appends
// nothing.
message JoinRequest {
  string name = 1;
  // Advertised gRPC address (host:port; DNS name or literal IP).
  string grpc_addr = 2;
  // Advertised HTTP address (scheme://host:port).
  string http_addr = 3;
  // The joiner binary's capability bit-field (m11p5 §3.1). A joiner that is
  // not kind-4 capable can never participate in conf-changes; the leader's
  // gate also requires every current voter to be capable.
  uint64 capabilities = 4;
}

// The leader's answer to a join (m11p5 §3.3).
message JoinResponse {
  // false = refused; `refusal_reason` says why and (for a non-leader seed)
  // `leader_*` names where to re-target.
  bool accepted = 1;
  string refusal_reason = 2;
  // The id the joiner was assigned (or its existing id on an idempotent
  // re-join). Meaningful only when `accepted`.
  uint32 assigned_id = 3;
  // The current leadership term + the leader's addresses, so the joiner can
  // persist the term and dial the leader directly.
  uint64 term = 4;
  string leader_region = 5;
  string leader_grpc_addr = 6;
  string leader_http_addr = 7;
  // The full roster after the join (the joiner's source of truth for its peer
  // tables), and the conf version it is at.
  repeated MemberInfo members = 8;
  uint64 membership_version = 9;
}

// WAL segment shipping service between tidalDB shards.
//
// ── CAPABILITY BIT REGISTRY (m11p5 §3.1) ───────────────────────────────────
// `capabilities` is a `uint64` bit-field on `HeartbeatResponse` and
// `AppliedReport`. proto3's zero-default means a pre-m11p5 binary reports `0`
// (no bits) — conservatively "incapable". Allocate bits append-only; NEVER
// reuse or renumber a retired bit.
//   bit 0 (0x1): kind-4 membership record capable — this binary can fold a
//                `MembershipRecord` WAL blob without halting its receiver. The
//                leader refuses `JoinCluster`/conf-changes until every voter
//                reports this bit (a kind-4 record to a pre-p5 follower is an
//                unknown batch kind → permanent receiver halt).
service WalShipping {
  // Ship a single WAL segment to a peer shard (unary).
  rpc ShipSegment(ShipSegmentRequest) returns (ShipSegmentResponse);

  // Stream WAL segments from a given sequence number (server-streaming).
  rpc StreamSegments(StreamRequest) returns (stream ShipSegmentRequest);

  // Stream a staged snapshot artifact to a joining/reseeding peer (m11p5).
  // The header chunk arrives first (needed / manifest), then file chunks.
  rpc FetchSnapshot(SnapshotRequest) returns (stream SnapshotChunk);

  // Periodic health check for the ControlPlane; with m11p4, the leader's
  // lease assertion and the failure detector's input.
  rpc Heartbeat(HeartbeatRequest) returns (HeartbeatResponse);

  // Follower -> leader durable-frontier report (m11p3 quorum acks).
  rpc ReportApplied(AppliedReport) returns (AppliedReportAck);

  // Pre-vote / vote (m11p4 leader election).
  rpc RequestVote(VoteRequest) returns (VoteResponse);

  // Fenced leadership transfer: start an immediate election (m11p4).
  rpc TimeoutNow(TimeoutNowRequest) returns (TimeoutNowResponse);

  // Join the cluster (m11p5 §3.3). Dialled by a joiner against any seed; a
  // non-leader seed refuses with a leader hint. The leader assigns a permanent
  // id, appends a Learner kind-4 record, waits for same-term quorum commit, and
  // answers with the roster. Idempotent by name.
  rpc JoinCluster(JoinRequest) returns (JoinResponse);
}