6651c14adc
m11p7 — secure the cluster, all opt-in (pre-m11p7 byte-for-byte):
- gRPC replication mTLS by default via a custom tokio-rustls acceptor +
DynamicCertResolver; zero-drop content-hash cert rotation (k8s ..data swap,
no pod restart, no inotify)
- inter-node HTTP TLS sharing the same resolver (one rotation, both planes) +
per-node keyed-BLAKE3 signed x-tidal-node-token; marker-without-token -> 403
- admin audit log (operator-leg only) + per-principal rate limit (engine
RateLimiter; sibling nodes exempt)
- k8s cert-manager manifest (certs.yaml) + scripts/gen-cluster-certs.sh fallback;
secret.example.yaml gains TIDAL_CLUSTER_KEY (file-mounted, hot-rotatable)
- exit gate verified real: mtls.rs (gRPC foreign-pod), cluster_security.rs
(HTTP foreign + zero-drop rotation under load), 7 security unit tests
perf — instrument floor (sweep Wave 1):
- new tidal/benches/wal.rs + tidal-server/benches/scatter.rs
- p99->mean honesty relabel; sweep manifest at docs/reviews/perf-sweep-2026-06-13.md
- add @tidal-performance agent (Martin Thompson)
new: cluster/{audit,http_tls,security}.rs, tests/cluster_security.rs,
docs/planning/milestone-11/phase-7.md
92 lines
4.4 KiB
Rust
92 lines
4.4 KiB
Rust
//! Cluster mode: multi-region tidalDB behind an HTTP surface.
|
||
//!
|
||
//! Two cluster modes share this module:
|
||
//!
|
||
//! * **Single-process** ([`ClusterState`], the m8p8 dev/demo default): every
|
||
//! region runs inside ONE process, wrapping [`SimulatedCluster`] with region
|
||
//! name ↔ [`RegionId`] mapping. Replication between regions still traverses a
|
||
//! real [`GrpcTransport`] on loopback (not in-process channels), but a crash
|
||
//! takes the whole "cluster" down — there is no process isolation.
|
||
//! * **Multi-process** ([`ClusterNode`], the m8p10→m11p6 mode, selected by
|
||
//! `--region`): this process is a [`ClusterNode`] hosting a
|
||
//! `BTreeMap<ShardId, Arc<ShardReplica>>` — one [`ShardReplica`] (one
|
||
//! [`TidalDb`] + [`GrpcTransport`] + election/commit/membership) per shard
|
||
//! group it replicates. Entity writes hash-route to the owning group's leader
|
||
//! (apply locally or forward); corpus reads scatter over hosted groups. With
|
||
//! `shards:` absent it holds ONE group spanning every region — byte-for-byte
|
||
//! the original one-region-per-process node. Process isolation is real;
|
||
//! quorum-ack writes (m11p3) and elected failover (m11p4) are wired per group.
|
||
//!
|
||
//! Both modes are gated behind the same explicit operator opt-in
|
||
//! ([`ensure_experimental_enabled`]).
|
||
//!
|
||
//! # Module layout (M0–M10 review Maintainability-S follow-up)
|
||
//!
|
||
//! The original single 1379-line `cluster.rs` carried four loosely-coupled
|
||
//! concerns; it was split (behavior-neutral) into:
|
||
//!
|
||
//! * [`topology`] — `TopologySpec` / `RegionSpec` / `load_topology` / validation
|
||
//! * [`transport`] — single-process self-loop gRPC transport wiring
|
||
//! * [`state`] — [`ClusterState`] (single-process) + the experimental gate
|
||
//! * [`routes`] — the single-process router + handlers
|
||
//! * [`node`] — [`ClusterNode`] (the process) hosting [`ShardReplica`] groups
|
||
//! (multi-process) + its gateway router/handlers
|
||
//!
|
||
//! [`GrpcTransport`]: tidal_net::GrpcTransport
|
||
//! [`RegionId`]: tidaldb::replication::shard::RegionId
|
||
//! [`SimulatedCluster`]: tidaldb::testing::SimulatedCluster
|
||
//! [`TidalDb`]: tidaldb::TidalDb
|
||
|
||
/// m11p7 admin-verb audit log: one structured record (principal, term, target,
|
||
/// outcome) per promote/partition/heal/conf-change, to a tracing target + an
|
||
/// optional append-only JSONL file.
|
||
pub(crate) mod audit;
|
||
pub(crate) mod election_driver;
|
||
pub(crate) mod forward;
|
||
/// m11p7 inter-node HTTP TLS: a `tokio-rustls` axum listener reusing tidal-net's
|
||
/// hot-swappable cert resolver (one rotation covers gRPC + HTTP).
|
||
///
|
||
/// Public so the binary's `serve_state` can serve the region surface over TLS.
|
||
pub mod http_tls;
|
||
/// Seed-join boot (m11p5 §3.4–§3.6): a node not declared in the local topology
|
||
/// joins an existing cluster by contacting a `--seed`.
|
||
///
|
||
/// Public so the binary's `run_seed_join_cluster` can invoke `seed_join_boot`
|
||
/// before `ShardReplica::new`.
|
||
pub mod join_boot;
|
||
/// The membership runtime (m11p5 §3.1–§3.3): effective roster, the one fenced
|
||
/// apply path, conf-change gates, and leader-side join/remove planning.
|
||
pub(crate) mod membership;
|
||
pub(crate) mod node;
|
||
/// Boot-time snapshot install + swap-recovery (m11p5 §2.2–§2.7).
|
||
///
|
||
/// Public so the binary's `run_region_cluster` can invoke `run_boot_install`
|
||
/// before `ShardReplica::new`.
|
||
pub mod reseed;
|
||
pub(crate) mod routes;
|
||
/// m11p7 cluster security: reloadable bearer + cluster keys, per-node signed
|
||
/// internal tokens, and the request principal.
|
||
///
|
||
/// Public so the binary's `serve_state` / rotation poller can build and reload
|
||
/// the shared [`security::ClusterCreds`].
|
||
pub mod security;
|
||
pub(crate) mod snapshot;
|
||
mod state;
|
||
mod topology;
|
||
mod transport;
|
||
|
||
// ── Public API (preserved across the split) ─────────────────────────────────
|
||
|
||
pub use node::{ClusterNode, ShardReplica, build_region_router};
|
||
pub use routes::build_cluster_router;
|
||
pub use state::{ClusterMode, ClusterState, EXPERIMENTAL_CLUSTER_ENV, ensure_experimental_enabled};
|
||
pub use topology::{
|
||
ElectionSpec, GrpcTlsSpec, RegionSpec, ReplicationSpec, ShardReplicaSpec, ShardSpec,
|
||
TimeoutsSpec, TopologySpec, WalSpec, load_topology, validate_multiproc,
|
||
};
|
||
|
||
// The OpenAPI documents (`crate::openapi`) reach the handlers/DTOs through
|
||
// `cluster::routes::…` / `cluster::node::…` directly, so the
|
||
// `#[utoipa::path]`-generated `__path_*` types resolve in the module that
|
||
// defines them. `node` reaches `ClusterAppError` via `super::routes`.
|