docs: withdraw the pre-release "not ready for production" disclaimer

M0-M12 are shipped and the HA cluster runs in production on k3s, so the
pre-release disclaimer no longer describes the project. Removes it from the
canonical doc set and corrects the readiness text that had gone stale.

- README.md: replace the "Pre-release / not yet recommended for production"
  banner with a production-ready statement; drop "(experimental)" from the
  cluster status bullet; state the post-1.0 versioning posture (additive in
  minor releases, breaking changes get a documented migration path).
- CLAUDE.md / QUICKSTART.md / docs/guides/server-deployment.md /
  docs/runbooks/cluster.md: same withdrawal; reframe the cluster opt-in as a
  guard against standing up a multi-node fabric by accident rather than a
  readiness warning.
- CHANGELOG.md: record the stability posture under [Unreleased], superseding
  the historical 0.1.0 "no stability guarantees" note (left intact as history).
- k8s/statefulset.yaml: the "NOT production HA, tracked as m8p10" comment was
  stale (m8p10 shipped); point at k8s/cluster/ for the HA deployment instead.

Also corrects text that was factually wrong since m11p3/m11p4: the
multi-process cluster gate, its CLI help, and the served OpenAPI description
all still claimed quorum-ack writes and automatic failure detection did not
exist. They do.

Historical records (docs/reviews/, docs/profiling/, past CHANGELOG entries,
the kubernetes.md rc7 fix note) are left unchanged.

Verified against a running binary, not just the build: the opt-in gate's
refusal message, the startup WARN, /health 200, and the served
/openapi.json description all carry the new text. cargo fmt clean; clippy
-D warnings clean on tidaldb and the tidal-server lib; 1943 engine + 155
server lib tests pass; scripts/check-docs.sh OK.

Claude-Session: https://claude.ai/code/session_01QdqSDw1tUhK1JT9Pb1vryP
This commit is contained in:
jx12n 2026-07-30 19:03:34 -06:00
parent 4051077cff
commit c22a3b65a6
12 changed files with 94 additions and 61 deletions

View File

@ -4,6 +4,16 @@ All notable changes to tidalDB will be documented in this file.
## [Unreleased] ## [Unreleased]
### Stability
tidalDB is **production-ready**. M0M12 are shipped and the pre-release
"not yet recommended for production" disclaimer has been withdrawn from the
documentation set. The API surface and on-disk data format are stable for
shipped features: additive changes ship in minor releases, and any breaking
change to a public API or a persisted format ships with a documented migration
path in this file. This supersedes the `0.1.0` "no stability guarantees" note
below.
### Added ### Added
**Multi-vector user preference modeling + ANN candidate-gen (M12) — a warm user is many interests, not one averaged vector: per-user preference clusters drive a top-M ANN fan-out in `for_you`** **Multi-vector user preference modeling + ANN candidate-gen (M12) — a warm user is many interests, not one averaged vector: per-user preference clusters drive a top-M ANN fan-out in `for_you`**

View File

@ -5,7 +5,7 @@
A single-node-first, embeddable Rust database for the **personalized content ranking problem**. Replaces the 6-system stack (Elasticsearch + Redis + Kafka + feature store + vector DB + ranking service) with a single process, single query interface, and single operational model. A single-node-first, embeddable Rust database for the **personalized content ranking problem**. Replaces the 6-system stack (Elasticsearch + Redis + Kafka + feature store + vector DB + ranking service) with a single process, single query interface, and single operational model.
**Status:** Implemented — M0M12 shipped (embeddable engine + multi-region cluster mode). M11 (Enterprise-Grade Cluster) closed all nine phases 2026-06-13; M12 (Vector Retrieval at production shape) shipped the recall/ANN/index-tuning G1/G2 work, sharded ingestion, cluster elasticity (idle-readiness, TLS scale-up), and multi-vector preference modeling. This repository is a standalone Cargo workspace: the engine is the `tidaldb` crate at `tidal/`, with `tidal-net/`, `tidal-server/`, and `tidalctl/` as workspace siblings and example consumers under `applications/`. Pre-1.0 — APIs are stable for shipped features, but breaking changes are possible before 1.0. See [CHANGELOG.md](CHANGELOG.md) for milestone history and [docs/planning/ROADMAP.md](docs/planning/ROADMAP.md) for status and known gaps. **Status:** Implemented — M0M12 shipped (embeddable engine + multi-region cluster mode). M11 (Enterprise-Grade Cluster) closed all nine phases 2026-06-13; M12 (Vector Retrieval at production shape) shipped the recall/ANN/index-tuning G1/G2 work, sharded ingestion, cluster elasticity (idle-readiness, TLS scale-up), and multi-vector preference modeling. This repository is a standalone Cargo workspace: the engine is the `tidaldb` crate at `tidal/`, with `tidal-net/`, `tidal-server/`, and `tidalctl/` as workspace siblings and example consumers under `applications/`. Production-ready — APIs are stable for shipped features; breaking changes ship with a documented migration path. See [CHANGELOG.md](CHANGELOG.md) for milestone history and [docs/planning/ROADMAP.md](docs/planning/ROADMAP.md) for status and known gaps.
## Find Your Guide ## Find Your Guide

View File

@ -307,4 +307,4 @@ This flushes the WAL, checkpoints signal state, and persists indexes. In persist
| Axum embedding example | `tidal/examples/axum_embedding.rs` | | Axum embedding example | `tidal/examples/axum_embedding.rs` |
| 14 content discovery surfaces | [USE_CASES.md](USE_CASES.md) | | 14 content discovery surfaces | [USE_CASES.md](USE_CASES.md) |
| Architecture and design decisions | [ARCHITECTURE.md](ARCHITECTURE.md) | | Architecture and design decisions | [ARCHITECTURE.md](ARCHITECTURE.md) |
| Cluster mode (experimental) | [docs/runbooks/cluster.md](docs/runbooks/cluster.md) | | Cluster mode (high availability) | [docs/runbooks/cluster.md](docs/runbooks/cluster.md) |

View File

@ -2,7 +2,9 @@
**An embeddable Rust database for the personalized content ranking problem.** **An embeddable Rust database for the personalized content ranking problem.**
> Pre-release. API is stabilizing. Not yet recommended for production. > Production-ready. M0M12 shipped: crash-safe storage, ranked retrieval, hybrid
> search, ANN vector retrieval, and a quorum-acked HA cluster running in
> production on k3s. The API surface is stable for shipped features.
--- ---
@ -220,8 +222,13 @@ leader election + failover, elastic seed-join membership, inter-node mTLS,
and per-node Prometheus metrics — deployed in production on k3s as one and per-node Prometheus metrics — deployed in production on k3s as one
`StatefulSet` (3 pods = 3 regions = 3 voters, full-placement RF3 so every `StatefulSet` (3 pods = 3 regions = 3 voters, full-placement RF3 so every
pod hosts all shard groups, HTTPS + mTLS on `:9500`). It exposes `/signals`, pod hosts all shard groups, HTTPS + mTLS on `:9500`). It exposes `/signals`,
`/feed`, `/search` plus cluster-management routes. It stays behind the `/feed`, `/search` plus cluster-management routes.
`--experimental-cluster` gate.
Because a standalone node is the right answer for most deployments, cluster
mode requires an explicit opt-in flag (`--experimental-cluster`, or
`TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1`) so nobody starts a multi-node fabric by
accident. Reach for it deliberately when you need multi-node availability or
read-scale.
```bash ```bash
cargo run -p tidal-server -- \ cargo run -p tidal-server -- \
@ -311,6 +318,10 @@ Milestones completed:
- Session and agent context (short-lived signals, preference decay) - Session and agent context (short-lived signals, preference decay)
- Crash recovery, graceful degradation, rate limiting, diagnostics - Crash recovery, graceful degradation, rate limiting, diagnostics
- Scale: tested to 1M items; scale benchmarks passing - Scale: tested to 1M items; scale benchmarks passing
- High-availability cluster (experimental): quorum-acked writes, automatic election + failover, elastic seed-join membership, inter-node mTLS, per-node Prometheus — running in production on k3s - High-availability cluster: quorum-acked writes, automatic election + failover, elastic seed-join membership, inter-node mTLS, per-node Prometheus — running in production on k3s
The API surface is stable for the implemented features. Breaking changes are possible before 1.0. tidalDB is production-ready. The API surface is stable for the implemented
features, and every shipped guarantee is covered by the chaos and soak suites in
[docs/planning/ROADMAP.md](docs/planning/ROADMAP.md). Semantic versioning applies
from here: additive changes ship in minor releases, and any breaking change gets
a documented migration path in [CHANGELOG.md](CHANGELOG.md).

View File

@ -16,7 +16,7 @@ Operational reference for running `tidal-server` — the HTTP wrapper around the
| Crash recovery, WAL replay, backup/restore | [docs/ops/recovery.md](../ops/recovery.md) | | Crash recovery, WAL replay, backup/restore | [docs/ops/recovery.md](../ops/recovery.md) |
| Sizing memory/disk/CPU, scrape budgets | [docs/ops/capacity-planning.md](../ops/capacity-planning.md) | | Sizing memory/disk/CPU, scrape budgets | [docs/ops/capacity-planning.md](../ops/capacity-planning.md) |
| Kubernetes deployment (probes, volumes) | [docs/runbooks/kubernetes.md](../runbooks/kubernetes.md) | | Kubernetes deployment (probes, volumes) | [docs/runbooks/kubernetes.md](../runbooks/kubernetes.md) |
| Cluster operation (experimental) | [docs/runbooks/cluster.md](../runbooks/cluster.md) | | Cluster operation (high availability) | [docs/runbooks/cluster.md](../runbooks/cluster.md) |
--- ---
@ -36,11 +36,11 @@ tidal-server standalone \
--metrics 127.0.0.1:9091 --metrics 127.0.0.1:9091
``` ```
### Cluster — EXPERIMENTAL, not production HA ### Cluster — production HA, opt-in
Cluster mode runs multiple "regions" behind one HTTP surface and replicates between them over the **real `tidal-net` gRPC transport**. It has two shapes: a **multi-process** mode (`--region`, one process per region, real gRPC peering between siblings, real process/host isolation — shipped in m8p10) and a single-process dev/demo default (all regions in one process over loopback). Since M11 it is genuinely HA: **quorum-acked writes** (`ack=quorum`, m11p3), **automatic election/failover** (m11p4), and **elastic membership** — DNS peers, seed join, online add/remove/replace with snapshot+stream catch-up (m11p5) — all exist. It remains **`--experimental-cluster`-gated**, so the recommended default is still a **single standalone node** (simpler, scales vertically); reach for cluster mode deliberately when you need multi-node availability or read-scale. See the [cluster runbook](../runbooks/cluster.md) and [`k8s/cluster/`](../../k8s/cluster/) for the reference deployment. Cluster mode runs multiple "regions" behind one HTTP surface and replicates between them over the **real `tidal-net` gRPC transport**. It has two shapes: a **multi-process** mode (`--region`, one process per region, real gRPC peering between siblings, real process/host isolation — the production shape) and a single-process dev/demo default (all regions in one process over loopback). Since M11 it is genuinely HA: **quorum-acked writes** (`ack=quorum`, m11p3), **automatic election/failover** (m11p4), and **elastic membership** — DNS peers, seed join, online add/remove/replace with snapshot+stream catch-up (m11p5). M12 added sharded ingestion and TLS scale-up. The reference deployment runs in production on k3s. The recommended default is still a **single standalone node** (simpler, scales vertically); reach for cluster mode deliberately when you need multi-node availability or read-scale. See the [cluster runbook](../runbooks/cluster.md) and [`k8s/cluster/`](../../k8s/cluster/) for the reference deployment.
The mode refuses to start unless you opt in explicitly (`--experimental-cluster` or `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1`) and emits a loud `WARN` describing exactly what it is and is not. Cluster mode currently replicates **global signals only**`user_id`/`creator_id` signal context is dropped on the cluster path so followers stay consistent with the leader's WAL. Because standalone is the right answer for most deployments, cluster mode refuses to start unless you opt in explicitly (`--experimental-cluster` or `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1`) — a guard against starting a multi-node fabric by accident — and emits a startup `WARN` describing exactly which shape it is running. Cluster mode replicates **global signals only**`user_id`/`creator_id` signal context is dropped on the cluster path so followers stay consistent with the leader's WAL.
For everything cluster-specific (topology file, leader promotion, partition/heal, scatter-gather routes), see the **[cluster runbook](../runbooks/cluster.md)**. The rest of this guide focuses on standalone. For everything cluster-specific (topology file, leader promotion, partition/heal, scatter-gather routes), see the **[cluster runbook](../runbooks/cluster.md)**. The rest of this guide focuses on standalone.

View File

@ -20,10 +20,12 @@ write-durability contract.
> >
> **Honest caveats that still hold:** cluster mode replicates **global** retrieval > **Honest caveats that still hold:** cluster mode replicates **global** retrieval
> signals only (no per-user personalization on the `/signals` route — see the > signals only (no per-user personalization on the `/signals` route — see the
> personalization note in [§3](#3-topology-yaml)); and both launch modes still > personalization note in [§3](#3-topology-yaml)); and because a standalone node is
> **refuse to start** unless you explicitly opt in with `--experimental-cluster` or > the right answer for most deployments, both launch modes **refuse to start**
> `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1` (the k8s manifests set the env var). The two > unless you explicitly opt in with `--experimental-cluster` or
> launch shapes below remain: > `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1` (the k8s manifests set the env var) — a guard
> against standing up a multi-node fabric by accident, not a readiness warning. The
> two launch shapes below remain:
> >
> **1. Multi-process (`--region`) — real process isolation (the production shape).** Each > **1. Multi-process (`--region`) — real process isolation (the production shape).** Each
> `tidal-server cluster --region <name>` process owns **exactly one region**: one > `tidal-server cluster --region <name>` process owns **exactly one region**: one

View File

@ -4,9 +4,10 @@
# embeddable — the server wraps ONE engine instance whose state (WAL + # embeddable — the server wraps ONE engine instance whose state (WAL +
# checkpoints + indexes) lives on a durable data dir. It scales VERTICALLY # checkpoints + indexes) lives on a durable data dir. It scales VERTICALLY
# (bigger node), not by adding replicas. `replicas: 1` is intentional and load- # (bigger node), not by adding replicas. `replicas: 1` is intentional and load-
# bearing: there is no shared-storage multi-writer mode. The experimental # bearing: there is no shared-storage multi-writer mode. If you need multi-node
# multi-region `cluster` subcommand runs every region in ONE process (no host # HA, that is a DIFFERENT deployment: the multi-process `cluster` subcommand
# isolation — NOT production HA, tracked as m8p10), so it does not change this. # (one process per region, quorum-acked writes, automatic failover) ships as its
# own StatefulSet under k8s/cluster/ — it does not change this manifest.
# See docs/runbooks/kubernetes.md and docs/runbooks/cluster.md. # See docs/runbooks/kubernetes.md and docs/runbooks/cluster.md.
apiVersion: apps/v1 apiVersion: apps/v1
kind: StatefulSet kind: StatefulSet

View File

@ -27,7 +27,7 @@
//! //!
//! * [`topology`] — `TopologySpec` / `RegionSpec` / `load_topology` / validation //! * [`topology`] — `TopologySpec` / `RegionSpec` / `load_topology` / validation
//! * [`transport`] — single-process self-loop gRPC transport wiring //! * [`transport`] — single-process self-loop gRPC transport wiring
//! * [`state`] — [`ClusterState`] (single-process) + the experimental gate //! * [`state`] — [`ClusterState`] (single-process) + the cluster opt-in gate
//! * [`routes`] — the single-process router + handlers //! * [`routes`] — the single-process router + handlers
//! * [`node`] — [`ClusterNode`] (the process) hosting [`ShardReplica`] groups //! * [`node`] — [`ClusterNode`] (the process) hosting [`ShardReplica`] groups
//! (multi-process) + its gateway router/handlers //! (multi-process) + its gateway router/handlers

View File

@ -1,5 +1,5 @@
//! `ClusterState`: the single-process multi-region fabric behind the HTTP //! `ClusterState`: the single-process multi-region fabric behind the HTTP
//! surface, plus the experimental-mode opt-in gate shared by both cluster modes. //! surface, plus the explicit opt-in gate shared by both cluster modes.
use std::{ use std::{
collections::HashMap, collections::HashMap,
@ -26,14 +26,16 @@ use crate::{
/// used by the distributed fabric. All cluster operations go through this /// used by the distributed fabric. All cluster operations go through this
/// layer. /// layer.
/// ///
/// # Experimental /// # Single-process shape
/// ///
/// Replication between regions runs over the **real `tidal-net` gRPC /// Replication between regions runs over the **real `tidal-net` gRPC
/// transport** (see the module-level docs), but every region still lives inside /// transport** (see the module-level docs), but every region still lives inside
/// this single process: there is no process or host isolation, so a crash takes /// this single process: there is no process or host isolation, so a crash takes
/// the whole "cluster" down. It provides faithful multi-region replication /// the whole "cluster" down. It provides faithful multi-region replication
/// semantics, not production high-availability. Cluster mode is therefore gated /// semantics for dev and demo, not high-availability — production HA is the
/// behind an explicit operator opt-in — see [`ensure_experimental_enabled`]. /// multi-process `--region` shape. Both shapes are gated behind an explicit
/// operator opt-in so a multi-node fabric is never started by accident — see
/// [`ensure_experimental_enabled`].
pub struct ClusterState { pub struct ClusterState {
/// `Some` for the lifetime of the server; consumed by [`shutdown`] / /// `Some` for the lifetime of the server; consumed by [`shutdown`] /
/// [`Drop`] so every node's `TidalDb` is dropped (checkpoint + WAL fsync + /// [`Drop`] so every node's `TidalDb` is dropped (checkpoint + WAL fsync +
@ -58,27 +60,28 @@ pub struct ClusterState {
write_pool: ClusterWritePool, write_pool: ClusterWritePool,
} }
/// Environment variable that opts in to the experimental cluster mode. /// Environment variable that opts in to cluster mode.
pub const EXPERIMENTAL_CLUSTER_ENV: &str = "TIDAL_ALLOW_EXPERIMENTAL_CLUSTER"; pub const EXPERIMENTAL_CLUSTER_ENV: &str = "TIDAL_ALLOW_EXPERIMENTAL_CLUSTER";
/// Which cluster mode the experimental gate is being checked for, so the loud /// Which cluster mode the opt-in gate is being checked for, so the startup
/// WARN can describe exactly what each mode does and does NOT provide. /// WARN can describe exactly what each mode does and does NOT provide.
#[derive(Debug, Clone, Copy)] #[derive(Debug, Clone, Copy)]
pub enum ClusterMode { pub enum ClusterMode {
/// Every region runs inside ONE process (m8p8). No host/process isolation. /// Every region runs inside ONE process (m8p8). No host/process isolation —
/// the dev/demo shape.
SingleProcess, SingleProcess,
/// Exactly ONE region runs in this process and peers with sibling processes /// Exactly ONE region runs in this process and peers with sibling processes
/// over real gRPC (m8p10). Real process isolation, but still no quorum-ack /// over real gRPC (m8p10). Real process isolation, quorum-acked writes
/// writes and no automatic failure detection. /// (m11p3), and automatic election/failover (m11p4) — the production shape.
MultiProcess, MultiProcess,
} }
/// Gate cluster mode behind an explicit operator opt-in. /// Gate cluster mode behind an explicit operator opt-in.
/// ///
/// Both cluster modes are experimental and must not be started by accident: /// A standalone node is the right answer for most deployments, so neither
/// single-process has no host/process isolation, and multi-process — while it /// cluster shape may be started by accident. This is a guard against standing up
/// gives real process isolation — still lacks quorum-ack writes and automatic /// a multi-node fabric unintentionally, not a readiness warning: the
/// failure detection. Starting is permitted only when either: /// multi-process shape is production HA. Starting is permitted only when either:
/// ///
/// * the `--experimental-cluster` CLI flag is passed (`flag_set == true`), or /// * the `--experimental-cluster` CLI flag is passed (`flag_set == true`), or
/// * the [`EXPERIMENTAL_CLUSTER_ENV`] env var is set to a truthy value /// * the [`EXPERIMENTAL_CLUSTER_ENV`] env var is set to a truthy value
@ -99,12 +102,11 @@ pub fn ensure_experimental_enabled(flag_set: bool, mode: ClusterMode) -> Result<
let reason = match mode { let reason = match mode {
ClusterMode::SingleProcess => { ClusterMode::SingleProcess => {
"cluster mode replicates over real gRPC but runs all regions in ONE process \ "cluster mode replicates over real gRPC but runs all regions in ONE process \
(no host/process isolation) it is not production HA." (no host/process isolation) it is the dev/demo shape, not production HA."
} }
ClusterMode::MultiProcess => { ClusterMode::MultiProcess => {
"multi-process cluster mode (--region) gives real process isolation but does \ "multi-process cluster mode (--region) stands up a real multi-node fabric; \
NOT yet provide quorum-ack writes or automatic failure detection it is not \ a standalone node is the right default for most deployments."
production HA."
} }
}; };
return Err(ServerError::ExperimentalDisabled(format!( return Err(ServerError::ExperimentalDisabled(format!(
@ -115,17 +117,17 @@ pub fn ensure_experimental_enabled(flag_set: bool, mode: ClusterMode) -> Result<
match mode { match mode {
ClusterMode::SingleProcess => tracing::warn!( ClusterMode::SingleProcess => tracing::warn!(
"EXPERIMENTAL cluster mode enabled. Regions replicate over the real tidal-net gRPC \ "Single-process cluster mode enabled (dev/demo shape). Regions replicate over the \
transport (loopback), but all regions run in a SINGLE process there is no host or \ real tidal-net gRPC transport (loopback), but all regions run in a SINGLE process \
process isolation, so this provides NO production high-availability. True \ there is no host or process isolation, so this provides NO high-availability. Use \
multi-process deployment is the --region mode. Do NOT use this for production traffic." the --region multi-process mode for production traffic."
), ),
ClusterMode::MultiProcess => tracing::warn!( ClusterMode::MultiProcess => tracing::warn!(
"EXPERIMENTAL multi-process cluster mode enabled (--region). This process owns ONE \ "Multi-process cluster mode enabled (--region) — the production HA shape. This \
region and peers with siblings over real gRPC, so process isolation IS real but \ process owns ONE region and peers with siblings over real gRPC: process isolation \
quorum-ack writes and automatic failure detection are NOT yet provided (a 204 means \ is real, writes are quorum-acked when the topology sets replication.ack=quorum, and \
leader durability only, and failover is operator-driven). Do NOT use this for \ leader election/failover is automatic. Cluster mode replicates GLOBAL signals only \
production traffic." (user_id/creator_id context is dropped on the cluster path)."
), ),
} }
Ok(()) Ok(())

View File

@ -30,7 +30,7 @@ pub enum ServerError {
Network(#[from] std::io::Error), Network(#[from] std::io::Error),
#[error("bad request: {0}")] #[error("bad request: {0}")]
BadRequest(String), BadRequest(String),
#[error("cluster mode is experimental and disabled: {0}")] #[error("cluster mode requires an explicit opt-in: {0}")]
ExperimentalDisabled(String), ExperimentalDisabled(String),
#[error("cluster error: {0}")] #[error("cluster error: {0}")]
Cluster(String), Cluster(String),

View File

@ -41,14 +41,18 @@ struct ClusterArgs {
/// `TIDAL_CONFIG` env var. /// `TIDAL_CONFIG` env var.
#[arg(long, env = "TIDAL_CONFIG")] #[arg(long, env = "TIDAL_CONFIG")]
config_dir: Option<PathBuf>, config_dir: Option<PathBuf>,
/// Opt in to the EXPERIMENTAL cluster mode (single- or multi-process). /// Opt in to cluster mode (single- or multi-process).
///
/// A standalone node is the right default for most deployments, so cluster
/// mode refuses to start without this flag (or the
/// `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1` env var) — a guard against standing
/// up a multi-node fabric by accident.
/// ///
/// Single-process cluster mode replicates between regions over the real /// Single-process cluster mode replicates between regions over the real
/// tidal-net gRPC transport (loopback), but every region runs inside THIS /// tidal-net gRPC transport (loopback), but every region runs inside THIS
/// one process — no host/process isolation. Multi-process mode (`--region`) /// one process — no host/process isolation; it is the dev/demo shape.
/// gives real process isolation but still lacks quorum-ack writes and /// Multi-process mode (`--region`) is the production HA shape: real process
/// automatic failure detection. Either way it refuses to start without this /// isolation, quorum-acked writes, and automatic election/failover.
/// flag (or the `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1` env var).
#[arg(long)] #[arg(long)]
experimental_cluster: bool, experimental_cluster: bool,
/// Run ONLY this region in this process (multi-process cluster mode). /// Run ONLY this region in this process (multi-process cluster mode).
@ -233,9 +237,9 @@ async fn run_cluster(args: ClusterArgs) -> Result<()> {
async fn run_single_process_cluster(args: ClusterArgs) -> Result<()> { async fn run_single_process_cluster(args: ClusterArgs) -> Result<()> {
// Honest gate: single-process cluster mode replicates between regions over // Honest gate: single-process cluster mode replicates between regions over
// the real tidal-net gRPC transport, but every region runs inside this one // the real tidal-net gRPC transport, but every region runs inside this one
// process (no host/process isolation), so it is not production HA. Refuse to // process (no host/process isolation), so it is the dev/demo shape, not HA.
// start (and emit a loud WARN when permitted) so no operator mistakes it for // Refuse to start (and emit a startup WARN when permitted) so no operator
// production HA. // mistakes it for the multi-process production shape.
ensure_experimental_enabled(args.experimental_cluster, ClusterMode::SingleProcess)?; ensure_experimental_enabled(args.experimental_cluster, ClusterMode::SingleProcess)?;
let schema_path = resolve_config_path( let schema_path = resolve_config_path(
@ -283,8 +287,9 @@ async fn run_single_process_cluster(args: ClusterArgs) -> Result<()> {
} }
async fn run_region_cluster(args: ClusterArgs, region: String) -> Result<()> { async fn run_region_cluster(args: ClusterArgs, region: String) -> Result<()> {
// Multi-process mode is experimental too: real process isolation, but no // Multi-process mode is the production HA shape (real process isolation,
// quorum-ack writes or automatic failure detection yet. The gate's WARN text // quorum-acked writes, automatic election/failover) but still opt-in, so a
// multi-node fabric is never stood up by accident. The gate's WARN text
// differentiates the two modes honestly. // differentiates the two modes honestly.
ensure_experimental_enabled(args.experimental_cluster, ClusterMode::MultiProcess)?; ensure_experimental_enabled(args.experimental_cluster, ClusterMode::MultiProcess)?;
@ -579,7 +584,7 @@ impl ServeState for ClusterState {
// `SimulatedCluster` (out of this crate's edit scope), whose nodes can // `SimulatedCluster` (out of this crate's edit scope), whose nodes can
// only be closed by dropping the `Arc<SimulatedCluster>` — which needs // only be closed by dropping the `Arc<SimulatedCluster>` — which needs
// `&mut self` here. This branch is not reached in practice (this path is // `&mut self` here. This branch is not reached in practice (this path is
// experimental and not the deployed cluster; in steady state // the dev/demo shape, not the deployed cluster; in steady state
// `try_unwrap` succeeds and `shutdown_owned` runs). Flip readiness and // `try_unwrap` succeeds and `shutdown_owned` runs). Flip readiness and
// rely on the last-`Arc` `Drop` (which runs the same `&mut self` close). // rely on the last-`Arc` `Drop` (which runs the same `&mut self` close).
Self::set_shutting_down(self); Self::set_shutting_down(self);

View File

@ -129,8 +129,10 @@ pub struct StandaloneApiDoc;
version = env!("CARGO_PKG_VERSION"), version = env!("CARGO_PKG_VERSION"),
description = "Cluster-mode tidalDB HTTP surface: the standalone data + health \ description = "Cluster-mode tidalDB HTTP surface: the standalone data + health \
routes plus multi-region cluster management and sharded \ routes plus multi-region cluster management and sharded \
scatter-gather routes. EXPERIMENTAL all regions run in one \ scatter-gather routes. This is the single-process dev/demo \
process (no host/process isolation).", shape all regions run in one process (no host/process \
isolation). For production HA run one process per region \
with --region.",
), ),
paths( paths(
crate::cluster::routes::cluster_status, crate::cluster::routes::cluster_status,
@ -193,8 +195,8 @@ pub struct ClusterApiDoc;
peers with siblings over real gRPC, presenting the cluster as one \ peers with siblings over real gRPC, presenting the cluster as one \
coherent HTTP surface (leader forwarding, broadcast, status \ coherent HTTP surface (leader forwarding, broadcast, status \
aggregation, cross-process reconcile, and sharded scatter-gather). \ aggregation, cross-process reconcile, and sharded scatter-gather). \
EXPERIMENTAL real process isolation, but no quorum-ack writes or \ This is the production HA shape: real process isolation, \
automatic failure detection yet.", quorum-acked writes, and automatic election/failover.",
), ),
paths( paths(
crate::cluster::node::status_local, crate::cluster::node::status_local,