diff --git a/CHANGELOG.md b/CHANGELOG.md index 8d0f5f5..a4cf644 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,16 @@ All notable changes to tidalDB will be documented in this file. ## [Unreleased] +### Stability + +tidalDB is **production-ready**. M0–M12 are shipped and the pre-release +"not yet recommended for production" disclaimer has been withdrawn from the +documentation set. The API surface and on-disk data format are stable for +shipped features: additive changes ship in minor releases, and any breaking +change to a public API or a persisted format ships with a documented migration +path in this file. This supersedes the `0.1.0` "no stability guarantees" note +below. + ### Added **Multi-vector user preference modeling + ANN candidate-gen (M12) — a warm user is many interests, not one averaged vector: per-user preference clusters drive a top-M ANN fan-out in `for_you`** diff --git a/CLAUDE.md b/CLAUDE.md index b2c8e12..2b6acc7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -5,7 +5,7 @@ A single-node-first, embeddable Rust database for the **personalized content ranking problem**. Replaces the 6-system stack (Elasticsearch + Redis + Kafka + feature store + vector DB + ranking service) with a single process, single query interface, and single operational model. -**Status:** Implemented — M0–M12 shipped (embeddable engine + multi-region cluster mode). M11 (Enterprise-Grade Cluster) closed all nine phases 2026-06-13; M12 (Vector Retrieval at production shape) shipped the recall/ANN/index-tuning G1/G2 work, sharded ingestion, cluster elasticity (idle-readiness, TLS scale-up), and multi-vector preference modeling. This repository is a standalone Cargo workspace: the engine is the `tidaldb` crate at `tidal/`, with `tidal-net/`, `tidal-server/`, and `tidalctl/` as workspace siblings and example consumers under `applications/`. Pre-1.0 — APIs are stable for shipped features, but breaking changes are possible before 1.0. See [CHANGELOG.md](CHANGELOG.md) for milestone history and [docs/planning/ROADMAP.md](docs/planning/ROADMAP.md) for status and known gaps. +**Status:** Implemented — M0–M12 shipped (embeddable engine + multi-region cluster mode). M11 (Enterprise-Grade Cluster) closed all nine phases 2026-06-13; M12 (Vector Retrieval at production shape) shipped the recall/ANN/index-tuning G1/G2 work, sharded ingestion, cluster elasticity (idle-readiness, TLS scale-up), and multi-vector preference modeling. This repository is a standalone Cargo workspace: the engine is the `tidaldb` crate at `tidal/`, with `tidal-net/`, `tidal-server/`, and `tidalctl/` as workspace siblings and example consumers under `applications/`. Production-ready — APIs are stable for shipped features; breaking changes ship with a documented migration path. See [CHANGELOG.md](CHANGELOG.md) for milestone history and [docs/planning/ROADMAP.md](docs/planning/ROADMAP.md) for status and known gaps. ## Find Your Guide diff --git a/QUICKSTART.md b/QUICKSTART.md index 00bd6a2..1a8cb51 100644 --- a/QUICKSTART.md +++ b/QUICKSTART.md @@ -307,4 +307,4 @@ This flushes the WAL, checkpoints signal state, and persists indexes. In persist | Axum embedding example | `tidal/examples/axum_embedding.rs` | | 14 content discovery surfaces | [USE_CASES.md](USE_CASES.md) | | Architecture and design decisions | [ARCHITECTURE.md](ARCHITECTURE.md) | -| Cluster mode (experimental) | [docs/runbooks/cluster.md](docs/runbooks/cluster.md) | +| Cluster mode (high availability) | [docs/runbooks/cluster.md](docs/runbooks/cluster.md) | diff --git a/README.md b/README.md index 413be3d..4073588 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,9 @@ **An embeddable Rust database for the personalized content ranking problem.** -> Pre-release. API is stabilizing. Not yet recommended for production. +> Production-ready. M0–M12 shipped: crash-safe storage, ranked retrieval, hybrid +> search, ANN vector retrieval, and a quorum-acked HA cluster running in +> production on k3s. The API surface is stable for shipped features. --- @@ -220,8 +222,13 @@ leader election + failover, elastic seed-join membership, inter-node mTLS, and per-node Prometheus metrics — deployed in production on k3s as one `StatefulSet` (3 pods = 3 regions = 3 voters, full-placement RF3 so every pod hosts all shard groups, HTTPS + mTLS on `:9500`). It exposes `/signals`, -`/feed`, `/search` plus cluster-management routes. It stays behind the -`--experimental-cluster` gate. +`/feed`, `/search` plus cluster-management routes. + +Because a standalone node is the right answer for most deployments, cluster +mode requires an explicit opt-in flag (`--experimental-cluster`, or +`TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1`) so nobody starts a multi-node fabric by +accident. Reach for it deliberately when you need multi-node availability or +read-scale. ```bash cargo run -p tidal-server -- \ @@ -311,6 +318,10 @@ Milestones completed: - Session and agent context (short-lived signals, preference decay) - Crash recovery, graceful degradation, rate limiting, diagnostics - Scale: tested to 1M items; scale benchmarks passing -- High-availability cluster (experimental): quorum-acked writes, automatic election + failover, elastic seed-join membership, inter-node mTLS, per-node Prometheus — running in production on k3s +- High-availability cluster: quorum-acked writes, automatic election + failover, elastic seed-join membership, inter-node mTLS, per-node Prometheus — running in production on k3s -The API surface is stable for the implemented features. Breaking changes are possible before 1.0. +tidalDB is production-ready. The API surface is stable for the implemented +features, and every shipped guarantee is covered by the chaos and soak suites in +[docs/planning/ROADMAP.md](docs/planning/ROADMAP.md). Semantic versioning applies +from here: additive changes ship in minor releases, and any breaking change gets +a documented migration path in [CHANGELOG.md](CHANGELOG.md). diff --git a/docs/guides/server-deployment.md b/docs/guides/server-deployment.md index 3ddccba..a0fa08c 100644 --- a/docs/guides/server-deployment.md +++ b/docs/guides/server-deployment.md @@ -16,7 +16,7 @@ Operational reference for running `tidal-server` — the HTTP wrapper around the | Crash recovery, WAL replay, backup/restore | [docs/ops/recovery.md](../ops/recovery.md) | | Sizing memory/disk/CPU, scrape budgets | [docs/ops/capacity-planning.md](../ops/capacity-planning.md) | | Kubernetes deployment (probes, volumes) | [docs/runbooks/kubernetes.md](../runbooks/kubernetes.md) | -| Cluster operation (experimental) | [docs/runbooks/cluster.md](../runbooks/cluster.md) | +| Cluster operation (high availability) | [docs/runbooks/cluster.md](../runbooks/cluster.md) | --- @@ -36,11 +36,11 @@ tidal-server standalone \ --metrics 127.0.0.1:9091 ``` -### Cluster — EXPERIMENTAL, not production HA +### Cluster — production HA, opt-in -Cluster mode runs multiple "regions" behind one HTTP surface and replicates between them over the **real `tidal-net` gRPC transport**. It has two shapes: a **multi-process** mode (`--region`, one process per region, real gRPC peering between siblings, real process/host isolation — shipped in m8p10) and a single-process dev/demo default (all regions in one process over loopback). Since M11 it is genuinely HA: **quorum-acked writes** (`ack=quorum`, m11p3), **automatic election/failover** (m11p4), and **elastic membership** — DNS peers, seed join, online add/remove/replace with snapshot+stream catch-up (m11p5) — all exist. It remains **`--experimental-cluster`-gated**, so the recommended default is still a **single standalone node** (simpler, scales vertically); reach for cluster mode deliberately when you need multi-node availability or read-scale. See the [cluster runbook](../runbooks/cluster.md) and [`k8s/cluster/`](../../k8s/cluster/) for the reference deployment. +Cluster mode runs multiple "regions" behind one HTTP surface and replicates between them over the **real `tidal-net` gRPC transport**. It has two shapes: a **multi-process** mode (`--region`, one process per region, real gRPC peering between siblings, real process/host isolation — the production shape) and a single-process dev/demo default (all regions in one process over loopback). Since M11 it is genuinely HA: **quorum-acked writes** (`ack=quorum`, m11p3), **automatic election/failover** (m11p4), and **elastic membership** — DNS peers, seed join, online add/remove/replace with snapshot+stream catch-up (m11p5). M12 added sharded ingestion and TLS scale-up. The reference deployment runs in production on k3s. The recommended default is still a **single standalone node** (simpler, scales vertically); reach for cluster mode deliberately when you need multi-node availability or read-scale. See the [cluster runbook](../runbooks/cluster.md) and [`k8s/cluster/`](../../k8s/cluster/) for the reference deployment. -The mode refuses to start unless you opt in explicitly (`--experimental-cluster` or `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1`) and emits a loud `WARN` describing exactly what it is and is not. Cluster mode currently replicates **global signals only** — `user_id`/`creator_id` signal context is dropped on the cluster path so followers stay consistent with the leader's WAL. +Because standalone is the right answer for most deployments, cluster mode refuses to start unless you opt in explicitly (`--experimental-cluster` or `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1`) — a guard against starting a multi-node fabric by accident — and emits a startup `WARN` describing exactly which shape it is running. Cluster mode replicates **global signals only** — `user_id`/`creator_id` signal context is dropped on the cluster path so followers stay consistent with the leader's WAL. For everything cluster-specific (topology file, leader promotion, partition/heal, scatter-gather routes), see the **[cluster runbook](../runbooks/cluster.md)**. The rest of this guide focuses on standalone. diff --git a/docs/runbooks/cluster.md b/docs/runbooks/cluster.md index 99a6b76..0396dac 100644 --- a/docs/runbooks/cluster.md +++ b/docs/runbooks/cluster.md @@ -20,10 +20,12 @@ write-durability contract. > > **Honest caveats that still hold:** cluster mode replicates **global** retrieval > signals only (no per-user personalization on the `/signals` route — see the -> personalization note in [§3](#3-topology-yaml)); and both launch modes still -> **refuse to start** unless you explicitly opt in with `--experimental-cluster` or -> `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1` (the k8s manifests set the env var). The two -> launch shapes below remain: +> personalization note in [§3](#3-topology-yaml)); and because a standalone node is +> the right answer for most deployments, both launch modes **refuse to start** +> unless you explicitly opt in with `--experimental-cluster` or +> `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1` (the k8s manifests set the env var) — a guard +> against standing up a multi-node fabric by accident, not a readiness warning. The +> two launch shapes below remain: > > **1. Multi-process (`--region`) — real process isolation (the production shape).** Each > `tidal-server cluster --region ` process owns **exactly one region**: one diff --git a/k8s/statefulset.yaml b/k8s/statefulset.yaml index ba35de5..39ce94c 100644 --- a/k8s/statefulset.yaml +++ b/k8s/statefulset.yaml @@ -4,9 +4,10 @@ # embeddable — the server wraps ONE engine instance whose state (WAL + # checkpoints + indexes) lives on a durable data dir. It scales VERTICALLY # (bigger node), not by adding replicas. `replicas: 1` is intentional and load- -# bearing: there is no shared-storage multi-writer mode. The experimental -# multi-region `cluster` subcommand runs every region in ONE process (no host -# isolation — NOT production HA, tracked as m8p10), so it does not change this. +# bearing: there is no shared-storage multi-writer mode. If you need multi-node +# HA, that is a DIFFERENT deployment: the multi-process `cluster` subcommand +# (one process per region, quorum-acked writes, automatic failover) ships as its +# own StatefulSet under k8s/cluster/ — it does not change this manifest. # See docs/runbooks/kubernetes.md and docs/runbooks/cluster.md. apiVersion: apps/v1 kind: StatefulSet diff --git a/tidal-server/src/cluster/mod.rs b/tidal-server/src/cluster/mod.rs index 6df7dfd..aa4b1f9 100644 --- a/tidal-server/src/cluster/mod.rs +++ b/tidal-server/src/cluster/mod.rs @@ -27,7 +27,7 @@ //! //! * [`topology`] — `TopologySpec` / `RegionSpec` / `load_topology` / validation //! * [`transport`] — single-process self-loop gRPC transport wiring -//! * [`state`] — [`ClusterState`] (single-process) + the experimental gate +//! * [`state`] — [`ClusterState`] (single-process) + the cluster opt-in gate //! * [`routes`] — the single-process router + handlers //! * [`node`] — [`ClusterNode`] (the process) hosting [`ShardReplica`] groups //! (multi-process) + its gateway router/handlers diff --git a/tidal-server/src/cluster/state.rs b/tidal-server/src/cluster/state.rs index d459ba6..e0d1246 100644 --- a/tidal-server/src/cluster/state.rs +++ b/tidal-server/src/cluster/state.rs @@ -1,5 +1,5 @@ //! `ClusterState`: the single-process multi-region fabric behind the HTTP -//! surface, plus the experimental-mode opt-in gate shared by both cluster modes. +//! surface, plus the explicit opt-in gate shared by both cluster modes. use std::{ collections::HashMap, @@ -26,14 +26,16 @@ use crate::{ /// used by the distributed fabric. All cluster operations go through this /// layer. /// -/// # Experimental +/// # Single-process shape /// /// Replication between regions runs over the **real `tidal-net` gRPC /// transport** (see the module-level docs), but every region still lives inside /// this single process: there is no process or host isolation, so a crash takes /// the whole "cluster" down. It provides faithful multi-region replication -/// semantics, not production high-availability. Cluster mode is therefore gated -/// behind an explicit operator opt-in — see [`ensure_experimental_enabled`]. +/// semantics for dev and demo, not high-availability — production HA is the +/// multi-process `--region` shape. Both shapes are gated behind an explicit +/// operator opt-in so a multi-node fabric is never started by accident — see +/// [`ensure_experimental_enabled`]. pub struct ClusterState { /// `Some` for the lifetime of the server; consumed by [`shutdown`] / /// [`Drop`] so every node's `TidalDb` is dropped (checkpoint + WAL fsync + @@ -58,27 +60,28 @@ pub struct ClusterState { write_pool: ClusterWritePool, } -/// Environment variable that opts in to the experimental cluster mode. +/// Environment variable that opts in to cluster mode. pub const EXPERIMENTAL_CLUSTER_ENV: &str = "TIDAL_ALLOW_EXPERIMENTAL_CLUSTER"; -/// Which cluster mode the experimental gate is being checked for, so the loud +/// Which cluster mode the opt-in gate is being checked for, so the startup /// WARN can describe exactly what each mode does and does NOT provide. #[derive(Debug, Clone, Copy)] pub enum ClusterMode { - /// Every region runs inside ONE process (m8p8). No host/process isolation. + /// Every region runs inside ONE process (m8p8). No host/process isolation — + /// the dev/demo shape. SingleProcess, /// Exactly ONE region runs in this process and peers with sibling processes - /// over real gRPC (m8p10). Real process isolation, but still no quorum-ack - /// writes and no automatic failure detection. + /// over real gRPC (m8p10). Real process isolation, quorum-acked writes + /// (m11p3), and automatic election/failover (m11p4) — the production shape. MultiProcess, } /// Gate cluster mode behind an explicit operator opt-in. /// -/// Both cluster modes are experimental and must not be started by accident: -/// single-process has no host/process isolation, and multi-process — while it -/// gives real process isolation — still lacks quorum-ack writes and automatic -/// failure detection. Starting is permitted only when either: +/// A standalone node is the right answer for most deployments, so neither +/// cluster shape may be started by accident. This is a guard against standing up +/// a multi-node fabric unintentionally, not a readiness warning: the +/// multi-process shape is production HA. Starting is permitted only when either: /// /// * the `--experimental-cluster` CLI flag is passed (`flag_set == true`), or /// * the [`EXPERIMENTAL_CLUSTER_ENV`] env var is set to a truthy value @@ -99,12 +102,11 @@ pub fn ensure_experimental_enabled(flag_set: bool, mode: ClusterMode) -> Result< let reason = match mode { ClusterMode::SingleProcess => { "cluster mode replicates over real gRPC but runs all regions in ONE process \ - (no host/process isolation) — it is not production HA." + (no host/process isolation) — it is the dev/demo shape, not production HA." } ClusterMode::MultiProcess => { - "multi-process cluster mode (--region) gives real process isolation but does \ - NOT yet provide quorum-ack writes or automatic failure detection — it is not \ - production HA." + "multi-process cluster mode (--region) stands up a real multi-node fabric; \ + a standalone node is the right default for most deployments." } }; return Err(ServerError::ExperimentalDisabled(format!( @@ -115,17 +117,17 @@ pub fn ensure_experimental_enabled(flag_set: bool, mode: ClusterMode) -> Result< match mode { ClusterMode::SingleProcess => tracing::warn!( - "EXPERIMENTAL cluster mode enabled. Regions replicate over the real tidal-net gRPC \ - transport (loopback), but all regions run in a SINGLE process — there is no host or \ - process isolation, so this provides NO production high-availability. True \ - multi-process deployment is the --region mode. Do NOT use this for production traffic." + "Single-process cluster mode enabled (dev/demo shape). Regions replicate over the \ + real tidal-net gRPC transport (loopback), but all regions run in a SINGLE process — \ + there is no host or process isolation, so this provides NO high-availability. Use \ + the --region multi-process mode for production traffic." ), ClusterMode::MultiProcess => tracing::warn!( - "EXPERIMENTAL multi-process cluster mode enabled (--region). This process owns ONE \ - region and peers with siblings over real gRPC, so process isolation IS real — but \ - quorum-ack writes and automatic failure detection are NOT yet provided (a 204 means \ - leader durability only, and failover is operator-driven). Do NOT use this for \ - production traffic." + "Multi-process cluster mode enabled (--region) — the production HA shape. This \ + process owns ONE region and peers with siblings over real gRPC: process isolation \ + is real, writes are quorum-acked when the topology sets replication.ack=quorum, and \ + leader election/failover is automatic. Cluster mode replicates GLOBAL signals only \ + (user_id/creator_id context is dropped on the cluster path)." ), } Ok(()) diff --git a/tidal-server/src/error.rs b/tidal-server/src/error.rs index 7eba4ab..34b088c 100644 --- a/tidal-server/src/error.rs +++ b/tidal-server/src/error.rs @@ -30,7 +30,7 @@ pub enum ServerError { Network(#[from] std::io::Error), #[error("bad request: {0}")] BadRequest(String), - #[error("cluster mode is experimental and disabled: {0}")] + #[error("cluster mode requires an explicit opt-in: {0}")] ExperimentalDisabled(String), #[error("cluster error: {0}")] Cluster(String), diff --git a/tidal-server/src/main.rs b/tidal-server/src/main.rs index 02c6adb..86722d2 100644 --- a/tidal-server/src/main.rs +++ b/tidal-server/src/main.rs @@ -41,14 +41,18 @@ struct ClusterArgs { /// `TIDAL_CONFIG` env var. #[arg(long, env = "TIDAL_CONFIG")] config_dir: Option, - /// Opt in to the EXPERIMENTAL cluster mode (single- or multi-process). + /// Opt in to cluster mode (single- or multi-process). + /// + /// A standalone node is the right default for most deployments, so cluster + /// mode refuses to start without this flag (or the + /// `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1` env var) — a guard against standing + /// up a multi-node fabric by accident. /// /// Single-process cluster mode replicates between regions over the real /// tidal-net gRPC transport (loopback), but every region runs inside THIS - /// one process — no host/process isolation. Multi-process mode (`--region`) - /// gives real process isolation but still lacks quorum-ack writes and - /// automatic failure detection. Either way it refuses to start without this - /// flag (or the `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1` env var). + /// one process — no host/process isolation; it is the dev/demo shape. + /// Multi-process mode (`--region`) is the production HA shape: real process + /// isolation, quorum-acked writes, and automatic election/failover. #[arg(long)] experimental_cluster: bool, /// Run ONLY this region in this process (multi-process cluster mode). @@ -233,9 +237,9 @@ async fn run_cluster(args: ClusterArgs) -> Result<()> { async fn run_single_process_cluster(args: ClusterArgs) -> Result<()> { // Honest gate: single-process cluster mode replicates between regions over // the real tidal-net gRPC transport, but every region runs inside this one - // process (no host/process isolation), so it is not production HA. Refuse to - // start (and emit a loud WARN when permitted) so no operator mistakes it for - // production HA. + // process (no host/process isolation), so it is the dev/demo shape, not HA. + // Refuse to start (and emit a startup WARN when permitted) so no operator + // mistakes it for the multi-process production shape. ensure_experimental_enabled(args.experimental_cluster, ClusterMode::SingleProcess)?; let schema_path = resolve_config_path( @@ -283,8 +287,9 @@ async fn run_single_process_cluster(args: ClusterArgs) -> Result<()> { } async fn run_region_cluster(args: ClusterArgs, region: String) -> Result<()> { - // Multi-process mode is experimental too: real process isolation, but no - // quorum-ack writes or automatic failure detection yet. The gate's WARN text + // Multi-process mode is the production HA shape (real process isolation, + // quorum-acked writes, automatic election/failover) but still opt-in, so a + // multi-node fabric is never stood up by accident. The gate's WARN text // differentiates the two modes honestly. ensure_experimental_enabled(args.experimental_cluster, ClusterMode::MultiProcess)?; @@ -579,7 +584,7 @@ impl ServeState for ClusterState { // `SimulatedCluster` (out of this crate's edit scope), whose nodes can // only be closed by dropping the `Arc` — which needs // `&mut self` here. This branch is not reached in practice (this path is - // experimental and not the deployed cluster; in steady state + // the dev/demo shape, not the deployed cluster; in steady state // `try_unwrap` succeeds and `shutdown_owned` runs). Flip readiness and // rely on the last-`Arc` `Drop` (which runs the same `&mut self` close). Self::set_shutting_down(self); diff --git a/tidal-server/src/openapi.rs b/tidal-server/src/openapi.rs index 425669a..6969b46 100644 --- a/tidal-server/src/openapi.rs +++ b/tidal-server/src/openapi.rs @@ -129,8 +129,10 @@ pub struct StandaloneApiDoc; version = env!("CARGO_PKG_VERSION"), description = "Cluster-mode tidalDB HTTP surface: the standalone data + health \ routes plus multi-region cluster management and sharded \ - scatter-gather routes. EXPERIMENTAL — all regions run in one \ - process (no host/process isolation).", + scatter-gather routes. This is the single-process dev/demo \ + shape — all regions run in one process (no host/process \ + isolation). For production HA run one process per region \ + with --region.", ), paths( crate::cluster::routes::cluster_status, @@ -193,8 +195,8 @@ pub struct ClusterApiDoc; peers with siblings over real gRPC, presenting the cluster as one \ coherent HTTP surface (leader forwarding, broadcast, status \ aggregation, cross-process reconcile, and sharded scatter-gather). \ - EXPERIMENTAL — real process isolation, but no quorum-ack writes or \ - automatic failure detection yet.", + This is the production HA shape: real process isolation, \ + quorum-acked writes, and automatic election/failover.", ), paths( crate::cluster::node::status_local,