tidaldb/docs/guides/server-deployment.md

Server Deployment

Operational reference for running tidal-server — the HTTP wrapper around the embeddable tidaldb engine. This is the doc to read before exposing tidalDB over the network.

tidal-server is a thin Axum surface over the engine. It does not change the engine's correctness model: every write still goes through the WAL, every shutdown still checkpoints and fsyncs. What the server adds is an HTTP contract, Bearer auth, request middleware (timeouts, concurrency cap, body limit, request IDs), and a YAML-driven schema/profile loader so you can deploy without writing Rust.

tidalDB owns retrieval and ranking only. It does not generate embeddings, store video/blobs, run a CDN/transcoder, or provide auth/moderation/payments. The server brings vectors in over POST /embeddings; your application produces them.

Cross-references

Topic	Doc
HTTP + Rust API contract (handwritten)	API.md
Get-started walkthrough	QUICKSTART.md
Metrics, alert thresholds, Grafana	docs/ops/monitoring.md
Crash recovery, WAL replay, backup/restore	docs/ops/recovery.md
Sizing memory/disk/CPU, scrape budgets	docs/ops/capacity-planning.md
Kubernetes deployment (probes, volumes)	docs/runbooks/kubernetes.md
Cluster operation (high availability)	docs/runbooks/cluster.md

---

1. Run Modes

tidal-server has two subcommands. Pick standalone unless you have a specific reason not to.

Standalone — the recommended path

tidalDB is single-node-first. One process wraps one TidalDb instance, serves the full data + health surface, and scales vertically. This is the production-ready mode. It is the default in every shipped Docker image and the only mode covered by the durability and recovery guarantees in docs/ops/recovery.md.

tidal-server standalone \
  --listen 0.0.0.0:9400 \
  --schema ./schema.yaml \
  --data-dir /var/lib/tidaldb \
  --metrics 127.0.0.1:9091

Cluster — production HA, opt-in

Cluster mode runs multiple "regions" behind one HTTP surface and replicates between them over the real tidal-net gRPC transport. It has two shapes: a multi-process mode (--region, one process per region, real gRPC peering between siblings, real process/host isolation — the production shape) and a single-process dev/demo default (all regions in one process over loopback). Since M11 it is genuinely HA: quorum-acked writes (ack=quorum, m11p3), automatic election/failover (m11p4), and elastic membership — DNS peers, seed join, online add/remove/replace with snapshot+stream catch-up (m11p5). M12 added sharded ingestion and TLS scale-up. The reference deployment runs in production on k3s. The recommended default is still a single standalone node (simpler, scales vertically); reach for cluster mode deliberately when you need multi-node availability or read-scale. See the cluster runbook and k8s/cluster/ for the reference deployment.

Because standalone is the right answer for most deployments, cluster mode refuses to start unless you opt in explicitly (--experimental-cluster or TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1) — a guard against starting a multi-node fabric by accident — and emits a startup WARN describing exactly which shape it is running. Cluster mode replicates global signals only — user_id/creator_id signal context is dropped on the cluster path so followers stay consistent with the leader's WAL.

For everything cluster-specific (topology file, leader promotion, partition/heal, scatter-gather routes), see the cluster runbook. The rest of this guide focuses on standalone.

---

2. CLI Flags and Environment Variables

Standalone flags

Flag	Env override	Default	Meaning
--listen <addr>	PORT	127.0.0.1:9400	HTTP bind address. A bare PORT value (e.g. PORT=8080) is normalized to 0.0.0.0:8080. A host:port string is used verbatim.
--schema <path>	—	compiled-in default	Path to the schema/profile YAML (see section 3).
--config-dir <dir>	TIDAL_CONFIG	unset	Directory holding default-schema.yaml. Used only when --schema is omitted; the explicit flag always wins. If the dir is set but the file is missing, startup fails loudly rather than silently serving compiled-in defaults.
--data-dir <dir>	—	unset → EPHEMERAL	Persistent data directory. If omitted, the server runs in-memory and loses every write on restart. Always set this in production.
--metrics <addr>	—	unset (disabled)	Bind address for the Prometheus endpoint. Bind to loopback only — it is unauthenticated (see section 7).

Cluster flags

Flag	Env override	Default	Meaning
--listen <addr>	PORT	127.0.0.1:9500	HTTP bind address (same bare-port normalization).
--schema <path>	—	compiled-in default	Schema/profile YAML.
--topology <path>	—	compiled-in default	Cluster topology YAML (regions, leader). See the cluster runbook.
--config-dir <dir>	TIDAL_CONFIG	unset	Directory holding default-schema.yaml and default-cluster.yaml.
--region <name>	TIDAL_REGION	unset	Run only this region in this process (multi-process mode). Omit for the single-process dev/demo default.
--data-dir <dir>	—	required (multi-process)	Persistent data directory. The durable WAL is the replicated log itself, so multi-process mode rejects a node without one.
--seed <url> (repeatable)	—	unset	Seed-join boot (m11p5): contact a running peer's HTTP base URL to learn the roster + assigned id + term and join online as a learner — no topology entry of this node's own required. Still needs --topology/TIDAL_CONFIG for the behavioral knob blocks.
--advertise-grpc <host:port>	—	unset	This node's advertised gRPC address (what siblings dial; DNS-capable). Required with --seed.
--advertise-http <host:port>	—	unset	This node's advertised HTTP gateway (what peers forward to). Required with --seed.
--metrics <addr>	—	unset	Prometheus /metrics bind for a seed-joined node (which has no topology metrics_addr). Bind to loopback / cluster-internal only — unauthenticated.
--experimental-cluster	TIDAL_ALLOW_EXPERIMENTAL_CLUSTER	off	Required opt-in. Without it (flag or env 1/true/yes), cluster mode refuses to start.

Environment variables (all modes)

Variable	Default	Meaning
TIDAL_API_KEY	unset	Bearer token for data routes. If unset, the server is UNAUTHENTICATED and logs a WARN at startup. See section 4.
TIDAL_API_KEY_FILE	unset	Path whose CONTENT is the bearer token (m11p7); takes precedence over TIDAL_API_KEY. The FILE form rotates without a restart (a credential poller re-reads it).
TIDAL_ADMIN_KEY	unset	OPERATOR credential gating the destructive /cluster/* verbs. If unset, those verbs accept the data bearer and the server logs a WARN. See section 4.
TIDAL_ADMIN_KEY_FILE	unset	Path whose CONTENT is the operator key; takes precedence over TIDAL_ADMIN_KEY and rotates without a restart.
TIDAL_CONFIG	unset	Config directory (backs --config-dir).
PORT	unset	Listen port/address (backs --listen); bare port → 0.0.0.0:PORT.
TIDAL_SERVER_LOG	info	tracing env-filter directive, e.g. TIDAL_SERVER_LOG=tidal_server=debug,info.
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER	unset	Truthy (1/true/yes) opts in to cluster mode.
TIDAL_CLUSTER_KEY / TIDAL_CLUSTER_KEY_FILE	unset	Cluster mode, m11p7. Shared secret (any random string) that mints/verifies per-node internal tokens authenticating inter-node HTTP. Absent ⇒ per-node tokens disabled (the x-tidal-internal marker keeps hint-only behavior; WARN). The _FILE form rotates without restart. Never give it to external clients. See the cluster runbook §12.
TIDAL_AUDIT_LOG	unset	Cluster mode, m11p7. Path to an append-only JSONL admin-verb audit log (promote/heal/partition/conf-change). Always also emitted to the tidal_audit tracing target. At-rest encryption is delegated to the volume.
TIDAL_RATE_LIMIT_RPS / TIDAL_RATE_LIMIT_BURST	unset (unlimited)	m11p7. Per-principal HTTP rate limit (sustained / burst, default burst 2×). A deny is 429 + Retry-After. Verified sibling nodes are exempt.
TIDAL_ROTATION_POLL_MS	30000	Cluster mode, m11p7. How often the cert + credential rotation poller re-reads the TLS/secret files.

Inter-node TLS (m11p7) is configured by the topology's per-region grpc_tls block (paths to ca_cert / server_cert / server_key / client_cert / client_key), NOT an env var. Present ⇒ gRPC mutual TLS + inter-node HTTP TLS (one cert, one hot rotation, both planes); absent ⇒ plaintext inter-node links with a loud startup WARN. See the cluster runbook §12 and k8s/cluster/ (cert-manager) / scripts/gen-cluster-certs.sh.

---

3. Schema YAML

The server loads its schema and ranking profiles from YAML at startup. This is the YAML projection of the Rust SchemaBuilder API. Four top-level keys: signals (required), text_fields, embedding_slots, profiles (all optional). At least one signal must be defined or startup fails.

Top-level shape

signals:        # required, >= 1
text_fields:    # optional
embedding_slots: # optional
profiles:       # optional — built-in profiles are available regardless

signals

signals:
  - name: view
    entity: item            # item | user | creator
    decay:
      exponential:
        half_life_seconds: 604800   # exclusive: exponential | linear | permanent
    windows: [one_hour, twenty_four_hours, seven_days]
    velocity: true
    positive_engagement: true   # optional; folds into the user preference vector

• decay is exactly one of:
  • exponential: { half_life_seconds: <f64> }
  • linear: { lifetime_seconds: <f64> }
  • permanent: true
• windows accepts: one_hour (1h), twenty_four_hours (24h), seven_days (7d), thirty_days (30d), all_time (alltime).
• velocity: true enables velocity tracking for the signal.
• positive_engagement: true makes engaging with an item via this signal fold the item's embedding into the user's preference vector (personalization). Omit it and the engine falls back to name-based detection — declare it explicitly on signals like view/like/completion for predictable taste.

text_fields

text_fields:
  - name: title
    kind: text       # text (analyzed, BM25) | keyword (exact)
  - name: category
    kind: keyword

embedding_slots

embedding_slots:
  - name: content_vector
    entity: item
    dimensions: 128

Embedding rules to respect from the engine side:

• The app brings vectors; the server's POST /embeddings L2-normalizes them and inserts into the HNSW index. tidalDB does not generate embeddings.
• Dimensions are strict: min 2, max 4096, and the submitted vector length must equal the slot's declared dimensions or the insert fails. Zero-norm vectors are rejected.
• Multi-slot caveat: RETRIEVE/SEARCH route through the first declared slot only. Multi-modal apps must fuse offline or use separate entity kinds.

profiles

Optional. The 25 built-in profiles (for_you, trending, hot, new, following, related, top_week, most_viewed, controversial, hidden_gems, shuffle, cohort_trending, etc.) are available without declaring anything. Use profiles to add custom ones or override behavior.

profiles:
  - name: for_you
    version: 1
    candidate_strategy:           # default: scan(created_at)
      ann:
        slot: content_vector
        limit: 200
    sort:                         # string OR parameterized map
      hot:
        gravity: 1.5
    boosts:
      - { signal: like, agg: decay_score, window: all_time, weight: 2.5 }
      - { signal: view, agg: velocity, window: one_hour, weight: 1.0 }
    gates:
      - { signal: skip, agg: value, window: all_time, min_threshold: 0.0 }
    penalties:
      - { signal: skip, agg: decay_score, window: seven_days, weight: 1.5 }
    excludes:
      - { signal: skip, agg: value, window: all_time, above: 1.0 }
    diversity:
      max_per_creator: 2
      format_mix_max_fraction: 0.4
    exploration: 0.1

Field reference:

• candidate_strategy — scan | relationship | hybrid | cohort_trending, or a map: ann: { slot, limit }, signal_ranked: { signal, window }, scan: { sort_field }. Default: scan over created_at.
• sort — string: trending, rising, controversial, hidden_gems, shuffle, new, most_followed, creator_engagement_rate, alphabetical_asc, alphabetical_desc, shortest, longest, live_viewer_count, date_saved; or map: hot: { gravity }, top_window: { window }, most_viewed: { window }, most_liked: { window }, most_commented: { window }, most_shared: { window }.
• boosts / penalties — { signal, agg, window, weight }. agg ∈ value | velocity | decay_score | ratio | relative_velocity.
• gates — { signal, agg, window, min_threshold } (drop candidates below the threshold).
• excludes — { signal, agg, window, above } (drop candidates above the threshold; e.g. hard-negative skip/hide/block).
• diversity — { max_per_creator, format_mix_max_fraction }.
• exploration — f64 in [0,1].

Every signal named in a profile's boosts/gates/penalties/excludes/decay must be declared in signals, or startup fails with a clear error naming the offending profile and signal.

Correctness rule — preference-vector personalization. The user "taste vector" that drives for_you is folded from item embeddings only for signals declared positive_engagement: true (the YAML field above; equivalently SignalBuilder::positive_engagement(true) in embedded mode). Declare it on your engaging signals (view/like/completion); if you omit it on every signal, the engine falls back to a name-based allowlist. If your personalization looks flat, confirm the engaging signals carry the flag. See API.md for the underlying signal_with_context semantics.

Worked example (the shipped default)

signals:
  - name: view
    entity: item
    decay:
      exponential:
        half_life_seconds: 604800    # 7 days
    windows: [one_hour, twenty_four_hours, seven_days]
    velocity: true
  - name: like
    entity: item
    decay:
      exponential:
        half_life_seconds: 1209600   # 14 days
    windows: [twenty_four_hours, seven_days, thirty_days, all_time]
    velocity: false
  - name: skip
    entity: item
    decay:
      permanent: true
    velocity: false
text_fields:
  - name: title
    kind: text
  - name: category
    kind: keyword
embedding_slots:
  - name: content_vector
    entity: item
    dimensions: 128

This is the compiled-in default at tidal-server/config/default-schema.yaml, used when neither --schema nor a TIDAL_CONFIG directory provides one.

---

4. Authentication

Data routes (/items, /embeddings, /signals, /feed, /search) require an Authorization: Bearer <token> header when TIDAL_API_KEY is set. The comparison is constant-time. A missing or wrong token returns 401 with WWW-Authenticate: Bearer.

Health probes (/health, /health/startup, /health/live) and the OpenAPI document (/openapi.json) are always unauthenticated — a client needs the spec to learn how to authenticate, and probes must work for orchestrators that cannot present a token.

WARNING — no key means an open server. If TIDAL_API_KEY is unset (or empty), every data route is served without authentication and the server logs a startup WARN. This is fine for local development. Never expose an unauthenticated server to any network. Set TIDAL_API_KEY before binding to anything other than 127.0.0.1.

export TIDAL_API_KEY="$(openssl rand -hex 32)"
curl -H "Authorization: Bearer $TIDAL_API_KEY" \
     "http://localhost:9400/feed?user_id=42&profile=for_you&limit=20"

Two credential tiers (cluster mode)

A cluster has two credentials, because one is not enough: every /cluster/* mutation used to sit behind the same bearer as /items and /search, so any application key could remove a member, force a partition, or transfer a shard.

Credential	Env	Grants
Data bearer	TIDAL_API_KEY / TIDAL_API_KEY_FILE	The data routes, /cluster/status, and the peer-callable verbs (/cluster/catchup, /cluster/join, /cluster/members, /cluster/reconcile*).
Operator key	TIDAL_ADMIN_KEY / TIDAL_ADMIN_KEY_FILE	Everything above plus the destructive verbs: /cluster/promote, /cluster/partition, /cluster/heal, /cluster/members/remove, /cluster/reseed, /cluster/shards/{id}/replicas, /cluster/shards/{id}/transfer.

Presenting the data bearer to a destructive verb returns 403 (authenticated, not authorized) — distinct from the 401 a missing/invalid token returns.

The operator key is a superset credential: it also satisfies the authentication gate, because a request carries a single Authorization header — if it did not authenticate, an operator presenting it would be rejected 401 before the authorization gate ran. Keep it off application hosts.

A verified sibling node token also clears the gate. That is load-bearing, not a convenience: nodes relay operator verbs to the leader/target carrying whatever credential the caller sent, so enabling an operator key can never partition the control plane.

WARNING — one key means every client is an operator. If TIDAL_ADMIN_KEY is unset, the destructive verbs accept the data bearer and the server logs a startup WARN naming the exposure. Behavior is unchanged from before the split, so this is safe to upgrade into, but set the key before handing a bearer to anything you do not operate.

export TIDAL_API_KEY="$(openssl rand -hex 32)"
export TIDAL_ADMIN_KEY="$(openssl rand -hex 32)"

# 403: the data key is not an operator credential
curl -X POST -H "Authorization: Bearer $TIDAL_API_KEY" \
     -d '{"region":"eu-west"}' http://localhost:9500/cluster/promote

# accepted
curl -X POST -H "Authorization: Bearer $TIDAL_ADMIN_KEY" \
     -d '{"region":"eu-west"}' http://localhost:9500/cluster/promote

/cluster/status and /cluster/status/local require a credential too. They report leader identity, membership, term, and per-shard applied/lag/commit seqnos — reconnaissance, not a probe — so they are not in the open group with /health.

---

5. The Served OpenAPI Reference (GET /openapi.json)

tidal-server serves a machine-readable OpenAPI 3.1 document at GET /openapi.json, unauthenticated. This is the canonical HTTP API contract — it is derived directly from the handler signatures and DTOs, so it can never drift from the running code (unlike a handwritten spec). API.md remains the prose companion, but /openapi.json is the source of truth for request/response shapes.

Inspect it:

curl -s localhost:9400/openapi.json | jq .info
# {
#   "title": "tidalDB HTTP API",
#   "version": "<crate version>",
#   "description": "Embeddable, single-node-first database for the personalized content ranking problem..."
# }

You can load /openapi.json into any OpenAPI viewer (Swagger UI, Redoc, Stoplight, Postman) or feed it to a client generator (openapi-generator, oapi-codegen, etc.) to produce typed clients in any language. The standalone document describes the data + health surface; the cluster document is a superset that also describes the /cluster/* and /sharded/* routes. The bearerAuth security scheme is declared on the data routes so generated clients know to send the token; /openapi.json and the probes carry no security requirement.

Route summary (standalone)

Method + path	Auth	Success	Notes
GET /health	no	200 ready / 503 draining	Readiness probe; reports item count.
GET /health/startup	no	200 always	Startup probe.
GET /health/live	no	200 always	Liveness probe.
GET /openapi.json	no	200	OpenAPI 3.1 document.
POST /items	yes	201	Body: { "entity_id": <u64>, "metadata": { ... } }.
POST /embeddings	yes	204	Body: { "entity_id": <u64>, "values": [<f32>, ...] }.
POST /signals	yes	204	Body: { "entity_id", "signal", "weight", "user_id"?, "creator_id"? }.
GET /feed	yes	200	?user_id=&profile=for_you&limit=20.
GET /search	yes	200	?query=&user_id=&limit=20.
POST /vector_search	yes	200	Pure k-NN. Body: { "vector": [<f32>, ...], "k"?, "ef_search"? }.

limit is clamped to 1000 at the trust boundary. Request middleware: 30 s timeout (408), 100 max in-flight (429), 2 MB body cap (413), and an x-request-id echoed on every response.

---

6. Running the Server

Via cargo (development)

# Ephemeral, loopback, default schema — quickest smoke test:
cargo run -p tidal-server -- standalone

# Durable, real schema, metrics on loopback:
cargo run -p tidal-server -- standalone \
  --listen 0.0.0.0:9400 \
  --schema tidal-server/config/default-schema.yaml \
  --data-dir /var/lib/tidaldb \
  --metrics 127.0.0.1:9091

For an end-to-end engine walkthrough (embedded, no server), run the verified example: cargo run -p tidaldb --example foryou_feed. See QUICKSTART.md.

Via Docker

Three images ship, all built from the repository root as build context:

Image	Dockerfile	Toolchain	Use
standalone	docker/standalone/Dockerfile	rust:1.91-bookworm	General single-node; bundles curl for healthcheck.
deploy	docker/deploy/Dockerfile	rust:1.91-slim-bookworm, uid 10001	Slim production single-node; bakes the schema, WORKDIR /data.
cluster	docker/cluster/Dockerfile	—	Experimental cluster mode; see the cluster runbook.

Both single-node images:

• Run as a non-root tidal user.
• Declare VOLUME ["/data"] and pass --data-dir /data in the default CMD. Durability lives on /data — you must mount a real volume there or every write is lost on restart (the container layer is ephemeral).
• Use ENTRYPOINT ["tidal-server"] + a CMD of subcommand args, so docker run <image> <subcommand> ... overrides work (e.g. running an inspect command).
• Set ENV TIDAL_CONFIG to the baked config dir (/etc/tidal-server for standalone, /config for deploy) so the container is self-contained.
• Expose 9400 (HTTP) and 9091 (metrics).

# Build (from repo root):
docker build -f docker/standalone/Dockerfile -t tidaldb:standalone .

# Run with a named volume for durability and an API key:
docker run -d --name tidaldb \
  -p 9400:9400 \
  -p 127.0.0.1:9091:9091 \
  -e TIDAL_API_KEY="$(openssl rand -hex 32)" \
  -v tidaldb-data:/data \
  tidaldb:standalone

Note the metrics port is published to 127.0.0.1 only — it is unauthenticated (see section 7).

Via docker-compose

docker/docker-compose.yml brings up the standalone image plus a Prometheus sidecar, with a named tidaldb-data volume mounted at /data:

cd docker
export TIDAL_API_KEY="$(openssl rand -hex 32)"
docker compose up -d

The compose healthcheck calls GET /health with the Bearer header. Adjust the tidaldb-data volume to a host bind mount if you want the data outside Docker's volume store. For Kubernetes equivalents (probes wired to /health/startup, /health/live, /health; a PersistentVolumeClaim at /data), see the Kubernetes runbook.

---

7. Metrics and Shutdown

Metrics

--metrics <addr> enables the engine's Prometheus endpoint on a separate port (default image: 9091). It serves /metrics (Prometheus text format) and /healthz (JSON). This endpoint is UNAUTHENTICATED — bind it to loopback or a cluster-internal interface only. The engine logs a WARN if you bind it to a non-loopback address. For the full metric catalog, scrape config, alert thresholds, and Grafana dashboard, see docs/ops/monitoring.md. For sizing the scrape budget and memory/disk/CPU, see docs/ops/capacity-planning.md.

Graceful shutdown

On SIGTERM (or Ctrl-C), the server:

1. Flips readiness to 503 — GET /health reports { "ok": false, "cause": "shutting down" } so a load balancer stops routing new traffic. Startup/liveness probes still return 200.
2. Drains in-flight requests (axum graceful shutdown).
3. Closes the database — checkpoints state and fsyncs the WAL before the process exits. The standalone path reclaims sole ownership and runs this deterministically; if a flush fails it is logged at error level. There is a Drop backstop that runs the same shutdown if ownership is unexpectedly still shared.

Give orchestrators a terminationGracePeriod long enough to cover drain + checkpoint (a few seconds is typically ample for single-node; size against your WAL volume per capacity planning). For crash recovery, WAL replay, and backup/restore, see docs/ops/recovery.md.

Reverse proxy and TLS termination

tidal-server speaks plain HTTP and does not terminate TLS. In production, front it with a reverse proxy (nginx, Caddy, Envoy, a cloud load balancer, or a Kubernetes Ingress) that:

• Terminates TLS and forwards to the server's --listen address.
• Performs health checks against GET /health (it returns 503 during drain, so the proxy stops sending traffic before the process exits).
• Optionally adds its own rate limiting in front of the server's 100-in-flight / 30 s-timeout middleware.
• Preserves or sets x-request-id if you want end-to-end tracing — the server generates one when absent and echoes it on the response.

Keep the metrics port off the public proxy entirely; scrape it over the internal network only.