tidaldb/docs/guides/server-deployment.md

Server Deployment

Operational reference for running tidal-server — the HTTP wrapper around the embeddable tidaldb engine. This is the doc to read before exposing tidalDB over the network.

tidal-server is a thin Axum surface over the engine. It does not change the engine's correctness model: every write still goes through the WAL, every shutdown still checkpoints and fsyncs. What the server adds is an HTTP contract, Bearer auth, request middleware (timeouts, concurrency cap, body limit, request IDs), and a YAML-driven schema/profile loader so you can deploy without writing Rust.

tidalDB owns retrieval and ranking only. It does not generate embeddings, store video/blobs, run a CDN/transcoder, or provide auth/moderation/payments. The server brings vectors in over POST /embeddings; your application produces them.

Cross-references

Topic	Doc
HTTP + Rust API contract (handwritten)	API.md
Get-started walkthrough	QUICKSTART.md
Metrics, alert thresholds, Grafana	docs/ops/monitoring.md
Crash recovery, WAL replay, backup/restore	docs/ops/recovery.md
Sizing memory/disk/CPU, scrape budgets	docs/ops/capacity-planning.md
Kubernetes deployment (probes, volumes)	docs/runbooks/kubernetes.md
Cluster operation (experimental)	docs/runbooks/cluster.md

---

1. Run Modes

tidal-server has two subcommands. Pick standalone unless you have a specific reason not to.

Standalone — the recommended path

tidalDB is single-node-first. One process wraps one TidalDb instance, serves the full data + health surface, and scales vertically. This is the production-ready mode. It is the default in every shipped Docker image and the only mode covered by the durability and recovery guarantees in docs/ops/recovery.md.

tidal-server standalone \
  --listen 0.0.0.0:9400 \
  --schema ./schema.yaml \
  --data-dir /var/lib/tidaldb \
  --metrics 127.0.0.1:9091

Cluster — EXPERIMENTAL, not production HA

Cluster mode runs multiple "regions" behind one HTTP surface and replicates between them over the real tidal-net gRPC transport. The critical caveat: all regions run inside this one process over loopback — there is no host or process isolation, so a process crash takes down every region at once. This is not production high availability. True multi-process HA is tracked as m8p10.

The mode refuses to start unless you opt in explicitly (--experimental-cluster or TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1) and emits a loud WARN describing exactly what it is and is not. Cluster mode currently replicates global signals only — user_id/creator_id signal context is dropped on the cluster path so followers stay consistent with the leader's WAL.

For everything cluster-specific (topology file, leader promotion, partition/heal, scatter-gather routes), see the cluster runbook. The rest of this guide focuses on standalone.

---

2. CLI Flags and Environment Variables

Standalone flags

Flag	Env override	Default	Meaning
--listen <addr>	PORT	127.0.0.1:9400	HTTP bind address. A bare PORT value (e.g. PORT=8080) is normalized to 0.0.0.0:8080. A host:port string is used verbatim.
--schema <path>	—	compiled-in default	Path to the schema/profile YAML (see section 3).
--config-dir <dir>	TIDAL_CONFIG	unset	Directory holding default-schema.yaml. Used only when --schema is omitted; the explicit flag always wins. If the dir is set but the file is missing, startup fails loudly rather than silently serving compiled-in defaults.
--data-dir <dir>	—	unset → EPHEMERAL	Persistent data directory. If omitted, the server runs in-memory and loses every write on restart. Always set this in production.
--metrics <addr>	—	unset (disabled)	Bind address for the Prometheus endpoint. Bind to loopback only — it is unauthenticated (see section 7).

Cluster flags

Flag	Env override	Default	Meaning
--listen <addr>	PORT	127.0.0.1:9500	HTTP bind address (same bare-port normalization).
--schema <path>	—	compiled-in default	Schema/profile YAML.
--topology <path>	—	compiled-in default	Cluster topology YAML (regions, leader). See the cluster runbook.
--config-dir <dir>	TIDAL_CONFIG	unset	Directory holding default-schema.yaml and default-cluster.yaml.
--experimental-cluster	TIDAL_ALLOW_EXPERIMENTAL_CLUSTER	off	Required opt-in. Without it (flag or env 1/true/yes), cluster mode refuses to start.

Environment variables (all modes)

Variable	Default	Meaning
TIDAL_API_KEY	unset	Bearer token for data routes. If unset, the server is UNAUTHENTICATED and logs a WARN at startup. See section 4.
TIDAL_CONFIG	unset	Config directory (backs --config-dir).
PORT	unset	Listen port/address (backs --listen); bare port → 0.0.0.0:PORT.
TIDAL_SERVER_LOG	info	tracing env-filter directive, e.g. TIDAL_SERVER_LOG=tidal_server=debug,info.
TIDAL_ALLOW_EXPERIMENTAL_CLUSTER	unset	Truthy (1/true/yes) opts in to cluster mode.

---

3. Schema YAML

The server loads its schema and ranking profiles from YAML at startup. This is the YAML projection of the Rust SchemaBuilder API. Four top-level keys: signals (required), text_fields, embedding_slots, profiles (all optional). At least one signal must be defined or startup fails.

Top-level shape

signals:        # required, >= 1
text_fields:    # optional
embedding_slots: # optional
profiles:       # optional — built-in profiles are available regardless

signals

signals:
  - name: view
    entity: item            # item | user | creator
    decay:
      exponential:
        half_life_seconds: 604800   # exclusive: exponential | linear | permanent
    windows: [one_hour, twenty_four_hours, seven_days]
    velocity: true
    positive_engagement: true   # optional; folds into the user preference vector

• decay is exactly one of:
  • exponential: { half_life_seconds: <f64> }
  • linear: { lifetime_seconds: <f64> }
  • permanent: true
• windows accepts: one_hour (1h), twenty_four_hours (24h), seven_days (7d), thirty_days (30d), all_time (alltime).
• velocity: true enables velocity tracking for the signal.
• positive_engagement: true makes engaging with an item via this signal fold the item's embedding into the user's preference vector (personalization). Omit it and the engine falls back to name-based detection — declare it explicitly on signals like view/like/completion for predictable taste.

text_fields

text_fields:
  - name: title
    kind: text       # text (analyzed, BM25) | keyword (exact)
  - name: category
    kind: keyword

embedding_slots

embedding_slots:
  - name: content_vector
    entity: item
    dimensions: 128

Embedding rules to respect from the engine side:

• The app brings vectors; the server's POST /embeddings L2-normalizes them and inserts into the HNSW index. tidalDB does not generate embeddings.
• Dimensions are strict: min 2, max 4096, and the submitted vector length must equal the slot's declared dimensions or the insert fails. Zero-norm vectors are rejected.
• Multi-slot caveat: RETRIEVE/SEARCH route through the first declared slot only. Multi-modal apps must fuse offline or use separate entity kinds.

profiles

Optional. The 25 built-in profiles (for_you, trending, hot, new, following, related, top_week, most_viewed, controversial, hidden_gems, shuffle, cohort_trending, etc.) are available without declaring anything. Use profiles to add custom ones or override behavior.

profiles:
  - name: for_you
    version: 1
    candidate_strategy:           # default: scan(created_at)
      ann:
        slot: content_vector
        limit: 200
    sort:                         # string OR parameterized map
      hot:
        gravity: 1.5
    boosts:
      - { signal: like, agg: decay_score, window: all_time, weight: 2.5 }
      - { signal: view, agg: velocity, window: one_hour, weight: 1.0 }
    gates:
      - { signal: skip, agg: value, window: all_time, min_threshold: 0.0 }
    penalties:
      - { signal: skip, agg: decay_score, window: seven_days, weight: 1.5 }
    excludes:
      - { signal: skip, agg: value, window: all_time, above: 1.0 }
    diversity:
      max_per_creator: 2
      format_mix_max_fraction: 0.4
    exploration: 0.1

Field reference:

• candidate_strategy — scan | relationship | hybrid | cohort_trending, or a map: ann: { slot, limit }, signal_ranked: { signal, window }, scan: { sort_field }. Default: scan over created_at.
• sort — string: trending, rising, controversial, hidden_gems, shuffle, new, most_followed, creator_engagement_rate, alphabetical_asc, alphabetical_desc, shortest, longest, live_viewer_count, date_saved; or map: hot: { gravity }, top_window: { window }, most_viewed: { window }, most_liked: { window }, most_commented: { window }, most_shared: { window }.
• boosts / penalties — { signal, agg, window, weight }. agg ∈ value | velocity | decay_score | ratio | relative_velocity.
• gates — { signal, agg, window, min_threshold } (drop candidates below the threshold).
• excludes — { signal, agg, window, above } (drop candidates above the threshold; e.g. hard-negative skip/hide/block).
• diversity — { max_per_creator, format_mix_max_fraction }.
• exploration — f64 in [0,1].

Every signal named in a profile's boosts/gates/penalties/excludes/decay must be declared in signals, or startup fails with a clear error naming the offending profile and signal.

Correctness rule — preference-vector personalization. The user "taste vector" that drives for_you is folded from item embeddings only for signals declared positive_engagement: true (the YAML field above; equivalently SignalBuilder::positive_engagement(true) in embedded mode). Declare it on your engaging signals (view/like/completion); if you omit it on every signal, the engine falls back to a name-based allowlist. If your personalization looks flat, confirm the engaging signals carry the flag. See API.md for the underlying signal_with_context semantics.

Worked example (the shipped default)

signals:
  - name: view
    entity: item
    decay:
      exponential:
        half_life_seconds: 604800    # 7 days
    windows: [one_hour, twenty_four_hours, seven_days]
    velocity: true
  - name: like
    entity: item
    decay:
      exponential:
        half_life_seconds: 1209600   # 14 days
    windows: [twenty_four_hours, seven_days, thirty_days, all_time]
    velocity: false
  - name: skip
    entity: item
    decay:
      permanent: true
    velocity: false
text_fields:
  - name: title
    kind: text
  - name: category
    kind: keyword
embedding_slots:
  - name: content_vector
    entity: item
    dimensions: 128

This is the compiled-in default at tidal-server/config/default-schema.yaml, used when neither --schema nor a TIDAL_CONFIG directory provides one.

---

4. Authentication

Data routes (/items, /embeddings, /signals, /feed, /search) require an Authorization: Bearer <token> header when TIDAL_API_KEY is set. The comparison is constant-time. A missing or wrong token returns 401 with WWW-Authenticate: Bearer.

Health probes (/health, /health/startup, /health/live) and the OpenAPI document (/openapi.json) are always unauthenticated — a client needs the spec to learn how to authenticate, and probes must work for orchestrators that cannot present a token.

WARNING — no key means an open server. If TIDAL_API_KEY is unset (or empty), every data route is served without authentication and the server logs a startup WARN. This is fine for local development. Never expose an unauthenticated server to any network. Set TIDAL_API_KEY before binding to anything other than 127.0.0.1.

export TIDAL_API_KEY="$(openssl rand -hex 32)"
curl -H "Authorization: Bearer $TIDAL_API_KEY" \
     "http://localhost:9400/feed?user_id=42&profile=for_you&limit=20"

---

5. The Served OpenAPI Reference (GET /openapi.json)

tidal-server serves a machine-readable OpenAPI 3.1 document at GET /openapi.json, unauthenticated. This is the canonical HTTP API contract — it is derived directly from the handler signatures and DTOs, so it can never drift from the running code (unlike a handwritten spec). API.md remains the prose companion, but /openapi.json is the source of truth for request/response shapes.

Inspect it:

curl -s localhost:9400/openapi.json | jq .info
# {
#   "title": "tidalDB HTTP API",
#   "version": "<crate version>",
#   "description": "Embeddable, single-node-first database for the personalized content ranking problem..."
# }

You can load /openapi.json into any OpenAPI viewer (Swagger UI, Redoc, Stoplight, Postman) or feed it to a client generator (openapi-generator, oapi-codegen, etc.) to produce typed clients in any language. The standalone document describes the data + health surface; the cluster document is a superset that also describes the /cluster/* and /sharded/* routes. The bearerAuth security scheme is declared on the data routes so generated clients know to send the token; /openapi.json and the probes carry no security requirement.

Route summary (standalone)

Method + path	Auth	Success	Notes
GET /health	no	200 ready / 503 draining	Readiness probe; reports item count.
GET /health/startup	no	200 always	Startup probe.
GET /health/live	no	200 always	Liveness probe.
GET /openapi.json	no	200	OpenAPI 3.1 document.
POST /items	yes	201	Body: { "entity_id": <u64>, "metadata": { ... } }.
POST /embeddings	yes	204	Body: { "entity_id": <u64>, "values": [<f32>, ...] }.
POST /signals	yes	204	Body: { "entity_id", "signal", "weight", "user_id"?, "creator_id"? }.
GET /feed	yes	200	?user_id=&profile=for_you&limit=20.
GET /search	yes	200	?query=&user_id=&limit=20.

limit is clamped to 1000 at the trust boundary. Request middleware: 30 s timeout (408), 100 max in-flight (429), 2 MB body cap (413), and an x-request-id echoed on every response.

---

6. Running the Server

Via cargo (development)

# Ephemeral, loopback, default schema — quickest smoke test:
cargo run -p tidal-server -- standalone

# Durable, real schema, metrics on loopback:
cargo run -p tidal-server -- standalone \
  --listen 0.0.0.0:9400 \
  --schema tidal-server/config/default-schema.yaml \
  --data-dir /var/lib/tidaldb \
  --metrics 127.0.0.1:9091

For an end-to-end engine walkthrough (embedded, no server), run the verified example: cargo run -p tidaldb --example foryou_feed. See QUICKSTART.md.

Via Docker

Three images ship, all built from the repository root as build context:

Image	Dockerfile	Toolchain	Use
standalone	docker/standalone/Dockerfile	rust:1.91-bookworm	General single-node; bundles curl for healthcheck.
deploy	docker/deploy/Dockerfile	rust:1.91-slim-bookworm, uid 10001	Slim production single-node; bakes the schema, WORKDIR /data.
cluster	docker/cluster/Dockerfile	—	Experimental cluster mode; see the cluster runbook.

Both single-node images:

• Run as a non-root tidal user.
• Declare VOLUME ["/data"] and pass --data-dir /data in the default CMD. Durability lives on /data — you must mount a real volume there or every write is lost on restart (the container layer is ephemeral).
• Use ENTRYPOINT ["tidal-server"] + a CMD of subcommand args, so docker run <image> <subcommand> ... overrides work (e.g. running an inspect command).
• Set ENV TIDAL_CONFIG to the baked config dir (/etc/tidal-server for standalone, /config for deploy) so the container is self-contained.
• Expose 9400 (HTTP) and 9091 (metrics).

# Build (from repo root):
docker build -f docker/standalone/Dockerfile -t tidaldb:standalone .

# Run with a named volume for durability and an API key:
docker run -d --name tidaldb \
  -p 9400:9400 \
  -p 127.0.0.1:9091:9091 \
  -e TIDAL_API_KEY="$(openssl rand -hex 32)" \
  -v tidaldb-data:/data \
  tidaldb:standalone

Note the metrics port is published to 127.0.0.1 only — it is unauthenticated (see section 7).

Via docker-compose

docker/docker-compose.yml brings up the standalone image plus a Prometheus sidecar, with a named tidaldb-data volume mounted at /data:

cd docker
export TIDAL_API_KEY="$(openssl rand -hex 32)"
docker compose up -d

The compose healthcheck calls GET /health with the Bearer header. Adjust the tidaldb-data volume to a host bind mount if you want the data outside Docker's volume store. For Kubernetes equivalents (probes wired to /health/startup, /health/live, /health; a PersistentVolumeClaim at /data), see the Kubernetes runbook.

---

7. Metrics and Shutdown

Metrics

--metrics <addr> enables the engine's Prometheus endpoint on a separate port (default image: 9091). It serves /metrics (Prometheus text format) and /healthz (JSON). This endpoint is UNAUTHENTICATED — bind it to loopback or a cluster-internal interface only. The engine logs a WARN if you bind it to a non-loopback address. For the full metric catalog, scrape config, alert thresholds, and Grafana dashboard, see docs/ops/monitoring.md. For sizing the scrape budget and memory/disk/CPU, see docs/ops/capacity-planning.md.

Graceful shutdown

On SIGTERM (or Ctrl-C), the server:

1. Flips readiness to 503 — GET /health reports { "ok": false, "cause": "shutting down" } so a load balancer stops routing new traffic. Startup/liveness probes still return 200.
2. Drains in-flight requests (axum graceful shutdown).
3. Closes the database — checkpoints state and fsyncs the WAL before the process exits. The standalone path reclaims sole ownership and runs this deterministically; if a flush fails it is logged at error level. There is a Drop backstop that runs the same shutdown if ownership is unexpectedly still shared.

Give orchestrators a terminationGracePeriod long enough to cover drain + checkpoint (a few seconds is typically ample for single-node; size against your WAL volume per capacity planning). For crash recovery, WAL replay, and backup/restore, see docs/ops/recovery.md.

Reverse proxy and TLS termination

tidal-server speaks plain HTTP and does not terminate TLS. In production, front it with a reverse proxy (nginx, Caddy, Envoy, a cloud load balancer, or a Kubernetes Ingress) that:

• Terminates TLS and forwards to the server's --listen address.
• Performs health checks against GET /health (it returns 503 during drain, so the proxy stops sending traffic before the process exits).
• Optionally adds its own rate limiting in front of the server's 100-in-flight / 30 s-timeout middleware.
• Preserves or sets x-request-id if you want end-to-end tracing — the server generates one when absent and echoes it on the response.

Keep the metrics port off the public proxy entirely; scrape it over the internal network only.