tidaldb/docs/guides/server-deployment.md

# Server Deployment

Operational reference for running `tidal-server` — the HTTP wrapper around the embeddable `tidaldb` engine. This is the doc to read before exposing tidalDB over the network.

`tidal-server` is a thin Axum surface over the engine. It does not change the engine's correctness model: every write still goes through the WAL, every shutdown still checkpoints and fsyncs. What the server adds is an HTTP contract, Bearer auth, request middleware (timeouts, concurrency cap, body limit, request IDs), and a YAML-driven schema/profile loader so you can deploy without writing Rust.

> tidalDB owns **retrieval and ranking only**. It does **not** generate embeddings, store video/blobs, run a CDN/transcoder, or provide auth/moderation/payments. The server brings vectors in over `POST /embeddings`; your application produces them.

**Cross-references**

| Topic | Doc |
|-------|-----|
| HTTP + Rust API contract (handwritten) | [API.md](../../API.md) |
| Get-started walkthrough | [QUICKSTART.md](../../QUICKSTART.md) |
| Metrics, alert thresholds, Grafana | [docs/ops/monitoring.md](../ops/monitoring.md) |
| Crash recovery, WAL replay, backup/restore | [docs/ops/recovery.md](../ops/recovery.md) |
| Sizing memory/disk/CPU, scrape budgets | [docs/ops/capacity-planning.md](../ops/capacity-planning.md) |
| Kubernetes deployment (probes, volumes) | [docs/runbooks/kubernetes.md](../runbooks/kubernetes.md) |
| Cluster operation (experimental) | [docs/runbooks/cluster.md](../runbooks/cluster.md) |

---

## 1. Run Modes

`tidal-server` has two subcommands. Pick `standalone` unless you have a specific reason not to.

### Standalone — the recommended path

tidalDB is **single-node-first**. One process wraps one `TidalDb` instance, serves the full data + health surface, and scales vertically. This is the production-ready mode. It is the default in every shipped Docker image and the only mode covered by the durability and recovery guarantees in [docs/ops/recovery.md](../ops/recovery.md).

```bash
tidal-server standalone \
  --listen 0.0.0.0:9400 \
  --schema ./schema.yaml \
  --data-dir /var/lib/tidaldb \
  --metrics 127.0.0.1:9091
```

### Cluster — EXPERIMENTAL, not production HA

Cluster mode runs multiple "regions" behind one HTTP surface and replicates between them over the **real `tidal-net` gRPC transport**. The critical caveat: **all regions run inside this one process over loopback** — there is no host or process isolation, so a process crash takes down every region at once. **This is not production high availability.** True multi-process HA is tracked as **m8p10**.

The mode refuses to start unless you opt in explicitly (`--experimental-cluster` or `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1`) and emits a loud `WARN` describing exactly what it is and is not. Cluster mode currently replicates **global signals only** — `user_id`/`creator_id` signal context is dropped on the cluster path so followers stay consistent with the leader's WAL.

For everything cluster-specific (topology file, leader promotion, partition/heal, scatter-gather routes), see the **[cluster runbook](../runbooks/cluster.md)**. The rest of this guide focuses on standalone.

---

## 2. CLI Flags and Environment Variables

### Standalone flags

| Flag | Env override | Default | Meaning |
|------|--------------|---------|---------|
| `--listen <addr>` | `PORT` | `127.0.0.1:9400` | HTTP bind address. A bare `PORT` value (e.g. `PORT=8080`) is normalized to `0.0.0.0:8080`. A `host:port` string is used verbatim. |
| `--schema <path>` | — | compiled-in default | Path to the schema/profile YAML (see [section 3](#3-schema-yaml)). |
| `--config-dir <dir>` | `TIDAL_CONFIG` | unset | Directory holding `default-schema.yaml`. Used only when `--schema` is omitted; the explicit flag always wins. If the dir is set but the file is missing, startup **fails loudly** rather than silently serving compiled-in defaults. |
| `--data-dir <dir>` | — | **unset → EPHEMERAL** | Persistent data directory. **If omitted, the server runs in-memory and loses every write on restart.** Always set this in production. |
| `--metrics <addr>` | — | unset (disabled) | Bind address for the Prometheus endpoint. **Bind to loopback only — it is unauthenticated** (see [section 7](#7-metrics-and-shutdown)). |

### Cluster flags

| Flag | Env override | Default | Meaning |
|------|--------------|---------|---------|
| `--listen <addr>` | `PORT` | `127.0.0.1:9500` | HTTP bind address (same bare-port normalization). |
| `--schema <path>` | — | compiled-in default | Schema/profile YAML. |
| `--topology <path>` | — | compiled-in default | Cluster topology YAML (`regions`, `leader`). See the [cluster runbook](../runbooks/cluster.md). |
| `--config-dir <dir>` | `TIDAL_CONFIG` | unset | Directory holding `default-schema.yaml` and `default-cluster.yaml`. |
| `--experimental-cluster` | `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER` | off | Required opt-in. Without it (flag or env `1`/`true`/`yes`), cluster mode refuses to start. |

### Environment variables (all modes)

| Variable | Default | Meaning |
|----------|---------|---------|
| `TIDAL_API_KEY` | unset | Bearer token for data routes. **If unset, the server is UNAUTHENTICATED** and logs a WARN at startup. See [section 4](#4-authentication). |
| `TIDAL_CONFIG` | unset | Config directory (backs `--config-dir`). |
| `PORT` | unset | Listen port/address (backs `--listen`); bare port → `0.0.0.0:PORT`. |
| `TIDAL_SERVER_LOG` | `info` | `tracing` env-filter directive, e.g. `TIDAL_SERVER_LOG=tidal_server=debug,info`. |
| `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER` | unset | Truthy (`1`/`true`/`yes`) opts in to cluster mode. |

---

## 3. Schema YAML

The server loads its schema and ranking profiles from YAML at startup. This is the YAML projection of the Rust `SchemaBuilder` API. Four top-level keys: `signals` (required), `text_fields`, `embedding_slots`, `profiles` (all optional). At least one signal must be defined or startup fails.

### Top-level shape

```yaml
signals:        # required, >= 1
text_fields:    # optional
embedding_slots: # optional
profiles:       # optional — built-in profiles are available regardless
```

### `signals`

```yaml
signals:
  - name: view
    entity: item            # item | user | creator
    decay:
      exponential:
        half_life_seconds: 604800   # exclusive: exponential | linear | permanent
    windows: [one_hour, twenty_four_hours, seven_days]
    velocity: true
    positive_engagement: true   # optional; folds into the user preference vector
```

- `decay` is exactly one of:
  - `exponential: { half_life_seconds: <f64> }`
  - `linear: { lifetime_seconds: <f64> }`
  - `permanent: true`
- `windows` accepts: `one_hour` (`1h`), `twenty_four_hours` (`24h`), `seven_days` (`7d`), `thirty_days` (`30d`), `all_time` (`alltime`).
- `velocity: true` enables velocity tracking for the signal.
- `positive_engagement: true` makes engaging with an item via this signal fold
  the item's embedding into the user's preference vector (personalization).
  Omit it and the engine falls back to name-based detection — declare it
  explicitly on signals like `view`/`like`/`completion` for predictable taste.

### `text_fields`

```yaml
text_fields:
  - name: title
    kind: text       # text (analyzed, BM25) | keyword (exact)
  - name: category
    kind: keyword
```

### `embedding_slots`

```yaml
embedding_slots:
  - name: content_vector
    entity: item
    dimensions: 128
```

Embedding rules to respect from the engine side:
- The app **brings vectors**; the server's `POST /embeddings` L2-normalizes them and inserts into the HNSW index. tidalDB does not generate embeddings.
- Dimensions are **strict**: min 2, max 4096, and the submitted vector length **must equal the slot's declared `dimensions`** or the insert fails. Zero-norm vectors are rejected.
- **Multi-slot caveat:** `RETRIEVE`/`SEARCH` route through the **first declared slot only**. Multi-modal apps must fuse offline or use separate entity kinds.

### `profiles`

Optional. The 25 built-in profiles (`for_you`, `trending`, `hot`, `new`, `following`, `related`, `top_week`, `most_viewed`, `controversial`, `hidden_gems`, `shuffle`, `cohort_trending`, etc.) are available without declaring anything. Use `profiles` to add custom ones or override behavior.

```yaml
profiles:
  - name: for_you
    version: 1
    candidate_strategy:           # default: scan(created_at)
      ann:
        slot: content_vector
        limit: 200
    sort:                         # string OR parameterized map
      hot:
        gravity: 1.5
    boosts:
      - { signal: like, agg: decay_score, window: all_time, weight: 2.5 }
      - { signal: view, agg: velocity, window: one_hour, weight: 1.0 }
    gates:
      - { signal: skip, agg: value, window: all_time, min_threshold: 0.0 }
    penalties:
      - { signal: skip, agg: decay_score, window: seven_days, weight: 1.5 }
    excludes:
      - { signal: skip, agg: value, window: all_time, above: 1.0 }
    diversity:
      max_per_creator: 2
      format_mix_max_fraction: 0.4
    exploration: 0.1
```

Field reference:
- **`candidate_strategy`** — `scan` | `relationship` | `hybrid` | `cohort_trending`, or a map: `ann: { slot, limit }`, `signal_ranked: { signal, window }`, `scan: { sort_field }`. Default: `scan` over `created_at`.
- **`sort`** — string: `trending`, `rising`, `controversial`, `hidden_gems`, `shuffle`, `new`, `most_followed`, `creator_engagement_rate`, `alphabetical_asc`, `alphabetical_desc`, `shortest`, `longest`, `live_viewer_count`, `date_saved`; or map: `hot: { gravity }`, `top_window: { window }`, `most_viewed: { window }`, `most_liked: { window }`, `most_commented: { window }`, `most_shared: { window }`.
- **`boosts` / `penalties`** — `{ signal, agg, window, weight }`. `agg` ∈ `value | velocity | decay_score | ratio | relative_velocity`.
- **`gates`** — `{ signal, agg, window, min_threshold }` (drop candidates below the threshold).
- **`excludes`** — `{ signal, agg, window, above }` (drop candidates above the threshold; e.g. hard-negative skip/hide/block).
- **`diversity`** — `{ max_per_creator, format_mix_max_fraction }`.
- **`exploration`** — `f64` in `[0,1]`.

Every `signal` named in a profile's boosts/gates/penalties/excludes/decay **must be declared in `signals`**, or startup fails with a clear error naming the offending profile and signal.

> **Correctness rule — preference-vector personalization.** The user "taste vector" that drives `for_you` is folded from item embeddings **only for signals declared `positive_engagement: true`** (the YAML field above; equivalently `SignalBuilder::positive_engagement(true)` in embedded mode). Declare it on your engaging signals (`view`/`like`/`completion`); if you omit it on *every* signal, the engine falls back to a name-based allowlist. If your personalization looks flat, confirm the engaging signals carry the flag. See [API.md](../../API.md) for the underlying `signal_with_context` semantics.

### Worked example (the shipped default)

```yaml
signals:
  - name: view
    entity: item
    decay:
      exponential:
        half_life_seconds: 604800    # 7 days
    windows: [one_hour, twenty_four_hours, seven_days]
    velocity: true
  - name: like
    entity: item
    decay:
      exponential:
        half_life_seconds: 1209600   # 14 days
    windows: [twenty_four_hours, seven_days, thirty_days, all_time]
    velocity: false
  - name: skip
    entity: item
    decay:
      permanent: true
    velocity: false
text_fields:
  - name: title
    kind: text
  - name: category
    kind: keyword
embedding_slots:
  - name: content_vector
    entity: item
    dimensions: 128
```

This is the compiled-in default at `tidal-server/config/default-schema.yaml`, used when neither `--schema` nor a `TIDAL_CONFIG` directory provides one.

---

## 4. Authentication

Data routes (`/items`, `/embeddings`, `/signals`, `/feed`, `/search`) require an `Authorization: Bearer <token>` header **when `TIDAL_API_KEY` is set**. The comparison is constant-time. A missing or wrong token returns `401` with `WWW-Authenticate: Bearer`.

Health probes (`/health`, `/health/startup`, `/health/live`) and the OpenAPI document (`/openapi.json`) are **always unauthenticated** — a client needs the spec to learn how to authenticate, and probes must work for orchestrators that cannot present a token.

> **WARNING — no key means an open server.** If `TIDAL_API_KEY` is unset (or empty), **every data route is served without authentication** and the server logs a startup WARN. This is fine for local development. **Never expose an unauthenticated server to any network.** Set `TIDAL_API_KEY` before binding to anything other than `127.0.0.1`.

```bash
export TIDAL_API_KEY="$(openssl rand -hex 32)"
curl -H "Authorization: Bearer $TIDAL_API_KEY" \
     "http://localhost:9400/feed?user_id=42&profile=for_you&limit=20"
```

---

## 5. The Served OpenAPI Reference (`GET /openapi.json`)

`tidal-server` serves a machine-readable **OpenAPI 3.1** document at `GET /openapi.json`, unauthenticated. **This is the canonical HTTP API contract** — it is derived directly from the handler signatures and DTOs, so it can never drift from the running code (unlike a handwritten spec). [API.md](../../API.md) remains the prose companion, but `/openapi.json` is the source of truth for request/response shapes.

Inspect it:

```bash
curl -s localhost:9400/openapi.json | jq .info
# {
#   "title": "tidalDB HTTP API",
#   "version": "<crate version>",
#   "description": "Embeddable, single-node-first database for the personalized content ranking problem..."
# }
```

You can load `/openapi.json` into any OpenAPI viewer (Swagger UI, Redoc, Stoplight, Postman) or feed it to a client generator (`openapi-generator`, `oapi-codegen`, etc.) to produce typed clients in any language. The standalone document describes the data + health surface; the cluster document is a superset that also describes the `/cluster/*` and `/sharded/*` routes. The `bearerAuth` security scheme is declared on the data routes so generated clients know to send the token; `/openapi.json` and the probes carry no security requirement.

### Route summary (standalone)

| Method + path | Auth | Success | Notes |
|---------------|------|---------|-------|
| `GET /health` | no | `200` ready / `503` draining | Readiness probe; reports item count. |
| `GET /health/startup` | no | `200` always | Startup probe. |
| `GET /health/live` | no | `200` always | Liveness probe. |
| `GET /openapi.json` | no | `200` | OpenAPI 3.1 document. |
| `POST /items` | yes | `201` | Body: `{ "entity_id": <u64>, "metadata": { ... } }`. |
| `POST /embeddings` | yes | `204` | Body: `{ "entity_id": <u64>, "values": [<f32>, ...] }`. |
| `POST /signals` | yes | `204` | Body: `{ "entity_id", "signal", "weight", "user_id"?, "creator_id"? }`. |
| `GET /feed` | yes | `200` | `?user_id=&profile=for_you&limit=20`. |
| `GET /search` | yes | `200` | `?query=&user_id=&limit=20`. |

`limit` is clamped to `1000` at the trust boundary. Request middleware: 30 s timeout (`408`), 100 max in-flight (`429`), 2 MB body cap (`413`), and an `x-request-id` echoed on every response.

---

## 6. Running the Server

### Via cargo (development)

```bash
# Ephemeral, loopback, default schema — quickest smoke test:
cargo run -p tidal-server -- standalone

# Durable, real schema, metrics on loopback:
cargo run -p tidal-server -- standalone \
  --listen 0.0.0.0:9400 \
  --schema tidal-server/config/default-schema.yaml \
  --data-dir /var/lib/tidaldb \
  --metrics 127.0.0.1:9091
```

For an end-to-end engine walkthrough (embedded, no server), run the verified example: `cargo run -p tidaldb --example foryou_feed`. See [QUICKSTART.md](../../QUICKSTART.md).

### Via Docker

Three images ship, all built from the **repository root** as build context:

| Image | Dockerfile | Toolchain | Use |
|-------|-----------|-----------|-----|
| standalone | `docker/standalone/Dockerfile` | `rust:1.91-bookworm` | General single-node; bundles `curl` for healthcheck. |
| deploy | `docker/deploy/Dockerfile` | `rust:1.91-slim-bookworm`, uid 10001 | Slim production single-node; bakes the schema, `WORKDIR /data`. |
| cluster | `docker/cluster/Dockerfile` | — | Experimental cluster mode; see the [cluster runbook](../runbooks/cluster.md). |

Both single-node images:
- Run as a **non-root** `tidal` user.
- Declare `VOLUME ["/data"]` and pass `--data-dir /data` in the default `CMD`. **Durability lives on `/data` — you must mount a real volume there or every write is lost on restart** (the container layer is ephemeral).
- Use `ENTRYPOINT ["tidal-server"]` + a `CMD` of subcommand args, so `docker run <image> <subcommand> ...` overrides work (e.g. running an inspect command).
- Set `ENV TIDAL_CONFIG` to the baked config dir (`/etc/tidal-server` for standalone, `/config` for deploy) so the container is self-contained.
- Expose `9400` (HTTP) and `9091` (metrics).

```bash
# Build (from repo root):
docker build -f docker/standalone/Dockerfile -t tidaldb:standalone .

# Run with a named volume for durability and an API key:
docker run -d --name tidaldb \
  -p 9400:9400 \
  -p 127.0.0.1:9091:9091 \
  -e TIDAL_API_KEY="$(openssl rand -hex 32)" \
  -v tidaldb-data:/data \
  tidaldb:standalone
```

Note the metrics port is published to `127.0.0.1` only — it is unauthenticated (see [section 7](#7-metrics-and-shutdown)).

### Via docker-compose

`docker/docker-compose.yml` brings up the standalone image plus a Prometheus sidecar, with a named `tidaldb-data` volume mounted at `/data`:

```bash
cd docker
export TIDAL_API_KEY="$(openssl rand -hex 32)"
docker compose up -d
```

The compose healthcheck calls `GET /health` with the Bearer header. Adjust the `tidaldb-data` volume to a host bind mount if you want the data outside Docker's volume store. For Kubernetes equivalents (probes wired to `/health/startup`, `/health/live`, `/health`; a `PersistentVolumeClaim` at `/data`), see the [Kubernetes runbook](../runbooks/kubernetes.md).

---

## 7. Metrics and Shutdown

### Metrics

`--metrics <addr>` enables the engine's Prometheus endpoint on a **separate port** (default image: `9091`). It serves `/metrics` (Prometheus text format) and `/healthz` (JSON). **This endpoint is UNAUTHENTICATED — bind it to loopback or a cluster-internal interface only.** The engine logs a WARN if you bind it to a non-loopback address. For the full metric catalog, scrape config, alert thresholds, and Grafana dashboard, see [docs/ops/monitoring.md](../ops/monitoring.md). For sizing the scrape budget and memory/disk/CPU, see [docs/ops/capacity-planning.md](../ops/capacity-planning.md).

### Graceful shutdown

On `SIGTERM` (or Ctrl-C), the server:
1. **Flips readiness to `503`** — `GET /health` reports `{ "ok": false, "cause": "shutting down" }` so a load balancer stops routing new traffic. Startup/liveness probes still return `200`.
2. **Drains** in-flight requests (axum graceful shutdown).
3. **Closes the database** — checkpoints state and **fsyncs the WAL** before the process exits. The standalone path reclaims sole ownership and runs this deterministically; if a flush fails it is logged at error level. There is a `Drop` backstop that runs the same shutdown if ownership is unexpectedly still shared.

Give orchestrators a `terminationGracePeriod` long enough to cover drain + checkpoint (a few seconds is typically ample for single-node; size against your WAL volume per [capacity planning](../ops/capacity-planning.md)). For crash recovery, WAL replay, and backup/restore, see [docs/ops/recovery.md](../ops/recovery.md).

### Reverse proxy and TLS termination

`tidal-server` speaks **plain HTTP** and does not terminate TLS. In production, front it with a reverse proxy (nginx, Caddy, Envoy, a cloud load balancer, or a Kubernetes Ingress) that:
- Terminates TLS and forwards to the server's `--listen` address.
- Performs health checks against `GET /health` (it returns `503` during drain, so the proxy stops sending traffic before the process exits).
- Optionally adds its own rate limiting in front of the server's 100-in-flight / 30 s-timeout middleware.
- Preserves or sets `x-request-id` if you want end-to-end tracing — the server generates one when absent and echoes it on the response.

Keep the metrics port off the public proxy entirely; scrape it over the internal network only.