tidaldb/docs/guides/server-deployment.md

# Server Deployment

Operational reference for running `tidal-server` — the HTTP wrapper around the embeddable `tidaldb` engine. This is the doc to read before exposing tidalDB over the network.

`tidal-server` is a thin Axum surface over the engine. It does not change the engine's correctness model: every write still goes through the WAL, every shutdown still checkpoints and fsyncs. What the server adds is an HTTP contract, Bearer auth, request middleware (timeouts, concurrency cap, body limit, request IDs), and a YAML-driven schema/profile loader so you can deploy without writing Rust.

> tidalDB owns **retrieval and ranking only**. It does **not** generate embeddings, store video/blobs, run a CDN/transcoder, or provide auth/moderation/payments. The server brings vectors in over `POST /embeddings`; your application produces them.

**Cross-references**

| Topic | Doc |
|-------|-----|
| HTTP + Rust API contract (handwritten) | [API.md](../../API.md) |
| Get-started walkthrough | [QUICKSTART.md](../../QUICKSTART.md) |
| Metrics, alert thresholds, Grafana | [docs/ops/monitoring.md](../ops/monitoring.md) |
| Crash recovery, WAL replay, backup/restore | [docs/ops/recovery.md](../ops/recovery.md) |
| Sizing memory/disk/CPU, scrape budgets | [docs/ops/capacity-planning.md](../ops/capacity-planning.md) |
| Kubernetes deployment (probes, volumes) | [docs/runbooks/kubernetes.md](../runbooks/kubernetes.md) |
| Cluster operation (high availability) | [docs/runbooks/cluster.md](../runbooks/cluster.md) |

---

## 1. Run Modes

`tidal-server` has two subcommands. Pick `standalone` unless you have a specific reason not to.

### Standalone — the recommended path

tidalDB is **single-node-first**. One process wraps one `TidalDb` instance, serves the full data + health surface, and scales vertically. This is the production-ready mode. It is the default in every shipped Docker image and the only mode covered by the durability and recovery guarantees in [docs/ops/recovery.md](../ops/recovery.md).

```bash
tidal-server standalone \
  --listen 0.0.0.0:9400 \
  --schema ./schema.yaml \
  --data-dir /var/lib/tidaldb \
  --metrics 127.0.0.1:9091
```

### Cluster — production HA, opt-in

Cluster mode runs multiple "regions" behind one HTTP surface and replicates between them over the **real `tidal-net` gRPC transport**. It has two shapes: a **multi-process** mode (`--region`, one process per region, real gRPC peering between siblings, real process/host isolation — the production shape) and a single-process dev/demo default (all regions in one process over loopback). Since M11 it is genuinely HA: **quorum-acked writes** (`ack=quorum`, m11p3), **automatic election/failover** (m11p4), and **elastic membership** — DNS peers, seed join, online add/remove/replace with snapshot+stream catch-up (m11p5). M12 added sharded ingestion and TLS scale-up. The reference deployment runs in production on k3s. The recommended default is still a **single standalone node** (simpler, scales vertically); reach for cluster mode deliberately when you need multi-node availability or read-scale. See the [cluster runbook](../runbooks/cluster.md) and [`k8s/cluster/`](../../k8s/cluster/) for the reference deployment.

Because standalone is the right answer for most deployments, cluster mode refuses to start unless you opt in explicitly (`--experimental-cluster` or `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1`) — a guard against starting a multi-node fabric by accident — and emits a startup `WARN` describing exactly which shape it is running. Cluster mode replicates **global signals only** — `user_id`/`creator_id` signal context is dropped on the cluster path so followers stay consistent with the leader's WAL.

For everything cluster-specific (topology file, leader promotion, partition/heal, scatter-gather routes), see the **[cluster runbook](../runbooks/cluster.md)**. The rest of this guide focuses on standalone.

---

## 2. CLI Flags and Environment Variables

### Standalone flags

| Flag | Env override | Default | Meaning |
|------|--------------|---------|---------|
| `--listen <addr>` | `PORT` | `127.0.0.1:9400` | HTTP bind address. A bare `PORT` value (e.g. `PORT=8080`) is normalized to `0.0.0.0:8080`. A `host:port` string is used verbatim. |
| `--schema <path>` | — | compiled-in default | Path to the schema/profile YAML (see [section 3](#3-schema-yaml)). |
| `--config-dir <dir>` | `TIDAL_CONFIG` | unset | Directory holding `default-schema.yaml`. Used only when `--schema` is omitted; the explicit flag always wins. If the dir is set but the file is missing, startup **fails loudly** rather than silently serving compiled-in defaults. |
| `--data-dir <dir>` | — | **unset → EPHEMERAL** | Persistent data directory. **If omitted, the server runs in-memory and loses every write on restart.** Always set this in production. |
| `--metrics <addr>` | — | unset (disabled) | Bind address for the Prometheus endpoint. **Bind to loopback only — it is unauthenticated** (see [section 7](#7-metrics-and-shutdown)). |

### Cluster flags

| Flag | Env override | Default | Meaning |
|------|--------------|---------|---------|
| `--listen <addr>` | `PORT` | `127.0.0.1:9500` | HTTP bind address (same bare-port normalization). |
| `--schema <path>` | — | compiled-in default | Schema/profile YAML. |
| `--topology <path>` | — | compiled-in default | Cluster topology YAML (`regions`, `leader`). See the [cluster runbook](../runbooks/cluster.md). |
| `--config-dir <dir>` | `TIDAL_CONFIG` | unset | Directory holding `default-schema.yaml` and `default-cluster.yaml`. |
| `--region <name>` | `TIDAL_REGION` | unset | Run **only this region** in this process (multi-process mode). Omit for the single-process dev/demo default. |
| `--data-dir <dir>` | — | **required (multi-process)** | Persistent data directory. The durable WAL is the replicated log itself, so multi-process mode rejects a node without one. |
| `--seed <url>` (repeatable) | — | unset | Seed-join boot (m11p5): contact a running peer's HTTP base URL to learn the roster + assigned id + term and join online as a learner — no topology entry of this node's own required. Still needs `--topology`/`TIDAL_CONFIG` for the behavioral knob blocks. |
| `--advertise-grpc <host:port>` | — | unset | This node's **advertised** gRPC address (what siblings dial; DNS-capable). Required with `--seed`. |
| `--advertise-http <host:port>` | — | unset | This node's advertised HTTP gateway (what peers forward to). Required with `--seed`. |
| `--metrics <addr>` | — | unset | Prometheus `/metrics` bind for a seed-joined node (which has no topology `metrics_addr`). Bind to loopback / cluster-internal only — unauthenticated. |
| `--experimental-cluster` | `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER` | off | Required opt-in. Without it (flag or env `1`/`true`/`yes`), cluster mode refuses to start. |

### Environment variables (all modes)

| Variable | Default | Meaning |
|----------|---------|---------|
| `TIDAL_API_KEY` | unset | Bearer token for data routes. **If unset, the server is UNAUTHENTICATED** and logs a WARN at startup. See [section 4](#4-authentication). |
| `TIDAL_API_KEY_FILE` | unset | Path whose CONTENT is the bearer token (m11p7); takes precedence over `TIDAL_API_KEY`. The FILE form rotates **without a restart** (a credential poller re-reads it). |
| `TIDAL_ADMIN_KEY` | unset | OPERATOR credential gating the destructive `/cluster/*` verbs. **If unset, those verbs accept the data bearer** and the server logs a WARN. See [section 4](#4-authentication). |
| `TIDAL_ADMIN_KEY_FILE` | unset | Path whose CONTENT is the operator key; takes precedence over `TIDAL_ADMIN_KEY` and rotates without a restart. |
| `TIDAL_CONFIG` | unset | Config directory (backs `--config-dir`). |
| `PORT` | unset | Listen port/address (backs `--listen`); bare port → `0.0.0.0:PORT`. |
| `TIDAL_SERVER_LOG` | `info` | `tracing` env-filter directive, e.g. `TIDAL_SERVER_LOG=tidal_server=debug,info`. |
| `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER` | unset | Truthy (`1`/`true`/`yes`) opts in to cluster mode. |
| `TIDAL_CLUSTER_KEY` / `TIDAL_CLUSTER_KEY_FILE` | unset | **Cluster mode, m11p7.** Shared secret (any random string) that mints/verifies per-node internal tokens authenticating inter-node HTTP. Absent ⇒ per-node tokens disabled (the `x-tidal-internal` marker keeps hint-only behavior; WARN). The `_FILE` form rotates without restart. Never give it to external clients. See the [cluster runbook §12](../runbooks/cluster.md#12-security-m11p7-mtls-rotation-identity-audit-rate-limits). |
| `TIDAL_AUDIT_LOG` | unset | **Cluster mode, m11p7.** Path to an append-only JSONL admin-verb audit log (promote/heal/partition/conf-change). Always also emitted to the `tidal_audit` tracing target. At-rest encryption is delegated to the volume. |
| `TIDAL_RATE_LIMIT_RPS` / `TIDAL_RATE_LIMIT_BURST` | unset (unlimited) | **m11p7.** Per-principal HTTP rate limit (sustained / burst, default burst 2×). A deny is `429` + `Retry-After`. Verified sibling nodes are exempt. |
| `TIDAL_ROTATION_POLL_MS` | `30000` | **Cluster mode, m11p7.** How often the cert + credential rotation poller re-reads the TLS/secret files. |

> **Inter-node TLS (m11p7)** is configured by the topology's per-region `grpc_tls`
> block (paths to `ca_cert` / `server_cert` / `server_key` / `client_cert` /
> `client_key`), NOT an env var. Present ⇒ gRPC mutual TLS + inter-node HTTP TLS
> (one cert, one hot rotation, both planes); absent ⇒ plaintext inter-node links
> with a loud startup WARN. See the [cluster runbook §12](../runbooks/cluster.md#12-security-m11p7-mtls-rotation-identity-audit-rate-limits)
> and `k8s/cluster/` (cert-manager) / `scripts/gen-cluster-certs.sh`.

---

## 3. Schema YAML

The server loads its schema and ranking profiles from YAML at startup. This is the YAML projection of the Rust `SchemaBuilder` API. Four top-level keys: `signals` (required), `text_fields`, `embedding_slots`, `profiles` (all optional). At least one signal must be defined or startup fails.

### Top-level shape

```yaml
signals:        # required, >= 1
text_fields:    # optional
embedding_slots: # optional
profiles:       # optional — built-in profiles are available regardless
```

### `signals`

```yaml
signals:
  - name: view
    entity: item            # item | user | creator
    decay:
      exponential:
        half_life_seconds: 604800   # exclusive: exponential | linear | permanent
    windows: [one_hour, twenty_four_hours, seven_days]
    velocity: true
    positive_engagement: true   # optional; folds into the user preference vector
```

- `decay` is exactly one of:
  - `exponential: { half_life_seconds: <f64> }`
  - `linear: { lifetime_seconds: <f64> }`
  - `permanent: true`
- `windows` accepts: `one_hour` (`1h`), `twenty_four_hours` (`24h`), `seven_days` (`7d`), `thirty_days` (`30d`), `all_time` (`alltime`).
- `velocity: true` enables velocity tracking for the signal.
- `positive_engagement: true` makes engaging with an item via this signal fold
  the item's embedding into the user's preference vector (personalization).
  Omit it and the engine falls back to name-based detection — declare it
  explicitly on signals like `view`/`like`/`completion` for predictable taste.

### `text_fields`

```yaml
text_fields:
  - name: title
    kind: text       # text (analyzed, BM25) | keyword (exact)
  - name: category
    kind: keyword
```

### `embedding_slots`

```yaml
embedding_slots:
  - name: content_vector
    entity: item
    dimensions: 128
```

Embedding rules to respect from the engine side:
- The app **brings vectors**; the server's `POST /embeddings` L2-normalizes them and inserts into the HNSW index. tidalDB does not generate embeddings.
- Dimensions are **strict**: min 2, max 4096, and the submitted vector length **must equal the slot's declared `dimensions`** or the insert fails. Zero-norm vectors are rejected.
- **Multi-slot caveat:** `RETRIEVE`/`SEARCH` route through the **first declared slot only**. Multi-modal apps must fuse offline or use separate entity kinds.

### `profiles`

Optional. The 25 built-in profiles (`for_you`, `trending`, `hot`, `new`, `following`, `related`, `top_week`, `most_viewed`, `controversial`, `hidden_gems`, `shuffle`, `cohort_trending`, etc.) are available without declaring anything. Use `profiles` to add custom ones or override behavior.

```yaml
profiles:
  - name: for_you
    version: 1
    candidate_strategy:           # default: scan(created_at)
      ann:
        slot: content_vector
        limit: 200
    sort:                         # string OR parameterized map
      hot:
        gravity: 1.5
    boosts:
      - { signal: like, agg: decay_score, window: all_time, weight: 2.5 }
      - { signal: view, agg: velocity, window: one_hour, weight: 1.0 }
    gates:
      - { signal: skip, agg: value, window: all_time, min_threshold: 0.0 }
    penalties:
      - { signal: skip, agg: decay_score, window: seven_days, weight: 1.5 }
    excludes:
      - { signal: skip, agg: value, window: all_time, above: 1.0 }
    diversity:
      max_per_creator: 2
      format_mix_max_fraction: 0.4
    exploration: 0.1
```

Field reference:
- **`candidate_strategy`** — `scan` | `relationship` | `hybrid` | `cohort_trending`, or a map: `ann: { slot, limit }`, `signal_ranked: { signal, window }`, `scan: { sort_field }`. Default: `scan` over `created_at`.
- **`sort`** — string: `trending`, `rising`, `controversial`, `hidden_gems`, `shuffle`, `new`, `most_followed`, `creator_engagement_rate`, `alphabetical_asc`, `alphabetical_desc`, `shortest`, `longest`, `live_viewer_count`, `date_saved`; or map: `hot: { gravity }`, `top_window: { window }`, `most_viewed: { window }`, `most_liked: { window }`, `most_commented: { window }`, `most_shared: { window }`.
- **`boosts` / `penalties`** — `{ signal, agg, window, weight }`. `agg` ∈ `value | velocity | decay_score | ratio | relative_velocity`.
- **`gates`** — `{ signal, agg, window, min_threshold }` (drop candidates below the threshold).
- **`excludes`** — `{ signal, agg, window, above }` (drop candidates above the threshold; e.g. hard-negative skip/hide/block).
- **`diversity`** — `{ max_per_creator, format_mix_max_fraction }`.
- **`exploration`** — `f64` in `[0,1]`.

Every `signal` named in a profile's boosts/gates/penalties/excludes/decay **must be declared in `signals`**, or startup fails with a clear error naming the offending profile and signal.

> **Correctness rule — preference-vector personalization.** The user "taste vector" that drives `for_you` is folded from item embeddings **only for signals declared `positive_engagement: true`** (the YAML field above; equivalently `SignalBuilder::positive_engagement(true)` in embedded mode). Declare it on your engaging signals (`view`/`like`/`completion`); if you omit it on *every* signal, the engine falls back to a name-based allowlist. If your personalization looks flat, confirm the engaging signals carry the flag. See [API.md](../../API.md) for the underlying `signal_with_context` semantics.

### Worked example (the shipped default)

```yaml
signals:
  - name: view
    entity: item
    decay:
      exponential:
        half_life_seconds: 604800    # 7 days
    windows: [one_hour, twenty_four_hours, seven_days]
    velocity: true
  - name: like
    entity: item
    decay:
      exponential:
        half_life_seconds: 1209600   # 14 days
    windows: [twenty_four_hours, seven_days, thirty_days, all_time]
    velocity: false
  - name: skip
    entity: item
    decay:
      permanent: true
    velocity: false
text_fields:
  - name: title
    kind: text
  - name: category
    kind: keyword
embedding_slots:
  - name: content_vector
    entity: item
    dimensions: 128
```

This is the compiled-in default at `tidal-server/config/default-schema.yaml`, used when neither `--schema` nor a `TIDAL_CONFIG` directory provides one.

---

## 4. Authentication

Data routes (`/items`, `/embeddings`, `/signals`, `/feed`, `/search`) require an `Authorization: Bearer <token>` header **when `TIDAL_API_KEY` is set**. The comparison is constant-time. A missing or wrong token returns `401` with `WWW-Authenticate: Bearer`.

Health probes (`/health`, `/health/startup`, `/health/live`) and the OpenAPI document (`/openapi.json`) are **always unauthenticated** — a client needs the spec to learn how to authenticate, and probes must work for orchestrators that cannot present a token.

> **WARNING — no key means an open server.** If `TIDAL_API_KEY` is unset (or empty), **every data route is served without authentication** and the server logs a startup WARN. This is fine for local development. **Never expose an unauthenticated server to any network.** Set `TIDAL_API_KEY` before binding to anything other than `127.0.0.1`.

```bash
export TIDAL_API_KEY="$(openssl rand -hex 32)"
curl -H "Authorization: Bearer $TIDAL_API_KEY" \
     "http://localhost:9400/feed?user_id=42&profile=for_you&limit=20"
```

### Two credential tiers (cluster mode)

A cluster has **two** credentials, because one is not enough: every `/cluster/*`
mutation used to sit behind the same bearer as `/items` and `/search`, so any
application key could remove a member, force a partition, or transfer a shard.

| Credential | Env | Grants |
|---|---|---|
| Data bearer | `TIDAL_API_KEY` / `TIDAL_API_KEY_FILE` | The data routes, `/cluster/status`, and the peer-callable verbs (`/cluster/catchup`, `/cluster/join`, `/cluster/members`, `/cluster/reconcile*`). |
| Operator key | `TIDAL_ADMIN_KEY` / `TIDAL_ADMIN_KEY_FILE` | Everything above **plus** the destructive verbs: `/cluster/promote`, `/cluster/partition`, `/cluster/heal`, `/cluster/members/remove`, `/cluster/reseed`, `/cluster/shards/{id}/replicas`, `/cluster/shards/{id}/transfer`. |

Presenting the data bearer to a destructive verb returns **`403`** (authenticated,
not authorized) — distinct from the `401` a missing/invalid token returns.

The operator key is a **superset** credential: it also satisfies the
authentication gate, because a request carries a single `Authorization` header —
if it did not authenticate, an operator presenting it would be rejected `401`
before the authorization gate ran. Keep it off application hosts.

A **verified sibling node token** also clears the gate. That is load-bearing, not
a convenience: nodes relay operator verbs to the leader/target carrying whatever
credential the caller sent, so enabling an operator key can never partition the
control plane.

> **WARNING — one key means every client is an operator.** If `TIDAL_ADMIN_KEY` is
> unset, the destructive verbs accept the data bearer and the server logs a
> startup WARN naming the exposure. Behavior is unchanged from before the split,
> so this is safe to upgrade into, but set the key before handing a bearer to
> anything you do not operate.

```bash
export TIDAL_API_KEY="$(openssl rand -hex 32)"
export TIDAL_ADMIN_KEY="$(openssl rand -hex 32)"

# 403: the data key is not an operator credential
curl -X POST -H "Authorization: Bearer $TIDAL_API_KEY" \
     -d '{"region":"eu-west"}' http://localhost:9500/cluster/promote

# accepted
curl -X POST -H "Authorization: Bearer $TIDAL_ADMIN_KEY" \
     -d '{"region":"eu-west"}' http://localhost:9500/cluster/promote
```

`/cluster/status` and `/cluster/status/local` require a credential too. They
report leader identity, membership, term, and per-shard applied/lag/commit
seqnos — reconnaissance, not a probe — so they are **not** in the open group with
`/health`.

---

## 5. The Served OpenAPI Reference (`GET /openapi.json`)

`tidal-server` serves a machine-readable **OpenAPI 3.1** document at `GET /openapi.json`, unauthenticated. **This is the canonical HTTP API contract** — it is derived directly from the handler signatures and DTOs, so it can never drift from the running code (unlike a handwritten spec). [API.md](../../API.md) remains the prose companion, but `/openapi.json` is the source of truth for request/response shapes.

Inspect it:

```bash
curl -s localhost:9400/openapi.json | jq .info
# {
#   "title": "tidalDB HTTP API",
#   "version": "<crate version>",
#   "description": "Embeddable, single-node-first database for the personalized content ranking problem..."
# }
```

You can load `/openapi.json` into any OpenAPI viewer (Swagger UI, Redoc, Stoplight, Postman) or feed it to a client generator (`openapi-generator`, `oapi-codegen`, etc.) to produce typed clients in any language. The standalone document describes the data + health surface. The cluster document adds `/cluster/*` and `/sharded/*`, but intentionally omits standalone-only `POST /rank` until exact-profile version authority has a distributed contract. The `bearerAuth` security scheme is declared on the data routes so generated clients know to send the token; `/openapi.json` and the probes carry no security requirement.

### Route summary (standalone)

| Method + path | Auth | Success | Notes |
|---------------|------|---------|-------|
| `GET /health` | no | `200` ready / `503` draining | Readiness probe; reports item count. |
| `GET /health/startup` | no | `200` always | Startup probe. |
| `GET /health/live` | no | `200` always | Liveness probe. |
| `GET /openapi.json` | no | `200` | OpenAPI 3.1 document. |
| `POST /items` | yes | `201` | Body: `{ "entity_id": <u64>, "metadata": { ... } }`. |
| `POST /embeddings` | yes | `204` | Body: `{ "entity_id": <u64>, "values": [<f32>, ...] }`. |
| `POST /signals` | yes | `204` | Body: `{ "entity_id", "signal", "weight", "user_id"?, "creator_id"? }`. |
| `GET /feed` | yes | `200` | `?user_id=&profile=for_you&limit=20`. |
| `GET /search` | yes | `200` | `?query=&user_id=&limit=20`. |
| `POST /vector_search` | yes | `200` | Pure k-NN. Body: `{ "vector": [<f32>, ...], "k"?, "ef_search"? }`. |
| `POST /rank` | yes | `200` | Standalone-only exact ranking over a complete caller-supplied candidate set; the profile, version, and decimal-string `as_of_nanos` are required. |

Feed/search `limit` is clamped to `1000`; exact rank accepts at most 10,000 candidates and returns each exactly once. Request middleware: 30 s timeout (`408`), 100 max in-flight (`429`), 2 MB body cap (`413`), and an `x-request-id` echoed on every response.

---

## 6. Running the Server

### Via cargo (development)

```bash
# Ephemeral, loopback, default schema — quickest smoke test:
cargo run -p tidal-server -- standalone

# Durable, real schema, metrics on loopback:
cargo run -p tidal-server -- standalone \
  --listen 0.0.0.0:9400 \
  --schema tidal-server/config/default-schema.yaml \
  --data-dir /var/lib/tidaldb \
  --metrics 127.0.0.1:9091
```

For an end-to-end engine walkthrough (embedded, no server), run the verified example: `cargo run -p tidaldb --example foryou_feed`. See [QUICKSTART.md](../../QUICKSTART.md).

### Via Docker

Three images ship, all built from the **repository root** as build context:

| Image | Dockerfile | Toolchain | Use |
|-------|-----------|-----------|-----|
| standalone | `docker/standalone/Dockerfile` | `rust:1.91-bookworm` | General single-node; bundles `curl` for healthcheck. |
| deploy | `docker/deploy/Dockerfile` | `rust:1.91-slim-bookworm`, uid 10001 | Slim production single-node; bakes the schema, `WORKDIR /data`. |
| cluster | `docker/cluster/Dockerfile` | — | Experimental cluster mode; see the [cluster runbook](../runbooks/cluster.md). |

Both single-node images:
- Run as a **non-root** `tidal` user.
- Declare `VOLUME ["/data"]` and pass `--data-dir /data` in the default `CMD`. **Durability lives on `/data` — you must mount a real volume there or every write is lost on restart** (the container layer is ephemeral).
- Use `ENTRYPOINT ["tidal-server"]` + a `CMD` of subcommand args, so `docker run <image> <subcommand> ...` overrides work (e.g. running an inspect command).
- Set `ENV TIDAL_CONFIG` to the baked config dir (`/etc/tidal-server` for standalone, `/config` for deploy) so the container is self-contained.
- Expose `9400` (HTTP) and `9091` (metrics).

```bash
# Build (from repo root):
docker build -f docker/standalone/Dockerfile -t tidaldb:standalone .

# Run with a named volume for durability and an API key:
docker run -d --name tidaldb \
  -p 9400:9400 \
  -p 127.0.0.1:9091:9091 \
  -e TIDAL_API_KEY="$(openssl rand -hex 32)" \
  -v tidaldb-data:/data \
  tidaldb:standalone
```

Note the metrics port is published to `127.0.0.1` only — it is unauthenticated (see [section 7](#7-metrics-and-shutdown)).

### Via docker-compose

`docker/docker-compose.yml` brings up the standalone image plus a Prometheus sidecar, with a named `tidaldb-data` volume mounted at `/data`:

```bash
cd docker
export TIDAL_API_KEY="$(openssl rand -hex 32)"
docker compose up -d
```

The compose healthcheck calls `GET /health` with the Bearer header. Adjust the `tidaldb-data` volume to a host bind mount if you want the data outside Docker's volume store. For Kubernetes equivalents (probes wired to `/health/startup`, `/health/live`, `/health`; a `PersistentVolumeClaim` at `/data`), see the [Kubernetes runbook](../runbooks/kubernetes.md).

---

## 7. Metrics and Shutdown

### Metrics

`--metrics <addr>` enables the engine's Prometheus endpoint on a **separate port** (default image: `9091`). It serves `/metrics` (Prometheus text format) and `/healthz` (JSON). **This endpoint is UNAUTHENTICATED — bind it to loopback or a cluster-internal interface only.** The engine logs a WARN if you bind it to a non-loopback address. For the full metric catalog, scrape config, alert thresholds, and Grafana dashboard, see [docs/ops/monitoring.md](../ops/monitoring.md). For sizing the scrape budget and memory/disk/CPU, see [docs/ops/capacity-planning.md](../ops/capacity-planning.md).

### Graceful shutdown

On `SIGTERM` (or Ctrl-C), the server:
1. **Flips readiness to `503`** — `GET /health` reports `{ "ok": false, "cause": "shutting down" }` so a load balancer stops routing new traffic. Startup/liveness probes still return `200`.
2. **Drains** in-flight requests (axum graceful shutdown).
3. **Closes the database** — checkpoints state and **fsyncs the WAL** before the process exits. The standalone path reclaims sole ownership and runs this deterministically; if a flush fails it is logged at error level. There is a `Drop` backstop that runs the same shutdown if ownership is unexpectedly still shared.

Give orchestrators a `terminationGracePeriod` long enough to cover drain + checkpoint (a few seconds is typically ample for single-node; size against your WAL volume per [capacity planning](../ops/capacity-planning.md)). For crash recovery, WAL replay, and backup/restore, see [docs/ops/recovery.md](../ops/recovery.md).

### Reverse proxy and TLS termination

`tidal-server` speaks **plain HTTP** and does not terminate TLS. In production, front it with a reverse proxy (nginx, Caddy, Envoy, a cloud load balancer, or a Kubernetes Ingress) that:
- Terminates TLS and forwards to the server's `--listen` address.
- Performs health checks against `GET /health` (it returns `503` during drain, so the proxy stops sending traffic before the process exits).
- Optionally adds its own rate limiting in front of the server's 100-in-flight / 30 s-timeout middleware.
- Preserves or sets `x-request-id` if you want end-to-end tracing — the server generates one when absent and echoes it on the response.

Keep the metrics port off the public proxy entirely; scrape it over the internal network only.