# m11p7 — Security Hardening (COMPLETE — 2026-06-13)

Phase spec and exit gate: [docs/roadmap-to-cluster.md §4/m11p7](../../roadmap-to-cluster.md).
Closes the ROADMAP **network-trust** gap ("the cluster trusts the network").
Predecessors: p1–p2 (the stable replication surface this hardens), p3 (quorum),
p4 (election), p5 (membership), p6 L0–L2 (sharding data plane).

**Goal:** the cluster stops trusting the network. Inter-node links are encrypted
and authenticated; certs and bearer keys rotate without a restart; admin verbs
are audited with the principal who issued them; external load is rate-limited
per principal. The exit gate: a reference deployment has **zero plaintext
inter-node links**, **rotation under load drops zero requests**, and a **foreign
pod on the cluster network can neither ship segments nor call internal routes**.

## Design (as adopted)

### 1. gRPC mTLS by default + zero-drop cert rotation (`tidal-net`)

mTLS already worked when `grpc_tls` was configured, but the default was
plaintext and the cert was fixed for the server's life (tonic 0.12 caches one
`Arc<rustls::ServerConfig>` from a static identity and exposes no resolver hook).
m11p7 makes TLS the intended posture and the cert hot-swappable:

- **The server is served over a custom `tokio-rustls` acceptor** (not tonic's
  `.tls_config()`), fed a rustls `ServerConfig` whose identity is a
  `DynamicCertResolver` — an `ArcSwap<CertifiedKey>`. mTLS is preserved EXACTLY:
  a `WebPkiClientVerifier` over the cluster CA roots (NOT `allow_unauthenticated`)
  + ALPN `h2`. The accept loop runs each TLS handshake in its OWN task and
  forwards only successfully-handshaken streams into tonic, so a foreign pod —
  no client cert, a foreign-CA cert, or a plaintext probe — fails the handshake
  and **never reaches an RPC** (and never head-of-line-blocks the next connect).
- **Plaintext is an explicit `insecure: true` with a loud startup WARN** on both
  the server (`start_server`) and the client (`PeerPool::new`).
- **Rotation without restart**: a content-hash polling reloader
  (`ServerCertReloader`, `rotation_poll_interval`, default 30s) re-reads the cert
  files and atomically swaps the resolver's cert; in-flight TLS sessions keep
  their negotiated keys (zero drop), only new handshakes pick up the new cert.
  The outbound peer channels are rebuilt from the refreshed files
  (`PeerPool::rebuild_all`), zero-drop in the CA-overlap window. **Polling, not
  inotify**, is deliberate: Kubernetes secret rotation swaps a `..data` symlink
  atomically — a mode inotify watchers routinely miss.

### 2. Inter-node HTTP: encryption + per-node identity (`tidal-server`)

The HTTP plane (forwards, broadcasts, scatter-gather, status, seed-join) was
plaintext `http://` with only the shared bearer relayed. m11p7 adds both halves:

- **Encryption (opt-in via the same `grpc_tls` material).** The axum listener is
  served over a `TlsListener` (a `tokio-rustls` acceptor reusing tidal-net's
  hot-swappable resolver — one rotation covers both planes); forwards/broadcasts/
  scatter/status/seed-join dial `https://` and the `reqwest` clients trust the
  cluster CA. A process is exactly one node with one TLS posture, so the scheme
  decision is a single process-global (`forward::set_inter_node_https`) set at
  node construction — no threading `https` through every `peer_url` call site or
  the membership view. Server-auth only (no client-cert requirement) because the
  same listener also serves external bearer clients.
- **Per-node identity via signed internal tokens.** A forwarding/broadcasting
  node mints an `x-tidal-node-token` naming itself + an expiry, MAC'd with a
  shared **cluster key** (keyed BLAKE3 — no new crypto dependency; a foreign pod
  without the key cannot forge one). The receiver verifies the MAC in constant
  time. This gives inter-node calls a VERIFIABLE node identity (audit attribution
  + defense-in-depth beyond the shared bearer).
- **The `x-tidal-internal` marker stays a hint, not a bypass — now enforced.**
  When a cluster key is configured, a request that sets the marker WITHOUT a
  valid node token is rejected (403): the marker is honored only from a verified
  sibling. When no cluster key is configured the marker keeps its pre-m11p7
  hint-only behavior (backward compatible). The bearer middleware still runs
  first, so the marker is never reached before authentication.
- **Bearer + cluster key rotate without restart.** Both live behind `ArcSwap` in
  a reloadable `ClusterCreds`, read PER REQUEST and re-read from their files
  (`TIDAL_API_KEY_FILE` / `TIDAL_CLUSTER_KEY_FILE`) by the rotation poller.

### 3. Admin-verb audit log (`tidal-server`)

Every admin verb (`promote` / `partition` / `heal` / conf-change=`join`,
`member_remove` / `reseed`) emits one structured audit record carrying the
**principal** (a verified sibling node, or an external operator), the **term**,
the **target**, and the **outcome** (`applied (status)` / `rejected (status)` /
`error: …`). Records go to a dedicated `tidal_audit` tracing target (always) and
an append-only JSONL file when `TIDAL_AUDIT_LOG` is set. Emitted ONLY on the
operator-originated leg (`x-tidal-internal` absent), so a follower that forwards
the verb to the leader audits the operator request ONCE — the leader's marked
re-apply does not double-audit, and the record carries the operator's principal,
not the forwarding node's. At-rest encryption of the JSONL file is delegated to
the volume (documented in the runbook).

### 4. Per-principal HTTP rate limit (`tidal-server`)

The engine's token-bucket `RateLimiter` (re-exported from the crate root) is
wired to all three routers' middleware, keyed by the resolved principal. Verified
sibling NODES are EXEMPT (replication/forward traffic must never be throttled by
the external-client budget); external principals consume their bucket. A deny is
a 429 with a `Retry-After` header (and the millisecond hint + limit in the body).
Default unlimited (`TIDAL_RATE_LIMIT_RPS` unset ⇒ no behavior change); the bucket
set is bounded (one "external" bucket today).

### 5. Reference deployment + tooling (`k8s/cluster/`, `scripts/`)

cert-manager `Issuer` (self-signed CA) + a shared node `Certificate` (every
pod's stable DNS as a SAN); the StatefulSet mounts the cert Secret at
`/etc/tidaldb/tls` and the cluster key as `TIDAL_CLUSTER_KEY_FILE`; the topology
ConfigMap carries the `grpc_tls` block per region. `scripts/gen-cluster-certs.sh`
(openssl) provisions the same Secret shape for clusters without cert-manager and
for local runs. cert-manager renewal → Secret rewrite → kubelet `..data` swap →
the cert poller hot-swaps with zero restart.

## Designs that did NOT ship (recorded because the reasoning is load-bearing)

- **Hot-swapping the cert through tonic's API.** tonic 0.12 caches a fixed
  `Arc<ServerConfig>`; there is no resolver hook. "Rotation without restart"
  through tonic would force a Server rebuild + re-serve, which drops in-flight
  connections — exactly what the exit gate forbids. The custom rustls acceptor
  with a `ResolvesServerCert` is the ONLY zero-drop path, so it shipped.
- **Requiring HTTP client certs (full HTTP mTLS).** The HTTP listener also serves
  EXTERNAL bearer clients, which do not hold cluster certs. Requiring client
  certs would break them. Per-node identity on the HTTP plane is therefore the
  signed token (the spec's explicit alternative), not a client cert; gRPC uses
  client certs (it is purely inter-node).
- **inotify cert watching.** Loses k8s atomic `..data` symlink swaps. Content-hash
  polling is the robust signal.

### Idempotency / attribution decisions (in-phase)

- **Audit attribution = the operator, recorded once.** Auditing on every node a
  verb touches (operator → follower → leader) would multi-record one action and
  attribute it to the forwarding node. Auditing only the operator-originated leg
  (`!is_internal`) records it once with the operator's principal.
- **No new crypto dependency.** The node-token MAC reuses BLAKE3's keyed-hash
  mode (already a dependency); the cluster key is BLAKE3-derived from any
  operator secret string.

## Exit gate (from the roadmap)

- Reference deployment has zero plaintext inter-node links.
- Rotation under load drops zero requests.
- A foreign pod can neither ship segments nor call internal routes (negative tests).

## Status

- [x] gRPC mTLS-default + custom rustls acceptor + `DynamicCertResolver` + loud insecure WARN
- [x] Zero-drop cert rotation (`ServerCertReloader` poll + `PeerPool::rebuild_all`)
- [x] Inter-node HTTP TLS (`TlsListener` + `https` forwards + cluster-CA reqwest clients), opt-in via `grpc_tls`
- [x] Per-node signed internal tokens (`ClusterCreds`, keyed BLAKE3) + marker-pinning (marker-without-token → 403)
- [x] Bearer + cluster-key rotation without restart (`ClusterCreds` + rotation poller)
- [x] Admin-verb audit log (`tidal_audit` tracing target + `TIDAL_AUDIT_LOG` JSONL)
- [x] Per-principal HTTP rate limit (engine `RateLimiter`, node-exempt, 429 + Retry-After)
- [x] k8s reference: cert-manager Issuer/Certificate, mounts, `grpc_tls` topology, cluster-key Secret
- [x] `scripts/gen-cluster-certs.sh` (openssl, non-cert-manager + local)
- [x] Exit-gate tests + docs/runbook/monitoring/CHANGELOG/ROADMAP/memory

## Exit-gate evidence (local; real TLS, real handshakes)

| Gate | Verified by | Result |
|------|-------------|--------|
| Foreign pod cannot SHIP segments (gRPC) | `tidal-net/tests/mtls.rs` — `untrusted_client_cert_is_rejected`, `absent_client_cert_is_rejected` run through the NEW custom rustls acceptor + `WebPkiClientVerifier` | **rejected at the handshake** (4/4 mtls tests green, incl. the mTLS round-trip) |
| Foreign pod cannot CALL internal routes (HTTP) | `cluster_security.rs::http_tls_serves_ca_trusting_client_and_rejects_foreign` (real `TlsListener`); `cluster::security` marker-pinning unit test | a foreign-CA client + a plaintext probe **fail before any route**; a marked request without a node token is **403** |
| Zero plaintext inter-node links | the HTTP listener serves TLS + `peer_url` emits `https` when `grpc_tls` is set; the k8s reference enables it cluster-wide | HTTP-TLS serve verified end-to-end (`cluster_security.rs`); k8s manifests carry cert-manager + `grpc_tls` |
| **Rotation under load drops zero requests** | `cluster_security.rs::http_tls_cert_rotation_under_load_drops_zero` — 6 concurrent CA-pinned clients (handshake-per-request) while the cert hot-swaps 5× | **0 dropped of N>100 requests** |
| Per-node tokens + key rotation | `cluster::security` unit tests (7) | mint/verify, foreign-key reject, tamper reject, expiry, marker-pinning, key rotation — all green |

Verification: workspace `cargo clippy --all-targets -D warnings` clean, `cargo fmt
--check` clean, tidal-net (50 lib + mtls 4 + integration suites) green, tidal-server
(124 lib incl. 7 security unit + `cluster_security` 2 + `cluster_region`/`cluster_routes`/
`middleware` regression suites) green. The cert script is smoke-tested with real
openssl (chain verifies, SANs correct). Ref-A (Linux k3s) re-run of the full mTLS
cluster + rotation-under-load remains blocked on k3s access (same as p1/p2/p3).