tidaldb/docs/planning/milestone-11/phase-7.md

m11p7 — Security Hardening (COMPLETE — 2026-06-13)

Phase spec and exit gate: docs/roadmap-to-cluster.md §4/m11p7. Closes the ROADMAP network-trust gap ("the cluster trusts the network"). Predecessors: p1–p2 (the stable replication surface this hardens), p3 (quorum), p4 (election), p5 (membership), p6 L0–L2 (sharding data plane).

Goal: the cluster stops trusting the network. Inter-node links are encrypted and authenticated; certs and bearer keys rotate without a restart; admin verbs are audited with the principal who issued them; external load is rate-limited per principal. The exit gate: a reference deployment has zero plaintext inter-node links, rotation under load drops zero requests, and a foreign pod on the cluster network can neither ship segments nor call internal routes.

Design (as adopted)

1. gRPC mTLS by default + zero-drop cert rotation (tidal-net)

mTLS already worked when grpc_tls was configured, but the default was plaintext and the cert was fixed for the server's life (tonic 0.12 caches one Arc<rustls::ServerConfig> from a static identity and exposes no resolver hook). m11p7 makes TLS the intended posture and the cert hot-swappable:

• The server is served over a custom tokio-rustls acceptor (not tonic's .tls_config()), fed a rustls ServerConfig whose identity is a DynamicCertResolver — an ArcSwap<CertifiedKey>. mTLS is preserved EXACTLY: a WebPkiClientVerifier over the cluster CA roots (NOT allow_unauthenticated)
  • ALPN h2. The accept loop runs each TLS handshake in its OWN task and forwards only successfully-handshaken streams into tonic, so a foreign pod — no client cert, a foreign-CA cert, or a plaintext probe — fails the handshake and never reaches an RPC (and never head-of-line-blocks the next connect).
• Plaintext is an explicit insecure: true with a loud startup WARN on both the server (start_server) and the client (PeerPool::new).
• Rotation without restart: a content-hash polling reloader (ServerCertReloader, rotation_poll_interval, default 30s) re-reads the cert files and atomically swaps the resolver's cert; in-flight TLS sessions keep their negotiated keys (zero drop), only new handshakes pick up the new cert. The outbound peer channels are rebuilt from the refreshed files (PeerPool::rebuild_all), zero-drop in the CA-overlap window. Polling, not inotify, is deliberate: Kubernetes secret rotation swaps a ..data symlink atomically — a mode inotify watchers routinely miss.

2. Inter-node HTTP: encryption + per-node identity (tidal-server)

The HTTP plane (forwards, broadcasts, scatter-gather, status, seed-join) was plaintext http:// with only the shared bearer relayed. m11p7 adds both halves:

• Encryption (opt-in via the same grpc_tls material). The axum listener is served over a TlsListener (a tokio-rustls acceptor reusing tidal-net's hot-swappable resolver — one rotation covers both planes); forwards/broadcasts/ scatter/status/seed-join dial https:// and the reqwest clients trust the cluster CA. A process is exactly one node with one TLS posture, so the scheme decision is a single process-global (forward::set_inter_node_https) set at node construction — no threading https through every peer_url call site or the membership view. Server-auth only (no client-cert requirement) because the same listener also serves external bearer clients.
• Per-node identity via signed internal tokens. A forwarding/broadcasting node mints an x-tidal-node-token naming itself + an expiry, MAC'd with a shared cluster key (keyed BLAKE3 — no new crypto dependency; a foreign pod without the key cannot forge one). The receiver verifies the MAC in constant time. This gives inter-node calls a VERIFIABLE node identity (audit attribution
  • defense-in-depth beyond the shared bearer).
• The x-tidal-internal marker stays a hint, not a bypass — now enforced. When a cluster key is configured, a request that sets the marker WITHOUT a valid node token is rejected (403): the marker is honored only from a verified sibling. When no cluster key is configured the marker keeps its pre-m11p7 hint-only behavior (backward compatible). The bearer middleware still runs first, so the marker is never reached before authentication.
• Bearer + cluster key rotate without restart. Both live behind ArcSwap in a reloadable ClusterCreds, read PER REQUEST and re-read from their files (TIDAL_API_KEY_FILE / TIDAL_CLUSTER_KEY_FILE) by the rotation poller.

3. Admin-verb audit log (tidal-server)

Every admin verb (promote / partition / heal / conf-change=join, member_remove / reseed) emits one structured audit record carrying the principal (a verified sibling node, or an external operator), the term, the target, and the outcome (applied (status) / rejected (status) / error: …). Records go to a dedicated tidal_audit tracing target (always) and an append-only JSONL file when TIDAL_AUDIT_LOG is set. Emitted ONLY on the operator-originated leg (x-tidal-internal absent), so a follower that forwards the verb to the leader audits the operator request ONCE — the leader's marked re-apply does not double-audit, and the record carries the operator's principal, not the forwarding node's. At-rest encryption of the JSONL file is delegated to the volume (documented in the runbook).

4. Per-principal HTTP rate limit (tidal-server)

The engine's token-bucket RateLimiter (re-exported from the crate root) is wired to all three routers' middleware, keyed by the resolved principal. Verified sibling NODES are EXEMPT (replication/forward traffic must never be throttled by the external-client budget); external principals consume their bucket. A deny is a 429 with a Retry-After header (and the millisecond hint + limit in the body). Default unlimited (TIDAL_RATE_LIMIT_RPS unset ⇒ no behavior change); the bucket set is bounded (one "external" bucket today).

5. Reference deployment + tooling (k8s/cluster/, scripts/)

cert-manager Issuer (self-signed CA) + a shared node Certificate (every pod's stable DNS as a SAN); the StatefulSet mounts the cert Secret at /etc/tidaldb/tls and the cluster key as TIDAL_CLUSTER_KEY_FILE; the topology ConfigMap carries the grpc_tls block per region. scripts/gen-cluster-certs.sh (openssl) provisions the same Secret shape for clusters without cert-manager and for local runs. cert-manager renewal → Secret rewrite → kubelet ..data swap → the cert poller hot-swaps with zero restart.

Designs that did NOT ship (recorded because the reasoning is load-bearing)

• Hot-swapping the cert through tonic's API. tonic 0.12 caches a fixed Arc<ServerConfig>; there is no resolver hook. "Rotation without restart" through tonic would force a Server rebuild + re-serve, which drops in-flight connections — exactly what the exit gate forbids. The custom rustls acceptor with a ResolvesServerCert is the ONLY zero-drop path, so it shipped.
• Requiring HTTP client certs (full HTTP mTLS). The HTTP listener also serves EXTERNAL bearer clients, which do not hold cluster certs. Requiring client certs would break them. Per-node identity on the HTTP plane is therefore the signed token (the spec's explicit alternative), not a client cert; gRPC uses client certs (it is purely inter-node).
• inotify cert watching. Loses k8s atomic ..data symlink swaps. Content-hash polling is the robust signal.

Idempotency / attribution decisions (in-phase)

• Audit attribution = the operator, recorded once. Auditing on every node a verb touches (operator → follower → leader) would multi-record one action and attribute it to the forwarding node. Auditing only the operator-originated leg (!is_internal) records it once with the operator's principal.
• No new crypto dependency. The node-token MAC reuses BLAKE3's keyed-hash mode (already a dependency); the cluster key is BLAKE3-derived from any operator secret string.

Exit gate (from the roadmap)

• Reference deployment has zero plaintext inter-node links.
• Rotation under load drops zero requests.
• A foreign pod can neither ship segments nor call internal routes (negative tests).

Status

• gRPC mTLS-default + custom rustls acceptor + DynamicCertResolver + loud insecure WARN
• Zero-drop cert rotation (ServerCertReloader poll + PeerPool::rebuild_all)
• Inter-node HTTP TLS (TlsListener + https forwards + cluster-CA reqwest clients), opt-in via grpc_tls
• Per-node signed internal tokens (ClusterCreds, keyed BLAKE3) + marker-pinning (marker-without-token → 403)
• Bearer + cluster-key rotation without restart (ClusterCreds + rotation poller)
• Admin-verb audit log (tidal_audit tracing target + TIDAL_AUDIT_LOG JSONL)
• Per-principal HTTP rate limit (engine RateLimiter, node-exempt, 429 + Retry-After)
• k8s reference: cert-manager Issuer/Certificate, mounts, grpc_tls topology, cluster-key Secret
• scripts/gen-cluster-certs.sh (openssl, non-cert-manager + local)
• Exit-gate tests + docs/runbook/monitoring/CHANGELOG/ROADMAP/memory

Exit-gate evidence (local; real TLS, real handshakes)

Gate	Verified by	Result
Foreign pod cannot SHIP segments (gRPC)	tidal-net/tests/mtls.rs — untrusted_client_cert_is_rejected, absent_client_cert_is_rejected run through the NEW custom rustls acceptor + WebPkiClientVerifier	rejected at the handshake (4/4 mtls tests green, incl. the mTLS round-trip)
Foreign pod cannot CALL internal routes (HTTP)	cluster_security.rs::http_tls_serves_ca_trusting_client_and_rejects_foreign (real TlsListener); cluster::security marker-pinning unit test	a foreign-CA client + a plaintext probe fail before any route; a marked request without a node token is 403
Zero plaintext inter-node links	the HTTP listener serves TLS + peer_url emits https when grpc_tls is set; the k8s reference enables it cluster-wide	HTTP-TLS serve verified end-to-end (cluster_security.rs); k8s manifests carry cert-manager + grpc_tls
Rotation under load drops zero requests	cluster_security.rs::http_tls_cert_rotation_under_load_drops_zero — 6 concurrent CA-pinned clients (handshake-per-request) while the cert hot-swaps 5×	0 dropped of N>100 requests
Per-node tokens + key rotation	cluster::security unit tests (7)	mint/verify, foreign-key reject, tamper reject, expiry, marker-pinning, key rotation — all green

Verification: workspace cargo clippy --all-targets -D warnings clean, cargo fmt --check clean, tidal-net (50 lib + mtls 4 + integration suites) green, tidal-server (124 lib incl. 7 security unit + cluster_security 2 + cluster_region/cluster_routes/ middleware regression suites) green. The cert script is smoke-tested with real openssl (chain verifies, SANs correct). Ref-A (Linux k3s) re-run of the full mTLS cluster + rotation-under-load remains blocked on k3s access (same as p1/p2/p3).