6651c14adc
m11p7 — secure the cluster, all opt-in (pre-m11p7 byte-for-byte):
- gRPC replication mTLS by default via a custom tokio-rustls acceptor +
DynamicCertResolver; zero-drop content-hash cert rotation (k8s ..data swap,
no pod restart, no inotify)
- inter-node HTTP TLS sharing the same resolver (one rotation, both planes) +
per-node keyed-BLAKE3 signed x-tidal-node-token; marker-without-token -> 403
- admin audit log (operator-leg only) + per-principal rate limit (engine
RateLimiter; sibling nodes exempt)
- k8s cert-manager manifest (certs.yaml) + scripts/gen-cluster-certs.sh fallback;
secret.example.yaml gains TIDAL_CLUSTER_KEY (file-mounted, hot-rotatable)
- exit gate verified real: mtls.rs (gRPC foreign-pod), cluster_security.rs
(HTTP foreign + zero-drop rotation under load), 7 security unit tests
perf — instrument floor (sweep Wave 1):
- new tidal/benches/wal.rs + tidal-server/benches/scatter.rs
- p99->mean honesty relabel; sweep manifest at docs/reviews/perf-sweep-2026-06-13.md
- add @tidal-performance agent (Martin Thompson)
new: cluster/{audit,http_tls,security}.rs, tests/cluster_security.rs,
docs/planning/milestone-11/phase-7.md
167 lines
11 KiB
Markdown
167 lines
11 KiB
Markdown
# m11p7 — Security Hardening (COMPLETE — 2026-06-13)
|
||
|
||
Phase spec and exit gate: [docs/roadmap-to-cluster.md §4/m11p7](../../roadmap-to-cluster.md).
|
||
Closes the ROADMAP **network-trust** gap ("the cluster trusts the network").
|
||
Predecessors: p1–p2 (the stable replication surface this hardens), p3 (quorum),
|
||
p4 (election), p5 (membership), p6 L0–L2 (sharding data plane).
|
||
|
||
**Goal:** the cluster stops trusting the network. Inter-node links are encrypted
|
||
and authenticated; certs and bearer keys rotate without a restart; admin verbs
|
||
are audited with the principal who issued them; external load is rate-limited
|
||
per principal. The exit gate: a reference deployment has **zero plaintext
|
||
inter-node links**, **rotation under load drops zero requests**, and a **foreign
|
||
pod on the cluster network can neither ship segments nor call internal routes**.
|
||
|
||
## Design (as adopted)
|
||
|
||
### 1. gRPC mTLS by default + zero-drop cert rotation (`tidal-net`)
|
||
|
||
mTLS already worked when `grpc_tls` was configured, but the default was
|
||
plaintext and the cert was fixed for the server's life (tonic 0.12 caches one
|
||
`Arc<rustls::ServerConfig>` from a static identity and exposes no resolver hook).
|
||
m11p7 makes TLS the intended posture and the cert hot-swappable:
|
||
|
||
- **The server is served over a custom `tokio-rustls` acceptor** (not tonic's
|
||
`.tls_config()`), fed a rustls `ServerConfig` whose identity is a
|
||
`DynamicCertResolver` — an `ArcSwap<CertifiedKey>`. mTLS is preserved EXACTLY:
|
||
a `WebPkiClientVerifier` over the cluster CA roots (NOT `allow_unauthenticated`)
|
||
+ ALPN `h2`. The accept loop runs each TLS handshake in its OWN task and
|
||
forwards only successfully-handshaken streams into tonic, so a foreign pod —
|
||
no client cert, a foreign-CA cert, or a plaintext probe — fails the handshake
|
||
and **never reaches an RPC** (and never head-of-line-blocks the next connect).
|
||
- **Plaintext is an explicit `insecure: true` with a loud startup WARN** on both
|
||
the server (`start_server`) and the client (`PeerPool::new`).
|
||
- **Rotation without restart**: a content-hash polling reloader
|
||
(`ServerCertReloader`, `rotation_poll_interval`, default 30s) re-reads the cert
|
||
files and atomically swaps the resolver's cert; in-flight TLS sessions keep
|
||
their negotiated keys (zero drop), only new handshakes pick up the new cert.
|
||
The outbound peer channels are rebuilt from the refreshed files
|
||
(`PeerPool::rebuild_all`), zero-drop in the CA-overlap window. **Polling, not
|
||
inotify**, is deliberate: Kubernetes secret rotation swaps a `..data` symlink
|
||
atomically — a mode inotify watchers routinely miss.
|
||
|
||
### 2. Inter-node HTTP: encryption + per-node identity (`tidal-server`)
|
||
|
||
The HTTP plane (forwards, broadcasts, scatter-gather, status, seed-join) was
|
||
plaintext `http://` with only the shared bearer relayed. m11p7 adds both halves:
|
||
|
||
- **Encryption (opt-in via the same `grpc_tls` material).** The axum listener is
|
||
served over a `TlsListener` (a `tokio-rustls` acceptor reusing tidal-net's
|
||
hot-swappable resolver — one rotation covers both planes); forwards/broadcasts/
|
||
scatter/status/seed-join dial `https://` and the `reqwest` clients trust the
|
||
cluster CA. A process is exactly one node with one TLS posture, so the scheme
|
||
decision is a single process-global (`forward::set_inter_node_https`) set at
|
||
node construction — no threading `https` through every `peer_url` call site or
|
||
the membership view. Server-auth only (no client-cert requirement) because the
|
||
same listener also serves external bearer clients.
|
||
- **Per-node identity via signed internal tokens.** A forwarding/broadcasting
|
||
node mints an `x-tidal-node-token` naming itself + an expiry, MAC'd with a
|
||
shared **cluster key** (keyed BLAKE3 — no new crypto dependency; a foreign pod
|
||
without the key cannot forge one). The receiver verifies the MAC in constant
|
||
time. This gives inter-node calls a VERIFIABLE node identity (audit attribution
|
||
+ defense-in-depth beyond the shared bearer).
|
||
- **The `x-tidal-internal` marker stays a hint, not a bypass — now enforced.**
|
||
When a cluster key is configured, a request that sets the marker WITHOUT a
|
||
valid node token is rejected (403): the marker is honored only from a verified
|
||
sibling. When no cluster key is configured the marker keeps its pre-m11p7
|
||
hint-only behavior (backward compatible). The bearer middleware still runs
|
||
first, so the marker is never reached before authentication.
|
||
- **Bearer + cluster key rotate without restart.** Both live behind `ArcSwap` in
|
||
a reloadable `ClusterCreds`, read PER REQUEST and re-read from their files
|
||
(`TIDAL_API_KEY_FILE` / `TIDAL_CLUSTER_KEY_FILE`) by the rotation poller.
|
||
|
||
### 3. Admin-verb audit log (`tidal-server`)
|
||
|
||
Every admin verb (`promote` / `partition` / `heal` / conf-change=`join`,
|
||
`member_remove` / `reseed`) emits one structured audit record carrying the
|
||
**principal** (a verified sibling node, or an external operator), the **term**,
|
||
the **target**, and the **outcome** (`applied (status)` / `rejected (status)` /
|
||
`error: …`). Records go to a dedicated `tidal_audit` tracing target (always) and
|
||
an append-only JSONL file when `TIDAL_AUDIT_LOG` is set. Emitted ONLY on the
|
||
operator-originated leg (`x-tidal-internal` absent), so a follower that forwards
|
||
the verb to the leader audits the operator request ONCE — the leader's marked
|
||
re-apply does not double-audit, and the record carries the operator's principal,
|
||
not the forwarding node's. At-rest encryption of the JSONL file is delegated to
|
||
the volume (documented in the runbook).
|
||
|
||
### 4. Per-principal HTTP rate limit (`tidal-server`)
|
||
|
||
The engine's token-bucket `RateLimiter` (re-exported from the crate root) is
|
||
wired to all three routers' middleware, keyed by the resolved principal. Verified
|
||
sibling NODES are EXEMPT (replication/forward traffic must never be throttled by
|
||
the external-client budget); external principals consume their bucket. A deny is
|
||
a 429 with a `Retry-After` header (and the millisecond hint + limit in the body).
|
||
Default unlimited (`TIDAL_RATE_LIMIT_RPS` unset ⇒ no behavior change); the bucket
|
||
set is bounded (one "external" bucket today).
|
||
|
||
### 5. Reference deployment + tooling (`k8s/cluster/`, `scripts/`)
|
||
|
||
cert-manager `Issuer` (self-signed CA) + a shared node `Certificate` (every
|
||
pod's stable DNS as a SAN); the StatefulSet mounts the cert Secret at
|
||
`/etc/tidaldb/tls` and the cluster key as `TIDAL_CLUSTER_KEY_FILE`; the topology
|
||
ConfigMap carries the `grpc_tls` block per region. `scripts/gen-cluster-certs.sh`
|
||
(openssl) provisions the same Secret shape for clusters without cert-manager and
|
||
for local runs. cert-manager renewal → Secret rewrite → kubelet `..data` swap →
|
||
the cert poller hot-swaps with zero restart.
|
||
|
||
## Designs that did NOT ship (recorded because the reasoning is load-bearing)
|
||
|
||
- **Hot-swapping the cert through tonic's API.** tonic 0.12 caches a fixed
|
||
`Arc<ServerConfig>`; there is no resolver hook. "Rotation without restart"
|
||
through tonic would force a Server rebuild + re-serve, which drops in-flight
|
||
connections — exactly what the exit gate forbids. The custom rustls acceptor
|
||
with a `ResolvesServerCert` is the ONLY zero-drop path, so it shipped.
|
||
- **Requiring HTTP client certs (full HTTP mTLS).** The HTTP listener also serves
|
||
EXTERNAL bearer clients, which do not hold cluster certs. Requiring client
|
||
certs would break them. Per-node identity on the HTTP plane is therefore the
|
||
signed token (the spec's explicit alternative), not a client cert; gRPC uses
|
||
client certs (it is purely inter-node).
|
||
- **inotify cert watching.** Loses k8s atomic `..data` symlink swaps. Content-hash
|
||
polling is the robust signal.
|
||
|
||
### Idempotency / attribution decisions (in-phase)
|
||
|
||
- **Audit attribution = the operator, recorded once.** Auditing on every node a
|
||
verb touches (operator → follower → leader) would multi-record one action and
|
||
attribute it to the forwarding node. Auditing only the operator-originated leg
|
||
(`!is_internal`) records it once with the operator's principal.
|
||
- **No new crypto dependency.** The node-token MAC reuses BLAKE3's keyed-hash
|
||
mode (already a dependency); the cluster key is BLAKE3-derived from any
|
||
operator secret string.
|
||
|
||
## Exit gate (from the roadmap)
|
||
|
||
- Reference deployment has zero plaintext inter-node links.
|
||
- Rotation under load drops zero requests.
|
||
- A foreign pod can neither ship segments nor call internal routes (negative tests).
|
||
|
||
## Status
|
||
|
||
- [x] gRPC mTLS-default + custom rustls acceptor + `DynamicCertResolver` + loud insecure WARN
|
||
- [x] Zero-drop cert rotation (`ServerCertReloader` poll + `PeerPool::rebuild_all`)
|
||
- [x] Inter-node HTTP TLS (`TlsListener` + `https` forwards + cluster-CA reqwest clients), opt-in via `grpc_tls`
|
||
- [x] Per-node signed internal tokens (`ClusterCreds`, keyed BLAKE3) + marker-pinning (marker-without-token → 403)
|
||
- [x] Bearer + cluster-key rotation without restart (`ClusterCreds` + rotation poller)
|
||
- [x] Admin-verb audit log (`tidal_audit` tracing target + `TIDAL_AUDIT_LOG` JSONL)
|
||
- [x] Per-principal HTTP rate limit (engine `RateLimiter`, node-exempt, 429 + Retry-After)
|
||
- [x] k8s reference: cert-manager Issuer/Certificate, mounts, `grpc_tls` topology, cluster-key Secret
|
||
- [x] `scripts/gen-cluster-certs.sh` (openssl, non-cert-manager + local)
|
||
- [x] Exit-gate tests + docs/runbook/monitoring/CHANGELOG/ROADMAP/memory
|
||
|
||
## Exit-gate evidence (local; real TLS, real handshakes)
|
||
|
||
| Gate | Verified by | Result |
|
||
|------|-------------|--------|
|
||
| Foreign pod cannot SHIP segments (gRPC) | `tidal-net/tests/mtls.rs` — `untrusted_client_cert_is_rejected`, `absent_client_cert_is_rejected` run through the NEW custom rustls acceptor + `WebPkiClientVerifier` | **rejected at the handshake** (4/4 mtls tests green, incl. the mTLS round-trip) |
|
||
| Foreign pod cannot CALL internal routes (HTTP) | `cluster_security.rs::http_tls_serves_ca_trusting_client_and_rejects_foreign` (real `TlsListener`); `cluster::security` marker-pinning unit test | a foreign-CA client + a plaintext probe **fail before any route**; a marked request without a node token is **403** |
|
||
| Zero plaintext inter-node links | the HTTP listener serves TLS + `peer_url` emits `https` when `grpc_tls` is set; the k8s reference enables it cluster-wide | HTTP-TLS serve verified end-to-end (`cluster_security.rs`); k8s manifests carry cert-manager + `grpc_tls` |
|
||
| **Rotation under load drops zero requests** | `cluster_security.rs::http_tls_cert_rotation_under_load_drops_zero` — 6 concurrent CA-pinned clients (handshake-per-request) while the cert hot-swaps 5× | **0 dropped of N>100 requests** |
|
||
| Per-node tokens + key rotation | `cluster::security` unit tests (7) | mint/verify, foreign-key reject, tamper reject, expiry, marker-pinning, key rotation — all green |
|
||
|
||
Verification: workspace `cargo clippy --all-targets -D warnings` clean, `cargo fmt
|
||
--check` clean, tidal-net (50 lib + mtls 4 + integration suites) green, tidal-server
|
||
(124 lib incl. 7 security unit + `cluster_security` 2 + `cluster_region`/`cluster_routes`/
|
||
`middleware` regression suites) green. The cert script is smoke-tested with real
|
||
openssl (chain verifies, SANs correct). Ref-A (Linux k3s) re-run of the full mTLS
|
||
cluster + rotation-under-load remains blocked on k3s access (same as p1/p2/p3).
|