tidaldb/docs/profiling/m12p4-t5-sharded-throughput.md

# m12p4 / T5 — Sharded ingestion at scale (real-cluster run, 2026-06-14)

**What ran.** A real 3-shard-group × RF=3 tidalDB cluster on a local `kind`
cluster (single node, 24 vCPU / 50 GB), full mTLS (cert-manager-issued certs),
the m12p4 server image, driven by **two** in-cluster `tidal-stress` generator
pods (the "second stress generator" m12p4 calls for). Not a simulation — real
binaries, real WAL + quorum, real gRPC replication, real inter-node mTLS HTTP.

This is the honest record of what the run **proved**, what it **found**, and why
the headline gate (`≥2.5× scaling AND ≥5000 quorum writes/s`) remains
**Ref-A/k3s-pending** — now with data and a sharper reason, not just a hardware
caveat.

## What the run proved (real, reproducible)

- **The 3-group topology deploys and serves on real k8s.** `shards:` enabled
  (`k8s/cluster/topology-configmap.yaml`), every pod binds a derived gRPC port
  per group (9601/9602/9603), `/cluster/status` shows **balanced leaders** —
  shard 0→tidaldb-0, 1→tidaldb-1, 2→tidaldb-2, all term 0. Fresh parallel boot
  converges in **~30–48 s**. (Reproduce: `k8s/cluster-local-kind/` overlay.)
- **Cross-group quorum writes work end-to-end** — a write for an entity owned by
  another group's leader is hash-routed + forwarded over the inter-node mTLS
  plane and quorum-acked, `x-tidal-seq` verdict header riding back.
- **Graceful behavior under overload** — at offered loads past the engine knee,
  the cluster sheds with 503 backpressure / request timeouts; **0 pod crashes,
  0 acked loss** (a timed-out write is never acked, so it is not acked loss).

## The blocker this run uncovered and fixed (HTTP/2 forward relay)

The FIRST cross-group write returned curl **exit 92 (HTTP/2 stream error)** /
HTTP 000; the identical write with `--http1.1` returned 204. Root cause: the
inter-node/client plane serves h2 (ALPN `h2`+`http/1.1`), `/signals`//`/embeddings`
return **204 No Content**, and the forward relay (`forward::relay_forwarded`)
attached a **synthesized JSON body to a 204** (`forward_json_with_headers` fills
an empty peer body). HTTP/2 RST_STREAMs a 204 that carries a DATA frame; HTTP/1.1
tolerated it, which masked the bug until the sharded forward ran over the **real
h2 plane** (never exercised before — was itself Ref-A/k3s-pending). Fixed in
`relay_forwarded` (skip the body for 1xx/204/304 via `status_forbids_body`) +
3 unit tests; the one direct relay site now routes through the helper too. After
the fix, all cross-group writes return 204 over h2.

This is a genuine correctness fix that only a real-h2-plane run surfaces —
in-process / plaintext multiproc e2e (reqwest→plaintext = h1.1) cannot catch it.

## Throughput numbers (measured)

Open-loop, coordinated-omission-corrected, `--ack quorum`, `--mix writes`,
`--embedding-dim 1536` (schema width). "Sustained" = highest ramp stage at
≤1% error; aggregate = sum of the two generator pods.

| Config (per-pod CPU) | Sustained quorum writes/s (≤1% err) | Behavior past the knee |
|----------------------|-------------------------------------|------------------------|
| **1 group, RF=3** (2) | **~5,500** (2×~2,775) | latency rises to p99 ~280 ms, 503s — single leader saturates |
| **3 groups, RF=3** (2) | **~4,000–5,000** | collapses earlier (stage 2: 78% shed, 40 s timeouts) |

Both generators are **identical** and cap at **~2,300–3,900 rps each** — the knee
is the generator's in-flight×latency limit (`client-shed`), not engine errors, at
the lower stages. Two generators offer **at most ~7,800 rps aggregate**.

## The finding: full-placement sharding does not scale writes at fixed per-pod CPU

The roadmap premise — "3 groups → ~3× writes" — assumes the bottleneck is the
**single-leader funnel** (one leader serializing WAL append + quorum), which
sharding parallelizes across S leaders. That holds **only when per-pod CPU is not
the binding constraint.**

With **full placement** (every pod replicates EVERY group — the shape chosen for
failover simplicity), each pod runs **all S groups**: leader for 1 + follower for
S−1. So a pod's total replication work scales with **S**, while its CPU is fixed.
At 2 vCPU/pod the 3-group pods are doing 3 groups' WAL/fsync/ship/apply in the
same budget as the 1-group pods do 1 group's — so the 3-group cluster saturates
at a **comparable or lower** aggregate than single-group, not 3× higher. Measured:
3-group ≈ single-group (~4–5k), then 3-group collapses *first* under overload.

**Implication.** Full-placement sharding buys **failover** (any pod loss keeps
every group's quorum) and **leader-funnel relief**, but it does **not** raise
aggregate write throughput at a fixed cluster CPU budget — the per-pod replication
overhead grows with S. Real write *scaling* needs **partitioned placement** (each
pod hosts a SUBSET of groups, so adding pods adds both groups and CPU) and/or
per-pod CPU headroom so the leader funnel — not per-pod CPU — is the bottleneck.

## Why the gate stays Ref-A/k3s-pending (now with data)

To demonstrate `≥2.5× scaling AND ≥5,000/s` you need ALL of:
1. **Per-pod CPU headroom** so the single-leader funnel (not per-pod CPU) binds —
   i.e. enough cores that a pod running S groups isn't CPU-saturated.
2. **Generators that can offer ≥14k rps** — 2.5× over the ~5.5k single-group
   ceiling. Two TLS generators cap at ~7.8k aggregate; that's ~1.4× at most.
3. **Or partitioned placement** across more nodes (the architecturally correct
   write-scaling shape), which a single kind node cannot host meaningfully.

A single shared 24-vCPU laptop cannot satisfy (1)+(2) simultaneously — the engine
and the TLS generators contend for the same cores. This is precisely the
hardware dependency the roadmap names; the run confirms it with numbers and adds
the full-placement insight above.

## Artifacts (committed)

- `k8s/cluster/topology-configmap.yaml` — 3-group `shards:` block enabled.
- `k8s/cluster/statefulset.yaml`, `service-peers.yaml` — derived per-shard ports.
- `k8s/cluster-local-kind/` — local-kind overlay (image + storageclass override).
- `tidal-stress/k8s/stress-job-t5.yaml` — the 2-generator T5 Job (production shape).
- `tidal-server/src/cluster/forward.rs` — the h2 204-relay fix + tests.

## Recommended follow-up (for the visionary)

The T5 gate as written assumes full-placement scales writes; the measured reality
says it scales failover, not write throughput, at fixed CPU. Re-scope T5 to either
(a) **partitioned placement** on multi-node k3s/Ref-A (the real write-scaling
proof), or (b) restate the gate as "leader-funnel relief at CPU headroom" and size
the pods accordingly. Either way the generator fleet must be ≥4 pods to offer the
load a scaled cluster can absorb.