tidaldb/docs/profiling/m12p4-t5-sharded-throughput.md

m12p4 / T5 — Sharded ingestion at scale (real-cluster run, 2026-06-14)

What ran. A real 3-shard-group × RF=3 tidalDB cluster on a local kind cluster (single node, 24 vCPU / 50 GB), full mTLS (cert-manager-issued certs), the m12p4 server image, driven by two in-cluster tidal-stress generator pods (the "second stress generator" m12p4 calls for). Not a simulation — real binaries, real WAL + quorum, real gRPC replication, real inter-node mTLS HTTP.

This is the honest record of what the run proved, what it found, and why the headline gate (≥2.5× scaling AND ≥5000 quorum writes/s) remains Ref-A/k3s-pending — now with data and a sharper reason, not just a hardware caveat.

What the run proved (real, reproducible)

• The 3-group topology deploys and serves on real k8s. shards: enabled (k8s/cluster/topology-configmap.yaml), every pod binds a derived gRPC port per group (9601/9602/9603), /cluster/status shows balanced leaders — shard 0→tidaldb-0, 1→tidaldb-1, 2→tidaldb-2, all term 0. Fresh parallel boot converges in ~30–48 s. (Reproduce: k8s/cluster-local-kind/ overlay.)
• Cross-group quorum writes work end-to-end — a write for an entity owned by another group's leader is hash-routed + forwarded over the inter-node mTLS plane and quorum-acked, x-tidal-seq verdict header riding back.
• Graceful behavior under overload — at offered loads past the engine knee, the cluster sheds with 503 backpressure / request timeouts; 0 pod crashes, 0 acked loss (a timed-out write is never acked, so it is not acked loss).

The blocker this run uncovered and fixed (HTTP/2 forward relay)

The FIRST cross-group write returned curl exit 92 (HTTP/2 stream error) / HTTP 000; the identical write with --http1.1 returned 204. Root cause: the inter-node/client plane serves h2 (ALPN h2+http/1.1), /signals///embeddings return 204 No Content, and the forward relay (forward::relay_forwarded) attached a synthesized JSON body to a 204 (forward_json_with_headers fills an empty peer body). HTTP/2 RST_STREAMs a 204 that carries a DATA frame; HTTP/1.1 tolerated it, which masked the bug until the sharded forward ran over the real h2 plane (never exercised before — was itself Ref-A/k3s-pending). Fixed in relay_forwarded (skip the body for 1xx/204/304 via status_forbids_body) + 3 unit tests; the one direct relay site now routes through the helper too. After the fix, all cross-group writes return 204 over h2.

This is a genuine correctness fix that only a real-h2-plane run surfaces — in-process / plaintext multiproc e2e (reqwest→plaintext = h1.1) cannot catch it.

Throughput numbers (measured)

Open-loop, coordinated-omission-corrected, --ack quorum, --mix writes, --embedding-dim 1536 (schema width). "Sustained" = highest ramp stage at ≤1% error; aggregate = sum of the two generator pods.

Config (per-pod CPU)	Sustained quorum writes/s (≤1% err)	Behavior past the knee
1 group, RF=3 (2)	~5,500 (2×~2,775)	latency rises to p99 ~280 ms, 503s — single leader saturates
3 groups, RF=3 (2)	~4,000–5,000	collapses earlier (stage 2: 78% shed, 40 s timeouts)

Both generators are identical and cap at ~2,300–3,900 rps each — the knee is the generator's in-flight×latency limit (client-shed), not engine errors, at the lower stages. Two generators offer at most ~7,800 rps aggregate.

The finding: full-placement sharding does not scale writes at fixed per-pod CPU

The roadmap premise — "3 groups → ~3× writes" — assumes the bottleneck is the single-leader funnel (one leader serializing WAL append + quorum), which sharding parallelizes across S leaders. That holds only when per-pod CPU is not the binding constraint.

With full placement (every pod replicates EVERY group — the shape chosen for failover simplicity), each pod runs all S groups: leader for 1 + follower for S−1. So a pod's total replication work scales with S, while its CPU is fixed. At 2 vCPU/pod the 3-group pods are doing 3 groups' WAL/fsync/ship/apply in the same budget as the 1-group pods do 1 group's — so the 3-group cluster saturates at a comparable or lower aggregate than single-group, not 3× higher. Measured: 3-group ≈ single-group (~4–5k), then 3-group collapses first under overload.

Implication. Full-placement sharding buys failover (any pod loss keeps every group's quorum) and leader-funnel relief, but it does not raise aggregate write throughput at a fixed cluster CPU budget — the per-pod replication overhead grows with S. Real write scaling needs partitioned placement (each pod hosts a SUBSET of groups, so adding pods adds both groups and CPU) and/or per-pod CPU headroom so the leader funnel — not per-pod CPU — is the bottleneck.

Why the gate stays Ref-A/k3s-pending (now with data)

To demonstrate ≥2.5× scaling AND ≥5,000/s you need ALL of:

1. Per-pod CPU headroom so the single-leader funnel (not per-pod CPU) binds — i.e. enough cores that a pod running S groups isn't CPU-saturated.
2. Generators that can offer ≥14k rps — 2.5× over the ~5.5k single-group ceiling. Two TLS generators cap at ~7.8k aggregate; that's ~1.4× at most.
3. Or partitioned placement across more nodes (the architecturally correct write-scaling shape), which a single kind node cannot host meaningfully.

A single shared 24-vCPU laptop cannot satisfy (1)+(2) simultaneously — the engine and the TLS generators contend for the same cores. This is precisely the hardware dependency the roadmap names; the run confirms it with numbers and adds the full-placement insight above.

Artifacts (committed)

• k8s/cluster/topology-configmap.yaml — 3-group shards: block enabled.
• k8s/cluster/statefulset.yaml, service-peers.yaml — derived per-shard ports.
• k8s/cluster-local-kind/ — local-kind overlay (image + storageclass override).
• tidal-stress/k8s/stress-job-t5.yaml — the 2-generator T5 Job (production shape).
• tidal-server/src/cluster/forward.rs — the h2 204-relay fix + tests.

Recommended follow-up (for the visionary)

The T5 gate as written assumes full-placement scales writes; the measured reality says it scales failover, not write throughput, at fixed CPU. Re-scope T5 to either (a) partitioned placement on multi-node k3s/Ref-A (the real write-scaling proof), or (b) restate the gate as "leader-funnel relief at CPU headroom" and size the pods accordingly. Either way the generator fleet must be ≥4 pods to offer the load a scaled cluster can absorb.