6651c14adc
m11p7 — secure the cluster, all opt-in (pre-m11p7 byte-for-byte):
- gRPC replication mTLS by default via a custom tokio-rustls acceptor +
DynamicCertResolver; zero-drop content-hash cert rotation (k8s ..data swap,
no pod restart, no inotify)
- inter-node HTTP TLS sharing the same resolver (one rotation, both planes) +
per-node keyed-BLAKE3 signed x-tidal-node-token; marker-without-token -> 403
- admin audit log (operator-leg only) + per-principal rate limit (engine
RateLimiter; sibling nodes exempt)
- k8s cert-manager manifest (certs.yaml) + scripts/gen-cluster-certs.sh fallback;
secret.example.yaml gains TIDAL_CLUSTER_KEY (file-mounted, hot-rotatable)
- exit gate verified real: mtls.rs (gRPC foreign-pod), cluster_security.rs
(HTTP foreign + zero-drop rotation under load), 7 security unit tests
perf — instrument floor (sweep Wave 1):
- new tidal/benches/wal.rs + tidal-server/benches/scatter.rs
- p99->mean honesty relabel; sweep manifest at docs/reviews/perf-sweep-2026-06-13.md
- add @tidal-performance agent (Martin Thompson)
new: cluster/{audit,http_tls,security}.rs, tests/cluster_security.rs,
docs/planning/milestone-11/phase-7.md
106 lines
4.7 KiB
Markdown
106 lines
4.7 KiB
Markdown
# Scale Benchmarks: 1M-Item Baselines
|
||
|
||
## Hardware
|
||
|
||
macOS Darwin 23.6.0 (Apple Silicon / x86-64 — see run date below).
|
||
|
||
**Run command:**
|
||
```bash
|
||
cargo bench --manifest-path tidal/Cargo.toml --bench scale
|
||
```
|
||
|
||
**Date:** 2026-02-23
|
||
|
||
## Dataset
|
||
|
||
| Parameter | Value |
|
||
|-----------|-------|
|
||
| Items | 1,000,000 |
|
||
| Creators | 10,000 (100 items/creator) |
|
||
| Categories | 20 |
|
||
| Embedding dim | 128 (not 1536 — reduced for bench RAM) |
|
||
| Signal coverage | 10% view, 5% like |
|
||
| Bench tool | Criterion (sample_size=10, 30s measurement, Flat mode) — **closed-loop, single-threaded; reports the mean, not the tail** |
|
||
|
||
## Measurement contract (read before trusting any number below)
|
||
|
||
> The `time:` figures Criterion reports are **single-threaded mean per-op cost
|
||
> under a closed loop** — the `[lower mean upper]` triple is a *confidence
|
||
> interval on the mean*, **not** a latency distribution. They are
|
||
> **regression tripwires**, not tail-SLO evidence.
|
||
>
|
||
> A `p99`/`p999`/tail SLO can only be honored by an **open-loop,
|
||
> coordinated-omission-corrected** measurement under real concurrency (the
|
||
> `tidal-stress` ramp). A closed-loop mean cannot observe the tail it hides:
|
||
> when the system stalls, a closed-loop harness simply *stops sending*, so the
|
||
> queue that would inflate p99 never forms. Reporting a mean under a `p99` label
|
||
> understates the tail by 10–100× under load.
|
||
>
|
||
> Therefore every row below is labelled **isolated per-op cost (mean)**. Tail
|
||
> SLOs are validated separately — see `tidal-stress` (open-loop) for the
|
||
> authoritative p99/p999 under the production workload.
|
||
|
||
## Acceptance Criteria (isolated per-op cost — regression tripwires, NOT tail SLOs)
|
||
|
||
| Benchmark | Tail SLO (validated open-loop) | Isolated per-op cost (mean, closed-loop) | Tripwire |
|
||
|-----------|--------------------------------|------------------------------------------|----------|
|
||
| RETRIEVE | < 50ms p99 | **152 µs** (for_you) | ✅ well under |
|
||
| SEARCH | < 100ms p99 | **28.9 ms** (text_only) | ✅ well under |
|
||
| Signal write | < 100µs p99 | **82 ns** (rotating 1K) | ✅ well under |
|
||
|
||
The mean per-op costs sit far under the tail targets — a necessary but **not
|
||
sufficient** condition for the p99 SLO. "Well under" means the *mean* clears the
|
||
target with headroom; the p99 itself is signed off only by the open-loop
|
||
`tidal-stress` run, never by this table.
|
||
|
||
## Benchmark Results
|
||
|
||
### RETRIEVE (1M items)
|
||
|
||
```
|
||
retrieve_1m/for_you time: [151.88 µs 152.13 µs 152.40 µs]
|
||
retrieve_1m/trending time: [127.96 µs 128.25 µs 128.52 µs]
|
||
retrieve_1m/new_filtered time: [ 7.5636 µs 7.5855 µs 7.6058 µs]
|
||
```
|
||
|
||
**All RETRIEVE queries < 200µs.** The 50ms target is beaten by 3 orders of magnitude.
|
||
|
||
- `for_you`: signal-scored ranking over full 1M-item universe — 152µs
|
||
- `trending`: windowed view count ranking — 128µs
|
||
- `new_filtered`: category filter at ~5% selectivity — 7.6µs (bitmap pre-filter eliminates 95% of candidates)
|
||
|
||
### SEARCH (1M items)
|
||
|
||
```
|
||
search_1m/text_only time: [28.844 ms 28.934 ms 29.021 ms]
|
||
search_1m/text_filtered time: [ 1.8972 ms 1.9104 ms 1.9220 ms]
|
||
```
|
||
|
||
**Both SEARCH queries < 30ms.** The 100ms target is beaten by 3-50×.
|
||
|
||
- `text_only`: BM25 over 1M documents — 28.9ms (most expensive path; dominated by Tantivy posting list traversal)
|
||
- `text_filtered`: BM25 with category filter reduces candidate set — 1.9ms
|
||
|
||
### Signal Write (1M-item DB, rotating 1K entities)
|
||
|
||
```
|
||
signal_write_1m/write_rotating_1k_entities time: [82.033 ns 82.286 ns 82.535 ns]
|
||
```
|
||
|
||
**82 ns per write.** The 100µs target is beaten by 1,200×. DashMap hot-path write amortises to sub-100ns across 1K rotating entity IDs.
|
||
|
||
## Setup Notes
|
||
|
||
The `LazyLock<TidalDb>` pattern ensures the 1M-item database is built exactly once per bench run. Build time ~30s on the reference hardware above. The text syncer waits 3s after ingestion.
|
||
|
||
## Database Build Time
|
||
|
||
Approximately **30 seconds** on reference hardware (observed from `[scale bench] Database ready` log line).
|
||
|
||
## Analysis
|
||
|
||
tidalDB's **isolated per-op mean cost** sits well within all three
|
||
acceptance-criteria targets at 1M items. The dominant cost is SEARCH text_only at ~29ms — driven by Tantivy posting list traversal across 1M documents. The LogMergePolicy tuning (< 20 segments at steady state) keeps this below the 100ms target with headroom. **The p99 tail SLOs themselves are signed off by the open-loop `tidal-stress` ramp, not by these closed-loop means** (see the measurement contract above).
|
||
|
||
Signal writes at 82ns confirm the DashMap hot-path is not a bottleneck at this scale. The 5M-entry LRU trimming threshold (DEFAULT_MAX_SIGNAL_ENTRIES) provides ample headroom for the 100K-item signal coverage in this benchmark (~200K entries = ~218MB).
|