tidaldb/docs/ops/capacity-planning.md

# Capacity Planning

This document provides RAM, disk, and startup time estimates for tidalDB deployments. Use these tables to provision hardware before going to production.

All estimates assume a single-node deployment with default configuration (30-second checkpoint interval, f16 vector quantization, DashMap-based hot tier).

---

## RAM Capacity

tidalDB is an in-memory-first database. USearch HNSW indexes, the signal ledger hot tier, and Tantivy reader segments all reside in RAM during operation. There is no swap tolerance for USearch -- if the process is swapped, ANN query latency degrades from microseconds to seconds.

| Items | Embedding Dims | USearch RAM | Signal Ledger RAM (10 active signals) | Tantivy RAM | Analytic Total |
|------:|---------------:|------------:|--------------------------------------:|------------:|---------------:|
| 100K  | 128D           | ~31 MB      | ~1.09 GB                              | ~50 MB      | ~1.17 GB       |
| 100K  | 768D           | ~184 MB     | ~1.09 GB                              | ~50 MB      | ~1.32 GB       |
| 100K  | 1536D          | ~369 MB     | ~1.09 GB                              | ~50 MB      | ~1.51 GB       |
| 1M    | 128D           | ~307 MB     | ~5.44 GB (hot-tier cap)               | ~200 MB     | ~5.95 GB       |
| 1M    | 768D           | ~1.84 GB    | ~5.44 GB (hot-tier cap)               | ~200 MB     | ~7.48 GB       |
| 1M    | 1536D          | ~3.69 GB    | ~5.44 GB (hot-tier cap)               | ~200 MB     | ~9.33 GB       |
| 10M   | 128D           | ~3.07 GB    | ~5.44 GB (hot-tier cap)               | ~500 MB     | ~9.01 GB       |
| 10M   | 768D           | ~18.43 GB   | ~5.44 GB (hot-tier cap)               | ~500 MB     | ~24.37 GB      |
| 10M   | 1536D          | ~36.86 GB   | ~5.44 GB (hot-tier cap)               | ~500 MB     | ~42.80 GB      |

### Formulas

**USearch HNSW index:**

```
items * dims * 2 bytes (f16 quantization) * 1.2 (HNSW graph overhead)
```

The 20% graph overhead accounts for HNSW neighbor lists (M=16 default, two layers). Actual overhead varies with M and ef_construction parameters.

**Signal ledger hot tier:**

```
min(items * active_signal_types_per_item, DEFAULT_MAX_SIGNAL_ENTRIES)
  * ~1,088 bytes/entry
```

Each `(entity_id, signal_type_id)` entry in the DashMap holds the running decay
score, windowed counters (BucketedCounter with minute and hour buckets),
velocity state, and the DashMap per-shard overhead. The 1,088 bytes/entry figure
was measured in the m7p3 scale benchmarks. The table uses ten active signal
types per item and the default 5M-entry hot-tier ceiling. Without trimming, the
signal column would be ~10.88 GB at 1M items and ~108.8 GB at 10M items.

The signal ledger has a memory budget of 5M entries
(`DEFAULT_MAX_SIGNAL_ENTRIES`, ~5.44 GB). When exceeded, the checkpoint thread
evicts cold entries (oldest `last_update` timestamp). If your workload has more
than 5M active `(entity, signal_type)` pairs, cold entries will be served from
fjall checkpoints (slower, but correct).

**Tantivy text index:**

Tantivy's RAM usage depends on the number of indexed documents, average document length, and the number of open reader segments. The estimates above assume short metadata fields (title + description, ~200 bytes average). Long-form content indexing will increase RAM proportionally.

### Notes

- Signal ledger RAM is for the in-memory hot tier only. The WAL and fjall checkpoints add disk usage, not RAM.
- The table assumes ten active signal types per item. Below the 5M-entry hot-tier
  cap it scales linearly; beyond the cap, additional cold entries trade RAM for
  checkpoint-backed lookup cost.
- USearch RAM is the dominant index cost at high dimensionality. If you use 1536D embeddings (e.g., OpenAI text-embedding-3-large), budget from the complete analytic total and measured process envelope, not the vector column alone.
- The analytic table excludes process/runtime overhead, replication and catch-up
  buffers, allocator fragmentation, and concurrent query/write working memory.
  It is a sizing input, not a container limit. Use the measured envelope below
  for the multi-process cluster.

---

## Disk Capacity

Disk usage comes from three sources: fjall LSM-tree storage (metadata, relationships, signal checkpoints), WAL segments (append-only signal event log), and Tantivy/USearch index files.

| Items | Metadata Size   | Signal Events/Day | Disk/Day (WAL) | Fjall (90 days) | Total (90 days) |
|------:|:----------------|------------------:|----------------:|----------------:|----------------:|
| 100K  | small (256B avg) | 50K               | ~2 MB           | ~1 GB           | ~1.2 GB         |
| 1M    | small            | 500K              | ~20 MB          | ~10 GB          | ~11.8 GB        |
| 10M   | small            | 5M                | ~200 MB         | ~100 GB         | ~118 GB         |

### Formulas

**WAL daily growth:**

```
signal_events_per_day * ~40 bytes/event
```

Each WAL entry contains: 4-byte magic, 8-byte sequence, 1-byte event type, 8-byte entity ID, 2-byte signal type ID, 8-byte timestamp, 8-byte weight (f64), 32-byte BLAKE3 checksum. WAL segments are compacted after each successful checkpoint (every 30 seconds), so WAL disk usage represents only the uncompacted tail, not cumulative growth.

**Fjall storage:**

```
items * metadata_avg_bytes * 1.5 (LSM write amplification)
```

The 1.5x amplification factor accounts for LSM-tree space amplification (multiple sorted runs before compaction merges them). Actual amplification depends on the compaction strategy and write pattern. Signal checkpoints are also stored in fjall -- add ~100 bytes per active `(entity, signal_type)` pair for the serialized checkpoint data.

**Tantivy and USearch on disk:**

- Tantivy: roughly 1.5-2x the raw text size after indexing (inverted index + postings + term dictionary).
- USearch: saved index files are approximately the same size as the in-memory representation (items * dims * 2 bytes + graph metadata).

### WAL Compaction

WAL segments older than the last successful checkpoint are automatically deleted by the checkpoint thread (every 30 seconds). Under normal operation, WAL disk usage stays bounded at roughly `signal_rate * 40 bytes * 30 seconds`. Monitor `tidaldb_wal_lag_bytes` -- if it grows unbounded, checkpointing may be failing (check `tidaldb_checkpoint_failures_total`).

---

## Startup Time

Startup involves: opening fjall keyspaces, restoring the signal ledger from checkpoint, replaying WAL events since the last checkpoint, rebuilding in-memory indexes (bitmap, range, universe, creator-items, collections, suggestions), and loading USearch vector indexes.

| Items | Vectors | Typical Startup |
|------:|--------:|:----------------|
| 100K  | 100K    | ~2-5 sec        |
| 1M    | 1M      | ~15-45 sec      |
| 10M   | 10M     | ~3-8 min        |

### Dominant Costs

1. **USearch index load** is the dominant startup cost at 1M+ vectors. USearch rebuilds the HNSW graph from its serialized format. Progress is logged every 10K vectors.

2. **Signal ledger restore** reads the checkpoint from fjall (a single prefix scan of `Tag::Sig` keys), then replays any WAL events with sequence numbers higher than the checkpoint's `wal_sequence`. Time is proportional to the number of active signal entries + unreplayed WAL events.

3. **Entity state rebuild** scans the items and users keyspaces to reconstruct creator-items bitmaps, relationship indexes (follows, blocks, hides), and interaction weights. Progress is logged every 10K items.

4. **Suggestion index rebuild** scans all item metadata for "title" fields and indexes terms for autocomplete. This is a sequential scan -- fast for 100K items, noticeable at 10M.

5. **Collection index rebuild** reconstructs collection membership bitmaps from fjall.

### Notes

- Startup time is I/O-bound, not CPU-bound. Fast NVMe storage reduces startup time significantly compared to spinning disk.
- WAL replay time depends on how many signals were written since the last checkpoint (at most ~30 seconds of writes under normal operation).
- Tantivy indexes are opened directly from disk (memory-mapped) and do not require a rebuild step.

---

## Recommended Provisioning

**General rule:** provision 2x the estimated RAM for headroom.

| Scale    | Recommended RAM | Recommended Disk | CPU Cores |
|:---------|:----------------|:-----------------|:----------|
| 100K items, 128D | 512 MB   | 5 GB SSD         | 2         |
| 100K items, 768D | 1 GB     | 5 GB SSD         | 2         |
| 1M items, 128D   | 4 GB     | 25 GB SSD        | 4         |
| 1M items, 768D   | 8 GB     | 25 GB SSD        | 4         |
| 10M items, 128D  | 32 GB    | 250 GB NVMe      | 8         |
| 10M items, 768D  | 64 GB    | 250 GB NVMe      | 8         |
| 10M items, 1536D | 96 GB    | 250 GB NVMe      | 16        |

### Why 2x headroom?

- Signal ledger entries grow as new `(entity, signal_type)` pairs are written. The hot tier can hold up to 5M entries before trimming kicks in.
- Tantivy segment merges temporarily double the index size during merge operations.
- USearch does not support incremental resize -- if you approach capacity, you need enough free RAM to hold both the old and new index during a potential rebuild.
- The Rust allocator (jemalloc or system) has its own fragmentation overhead.

### Swap

Do not configure swap for production tidalDB instances. USearch HNSW traversal accesses memory in a random-access pattern that defeats page-level caching. A single swapped page in the HNSW graph can turn a 50-microsecond ANN query into a 50-millisecond disk seek.

### Disk Type

SSD is strongly recommended for all deployments. NVMe is recommended at 10M+ items. The WAL uses synchronous `fsync` on every segment rotation, and fjall's journal uses `persist(SyncAll)` during checkpoint. Spinning disk latency on these operations directly impacts signal write throughput.

---

## Cluster (Ref-A 3-node fleet) — measured capacity

The tables above are single-node, analytic estimates. This section is the **measured operating envelope** of the live cluster: **Ref-A = 3 nodes × 4 vCPU / 16 GiB**, full-placement RF3 (every pod replicates every shard group), 100k × 1536-D corpus. These are real `tidal-stress` numbers, not formulas. They are **Ref-A figures** — the enterprise **Ref-B** target is **≥5 × 8 vCPU / ≥16 GiB** nodes with **partitioned** placement (each pod hosts a subset of groups). See `docs/profiling/m12-cluster-deploy-findings.md` and `docs/profiling/m12p4-t5-sharded-throughput.md`.

### Read throughput

| metric | value | gate | verdict |
|--------|-------|------|---------|
| read p99 @ 100–500 rps | **7.97–11.47 ms** | ≤ 10 ms (G1) | **MET** @ 100k/1536-D |
| recall@10 | **0.9989** | ≥ 0.95 | **MET** |
| read ceiling (clean) | **~1000 read-ops/s** | — | CPU-bound |
| read ceiling (saturated) | **~1500 read-ops/s** | — | shed/error past here |

- Reads are CPU-bound: each read scatters **3 parallel HNSW searches** (one per shard group, full placement).
- Read throughput scales **~linearly with node count** on full placement — every node serves reads from its local replica of every group.
- Spread reads **round-robin across all 3 pods**; do not pin to the leader.

### Write throughput

| metric | value | note |
|--------|-------|------|
| write knee | **~250 rps in the clean sweep** | one benchmark, not a sustained-safe rate |
| nightly 200 rps soak | **23 PASS / 32 FAIL (42% green)** | not sustained with margin; gate parked |
| write scaling vs node count | **~1.0×** | does NOT scale at full-placement RF3 |

- Write tput does **NOT** scale with node count at full-placement RF3: every per-shard quorum spans all 3 nodes, so **every follower applies every 1536-D write**. Adding nodes adds replication work, not write capacity (~1.0×, not 2.5×).
- The old T5 **2.5× write-scaling** target needs **≥5 nodes + PARTITIONED placement**. T5 / G-S has been **re-scoped to read-throughput scaling**.

### Per-pod memory at 1536-D

| corpus / workload | observed per-pod working set | configured limit at observation | verdict |
|-------------------|------------------------------|---------------------------------|---------|
| 100k idle baseline | **~3.58 GiB** | 4 GiB | only ~420 MiB load/recovery headroom |
| 100k / 200 rps soak | **3.97-4.00 GiB before OOMKill** | 4 GiB | limit is insufficient |
| 1M × 1536-D estimate | **~7-8 GiB before runtime overhead** | — | requires Ref-B measurement |

- Full placement means each pod holds the whole corpus.
- Four independent historical windows show `OOMKilled` at 3.97-4.00 GiB, followed
  by snapshot-required reseed self-restarts. The exact internal growth source is
  not yet isolated; signal hot-entry count stayed flat, so do not label it a leak
  without allocation/heap evidence.
- The corrected canary envelope is a 4 GiB request and 6 GiB limit (measured peak
  plus 50% recovery/profiling headroom). Profile the exact 100k/1536-D/200-rps
  trajectory before treating 6 GiB as a production ceiling.

### Startup / boot

- HNSW rebuild/load at 1536-D is **CPU-bound** (~5 min single-core at 100k); 1M scales up from there.
- `startupProbe` budget is **~20 min** (`failureThreshold` 240 × 5 s).
- Graceful shutdown **saves the graphs** (grace **600 s**), so a clean restart **skips the rebuild** (load, not rebuild). A SIGKILL/crash skips the save → next boot rebuilds.

### Pod resources (corrected canary manifest)

| resource | request | limit |
|----------|---------|-------|
| CPU | 2 | 3 |
| memory | 4 GiB | 6 GiB |
| PVC | — | 5 GiB/pod, `local-path` |

- PVC is **local NVMe** (`local-path`) — longhorn's fsync overhead was unacceptable for the WAL path.

### Ref-B target (enterprise)

These are Ref-A figures. The enterprise **Ref-B** target is **≥5 × 8 vCPU / ≥16 GiB** nodes with **partitioned** placement — the shape required to (a) scale writes past ~1.0×, and (b) hold a 1M × 1536-D corpus without OOM.