9.0 KiB
Capacity Planning
This document provides RAM, disk, and startup time estimates for tidalDB deployments. Use these tables to provision hardware before going to production.
All estimates assume a single-node deployment with default configuration (30-second checkpoint interval, f16 vector quantization, DashMap-based hot tier).
RAM Capacity
tidalDB is an in-memory-first database. USearch HNSW indexes, the signal ledger hot tier, and Tantivy reader segments all reside in RAM during operation. There is no swap tolerance for USearch -- if the process is swapped, ANN query latency degrades from microseconds to seconds.
| Items | Embedding Dims | USearch RAM | Signal Ledger RAM (10 signals) | Tantivy RAM | Total Estimate |
|---|---|---|---|---|---|
| 100K | 128D | ~26 MB | ~110 MB | ~50 MB | ~200 MB |
| 100K | 768D | ~154 MB | ~110 MB | ~50 MB | ~320 MB |
| 100K | 1536D | ~307 MB | ~110 MB | ~50 MB | ~470 MB |
| 1M | 128D | ~256 MB | ~1.1 GB | ~200 MB | ~1.6 GB |
| 1M | 768D | ~1.5 GB | ~1.1 GB | ~200 MB | ~2.8 GB |
| 1M | 1536D | ~3.1 GB | ~1.1 GB | ~200 MB | ~4.4 GB |
| 10M | 128D | ~2.6 GB | ~11 GB | ~500 MB | ~14 GB |
| 10M | 768D | ~15 GB | ~11 GB | ~500 MB | ~27 GB |
| 10M | 1536D | ~31 GB | ~11 GB | ~500 MB | ~43 GB |
Formulas
USearch HNSW index:
items * dims * 2 bytes (f16 quantization) * 1.2 (HNSW graph overhead)
The 20% graph overhead accounts for HNSW neighbor lists (M=16 default, two layers). Actual overhead varies with M and ef_construction parameters.
Signal ledger hot tier:
items * signal_count * ~1,088 bytes/entry
Each (entity_id, signal_type_id) entry in the DashMap holds the running decay score, windowed counters (BucketedCounter with minute and hour buckets), velocity state, and the DashMap per-shard overhead. The 1,088 bytes/entry figure was measured in the m7p3 scale benchmarks.
The signal ledger has a memory budget of 5M entries (DEFAULT_MAX_SIGNAL_ENTRIES). When exceeded, the checkpoint thread evicts cold entries (oldest last_update timestamp). If your workload has more than 5M active (entity, signal_type) pairs, cold entries will be served from fjall checkpoints (slower, but correct).
Tantivy text index:
Tantivy's RAM usage depends on the number of indexed documents, average document length, and the number of open reader segments. The estimates above assume short metadata fields (title + description, ~200 bytes average). Long-form content indexing will increase RAM proportionally.
Notes
- Signal ledger RAM is for the in-memory hot tier only. The WAL and fjall checkpoints add disk usage, not RAM.
- The "10 signals" column assumes 10 distinct signal types per entity. Scale linearly for more signal types.
- USearch RAM is the dominant cost at high dimensionality. If you use 1536D embeddings (e.g., OpenAI text-embedding-3-large), plan for USearch to consume 70%+ of total RAM at 10M items.
Disk Capacity
Disk usage comes from three sources: fjall LSM-tree storage (metadata, relationships, signal checkpoints), WAL segments (append-only signal event log), and Tantivy/USearch index files.
| Items | Metadata Size | Signal Events/Day | Disk/Day (WAL) | Fjall (90 days) | Total (90 days) |
|---|---|---|---|---|---|
| 100K | small (256B avg) | 50K | ~2 MB | ~1 GB | ~1.2 GB |
| 1M | small | 500K | ~20 MB | ~10 GB | ~11.8 GB |
| 10M | small | 5M | ~200 MB | ~100 GB | ~118 GB |
Formulas
WAL daily growth:
signal_events_per_day * ~40 bytes/event
Each WAL entry contains: 4-byte magic, 8-byte sequence, 1-byte event type, 8-byte entity ID, 2-byte signal type ID, 8-byte timestamp, 8-byte weight (f64), 32-byte BLAKE3 checksum. WAL segments are compacted after each successful checkpoint (every 30 seconds), so WAL disk usage represents only the uncompacted tail, not cumulative growth.
Fjall storage:
items * metadata_avg_bytes * 1.5 (LSM write amplification)
The 1.5x amplification factor accounts for LSM-tree space amplification (multiple sorted runs before compaction merges them). Actual amplification depends on the compaction strategy and write pattern. Signal checkpoints are also stored in fjall -- add ~100 bytes per active (entity, signal_type) pair for the serialized checkpoint data.
Tantivy and USearch on disk:
- Tantivy: roughly 1.5-2x the raw text size after indexing (inverted index + postings + term dictionary).
- USearch: saved index files are approximately the same size as the in-memory representation (items * dims * 2 bytes + graph metadata).
WAL Compaction
WAL segments older than the last successful checkpoint are automatically deleted by the checkpoint thread (every 30 seconds). Under normal operation, WAL disk usage stays bounded at roughly signal_rate * 40 bytes * 30 seconds. Monitor tidaldb_wal_lag_bytes -- if it grows unbounded, checkpointing may be failing (check tidaldb_checkpoint_failures_total).
Startup Time
Startup involves: opening fjall keyspaces, restoring the signal ledger from checkpoint, replaying WAL events since the last checkpoint, rebuilding in-memory indexes (bitmap, range, universe, creator-items, collections, suggestions), and loading USearch vector indexes.
| Items | Vectors | Typical Startup |
|---|---|---|
| 100K | 100K | ~2-5 sec |
| 1M | 1M | ~15-45 sec |
| 10M | 10M | ~3-8 min |
Dominant Costs
-
USearch index load is the dominant startup cost at 1M+ vectors. USearch rebuilds the HNSW graph from its serialized format. Progress is logged every 10K vectors.
-
Signal ledger restore reads the checkpoint from fjall (a single prefix scan of
Tag::Sigkeys), then replays any WAL events with sequence numbers higher than the checkpoint'swal_sequence. Time is proportional to the number of active signal entries + unreplayed WAL events. -
Entity state rebuild scans the items and users keyspaces to reconstruct creator-items bitmaps, relationship indexes (follows, blocks, hides), and interaction weights. Progress is logged every 10K items.
-
Suggestion index rebuild scans all item metadata for "title" fields and indexes terms for autocomplete. This is a sequential scan -- fast for 100K items, noticeable at 10M.
-
Collection index rebuild reconstructs collection membership bitmaps from fjall.
Notes
- Startup time is I/O-bound, not CPU-bound. Fast NVMe storage reduces startup time significantly compared to spinning disk.
- WAL replay time depends on how many signals were written since the last checkpoint (at most ~30 seconds of writes under normal operation).
- Tantivy indexes are opened directly from disk (memory-mapped) and do not require a rebuild step.
Recommended Provisioning
General rule: provision 2x the estimated RAM for headroom.
| Scale | Recommended RAM | Recommended Disk | CPU Cores |
|---|---|---|---|
| 100K items, 128D | 512 MB | 5 GB SSD | 2 |
| 100K items, 768D | 1 GB | 5 GB SSD | 2 |
| 1M items, 128D | 4 GB | 25 GB SSD | 4 |
| 1M items, 768D | 8 GB | 25 GB SSD | 4 |
| 10M items, 128D | 32 GB | 250 GB NVMe | 8 |
| 10M items, 768D | 64 GB | 250 GB NVMe | 8 |
| 10M items, 1536D | 96 GB | 250 GB NVMe | 16 |
Why 2x headroom?
- Signal ledger entries grow as new
(entity, signal_type)pairs are written. The hot tier can hold up to 5M entries before trimming kicks in. - Tantivy segment merges temporarily double the index size during merge operations.
- USearch does not support incremental resize -- if you approach capacity, you need enough free RAM to hold both the old and new index during a potential rebuild.
- The Rust allocator (jemalloc or system) has its own fragmentation overhead.
Swap
Do not configure swap for production tidalDB instances. USearch HNSW traversal accesses memory in a random-access pattern that defeats page-level caching. A single swapped page in the HNSW graph can turn a 50-microsecond ANN query into a 50-millisecond disk seek.
Disk Type
SSD is strongly recommended for all deployments. NVMe is recommended at 10M+ items. The WAL uses synchronous fsync on every segment rotation, and fjall's journal uses persist(SyncAll) during checkpoint. Spinning disk latency on these operations directly impacts signal write throughput.