tidaldb/tidal/src/storage
jordan 7c1c80dd90 fix(vector): grow the HNSW graph on insert; it could never accept a new vector once full
Measured on the live RF3 cluster 2026-08-30 while verifying it end to end. Every
NEW embedding returned HTTP 500:

  POST /sharded/embeddings -> 500
  {"error":"... [op=write_item_embedding] backend error:
    USearch insert failed: Reserve capacity ahead of insertions!"}

while POST /sharded/items returned 201, POST /sharded/signals 204, text search
and feed 200, and re-embedding an ALREADY-INDEXED entity returned 204. So the
store had silently become read/update-only for vectors: a consumer could write
items and signals all day and only its embeddings would fail, with nothing
alerting on it.

Cause: USearch's add() cannot grow the graph. build_slot_index() reserves the
rebuild's expected count once and its comment claimed 'the write path grows it
further as needed' - the write path never reserved anything. insert() went
straight to add() for a new key, so the moment size() reached the reservation
every new key failed permanently. The upsert path kept working because remove()
tombstones and size() excludes tombstones, so the re-add lands in the slot just
freed - which is exactly why this looked healthy from the outside.

insert() now reserves before adding a new key at capacity, growing by
max(size/8, 1024) so reserve's reallocation is amortised rather than per-insert.

The regression test drives off the REPORTED capacity, not the requested one:
USearch rounds a reservation up (reserve(4) reported 64), so a hardcoded insert
count sits inside the reservation and exercises nothing. Verified against the
pre-fix source it fails with the production error at insert 64 of capacity 64,
and it asserts the grown graph still answers searches and still upserts.

storage::vector 101 passed, db::items 7 passed.
2026-08-30 11:33:03 -06:00
..
indexes fix: M0-M10 code-review pass2 remediation — all 91 findings 2026-06-09 12:21:00 -06:00
vector fix(vector): grow the HNSW graph on insert; it could never accept a new vector once full 2026-08-30 11:33:03 -06:00
batch.rs feat: implement Milestone 1 phases 1-3 — schema, WAL, and storage layer 2026-02-20 16:43:24 -07:00
engine.rs fix: M0-M10 code-review pass2 remediation — all 91 findings 2026-06-09 12:21:00 -06:00
error.rs fix: M0-M10 code-review pass2 remediation — all 91 findings 2026-06-09 12:21:00 -06:00
fjall.rs fix: M0-M10 code-review pass2 remediation — all 91 findings 2026-06-09 12:21:00 -06:00
iterator.rs feat: implement Milestone 1 phases 1-3 — schema, WAL, and storage layer 2026-02-20 16:43:24 -07:00
keys.rs Merge remote-tracking branch 'origin/main' (m11/m12 cluster) into m9/m10 2026-08-03 02:16:04 -06:00
memory.rs fix: M0-M10 code-review pass2 remediation — all 91 findings 2026-06-09 12:21:00 -06:00
mod.rs feat: complete Milestones 2–4 — RETRIEVE query, vector index, ranking profiles, diversity, entity system, sessions 2026-02-21 16:24:48 -07:00