a0399550d6
Boot now LOADS the per-slot HNSW graph instead of rebuilding it. Clean
shutdown writes {data_dir}/vector/<kind>__<slot>.usearch; the next open loads
it when it matches the durable corpus (seconds), falling back to a full rebuild
only when the graph is missing/stale/corrupt. Eliminates the multi-minute boot
rebuild (~50-70 min at 1M/1536-D) that let the WAL compact past a restarting
node and triggered the reseed cascade.
Graceful SIGTERM now actually runs the close: bounded_drain caps the post-signal
HTTP drain (TIDAL_SHUTDOWN_DRAIN_MS, default 15s) then runs the deterministic
close regardless — sibling keep-alive connections no longer block the drain past
the k8s 60s grace into a SIGKILL (which cannot run Drop). ClusterNode and
ShardReplica::shutdown are now &self (db handle is an ArcSwapOption) so the close
fires even when a stuck connection task holds an Arc.
Fix USearch insert to be a true upsert (remove+add): it was unconditional add,
which a multi:false index rejects on a reseeding follower's post-snapshot WAL
replay -> applied_events stalls -> catch-up deadlock -> unrecoverable cluster.
Also: circuit-breaker peer last-contact tracking; real k3s 1536-dim deploy +
recall findings (recall@10 0.9869, read p99 8.71ms @ 200rps @ 100k) in
docs/profiling/m12-cluster-deploy-findings.md; new tidal-stress k8s jobs and
m12p6 graph-persistence + SIGTERM tier-3 regression tests.
220 lines
13 KiB
Markdown
220 lines
13 KiB
Markdown
# M12 cluster deploy + recall findings (k3s, 1536-dim)
|
||
|
||
Date: 2026-06-15. Cluster: orchard9-k3sf, namespace `tidaldb-cluster`, 3 nodes
|
||
(4 vCPU / 16 GiB each), local-path PVCs. Image built from HEAD `8e39ee1` + the
|
||
working-tree T4-TLS fixes. **Goal: deploy M12 to the real cluster and get the
|
||
real recall@10 + read p99 at the 1536-dim production shape** (the m11p6 image
|
||
404'd `/vector_search`).
|
||
|
||
## Headline result (real, live cluster)
|
||
|
||
100k items / 1536-dim, recall@10 vs a brute-force cosine oracle (1000-query pool),
|
||
`/vector_search` pure k-NN against one node:
|
||
|
||
| metric | value | gate | verdict |
|
||
|--------|-------|------|---------|
|
||
| recall@10 | **0.9869** | ≥ 0.95 | **MET** |
|
||
| read p99 @ knee | **8.71 ms** @ 200 rps | ≤ 10 ms | **MET (at 100k)** |
|
||
| read knee (single node) | **~200 rps** | — | saturates fast |
|
||
|
||
Beyond the knee a single node sheds/errors hard: 62% error at 500 rps (achieved
|
||
267), 100% at 1000+. 1536-dim k-NN reads saturate one node at ~250 rps.
|
||
|
||
Honest scope:
|
||
- This is **100k, not 1M**. The G1 gate is p99 ≤ 10 ms @ **1M**/1536-D — still
|
||
unproven; at 1M the candidate set grows and p99 rises. 100k clears it at 8.71 ms.
|
||
- recall 0.9869 is **below the standalone-laptop 0.9997** because the measured
|
||
node's HNSW carries **tombstones from the upsert-reseed** (remove+add, below) —
|
||
a freshly-built index has no tombstones and recalls higher. Re-measure on a
|
||
fresh index for the canonical number.
|
||
- Measured against a **single healthy replica** with the corpus already resident
|
||
(`--skip-seed`), because the cluster's **write path is unstable at 1536-dim**
|
||
(below) — a full clean seed+query could not complete.
|
||
|
||
## Bug found + fixed: USearch insert was not an upsert (reseed wedge)
|
||
|
||
`VectorIndex::insert`'s contract is "Replaces any existing vector with the same
|
||
ID." The USearch backend (`tidal/src/storage/vector/usearch_index.rs`) violated
|
||
it: it computed `is_new` but then **unconditionally `add`**, which a `multi:false`
|
||
index rejects with `Duplicate keys not allowed in high-level wrappers` whenever
|
||
the key exists.
|
||
|
||
Impact: a reseeding follower's post-snapshot WAL replay re-applies embeddings the
|
||
snapshot already loaded. Every overlapping embedding failed `add` →
|
||
`WAL blob replay failed` → applied_events never advanced past the snapshot
|
||
frontier → the catch-up receiver deadlocked on a backlog it could not apply
|
||
(`inbound channel full` forever). With all three replicas behind the compaction
|
||
horizon simultaneously, none would self-restart (quorum protection) → the cluster
|
||
was unrecoverable.
|
||
|
||
Fix: make `insert` a true upsert — `remove` the existing key, then `add`. Added a
|
||
regression test (`usearch_insert_replaces_existing_key`). Verified in production:
|
||
operator-forced rolling reseed (followers first from the ahead leader, leader
|
||
last) completed with `dupkey=0` and converged 3/3 to lag=0, no data loss.
|
||
|
||
Tradeoff: `remove`+`add` tombstones the old slot, so a reseed leaves the index
|
||
tombstoned (the recall dip above). A periodic compaction/rebuild reclaims it.
|
||
|
||
## Open upstream issues (NOT bandaided — flagged for the devs)
|
||
|
||
1. **1536-dim write path destabilizes the cluster.** The recall seed's quorum
|
||
write-burst (100k embeddings) trips the leader's partition detector: it marks
|
||
the followers `partitioned:true, reachable:false` (the followers actually
|
||
applied the data and self-report healthy) → leader believes it lost quorum →
|
||
writes stall AND reads on the leader time out (HTTP 408, the cross-region
|
||
gather waits on followers it thinks are gone). The view does not self-heal;
|
||
only a leader restart + re-election clears it. Root cause is almost certainly
|
||
the replication ship/ack path not keeping pace with 1536-dim apply (each apply
|
||
is an HNSW insert at ef_construction=400), tripping heartbeat/ack timeouts.
|
||
**This blocks ingesting the production-shape corpus and is the top blocker to
|
||
"M12 production-ready at 1536-D."**
|
||
|
||
2. **HNSW index is rebuilt on every boot, not persisted.** Open logs
|
||
`using USearch (HNSW) backend ... count=100000` then spends ~5.5 min
|
||
single-core rebuilding the graph before binding HTTP. 1M would imply a ~50–70
|
||
min boot. The long outage is what let the WAL compact past restarting nodes
|
||
and triggered the reseed cascade in the first place. Persist/mmap the graph so
|
||
boot is a load, not a rebuild. (Mitigation applied: startup probe budget
|
||
raised to 20 min; pod memory limit 2Gi→4Gi — 100k/1536 peaks ~1.9 GiB.)
|
||
|
||
3. **The 3-group `shards:` topology runs as a single shard at runtime.**
|
||
`/cluster/status` reports one shard (shard 0, RF=3) on every node despite the
|
||
topology's `shards:` block. Relevant to the T5 sharded-throughput gate — the
|
||
sharding is not actually engaging.
|
||
|
||
## Deploy mechanics (for repeat)
|
||
|
||
- Cross-compile on macOS-arm64 → x86_64-linux (GCC 15.2 = glibc 2.41 → trixie
|
||
runtime; binary NEEDs `libmvec.so.1`, absent on bookworm). Runtime image
|
||
`debian:trixie-slim` + `libstdc++6`/`libgcc-s1`, built via the `amd64builder`
|
||
docker-container buildx builder (QEMU). Pushed to `registry.threesix.ai`.
|
||
- Two-tier PKI (`k8s/cluster/certs.yaml`, working-tree) applied cleanly: `ca.crt`
|
||
is now a real CA (CA:TRUE); leaf carries the wildcard SAN + ready-only client
|
||
Service SAN (needed for T4 scale-to-5).
|
||
- Deployed image: `registry.threesix.ai/tidal/server:m12-rc2`
|
||
(`sha256:1cfa30e5938002a881545b60f21cdfa50af6f87be8190d1ed4b1ca5a83d5bc3b`) =
|
||
m12 + the upsert fix.
|
||
|
||
## rc3 update (2026-06-15): three fixes + verification
|
||
|
||
`m12-rc3` (`sha256:bceadce5419d5fcd78d3491f9d53cbd4055ee131f5ec7e9efa75b0f36474d1f7`)
|
||
= rc2 + three changes (built/deployed; combined `cargo check` green; per-area
|
||
unit/chaos tests pass):
|
||
|
||
1. **G1 index tuning — DONE & verified at 100k.** Grid search (in-process, clean
|
||
index) at 1536-D: `ef_search` 200→64 holds recall@10 0.998 at ~half the p99
|
||
(2434µs→1221µs); Int8 disqualified (recall collapses to ~0.71). Default set:
|
||
`ANN_DEFAULT_EF_SEARCH=64` (candidate_gen.rs). End-to-end on the cluster
|
||
(100k/1536, follower target): read-knee moved **200→500 rps**, p99 **7.06ms**,
|
||
recall@10 **0.967** (tombstone-depressed; clean index ≈0.998) — **G1+G2 MET at
|
||
the 100k knee**. 1M still unproven (blocked by the ingest issue below).
|
||
|
||
2. **False-partition status view — FIXED & verified.** Under the 100k/1536 write
|
||
burst, `partitioned_peers` stayed **0** throughout (previously the leader
|
||
falsely flagged followers partitioned). The detector now uses a gRPC
|
||
last-contact liveness signal; genuine-partition detection intact (chaos tests
|
||
pass).
|
||
|
||
3. **HNSW persist-on-boot — IMPLEMENTED, NOT yet effective on this cluster.** Two
|
||
real gaps found in verification: (a) the save is wired into `Db::shutdown_inner`
|
||
but the k8s SIGTERM path doesn't reach it — no `{data_dir}/vector/` file
|
||
appeared after a `kubectl delete pod` (fix in progress: wire the save into the
|
||
cluster serve SIGTERM shutdown); (b) this cluster reseeds on almost every
|
||
restart (term-mismatch / accumulated PVC churn), and the reseed path rebuilds
|
||
from the snapshot's raw vectors, bypassing graph-load. Both must be closed for
|
||
the persist fix to pay off here.
|
||
|
||
### NEW blockers surfaced by rc3 verification (the real 1536-D ingest wall)
|
||
|
||
The false-partition fix removed the *view* symptom, but the underlying defect —
|
||
**the leader's HTTP/gather plane degrades under 1536-D load** — has two more faces
|
||
that still block production-shape ingest:
|
||
|
||
- **Write-forward timeout under burst.** Seeding 100k to a *follower* target: the
|
||
follower→leader HTTP forward (`/items`,`/embeddings`) succeeds ~1950 writes then
|
||
every forward times out (`leader unreachable ... 503, latency 15000ms`) and the
|
||
seed stalls. On an IDLE cluster the same forward takes ~47ms — it is purely
|
||
load-induced HTTP-plane saturation on the leader. (Fix in progress.)
|
||
- **Leader read-gather hang.** Even idle, `/vector_search` on the **leader**
|
||
times out (HTTP 000 @ 8s) while **followers** serve it in ~50ms — the leader's
|
||
cross-region/shard read gather hangs. Reads must currently be sent to followers.
|
||
|
||
Net: at 1536-D the leader cannot ingest a sustained quorum write load nor serve
|
||
vector reads; followers serve reads fine. This is the top blocker to T4/T5/1M and
|
||
to a clean (tombstone-free) recall number. Root cause is the apply/HTTP-plane
|
||
contention; the status-view fix was necessary but not sufficient.
|
||
|
||
## rc4 (2026-06-15): ingest admission-control + SIGTERM bounded-drain
|
||
|
||
`m12-rc4` (`sha256:28d0c45e59c487cf0d5e6a574c4c9997b82c6a358d67e74f326ff2c0c0925193`)
|
||
adds the two gap fixes:
|
||
- **Ingest admission control + forward retry (fixes the write-forward timeout).**
|
||
Leader `/items`+`/embeddings` apply now goes through the bounded `ClusterWritePool`
|
||
(was unbounded `spawn_blocking`) → sheds as fast 429+retry_after instead of
|
||
timing out; the follower→leader forwarder retries 3× honoring 429/backoff within
|
||
a 28s budget (< the 30s route timeout). Idempotent upserts make retry safe. 147
|
||
tidal-server tests pass.
|
||
- **SIGTERM bounded-drain (fixes the persist-save not running).** The cluster
|
||
`with_graceful_shutdown` blocked forever on sibling keep-alive connections →
|
||
60s grace → SIGKILL → no `Drop` → no graph save. rc4 adds a 15s drain deadline
|
||
+ an `&self`-reachable deterministic close (`ShardReplica.db` → `ArcSwapOption`)
|
||
so `checkpoint_embedding_graphs` always runs. Real multi-process SIGTERM e2e test
|
||
asserts the shutdown checkpoint is written.
|
||
|
||
### STILL-OPEN catch-up/term bugs (surface under restart churn; NOT yet fixed)
|
||
|
||
The rc4 rollout exposed two more pre-existing replication bugs that make every
|
||
rolling restart a slog on these (heavily-churned) PVCs — distinct from anything
|
||
the four fixes above address:
|
||
- **Term-gate rejects same-term catch-up chunks.** A reseeding/catching-up node
|
||
pulls a catch-up stream and the chunks are refused:
|
||
`catch-up chunk refused by the term gate ... chunk_term=10 rejection=Stale { current_term: 10 }`
|
||
— `chunk_term == current_term` should NOT be stale, but it is, so the pull
|
||
retry-loops forever (the node never converges via the stream). Workaround:
|
||
restart the node so it takes the snapshot-install path instead of the stream.
|
||
- **Reseed-on-restart churn.** A node restart that coincides with a term bump
|
||
(e.g. the leader restarting during a rolling update) term-mismatches on rejoin
|
||
and does a full snapshot reseed (rebuild-from-snapshot, ~5-7min) rather than a
|
||
cheap WAL catch-up — even when only a few events behind. This both slows every
|
||
rollout AND bypasses the new HNSW graph-load (reseed rebuilds from raw vectors).
|
||
|
||
These two are the next upstream targets: a correct term-gate (accept `==` term)
|
||
plus a streaming catch-up that actually converges would remove the reseed churn,
|
||
let restarts be fast clean reloads, and make the HNSW-persist fix pay off.
|
||
|
||
### rc4 cluster verification (2026-06-15)
|
||
|
||
- **Ingest admission-control + forward-retry — VERIFIED.** The 100k/1536 seed
|
||
directed at a FOLLOWER (forwards to leader) now commits **steadily** (leader
|
||
applied climbed monotonically ~52k events over 10 min) with **zero
|
||
`forward to leader failed` 503s**, **zero false partition** (`partitioned`
|
||
stayed 0), and only brief graceful 429 backpressure — exactly the design. The
|
||
pre-fix build hard-stalled at ~1500 writes in 90s. (Throughput is slow only
|
||
because it upserts the existing corpus; correctness/liveness is the point.)
|
||
This unblocks the 1536-D ingest path for T4/T5/1M.
|
||
- **HNSW persist-on-boot — NOT yet effective on the cluster.** Despite the rc4
|
||
SIGTERM bounded-drain fix (which makes the shutdown path RUN — the e2e test
|
||
asserts `checkpoint.meta` is written), a clean follower restart on the real
|
||
1536-D cluster still showed `using USearch backend` (rebuild) and **no
|
||
`{data_dir}/vector/` file** — the graph save does not materialize via the
|
||
SIGTERM→bounded-drain→`ArcSwap` close→`Drop`→`shutdown_inner` path, even though
|
||
Agent's direct-`shutdown()` unit test (dim=4000) writes the file. This is a
|
||
reconciliation gap between the bounded-drain/ArcSwap-close and the
|
||
graph-save-in-`shutdown_inner` (Drop may not run the full shutdown body), on
|
||
top of the reseed-churn that would bypass graph-LOAD anyway. Implemented +
|
||
unit-tested, but not production-verified.
|
||
|
||
### Net state after the fan-out (G1 + 2 cluster fixes)
|
||
|
||
| item | status |
|
||
|------|--------|
|
||
| G1 ef_search=64 tuning | **DONE** — verified 100k (knee 500 rps, p99 7.06ms, recall 0.967≥0.95); 1M unproven |
|
||
| Fix #1a false-partition view | **DONE** — verified (partitioned_peers=0 under the write burst) |
|
||
| Fix #1b ingest saturation (forward-503) | **DONE** — verified (seed commits steadily, 0 forward-503) |
|
||
| Fix #2 HNSW persist | implemented + unit-tested; **cluster save doesn't materialize** (open) |
|
||
| term-gate same-term `Stale` reject | **open** (workaround: restart→snapshot-install) |
|
||
| reseed-on-restart churn | **open** (term-mismatch → full snapshot reseed) |
|
||
|
||
Deployed: `m12-rc4`
|
||
(`sha256:28d0c45e59c487cf0d5e6a574c4c9997b82c6a358d67e74f326ff2c0c0925193`).
|