a0399550d6
Boot now LOADS the per-slot HNSW graph instead of rebuilding it. Clean
shutdown writes {data_dir}/vector/<kind>__<slot>.usearch; the next open loads
it when it matches the durable corpus (seconds), falling back to a full rebuild
only when the graph is missing/stale/corrupt. Eliminates the multi-minute boot
rebuild (~50-70 min at 1M/1536-D) that let the WAL compact past a restarting
node and triggered the reseed cascade.
Graceful SIGTERM now actually runs the close: bounded_drain caps the post-signal
HTTP drain (TIDAL_SHUTDOWN_DRAIN_MS, default 15s) then runs the deterministic
close regardless — sibling keep-alive connections no longer block the drain past
the k8s 60s grace into a SIGKILL (which cannot run Drop). ClusterNode and
ShardReplica::shutdown are now &self (db handle is an ArcSwapOption) so the close
fires even when a stuck connection task holds an Arc.
Fix USearch insert to be a true upsert (remove+add): it was unconditional add,
which a multi:false index rejects on a reseeding follower's post-snapshot WAL
replay -> applied_events stalls -> catch-up deadlock -> unrecoverable cluster.
Also: circuit-breaker peer last-contact tracking; real k3s 1536-dim deploy +
recall findings (recall@10 0.9869, read p99 8.71ms @ 200rps @ 100k) in
docs/profiling/m12-cluster-deploy-findings.md; new tidal-stress k8s jobs and
m12p6 graph-persistence + SIGTERM tier-3 regression tests.
|
||
|---|---|---|
| .. | ||
| guides | ||
| ops | ||
| planning | ||
| profiling | ||
| research | ||
| reviews | ||
| runbooks | ||
| specs | ||
| content-strategy.md | ||
| personal-briefing-beachhead.md | ||
| README.md | ||
| roadmap-to-cluster.md | ||
tidalDB Engineering Docs
The engineering documentation home. Top-level product docs (VISION, USE_CASES, SEQUENCE, ARCHITECTURE, API, QUICKSTART, CODING_GUIDELINES, thoughts) live at the repository root; everything below is the deeper engineering record.
This and the repo root are the two canonical doc homes. There is intentionally no per-crate doc mirror (no
tidal/docs/). Edit the canonical file, never a copy.
Component specs — specs/
The authoritative component specifications (status: Implemented, M0–M8).
| # | Spec | # | Spec |
|---|---|---|---|
| 00 | Architecture overview | 08 | Query engine |
| 01 | Storage engine | 09 | Ranking & scoring |
| 02 | Entity model | 10 | Feedback loop |
| 03 | Signal system | 11 | Schema |
| 04 | Relationships | 12 | Cold start |
| 05 | Cohorts | 13 | Concurrency |
| 06 | Text retrieval | 14 | Scale architecture |
| 07 | Vector retrieval |
Planning — planning/
- ROADMAP.md — milestones M0–M11, phase status, known gaps
- roadmap-to-cluster.md — adopted M11 plan: gap analysis + phase specs taking the multi-process cluster from experimental to enterprise-grade (m11p1–p5 ✅; p6–p9 planned), grounded in the 2026-06-10 live stress-test baselines
- PRODUCT_ROADMAP.md · architecture-review.md · roadmap-cohort-analysis.md · site-cohort-analysis.md
- Per-milestone phase/task archive:
planning/milestone-0,1,2,3,5,7,8,9,10,11,p/
Code reviews — reviews/
- M0–M10 code review — 2026-06-07 — seven-dimension re-review, 88 verified findings
- M0–M10 code review — 2026-06-08 — seven-dimension review, 142 findings (latest pass)
- M0–M10 seven-dimension review — additional pass (2 BLOCKERs: signal-checkpoint trim, 30-day window)
Guides — guides/
Task-oriented, build-an-app docs (complements the root QUICKSTART.md and API.md):
- Build a feed app — end-to-end TikTok/Reels-style "For You" feed, embedded and over HTTP
- Embedding integration — wiring a real embedding model (OpenAI / Cohere / local) into the write + query paths
- Server deployment — running the
tidal-serverHTTP service: config, auth, the served OpenAPI spec, Docker - Ranking-profile reference: ai-lookup/services/ranking-profiles.md — all 25 built-in profiles
Operations — ops/ and runbooks/
- Monitoring · Prometheus alerts · Grafana dashboard · Capacity planning · Recovery
- Runbooks: Kubernetes · Cluster (experimental)
Research — research/
ANN (1, 2) · Tantivy (1, 2) · Signal ledger (1, 2) · WAL · Type system · Tooling & diagnostics · Enterprise-readiness risks
Profiling — profiling/
Hotspot analysis · Scale baselines · Signal memory · Signal rollup eval · Social scale · Tantivy merge tuning · USearch tuning