jordan/tidaldb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
a946c6128c
tidaldb/docs
History
jx12n 727fbfcb6b fix(m12p6): 6-bug k3s 3-shard cluster repair (rc8+rc9) 
Root-caused and fixed five sharding bugs exposed on the real k3s 3-shard
cluster (rc5→rc7), plus a divergent-rejoin reseed loop found in rc9:

1. reseed shard-awareness (Bug 3, keystone): `run_boot_install_for_region`
   visits each hosted group's own shard subdir; per-group leader discovery
   appends `?shard=N` so a divergent shard heals from its own leader (not
   shard-0's WAL/term — cross-shard contamination).
2. leader self-join term (Bug 4): `become_leader_for_term` now calls
   `note_self_won_term` so the elected shard's `joined_term` is set and
   `cluster_promote` routes rebalances correctly (was: topology-era mis-read
   → legacy fenced promote → 500).
3. boot self-heal self-pull guard (Bug 2): `leader_shard != my_shard` gate
   prevents a node pulling its own stream (its stream isn't a registered peer)
   → eliminates the `PeerUnreachable(self)` loop.
4. scatter-merge degraded partial (Bug 1): failed shard logs + continues
   instead of `?`-failing the whole read; bounded read-admission semaphore
   (`offload.rs`) sheds as 429 instead of piling into a 36s p99.
5. WAL retention (Bug 5): `compact_wal_retained` keeps `WAL_RETENTION_SEGMENTS=4`
   most-recent sealed segments; online path gets the same retention clamp.
   Prevents brief-restart forced-reseed.
6. divergent-rejoin reseed loop (Bug 6, rc9): `note_quarantined` latches
   `from_seqno = stream_baseline` (not `frontier + 1`) so `wal_covers`
   returns `needed=true` and the snapshot installs instead of looping.

Also: `TidalDb::close_shared` for deterministic HNSW save on cluster SIGTERM
(HNSW graph was not saved when request-scoped Arc clones were alive at shutdown);
updated profiling doc with full rc8/rc9 fix narrative; k8s recall job YAMLs.
2026-06-16 22:34:21 -06:00
..
guides feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
ops bench(1536): production-shape capacity — read path is cheap, quorum write is the ceiling 2026-06-13 21:40:58 -06:00
planning feat(m11): sharding × replication + rebalancing (m11p6 L3-L5) 2026-06-13 18:23:43 -06:00
profiling fix(m12p6): 6-bug k3s 3-shard cluster repair (rc8+rc9) 2026-06-16 22:34:21 -06:00
research chore: doc consolidation, seven-dimension review fixes, and commit hooks 2026-06-08 22:46:28 -06:00
reviews feat(m11): observability+ops (m11p8) + perf-sweep wave 2 T2 2026-06-13 09:17:49 -06:00
runbooks fix(m12p6): T4 TLS scale-up enablement — https seed-join via ready-only Service + up-front rustls provider 2026-06-14 20:29:37 -06:00
specs feat(m12): vector retrieval G1/G2 — recall harness, ANN in RETRIEVE, index tuning 2026-06-14 11:07:09 -06:00
content-strategy.md feat: complete Milestone 5 — full-text search, RRF fusion, and creator search 2026-02-21 23:53:16 -07:00
personal-briefing-beachhead.md feat: M0p1 runtime skeleton, M0p2 tooling & diagnostics, m1p4 signal ledger 2026-02-20 20:32:00 -07:00
README.md feat(m11): membership, snapshot install, and reseed (m11p5) 2026-06-12 19:55:54 -06:00
roadmap-to-cluster.md feat(m11): sharding × replication + rebalancing (m11p6 L3-L5) 2026-06-13 18:23:43 -06:00

README.md

tidalDB Engineering Docs

The engineering documentation home. Top-level product docs (VISION, USE_CASES, SEQUENCE, ARCHITECTURE, API, QUICKSTART, CODING_GUIDELINES, thoughts) live at the repository root; everything below is the deeper engineering record.

This and the repo root are the two canonical doc homes. There is intentionally no per-crate doc mirror (no tidal/docs/). Edit the canonical file, never a copy.

Component specs — specs/

The authoritative component specifications (status: Implemented, M0–M8).

# Spec # Spec
00 Architecture overview 08 Query engine
01 Storage engine 09 Ranking & scoring
02 Entity model 10 Feedback loop
03 Signal system 11 Schema
04 Relationships 12 Cold start
05 Cohorts 13 Concurrency
06 Text retrieval 14 Scale architecture
07 Vector retrieval

Planning — planning/

Code reviews — reviews/

Guides — guides/

Task-oriented, build-an-app docs (complements the root QUICKSTART.md and API.md):

Operations — ops/ and runbooks/

Research — research/

ANN (1, 2) · Tantivy (1, 2) · Signal ledger (1, 2) · WAL · Type system · Tooling & diagnostics · Enterprise-readiness risks

Profiling — profiling/

Hotspot analysis · Scale baselines · Signal memory · Signal rollup eval · Social scale · Tantivy merge tuning · USearch tuning

Strategy

Content strategy · Personal-briefing beachhead