jordan/tidaldb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
31ee612f27
tidaldb/docs
History
jx12n 31ee612f27 feat(m12p4): sharded ingestion — scatter-gather pool + cross-shard unified reads (L4) 
Scale write throughput across data-shard groups while keeping a single unified
read surface:

- scatter_gather.rs: pooled fan-out across shard groups (replaces per-request
  client construction); cross-shard query results merged on one node
- cluster/node.rs: cross-shard read routing — a read on any node gathers from
  every shard group's leader and unions results
- cluster/forward.rs: fix h2 204 forward-relay bug (relay_forwarded skips body
  for 1xx/204/304 — synthesized JSON body on a 204 triggered HTTP/2 RST_STREAM
  on the real mTLS plane)
- dto.rs: cross-shard query/result DTOs
- k8s/cluster/: enable 3-group `shards:` topology (statefulset, service-peers,
  topology-configmap)
- k8s/cluster-local-kind/: local-kind overlay to run the T5 gate without Ref-A
- tidal-stress/k8s/stress-job-t5.yaml: 2-generator sharded throughput job
- tests: cluster_cross_shard_reads.rs + multiproc support; ran real on kind
- docs/profiling/m12p4-t5-sharded-throughput.md: T5 throughput findings
2026-06-14 15:17:35 -06:00
..
guides feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
ops bench(1536): production-shape capacity — read path is cheap, quorum write is the ceiling 2026-06-13 21:40:58 -06:00
planning feat(m11): sharding × replication + rebalancing (m11p6 L3-L5) 2026-06-13 18:23:43 -06:00
profiling feat(m12p4): sharded ingestion — scatter-gather pool + cross-shard unified reads (L4) 2026-06-14 15:17:35 -06:00
research chore: doc consolidation, seven-dimension review fixes, and commit hooks 2026-06-08 22:46:28 -06:00
reviews feat(m11): observability+ops (m11p8) + perf-sweep wave 2 T2 2026-06-13 09:17:49 -06:00
runbooks feat(m11): sharding × replication + rebalancing (m11p6 L3-L5) 2026-06-13 18:23:43 -06:00
specs feat(m12): vector retrieval G1/G2 — recall harness, ANN in RETRIEVE, index tuning 2026-06-14 11:07:09 -06:00
content-strategy.md feat: complete Milestone 5 — full-text search, RRF fusion, and creator search 2026-02-21 23:53:16 -07:00
personal-briefing-beachhead.md feat: M0p1 runtime skeleton, M0p2 tooling & diagnostics, m1p4 signal ledger 2026-02-20 20:32:00 -07:00
README.md feat(m11): membership, snapshot install, and reseed (m11p5) 2026-06-12 19:55:54 -06:00
roadmap-to-cluster.md feat(m11): sharding × replication + rebalancing (m11p6 L3-L5) 2026-06-13 18:23:43 -06:00

README.md

tidalDB Engineering Docs

The engineering documentation home. Top-level product docs (VISION, USE_CASES, SEQUENCE, ARCHITECTURE, API, QUICKSTART, CODING_GUIDELINES, thoughts) live at the repository root; everything below is the deeper engineering record.

This and the repo root are the two canonical doc homes. There is intentionally no per-crate doc mirror (no tidal/docs/). Edit the canonical file, never a copy.

Component specs — specs/

The authoritative component specifications (status: Implemented, M0–M8).

# Spec # Spec
00 Architecture overview 08 Query engine
01 Storage engine 09 Ranking & scoring
02 Entity model 10 Feedback loop
03 Signal system 11 Schema
04 Relationships 12 Cold start
05 Cohorts 13 Concurrency
06 Text retrieval 14 Scale architecture
07 Vector retrieval

Planning — planning/

Code reviews — reviews/

Guides — guides/

Task-oriented, build-an-app docs (complements the root QUICKSTART.md and API.md):

Operations — ops/ and runbooks/

Research — research/

ANN (1, 2) · Tantivy (1, 2) · Signal ledger (1, 2) · WAL · Type system · Tooling & diagnostics · Enterprise-readiness risks

Profiling — profiling/

Hotspot analysis · Scale baselines · Signal memory · Signal rollup eval · Social scale · Tantivy merge tuning · USearch tuning

Strategy

Content strategy · Personal-briefing beachhead