History

jx12n 4db3f1e597 fix(m12p6): complete T4 TLS scale-up — two-tier PKI + join_boot grpc_tls fallback Completes the seed-join-over-TLS enablement begun in `8e39ee1`. A real kubectl scale 3->5 on a real mTLS k8s cluster (kind) exercised the seed-join path over TLS for the first time and surfaced two more blockers beyond 8e39ee1's https-seed / ready-only-Service / up-front-rustls-provider fixes — both of which crash-looped every scale-up joiner with the same opaque 'could not join within 120s'. The plaintext in-process harness is blind to all of them. - certs.yaml: a real TWO-TIER PKI. The leaf was issued DIRECTLY from a selfSigned Issuer (a self-signed CA:FALSE end-entity whose ca.crt is a copy of the leaf); the joiner's strict webpki verifier rejected the peer cert as UnknownIssuer. Now: selfSigned Issuer -> CA cert (CA:TRUE) -> ca: Issuer signs the leaf. (scripts/gen-cluster-certs.sh already did this; the two were inconsistent.) - join_boot.rs: grpc_tls_for() fallback. own_grpc_tls/self_tls_spec looked up the joiner's OWN region in the knob file to find its TLS material, but a seed-joiner is NEVER in the shared-ConfigMap regions: list -> None -> the seed client built with NO CA (the real UnknownIssuer cause) and a plaintext synthesized topology. Fall back to ANY region's block (every pod mounts the same cert files). - join_boot.rs: STATUS_POLL_TIMEOUT 500ms -> 5s (env TIDAL_SEED_STATUS_TIMEOUT_MS); a cold TLS handshake under contention blew the sub-second budget. Discovery now logs each poll failure at WARN with the full error source chain (a silent loop made every bug present as the same 120s timeout). - statefulset.yaml: pin the m12-8e39ee1 server image (carries these fixes). - k8s/cluster-t4-kind + tidal-stress/k8s/t4-*: local-kind T4 overlay + seed/load. Verified GREEN on kind: idle scale 3->5, both joiners seed-join over mTLS, catch up, and flip /health Ready in 13s via the idle-readiness heartbeat convergence; auto-promote to Voter; full content parity; all 5 regions lag=0. clippy clean; mp_seed_join_snapshot_catchup + mp_idle_cluster_..._without_traffic green; tidal-server/tidal-net lib green. A separate, root-caused snapshot-frontier bug on a DEEPLY-compacted WAL (node.rs:734 last_wal_seq=0 for a state-only artifact) is documented as a follow-up — left unfixed because a naive patch broke the in-process snapshot test (own-WAL<->stream numbering); the GREEN run uses a small corpus (stream catch-up) to keep that path out of scope. See docs/profiling/m12p5-idle-readiness-elasticity.md §6.		2026-06-14 22:41:59 -06:00
..
guides	feat(m11): cluster security (m11p7) + perf instrumentation floor	2026-06-13 01:25:35 -06:00
ops	bench(1536): production-shape capacity — read path is cheap, quorum write is the ceiling	2026-06-13 21:40:58 -06:00
planning	feat(m11): sharding × replication + rebalancing (m11p6 L3-L5)	2026-06-13 18:23:43 -06:00
profiling	fix(m12p6): complete T4 TLS scale-up — two-tier PKI + join_boot grpc_tls fallback	2026-06-14 22:41:59 -06:00
research	chore: doc consolidation, seven-dimension review fixes, and commit hooks	2026-06-08 22:46:28 -06:00
reviews	feat(m11): observability+ops (m11p8) + perf-sweep wave 2 T2	2026-06-13 09:17:49 -06:00
runbooks	fix(m12p6): T4 TLS scale-up enablement — https seed-join via ready-only Service + up-front rustls provider	2026-06-14 20:29:37 -06:00
specs	feat(m12): vector retrieval G1/G2 — recall harness, ANN in RETRIEVE, index tuning	2026-06-14 11:07:09 -06:00
content-strategy.md	feat: complete Milestone 5 — full-text search, RRF fusion, and creator search	2026-02-21 23:53:16 -07:00
personal-briefing-beachhead.md	feat: M0p1 runtime skeleton, M0p2 tooling & diagnostics, m1p4 signal ledger	2026-02-20 20:32:00 -07:00
README.md	feat(m11): membership, snapshot install, and reseed (m11p5)	2026-06-12 19:55:54 -06:00
roadmap-to-cluster.md	feat(m11): sharding × replication + rebalancing (m11p6 L3-L5)	2026-06-13 18:23:43 -06:00

README.md

tidalDB Engineering Docs

The engineering documentation home. Top-level product docs (VISION, USE_CASES, SEQUENCE, ARCHITECTURE, API, QUICKSTART, CODING_GUIDELINES, thoughts) live at the repository root; everything below is the deeper engineering record.

This and the repo root are the two canonical doc homes. There is intentionally no per-crate doc mirror (no tidal/docs/). Edit the canonical file, never a copy.

Component specs — `specs/`

The authoritative component specifications (status: Implemented, M0–M8).

#	Spec	#	Spec
00	Architecture overview	08	Query engine
01	Storage engine	09	Ranking & scoring
02	Entity model	10	Feedback loop
03	Signal system	11	Schema
04	Relationships	12	Cold start
05	Cohorts	13	Concurrency
06	Text retrieval	14	Scale architecture
07	Vector retrieval

Planning — `planning/`

ROADMAP.md — milestones M0–M11, phase status, known gaps
roadmap-to-cluster.md — adopted M11 plan: gap analysis + phase specs taking the multi-process cluster from experimental to enterprise-grade (m11p1–p5 ✅; p6–p9 planned), grounded in the 2026-06-10 live stress-test baselines
PRODUCT_ROADMAP.md · architecture-review.md · roadmap-cohort-analysis.md · site-cohort-analysis.md
Per-milestone phase/task archive: planning/milestone-0,1,2,3,5,7,8,9,10,11,p/

Code reviews — `reviews/`

M0–M10 code review — 2026-06-07 — seven-dimension re-review, 88 verified findings
M0–M10 code review — 2026-06-08 — seven-dimension review, 142 findings (latest pass)
M0–M10 seven-dimension review — additional pass (2 BLOCKERs: signal-checkpoint trim, 30-day window)

Guides — `guides/`

Task-oriented, build-an-app docs (complements the root QUICKSTART.md and API.md):

Build a feed app — end-to-end TikTok/Reels-style "For You" feed, embedded and over HTTP
Embedding integration — wiring a real embedding model (OpenAI / Cohere / local) into the write + query paths
Server deployment — running the tidal-server HTTP service: config, auth, the served OpenAPI spec, Docker
Ranking-profile reference: ai-lookup/services/ranking-profiles.md — all 25 built-in profiles