c97aaa8e5b
`cargo test --workspace` could not run at all: dependency resolution failed with "aws-types@1.3.16 requires rustc 1.91.1" on the 1.91.0 default toolchain, so the gate the project documents was dead. Making it run exposed a compile break and two wrong tests that had been invisible for months. Now green end to end: 143 suites, 3155 tests, exit 0. Toolchain - rust-toolchain.toml pins the DEV toolchain to 1.91.1. The published MSRV stays `rust-version = "1.91"` (the engine builds on 1.91.0); only tidalctl's AWS SDK chain needs the patch release, and it now declares that itself. Consumer crates migrated to the current engine API (clean cutover) - iknowyou-engine: `AgentPolicy` gained five m10 read/profile-override fields; the literal now spreads `..AgentPolicy::default()` as the engine's own doc example does, so future fields do not break it again. - forage-engine: `RetrieveResult` gained p1 `reasons`. The app builds its own candidate pool, so it now tags what it knows: PreferenceMatch for the preference-vector blend, SemanticMatch (with the seed item) for similar-to-saved, ExplorationBudget for pinned discoveries. - forage-engine: `url_to_item_id` folded into the u32 item universe. The engine narrows item IDs to a u32 slot in durable per-user state and rejects anything above u32::MAX rather than alias two items forever, so every add_item with a 64-bit FNV hash failed. 9 of 28 smoke tests were failing on this alone. - forage-engine: bridge items read the top-2 preference CLUSTERS via `query_vectors`, not the single centroid from `preference_vectors().get()`. Since m12 that accessor returns only the strongest cluster, so a tech+jazz user whose interests split into two clusters looked single-interest and never bridged. Falls back to top-2 dimensions when a user has one cluster. Reconcile tests corrected to the shipped contract - tidal/tests/m8p3_reconcile_production.rs asserted `3 + 5 == 8` for a windowed count after heal. `take_crdt_snapshot` deliberately keys signal contributions to ONE canonical contributor (ShardId::SINGLE) because signals are relayed from a single writer, so per-node attribution double-counted every replicated event on every reconcile. Merge is therefore LWW on (last_update_ns, score) plus PN-counter per-node max: nodes converge on the more complete accumulator. The old expectation was asserting the bug that fix removed. - Rewrote to assert convergence, count survival (not 0), and no inflation, and added `repeated_reconcile_of_converged_nodes_does_not_creep` - the regression guard for the creep itself, which nothing covered. Pre-commit hook unified - hooks/pre-commit dropped `-D warnings`: each crate's `[lints]` table is the source of truth (`clippy::all`/`unwrap_used` deny, `pedantic` warn), and the flag promoted ~58 deliberate pedantic warnings in integration tests to errors, making every Rust commit impossible. - It now lints all five tidal crates instead of path-matching `tidal/`, which silently skipped tidal-server, tidal-net, tidal-stress, tidalctl and applications/ - the rot above lived in exactly those crates. Ported the CODING_GUIDELINES file-length, println, and unsafe-SAFETY checks from the divergent untracked copy that this replaces. - CONTRIBUTING.md now documents the real commands and the toolchain/MSRV split. Fleet recovery and soak - scripts/restore-fleet.sh: the fail-closed selective restore, promoted out of an ignored tmp/ directory into the repository. Preflights retained storage, digest-pinned images, parked state, and aggregate plus per-PV-node scheduler headroom before the first scale; writes a durable transcript under tmp/restore-logs/ with structured start/error/rollback/complete events. - k8s manifests park the standalone store, the RF3 cluster, and the soak monitor at zero replicas with restore-fleet.sh as the only supported scale-up path. - soak-eval/soak-watch and the nightly CronJob fail closed on stale or missing restart evidence instead of silently skipping the restart-aware half of the gate. - docs/ops/capacity-planning.md corrects the RAM envelope to the real hot-tier formula and separates analytic totals from the measured process envelope. |
||
|---|---|---|
| .. | ||
| guides | ||
| legal | ||
| ops | ||
| planning | ||
| profiling | ||
| research | ||
| reviews | ||
| runbooks | ||
| specs | ||
| content-strategy.md | ||
| personal-briefing-beachhead.md | ||
| README.md | ||
| roadmap-to-cluster.md | ||
tidalDB Engineering Docs
The engineering documentation home. Top-level product docs (VISION, USE_CASES, SEQUENCE, ARCHITECTURE, API, QUICKSTART, CODING_GUIDELINES, thoughts) live at the repository root; everything below is the deeper engineering record.
This and the repo root are the two canonical doc homes. There is intentionally no per-crate doc mirror (no
tidal/docs/). Edit the canonical file, never a copy.
Component specs — specs/
The authoritative component specifications (status: Implemented, M0–M12).
| # | Spec | # | Spec |
|---|---|---|---|
| 00 | Architecture overview | 08 | Query engine |
| 01 | Storage engine | 09 | Ranking & scoring |
| 02 | Entity model | 10 | Feedback loop |
| 03 | Signal system | 11 | Schema |
| 04 | Relationships | 12 | Cold start |
| 05 | Cohorts | 13 | Concurrency |
| 06 | Text retrieval | 14 | Scale architecture |
| 07 | Vector retrieval |
Planning — planning/
- ROADMAP.md — milestones M0–M12, phase status, known gaps
- roadmap-to-cluster.md — adopted M11 plan: gap analysis + phase specs taking the multi-process cluster from experimental to enterprise-grade (all nine phases m11p1–p9 ✅, 2026-06-13), grounded in the 2026-06-10 live stress-test baselines
- PRODUCT_ROADMAP.md · architecture-review.md · roadmap-cohort-analysis.md · site-cohort-analysis.md
- Per-milestone phase/task archive:
planning/milestone-0,1,2,3,5,7,8,9,10,11,p/
Code reviews — reviews/
- M0–M10 code review — 2026-06-07 — seven-dimension re-review, 88 verified findings
- M0–M10 code review — 2026-06-08 — seven-dimension review, 142 findings (latest pass)
- M0–M10 seven-dimension review — additional pass (2 BLOCKERs: signal-checkpoint trim, 30-day window)
Guides — guides/
Task-oriented, build-an-app docs (complements the root QUICKSTART.md and API.md):
- Build a feed app — end-to-end TikTok/Reels-style "For You" feed, embedded and over HTTP
- Embedding integration — wiring a real embedding model (OpenAI / Cohere / local) into the write + query paths
- Server deployment — running the
tidal-serverHTTP service: config, auth, the served OpenAPI spec, Docker - Ranking-profile reference: ai-lookup/services/ranking-profiles.md — all 25 built-in profiles
Operations — ops/ and runbooks/
- Monitoring · Prometheus alerts · Grafana dashboard · Capacity planning · Recovery
- Runbooks: Kubernetes · Cluster (experimental)
Research — research/
ANN (1, 2) · Tantivy (1, 2) · Signal ledger (1, 2) · WAL · Type system · Tooling & diagnostics · Enterprise-readiness risks
Profiling — profiling/
Hotspot analysis · Scale baselines · Signal memory · Signal rollup eval · Social scale · Tantivy merge tuning · USearch tuning