Commit Graph

9 Commits

Author SHA1 Message Date
jx12n
b55ad70141 fix: M0-M10 third-pass remediation — durability, replication, and CLI hardening
Resolves the 142 findings from tidal/docs/reviews/CODE_REVIEW_m0-m10.md across
the engine, server, net, and CLI surfaces:

- WAL/session-journal durability, checkpoint format, and crash-recovery hardening
- Replication shipper/receiver, tenant isolation, and migration paths
- Cluster scatter-gather, router, standalone server + health/offload endpoints
- tidalctl refactored into command modules with JSON output and WAL-state tooling
- Cohort, governance, signal-ledger, and vector-registry correctness fixes
- Expanded UAT/integration/durability test coverage across all milestones
2026-06-08 10:28:34 -06:00
jx12n
3bcfb3c576 feat: Bazel build, crate docs/ai-lookup, docker images, and engine hardening
- Add BUILD.bazel across tidal, tidal-net, tidal-server, tidalctl for bzlmod build
- Add tidal/ crate docs (README, CHANGELOG, CONTRIBUTING, AGENTS, CLAUDE, API, ARCHITECTURE) and ai-lookup reference
- Add docker standalone/cluster/deploy images, compose, and prometheus config
- Harden WAL (batch format, writer, dedup, diagnostics), text syncer/collectors, and vector registry
- Expand tidalctl CLI and tests; restructure WAL/visibility integration test suites
- Refine tidal-net transport/client/server and tidal-server cluster/scatter-gather
2026-06-07 18:29:38 -06:00
jordan.washburn
fe711870be feat: M8 phases 7-10 — gRPC transport, cluster server, scatter-gather, multi-node UAT
Delivers the distributed fabric's network layer and HTTP cluster surface:

**m8p7: tidal-net crate (gRPC transport)**
- GrpcTransport implementing Transport trait via tonic 0.12
- Per-peer circuit breaker (Closed/Open/HalfOpen), mutual TLS via rustls
- Boxed error types (clippy-clean), graceful mutex recovery, debug_assert
  against calling block_on from tokio context
- Proto: WalShipping service (ShipSegment, StreamSegments stub, Heartbeat)
- 19 tests: contract, mTLS, reconnection, multi-node UAT, benchmarks

**m8p8: cluster subcommand + HTTP routes**
- ClusterState wrapping SimulatedCluster with region name mapping
- Routes: /health, /cluster/status, /cluster/promote, /partition, /heal
- Data routes: /items, /embeddings, /signals, /feed, /search (region-aware)
- Ranking profiles wired through ClusterConfig to all cluster nodes
- Topology YAML config, docker/cluster/Dockerfile (ENTRYPOINT+CMD, non-root)

**m8p9: scatter-gather query routing**
- Entity-sharded writes via Knuth multiplicative hash
- Scatter-gather RETRIEVE and SEARCH with deadline propagation (50ms-5ms)
- Partial failure: degraded=true with unavailable_shards metadata
- 6 tests: distribution, determinism, multi-shard retrieve, degraded
  partial results, deadline propagation, scatter-gather search

**m8p10: gRPC transport integration tests**
- 8 tests over real gRPC: replication convergence, idempotent replay,
  mixed signals, 3-node fan-out, partition/heal, degraded follower, perf
- Documented as tier-2 (in-process+gRPC); tier-3 multi-process pending
2026-04-11 13:51:08 -06:00
jordan
006d3d058a fix: populate signal_snapshot in ranking results and fix quickstart demo
- Score and boost contributions now captured per candidate in signal_snapshot:
  score_by_sort() returns (f64, Vec<(String, f64)>) with named signal values
  (e.g. view_velocity, share_velocity, view, like). Boost contributions are
  appended as `{signal}_boost`. compute_raw_score() threads the snapshot
  through; score_inner() and score_personalized() write it onto ScoredCandidate
  instead of always initialising to vec![].

- Quickstart switched from `trending` to `hot` profile. The trending profile
  scores by 24h velocity, which requires signals to arrive over real elapsed
  time to populate hour-level bucket aggregates — impossible in a self-contained
  demo. The hot profile (AllTime view count + age decay) works with any timestamp
  and produces clearly differentiated scores. Signals now use different counts
  per item to drive meaningful ranking output.
2026-03-10 17:06:21 -06:00
Alan Kahn
1d826c87b2 feat: schema-level ranking profile definitions
Add support for defining ranking profiles in schema YAML, allowing
deployments to override builtin profiles with deployment-specific
signal names and tuning parameters.

- Add override_register() to ProfileRegistry for clean builtin replacement
- Add with_profiles() to TidalDbBuilder to thread schema profiles
- Parse profiles section in config.rs with full sort/strategy/agg support
- Validate that profile signal references exist in schema at startup
- Change load_schema() to return (Schema, Vec<RankingProfile>)

This closes the gap where the builtin for_you profile referenced
signals (view, like, share) that don't exist in deployment schemas,
causing all feed scores to normalize to 1.0.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 12:12:54 -05:00
jordan
213b8efcca feat: complete M6-M7 + Enterprise Readiness milestones; split oversized source files per CODING_GUIDELINES §9
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-23 22:41:16 -07:00
jordan
192c473f55 feat: complete Milestone 5 — full-text search, RRF fusion, and creator search
- M5p1: BM25 text indexing via Tantivy with background syncer (0.26ms @ 10K docs)
- M5p2: RRF fusion layer combining BM25 + ANN scores (46µs @ 1K candidates)
- M5p3: unified Search query API (8-stage pipeline, BM25 + vector + ranking)
- M5p4: creator text + vector indexing and creator search executor (< 20ms @ 200 creators)
- Refactor db/mod.rs into focused sub-modules (creators, items, sessions, signals, etc.)
- Decompose monolithic files into directory modules (query/executor, ranking/diversity, etc.)
- Split brute.rs → brute/mod.rs + brute/tests.rs; extract search executor helpers
- Add benches: fusion, search, session, text_index
- Add M5 UAT test suites (m5_uat, m5_search, m5p4_creator_search, text_index)
- Update blog posts, roadmap, content strategy, and M5 planning docs
- Add tmp/ and .claude/worktrees/ to .gitignore

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-21 23:53:16 -07:00
jordan
39ada28c6e feat: complete Milestones 2–4 — RETRIEVE query, vector index, ranking profiles, diversity, entity system, sessions
M2: RETRIEVE query pipeline with 5-stage execution (candidate → filter → score → diversify → limit),
    usearch HNSW vector index, bitmap/range/universe filters, ranking profiles with signal scoring,
    MMR diversity enforcement, and m2_uat integration tests.

M3: Entity system with typed metadata, relationship graph (follows/blocks/interactions),
    creator entities, session tracking, and m3_uat integration tests.

M4: Advanced ranking with builtin functions (freshness, trending, controversy, wilson),
    ranking executor with explain mode, query executor integration, benchmarks for
    query/ranking/vector/filters/diversity, and m4_uat integration tests.

Includes: 9 new blog posts, marketing site updates, updated roadmap, and updated vision doc.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-21 16:24:48 -07:00
jordan
413b712c0a chore: initialize tidalDB repository with schema foundation and standards
- Schema phase 1 (tasks 01-02): EntityId, EntityKind, Timestamp, Score, SignalTypeDef, DecayModel, Window, WindowSet — all with property tests and benchmarks scaffolding
- Stub modules for storage, signals, query, ranking
- Full documentation suite: VISION, USE_CASES, SEQUENCE, API, CODING_GUIDELINES, ai-lookup, research docs, specs, roadmap, planning docs
- Marketing site (Next.js) with blog infrastructure
- .claude/ agents and skills for the tidalDB development workflow
- Foundation standards enforced: thiserror + tracing declared as dependencies, clippy::unwrap_used = deny added to lint config
- .gitignore hardened: .next/, node_modules/, .env, secrets, logs

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-20 12:52:20 -07:00