Commit Graph

11 Commits

Author SHA1 Message Date
jordan
cdbe9cb453 Merge remote-tracking branch 'origin/main' (m11/m12 cluster) into m9/m10
Reconciles two independently-developed lines from base 006d3d0:
  ours   — M9/M10 community layers, retroactive purge + re-materialization,
           signal revocation, agent capability boundaries, P1 feedback loop,
           reason labels, instrumented metrics
  theirs — M11/M12 cluster mode (tidal-net gRPC transport, tidal-server
           cluster/scatter-gather, tidal-stress), multi-vector preference,
           ANN candidate-gen, warm-tier day buckets, keyed signal snapshots

Notable semantic resolutions:

* storage::keys::Tag — both sides allocated 0x0E..0x11 for different
  records. Kept theirs' 0x0E..0x1A (shipped on-disk format) and renumbered
  ours to 0x1B..0x1E (CommunityMembership/Revocation/PurgeManifest/
  CommunityLeave); Tag::ALL grown to 30 so the contiguity drift guard holds.

* ranking executor — took theirs' rewrite (SignalReadPlan pre-pass, keyed
  SignalKey snapshots, Result-returning reads, finalize()) and re-applied
  ours' M10 read suppression at the chokepoints it introduced:
  single_signal_score, score_hot/trending/controversial,
  CreatorEngagementRate, and the Stage-4 boost loop.

* signals::warm — theirs' day-bucket/read-time-rotation rewrite, with ours'
  subtract_bucket and Clone extended to the new day tier; ours' test split
  kept (warm/tests.rs, warm/proptests.rs) carrying theirs' updated bodies.

* db::signals — kept ours' contribution-logging try_cohort_attribution in
  signal_dispatch.rs and theirs' event-time try_update_preference_vector;
  dropped the superseded duplicates.

* db::mod / from_parts — theirs' constructors, with ours' purge/
  re-materialization/revocation/community/skip-counter fields and restart
  rebuilds; from_parts kept in its own file per the 600-line guideline.

* schema::validation::builders — ours' module split with theirs' expanded
  tests; policy validation runs both sides' checks (read-signal lists +
  profile overrides, then the zero-duration limit guard).

* feedback Unhide no longer writes a -1.0 "hide" signal: theirs' engine
  rejects negative weights (spec §8). Reverses index state only, matching
  every other undo action.

* SessionState::new is now the single construction path (gains
  overrides_rejected/default_profile); AuditEntry gains kind on the
  deserialize path, inferred from the accepted flag as before.

* Removed tidal/src/replication/tcp_transport.rs and its test: never
  declared in replication/mod.rs on either branch, so it had never
  compiled and nothing referenced it. Superseded by tidal-net's
  GrpcTransport.

Verified: cargo clippy -p tidaldb (lib) clean; --all-targets compiles for
tidaldb/tidal-net/tidal-server/tidal-stress; 2094/2094 lib tests and the
integration suite pass except m8p3_reconcile_production's two CRDT-count
assertions, which fail identically on MERGE_HEAD (pre-existing).
tidalctl cannot build locally: its aws-sdk deps need rustc 1.91.1, local
toolchain is 1.91.0.
2026-08-03 02:16:04 -06:00
jx12n
6a937fc4bc feat(m12): multi-vector user preference modeling + ANN candidate-gen
Add multi-vector preference entity (per-signal-type preference vectors with
event-time decay) feeding ANN candidate generation in the query executor.

- entities: multi_preference vectors + event-time-aware preference updates
- query/executor: ANN candidate-gen + personalization/pipeline integration
- storage/keys, db ops, state_rebuild: persist & rebuild multi-vector prefs
- ranking: profile + builtins support for multi-vector scoring
- tidal-server/config: expose multi-preference knobs
- tests/bench: m12_preference_event_time integration + multi_preference bench
- docs: multi-vector-preference research, ROADMAP/ARCHITECTURE refresh,
  legal/tidaldb-patent-proposal
- .codex/agents: codex agent definitions
- chore: gitignore tool-regenerated .agents/ mirror (doc-guard rejects it)
2026-06-23 09:52:36 -06:00
jx12n
bb21e69ae6 feat(m12): vector retrieval G1/G2 — recall harness, ANN in RETRIEVE, index tuning
m12p1 (measurement truth): TidalDb::vector_search_items pure k-NN probe +
POST /vector_search (standalone + region node, merge-by-distance) +
tidal-stress --verify-recall (deterministic id-keyed corpus, in-RAM brute-force
cosine oracle, open-loop ramp → recall@k + true p99 + read-knee + JSON/gate exit).
Repaired fabricated p99 columns (mean-as-p99) in social-scale.md / scale.rs.
Verified real: recall@10=0.9997 at 20k/1536-D vs brute-force.

m12p2 (G1 unblock): ANN candidate-gen wired into RETRIEVE — for_you=preference
vector, related=seed embedding (similar_to), graceful scan-fallback. Cached
per-signal-type top-K (signals/ledger/hot_top_k.rs, decay-order-invariant) so
trending serves O(K). related over HTTP (FeedQuery.similar_to). Harness gains
--feed-profile / --seed-preferences. Verified: trending retrieve p99 3.5-7.7ms.

m12p3 (G2): per-query ef_search now honored (RwLock epoch-guard with_expansion,
shared guard for same-ef concurrency) + dimension-aware brute→HNSW crossover
usearch_min_vectors(dim) + memory_usage() + examples/ann_grid_search.rs.
Measured 1536-D/100k clustered: default M=16/ef_c=400/F16/ef_s=200 clears
G1+G2 (recall 0.997, p99 1.4ms); F16 -0.25% vs F32; Int8 rejected (-28%).
Recall corpus is now clustered (Gaussian mixture) in grid + harness.
2026-06-14 11:07:09 -06:00
jx12n
8673723319 perf(m11): kill signal_snapshot allocation cascade (perf-sweep wave 2 T1)
Replace ScoredCandidate.signal_snapshot Vec<(String,f64)> with
SmallVec<[(SignalKey,f64); 4]> where SignalKey is Static(&'static str)
| Owned(Arc<str>):

- Compile-time-constant labels (sort bases, relevance, co_engagement,
  preference_affinity) -> Static: zero allocation, pointer-copy clone.
- Dynamic {signal}_boost/_penalty/_decay labels built once per query in a
  RuleLabels hoist (was format! per-candidate-per-rule), shared by Arc.
  Cohort rescore hoisted the same way.
- scored accumulator pre-sized to candidates.len().
- Owned Strings rebuilt only at the two response-assembly sites (<= limit).

Byte-identical output: full 1896-test lib suite green. Measured win
(cargo bench --bench ranking, committed-base vs working-tree):
score_200_hot 23.89->21.04us (-11.9%), score_200_trending
27.66->25.85us (-6.5%), score_200_full_pipeline 27.88->26.97us (-3.3%).

smallvec promoted from the lock to a direct dep (no new dependency
surface). perf-sweep doc updated; T2 (per-term DashMap collapse) next.
2026-06-13 01:53:08 -06:00
jx12n
9728194f16 fix: M0-M10 code-review pass2 remediation — all 91 findings
Resolves every finding in docs/reviews/M0-M10-code-review-2026-06-08-pass2.md
across the engine, network, server, and CLI crates: session restore,
replication/CRDT, WAL format and recovery, storage indexes, query/ranking
executors, cohort/community governance, and scatter-gather routing.

Adds regression tests:
- review_pass2_creator_search_filter
- review_pass2_d_replication
- review_pass2_query_for_session
- review_pass2_storage_indexes_bitmap_cache
- review_pass2_zone_a_sessions

Verified: cargo clippy -D warnings and full test suite green across all crates.
2026-06-09 12:21:00 -06:00
jx12n
ad4134e280 chore: doc consolidation, seven-dimension review fixes, and commit hooks
- Eliminate the tidal/ self-contained doc mirror; docs now have two canonical
  homes (root *.md and docs/), with planning/specs/research/reviews moved up
- Remove stale .agents/skills and .ai mirrors; canonicalize skills under .claude/
- Add pre-commit hook + scripts/check-docs.sh doc-guard + scripts/install-hooks.sh
- Implement M0-M10 seven-dimension review findings across engine, net, server,
  and tidalctl (durability, replication, query, WAL, storage, CLI hardening)
2026-06-08 22:46:28 -06:00
jx12n
b55ad70141 fix: M0-M10 third-pass remediation — durability, replication, and CLI hardening
Resolves the 142 findings from tidal/docs/reviews/CODE_REVIEW_m0-m10.md across
the engine, server, net, and CLI surfaces:

- WAL/session-journal durability, checkpoint format, and crash-recovery hardening
- Replication shipper/receiver, tenant isolation, and migration paths
- Cluster scatter-gather, router, standalone server + health/offload endpoints
- tidalctl refactored into command modules with JSON output and WAL-state tooling
- Cohort, governance, signal-ledger, and vector-registry correctness fixes
- Expanded UAT/integration/durability test coverage across all milestones
2026-06-08 10:28:34 -06:00
jx12n
69ae6e64a1 fix: M0-M10 remediation — deferred post-filters, WAL hardening, docker build fixes
- Extract shared deferred post-filter (InCollection/MinSignal/MaxSignal/
  NearLocation/SocialGraph) into query/executor/post_filter.rs so RETRIEVE and
  SEARCH apply identical predicates; SEARCH previously ignored four variants
- Harden WAL writer/compaction/dedup paths
- Brute-force vector store fixes + tests
- Governance community ledger updates
- Pin docker builders to bookworm (glibc/libmvec match) and add
  protobuf-compiler for tidal-net's tonic-build
- Add M0-M10 code-review document
2026-06-07 20:47:17 -06:00
jx12n
3bcfb3c576 feat: Bazel build, crate docs/ai-lookup, docker images, and engine hardening
- Add BUILD.bazel across tidal, tidal-net, tidal-server, tidalctl for bzlmod build
- Add tidal/ crate docs (README, CHANGELOG, CONTRIBUTING, AGENTS, CLAUDE, API, ARCHITECTURE) and ai-lookup reference
- Add docker standalone/cluster/deploy images, compose, and prometheus config
- Harden WAL (batch format, writer, dedup, diagnostics), text syncer/collectors, and vector registry
- Expand tidalctl CLI and tests; restructure WAL/visibility integration test suites
- Refine tidal-net transport/client/server and tidal-server cluster/scatter-gather
2026-06-07 18:29:38 -06:00
jordan
d8e408350c feat(m9/m10/p1): community policy engine, signal revocation, agent capability boundaries, feedback loop, metrics instrumentation 2026-03-16 05:59:42 -06:00
jordan
213b8efcca feat: complete M6-M7 + Enterprise Readiness milestones; split oversized source files per CODING_GUIDELINES §9
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-23 22:41:16 -07:00