jordan/tidaldb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
143 Commits 1 Branch 0 Tags 14 MiB
6ad8c51cfa
Commit Graph

4 Commits

Author SHA1 Message Date
jx12n
4051077cff docs(m12): refresh API, specs, ops, and roadmap to the shipped M12 reality 
- API.md: document `similar_to`/`region`/`unavailable_shards` on /feed and
  /search, the new POST /vector_search k-NN probe, and the cluster-node-only
  routes (/cluster/*, /sharded/*, /hardnegs)
- CHANGELOG.md: M12 entries — multi-vector preference + ANN candidate-gen,
  idle-readiness + TLS scale-up (m12p5/p6), sharded ingestion (m12p4)
- ROADMAP.md: mark M11 + M12 COMPLETE; restate the v1.0 bar (30-day-green
  nightly calendar + Ref-A/k3s throughput re-runs)
- prometheus-alerts.yaml: add ship-stall, quorum-lag, divergence-quarantine,
  reseed-pending, and snapshot-pin-force-drop cluster alerts
- check-docs.sh: self-updating milestone-status freshness guard derived from
  ROADMAP's latest COMPLETE milestone
- refresh specs (00-14), ai-lookup, guides, and runbooks to M0-M12
 2026-06-23 21:39:55 -06:00
jx12n
6a937fc4bc feat(m12): multi-vector user preference modeling + ANN candidate-gen 
Add multi-vector preference entity (per-signal-type preference vectors with
event-time decay) feeding ANN candidate generation in the query executor.

- entities: multi_preference vectors + event-time-aware preference updates
- query/executor: ANN candidate-gen + personalization/pipeline integration
- storage/keys, db ops, state_rebuild: persist & rebuild multi-vector prefs
- ranking: profile + builtins support for multi-vector scoring
- tidal-server/config: expose multi-preference knobs
- tests/bench: m12_preference_event_time integration + multi_preference bench
- docs: multi-vector-preference research, ROADMAP/ARCHITECTURE refresh,
  legal/tidaldb-patent-proposal
- .codex/agents: codex agent definitions
- chore: gitignore tool-regenerated .agents/ mirror (doc-guard rejects it)
 2026-06-23 09:52:36 -06:00
jordan
4f076c927d feat: M0p1 runtime skeleton, M0p2 tooling & diagnostics, m1p4 signal ledger 
## M0p1 — Embeddable Runtime Skeleton (329 tests)
- TidalDb with builder(), health_check(), close(), and Drop-based cleanup
- TidalDbBuilder fluent API: ephemeral(), with_data_dir(), wal_dir(), cache_dir()
- Config, StorageMode, ConfigError types; Config(ConfigError) variant on LumenError
- Paths: single source of truth for directory layout (wal, items, users, creators, cache)
- TempTidalHome: test isolation helper gated behind #[cfg(test)] / test-utils feature
- 8 integration tests: tests/sandboxed_storage.rs

## M0p2 — Tooling & Diagnostics (349 tests)
- Workspace root Cargo.toml (members: ["tidal", "tidalctl"])
- tidal/build.rs: BUILD_HASH from GIT_HASH with option_env!() fallback to "dev"
- MetricsState: always-compiled Arc-shared atomics (uptime, health_ok)
- MetricsHandle (metrics feature): hand-rolled TcpListener HTTP, zero new deps
  - GET /healthz → {"status":"ok","uptime_secs":N}
  - GET /metrics → Prometheus text (tidaldb_uptime_seconds, health_ok, info)
- TidalDbBuilder.enable_metrics(addr) starts background metrics thread
- tidalctl binary: status + paths commands, manual std::env::args() parsing
- 7 metrics integration tests, 9 tidalctl CLI tests

## m1p4 Signal Ledger (in-progress)
- SignalLedger: DashMap<(EntityId, SignalTypeId), EntitySignalEntry>, WAL-first writes
- HotSignalState: #[repr(C, align(64))], lock-free CAS decay, out-of-order handling
- BucketedCounter: 60 per-minute + 168 per-hour circular buffers, trigger-based rotation
- CheckpointMeta + serialize/restore: 983-byte fixed records, atomic WriteBatch
- Property tests: running score matches analytical to 1e-6, decay monotonic, non-negative
- Proptest regression: signals/warm.txt

## Documentation and planning
- ROADMAP: m0p1 COMPLETE (329), m0p2 COMPLETE (349), product track milestones
- PRODUCT_ROADMAP: P0-P4 product milestone track (personal briefing beachhead)
- Milestone planning docs: milestone-0 (phases 1-3), milestone-p (phases 1-5)
- docs/research/tidaldb_tooling_and_diagnostics.md
- ARCHITECTURE.md, CLAUDE.md, VISION.md updates

## Site
- Blog: every-platform-builds-the-same-6-systems.mdx (new)
- Blog: why-tidaldb.mdx (updated)
- next.config.ts, layout.tsx, blog/page.tsx updates

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-02-20 20:32:00 -07:00
jordan
29400d48db feat: implement Milestone 1 phases 1-3 — schema, WAL, and storage layer 
Implements the foundation of tidalDB's data pipeline:

**Phase 1 – Schema primitives**
- EntityId newtype (u64, big-endian ordering)
- SignalTypeDefinition with pre-computed decay λ, deduped/sorted windows
- SchemaBuilder with full constraint validation (duplicates, identifiers,
  half-life, windows, velocity)
- LumenError wrapping all subsystems with required From impls

**Phase 2 – Write-Ahead Log**
- Length-prefixed, BLAKE3-protected entry format
- Group-commit writer (batch up to 100 events / 10 ms)
- Double-buffered content-hash deduplication
- Checkpoint, truncation, and crash-recovery with full replay
- Integration, property, and UAT tests (incl. 5,500-event deterministic UAT)
- Proptest coverage scaled to 10 000 events/run (was ≤500) to meet
  acceptance criterion; cases reduced 100→10 to keep runtime comparable

**Phase 3 – Storage engine**
- StorageEngine trait (get/put/delete/scan/batch/flush)
- Key encoding: [EntityId][0x00][Tag][suffix] with ordering/prefix helpers
- InMemoryBackend (BTreeMap + RwLock)
- FjallStorage with three isolated keyspaces and atomic batch helper
- Property tests for key ordering and round-trip correctness

Also adds planning docs for phases 4-5, research docs, architecture
overview, and roadmap updates.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-02-20 16:43:24 -07:00