jx12n
005e292cbb
fix(m11): review remediation + tidal-stress perf sweep + perf wave 2
...
Resolve all BLOCKER/CRITICAL/WARNING findings from the m11p7/p8 review:
- tidalctl restore: safe_join path-traversal/Zip-Slip guard + fsync on write
- corrupt-WAL checkpoint_seq guard; PITR archive-before-delete
- cluster: x-tidal-relayed audit-dedup marker; forward_failures counts 5xx
- mTLS/HTTP-TLS handshake hardening; accept-loop EMFILE backoff
- per-principal rate-limit + node-token marker-pinning tests
- self-heal tier-3 coverage; 5 router-auth tests
tidal-stress: measurement-fidelity fixes (schedule-lag p99/max, exact
feed-over-SLO verdict, shed annotation) + typed Body, workload.next
184ns->68ns, RoundRobin len==1 short-circuit, HeaderValue cache;
new benches/hotpath.rs + lib.rs.
perf wave 2: signal_snapshot SmallVec/SignalKey carrier; one-get-per-type
ranking pre-pass.
2026-06-13 12:28:04 -06:00
jx12n
d5d1e7d81a
feat(m11): observability+ops (m11p8) + perf-sweep wave 2 T2
...
m11p8 closes G-O + §1.4-3:
- Cluster metrics: breaker state, forwards, self-heal on /metrics; multi-shard sibling render (shard="N")
- Grafana cluster row + 8-rule Prometheus alert group
- Request-id / TraceLayer on both cluster routers; id rides forward hop
- Truthful status: flushed leader applied_events frontier; post-promote ShardId(0) keying fix
- Self-driving heal: tick_self_heal re-arms stuck-peer backlog every ~3s
- WAL PITR: wal.archive_dir, archive-before-delete gap-free
- tidalctl backup/restore with BLAKE3 content-hash verification
- Rolling-upgrade build_version handshake (N/N+1, never rejects) + Woodpecker release gate
perf-sweep wave 2 T2: one-get-per-type pre-pass in ranking executor
- signal_values.rs pre-fetches all signal kinds before scoring loop
- Eliminates per-item repeated DashMap lookups: −18.8% for_you, −31% under writes
- Byte-identical output verified with A/B test harness
2026-06-13 09:17:49 -06:00
jx12n
225751d34d
feat(m11): WAL-as-stream replication + perf floor (m11p1+m11p2)
...
m11p1 — decoupled ack/ship path: staged writes (seqno+WAL+relay-push,
microseconds) separate from group-commit fsync; ShipQueue batches+windows
outbound segments; receiver coalesces inbound chunks before applying.
Adds first tidaldb_cluster_* metrics.
m11p2 — leader WAL is now THE replicated log: fsynced batches feed a
bounded WalShipFeed and ship byte-identical to followers; WAL seqnos
survive restarts (relay-reset hazard gone). Item metadata and embeddings
journal kind-1/2 blob records on the same stream as signals; the m8p10
HTTP broadcast is deleted. StreamSegments catch-up is follower-pulled via
server-streaming RPC, triggered on gap detection, follower boot, and
leader heal nudge. Promote carries a stream baseline so peers skip
pre-stream history.
2026-06-11 09:10:06 -06:00
jx12n
b55ad70141
fix: M0-M10 third-pass remediation — durability, replication, and CLI hardening
...
Resolves the 142 findings from tidal/docs/reviews/CODE_REVIEW_m0-m10.md across
the engine, server, net, and CLI surfaces:
- WAL/session-journal durability, checkpoint format, and crash-recovery hardening
- Replication shipper/receiver, tenant isolation, and migration paths
- Cluster scatter-gather, router, standalone server + health/offload endpoints
- tidalctl refactored into command modules with JSON output and WAL-state tooling
- Cohort, governance, signal-ledger, and vector-registry correctness fixes
- Expanded UAT/integration/durability test coverage across all milestones
2026-06-08 10:28:34 -06:00
jx12n
3bcfb3c576
feat: Bazel build, crate docs/ai-lookup, docker images, and engine hardening
...
- Add BUILD.bazel across tidal, tidal-net, tidal-server, tidalctl for bzlmod build
- Add tidal/ crate docs (README, CHANGELOG, CONTRIBUTING, AGENTS, CLAUDE, API, ARCHITECTURE) and ai-lookup reference
- Add docker standalone/cluster/deploy images, compose, and prometheus config
- Harden WAL (batch format, writer, dedup, diagnostics), text syncer/collectors, and vector registry
- Expand tidalctl CLI and tests; restructure WAL/visibility integration test suites
- Refine tidal-net transport/client/server and tidal-server cluster/scatter-gather
2026-06-07 18:29:38 -06:00
jordan
4f076c927d
feat: M0p1 runtime skeleton, M0p2 tooling & diagnostics, m1p4 signal ledger
...
## M0p1 — Embeddable Runtime Skeleton (329 tests)
- TidalDb with builder(), health_check(), close(), and Drop-based cleanup
- TidalDbBuilder fluent API: ephemeral(), with_data_dir(), wal_dir(), cache_dir()
- Config, StorageMode, ConfigError types; Config(ConfigError) variant on LumenError
- Paths: single source of truth for directory layout (wal, items, users, creators, cache)
- TempTidalHome: test isolation helper gated behind #[cfg(test)] / test-utils feature
- 8 integration tests: tests/sandboxed_storage.rs
## M0p2 — Tooling & Diagnostics (349 tests)
- Workspace root Cargo.toml (members: ["tidal", "tidalctl"])
- tidal/build.rs: BUILD_HASH from GIT_HASH with option_env!() fallback to "dev"
- MetricsState: always-compiled Arc-shared atomics (uptime, health_ok)
- MetricsHandle (metrics feature): hand-rolled TcpListener HTTP, zero new deps
- GET /healthz → {"status":"ok","uptime_secs":N}
- GET /metrics → Prometheus text (tidaldb_uptime_seconds, health_ok, info)
- TidalDbBuilder.enable_metrics(addr) starts background metrics thread
- tidalctl binary: status + paths commands, manual std::env::args() parsing
- 7 metrics integration tests, 9 tidalctl CLI tests
## m1p4 Signal Ledger (in-progress)
- SignalLedger: DashMap<(EntityId, SignalTypeId), EntitySignalEntry>, WAL-first writes
- HotSignalState: #[repr(C, align(64))], lock-free CAS decay, out-of-order handling
- BucketedCounter: 60 per-minute + 168 per-hour circular buffers, trigger-based rotation
- CheckpointMeta + serialize/restore: 983-byte fixed records, atomic WriteBatch
- Property tests: running score matches analytical to 1e-6, decay monotonic, non-negative
- Proptest regression: signals/warm.txt
## Documentation and planning
- ROADMAP: m0p1 COMPLETE (329), m0p2 COMPLETE (349), product track milestones
- PRODUCT_ROADMAP: P0-P4 product milestone track (personal briefing beachhead)
- Milestone planning docs: milestone-0 (phases 1-3), milestone-p (phases 1-5)
- docs/research/tidaldb_tooling_and_diagnostics.md
- ARCHITECTURE.md, CLAUDE.md, VISION.md updates
## Site
- Blog: every-platform-builds-the-same-6-systems.mdx (new)
- Blog: why-tidaldb.mdx (updated)
- next.config.ts, layout.tsx, blog/page.tsx updates
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-20 20:32:00 -07:00
