afdda7cc0f
5 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
jordan
|
c97aaa8e5b |
fleet remediation: make the workspace gate runnable, then fix what it caught
`cargo test --workspace` could not run at all: dependency resolution failed with "aws-types@1.3.16 requires rustc 1.91.1" on the 1.91.0 default toolchain, so the gate the project documents was dead. Making it run exposed a compile break and two wrong tests that had been invisible for months. Now green end to end: 143 suites, 3155 tests, exit 0. Toolchain - rust-toolchain.toml pins the DEV toolchain to 1.91.1. The published MSRV stays `rust-version = "1.91"` (the engine builds on 1.91.0); only tidalctl's AWS SDK chain needs the patch release, and it now declares that itself. Consumer crates migrated to the current engine API (clean cutover) - iknowyou-engine: `AgentPolicy` gained five m10 read/profile-override fields; the literal now spreads `..AgentPolicy::default()` as the engine's own doc example does, so future fields do not break it again. - forage-engine: `RetrieveResult` gained p1 `reasons`. The app builds its own candidate pool, so it now tags what it knows: PreferenceMatch for the preference-vector blend, SemanticMatch (with the seed item) for similar-to-saved, ExplorationBudget for pinned discoveries. - forage-engine: `url_to_item_id` folded into the u32 item universe. The engine narrows item IDs to a u32 slot in durable per-user state and rejects anything above u32::MAX rather than alias two items forever, so every add_item with a 64-bit FNV hash failed. 9 of 28 smoke tests were failing on this alone. - forage-engine: bridge items read the top-2 preference CLUSTERS via `query_vectors`, not the single centroid from `preference_vectors().get()`. Since m12 that accessor returns only the strongest cluster, so a tech+jazz user whose interests split into two clusters looked single-interest and never bridged. Falls back to top-2 dimensions when a user has one cluster. Reconcile tests corrected to the shipped contract - tidal/tests/m8p3_reconcile_production.rs asserted `3 + 5 == 8` for a windowed count after heal. `take_crdt_snapshot` deliberately keys signal contributions to ONE canonical contributor (ShardId::SINGLE) because signals are relayed from a single writer, so per-node attribution double-counted every replicated event on every reconcile. Merge is therefore LWW on (last_update_ns, score) plus PN-counter per-node max: nodes converge on the more complete accumulator. The old expectation was asserting the bug that fix removed. - Rewrote to assert convergence, count survival (not 0), and no inflation, and added `repeated_reconcile_of_converged_nodes_does_not_creep` - the regression guard for the creep itself, which nothing covered. Pre-commit hook unified - hooks/pre-commit dropped `-D warnings`: each crate's `[lints]` table is the source of truth (`clippy::all`/`unwrap_used` deny, `pedantic` warn), and the flag promoted ~58 deliberate pedantic warnings in integration tests to errors, making every Rust commit impossible. - It now lints all five tidal crates instead of path-matching `tidal/`, which silently skipped tidal-server, tidal-net, tidal-stress, tidalctl and applications/ - the rot above lived in exactly those crates. Ported the CODING_GUIDELINES file-length, println, and unsafe-SAFETY checks from the divergent untracked copy that this replaces. - CONTRIBUTING.md now documents the real commands and the toolchain/MSRV split. Fleet recovery and soak - scripts/restore-fleet.sh: the fail-closed selective restore, promoted out of an ignored tmp/ directory into the repository. Preflights retained storage, digest-pinned images, parked state, and aggregate plus per-PV-node scheduler headroom before the first scale; writes a durable transcript under tmp/restore-logs/ with structured start/error/rollback/complete events. - k8s manifests park the standalone store, the RF3 cluster, and the soak monitor at zero replicas with restore-fleet.sh as the only supported scale-up path. - soak-eval/soak-watch and the nightly CronJob fail closed on stale or missing restart evidence instead of silently skipping the restart-aware half of the gate. - docs/ops/capacity-planning.md corrects the RAM envelope to the real hot-tier formula and separates analytic totals from the measured process envelope. |
||
|
jx12n
|
a946c6128c |
fix(m12-rc13): read-SLA collapse + WAL_RETENTION_SEGMENTS 16 + tidalctl S3 DR
Read-SLA fix (rc12→rc13 — cpu-cgroup starvation → multi-second p99 + churning elections): - offload.rs: add SEARCH_GATE semaphore (core_count+1 permits, 50ms shed to 429) so per-shard searches gate on CPU, not reactor threads; concurrent scatter_merge fan-out (join_all) replaces the serial blocking offload_region_read loop - node.rs: scatter_merge → async; per-shard futures run via offload_search (each acquires one SEARCH_GATE permit, moves it into spawn_blocking so the permit is held for the search's full CPU lifetime) - main.rs: explicit tokio runtime with worker_threads floored at 4, independent of the cgroup quota — keeps the control plane (heartbeat/election/apply) on its own workers even when quota < 4 - k8s statefulset: CPU limit 2→3 (was: available_parallelism()=2 → only 2 async workers; search burst starved the reactor) - tidal/wal/compaction.rs: WAL_RETENTION_SEGMENTS 4→16 (64 MiB→256 MiB per-shard catch-up window; a briefly-down follower across a rolling restart streams up instead of forcing snapshot reseed; disk floor 768 MiB/pod, self-trimming) - cluster_reseed.rs: OFFLINE_ITEMS 1800→5600 to exceed the new 16-segment retention window (19 segs > 17); fix sequential quarantine/reseed race via await_status_bool tidalctl S3/R2 backup DR: - tidalctl/Cargo.toml: aws-config, aws-sdk-s3, aws-credential-types, tokio, tempfile - commands/s3.rs: S3Target + export_dir (upload every file, manifest last as atomicity marker) + import_to_dir (download prefix into temp staging dir) - commands/backup.rs: run_backup/run_restore accept Option<&S3Target>; S3 export is additive after local fsync barrier; S3 import stages into TempDir then runs the unchanged verified restore on it - main.rs: --s3-endpoint / --s3-bucket / --s3-prefix flags; all-or-nothing endpoint+bucket validation; usage updated tidal-stress/k8s: recall-rc12-spread-job, soak-nightly-cronjob, soak-monitor, soak-results-pvc, t5-readtput-job manifests |
||
|
jx12n
|
d5d1e7d81a |
feat(m11): observability+ops (m11p8) + perf-sweep wave 2 T2
m11p8 closes G-O + §1.4-3: - Cluster metrics: breaker state, forwards, self-heal on /metrics; multi-shard sibling render (shard="N") - Grafana cluster row + 8-rule Prometheus alert group - Request-id / TraceLayer on both cluster routers; id rides forward hop - Truthful status: flushed leader applied_events frontier; post-promote ShardId(0) keying fix - Self-driving heal: tick_self_heal re-arms stuck-peer backlog every ~3s - WAL PITR: wal.archive_dir, archive-before-delete gap-free - tidalctl backup/restore with BLAKE3 content-hash verification - Rolling-upgrade build_version handshake (N/N+1, never rejects) + Woodpecker release gate perf-sweep wave 2 T2: one-get-per-type pre-pass in ranking executor - signal_values.rs pre-fetches all signal kinds before scoring loop - Eliminates per-item repeated DashMap lookups: −18.8% for_you, −31% under writes - Byte-identical output verified with A/B test harness |
||
|
jx12n
|
3bcfb3c576 |
feat: Bazel build, crate docs/ai-lookup, docker images, and engine hardening
- Add BUILD.bazel across tidal, tidal-net, tidal-server, tidalctl for bzlmod build - Add tidal/ crate docs (README, CHANGELOG, CONTRIBUTING, AGENTS, CLAUDE, API, ARCHITECTURE) and ai-lookup reference - Add docker standalone/cluster/deploy images, compose, and prometheus config - Harden WAL (batch format, writer, dedup, diagnostics), text syncer/collectors, and vector registry - Expand tidalctl CLI and tests; restructure WAL/visibility integration test suites - Refine tidal-net transport/client/server and tidal-server cluster/scatter-gather |
||
|
jordan
|
4f076c927d |
feat: M0p1 runtime skeleton, M0p2 tooling & diagnostics, m1p4 signal ledger
## M0p1 — Embeddable Runtime Skeleton (329 tests)
- TidalDb with builder(), health_check(), close(), and Drop-based cleanup
- TidalDbBuilder fluent API: ephemeral(), with_data_dir(), wal_dir(), cache_dir()
- Config, StorageMode, ConfigError types; Config(ConfigError) variant on LumenError
- Paths: single source of truth for directory layout (wal, items, users, creators, cache)
- TempTidalHome: test isolation helper gated behind #[cfg(test)] / test-utils feature
- 8 integration tests: tests/sandboxed_storage.rs
## M0p2 — Tooling & Diagnostics (349 tests)
- Workspace root Cargo.toml (members: ["tidal", "tidalctl"])
- tidal/build.rs: BUILD_HASH from GIT_HASH with option_env!() fallback to "dev"
- MetricsState: always-compiled Arc-shared atomics (uptime, health_ok)
- MetricsHandle (metrics feature): hand-rolled TcpListener HTTP, zero new deps
- GET /healthz → {"status":"ok","uptime_secs":N}
- GET /metrics → Prometheus text (tidaldb_uptime_seconds, health_ok, info)
- TidalDbBuilder.enable_metrics(addr) starts background metrics thread
- tidalctl binary: status + paths commands, manual std::env::args() parsing
- 7 metrics integration tests, 9 tidalctl CLI tests
## m1p4 Signal Ledger (in-progress)
- SignalLedger: DashMap<(EntityId, SignalTypeId), EntitySignalEntry>, WAL-first writes
- HotSignalState: #[repr(C, align(64))], lock-free CAS decay, out-of-order handling
- BucketedCounter: 60 per-minute + 168 per-hour circular buffers, trigger-based rotation
- CheckpointMeta + serialize/restore: 983-byte fixed records, atomic WriteBatch
- Property tests: running score matches analytical to 1e-6, decay monotonic, non-negative
- Proptest regression: signals/warm.txt
## Documentation and planning
- ROADMAP: m0p1 COMPLETE (329), m0p2 COMPLETE (349), product track milestones
- PRODUCT_ROADMAP: P0-P4 product milestone track (personal briefing beachhead)
- Milestone planning docs: milestone-0 (phases 1-3), milestone-p (phases 1-5)
- docs/research/tidaldb_tooling_and_diagnostics.md
- ARCHITECTURE.md, CLAUDE.md, VISION.md updates
## Site
- Blog: every-platform-builds-the-same-6-systems.mdx (new)
- Blog: why-tidaldb.mdx (updated)
- next.config.ts, layout.tsx, blog/page.tsx updates
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|