be37c72b12
14 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
jordan
|
fe8d0c87e7 |
harden: restore CI verification, remove four wire-level fabrications, instrument the 401 path
Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement). Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release gate past its budget. Both push-path steps now declare backend_options.kubernetes.resources as two YAML anchors declared once on their first consuming step. The values are CALIBRATED against measured free node capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's original figure) fits on NO node and would sit Pending forever, because `ci-build-bounds` grants permission and the nodes supply capacity, and those are not the same thing. The `nightly` cron described in this file for 216 days was never created, so tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero signal while reading like standing coverage. nightly-chaos and nightly-security-ops now alias the anchors and have budgets matching the gate (their 120/90 were TIGHTER on the same runner, so they would have failed nightly for a budget reason, not a correctness one). nightly-soak is REMOVED, not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best node has 1700m free CPU, so it would fail on starvation rather than regression — manufacturing a nightly false alarm. Its commands move verbatim to docs/runbooks/nightly-soak.md. Ring 1 — four fabrications removed from the wire. - scatter_merge sorted and truncated without re-stamping rank, so /feed and /search returned 1,1,2 under full placement. Reuses merge_cross_shard's existing stamp; asserted on BOTH the multi-group merge path and the single-group [only] fast path that bypasses it. - aggregate_region_row's None arm invented `applied_events: 0` plus a deficit derived from it. applied_events/lag_events are now Option<u64>, null on the wire. leader_last_seq was also unwrap_or(0), so a node that could not reach the LEADER computed 0 - applied = 0 for every region and reported a converged cluster it had never measured — a fabrication pointing the dangerous way. - tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was actively hiding the PVC-wipe shape: a measured zero with a real deficit rendered as "no report" instead of BEHIND. Now read off the wire; converged exits 0, partitioned still exits nonzero. - /sharded/* answered 201/204 for single-copy writes with nothing anywhere saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the existing invalid_input path. Six call sites migrated, not the two this roadmap predicted — including docs/runbooks/cluster.md §16.3, which told operators to run a quorum-write probe via POST /sharded/items. That probe cannot verify quorum: the surface applies locally with no WAL append. It was used as the safety check between every step of a staged deploy earlier today. Ring 2 — observability. JSON_LOGS was already implemented and the deployment simply never asked for it; the StatefulSet now sets it, plus TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the VictoriaLogs `service` stream field and would have blinded every query keyed on it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only the origin would mean each vector lands on exactly one node, replicas never agree, and the alert built on it pages forever. Found by measurement, not planned: the 401 path discarded every fact about every rejection. Traefik has served 101,858 rejected requests to the public ingress — 87.6% of all its traffic — with no record of who or why anywhere. unauthorized_response now emits reason (missing_token vs invalid_token, the distinction that separates a scanner from a rotation that missed a consumer) and the forwarded client. The token is never logged. Also: scripts/restore-fleet.sh --cluster started the soak monitor while deliberately leaving its gate suspended, orphaning a watcher that has reported "0/30 green nights" for 13 days. The pair now moves together. Doc-guard's three-warning backlog is cleared with real backfill for M4/M6/M12. Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline, counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5; cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with the two remaining failures asserting the rank fix against the not-yet-rolled image — they are the post-deploy proof. |
||
|
jordan
|
4766f566de |
feat(observability): HTTP metrics, structured logs, dashboard, live tidalctl
There was no metric anywhere that could answer "how much traffic are we serving"
or "what is our error rate". The engine published a rich DOMAIN surface (search
latency, WAL fsync, quorum timeouts, replication lag) and nothing about HTTP, so
a cluster could serve 401s or 503s indefinitely with every existing gauge looking
healthy. Logs were collected but unusable. There was no way to ask a RUNNING node
anything.
1. HTTP metrics. tidaldb_http_requests_total{route,method,status} plus a
per-route duration histogram, recorded by one layer placed OUTSIDE the auth,
timeout and rate-limit layers so it sees the status actually returned to the
client. Cardinality is the whole design: the route label is axum's MatchedPath
TEMPLATE, not the path, and unmatched requests collapse into one <unmatched>
bucket so a 404 flood cannot mint series. A hard cap folds anything past it
into an overflow bucket while established series keep counting.
The engine owns the /metrics listener but must not learn what a route or a
status code is, so it gained one registration hook
(MetricsState::set_extra_renderer) and tidal-server publishes through it. One
scrape target per node, not two.
2. Structured logs. The previous init was a bare tracing_subscriber::fmt(), which
produced two real defects: ANSI escapes leaked into collected logs, and every
line failed the collector's JSON parse and was stamped level=info — so
`level:error` matched NOTHING and errors were invisible to the log platform
while being collected. JSON_LOGS=1 emits the collector's exact wire format
(ts/level/service/env/msg), span fields are lifted so request_id lands on every
line of a request, and ANSI is off unconditionally in both formats.
Verified against the running binary, which caught a defect no unit test would
have: dependencies logging through the `log` crate arrived with target="log"
and four log.* metadata fields (absolute cargo registry paths, indexed
forever). The real module is now lifted into target and the bridge metadata
pruned.
3. Dashboard. docs/ops/grafana-tidaldb.json, 13 panels, mirrored into the fleet
as a grafana-database-dashboards key. Every metric name was checked against a
live endpoint and all 26 PromQL expressions were executed against the live
TSDB before commit, because a dashboard full of "No data" is worse than none.
Confirmed loaded in Grafana (uid tidaldb-overview, Databases folder).
4. tidalctl live mode. Every other subcommand reads a data dir AT REST, some
requiring a stopped node. `search`, `feed`, `cluster-status` and `watch` take
--url and talk to a running server, with --ca/--insecure because a cluster's
client port is served with the INTERNAL cluster CA. Exit codes follow the crate
contract, so `tidalctl cluster-status && deploy` gates on convergence.
Its first real run immediately found a reporting defect: the aggregated
/cluster/status reported two HEALTHY peers as UNREACHABLE PARTITIONED at 13.3M
lag, having derived lag against an uninitialised applied=0, while every node's
own status reported lag=0, reseed=false and identical frontiers, with
pod-to-pod connectivity open and nothing logged. cluster-status now names that
signature "NO REPORT (aggregated view; query the node directly)" instead of
repeating it as replication lag; a genuine non-zero-applied lag still reports
BEHIND. The underlying gap is documented as open work in
docs/ops/observability.md.
Verified: 2101 + 175 engine/server unit tests, 8 standalone integration (3 new,
including the cardinality proof and the cross-crate metrics seam), 23 tidalctl
(10 new), reseed + catchup + admin-gate e2e green, clippy clean, and both the
metrics and the log format exercised against a real running binary.
|
||
|
jordan
|
c97aaa8e5b |
fleet remediation: make the workspace gate runnable, then fix what it caught
`cargo test --workspace` could not run at all: dependency resolution failed with "aws-types@1.3.16 requires rustc 1.91.1" on the 1.91.0 default toolchain, so the gate the project documents was dead. Making it run exposed a compile break and two wrong tests that had been invisible for months. Now green end to end: 143 suites, 3155 tests, exit 0. Toolchain - rust-toolchain.toml pins the DEV toolchain to 1.91.1. The published MSRV stays `rust-version = "1.91"` (the engine builds on 1.91.0); only tidalctl's AWS SDK chain needs the patch release, and it now declares that itself. Consumer crates migrated to the current engine API (clean cutover) - iknowyou-engine: `AgentPolicy` gained five m10 read/profile-override fields; the literal now spreads `..AgentPolicy::default()` as the engine's own doc example does, so future fields do not break it again. - forage-engine: `RetrieveResult` gained p1 `reasons`. The app builds its own candidate pool, so it now tags what it knows: PreferenceMatch for the preference-vector blend, SemanticMatch (with the seed item) for similar-to-saved, ExplorationBudget for pinned discoveries. - forage-engine: `url_to_item_id` folded into the u32 item universe. The engine narrows item IDs to a u32 slot in durable per-user state and rejects anything above u32::MAX rather than alias two items forever, so every add_item with a 64-bit FNV hash failed. 9 of 28 smoke tests were failing on this alone. - forage-engine: bridge items read the top-2 preference CLUSTERS via `query_vectors`, not the single centroid from `preference_vectors().get()`. Since m12 that accessor returns only the strongest cluster, so a tech+jazz user whose interests split into two clusters looked single-interest and never bridged. Falls back to top-2 dimensions when a user has one cluster. Reconcile tests corrected to the shipped contract - tidal/tests/m8p3_reconcile_production.rs asserted `3 + 5 == 8` for a windowed count after heal. `take_crdt_snapshot` deliberately keys signal contributions to ONE canonical contributor (ShardId::SINGLE) because signals are relayed from a single writer, so per-node attribution double-counted every replicated event on every reconcile. Merge is therefore LWW on (last_update_ns, score) plus PN-counter per-node max: nodes converge on the more complete accumulator. The old expectation was asserting the bug that fix removed. - Rewrote to assert convergence, count survival (not 0), and no inflation, and added `repeated_reconcile_of_converged_nodes_does_not_creep` - the regression guard for the creep itself, which nothing covered. Pre-commit hook unified - hooks/pre-commit dropped `-D warnings`: each crate's `[lints]` table is the source of truth (`clippy::all`/`unwrap_used` deny, `pedantic` warn), and the flag promoted ~58 deliberate pedantic warnings in integration tests to errors, making every Rust commit impossible. - It now lints all five tidal crates instead of path-matching `tidal/`, which silently skipped tidal-server, tidal-net, tidal-stress, tidalctl and applications/ - the rot above lived in exactly those crates. Ported the CODING_GUIDELINES file-length, println, and unsafe-SAFETY checks from the divergent untracked copy that this replaces. - CONTRIBUTING.md now documents the real commands and the toolchain/MSRV split. Fleet recovery and soak - scripts/restore-fleet.sh: the fail-closed selective restore, promoted out of an ignored tmp/ directory into the repository. Preflights retained storage, digest-pinned images, parked state, and aggregate plus per-PV-node scheduler headroom before the first scale; writes a durable transcript under tmp/restore-logs/ with structured start/error/rollback/complete events. - k8s manifests park the standalone store, the RF3 cluster, and the soak monitor at zero replicas with restore-fleet.sh as the only supported scale-up path. - soak-eval/soak-watch and the nightly CronJob fail closed on stale or missing restart evidence instead of silently skipping the restart-aware half of the gate. - docs/ops/capacity-planning.md corrects the RAM envelope to the real hot-tier formula and separates analytic totals from the measured process envelope. |
||
|
jx12n
|
1b5bcbacd7 |
fix(net): classify ship deadline as timeout, not partition (write-burst false-partition)
A client-side ship DEADLINE means the RPC did not round-trip within
request_timeout — which a slow-but-ALIVE follower produces under a sustained
1536-D ack=quorum apply burst (transport runtime momentarily starved by the
CPU-heavy HNSW apply on its single segment-receiver thread) exactly as a
genuinely blackholed peer does. Counting that as record_failure was the
write-burst false-partition: 5 such opened both followers' breakers, the commit
index stalled, ack=quorum 503'd, and retries re-burst the same starved peers
with no self-heal.
- CircuitBreaker::record_timeout: opens ONLY when the peer shows no recent proof
of life (no round-tripped success/backpressure within reset_duration); neutral
no-op when liveness is fresh; re-opens (never wedges) HalfOpen; never refreshes
the liveness stamp (no reply arrived).
- PeerPool::send_to routes tonic DeadlineExceeded/Cancelled -> record_timeout;
genuine severance still surfaces as connect-level Unavailable/transport reset
-> record_failure and still opens the breaker.
- ship_timeout_breaker.rs: end-to-end proof over a REAL tonic WalShipping server
(handler succeeds once then hangs past the client deadline) + 6 unit tests.
Also: re-scope G-S Scalability guarantee to read-throughput with the Ref-A
tidal-t5-readtput owner-test (write 2.5x is structurally impossible on 3-node
full-placement RF3); bump k8s image to m12-rc6 (live, commit
|
||
|
jx12n
|
a946c6128c |
fix(m12-rc13): read-SLA collapse + WAL_RETENTION_SEGMENTS 16 + tidalctl S3 DR
Read-SLA fix (rc12→rc13 — cpu-cgroup starvation → multi-second p99 + churning elections): - offload.rs: add SEARCH_GATE semaphore (core_count+1 permits, 50ms shed to 429) so per-shard searches gate on CPU, not reactor threads; concurrent scatter_merge fan-out (join_all) replaces the serial blocking offload_region_read loop - node.rs: scatter_merge → async; per-shard futures run via offload_search (each acquires one SEARCH_GATE permit, moves it into spawn_blocking so the permit is held for the search's full CPU lifetime) - main.rs: explicit tokio runtime with worker_threads floored at 4, independent of the cgroup quota — keeps the control plane (heartbeat/election/apply) on its own workers even when quota < 4 - k8s statefulset: CPU limit 2→3 (was: available_parallelism()=2 → only 2 async workers; search burst starved the reactor) - tidal/wal/compaction.rs: WAL_RETENTION_SEGMENTS 4→16 (64 MiB→256 MiB per-shard catch-up window; a briefly-down follower across a rolling restart streams up instead of forcing snapshot reseed; disk floor 768 MiB/pod, self-trimming) - cluster_reseed.rs: OFFLINE_ITEMS 1800→5600 to exceed the new 16-segment retention window (19 segs > 17); fix sequential quarantine/reseed race via await_status_bool tidalctl S3/R2 backup DR: - tidalctl/Cargo.toml: aws-config, aws-sdk-s3, aws-credential-types, tokio, tempfile - commands/s3.rs: S3Target + export_dir (upload every file, manifest last as atomicity marker) + import_to_dir (download prefix into temp staging dir) - commands/backup.rs: run_backup/run_restore accept Option<&S3Target>; S3 export is additive after local fsync barrier; S3 import stages into TempDir then runs the unchanged verified restore on it - main.rs: --s3-endpoint / --s3-bucket / --s3-prefix flags; all-or-nothing endpoint+bucket validation; usage updated tidal-stress/k8s: recall-rc12-spread-job, soak-nightly-cronjob, soak-monitor, soak-results-pvc, t5-readtput-job manifests |
||
|
jx12n
|
005e292cbb |
fix(m11): review remediation + tidal-stress perf sweep + perf wave 2
Resolve all BLOCKER/CRITICAL/WARNING findings from the m11p7/p8 review: - tidalctl restore: safe_join path-traversal/Zip-Slip guard + fsync on write - corrupt-WAL checkpoint_seq guard; PITR archive-before-delete - cluster: x-tidal-relayed audit-dedup marker; forward_failures counts 5xx - mTLS/HTTP-TLS handshake hardening; accept-loop EMFILE backoff - per-principal rate-limit + node-token marker-pinning tests - self-heal tier-3 coverage; 5 router-auth tests tidal-stress: measurement-fidelity fixes (schedule-lag p99/max, exact feed-over-SLO verdict, shed annotation) + typed Body, workload.next 184ns->68ns, RoundRobin len==1 short-circuit, HeaderValue cache; new benches/hotpath.rs + lib.rs. perf wave 2: signal_snapshot SmallVec/SignalKey carrier; one-get-per-type ranking pre-pass. |
||
|
jx12n
|
d5d1e7d81a |
feat(m11): observability+ops (m11p8) + perf-sweep wave 2 T2
m11p8 closes G-O + §1.4-3: - Cluster metrics: breaker state, forwards, self-heal on /metrics; multi-shard sibling render (shard="N") - Grafana cluster row + 8-rule Prometheus alert group - Request-id / TraceLayer on both cluster routers; id rides forward hop - Truthful status: flushed leader applied_events frontier; post-promote ShardId(0) keying fix - Self-driving heal: tick_self_heal re-arms stuck-peer backlog every ~3s - WAL PITR: wal.archive_dir, archive-before-delete gap-free - tidalctl backup/restore with BLAKE3 content-hash verification - Rolling-upgrade build_version handshake (N/N+1, never rejects) + Woodpecker release gate perf-sweep wave 2 T2: one-get-per-type pre-pass in ranking executor - signal_values.rs pre-fetches all signal kinds before scoring loop - Eliminates per-item repeated DashMap lookups: −18.8% for_you, −31% under writes - Byte-identical output verified with A/B test harness |
||
|
jx12n
|
225751d34d |
feat(m11): WAL-as-stream replication + perf floor (m11p1+m11p2)
m11p1 — decoupled ack/ship path: staged writes (seqno+WAL+relay-push, microseconds) separate from group-commit fsync; ShipQueue batches+windows outbound segments; receiver coalesces inbound chunks before applying. Adds first tidaldb_cluster_* metrics. m11p2 — leader WAL is now THE replicated log: fsynced batches feed a bounded WalShipFeed and ship byte-identical to followers; WAL seqnos survive restarts (relay-reset hazard gone). Item metadata and embeddings journal kind-1/2 blob records on the same stream as signals; the m8p10 HTTP broadcast is deleted. StreamSegments catch-up is follower-pulled via server-streaming RPC, triggered on gap detection, follower boot, and leader heal nudge. Promote carries a stream baseline so peers skip pre-stream history. |
||
|
jx12n
|
9728194f16 |
fix: M0-M10 code-review pass2 remediation — all 91 findings
Resolves every finding in docs/reviews/M0-M10-code-review-2026-06-08-pass2.md across the engine, network, server, and CLI crates: session restore, replication/CRDT, WAL format and recovery, storage indexes, query/ranking executors, cohort/community governance, and scatter-gather routing. Adds regression tests: - review_pass2_creator_search_filter - review_pass2_d_replication - review_pass2_query_for_session - review_pass2_storage_indexes_bitmap_cache - review_pass2_zone_a_sessions Verified: cargo clippy -D warnings and full test suite green across all crates. |
||
|
jx12n
|
ad4134e280 |
chore: doc consolidation, seven-dimension review fixes, and commit hooks
- Eliminate the tidal/ self-contained doc mirror; docs now have two canonical homes (root *.md and docs/), with planning/specs/research/reviews moved up - Remove stale .agents/skills and .ai mirrors; canonicalize skills under .claude/ - Add pre-commit hook + scripts/check-docs.sh doc-guard + scripts/install-hooks.sh - Implement M0-M10 seven-dimension review findings across engine, net, server, and tidalctl (durability, replication, query, WAL, storage, CLI hardening) |
||
|
jx12n
|
b55ad70141 |
fix: M0-M10 third-pass remediation — durability, replication, and CLI hardening
Resolves the 142 findings from tidal/docs/reviews/CODE_REVIEW_m0-m10.md across the engine, server, net, and CLI surfaces: - WAL/session-journal durability, checkpoint format, and crash-recovery hardening - Replication shipper/receiver, tenant isolation, and migration paths - Cluster scatter-gather, router, standalone server + health/offload endpoints - tidalctl refactored into command modules with JSON output and WAL-state tooling - Cohort, governance, signal-ledger, and vector-registry correctness fixes - Expanded UAT/integration/durability test coverage across all milestones |
||
|
jx12n
|
3bcfb3c576 |
feat: Bazel build, crate docs/ai-lookup, docker images, and engine hardening
- Add BUILD.bazel across tidal, tidal-net, tidal-server, tidalctl for bzlmod build - Add tidal/ crate docs (README, CHANGELOG, CONTRIBUTING, AGENTS, CLAUDE, API, ARCHITECTURE) and ai-lookup reference - Add docker standalone/cluster/deploy images, compose, and prometheus config - Harden WAL (batch format, writer, dedup, diagnostics), text syncer/collectors, and vector registry - Expand tidalctl CLI and tests; restructure WAL/visibility integration test suites - Refine tidal-net transport/client/server and tidal-server cluster/scatter-gather |
||
|
jordan
|
213b8efcca |
feat: complete M6-M7 + Enterprise Readiness milestones; split oversized source files per CODING_GUIDELINES §9
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
jordan
|
4f076c927d |
feat: M0p1 runtime skeleton, M0p2 tooling & diagnostics, m1p4 signal ledger
## M0p1 — Embeddable Runtime Skeleton (329 tests)
- TidalDb with builder(), health_check(), close(), and Drop-based cleanup
- TidalDbBuilder fluent API: ephemeral(), with_data_dir(), wal_dir(), cache_dir()
- Config, StorageMode, ConfigError types; Config(ConfigError) variant on LumenError
- Paths: single source of truth for directory layout (wal, items, users, creators, cache)
- TempTidalHome: test isolation helper gated behind #[cfg(test)] / test-utils feature
- 8 integration tests: tests/sandboxed_storage.rs
## M0p2 — Tooling & Diagnostics (349 tests)
- Workspace root Cargo.toml (members: ["tidal", "tidalctl"])
- tidal/build.rs: BUILD_HASH from GIT_HASH with option_env!() fallback to "dev"
- MetricsState: always-compiled Arc-shared atomics (uptime, health_ok)
- MetricsHandle (metrics feature): hand-rolled TcpListener HTTP, zero new deps
- GET /healthz → {"status":"ok","uptime_secs":N}
- GET /metrics → Prometheus text (tidaldb_uptime_seconds, health_ok, info)
- TidalDbBuilder.enable_metrics(addr) starts background metrics thread
- tidalctl binary: status + paths commands, manual std::env::args() parsing
- 7 metrics integration tests, 9 tidalctl CLI tests
## m1p4 Signal Ledger (in-progress)
- SignalLedger: DashMap<(EntityId, SignalTypeId), EntitySignalEntry>, WAL-first writes
- HotSignalState: #[repr(C, align(64))], lock-free CAS decay, out-of-order handling
- BucketedCounter: 60 per-minute + 168 per-hour circular buffers, trigger-based rotation
- CheckpointMeta + serialize/restore: 983-byte fixed records, atomic WriteBatch
- Property tests: running score matches analytical to 1e-6, decay monotonic, non-negative
- Proptest regression: signals/warm.txt
## Documentation and planning
- ROADMAP: m0p1 COMPLETE (329), m0p2 COMPLETE (349), product track milestones
- PRODUCT_ROADMAP: P0-P4 product milestone track (personal briefing beachhead)
- Milestone planning docs: milestone-0 (phases 1-3), milestone-p (phases 1-5)
- docs/research/tidaldb_tooling_and_diagnostics.md
- ARCHITECTURE.md, CLAUDE.md, VISION.md updates
## Site
- Blog: every-platform-builds-the-same-6-systems.mdx (new)
- Blog: why-tidaldb.mdx (updated)
- next.config.ts, layout.tsx, blog/page.tsx updates
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|