jx12n
|
bb21e69ae6
|
feat(m12): vector retrieval G1/G2 — recall harness, ANN in RETRIEVE, index tuning
m12p1 (measurement truth): TidalDb::vector_search_items pure k-NN probe +
POST /vector_search (standalone + region node, merge-by-distance) +
tidal-stress --verify-recall (deterministic id-keyed corpus, in-RAM brute-force
cosine oracle, open-loop ramp → recall@k + true p99 + read-knee + JSON/gate exit).
Repaired fabricated p99 columns (mean-as-p99) in social-scale.md / scale.rs.
Verified real: recall@10=0.9997 at 20k/1536-D vs brute-force.
m12p2 (G1 unblock): ANN candidate-gen wired into RETRIEVE — for_you=preference
vector, related=seed embedding (similar_to), graceful scan-fallback. Cached
per-signal-type top-K (signals/ledger/hot_top_k.rs, decay-order-invariant) so
trending serves O(K). related over HTTP (FeedQuery.similar_to). Harness gains
--feed-profile / --seed-preferences. Verified: trending retrieve p99 3.5-7.7ms.
m12p3 (G2): per-query ef_search now honored (RwLock epoch-guard with_expansion,
shared guard for same-ef concurrency) + dimension-aware brute→HNSW crossover
usearch_min_vectors(dim) + memory_usage() + examples/ann_grid_search.rs.
Measured 1536-D/100k clustered: default M=16/ef_c=400/F16/ef_s=200 clears
G1+G2 (recall 0.997, p99 1.4ms); F16 -0.25% vs F32; Int8 rejected (-28%).
Recall corpus is now clustered (Gaussian mixture) in grid + harness.
|
2026-06-14 11:07:09 -06:00 |
|
jx12n
|
1265140e28
|
feat(m11): continuous correctness (m11p9) — fault classes, invariant checkers, soak gates, nightly pipeline
- fault-injection cargo feature (compiled OUT of prod): slow-fsync + disk-full
WAL hooks in tidal/src/fault.rs, inert until armed, tier-3 builds with feature
- first-class invariant checkers (tests/support/invariants.rs): AckLedger
no-acked-loss (now consumed by m11p3 gate), feed parity, single-leader-per-term,
monotonic frontiers
- cluster_faults.rs tier-3 suite 4/4: disk-full degrade+recover, slow-fsync
lag+converge, both-slow quorum 503, asymmetric partition no-split-brain
- tidal-stress soak gates: --json-summary + --max-p99-ms/--max-error-pct/
--fail-on-knee → non-zero exit on regression
- Woodpecker cron nightly flow (chaos + gated soak), event-routed, not GH Actions
- guarantee-traceability.md: roadmap §2 guarantees → named tests (closes G-C
apparatus; 30-day-green is a calendar criterion)
|
2026-06-13 15:23:59 -06:00 |
|
jx12n
|
005e292cbb
|
fix(m11): review remediation + tidal-stress perf sweep + perf wave 2
Resolve all BLOCKER/CRITICAL/WARNING findings from the m11p7/p8 review:
- tidalctl restore: safe_join path-traversal/Zip-Slip guard + fsync on write
- corrupt-WAL checkpoint_seq guard; PITR archive-before-delete
- cluster: x-tidal-relayed audit-dedup marker; forward_failures counts 5xx
- mTLS/HTTP-TLS handshake hardening; accept-loop EMFILE backoff
- per-principal rate-limit + node-token marker-pinning tests
- self-heal tier-3 coverage; 5 router-auth tests
tidal-stress: measurement-fidelity fixes (schedule-lag p99/max, exact
feed-over-SLO verdict, shed annotation) + typed Body, workload.next
184ns->68ns, RoundRobin len==1 short-circuit, HeaderValue cache;
new benches/hotpath.rs + lib.rs.
perf wave 2: signal_snapshot SmallVec/SignalKey carrier; one-get-per-type
ranking pre-pass.
|
2026-06-13 12:28:04 -06:00 |
|