Commit Graph

7 Commits

Author SHA1 Message Date
jx12n
d5d1e7d81a feat(m11): observability+ops (m11p8) + perf-sweep wave 2 T2
m11p8 closes G-O + §1.4-3:
- Cluster metrics: breaker state, forwards, self-heal on /metrics; multi-shard sibling render (shard="N")
- Grafana cluster row + 8-rule Prometheus alert group
- Request-id / TraceLayer on both cluster routers; id rides forward hop
- Truthful status: flushed leader applied_events frontier; post-promote ShardId(0) keying fix
- Self-driving heal: tick_self_heal re-arms stuck-peer backlog every ~3s
- WAL PITR: wal.archive_dir, archive-before-delete gap-free
- tidalctl backup/restore with BLAKE3 content-hash verification
- Rolling-upgrade build_version handshake (N/N+1, never rejects) + Woodpecker release gate

perf-sweep wave 2 T2: one-get-per-type pre-pass in ranking executor
- signal_values.rs pre-fetches all signal kinds before scoring loop
- Eliminates per-item repeated DashMap lookups: −18.8% for_you, −31% under writes
- Byte-identical output verified with A/B test harness
2026-06-13 09:17:49 -06:00
jx12n
8673723319 perf(m11): kill signal_snapshot allocation cascade (perf-sweep wave 2 T1)
Replace ScoredCandidate.signal_snapshot Vec<(String,f64)> with
SmallVec<[(SignalKey,f64); 4]> where SignalKey is Static(&'static str)
| Owned(Arc<str>):

- Compile-time-constant labels (sort bases, relevance, co_engagement,
  preference_affinity) -> Static: zero allocation, pointer-copy clone.
- Dynamic {signal}_boost/_penalty/_decay labels built once per query in a
  RuleLabels hoist (was format! per-candidate-per-rule), shared by Arc.
  Cohort rescore hoisted the same way.
- scored accumulator pre-sized to candidates.len().
- Owned Strings rebuilt only at the two response-assembly sites (<= limit).

Byte-identical output: full 1896-test lib suite green. Measured win
(cargo bench --bench ranking, committed-base vs working-tree):
score_200_hot 23.89->21.04us (-11.9%), score_200_trending
27.66->25.85us (-6.5%), score_200_full_pipeline 27.88->26.97us (-3.3%).

smallvec promoted from the lock to a direct dep (no new dependency
surface). perf-sweep doc updated; T2 (per-term DashMap collapse) next.
2026-06-13 01:53:08 -06:00
jx12n
6651c14adc feat(m11): cluster security (m11p7) + perf instrumentation floor
m11p7 — secure the cluster, all opt-in (pre-m11p7 byte-for-byte):
- gRPC replication mTLS by default via a custom tokio-rustls acceptor +
  DynamicCertResolver; zero-drop content-hash cert rotation (k8s ..data swap,
  no pod restart, no inotify)
- inter-node HTTP TLS sharing the same resolver (one rotation, both planes) +
  per-node keyed-BLAKE3 signed x-tidal-node-token; marker-without-token -> 403
- admin audit log (operator-leg only) + per-principal rate limit (engine
  RateLimiter; sibling nodes exempt)
- k8s cert-manager manifest (certs.yaml) + scripts/gen-cluster-certs.sh fallback;
  secret.example.yaml gains TIDAL_CLUSTER_KEY (file-mounted, hot-rotatable)
- exit gate verified real: mtls.rs (gRPC foreign-pod), cluster_security.rs
  (HTTP foreign + zero-drop rotation under load), 7 security unit tests

perf — instrument floor (sweep Wave 1):
- new tidal/benches/wal.rs + tidal-server/benches/scatter.rs
- p99->mean honesty relabel; sweep manifest at docs/reviews/perf-sweep-2026-06-13.md
- add @tidal-performance agent (Martin Thompson)

new: cluster/{audit,http_tls,security}.rs, tests/cluster_security.rs,
docs/planning/milestone-11/phase-7.md
2026-06-13 01:25:35 -06:00
jx12n
9728194f16 fix: M0-M10 code-review pass2 remediation — all 91 findings
Resolves every finding in docs/reviews/M0-M10-code-review-2026-06-08-pass2.md
across the engine, network, server, and CLI crates: session restore,
replication/CRDT, WAL format and recovery, storage indexes, query/ranking
executors, cohort/community governance, and scatter-gather routing.

Adds regression tests:
- review_pass2_creator_search_filter
- review_pass2_d_replication
- review_pass2_query_for_session
- review_pass2_storage_indexes_bitmap_cache
- review_pass2_zone_a_sessions

Verified: cargo clippy -D warnings and full test suite green across all crates.
2026-06-09 12:21:00 -06:00
jx12n
5d211abce0 docs: record post-remediation re-review fixes in seven-dimension review
Document the six issues a follow-up adversarial sweep surfaced in the
M0-M10 remediation diff and their fixes (code already landed):

- BLOCKER warm-tier double-count was a half-fix (read side only); now
  bounded on read + write (hour_agg anchored at last_min) + day tier.
- CRITICAL W22 stale-index scrub now wired into the item overwrite path.
- CRITICAL checkpoint read() made side-effect-free; temp-sweep moved to
  the writable recover() path.
- WARNING W13 partition_id wired at construction; W22-class social-graph
  /collection u32 truncation routed through checked entity_as_u32.

Updates the green-state line (1672 lib tests, clippy -D on all crates).
2026-06-08 22:53:09 -06:00
jx12n
ad4134e280 chore: doc consolidation, seven-dimension review fixes, and commit hooks
- Eliminate the tidal/ self-contained doc mirror; docs now have two canonical
  homes (root *.md and docs/), with planning/specs/research/reviews moved up
- Remove stale .agents/skills and .ai mirrors; canonicalize skills under .claude/
- Add pre-commit hook + scripts/check-docs.sh doc-guard + scripts/install-hooks.sh
- Implement M0-M10 seven-dimension review findings across engine, net, server,
  and tidalctl (durability, replication, query, WAL, storage, CLI hardening)
2026-06-08 22:46:28 -06:00
jx12n
69ae6e64a1 fix: M0-M10 remediation — deferred post-filters, WAL hardening, docker build fixes
- Extract shared deferred post-filter (InCollection/MinSignal/MaxSignal/
  NearLocation/SocialGraph) into query/executor/post_filter.rs so RETRIEVE and
  SEARCH apply identical predicates; SEARCH previously ignored four variants
- Harden WAL writer/compaction/dedup paths
- Brute-force vector store fixes + tests
- Governance community ledger updates
- Pin docker builders to bookworm (glibc/libmvec match) and add
  protobuf-compiler for tidal-net's tonic-build
- Add M0-M10 code-review document
2026-06-07 20:47:17 -06:00