Commit Graph

7 Commits

Author SHA1 Message Date
jx12n
bd2b270dee build(standalone): committed multi-arch pullable standalone dev image path
Consumers (e.g. thepeach) need a PULLABLE tidaldb image — not every dev has the
source for a compose `build:` context. Publishes registry.threesix.ai/tidal/standalone
as a manifest list: linux/amd64 (Linux/CI) + linux/arm64 (Apple-silicon Macs),
defaulting to STANDALONE mode (:9400, --data-dir /data) so it drops into a consumer's
docker-compose like postgres.

Hybrid build because `rustc` SIGSEGVs under QEMU (an in-container amd64 cross-build on
an arm64 host fails):
- amd64: docker/standalone/amd64.Dockerfile packages the HOST cross-compiled x86_64
  binary (the proven build-release path; trixie-slim for glibc 2.41/libmvec) — apt+COPY only.
- arm64: native in-container build from docker/standalone/Dockerfile.
- scripts/build-standalone-image.sh stitches both into one manifest list (:m12 + :latest).

Verified: pulled :m12 (arm64), booted standalone, GET /health -> {ok:true,mode:standalone}.
2026-06-19 21:34:06 -06:00
jx12n
580142df49 feat(m12): election-divergence-fix + soak-eval streak + release tooling
Durable `leader_acked` frontier in `ShardReplica` tracks the highest seqno
acked under `ack=leader` (journal-only, un-replicated); `decide_join` now
quarantines on THIS node's own frontier rather than comparing stream numbers
across stream boundaries — eliminates false-quarantine churn on rolling
restarts. `SHUTDOWN_HANDOFF_WAIT` (3s) drains the leader's tail to quorum
before step-down so the next leader inherits a clean prefix. New
`load_leader_acked`/`persist_leader_acked` helpers; `cluster_reseed.rs` gains
the divergence-fix regression suite; `replication_ops.rs` threads the signal.

Soak-eval: `tidal_stress::soak_eval` + `soak-eval` binary implement the
30-night streak (ledger.tsv × restarts.tsv → streak.tsv); monitor and nightly
CronJob k8s YAMLs updated; phase-9 doc clarifies the dual-stream streak
definition (ledger PASS AND zero pod restarts in window). `run-reliability.sh`
gates the election-divergence suite before any k8s push.

Release tooling: `docker/release/` multi-stage Dockerfile + DR image;
`scripts/build-release.sh` single repeatable cross-compile+buildx path.
2026-06-18 13:08:53 -06:00
jx12n
aa94fd9b1f feat(m12p5): idle-readiness convergence via heartbeat live frontier + wildcard cert SAN
Leader heartbeat now carries its live flushed WAL frontier (leader_last_seq,
proto field 14) so a snapshot-installed joiner converges its sticky readiness
latch from the heartbeat — which flows even on a fully idle cluster — instead of
only from observed ship traffic or an external status poll. Fixes the
idle-readiness stall (WORKLOG 2026-06-13: an 11.5h /health 503 hang where a
caught-up joiner never joined the Service VIP).

- proto: HeartbeatRequest.leader_last_seq (field 14); 0 = pre-m12p5 leader → fall
  back to the status-poll readiness path
- ElectionHooks::on_heartbeat threads leader_last_seq through net + driver
- ShardReplica::note_leader_frontier_for_readiness folds the frontier into the
  lag gauge (monotonic per shard) and drives the readiness latch using a REAL
  leader frontier (never the uninitialized-0 gauge, which would false-converge a
  still-behind joiner); a joiner that WINS leadership converges trivially
- tier-3 regression: mp_idle_cluster_snapshot_joiner_flips_ready_without_traffic
  — snapshot joiner flips /health ready on an idle cluster with zero writes and
  no status poll, then proves content parity (honest convergence)
- certs: wildcard pod SAN (*.tidaldb-peers...) in k8s/cluster/certs.yaml and
  scripts/gen-cluster-certs.sh so StatefulSet scale-up/down with --seed needs no
  cert re-issue (T4 scale-to-5 broke mTLS on tidaldb-3/4); explicit per-pod
  names kept as belt-and-suspenders
- docs/profiling/m12p5-idle-readiness-elasticity.md: root-cause + fix writeup
2026-06-14 16:21:00 -06:00
jx12n
005e292cbb fix(m11): review remediation + tidal-stress perf sweep + perf wave 2
Resolve all BLOCKER/CRITICAL/WARNING findings from the m11p7/p8 review:
- tidalctl restore: safe_join path-traversal/Zip-Slip guard + fsync on write
- corrupt-WAL checkpoint_seq guard; PITR archive-before-delete
- cluster: x-tidal-relayed audit-dedup marker; forward_failures counts 5xx
- mTLS/HTTP-TLS handshake hardening; accept-loop EMFILE backoff
- per-principal rate-limit + node-token marker-pinning tests
- self-heal tier-3 coverage; 5 router-auth tests

tidal-stress: measurement-fidelity fixes (schedule-lag p99/max, exact
feed-over-SLO verdict, shed annotation) + typed Body, workload.next
184ns->68ns, RoundRobin len==1 short-circuit, HeaderValue cache;
new benches/hotpath.rs + lib.rs.

perf wave 2: signal_snapshot SmallVec/SignalKey carrier; one-get-per-type
ranking pre-pass.
2026-06-13 12:28:04 -06:00
jx12n
6651c14adc feat(m11): cluster security (m11p7) + perf instrumentation floor
m11p7 — secure the cluster, all opt-in (pre-m11p7 byte-for-byte):
- gRPC replication mTLS by default via a custom tokio-rustls acceptor +
  DynamicCertResolver; zero-drop content-hash cert rotation (k8s ..data swap,
  no pod restart, no inotify)
- inter-node HTTP TLS sharing the same resolver (one rotation, both planes) +
  per-node keyed-BLAKE3 signed x-tidal-node-token; marker-without-token -> 403
- admin audit log (operator-leg only) + per-principal rate limit (engine
  RateLimiter; sibling nodes exempt)
- k8s cert-manager manifest (certs.yaml) + scripts/gen-cluster-certs.sh fallback;
  secret.example.yaml gains TIDAL_CLUSTER_KEY (file-mounted, hot-rotatable)
- exit gate verified real: mtls.rs (gRPC foreign-pod), cluster_security.rs
  (HTTP foreign + zero-drop rotation under load), 7 security unit tests

perf — instrument floor (sweep Wave 1):
- new tidal/benches/wal.rs + tidal-server/benches/scatter.rs
- p99->mean honesty relabel; sweep manifest at docs/reviews/perf-sweep-2026-06-13.md
- add @tidal-performance agent (Martin Thompson)

new: cluster/{audit,http_tls,security}.rs, tests/cluster_security.rs,
docs/planning/milestone-11/phase-7.md
2026-06-13 01:25:35 -06:00
jx12n
1092d34c39 feat: kubernetes deployment, OpenAPI spec, guides, and docker consolidation
- Add k8s/ manifests (StatefulSet, kustomize, PDB, ServiceMonitor) + docs/runbooks/kubernetes.md
- Add tidal-server/src/openapi.rs (utoipa OpenAPI spec) and wire into router
- Add docs/guides/ (build-a-feed-app, embeddings, server-deployment) + foryou_feed example
- Consolidate tidal/docker/ into root docker/ (single canonical home)
- Update API.md, QUICKSTART.md, README.md, CLAUDE.md, check-docs.sh accordingly
2026-06-09 17:06:34 -06:00
jx12n
ad4134e280 chore: doc consolidation, seven-dimension review fixes, and commit hooks
- Eliminate the tidal/ self-contained doc mirror; docs now have two canonical
  homes (root *.md and docs/), with planning/specs/research/reviews moved up
- Remove stale .agents/skills and .ai mirrors; canonicalize skills under .claude/
- Add pre-commit hook + scripts/check-docs.sh doc-guard + scripts/install-hooks.sh
- Implement M0-M10 seven-dimension review findings across engine, net, server,
  and tidalctl (durability, replication, query, WAL, storage, CLI hardening)
2026-06-08 22:46:28 -06:00