Read-SLA fix (rc12→rc13 — cpu-cgroup starvation → multi-second p99 + churning
elections):
- offload.rs: add SEARCH_GATE semaphore (core_count+1 permits, 50ms shed to 429)
so per-shard searches gate on CPU, not reactor threads; concurrent scatter_merge
fan-out (join_all) replaces the serial blocking offload_region_read loop
- node.rs: scatter_merge → async; per-shard futures run via offload_search
(each acquires one SEARCH_GATE permit, moves it into spawn_blocking so the
permit is held for the search's full CPU lifetime)
- main.rs: explicit tokio runtime with worker_threads floored at 4, independent
of the cgroup quota — keeps the control plane (heartbeat/election/apply) on its
own workers even when quota < 4
- k8s statefulset: CPU limit 2→3 (was: available_parallelism()=2 → only 2 async
workers; search burst starved the reactor)
- tidal/wal/compaction.rs: WAL_RETENTION_SEGMENTS 4→16 (64 MiB→256 MiB per-shard
catch-up window; a briefly-down follower across a rolling restart streams up
instead of forcing snapshot reseed; disk floor 768 MiB/pod, self-trimming)
- cluster_reseed.rs: OFFLINE_ITEMS 1800→5600 to exceed the new 16-segment
retention window (19 segs > 17); fix sequential quarantine/reseed race via
await_status_bool
tidalctl S3/R2 backup DR:
- tidalctl/Cargo.toml: aws-config, aws-sdk-s3, aws-credential-types, tokio, tempfile
- commands/s3.rs: S3Target + export_dir (upload every file, manifest last as
atomicity marker) + import_to_dir (download prefix into temp staging dir)
- commands/backup.rs: run_backup/run_restore accept Option<&S3Target>; S3 export
is additive after local fsync barrier; S3 import stages into TempDir then runs
the unchanged verified restore on it
- main.rs: --s3-endpoint / --s3-bucket / --s3-prefix flags; all-or-nothing
endpoint+bucket validation; usage updated
tidal-stress/k8s: recall-rc12-spread-job, soak-nightly-cronjob, soak-monitor,
soak-results-pvc, t5-readtput-job manifests
Root-caused and fixed five sharding bugs exposed on the real k3s 3-shard
cluster (rc5→rc7), plus a divergent-rejoin reseed loop found in rc9:
1. reseed shard-awareness (Bug 3, keystone): `run_boot_install_for_region`
visits each hosted group's own shard subdir; per-group leader discovery
appends `?shard=N` so a divergent shard heals from its own leader (not
shard-0's WAL/term — cross-shard contamination).
2. leader self-join term (Bug 4): `become_leader_for_term` now calls
`note_self_won_term` so the elected shard's `joined_term` is set and
`cluster_promote` routes rebalances correctly (was: topology-era mis-read
→ legacy fenced promote → 500).
3. boot self-heal self-pull guard (Bug 2): `leader_shard != my_shard` gate
prevents a node pulling its own stream (its stream isn't a registered peer)
→ eliminates the `PeerUnreachable(self)` loop.
4. scatter-merge degraded partial (Bug 1): failed shard logs + continues
instead of `?`-failing the whole read; bounded read-admission semaphore
(`offload.rs`) sheds as 429 instead of piling into a 36s p99.
5. WAL retention (Bug 5): `compact_wal_retained` keeps `WAL_RETENTION_SEGMENTS=4`
most-recent sealed segments; online path gets the same retention clamp.
Prevents brief-restart forced-reseed.
6. divergent-rejoin reseed loop (Bug 6, rc9): `note_quarantined` latches
`from_seqno = stream_baseline` (not `frontier + 1`) so `wal_covers`
returns `needed=true` and the snapshot installs instead of looping.
Also: `TidalDb::close_shared` for deterministic HNSW save on cluster SIGTERM
(HNSW graph was not saved when request-scoped Arc clones were alive at shutdown);
updated profiling doc with full rc8/rc9 fix narrative; k8s recall job YAMLs.
m11p1 — decoupled ack/ship path: staged writes (seqno+WAL+relay-push,
microseconds) separate from group-commit fsync; ShipQueue batches+windows
outbound segments; receiver coalesces inbound chunks before applying.
Adds first tidaldb_cluster_* metrics.
m11p2 — leader WAL is now THE replicated log: fsynced batches feed a
bounded WalShipFeed and ship byte-identical to followers; WAL seqnos
survive restarts (relay-reset hazard gone). Item metadata and embeddings
journal kind-1/2 blob records on the same stream as signals; the m8p10
HTTP broadcast is deleted. StreamSegments catch-up is follower-pulled via
server-streaming RPC, triggered on gap detection, follower boot, and
leader heal nudge. Promote carries a stream baseline so peers skip
pre-stream history.
- Eliminate the tidal/ self-contained doc mirror; docs now have two canonical
homes (root *.md and docs/), with planning/specs/research/reviews moved up
- Remove stale .agents/skills and .ai mirrors; canonicalize skills under .claude/
- Add pre-commit hook + scripts/check-docs.sh doc-guard + scripts/install-hooks.sh
- Implement M0-M10 seven-dimension review findings across engine, net, server,
and tidalctl (durability, replication, query, WAL, storage, CLI hardening)
Resolves the 142 findings from tidal/docs/reviews/CODE_REVIEW_m0-m10.md across
the engine, server, net, and CLI surfaces:
- WAL/session-journal durability, checkpoint format, and crash-recovery hardening
- Replication shipper/receiver, tenant isolation, and migration paths
- Cluster scatter-gather, router, standalone server + health/offload endpoints
- tidalctl refactored into command modules with JSON output and WAL-state tooling
- Cohort, governance, signal-ledger, and vector-registry correctness fixes
- Expanded UAT/integration/durability test coverage across all milestones
