tidaldb

Author SHA1 Message Date

Author	SHA1	Message	Date
jx12n	727fbfcb6b	fix(m12p6): 6-bug k3s 3-shard cluster repair (rc8+rc9) Root-caused and fixed five sharding bugs exposed on the real k3s 3-shard cluster (rc5→rc7), plus a divergent-rejoin reseed loop found in rc9: 1. reseed shard-awareness (Bug 3, keystone): `run_boot_install_for_region` visits each hosted group's own shard subdir; per-group leader discovery appends `?shard=N` so a divergent shard heals from its own leader (not shard-0's WAL/term — cross-shard contamination). 2. leader self-join term (Bug 4): `become_leader_for_term` now calls `note_self_won_term` so the elected shard's `joined_term` is set and `cluster_promote` routes rebalances correctly (was: topology-era mis-read → legacy fenced promote → 500). 3. boot self-heal self-pull guard (Bug 2): `leader_shard != my_shard` gate prevents a node pulling its own stream (its stream isn't a registered peer) → eliminates the `PeerUnreachable(self)` loop. 4. scatter-merge degraded partial (Bug 1): failed shard logs + continues instead of `?`-failing the whole read; bounded read-admission semaphore (`offload.rs`) sheds as 429 instead of piling into a 36s p99. 5. WAL retention (Bug 5): `compact_wal_retained` keeps `WAL_RETENTION_SEGMENTS=4` most-recent sealed segments; online path gets the same retention clamp. Prevents brief-restart forced-reseed. 6. divergent-rejoin reseed loop (Bug 6, rc9): `note_quarantined` latches `from_seqno = stream_baseline` (not `frontier + 1`) so `wal_covers` returns `needed=true` and the snapshot installs instead of looping. Also: `TidalDb::close_shared` for deterministic HNSW save on cluster SIGTERM (HNSW graph was not saved when request-scoped Arc clones were alive at shutdown); updated profiling doc with full rc8/rc9 fix narrative; k8s recall job YAMLs.	2026-06-16 22:34:21 -06:00
jx12n	4db3f1e597	fix(m12p6): complete T4 TLS scale-up — two-tier PKI + join_boot grpc_tls fallback Completes the seed-join-over-TLS enablement begun in `8e39ee1`. A real kubectl scale 3->5 on a real mTLS k8s cluster (kind) exercised the seed-join path over TLS for the first time and surfaced two more blockers beyond 8e39ee1's https-seed / ready-only-Service / up-front-rustls-provider fixes — both of which crash-looped every scale-up joiner with the same opaque 'could not join within 120s'. The plaintext in-process harness is blind to all of them. - certs.yaml: a real TWO-TIER PKI. The leaf was issued DIRECTLY from a selfSigned Issuer (a self-signed CA:FALSE end-entity whose ca.crt is a copy of the leaf); the joiner's strict webpki verifier rejected the peer cert as UnknownIssuer. Now: selfSigned Issuer -> CA cert (CA:TRUE) -> ca: Issuer signs the leaf. (scripts/gen-cluster-certs.sh already did this; the two were inconsistent.) - join_boot.rs: grpc_tls_for() fallback. own_grpc_tls/self_tls_spec looked up the joiner's OWN region in the knob file to find its TLS material, but a seed-joiner is NEVER in the shared-ConfigMap regions: list -> None -> the seed client built with NO CA (the real UnknownIssuer cause) and a plaintext synthesized topology. Fall back to ANY region's block (every pod mounts the same cert files). - join_boot.rs: STATUS_POLL_TIMEOUT 500ms -> 5s (env TIDAL_SEED_STATUS_TIMEOUT_MS); a cold TLS handshake under contention blew the sub-second budget. Discovery now logs each poll failure at WARN with the full error source chain (a silent loop made every bug present as the same 120s timeout). - statefulset.yaml: pin the m12-8e39ee1 server image (carries these fixes). - k8s/cluster-t4-kind + tidal-stress/k8s/t4-*: local-kind T4 overlay + seed/load. Verified GREEN on kind: idle scale 3->5, both joiners seed-join over mTLS, catch up, and flip /health Ready in 13s via the idle-readiness heartbeat convergence; auto-promote to Voter; full content parity; all 5 regions lag=0. clippy clean; mp_seed_join_snapshot_catchup + mp_idle_cluster_..._without_traffic green; tidal-server/tidal-net lib green. A separate, root-caused snapshot-frontier bug on a DEEPLY-compacted WAL (node.rs:734 last_wal_seq=0 for a state-only artifact) is documented as a follow-up — left unfixed because a naive patch broke the in-process snapshot test (own-WAL<->stream numbering); the GREEN run uses a small corpus (stream catch-up) to keep that path out of scope. See docs/profiling/m12p5-idle-readiness-elasticity.md §6.	2026-06-14 22:41:59 -06:00
jx12n	8e39ee1078	fix(m12p6): T4 TLS scale-up enablement — https seed-join via ready-only Service + up-front rustls provider The m12p5 idle-readiness work converged on an idle cluster, but the real T4 1M/1536 scale-up over mTLS still failed to admit new pods. Three real blockers, all invisible to the plaintext in-process tests: - CryptoProvider crash-loop: the seed-join/reseed boot path builds a blocking reqwest (rustls) HTTPS client on a dedicated boot thread BEFORE GrpcTransport::new installs the process-wide provider, so every TLS joiner panicked. Install it at the top of main(); ensure_crypto_provider() is now pub, idempotent, harmless on the plaintext standalone path. - Wrong seed scheme + target: peer_url honors an explicit URL scheme verbatim, so http:// dialed plaintext at the TLS :9500 port. Seed is now https:// AND points at the ready-only client Service (ClusterIP VIP), not the headless peers Service — so a joiner never round-robins onto a not-ready pod (incl. itself) and burns the 120s discovery window. - Too-tight poll budget: a cold status poll pays a full rustls handshake on top of DNS+TCP; under CPU contention that alone blew the 500ms budget, so the joiner timed out every poll for the whole window despite the peer being reachable. Status-poll timeout is now 5s (env: TIDAL_SEED_STATUS_TIMEOUT_MS) with a separate 2s connect timeout (dead seeds still fail fast) and debug-level logging on every discovery failure mode. Refactors riding along: - on_heartbeat takes a HeartbeatContext struct (additive fields, no silent u64 transposition) across tidal-net, election_driver, and both test hooks. - ShardReplica::applied_for_leader_shard centralizes per-source-shard keying (BUG 1) shared by the readiness drive and local_status. - idle-readiness test now asserts convergence within ½ budget — a slow-path regression (periodic self-heal / status-poll dependency) the binary budget check would otherwise wave through. New k8s T4 manifests: cluster-t4-kind kustomization + single-group topology patch; tidal-stress t4 seed/load Jobs.	2026-06-14 20:29:37 -06:00

jx12n

727fbfcb6b

fix(m12p6): 6-bug k3s 3-shard cluster repair (rc8+rc9)

Root-caused and fixed five sharding bugs exposed on the real k3s 3-shard
cluster (rc5→rc7), plus a divergent-rejoin reseed loop found in rc9:

1. reseed shard-awareness (Bug 3, keystone): `run_boot_install_for_region`
   visits each hosted group's own shard subdir; per-group leader discovery
   appends `?shard=N` so a divergent shard heals from its own leader (not
   shard-0's WAL/term — cross-shard contamination).
2. leader self-join term (Bug 4): `become_leader_for_term` now calls
   `note_self_won_term` so the elected shard's `joined_term` is set and
   `cluster_promote` routes rebalances correctly (was: topology-era mis-read
   → legacy fenced promote → 500).
3. boot self-heal self-pull guard (Bug 2): `leader_shard != my_shard` gate
   prevents a node pulling its own stream (its stream isn't a registered peer)
   → eliminates the `PeerUnreachable(self)` loop.
4. scatter-merge degraded partial (Bug 1): failed shard logs + continues
   instead of `?`-failing the whole read; bounded read-admission semaphore
   (`offload.rs`) sheds as 429 instead of piling into a 36s p99.
5. WAL retention (Bug 5): `compact_wal_retained` keeps `WAL_RETENTION_SEGMENTS=4`
   most-recent sealed segments; online path gets the same retention clamp.
   Prevents brief-restart forced-reseed.
6. divergent-rejoin reseed loop (Bug 6, rc9): `note_quarantined` latches
   `from_seqno = stream_baseline` (not `frontier + 1`) so `wal_covers`
   returns `needed=true` and the snapshot installs instead of looping.

Also: `TidalDb::close_shared` for deterministic HNSW save on cluster SIGTERM
(HNSW graph was not saved when request-scoped Arc clones were alive at shutdown);
updated profiling doc with full rc8/rc9 fix narrative; k8s recall job YAMLs.

2026-06-16 22:34:21 -06:00

jx12n

4db3f1e597

fix(m12p6): complete T4 TLS scale-up — two-tier PKI + join_boot grpc_tls fallback

Completes the seed-join-over-TLS enablement begun in 8e39ee1. A real
kubectl scale 3->5 on a real mTLS k8s cluster (kind) exercised the seed-join
path over TLS for the first time and surfaced two more blockers beyond 8e39ee1's
https-seed / ready-only-Service / up-front-rustls-provider fixes — both of which
crash-looped every scale-up joiner with the same opaque 'could not join within
120s'. The plaintext in-process harness is blind to all of them.

- certs.yaml: a real TWO-TIER PKI. The leaf was issued DIRECTLY from a selfSigned
  Issuer (a self-signed CA:FALSE end-entity whose ca.crt is a copy of the leaf);
  the joiner's strict webpki verifier rejected the peer cert as UnknownIssuer.
  Now: selfSigned Issuer -> CA cert (CA:TRUE) -> ca: Issuer signs the leaf.
  (scripts/gen-cluster-certs.sh already did this; the two were inconsistent.)
- join_boot.rs: grpc_tls_for() fallback. own_grpc_tls/self_tls_spec looked up the
  joiner's OWN region in the knob file to find its TLS material, but a seed-joiner
  is NEVER in the shared-ConfigMap regions: list -> None -> the seed client built
  with NO CA (the real UnknownIssuer cause) and a plaintext synthesized topology.
  Fall back to ANY region's block (every pod mounts the same cert files).
- join_boot.rs: STATUS_POLL_TIMEOUT 500ms -> 5s (env TIDAL_SEED_STATUS_TIMEOUT_MS);
  a cold TLS handshake under contention blew the sub-second budget. Discovery now
  logs each poll failure at WARN with the full error source chain (a silent loop
  made every bug present as the same 120s timeout).
- statefulset.yaml: pin the m12-8e39ee1 server image (carries these fixes).
- k8s/cluster-t4-kind + tidal-stress/k8s/t4-*: local-kind T4 overlay + seed/load.

Verified GREEN on kind: idle scale 3->5, both joiners seed-join over mTLS, catch
up, and flip /health Ready in 13s via the idle-readiness heartbeat convergence;
auto-promote to Voter; full content parity; all 5 regions lag=0. clippy clean;
mp_seed_join_snapshot_catchup + mp_idle_cluster_..._without_traffic green;
tidal-server/tidal-net lib green. A separate, root-caused snapshot-frontier bug
on a DEEPLY-compacted WAL (node.rs:734 last_wal_seq=0 for a state-only artifact)
is documented as a follow-up — left unfixed because a naive patch broke the
in-process snapshot test (own-WAL<->stream numbering); the GREEN run uses a small
corpus (stream catch-up) to keep that path out of scope. See
docs/profiling/m12p5-idle-readiness-elasticity.md §6.

2026-06-14 22:41:59 -06:00

jx12n

8e39ee1078

fix(m12p6): T4 TLS scale-up enablement — https seed-join via ready-only Service + up-front rustls provider

The m12p5 idle-readiness work converged on an idle cluster, but the real
T4 1M/1536 scale-up over mTLS still failed to admit new pods. Three real
blockers, all invisible to the plaintext in-process tests:

- CryptoProvider crash-loop: the seed-join/reseed boot path builds a
  blocking reqwest (rustls) HTTPS client on a dedicated boot thread BEFORE
  GrpcTransport::new installs the process-wide provider, so every TLS joiner
  panicked. Install it at the top of main(); ensure_crypto_provider() is now
  pub, idempotent, harmless on the plaintext standalone path.

- Wrong seed scheme + target: peer_url honors an explicit URL scheme
  verbatim, so http:// dialed plaintext at the TLS :9500 port. Seed is now
  https:// AND points at the ready-only client Service (ClusterIP VIP), not
  the headless peers Service — so a joiner never round-robins onto a
  not-ready pod (incl. itself) and burns the 120s discovery window.

- Too-tight poll budget: a cold status poll pays a full rustls handshake on
  top of DNS+TCP; under CPU contention that alone blew the 500ms budget, so
  the joiner timed out every poll for the whole window despite the peer being
  reachable. Status-poll timeout is now 5s (env: TIDAL_SEED_STATUS_TIMEOUT_MS)
  with a separate 2s connect timeout (dead seeds still fail fast) and
  debug-level logging on every discovery failure mode.

Refactors riding along:
- on_heartbeat takes a HeartbeatContext struct (additive fields, no silent
  u64 transposition) across tidal-net, election_driver, and both test hooks.
- ShardReplica::applied_for_leader_shard centralizes per-source-shard keying
  (BUG 1) shared by the readiness drive and local_status.
- idle-readiness test now asserts convergence within ½ budget — a slow-path
  regression (periodic self-heal / status-poll dependency) the binary budget
  check would otherwise wave through.

New k8s T4 manifests: cluster-t4-kind kustomization + single-group topology
patch; tidal-stress t4 seed/load Jobs.

2026-06-14 20:29:37 -06:00

3 Commits