8e39ee1078
The m12p5 idle-readiness work converged on an idle cluster, but the real T4 1M/1536 scale-up over mTLS still failed to admit new pods. Three real blockers, all invisible to the plaintext in-process tests: - CryptoProvider crash-loop: the seed-join/reseed boot path builds a blocking reqwest (rustls) HTTPS client on a dedicated boot thread BEFORE GrpcTransport::new installs the process-wide provider, so every TLS joiner panicked. Install it at the top of main(); ensure_crypto_provider() is now pub, idempotent, harmless on the plaintext standalone path. - Wrong seed scheme + target: peer_url honors an explicit URL scheme verbatim, so http:// dialed plaintext at the TLS :9500 port. Seed is now https:// AND points at the ready-only client Service (ClusterIP VIP), not the headless peers Service — so a joiner never round-robins onto a not-ready pod (incl. itself) and burns the 120s discovery window. - Too-tight poll budget: a cold status poll pays a full rustls handshake on top of DNS+TCP; under CPU contention that alone blew the 500ms budget, so the joiner timed out every poll for the whole window despite the peer being reachable. Status-poll timeout is now 5s (env: TIDAL_SEED_STATUS_TIMEOUT_MS) with a separate 2s connect timeout (dead seeds still fail fast) and debug-level logging on every discovery failure mode. Refactors riding along: - on_heartbeat takes a HeartbeatContext struct (additive fields, no silent u64 transposition) across tidal-net, election_driver, and both test hooks. - ShardReplica::applied_for_leader_shard centralizes per-source-shard keying (BUG 1) shared by the readiness drive and local_status. - idle-readiness test now asserts convergence within ½ budget — a slow-path regression (periodic self-heal / status-poll dependency) the binary budget check would otherwise wave through. New k8s T4 manifests: cluster-t4-kind kustomization + single-group topology patch; tidal-stress t4 seed/load Jobs.
54 lines
2.2 KiB
YAML
54 lines
2.2 KiB
YAML
# m12p5 T4 overlay — SINGLE replication group (no `shards:`).
|
|
#
|
|
# T4 is REPLICA elasticity + idle-readiness (scale the voter set 3→5→3), NOT
|
|
# sharding (that is m12p4/T5). A single group RF=all is the faithful T4 shape:
|
|
# it mirrors the in-process exit gate `mp_scale_3_5_3_under_load_zero_loss` and
|
|
# uses the well-proven m11p5 seed-join path (the sharded-cluster *replica*
|
|
# scale-up — 5 replicas per group — is an untested path outside m12p5's scope).
|
|
#
|
|
# Strategic-merge replaces the base ConfigMap's `cluster-topology.yaml` value:
|
|
# identical to k8s/cluster/topology-configmap.yaml MINUS the `shards:` block, so
|
|
# every pod runs ONE group binding gRPC 9601 (the extra 9602/9603 containerPorts
|
|
# in the StatefulSet are simply unbound — harmless).
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: tidaldb-cluster-topology
|
|
namespace: tidaldb-cluster
|
|
data:
|
|
cluster-topology.yaml: |
|
|
regions:
|
|
- name: tidaldb-0
|
|
grpc_addr: tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9601
|
|
grpc_bind: 0.0.0.0:9601
|
|
http_addr: tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
|
|
metrics_addr: 0.0.0.0:9091
|
|
grpc_tls: &grpc_tls
|
|
ca_cert: /etc/tidaldb/tls/ca.crt
|
|
server_cert: /etc/tidaldb/tls/tls.crt
|
|
server_key: /etc/tidaldb/tls/tls.key
|
|
client_cert: /etc/tidaldb/tls/tls.crt
|
|
client_key: /etc/tidaldb/tls/tls.key
|
|
- name: tidaldb-1
|
|
grpc_addr: tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9601
|
|
grpc_bind: 0.0.0.0:9601
|
|
http_addr: tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
|
|
metrics_addr: 0.0.0.0:9091
|
|
grpc_tls: *grpc_tls
|
|
- name: tidaldb-2
|
|
grpc_addr: tidaldb-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9601
|
|
grpc_bind: 0.0.0.0:9601
|
|
http_addr: tidaldb-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
|
|
metrics_addr: 0.0.0.0:9091
|
|
grpc_tls: *grpc_tls
|
|
leader: tidaldb-0
|
|
replication:
|
|
ack: quorum
|
|
reseed_self_restart: true
|
|
election:
|
|
heartbeat_interval_ms: 300
|
|
election_timeout_min_ms: 1500
|
|
election_timeout_max_ms: 3000
|
|
leader_lease_ms: 900
|
|
auto_election: true
|