a6f663f002
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
Fixes the two defects a malformed probe exposed on the live cluster, plus the
coverage gap that let a stale assertion survive the same day it was falsified.
TASK 17 — validate before the WAL append. A 128-dim vector against a 1536-dim
slot was appended to the WAL FIRST, then validated, then answered 500 — so an
already-durable, unapplicable record shipped to both followers, halted both
receivers, and put shard 1 into a quorum-write outage. Validation now runs before
the append and returns 400 via invalid_input; nothing enters the log.
`storage::vector::validate_dimensions` is now the single comparison, replacing an
inline duplicate of the same rule in lifecycle/ops.rs:57-62 — two copies of a
dimension check drift, and the apply-path copy is the one that halts replication
when it disagrees.
The receiver's halt-vs-skip decision is now explicit instead of "halt on
anything". A record whose failure is deterministic and node-independent (schema
width) is skipped, counted on blobs_apply_failed_total and ERROR-logged, so the
frontier advances; a record that could become applicable after a binary upgrade
(unknown batch kind, capability skew) still halts, because skipping那 would
silently drop replicated data. Both branches are proven reachable by tests.
TASK 18 — the reseed latch outlived its discharge. A node hosting 3 shard groups
latched a marker per group but discharged on a single seqno, so two latches meant
permanent 503 on a node whose every shard read lag 0 — it hit all three pods
during the roll and each needed a manual delete. Gaps are now tracked per group
in a ReseedGapSet and cleared on evidence about themselves; a REFUSED
reseed_self_restart re-evaluates every 15s instead of waiting for a latch that
never arrives. /health's cause ladder was also lying: it printed "joiner boot not
yet converged" for a node whose groups had all converged, because the fallback
asserted a state it never tested. It now names the outstanding gaps, gained the
decommissioned-by-signal arm that is_ready checked but the ladder did not, and
its terminal arm says "reason unavailable" rather than inventing one.
COVERAGE — 14 of 23 integration suites were run by NO pipeline. Not theoretical:
cluster_routes still asserted the wire fabrication removed hours earlier
(applied_events == 0 with a lag derived from it) and nothing caught it because
nothing ran it. cluster_sharding (dense-rank, /sharded/* opt-in), vector_search
(distance contract) and cluster_poison_embedding (task 17's own gate) were in the
same position, so those guards would have rotted identically. Every suite now has
a runner: 8 in-process ones in a new `fast-suites` push step (measured 71s, runs
FIRST so a cheap failure precedes the 6.5-min gate), 6 multiproc ones in the
nightly. All 23 scheduled; all 4 never-before-run heavy suites verified passing
before being scheduled.
Also fixes cluster_chaos.rs:329, which the nightly's FIRST EVER run caught 13
minutes in — it demanded an unreachable peer report worst-case lag, i.e. it
required the fabrication task 04a deleted.
Verified: fmt clean; clippy 72 vs 73 baseline (one FEWER, zero added, measured on
touched trees at
|
||
|---|---|---|
| .. | ||
| cluster.md | ||
| deploy-verification.md | ||
| disaster-recovery.md | ||
| kubernetes.md | ||
| nightly-soak.md | ||
| on-call.md | ||
| README.md | ||
tidalDB Runbooks — index
Operational documentation for the cluster deployment (ns tidaldb-cluster,
single StatefulSet, full-placement RF3, quorum-ack + automatic election). Live state
and history live in the orchard9-k3sf repo (cluster-state.yaml,
deployments/history/tidaldb.md).
| When you need to… | Open |
|---|---|
| Verify a deploy end to end — every layer, with the command and its expected output | deploy-verification.md |
| Respond to an incident / page — symptom → signal → fix | on-call.md |
| Operate the cluster — API, failover, partition, membership, scale, rolling upgrade | cluster.md |
| Run it on Kubernetes — deploy, digest-pin, probes, scale up/down, troubleshoot pods | kubernetes.md |
| Back up / restore / DR — object-store export, restore, byte-verify, query-proof, R2, rebuild | disaster-recovery.md |
| Recover a single node / standalone engine — corrupt keyspace, WAL, lock, schema | ../ops/recovery.md |
| Size a deployment — single-node tables + the measured Ref-A cluster envelope | ../ops/capacity-planning.md |
| Soak before a release — 1000 rps load with p99 / error-rate gates (moved out of CI on 2026-08-30; the shared runner cannot meet a 250ms p99) | nightly-soak.md |
| Read the metrics / wire dashboards & alerts | ../ops/observability.md, ../ops/grafana-tidaldb.json, ../ops/prometheus-alerts.yaml |
| Understand the live perf/topology findings (dev handoff) | ../profiling/m12-cluster-deploy-findings.md |
Current production facts (2026-06-19)
- Image:
registry.threesix.ai/tidal/server@sha256:171505745b…(m12-writeburst-rc7). - Shape: 1 StatefulSet
tidaldb, 3 pods = 3 regions = 3 voters, full-placement RF3 (every pod hosts all 3 shard groups; gRPC 9601/9602/9603), HTTPS+mTLS on:9500, metrics:9091. - Guarantees live: quorum-ack writes, automatic election/failover, elastic seed-join (learner→voter), inter-node mTLS, per-node Prometheus.
- Shipped fixes in this image: reseed-loop resolved, seed-join promotion, election-divergence/quarantine, read-SLA (CPU oversubscription), and the write-burst false-partition fix.
- Known limits: read p99 ≤10 ms to ~1000 rps (ceiling ~1–1.5k/s, CPU-bound); write knee ~250 rps; soak at 200 rps; 1M corpus needs >16 GiB nodes; 2.5× write-scaling needs ≥5 nodes + partitioned placement.
- Must-watch signal:
tidaldb_cluster_peer_breaker_state(0/1/2) — a live peer stuck at 1 is the write-burst/partition pattern.
Alert rules in
../ops/prometheus-alerts.yamlare accurate but design-reference — not yet loaded by the live Prometheus. Promoting them to aPrometheusRuleis the one open observability step (seeon-call.md§3).