Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement).
Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace
default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release
gate past its budget. Both push-path steps now declare
backend_options.kubernetes.resources as two YAML anchors declared once on their
first consuming step. The values are CALIBRATED against measured free node
capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's
original figure) fits on NO node and would sit Pending forever, because
`ci-build-bounds` grants permission and the nodes supply capacity, and those are
not the same thing.
The `nightly` cron described in this file for 216 days was never created, so
tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero
signal while reading like standing coverage. nightly-chaos and
nightly-security-ops now alias the anchors and have budgets matching the gate
(their 120/90 were TIGHTER on the same runner, so they would have failed
nightly for a budget reason, not a correctness one). nightly-soak is REMOVED,
not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best
node has 1700m free CPU, so it would fail on starvation rather than regression —
manufacturing a nightly false alarm. Its commands move verbatim to
docs/runbooks/nightly-soak.md.
Ring 1 — four fabrications removed from the wire.
- scatter_merge sorted and truncated without re-stamping rank, so /feed and
/search returned 1,1,2 under full placement. Reuses merge_cross_shard's
existing stamp; asserted on BOTH the multi-group merge path and the
single-group [only] fast path that bypasses it.
- aggregate_region_row's None arm invented `applied_events: 0` plus a deficit
derived from it. applied_events/lag_events are now Option<u64>, null on the
wire. leader_last_seq was also unwrap_or(0), so a node that could not reach
the LEADER computed 0 - applied = 0 for every region and reported a converged
cluster it had never measured — a fabrication pointing the dangerous way.
- tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was
actively hiding the PVC-wipe shape: a measured zero with a real deficit
rendered as "no report" instead of BEHIND. Now read off the wire; converged
exits 0, partitioned still exits nonzero.
- /sharded/* answered 201/204 for single-copy writes with nothing anywhere
saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the
existing invalid_input path. Six call sites migrated, not the two this
roadmap predicted — including docs/runbooks/cluster.md §16.3, which told
operators to run a quorum-write probe via POST /sharded/items. That probe
cannot verify quorum: the surface applies locally with no WAL append. It was
used as the safety check between every step of a staged deploy earlier today.
Ring 2 — observability. JSON_LOGS was already implemented and the deployment
simply never asked for it; the StatefulSet now sets it, plus
TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the
VictoriaLogs `service` stream field and would have blinded every query keyed on
it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the
origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only
the origin would mean each vector lands on exactly one node, replicas never
agree, and the alert built on it pages forever.
Found by measurement, not planned: the 401 path discarded every fact about
every rejection. Traefik has served 101,858 rejected requests to the public
ingress — 87.6% of all its traffic — with no record of who or why anywhere.
unauthorized_response now emits reason (missing_token vs invalid_token, the
distinction that separates a scanner from a rotation that missed a consumer)
and the forwarded client. The token is never logged.
Also: scripts/restore-fleet.sh --cluster started the soak monitor while
deliberately leaving its gate suspended, orphaning a watcher that has reported
"0/30 green nights" for 13 days. The pair now moves together. Doc-guard's
three-warning backlog is cleared with real backfill for M4/M6/M12.
Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline,
counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5;
cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with
the two remaining failures asserting the rank fix against the not-yet-rolled
image — they are the post-deploy proof.
Three real defects, plus a retracted fourth that was a probe artifact.
P3 (fixed) - query/stored normalization asymmetry. The write path L2-normalized
every stored vector; the read path passed the caller's raw query straight to the
index, so the two sides lived in different spaces. With unit v,
d = |q|^2 - 2q.v + 1, so a non-unit query shifted and scaled every distance by
|q|^2. Measured live: 591-1174 against a documented [0,4], and an exact match
scoring |q|^2 - 1 instead of ~0. vector_search_items now normalizes with the
canonical l2_normalize; a zero-norm query (no direction, so nearest-by-cosine is
undefined) is rejected with 400. Ranking is unchanged - |q|^2 and 1 are constant
across candidates - which is why it went unnoticed; what broke was every absolute
use of the number. WIRE-VISIBLE, recorded in CHANGELOG.
P2 (fixed) - the blob path had zero instrumentation. Added per-kind
tidaldb_cluster_blobs_originated/applied/apply_failed totals. Label cardinality
is fixed at 4 by construction via a new BlobKind enum, and BlobRecord::blob_kind
is now the ONE exhaustive match over the variants (kind() derives from it), so a
new variant is a compile error in one place instead of a silent zero in three.
Only the live apply path is counted - boot replay would inflate applied past
originated on every restart.
Coverage gap (fixed) - tidaldb_usearch_vector_count rendered only the metrics
owner's shard group, so on a 3-group node two thirds of the corpus had no
vector-count series at all. Co-located groups now render shard="N"; the owner
stays unlabeled for wire compatibility, so an alert grouped by (shard) buckets
each replica set separately without double-counting.
P1 (RETRACTED) - the "replica-divergent vector index" does not exist. Every probe
wrote through the /sharded/ surface, which hash-partitions and applies to the
owning region's local store with no WAL append, and therefore does not replicate
BY DESIGN (cluster/node.rs:8828-8829). A controlled A/B settled it: on /items plus
/embeddings all 6 entities reach all 3 replicas; on the sharded surface four of six
reach exactly one node. Both are now pinned by tests. See
tmp/vector-search-correctness/diagnosis.md and the k3s-fleet cluster-state.yaml
entry RETRACTED_blob_replication_rf1_2026_08_30.
Pre-work: usearch_index.rs 872 to 503 lines by extracting its tests to a sibling
(the project's existing path-attribute convention), and the three hand-rolled
l2_normalize copies collapsed to one. The two entity copies used a zero threshold
about 2900x looser than the canonical one; normalize_centroid now names the
centroid zero-tolerance policy once, and a test pins the tightened behavior.
Tests: 2107 lib (+5), 8 vector_search e2e (+4, three of which fail without the
P3 fix), 4 cluster_sharding e2e (+2). The heavy multiproc tests in
cluster_sharding are now serialized - four concurrent 3-node clusters made the
pre-existing failover test miss its 10s budget.
M0-M12 are shipped and the HA cluster runs in production on k3s, so the
pre-release disclaimer no longer describes the project. Removes it from the
canonical doc set and corrects the readiness text that had gone stale.
- README.md: replace the "Pre-release / not yet recommended for production"
banner with a production-ready statement; drop "(experimental)" from the
cluster status bullet; state the post-1.0 versioning posture (additive in
minor releases, breaking changes get a documented migration path).
- CLAUDE.md / QUICKSTART.md / docs/guides/server-deployment.md /
docs/runbooks/cluster.md: same withdrawal; reframe the cluster opt-in as a
guard against standing up a multi-node fabric by accident rather than a
readiness warning.
- CHANGELOG.md: record the stability posture under [Unreleased], superseding
the historical 0.1.0 "no stability guarantees" note (left intact as history).
- k8s/statefulset.yaml: the "NOT production HA, tracked as m8p10" comment was
stale (m8p10 shipped); point at k8s/cluster/ for the HA deployment instead.
Also corrects text that was factually wrong since m11p3/m11p4: the
multi-process cluster gate, its CLI help, and the served OpenAPI description
all still claimed quorum-ack writes and automatic failure detection did not
exist. They do.
Historical records (docs/reviews/, docs/profiling/, past CHANGELOG entries,
the kubernetes.md rc7 fix note) are left unchanged.
Verified against a running binary, not just the build: the opt-in gate's
refusal message, the startup WARN, /health 200, and the served
/openapi.json description all carry the new text. cargo fmt clean; clippy
-D warnings clean on tidaldb and the tidal-server lib; 1943 engine + 155
server lib tests pass; scripts/check-docs.sh OK.
Claude-Session: https://claude.ai/code/session_01QdqSDw1tUhK1JT9Pb1vryP
End the "replicated XOR sharded" split: S shard groups, each a
replication group at RF with its own elected leader, leaders balanced
across nodes; any gateway hash-routes.
- One unified write surface: /items,/embeddings,/signals hash-route to
the owning shard group's leader (ShardRouter FNV-1a) AND replicate at
RF. x-tidal-ack/x-tidal-seq, quorum await, NotLeader/QuorumTimeout are
per-group; NotLeader names the group.
- Rebalance verbs (L3): POST /cluster/shards/{id}/transfer (fenced
leadership move) + /cluster/shards/{id}/replicas (add/remove replica).
A ?shard= selector threads through every per-shard admin verb and is
propagated on intra-group forwards (ShardReplica::admin_path). S=1 is
byte-for-byte (no selector, no shard in NotLeader body).
- Tier-3 exit gate (cluster_sharding.rs): 3 nodes × 3 shards × RF=3 over
real OS processes — SIGKILL a node under ack=quorum load → only its
shard-leaderships re-elect, reads never stop, zero acked loss across
random kill points; plus a rebalance-verb test. Harness:
MultiProcCluster::start_sharded.
- tidal-stress drives the single path (WritePath::Leader|Sharded gone),
spreading writes round-robin across gateways or pinning --leader-url.
- Throughput: local 3×3 sustains 3,000 quorum signal-writes/s @ 0% err,
~30% CPU, lag ~0 (generator-bound). ≥5,000/s + ≥2.5× scaling is Ref-A.
Known follow-up (tracked): per-group-aware node readiness and cross-node
read fan-out under PARTIAL placement.
Kind-3 term markers in the WAL stream, STREAM-relative vote frontiers,
heartbeat-only divergence detection + quarantine, and fenced promote.
Elections converge in 0.6–1.0s; zero acked-write loss across all kill points.
Closes G5 (leaderless recovery) from the v0.9 wave.
WAL segment format: 8-byte TSEG header (magic + version byte + 3 reserved)
prepended to every new segment. Legacy headerless segments (m0-m11p3) read
as implicit v0 — no migration. Unknown magic/version surfaces as
WalError::SegmentFormatUnknown at open time; foreign files are never
repaired or truncated (fixes the silent data-loss path from the p3 rollout
incident where torn-tail repair zeroed a follower's unreadable segments).
Catch-up transport: FAILED_PRECONDITION ("snapshot required") and stream
errors that skip the shard now arm a timer retry (re-arm-on-skip is the
load-bearing liveness fix — without it a skipped pull never re-fires and
the follower stays permanently behind). Single retry pending per shard;
CatchupRunner owns the Arc'd state shared between the retry tasks and the
transport. Test: tidal-net/tests/catchup_retry.rs covers the retry path.
Stress: k8s stress-job-t2a/t2b yaml + ops/stress-test-p3-t2 runbook.
ack=quorum gates replicated writes on a majority of the replica set durably
holding them: followers push their durably-applied frontier (ReportApplied,
once per apply round, decoupled from ship acks), the leader folds frontier
reports + ship-ack hints + heal resumes into a leadership-scoped CommitIndex
(k-th-largest durable mark), and handlers await it through an async
watch-channel bridge (zero parked threads per waiter). Honest timeouts:
retryable 503 naming the laggards; x-tidal-seq on every cluster write.
Follower blob applies are batched under group-commit fsyncs (22x seeding).
Exit gate: 167/167 leader-SIGKILL kill points, zero acked-write loss.
Seven-dimension review pass (all confirmed findings fixed):
- WAL blob drain now ABORTS on the first write failure instead of reusing
the failed seqno mid-drain (a torn record buried mid-segment would
truncate every later acked record on replay)
- apply_replicated_blobs waits every staged append even after a mid-batch
failure, parses metadata once, and moves records into Arcs shared with
the WAL writer (no deep clone per record on the follower apply path)
- CommitIndex: zero-peer fast path now respects demotion (active checked
under lock before the single-replica return), k-th-largest uses
select_nth over a reused scratch buffer
- await_quorum: re-reads the index once after the deadline fires (no false
503 for a write that committed in the race window), warns when the
commit-watch bridge dies outside shutdown, zero-peer path checks active
- notify_applied report failures: WARN on the first failure of a streak,
INFO on recovery (a silently stalling frontier reads as unexplained
quorum 503s); receiver skips re-notifying unadvanced frontiers
- x-tidal-deduplicated: 1 marks dedup-suppressed signal writes (relayed
through forwards) so durability cursors can tell dedup from no-seqno
- docs: 167/167 kill-point record corrected in CHANGELOG; rolling-upgrade
order (leader first — a pre-m11p3 leader silently downgrades quorum
requests to leader-ack) in CHANGELOG + runbook §8; monitoring note for
report-loss diagnosis on the quorum-timeout alert
Verified: workspace clippy -D warnings (incl. cluster-e2e targets), full
tidaldb/tidal-net/tidal-server/tidalctl suites green, tier-3 multi-process
quorum suite green (8/8 kill points, zero acked loss, partition gate/recover).
m11p1 — decoupled ack/ship path: staged writes (seqno+WAL+relay-push,
microseconds) separate from group-commit fsync; ShipQueue batches+windows
outbound segments; receiver coalesces inbound chunks before applying.
Adds first tidaldb_cluster_* metrics.
m11p2 — leader WAL is now THE replicated log: fsynced batches feed a
bounded WalShipFeed and ship byte-identical to followers; WAL seqnos
survive restarts (relay-reset hazard gone). Item metadata and embeddings
journal kind-1/2 blob records on the same stream as signals; the m8p10
HTTP broadcast is deleted. StreamSegments catch-up is follower-pulled via
server-streaming RPC, triggered on gap detection, follower boot, and
leader heal nudge. Promote carries a stream baseline so peers skip
pre-stream history.
Splits monolithic cluster.rs into tidal-server/src/cluster/ modules. Adds redeliver-missed
relay, bounded HLC drift, lag tracking, and reconcile idempotence. Five new tier-3 test suites
(chaos, lifecycle, multiproc, region, routes, runbook) all green. Docs, CHANGELOG, and ROADMAP
updated with G4/G5/G6 known gaps.
- Eliminate the tidal/ self-contained doc mirror; docs now have two canonical
homes (root *.md and docs/), with planning/specs/research/reviews moved up
- Remove stale .agents/skills and .ai mirrors; canonicalize skills under .claude/
- Add pre-commit hook + scripts/check-docs.sh doc-guard + scripts/install-hooks.sh
- Implement M0-M10 seven-dimension review findings across engine, net, server,
and tidalctl (durability, replication, query, WAL, storage, CLI hardening)
