jordan/tidaldb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
108 Commits 1 Branch 0 Tags 14 MiB
54d1353103
Commit Graph

7 Commits

Author SHA1 Message Date
jordan
54d1353103 fix(cluster): make a multi-group node's status readable 
Two status defects turned this incident into a day of misreading. Both are
observability, both are why the functional bug survived, and neither changes
readiness or replication behaviour.

1. PER-GROUP RESEED STATE. `reseed_required` / `reseeding` existed only as flat
   fields on LocalStatusResponse, and `status_local` fills those from
   `replica_for(sel.shard_id())` — the LOWEST hosted group id when no `?shard=` is
   given. On a 3-group node they therefore describe one group and say nothing about
   the other two. tidaldb-0 answered `reseed_required: false` while a different
   hosted group sat behind a compacted leader, and every operator reading and every
   diagnosis in this incident took that as converged. ShardStatusRow now carries
   both per group.

2. `lag_events: 0` WAS UNREADABLE. Lag is `leader_seqno_for(key) - applied`, an
   unsigned subtraction against a gauge that is 0 until this node learns the
   leader's frontier. A freshly-booted node that knows NOTHING computes 0 - 0 = 0
   and reports itself perfectly caught up. Measured in the multi-group repro at the
   moment the node declared itself settled:

     shard 0: applied_events 24, lag_events 0, leader us-east
     shard 1: applied_events 14, lag_events 0, leader eu-west
     shard 2: applied_events  0, lag_events 0, leader null

   5600 items had been written. All three groups claimed zero lag. Expose
   `leader_seqno` (the value lag subtracts from) on both the flat response and each
   shard row, so `lag_events: 0` with `leader_seqno: 0` reads as NO INFORMATION
   rather than converged. This is additive: `lag_events` keeps its value and
   readiness keeps its semantics, deliberately, because changing the readiness
   predicate during a live incident is not a change worth bundling here.

Also tightens the multi-group repro's settle predicate to require EVERY hosted
group's row to be clean. The first version trusted the flat fields, so it announced
"settled after 0 restarts" and then failed the content probe — fooled by exactly
the under-reporting above.
 2026-08-21 03:01:58 -06:00
jordan
fab5467b8f test(cluster): reproduce the multi-group reseed silent hole 
The served-evidence marker fix (afdda7c) closes the SINGLE-group case, proven by
mp_follower_reseeds_via_snapshot_after_compaction passing with its content probe.
It does not close the multi-group case, and nothing in the suite covered that: the
one reseed gate was single-group, and the harness leaves reseed_self_restart at
false, so a per-group self-restart that never reaches a fixpoint was invisible.

New mp_multi_group_node_converges_after_reseeding_several_groups reproduces the
production shape from k8s/cluster/topology-configmap.yaml: 3 nodes x 3 groups,
full placement, production election timers, reseed_self_restart TRUE. It stops one
node so its group leadership moves and a survivor ends up leading two groups (the
live tidaldb-1 arrangement), writes past WAL_RETENTION_SEGMENTS, gracefully
restarts the survivors to compact, then brings the node back.

The test also stands in for the ORCHESTRATOR. reseed_self_restart drains and
exits(0) expecting a reboot; the harness has no supervisor and `is_alive` only
checks that the handle is retained, so an exited node just stays down. Sustained
HTTP unreachability is the exit signal and `restart` is the reboot, counted
against a finite ceiling. The content probe stays supervised too, because the
first run settled, then re-latched and exited, and an unsupervised probe merely
panicked on a connection error and hid it.

Observed failure, the local twin of the production incident:

  [multi] node 2 settled after 0 orchestrator restart(s)
  [multi] node 2 exited AFTER settling; orchestrator reboot #1
  missing item 500 (reboots=1) ... reseed_required: false, lag_events: 0,
                                   applied_events: 3798, election_tail_term: 2

The node reports no marker and zero lag while an item written before its outage is
absent. That is the same silent hole tidaldb-0 showed at lag_events: 0.

Marked #[ignore] with the reason and the invocation, so the nightly chaos gate
keeps its signal instead of going permanently red on a known-open defect. Removing
the attribute is the gate for the fix.

Also adds write_heavy_item_retrying: which survivor inherits a stopped node's
groups varies per run, so a write may be local for one group and a cross-group
forward for another, and a forward inside an election window legitimately answers
a retryable 503. Retrying keeps the fixture deterministic without masking a hard
failure.
 2026-08-21 02:15:35 -06:00
jordan
afdda7cc0f fix(cluster): discharge a reseed marker on served evidence, never on a frontier 
da736b8 replaced `applied >= leader_last_seq` with `applied >= marker.from_seqno`
and was still wrong, for the same underlying reason: the applied frontier is a
HIGH-WATER-MARK, not a contiguity proof. A term join re-bases it onto the new
leader's stream (`replication_state().advance(.., baseline + 1)`), so it leaps
across history the node never received. Any predicate built on it discharges
markers for nodes that still have a hole.

Measured, not argued. `mp_follower_reseeds_via_snapshot_after_compaction` stops a
follower at frontier 9, compacts the leader so it retains only from 15722, and
the follower's frontier is re-based to 16810. Both predicates discharge the
marker there; the node skips its reseed and then reports `lag_events: 0` while
missing 10..15721 and serving reads from a log with a hole. Production showed the
identical shape: `applied 13540660` against a marker resuming at 13540653 that no
live WAL could serve.

The marker is now discharged only on POSITIVE EVIDENCE that the stream served the
latching range: a `StreamSegments` pull that began at or below the marker's
`from_seqno` and ran to completion. New `CatchupServedSink` in tidal-net fires on
`PullOutcome::Complete`; `NodeCatchupServedSink` routes it to
`discharge_reseed_marker_if_served`. `ReseedMarker::discharged_by_served_range`
replaces `discharged_by`. The other sound discharge is unchanged: a snapshot
install replaces the data dir and takes the marker with it.

Both frontier-based call sites are gone, with the reasoning recorded where they
were. The election-won site is deliberately NOT replaced: winning proves the log
beats a quorum's under the vote restriction, which is not contiguity, so
discharging there could promote a leader with a hole.

The owner-test for this mechanism was RED ON BASELINE and is now green. It also
gained the premise assertion it never had: it used to assert only the consequence
(`reseed_required == true`), so when its fixture stopped forcing compaction it
failed 40s later looking like a follower bug. `assert_history_compacted_past` now
checks the leader actually dropped the follower's resume seq, and prints the
retained segment floors. Its content probe ("every probed offline item is
searchable on the reseeded follower") is what proves the hole is really gone.

Suite state: cluster_reseed's other tests pass individually.
mp_graceful_rolling_restart_under_load_no_reseed remains red on baseline
(pre-existing, verified by stash). mp_quarantined_node_reseeds_without_wipe and
mp_election_position_consistent_across_roles_after_failover pass alone but can
fail in-suite: this fix makes the compaction test run its full 565s reseed
instead of failing fast at 40s, which shifts timing for later tests on shared
fixed ports. Order sensitivity is pre-existing, not introduced here.
 2026-08-21 00:40:06 -06:00
jx12n
580142df49 feat(m12): election-divergence-fix + soak-eval streak + release tooling 
Durable `leader_acked` frontier in `ShardReplica` tracks the highest seqno
acked under `ack=leader` (journal-only, un-replicated); `decide_join` now
quarantines on THIS node's own frontier rather than comparing stream numbers
across stream boundaries — eliminates false-quarantine churn on rolling
restarts. `SHUTDOWN_HANDOFF_WAIT` (3s) drains the leader's tail to quorum
before step-down so the next leader inherits a clean prefix. New
`load_leader_acked`/`persist_leader_acked` helpers; `cluster_reseed.rs` gains
the divergence-fix regression suite; `replication_ops.rs` threads the signal.

Soak-eval: `tidal_stress::soak_eval` + `soak-eval` binary implement the
30-night streak (ledger.tsv × restarts.tsv → streak.tsv); monitor and nightly
CronJob k8s YAMLs updated; phase-9 doc clarifies the dual-stream streak
definition (ledger PASS AND zero pod restarts in window). `run-reliability.sh`
gates the election-divergence suite before any k8s push.

Release tooling: `docker/release/` multi-stage Dockerfile + DR image;
`scripts/build-release.sh` single repeatable cross-compile+buildx path.
 2026-06-18 13:08:53 -06:00
jx12n
a946c6128c fix(m12-rc13): read-SLA collapse + WAL_RETENTION_SEGMENTS 16 + tidalctl S3 DR 
Read-SLA fix (rc12→rc13 — cpu-cgroup starvation → multi-second p99 + churning
elections):
- offload.rs: add SEARCH_GATE semaphore (core_count+1 permits, 50ms shed to 429)
  so per-shard searches gate on CPU, not reactor threads; concurrent scatter_merge
  fan-out (join_all) replaces the serial blocking offload_region_read loop
- node.rs: scatter_merge → async; per-shard futures run via offload_search
  (each acquires one SEARCH_GATE permit, moves it into spawn_blocking so the
  permit is held for the search's full CPU lifetime)
- main.rs: explicit tokio runtime with worker_threads floored at 4, independent
  of the cgroup quota — keeps the control plane (heartbeat/election/apply) on its
  own workers even when quota < 4
- k8s statefulset: CPU limit 2→3 (was: available_parallelism()=2 → only 2 async
  workers; search burst starved the reactor)
- tidal/wal/compaction.rs: WAL_RETENTION_SEGMENTS 4→16 (64 MiB→256 MiB per-shard
  catch-up window; a briefly-down follower across a rolling restart streams up
  instead of forcing snapshot reseed; disk floor 768 MiB/pod, self-trimming)
- cluster_reseed.rs: OFFLINE_ITEMS 1800→5600 to exceed the new 16-segment
  retention window (19 segs > 17); fix sequential quarantine/reseed race via
  await_status_bool

tidalctl S3/R2 backup DR:
- tidalctl/Cargo.toml: aws-config, aws-sdk-s3, aws-credential-types, tokio, tempfile
- commands/s3.rs: S3Target + export_dir (upload every file, manifest last as
  atomicity marker) + import_to_dir (download prefix into temp staging dir)
- commands/backup.rs: run_backup/run_restore accept Option<&S3Target>; S3 export
  is additive after local fsync barrier; S3 import stages into TempDir then runs
  the unchanged verified restore on it
- main.rs: --s3-endpoint / --s3-bucket / --s3-prefix flags; all-or-nothing
  endpoint+bucket validation; usage updated

tidal-stress/k8s: recall-rc12-spread-job, soak-nightly-cronjob, soak-monitor,
soak-results-pvc, t5-readtput-job manifests
 2026-06-17 15:47:37 -06:00
jx12n
727fbfcb6b fix(m12p6): 6-bug k3s 3-shard cluster repair (rc8+rc9) 
Root-caused and fixed five sharding bugs exposed on the real k3s 3-shard
cluster (rc5→rc7), plus a divergent-rejoin reseed loop found in rc9:

1. reseed shard-awareness (Bug 3, keystone): `run_boot_install_for_region`
   visits each hosted group's own shard subdir; per-group leader discovery
   appends `?shard=N` so a divergent shard heals from its own leader (not
   shard-0's WAL/term — cross-shard contamination).
2. leader self-join term (Bug 4): `become_leader_for_term` now calls
   `note_self_won_term` so the elected shard's `joined_term` is set and
   `cluster_promote` routes rebalances correctly (was: topology-era mis-read
   → legacy fenced promote → 500).
3. boot self-heal self-pull guard (Bug 2): `leader_shard != my_shard` gate
   prevents a node pulling its own stream (its stream isn't a registered peer)
   → eliminates the `PeerUnreachable(self)` loop.
4. scatter-merge degraded partial (Bug 1): failed shard logs + continues
   instead of `?`-failing the whole read; bounded read-admission semaphore
   (`offload.rs`) sheds as 429 instead of piling into a 36s p99.
5. WAL retention (Bug 5): `compact_wal_retained` keeps `WAL_RETENTION_SEGMENTS=4`
   most-recent sealed segments; online path gets the same retention clamp.
   Prevents brief-restart forced-reseed.
6. divergent-rejoin reseed loop (Bug 6, rc9): `note_quarantined` latches
   `from_seqno = stream_baseline` (not `frontier + 1`) so `wal_covers`
   returns `needed=true` and the snapshot installs instead of looping.

Also: `TidalDb::close_shared` for deterministic HNSW save on cluster SIGTERM
(HNSW graph was not saved when request-scoped Arc clones were alive at shutdown);
updated profiling doc with full rc8/rc9 fix narrative; k8s recall job YAMLs.
 2026-06-16 22:34:21 -06:00
jx12n
bf57be18e1 feat(m11): membership, snapshot install, and reseed (m11p5) 2026-06-12 19:55:54 -06:00