The served-evidence marker fix (afdda7c) closes the SINGLE-group case, proven by
mp_follower_reseeds_via_snapshot_after_compaction passing with its content probe.
It does not close the multi-group case, and nothing in the suite covered that: the
one reseed gate was single-group, and the harness leaves reseed_self_restart at
false, so a per-group self-restart that never reaches a fixpoint was invisible.
New mp_multi_group_node_converges_after_reseeding_several_groups reproduces the
production shape from k8s/cluster/topology-configmap.yaml: 3 nodes x 3 groups,
full placement, production election timers, reseed_self_restart TRUE. It stops one
node so its group leadership moves and a survivor ends up leading two groups (the
live tidaldb-1 arrangement), writes past WAL_RETENTION_SEGMENTS, gracefully
restarts the survivors to compact, then brings the node back.
The test also stands in for the ORCHESTRATOR. reseed_self_restart drains and
exits(0) expecting a reboot; the harness has no supervisor and `is_alive` only
checks that the handle is retained, so an exited node just stays down. Sustained
HTTP unreachability is the exit signal and `restart` is the reboot, counted
against a finite ceiling. The content probe stays supervised too, because the
first run settled, then re-latched and exited, and an unsupervised probe merely
panicked on a connection error and hid it.
Observed failure, the local twin of the production incident:
[multi] node 2 settled after 0 orchestrator restart(s)
[multi] node 2 exited AFTER settling; orchestrator reboot #1
missing item 500 (reboots=1) ... reseed_required: false, lag_events: 0,
applied_events: 3798, election_tail_term: 2
The node reports no marker and zero lag while an item written before its outage is
absent. That is the same silent hole tidaldb-0 showed at lag_events: 0.
Marked #[ignore] with the reason and the invocation, so the nightly chaos gate
keeps its signal instead of going permanently red on a known-open defect. Removing
the attribute is the gate for the fix.
Also adds write_heavy_item_retrying: which survivor inherits a stopped node's
groups varies per run, so a write may be local for one group and a cross-group
forward for another, and a forward inside an election window legitimately answers
a retryable 503. Retrying keeps the fixture deterministic without masking a hard
failure.
fab5467b8f