Two status defects turned this incident into a day of misreading. Both are
observability, both are why the functional bug survived, and neither changes
readiness or replication behaviour.
1. PER-GROUP RESEED STATE. `reseed_required` / `reseeding` existed only as flat
fields on LocalStatusResponse, and `status_local` fills those from
`replica_for(sel.shard_id())` — the LOWEST hosted group id when no `?shard=` is
given. On a 3-group node they therefore describe one group and say nothing about
the other two. tidaldb-0 answered `reseed_required: false` while a different
hosted group sat behind a compacted leader, and every operator reading and every
diagnosis in this incident took that as converged. ShardStatusRow now carries
both per group.
2. `lag_events: 0` WAS UNREADABLE. Lag is `leader_seqno_for(key) - applied`, an
unsigned subtraction against a gauge that is 0 until this node learns the
leader's frontier. A freshly-booted node that knows NOTHING computes 0 - 0 = 0
and reports itself perfectly caught up. Measured in the multi-group repro at the
moment the node declared itself settled:
shard 0: applied_events 24, lag_events 0, leader us-east
shard 1: applied_events 14, lag_events 0, leader eu-west
shard 2: applied_events 0, lag_events 0, leader null
5600 items had been written. All three groups claimed zero lag. Expose
`leader_seqno` (the value lag subtracts from) on both the flat response and each
shard row, so `lag_events: 0` with `leader_seqno: 0` reads as NO INFORMATION
rather than converged. This is additive: `lag_events` keeps its value and
readiness keeps its semantics, deliberately, because changing the readiness
predicate during a live incident is not a change worth bundling here.
Also tightens the multi-group repro's settle predicate to require EVERY hosted
group's row to be clean. The first version trusted the flat fields, so it announced
"settled after 0 restarts" and then failed the content probe — fooled by exactly
the under-reporting above.
54d1353103