jordan/tidaldb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
138 Commits 1 Branch 0 Tags 14 MiB
0f18243a64
Commit Graph

6 Commits

Author SHA1 Message Date
jordan
a588f01f63 ranking: fix two BLOCKERs in the age-aware sorts, and stop trusting created_at units
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Details 
Three parallel reviews of 6385425 found two BLOCKERs I introduced, one CRITICAL,
and a CHANGELOG that named profiles that do not exist. All verified before fixing.

BLOCKER 1 -- an undated row ranked #1 instead of last. `Sort::New` mapped a
missing age to 0.0, but every DATED candidate scores `-age_hours`, i.e. <= 0.0.
So 0.0 was the MAXIMUM of the scale, not a neutral value: a row with no readable
`created_at` outranked the genuinely newest item and `normalize` reported its
score as 1.0.

The codebase had already ruled on this and gone the other way. `score_shortest`,
`score_longest` and `score_date_saved` all use the NEG_INFINITY sentinel, and
`helpers.rs` carries a regression test named for the exact anomaly --
`normalize_neg_inf_sentinel_on_negated_scale_folds_to_bottom` -- because
`Shortest` hit it first: "a pre-clamped 0.0 floated above them and the
last-ranked (missing) item reported the highest score 1.0". I reintroduced it on
a new negated scale.

Three reachable data classes, none hypothetical: a legacy row written before
`created_at` was materialized (`state_rebuild.rs` documents the class), an empty
map from `deserialize_metadata` on a short or corrupt row, and an entity dropped
by the `.ok().flatten()` in the metadata map builder -- which means a TRANSIENT
STORAGE READ ERROR could promote an item to the top of the feed.

Now the two `None` causes are distinguished. Map absent entirely -> 0.0, the whole
set ties, unchanged. Map present but this entity has no usable `created_at` ->
NEG_INFINITY, sorts last. A future-dated timestamp still clamps to age 0, because
that is a real value that legitimately means "newest"; an absent one carries no
recency claim at all.

Same conflation in `score_hot`, and worse there because `Sort::Hot` now reports
`needs_item_metadata`, so the map is loaded on essentially every Hot query and the
per-entity branch is the LIVE one. `unwrap_or(DEFAULT_HOT_AGE_HOURS)` handed a
single undated row the freshest divisor in the set: against a year-old cohort at
the builtin gravity 1.8 the ratio `(8762/26)^1.8` is ~3e4, so a corrupt row with
two views outranked correctly-dated items with tens of thousands. Across all four
Hot builtins. Now scored at Hot's floor.

BLOCKER 2 -- my own load-shedding "fix" INVERTED the survivor set. `truncate_to_
newest` ranked candidates against the newest ids of the WHOLE universe, but
`scan_candidates` iterates the universe bitmap ASCENDING and breaks at
`(limit * multiplier).max(200)`, so `candidates` is the LOW-id prefix. On a
catalog whose ids are assigned in creation order -- which is exactly what
`metadata_with_created_at`'s `Timestamp::now()` default produces -- the global
newest are the HIGH ids. Above roughly `max_candidates + 4*cap` items the two sets
stop intersecting, every candidate tied at `usize::MAX`, the stable sort became a
NO-OP, and `truncate(cap)` kept the OLDEST candidates. The `select_nth_unstable_by`
I replaced kept the newest available. Strictly worse than the bug I set out to fix,
and only visible when degraded.

The key is now composite -- recency rank, then DESCENDING id for anything the
oversample did not cover -- so where index and candidate window overlap the
survivors are genuinely newest, and where they do not it degrades to the documented
pre-existing approximation instead of inverting. Back to `select_nth_unstable_by_
key`: this path runs ONLY when the load shedder has already decided the node cannot
afford the work, so it must stay a linear partition, and only membership of
`[0, cap)` matters because Stage 3 re-orders the survivors anyway.

Mutation-proven, and the numbers show why the old test was blind: rank-only key
returns `EntityId(100)` where the composite key returns `EntityId(200)`, and the
pre-existing 150-item test PASSES under that same mutation -- its fixture had the
oversample covering the whole universe AND the newest items at the low ids, the one
configuration where the defect cannot appear. New test uses a 3000-item
time-ordered catalog; limit=50 does NOT exercise it (max_candidates == cap == 200
so the guard skips), limit=25 does.

Three of my ordering fixtures pinned nothing. `finalize`'s tie-break 2 is ASCENDING
entity id, and I had made newest-first coincide with it, so any mutation that
merely TIED the set still produced the asserted vector. The gate mutation proves
it: reverting `needs_metadata_for_sort` to its old four-variant form yielded
`left: [1, 2, 3, 4, 5]` -- literally the old fixture's expectation. No test in the
suite failed if the retrieve executor stopped consulting `Sort::needs_item_metadata`,
which is the exact drift this work exists to prevent. Every ordering fixture is now
non-monotonic in id, so it disagrees with BOTH descending id (the old proxy) and
ascending id (the tie collapse).

CHANGELOG named profiles that do not exist. `recent_uploads` -- zero hits
repo-wide; I invented it. `following` is Sort::New, not Hot. `brief` is Hot, not
New. `related` (Hot{1.2}) and `chronological` (New) were omitted entirely. The real
blast radius is SEVEN profiles, not the four-plus-three I claimed, and five of them
carry a diversity block so they already loaded item metadata and change behaviour
the instant this image rolls. Replaced with a verified table. Also cited the spec
conformance this brings: docs/specs/09-ranking-scoring.md:1214 already specified
`Sort::New` as "created_at DESC".

created_at units are now trusted with a warning instead of silently. A
seconds-unit value parses as u64 and was stored verbatim; `read_age_hours` divides
by nanos-per-hour, so `1700000000` reads as 56 years old. `Sort::New` then scores
-496731 against boost sums in single digits -- re-entering through DATA the exact
"recency annihilates every boost" defect this work removed -- and Hot buries the
item by a factor of 5.1e7. Pre-change both sorts ignored the value, so it was
inert; this work made it live. The repo's OWN fixtures made that mistake in three
places, which is the proof it is the natural one.

`metadata_with_created_at` now warns when the value is too small to be nanoseconds,
logging entity id, value, age_hours and age_years so an operator can act. It does
NOT rewrite the value -- guessing the unit would corrupt what the `created_at`
range index already reads as nanoseconds -- and does NOT reject the write, which
would break an API that currently accepts it. Threshold 6e17 ns (1989): nanosecond
timestamps after 1990 exceed 6.31e17 while seconds/millis/micros for any plausible
date stay under 1e16, so the ranges cannot overlap. VERIFIED on a live server:
seconds, millis and micros each warn with age_years=56; a real nanosecond value is
silent. The nanosecond contract is now documented on the public DTO and propagates
to the OpenAPI schema, where it was invisible before.

Stale docs corrected: two published blog posts and two claim-verification ledgers
were certifying "entity recency (higher ID = newer)"; the e2e fixture contract
justified an interleaving decision with reasoning that is now false (its conclusion
still holds, for a different reason); and k8s/discover/schema.yaml's "NO sort:"
rationale cited behaviour this work removed, so it is now marked PENDING
RE-MEASUREMENT with the three specific measurements named rather than left reading
as justified.

Full lib suite 2133 passed. Clippy 66 vs 66 at baseline, zero added, zero errors.
Fast integration suites all green. Real-server e2e re-verified: `new` and `hot`
both return newest-first, `new`'s scores now evenly spaced across evenly spaced
ages.
 2026-08-31 21:31:31 -06:00
jordan
77f68d181c verify: flip the two log tripwires post-roll, calibrate the backup assertions
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Details 
Post-deploy half of the deploy-verification contract for m12-harden-20260831.

Flipped, exactly as each assertion instructed its own successor to do:
- 06-logs.spec.ts: asserted `jsonLines === 0`. JSON_LOGS is live, so it now
  asserts every sampled line parses as JSON. The ANSI check stays pinned at 0.
- 09-operator-authority.spec.ts: asserted JSON_LOGS was absent from the
  StatefulSet. Now asserts JSON_LOGS=1 AND TIDAL_SERVICE_NAME=tidaldb, because
  the second is load-bearing: enabling structured logs makes the app's own
  `service` field win in the fleet's Vector normalize transform, silently
  renaming the log stream tidaldb -> tidal-server and blinding every query keyed
  on it. The fleet's _stream_fields contract pins field names but no legal
  values, so nothing there would have caught the flip.

Calibrated, NOT loosened — the two backup assertions were unpassable by
construction for ~25 minutes every day:
- The schedule fires at 03:30 and measured runs take 9.1-24.8 min (n=15), so the
  newest object is legitimately InProgress during its own window. The "newest
  backup completed cleanly" test now selects the newest FINISHED backup; a
  namespace where nothing has ever finished still fails.
- "no backup stuck in progress" asserted InProgress -> fail, full stop. It now
  bounds in-flight age at 60 min: ~2.4x the slowest success and a quarter of the
  240.0 min timeout that the observed PartiallyFailed runs (2026-08-17/19/25) all
  hit. A gate that cries wolf on a schedule gets muted, and then it is not a gate.

Both thresholds come from reading every backup in the namespace, not from a
guess. Playwright 34/34 and hermetic semantics 5/5 against the deployed image.
 2026-08-30 22:06:20 -06:00
jordan
fe8d0c87e7 harden: restore CI verification, remove four wire-level fabrications, instrument the 401 path 
Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement).

Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace
default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release
gate past its budget. Both push-path steps now declare
backend_options.kubernetes.resources as two YAML anchors declared once on their
first consuming step. The values are CALIBRATED against measured free node
capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's
original figure) fits on NO node and would sit Pending forever, because
`ci-build-bounds` grants permission and the nodes supply capacity, and those are
not the same thing.

The `nightly` cron described in this file for 216 days was never created, so
tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero
signal while reading like standing coverage. nightly-chaos and
nightly-security-ops now alias the anchors and have budgets matching the gate
(their 120/90 were TIGHTER on the same runner, so they would have failed
nightly for a budget reason, not a correctness one). nightly-soak is REMOVED,
not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best
node has 1700m free CPU, so it would fail on starvation rather than regression —
manufacturing a nightly false alarm. Its commands move verbatim to
docs/runbooks/nightly-soak.md.

Ring 1 — four fabrications removed from the wire.
- scatter_merge sorted and truncated without re-stamping rank, so /feed and
  /search returned 1,1,2 under full placement. Reuses merge_cross_shard's
  existing stamp; asserted on BOTH the multi-group merge path and the
  single-group [only] fast path that bypasses it.
- aggregate_region_row's None arm invented `applied_events: 0` plus a deficit
  derived from it. applied_events/lag_events are now Option<u64>, null on the
  wire. leader_last_seq was also unwrap_or(0), so a node that could not reach
  the LEADER computed 0 - applied = 0 for every region and reported a converged
  cluster it had never measured — a fabrication pointing the dangerous way.
- tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was
  actively hiding the PVC-wipe shape: a measured zero with a real deficit
  rendered as "no report" instead of BEHIND. Now read off the wire; converged
  exits 0, partitioned still exits nonzero.
- /sharded/* answered 201/204 for single-copy writes with nothing anywhere
  saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the
  existing invalid_input path. Six call sites migrated, not the two this
  roadmap predicted — including docs/runbooks/cluster.md §16.3, which told
  operators to run a quorum-write probe via POST /sharded/items. That probe
  cannot verify quorum: the surface applies locally with no WAL append. It was
  used as the safety check between every step of a staged deploy earlier today.

Ring 2 — observability. JSON_LOGS was already implemented and the deployment
simply never asked for it; the StatefulSet now sets it, plus
TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the
VictoriaLogs `service` stream field and would have blinded every query keyed on
it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the
origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only
the origin would mean each vector lands on exactly one node, replicas never
agree, and the alert built on it pages forever.

Found by measurement, not planned: the 401 path discarded every fact about
every rejection. Traefik has served 101,858 rejected requests to the public
ingress — 87.6% of all its traffic — with no record of who or why anywhere.
unauthorized_response now emits reason (missing_token vs invalid_token, the
distinction that separates a scanner from a rotation that missed a consumer)
and the forwarded client. The token is never logged.

Also: scripts/restore-fleet.sh --cluster started the soak monitor while
deliberately leaving its gate suspended, orphaning a watcher that has reported
"0/30 green nights" for 13 days. The pair now moves together. Doc-guard's
three-warning backlog is cleared with real backfill for M4/M6/M12.

Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline,
counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5;
cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with
the two remaining failures asserting the rank fix against the not-yet-rolled
image — they are the post-deploy proof.
 2026-08-30 20:55:58 -06:00
jordan
71e80ef655 e2e: get the Playwright harness green end to end, and close the stale-evidence gap 
The suite had not been run since 2026-08-23 and deps were not installed. Running it
against the freshly rolled m12-vsc-20260830 found four failures. Every one was the
harness doing its job; three were stale pins it explicitly told me to invert.

REAL FINDING, caught by the suite and nothing else: tidaldb-2 was NotReady mid-run.
It had exited(0) with {"reason":"reseed_self_restart","shard":1}, reinstalled a
snapshot and converged. Designed behavior - but the suite sampled readiness ONCE and
reported a self-healing cluster as broken. Readiness is now polled via
waitForPodsReady with a bounded budget and the whole timeline attached as evidence.
Deliberately not Playwright retries: retries:0 is correct here, because a live check
that only passes on attempt two has told you something true.

STALE PINS INVERTED (each verified live first, not taken on the message's word):
  - 06-logs: ANSI escapes are gone (0 in a 5-line sample), BUG-006 resolved on this
    image. Now pinned so a regression to coloured output fails.
  - 09-operator-authority + CAP-015 capture: tidaldb_http_* exists (185 series
    against a 552 baseline). Runbook 9.1 moved from inert to LIVE. CAP-015 keeps its
    purpose - state the gaps - and now names the one that is still real: no JSON_LOGS.
  - The transient /search 500 and public 502 were tidaldb-2's restart window, not
    defects; both surfaces returned 200 on eight retries afterwards.

THRESHOLD CALIBRATED AGAINST A MEASUREMENT, TWICE. My first fix capped
consecutive ship failures at 500, guessing a restart burst was ~100. Measurement
killed it: a reseed restart is a ~2 minute absence, which at the shipper's 100ms
cadence is ~1200-2000 failures - observed exactly 1950, then "peer recovered", with
peer_acked_seqno back at the frontier. A COUNT cannot separate "a peer restarted"
from "shipping is stuck"; it only encodes how long the peer was away. The test now
compares the newest distress line against the newest recovery line and fails only
when distress is newer. Same correction applied to the alert in k3s-fleet.

STALE EVIDENCE WAS THE WORST GAP. demo/public/captures and capture-manifest.json
still described m12-admin-gate-20260823 - two image rolls stale - while
demo:preflight reported "audited perfect" about week-old frames, and the rendered
title card read "image m12-admin-gate-20260823 - 32 checks green". The capture suite
writes to test-results/demo-captures/ and the copy-and-merge step into the published
set simply did not exist; it was done by hand once. Added demo/promote.ts: copies
frames, verifies each PNG against its fragment hash, and stamps buildRevision and
verifiedImage from the live StatefulSet. Verdicts land `pending`, so preflight fails
until the frames are audited - that failure is the gate. scenes.ts now derives the
image tag and check count from the manifest, and preflight fails if a literal is
pasted back in (proven by pasting one back in).

All 10 captures were opened individually at full resolution; the audit note is stored
in the manifest beside each verdict rather than only in prose.

Green: 34 e2e + 5 hermetic semantics + 10 captures + preflight + 2107 lib.
Video: demo/out/deploy-verification.mp4, 90.05s 1920x1080 h264, title card now
reading "image m12-vsc-20260830 - 34 checks green".

CLAUDE.md gains a Deploy Verification section and AGENTS.md a short mandatory
pointer: every deploy is verified through this harness, and maintaining it is part
of the change, not follow-up. The suite pins current reality including defects, so a
correct improvement WILL turn it red - and that is the harness working.
 2026-08-30 15:27:56 -06:00
jordan
9523f6da43 test(e2e): verify ranking semantics with a content-feed app, and route three product findings 
The existing 32 checks prove the deployment answers -- TLS, auth, quorum commit,
convergence, isolation, dashboards, backups. Not one wrote a signal and observed
an order change, so VISION.md:17 "Ranking is not a feature. It is a primitive."
was unverified. This adds a 60-item content-feed app and five assertions that
verify the product's semantics, on a hermetic standalone node.

Added
- tests/e2e/app/: fixture contract (60 items, 4 categories, each owning one
  unoccupied 100-id embedding cluster), a deep-module harness owning the whole
  lifecycle behind startApp(), the product page, and an app:dev entry point.
- tidal-stress/src/bin/feed-fixture.rs: seeds the catalog and emits brute-force
  ground truth, reusing recall::embedding_for rather than adding a third copy of
  the corpus generator (tidal/src/db/items.rs already holds a second).
- GroundTruth::from_ids: the oracle now serves sparse id sets. build() delegates,
  so there is no transient copy even at 1M, and top_k indexes positionally.
- 10-ranking-semantics.spec.ts (5 hermetic checks) and
  11-ranking-integrity.spec.ts (2 cluster tripwires).
- playwright.semantics.config.ts + CAP-016 demo beat (walkthrough 82s -> 90s).

Measured, not merely green
- like: index 59 -> 0, like_boost 2.0, with no sleep between write and read.
- decay: implied half-lives 7.0007 d and 14.0014 d against a schema declaring
  7 d and 14 d, recovered from a 4-second window via H = t*ln2 / -ln(v2/v1) and
  compared against the schema the node actually loaded, not a hardcoded copy.
- ANN: top-10 identical to brute-force cosine on all four probes; self-distance
  0.0148-0.0197 against a 0.05 tolerance.
- rank: dense 1..60 on standalone vs [1,1,1,2,2,3,4,3,4,5,6,5] on the cluster.

Three product findings, pinned and routed to @tidal-engineer
- BUG-018 (High) skip is durably accepted and query-time inert. Penalty is fully
  implemented (ranking/profile.rs:227 -> executor/signal_values.rs:183, labelled
  {signal}_penalty at executor/mod.rs:65) but skeleton() sets penalties: vec![]
  (ranking/builtins.rs:62) and none of the 27 built-ins overrides it. So
  VISION.md:187 "negative signals are equal citizens" holds for no shipped
  profile. Same anti-pattern as the reseed defects and scatter_merge: a guard
  present on one path, absent on its sibling.
- BUG-019 (Medium) three built-ins read signals this schema does not declare --
  trending/share_velocity, hidden_gems/completion, controversial/dislike -- so
  those terms are permanently 0 and trending ranks on view_velocity alone.
- BUG-020 (Low) for_you declares Scan{sort_field:"created_at"} but ignores a
  created_at metadata value; an order matching neither id-asc nor
  created_at-desc came back strictly id-ascending.

Two assertions therefore report a gap rather than a success, written as tripwires
whose failure message says what to do when the gap closes. The rank defect is
localised, not fixed: scatter_merge (cluster/node.rs:7542) returns a merged slice
without re-stamping rank while scores stay correctly ordered, so the fault is the
missing stamp and not the merge's sort.

Notes
- Hermetic by construction: its own config, because FullConfig.projects is not
  filtered by --project and globalSetup publishes credentials into the main
  process that forked workers inherit -- so a setup project cannot replace it,
  and weakening globalSetup would destroy the fail-loud behaviour that is its
  purpose. Verified with KUBECONFIG=/nonexistent and all E2E_* unset.
- Never touches the deployed corpus: skip is permanent: true, so seeding it into
  production would be irreversible.
- The page contains no sort, no hostname and no credential; the harness proxy
  injects auth server-side so no bearer reaches a browser or a capture.
- Schema comes from k8s/cluster/schema-configmap.yaml, asserted at 1536 dims;
  tidal-server/config/default-schema.yaml declares 128 and would 422 every write.

Verification: 5 semantics + 34 regression + 10 demo captures green; tsc clean;
tidal-stress clippy clean under clippy::all=deny with unwrap_used=deny; 2101
tidaldb lib tests; preflight 10/10 perfect; render 90.05s/2700 frames with zero
empty boundary frames; zero orphan processes or temp dirs after teardown.
 2026-08-23 22:42:02 -06:00
jordan
15f6b11187 test(e2e): Playwright evidence harness for the deploy-verification runbook 
Turns docs/runbooks/deploy-verification.md from prose into 32 executable checks
against the live orchard9-k3sf cluster, and it found real defects on its first
run — including in the runbook it verifies.

WHY PLAYWRIGHT, HONESTLY
tidalDB serves zero HTML (no text/html, no Html(), 10 JSON routes), so this uses
Playwright in three distinct roles rather than pretending there is a UI:
  * request fixture as a real HTTP client for DNS/TLS/auth/quorum/404;
  * a browser for the only genuine screens in the chain, Grafana;
  * a test harness for cluster-plane checks with no HTTP surface, shelling out
    to kubectl and attaching the real transcript as evidence.

WHAT IT CAUGHT
  * The runbook asserted the operator/data credential split was "not active yet
    - requires an image roll". globalSetup read the live image and the live
    secret; a probe returned data->403, admin->200. It had been enforcing the
    whole time. Section 9 rewritten. (BUG-001)
  * docs/ops/grafana-tidaldb.json shipped datasource uid ${DS_PROMETHEUS} - a
    Grafana export-for-sharing placeholder with no __inputs block to resolve it.
    Under ConfigMap provisioning every panel queried a datasource that did not
    exist, so the whole board was blank. The API said "loaded" and I had only
    ever checked the API. 41 refs fixed here, 58 across the fleet ConfigMap,
    which was also blanking the postgres and redis dashboards. (BUG-007)
  * Stat panels used calcs "lastNonNull". Grafana's reducer is "lastNotNull", so
    no value was ever computed and Cluster health / Reseed pending / Indexed
    vectors rendered as empty boxes. I chased panel width and then panel height
    before comparing against a working stat panel elsewhere in the same Grafana.
    A spelling error wearing a layout bug's clothes. (BUG-009)
  * The namespace variable defaulted to All, so cluster panels silently included
    tidaldb-586b544c8-vpkmw from the superseded standalone deployment. Latency
    legends read "p50 p50 p50" with no way to tell the nodes apart. Both fixed.
  * "5xx ratio" rendered "No data" as large green text - at a glance a healthy
    value. And Fleet state gave three fields one shared green threshold, so
    reseed_required=1 would have shown GREEN during the exact incident the panel
    exists to surface. Split into three panels with per-field mappings.
  * tidalctl cluster-status exits 2 on a FULLY CONVERGED cluster, because the
    aggregated endpoint reports healthy peers as region=null applied=0
    reachable=false. The runbook claimed `cluster-status && deploy` was a safe
    gate; that claim came from an exit code masked by a shell pipeline. The gate
    can never pass here. Documented, test pins it, engine defect recorded.
    (BUG-005)
  * The deployed image writes ANSI colour into container logs, which the
    collector stores verbatim. Already fixed in logging.rs, not yet rolled;
    pinned as a tripwire. (BUG-006)
  * The runbook's own backup command sorted ALL backups by timestamp and
    selected a restore-canary run: 20 items, one volume, a meaningless pass.
    Now filters on the schedule label the freshness alert actually watches.

DEFECTS FOUND BY LOOKING AT THE SCREENS
Six of the first eight captures were slop and were fixed, not promoted:
230-350px of dead space; a verdict that rendered "exit code 2" in green; the
1600x1800 dashboard scaled into 16:9 until illegible (now clipped to the
evidence band using real element bounds); the dream beat whose caption described
a contradiction the image did not show (now a purpose-built capture holding the
committed doc text, the running image, and the live 403/200 side by side); and a
one-frame blink to bare background at every scene boundary, because Remotion
Sequences do not overlap and both scenes sat at opacity 0 on the boundary frame.

TRIPWIRES IN THE HONEST DIRECTION
Three tests assert what is ABSENT - zero tidaldb_http_* families, JSON_LOGS
unset, plain-text logs - and each carries the message "good news, roll the
runbook section from pending to live". The metric-absence test also asserts the
baseline family count, so "absent" cannot pass for "the scrape failed". That is
the drift that made section 9 stale in the first place.

Regression config uses workers:1 and retries:0 deliberately: a live-cluster
check that only passes on the second attempt has told you something true.

Verified: 32 passed (46.8s); 9 demo captures each asserting before photographing;
tsc clean; render 82.05s 1920x1080 h264, 0 empty frames across 10 boundaries;
every promoted image inspected individually and judged perfect; walk-the-render
ledger complete with no fails.
 2026-08-23 14:03:29 -06:00