jordan 71e80ef655 e2e: get the Playwright harness green end to end, and close the stale-evidence gap

The suite had not been run since 2026-08-23 and deps were not installed. Running it
against the freshly rolled m12-vsc-20260830 found four failures. Every one was the
harness doing its job; three were stale pins it explicitly told me to invert.

REAL FINDING, caught by the suite and nothing else: tidaldb-2 was NotReady mid-run.
It had exited(0) with {"reason":"reseed_self_restart","shard":1}, reinstalled a
snapshot and converged. Designed behavior - but the suite sampled readiness ONCE and
reported a self-healing cluster as broken. Readiness is now polled via
waitForPodsReady with a bounded budget and the whole timeline attached as evidence.
Deliberately not Playwright retries: retries:0 is correct here, because a live check
that only passes on attempt two has told you something true.

STALE PINS INVERTED (each verified live first, not taken on the message's word):
  - 06-logs: ANSI escapes are gone (0 in a 5-line sample), BUG-006 resolved on this
    image. Now pinned so a regression to coloured output fails.
  - 09-operator-authority + CAP-015 capture: tidaldb_http_* exists (185 series
    against a 552 baseline). Runbook 9.1 moved from inert to LIVE. CAP-015 keeps its
    purpose - state the gaps - and now names the one that is still real: no JSON_LOGS.
  - The transient /search 500 and public 502 were tidaldb-2's restart window, not
    defects; both surfaces returned 200 on eight retries afterwards.

THRESHOLD CALIBRATED AGAINST A MEASUREMENT, TWICE. My first fix capped
consecutive ship failures at 500, guessing a restart burst was ~100. Measurement
killed it: a reseed restart is a ~2 minute absence, which at the shipper's 100ms
cadence is ~1200-2000 failures - observed exactly 1950, then "peer recovered", with
peer_acked_seqno back at the frontier. A COUNT cannot separate "a peer restarted"
from "shipping is stuck"; it only encodes how long the peer was away. The test now
compares the newest distress line against the newest recovery line and fails only
when distress is newer. Same correction applied to the alert in k3s-fleet.

STALE EVIDENCE WAS THE WORST GAP. demo/public/captures and capture-manifest.json
still described m12-admin-gate-20260823 - two image rolls stale - while
demo:preflight reported "audited perfect" about week-old frames, and the rendered
title card read "image m12-admin-gate-20260823 - 32 checks green". The capture suite
writes to test-results/demo-captures/ and the copy-and-merge step into the published
set simply did not exist; it was done by hand once. Added demo/promote.ts: copies
frames, verifies each PNG against its fragment hash, and stamps buildRevision and
verifiedImage from the live StatefulSet. Verdicts land `pending`, so preflight fails
until the frames are audited - that failure is the gate. scenes.ts now derives the
image tag and check count from the manifest, and preflight fails if a literal is
pasted back in (proven by pasting one back in).

All 10 captures were opened individually at full resolution; the audit note is stored
in the manifest beside each verdict rather than only in prose.

Green: 34 e2e + 5 hermetic semantics + 10 captures + preflight + 2107 lib.
Video: demo/out/deploy-verification.mp4, 90.05s 1920x1080 h264, title card now
reading "image m12-vsc-20260830 - 34 checks green".

CLAUDE.md gains a Deploy Verification section and AGENTS.md a short mandatory
pointer: every deploy is verified through this harness, and maintaining it is part
of the change, not follow-up. The suite pins current reality including defects, so a
correct improvement WILL turn it red - and that is the harness working.

2026-08-30 15:27:56 -06:00

7.2 KiB

Raw Blame History

AGENTS.md

Agent instructions for tidalDB.

Deploy Verification — mandatory

No deploy is done until npm run test:e2e is green against the live cluster. 34 checks, ~50s. Then npm run test:e2e:semantics (5, hermetic).

Video: npm run test:demo && npm run demo:promote && npm run demo:preflight && npm run demo:render. demo:promote is NOT optional — it re-stamps the manifest from the live StatefulSet. Skipping it renders stale frames that preflight will still call "audited perfect".

Maintaining the suite is part of the change, not follow-up. It pins current reality including defects, so a correct improvement WILL turn it red — that is the harness working. When a test says "good news — invert this assertion", invert it and fix the runbook section it names. Never add retries, never loosen a threshold to make red go away, and calibrate every threshold against a measurement rather than a guess.

Full contract: CLAUDE.md §Deploy Verification · detail: docs/runbooks/deploy-verification.md.

Team

Agent	Identity	Model	Invoke when
`@tidal-engineer`	Jon Gjengset — principal Rust database engineer	opus	Implementing features, storage internals, signal system, query engine, debugging correctness
`@tidal-visionary`	Spencer Kimball — product and roadmap strategist	opus	Planning milestones, scoping phases, build-vs-defer decisions, roadmap sequencing
`@tidal-researcher`	Andy Pavlo — database systems researcher	opus	Prior art surveys, library evaluation, architectural research, producing `docs/research/` docs
`@tidal-distributed`	Kyle Kingsbury — distributed-systems engineer	opus	Network transports, cluster coordination, multi-node deployment, cross-node query routing, HA
`@tidal-performance`	Martin Thompson — low-latency performance engineer	opus	Hot-path optimization, lock-free concurrency, latency measurement (coordinated-omission), throughput funnels, mechanical sympathy, scale validation
`@tidal-storyteller`	Marketing and technical writer	sonnet	Marketing site (`site/`), blog posts, public-facing copy

Agent definitions live in .claude/agents/. CLAUDE.md §Agents is the canonical roster (this table mirrors it — keep them in sync); see it for the @knowledge-librarian utility agent and the vendored @kai-park/@kaya-osei/@mira-vasquez Aeries team that backs the aeries-* skills.

SDLC

Required reading: .sdlc/guidance.md — engineering principles that govern all implementation decisions on this project.

This project uses sdlc as its SDLC state machine. sdlc manages feature lifecycle, artifacts, tasks, and milestones. It emits structured directives via sdlc next --json that any consumer (Claude Code, custom scripts, or humans) acts on to decide what to do next.

Consumer scaffolding is installed globally under ~/.claude/commands/, ~/.gemini/commands/, ~/.opencode/command/, and ~/.agents/skills/ — available across all projects. Use /sdlc-specialize in Claude Code to generate a project-specific AI team (agents + skills) tailored to this project's tech stack and roles.

Key Commands

sdlc feature create <slug> --title "..." — create a new feature
sdlc next --for <slug> --json — get the next action directive (JSON)
sdlc next — show all active features and their next actions
sdlc artifact approve <slug> <type> — approve an artifact to advance the phase
sdlc state — show project state
sdlc feature list — list all features and their phases
sdlc task list [<slug>] — list tasks for a feature (or all tasks)

Lifecycle

draft → specified → planned → ready → implementation → review → audit → qa → merge → released

Treat this lifecycle as the default pathway. You can use explicit manual transitions when needed, but approvals/artifacts are the recommended way to keep quality and traceability.

Artifact Types

spec design tasks qa_plan review audit qa_results

CRITICAL: Never edit .sdlc/ YAML directly

All state changes go through sdlc CLI commands. See §6 of .sdlc/guidance.md for the full command reference. Direct YAML edits corrupt state.

Directive Interface

Use sdlc next --for <slug> --json to get the next directive. The JSON output tells the consumer what to do next (action, message, output_path, is_heavy, gates).

Consumer Commands

/sdlc-next <slug> — execute one step, then stop (human controls cadence)
/sdlc-run <slug> — run autonomously to completion
/sdlc-status [<slug>] — show current state
/sdlc-plan — distribute a plan into milestones, features, and tasks
/sdlc-milestone-uat <milestone-slug> — run the acceptance test for a milestone
/sdlc-pressure-test <milestone-slug> — pressure-test a milestone against user perspectives
/sdlc-vision-adjustment [description] — align all docs, sdlc state, and code to a vision change
/sdlc-architecture-adjustment [description] — align all docs, code, and sdlc state to an architecture change
/sdlc-enterprise-readiness [--stage <stage>] — analyze production readiness
/sdlc-setup-quality-gates — set up pre-commit hooks and quality gates
/sdlc-cookbook <milestone-slug> — create developer-scenario cookbook recipes
/sdlc-cookbook-run <milestone-slug> — execute cookbook recipes and record results
/sdlc-ponder [slug] — open the ideation workspace for exploring and committing ideas
/sdlc-ponder-commit <slug> — crystallize a pondered idea into milestones and features
/sdlc-guideline <slug-or-problem> — build an evidence-backed guideline through five research perspectives and TOC-first distillation
/sdlc-suggest — analyze project state and suggest 3-5 ponder topics to explore next
/sdlc-beat [domain | feature:<slug> | --week] — step back with a senior leadership lens; evaluate if we're building the right thing in the right direction; stores history in .sdlc/beat.yaml
/sdlc-recruit <role> — recruit an expert thought partner as a persistent agent
/sdlc-empathy <subject> — deep user perspective interviews before decisions
/sdlc-spike <slug> — <need>; [see <ref>] — research, prototype, validate, and report; produces working prototype + findings in .sdlc/spikes/<slug>/findings.md
/sdlc-convo-mine [file or text] — mine conversation dumps for signal; apply 5 perspective lenses, group themes, launch parallel ponder sessions per group

Tool Suite

Project-scoped TypeScript tools in .sdlc/tools/ — callable by agents and humans during any lifecycle phase. Read .sdlc/tools/tools.md for the full help menu.

sdlc tool list — show installed tools
sdlc tool run <name> [args] — run a tool; pass --json '{...}' for complex input
sdlc tool sync — regenerate tools.md after adding a custom tool
sdlc tool scaffold <name> "desc" — create a new tool skeleton

Core tools: ama (codebase Q&A), quality-check (runs platform shell gates)

Use /sdlc-tool-run, /sdlc-tool-build, /sdlc-tool-audit, /sdlc-tool-uat in Claude Code for guided tool workflows.

Project: tidalDB

7.2 KiB Raw Blame History