[ci] Benchmark preflight + fail-fast for broken deployments#2846
Conversation
The first production benchmark runs (push: main baselines) failed with every stream read timing out at 120s, burning the full 60-minute job timeout without a useful error. Three changes make that mode cheap and diagnosable: - Preflight: a trivial 1-step run must complete within 3 minutes before any scenario runs; otherwise all scenarios abort immediately with a clear "deployment accepted the run but did not execute it" error. - Zero-success circuit breaker: a scenario aborts after 3 attempts with no successful iterations instead of draining its full retry budget. - Stream-read timeouts now probe and report the run state (completed / still not finished / failed), distinguishing "run never executed" from "read path broken".
🦋 Changeset detectedLatest commit: 190824d The changes in this PR will be included in the next version bump. This PR includes changesets to release 0 packagesWhen changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results✅ All tests passed Summary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
|
📊 Workflow Benchmarkscommit Backend:
Metrics — TTFS: time to first step body execution · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (time outside step bodies, client start → last step body exit) · SL: stream latency (first chunk write → visible to the reader) Scenarios — stream: one step that streams chunks back to the client; no hooks, so the run stays in turbo mode · hook + stream: registers a hook before the same streaming step, which exits turbo mode · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges 🟢/🔴 mark percentiles within/above target. Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120 TTFS/WO compare client vs deployment clocks and SL compares the step runner’s clock vs the client’s (NTP-synced in CI). WO ends at the last step body exit, the closest observable proxy for the final step-completion request. |
|
No backport to This commit only modifies To override, re-run the Backport to stable workflow manually via |
1 participant
The first production baseline runs from #2842's
push: maintrigger failed pathologically: every stream read timed out at the 120s guard, both stream scenarios ran out their 30-minute vitest timeouts, and the job died at its 60-minute limit with no useful error (runs 28981485638, 28983634194). Notably, the e2e Vercel Prod lane passed on main ~20 minutes earlier with the same client code, so this looks specific to how the benchmark runs behave against the production deployment — but the logs can't currently tell what is stuck.Three changes make this failure mode cheap and self-diagnosing:
BENCH_PREFLIGHT_TIMEOUT_MS). If it doesn't, all scenarios abort immediately with a clear "deployment accepted the run but did not execute it to completion" error. Worst-case job time for a broken target drops from 60+ minutes to ~5.Verified locally: preflight failure aborts the whole file in seconds with all scenarios skipped; typecheck and lint clean.