[ci] Benchmark comment: show avg-latency deltas vs main#2842
Conversation
Pushes to main now run the benchmarks against the production deployment and upload their results as artifacts. PR comment rendering downloads the most recent main baseline and annotates each Avg cell with the percentage change (e.g. "1035 (+5.6%)"), with a footer note explaining the comparison. The baseline average is embedded in the comment's history data so collapsed history entries keep the delta they were originally compared against.
🦋 Changeset detectedLatest commit: 9d64dd6 The changes in this PR will be included in the next version bump. This PR includes changesets to release 0 packagesWhen changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
📊 Workflow Benchmarkscommit Backend:
📜 Previous results (1)e9ebdc3Wed, 08 Jul 2026 23:26:06 GMT · run logs
Metrics — TTFS: time to first step body execution · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (time outside step bodies, client start → last step body exit) · SL: stream latency (first chunk write → visible to the reader) Scenarios — stream: one step that streams chunks back to the client; no hooks, so the run stays in turbo mode · hook + stream: registers a hook before the same streaming step, which exits turbo mode · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges 🟢/🔴 mark percentiles within/above target. Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120 TTFS/WO compare client vs deployment clocks and SL compares the step runner’s clock vs the client’s (NTP-synced in CI). WO ends at the last step body exit, the closest observable proxy for the final step-completion request. |
🧪 E2E Test Results✅ All tests passed Summary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
|
| # successful run of this workflow on main. | ||
| - name: Download baseline results from main | ||
| continue-on-error: true | ||
| uses: dawidd6/action-download-artifact@bf251b5aa9c2f7eeb574a96ee720e24f801b7c11 # v6 |
There was a problem hiding this comment.
Flagged as a potential supply chain problem 🤔
Socket flagged dawidd6/action-download-artifact (supply-chain score 51). gh run list/download is first-party and lets us filter baseline runs to push/workflow_dispatch events, which fork PR runs can never produce — closing the artifact-poisoning hole where a fork branch named "main" could upload a crafted bench-results artifact.
|
No backport to This commit is a follow-up to the benchmark-comment overhaul (#2820) and edits To override, re-run the Backport to stable workflow manually via |
2 participants
Follow-up to #2820. The benchmark PR comment now shows how average latencies compare to
main.How it works
push: maintrigger: pushes tomainrun the benchmark suite against the production deployment, purely to producebench-results-*artifacts (the comment jobs are gated topull_requestand skip on push).bench-results-*artifacts from the most recent successful run of this workflow onmain(viadawidd6/action-download-artifact, same mechanism the pre-overhaul benchmarks used) and passes them to the renderer via--baseline-dir.1035 (+5.6%);(±0%)under 0.5%. Rows without a matching baseline row render unchanged. A footer note explains the comparison.Deltas apply only to averages, per the percentile 🟢/🔴 target marks already covering the tail latencies.
Notes