[benchmarks] Log the run id and Datadog trace for each sequential-steps run - #3248
[benchmarks] Log the run id and Datadog trace for each sequential-steps run#3248shalabhc wants to merge 4 commits into
Conversation
🦋 Changeset detectedLatest commit: 6896fb3 The changes in this PR will be included in the next version bump. This PR includes changesets to release 0 packagesWhen changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results✅ All tests passed E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
✅ vercel-multi-region
|
📊 Workflow Benchmarkscommit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 385272ms → this run 397249ms (Δ +11977ms, +3%) 1020 steps (queue-hop) Cumulative STSO time: main 7646ms → this run 9252ms (Δ +1606ms, +21%) 📜 Previous results (4)08567fdFri, 31 Jul 2026 23:21:18 GMT · run logs
75e3abeFri, 31 Jul 2026 18:10:29 GMT · run logs
c784f6cFri, 31 Jul 2026 16:59:19 GMT · run logs
fedfc7aFri, 31 Jul 2026 01:20:43 GMT · run logs
ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
TooTallNate
left a comment
TooTallNate
left a comment
There was a problem hiding this comment.
Reviewed the capture, the threading, and the renderer. Approving — this closes a real gap in the histogram section (I've done the deployment-id + time-window hunt this replaces), and the degradation matrix is genuinely complete: every optional field falls back the way the table claims, and the three new renderer tests cover the full/degraded/omitted cases (40/40 pass locally).
One correction worth making before merge — the route comment overclaims what the trace contains. route.ts says the trigger request's span "propagates into the workflow's own spans". That's only true under WORKFLOW_TRACE_MODE=continuous. Nothing in the workbench or benchmarks.yml sets the mode, so the benchmark deployment runs the default linked mode — where each workflow/step invocation is its own trace root, and the trigger request's trace contains only the workflow.start span plus span links to the execution roots. The link is still useful (one hop via span links, and Datadog renders those), but:
- the comment should say what's actually there, so nobody debugging a weird histogram concludes the run "has no spans" when they open a trigger-only trace; and
- consider making the link a Datadog APM search on
workflow.run.id:<runId>instead of (or alongside) the trace permalink — under linked mode that lands on the run's actual execution spans directly, which is where an STSO investigation actually goes. The run id is already in hand.
Fine to take the comment fix now and the search-link idea as a follow-up — the current link is strictly better than nothing either way.
Verified:
- The capture is correctly optional end-to-end: optional-chained
getActiveSpan()(undefined without OTEL → bare run id), type-guarded on the runner side (typeof data.traceId === 'string'), andsequentialRunsomitted rather than[]when the scenario didn't run — so old artifacts, old deployments, and history re-renders all degrade exactly per the table. - Recording all iterations' run refs matches the data: the histograms aggregate samples across iterations, so with
BENCH_SEQUENTIAL_ITERATIONS > 1the comma-joined list is the honest answer. The doc phrase "both histograms come out of the same iteration" is only true for the default single-iteration config — worth a "(iterations, when configured)" tweak since the code already handles N correctly. - Baseline matching reuses the same methodology/backend/app key the percentile baselines use (
resultKeyFor= the metric key minus the row), so the two sides of the diff can't come from mismatched configurations. - The empty changeset is correct — e2e runner, CI scripts, and a workbench route; nothing published changes.
One robustness nit, non-blocking: runId/traceId are spliced into the markdown link unvalidated. The artifact-trust model is consistent with the rest of this renderer, so it's not a security gap — but a cheap shape check (/^[a-z0-9_]+$/i) before rendering the link would keep a malformed value from a buggy deployment from breaking the whole comment's markdown, which is this section's own failure story.
CI: the 2 failures are the nextjs-webpack canary HMR rebuild-count test and its required-check aggregator — the long-running baseline flake, unreachable from a benchmark-comment change.
shalabhc
changed the title
The route comment claimed the trigger request's span "propagates into the workflow's own spans". That only holds under WORKFLOW_TRACE_MODE=continuous. Nothing in the workbench or benchmarks.yml sets the mode, so the benchmark deployment runs the default `linked` (packages/core/src/telemetry.ts), where each workflow/step invocation is its own trace root and the trigger's trace carries `workflow.start` plus span links out to those roots. The logged link is still the right entry point -- one hop through the links, which Datadog renders -- but the comment should describe that, so nobody opening a trigger-only trace while debugging a histogram concludes the run produced no spans. Raised by @TooTallNate in review of #3248. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>
The STSO distribution section added in #3213 shows the shape of the sequential-steps run but not which run produced it, so investigating an odd-looking bucket meant hunting for the run by deployment id and time window. Capture the identity alongside the samples (the mechanism prototyped on the WIP variance branch, #3107): `/api/bench` returns the trace id of the span @vercel/otel opened for the trigger request, the runner threads it through the sequential iteration and records `sequentialRuns` in the result file, and the renderer prints one line under the histograms with the run id + Datadog trace link for this run and for the `main` run it is diffed against. Every part is optional — a deployment predating the route change yields a bare run id, and a `main` baseline predating this yields only this run's side — so the section degrades instead of breaking on mixed-vintage artifacts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
The run id and Datadog trace are debugging aids, not part of the benchmark's reported result, so they belong in the job's own output rather than in the PR comment body. Logging them where the runs are produced also makes them available in two cases the comment could never cover: a local `pnpm bench`, and a job that fails before the comment step runs. This drops the comment-rendering side entirely -- `renderSequentialRunLinks`, the `baselineSequentialRuns` baseline plumbing in `annotateWithBaseline`, and the `sequentialRuns` field on the result artifact, which existed only to carry the data to the renderer. The `main`-baseline side of the link goes away with it: which run produced the baseline histogram is only knowable at comment-render time, where the two artifacts are matched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
The route comment claimed the trigger request's span "propagates into the workflow's own spans". That only holds under WORKFLOW_TRACE_MODE=continuous. Nothing in the workbench or benchmarks.yml sets the mode, so the benchmark deployment runs the default `linked` (packages/core/src/telemetry.ts), where each workflow/step invocation is its own trace root and the trigger's trace carries `workflow.start` plus span links out to those roots. The logged link is still the right entry point -- one hop through the links, which Datadog renders -- but the comment should describe that, so nobody opening a trigger-only trace while debugging a histogram concludes the run produced no spans. Raised by @TooTallNate in review of #3248. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
Under the default linked trace mode the trigger's trace holds only `workflow.start` plus span links, so opening it lands one hop away from the spans an STSO investigation needs. Log an APM search on `@workflow.run.id:<runId>` next to it, which goes straight to the run's execution spans. Both links are logged rather than one replacing the other: the search depends on `workflow.run.id` being an indexed span tag in the org, and the permalink works regardless. Suggested by @TooTallNate in review of #3248. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
2 participants
Logs the run id and Datadog APM links for each sequential-steps benchmark iteration, in the benchmark job's own output:
Follow-up to #3213 (which added the STSO histograms). Uses the run-id/trace capture prototyped on the WIP variance branch, #3107.
Why
The
📈 STSO distributionsection shows the shape of the sequential-steps run but not which run produced it. When a bucket looks wrong, the next step is always the same manual hunt: find the run by deployment id + time window, then find its trace. This puts both in the log, next to the run that produced them.These are debugging aids rather than part of the benchmark's reported result, so they go in the job output rather than the PR comment body. That also covers two cases a comment can't: a local
pnpm bench, and a job that fails before the comment step runs.What
workbench/nextjs-turbopack/app/api/bench/route.ts— returnstraceIdfrom the span@vercel/otelopened for the trigger request.@opentelemetry/apiwas already a dependency; no lockfile change.packages/core/e2e/benchmark.test.ts— threadstraceIdthroughtriggerBenchRun→runSequentialIteration, and logs one line per iteration as soon as the scenario completes.A deployment built before the route reported a trace id logs the run id and span search, with no trace permalink.
Why two links
This deployment runs the default
WORKFLOW_TRACE_MODE=linked— nothing in the workbench orbenchmarks.ymlsets the mode. Underlinked, each workflow/step invocation is its own trace root, and the trigger's trace holds onlyworkflow.startplus span links out to those roots. So the trace permalink is an entry point to the run (one hop through the links, which Datadog renders), not a single trace containing every step's spans.The
@workflow.run.id:<runId>span search lands directly on the run's execution spans instead, which is where an STSO investigation actually goes. It depends onworkflow.run.idbeing an indexed span tag in the org, so it's logged alongside the permalink rather than replacing it — still to be confirmed against a real run.Notes
The comment renderer is untouched, and the result artifact gains no new fields — the run identities are logged, not serialized.
One thing this deliberately drops relative to putting the line in the comment: there's no link to the
mainbaseline's run. Which run produced the baseline histogram is only knowable at comment-render time, where the two artifacts are matched up; the benchmark job only knows about its own run.Empty changeset — no package behavior changes.
🤖 Generated with Claude Code