feat(world-vercel): synthesize per-event client spans on the WS transport - #3452
Conversation
🦋 Changeset detectedLatest commit: daa7259 The changes in this PR will be included in the next version bump. This PR includes changesets to release 17 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results✅ All tests passed E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ vercel-multi-region
✅ vercel-ws-transport
|
📊 Workflow Benchmarks⏳ Benchmarks are running for
commit Backend:
📜 Previous results (4)eea5ff1Wed, 12 Aug 2026 15:51:27 GMT · run logs
ae04a56Tue, 11 Aug 2026 16:38:40 GMT · run logs
080e4b2Tue, 11 Aug 2026 15:28:43 GMT · run logs
847a1c3Tue, 11 Aug 2026 06:37:30 GMT · run logs
ℹ️ Metric definitions & methodologyBest/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
shalabhc
changed the title
shalabhc
added
the
ws-transport-test
ae04a56 to
eea5ff1
Compare
| // Named on both transports so a trace or a latency dashboard can tell which | ||
| // one served a write — they are otherwise deliberately indistinguishable, | ||
| // right down to the span name and `url.full`. See `postEventFrameOverWs`. | ||
| attributes: { ...WorkflowEventsTransport('http'), ...attributes }, |
There was a problem hiding this comment.
the GET call sites will also be tagged with this attribute - is that intentional?
There was a problem hiding this comment.
Actually, no was a mistake. fixed.
karthikscale3
left a comment
karthikscale3
left a comment
There was a problem hiding this comment.
LGTM. Just left one comment
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
|
Rebased cleanly onto current main with only the intended WS-span commits. The branch was updated with force-with-lease after verifying the 9-file diff and git diff --check. |
…port PR #3084 added the opt-in `WORKFLOW_EVENTS_TRANSPORT=ws` path and listed "no client-side span on the WS path" as a known limitation. Because event writes become multiplexed frames on one long-lived socket rather than individual `fetch` calls, the per-event `http POST` CLIENT span that the HTTP transport produced simply disappeared — traces went from one span per event to nothing between the invocation and the server. Restore it by synthesizing a request-shaped span around each frame, and give the upgrade its own span: - Extract `withHttpClientSpan` / `recordClientSpanStatus` from `instrumentedFetch` in `http-core.ts` so the synthetic span is emitted by the same envelope as the real one and cannot drift from it. `InstrumentedFetchOptions` now extends `HttpClientSpanOptions`. - `postEventFrameOverWs` opens `http POST` with `url.full` pointing at the v4 REST endpoint the frame is forwarded into, so per-event traces and latency dashboards keep working across the flag. Extract `eventsV4Url` so that URL cannot drift from the one the HTTP path actually requests. - Tag both transports with `workflow.events.transport` (`http` | `ws`) and `workflow.event.type`; the WS path additionally sets `network.protocol.name=websocket`, `workflow.events.ws.url` (the real wire destination) and `workflow.events.ws.req_id` (join key to the server's log line for the frame), so the span is never mistaken for a real HTTP request. - Add a `workflow.events.ws.connect` span around the upgrade — the one genuinely-HTTP request here, previously the invisible half of every WS write's latency — carrying `workflow.events.ws.reconnect_attempt`. This also puts `resolveUpgradeHeaders`' trace-context injection inside a client span, as AGENTS.md requires. - Fix `parseServer` to treat `wss:` as TLS (port 443, not 80). Out of scope, deliberately: per-frame `traceparent` (needs a frame-meta field plus a server change) and Vercel's outgoing-requests view (that instruments global `fetch`, so a frame structurally cannot appear there). Covered by `ws-transport-spans.test.ts`, which drives the real selection + transport + adapter stack over a fake socket and asserts span shape, failure reporting, retry behaviour and HTTP/WS parity. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhchaturvedi-7802 <shalabh.chaturvedi@vercel.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhchaturvedi-7802 <shalabh.chaturvedi@vercel.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
Signed-off-by: Shalabh Chaturvedi <shalabh.chaturvedi@vercel.com> Co-Authored-By: shalabhchaturvedi-7802 <shalabh.chaturvedi@vercel.com>
Signed-off-by: Shalabh Chaturvedi <shalabh.chaturvedi@vercel.com> Co-Authored-By: Shalabh Chaturvedi <shalabh.chaturvedi@vercel.com>
|
No backport to This is observability feature work: it adds new synthetic per-event client spans, a To override, re-run the Backport to stable workflow manually via |
2 participants
Problem
Websocket transport (#3084) does not produce per-message spans. In HTTP we got per request spans. In the WS transport writes become multiplexed frames on one long-lived socket, so the per-event spans dont exist.
This restores per-event spans by synthesizing a request-shaped span around each frame.
AI Details
Approach
The guiding constraint was comparability: a trace collected with the flag on should line up with one collected with it off, so dashboards and per-event traces survive flipping
WORKFLOW_EVENTS_TRANSPORT. The guiding counter-constraint was honesty: a synthetic span must never be mistaken for a real HTTP request.withHttpClientSpan/recordClientSpanStatusout ofinstrumentedFetchinhttp-core.ts, so the synthetic span is emitted by the same code as the real one and cannot drift from it.InstrumentedFetchOptionsnow extendsHttpClientSpanOptions.postEventFrameOverWsopenshttp POSTwithurl.fullpointing at the v4 REST endpoint the frame is forwarded into — that's what keeps traces and latency dashboards comparable across the flag. ExtractedeventsV4Urlso this URL cannot drift from the one the HTTP path actually requests.workflow.events.transport(http|ws) andworkflow.event.type. The WS path additionally setsnetwork.protocol.name=websocket,workflow.events.ws.url(the real wire destination) andworkflow.events.ws.req_id(the join key to the server's log line for that frame). Settingworkflow.events.transporton both paths is deliberate — it's the only way to slice two otherwise-identical spans against each other.workflow.events.ws.connectaround the upgrade — the one genuinely-HTTP request in this path — carryingworkflow.events.ws.reconnect_attempt. Named for the operation so it doesn't bucket with the event writes. This also putsresolveUpgradeHeaders' trace-context injection inside a client span, which AGENTS.md requires of every outgoing world-vercel request.parseServertreatedwss:as plaintext and defaultedserver.portto 80; it's now TLS-aware (443).Deliberately out of scope
traceparent. Each frame can't carry its own trace context without a frame-meta field plus a coordinated server change. Only the upgrade propagates context today. Happy to spec it separately.fetch. A WS frame is structurally not afetch, so it cannot appear there no matter what spans we emit. OTEL traces are the recoverable surface; this closes that one.Behaviour changes worth flagging
Two assertions in
trace-propagation.test.tschanged, both intended:traceparentnow names the connect span rather than the invocation span (it's a child, as every HTTP path already does).traceparentis now injected even when no span is active — parity with every other world-vercel request path, where previously the WS upgrade injected nothing.Tests
New
ws-transport-spans.test.ts(12 tests). Unlikeevents-v4-ws.test.tsit does not mockresolveWsTransport, so spans come from the real selection + transport + adapter stack over a fake socket. Covers span shape and kind,url.full, the honesty attributes,reqIdsequencing across writes, parenting to the caller rather than the connection, absence of a span when falling back to HTTP,HTTP 409andTRANSPORTfailure reporting, one span per retry attempt throughwithEventPostRetry, connect-span attributes, a refused upgrade, and HTTP/WS parity.Verified:
tsc --noEmitclean;vitest run src→ 487/487 passing across 24 files.Docs updated in
docs/content/worlds/v5/vercel.mdx(attribute table comparing the two transports), plus an AGENTS.md note that non-fetchrequest paths must still open their client span viawithHttpClientSpan. Changeset included (@workflow/world-vercel: patch).🤖 Generated with Claude Code