fix(otel): ship each trace's spans in one report through its owning request context#214
Draft
lucleray wants to merge 1 commit into
Draft
fix(otel): ship each trace's spans in one report through its owning request context#214lucleray wants to merge 1 commit into
lucleray wants to merge 1 commit into
Conversation
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
Comment on lines
+114
to
+116
| for (const finalizer of traceFinalizers) { | ||
| finalizer(traceId); | ||
| } |
Contributor
There was a problem hiding this comment.
Direct for...of iteration over a Set in finalizeTrace causes build failure TS2802 under the build-types tsc invocation, which ignores tsconfig.json's ES2021 target.
…equest context Long-running invocations (e.g. Vercel Workflows) lose most spans from the first part of the run: the trace looks blank early on, with all step spans bunched into the last few seconds. Root cause, in layers: 1. The BatchSpanProcessor's scheduled flush runs in a bare setTimeout, outside the request's AsyncLocalStorage, so the runtime exporter resolved no request context and silently dropped every mid-run batch. 2. The BSP queue is shared by all concurrent requests on the instance, so even an in-context flush ships other invocations' spans through a foreign telemetry channel, where they are not reliably persisted (verified empirically: per-trace re-routing alone still lost most spans). 3. Only a single report delivered near the end of the request is reliably persisted end to end (verified empirically: mid-run reports through the correct owning context still did not land; a single report at the end always does). Fix: - Capture the request context per trace id at root-span start (always in-request), bounded by a hard cap against never-ending traces. - The exporter ACCUMULATES spans of registered traces into per-trace buffers instead of reporting mid-run; untracked traces keep shipping immediately through the ambient context; unattributable spans are retained and re-attempted instead of silently acked. - When the trace's final waitUntil flush runs (inside the owning request), finalizeTrace ships the whole buffer in a single report through the owning context and drops the registration. - The composite processor drains the BSP queue in-context on span end (debounced on OTEL_BSP_MAX_EXPORT_BATCH_SIZE / OTEL_BSP_SCHEDULE_DELAY) so long runs cannot overflow its maxQueueSize. Off-Vercel behavior is unchanged. Verified on a long-running workflow reproduction: with this change all step spans consistently survive across repeated runs (previously only the last stretch of the trace did).
lucleray
force-pushed
the
luc/span-context-routing
branch
from
051c6e6 to
1a9cd87
Compare
lucleray
added a commit
that referenced
this pull request
Jul 8, 2026
…ontexts Variation of the accumulate-and-ship-once approach (#214) that also supports exporting spans DURING the invocation, not only at its end. Empirical finding behind the design: payloads composed of a SINGLE trace that belongs to an active request are the shape the delivery pipeline handles reliably. What poisoned mid-run delivery before was payload mixing: the BatchSpanProcessor's queue is shared by all concurrent requests, so its flushes shipped mixed-trace payloads whose delivery was non-deterministic, and stale retained spans leaking into active payloads recreated the same poisoning even with client-side routing. The exporter now enforces strict per-trace payload discipline: - Every flush groups incoming spans by their own trace id and ships each trace's spans as their own single-trace payload, through the telemetry channel captured for that trace at root-span start (ambient fallback for untracked traces). Traces never share a payload. - Spans with no reachable channel are buffered per trace (bounded, oldest dropped with a warning) and retried on the next flush; the trace's finalize (from the owning request's waitUntil) ships whatever is left. - The composite processor still drains the BSP queue in-context on span end (debounced on OTEL_BSP_MAX_EXPORT_BATCH_SIZE / OTEL_BSP_SCHEDULE_DELAY) so long runs cannot overflow maxQueueSize, and registers/finalizes the per-trace context as in #214. Streaming is opt-in via VERCEL_OTEL_STREAM_SPANS=1 and not recommended yet: parts of the platform-side delivery pipeline currently only reliably persist batches delivered near the end of the request, so the default remains accumulate-and-ship-once. Once that is addressed, flipping the flag gives mid-run streaming without loss; spans then leave the process every flush interval, bounding client memory on long runs.
lucleray
added a commit
that referenced
this pull request
Jul 8, 2026
…ontexts Variation of the accumulate-and-ship-once approach (#214) that also supports exporting spans DURING the invocation, not only at its end. Empirical finding behind the design: payloads composed of a SINGLE trace that belongs to an active request are the shape the delivery pipeline handles reliably. What poisoned mid-run delivery before was payload mixing: the BatchSpanProcessor's queue is shared by all concurrent requests, so its flushes shipped mixed-trace payloads whose delivery was non-deterministic, and stale retained spans leaking into active payloads recreated the same poisoning even with client-side routing. The exporter now enforces strict per-trace payload discipline: - Every flush groups incoming spans by their own trace id and ships each trace's spans as their own single-trace payload, through the telemetry channel captured for that trace at root-span start (ambient fallback for untracked traces). Traces never share a payload. - Spans with no reachable channel are buffered per trace (bounded, oldest dropped with a warning) and retried on the next flush; the trace's finalize (from the owning request's waitUntil) ships whatever is left. - The composite processor still drains the BSP queue in-context on span end (debounced on OTEL_BSP_MAX_EXPORT_BATCH_SIZE / OTEL_BSP_SCHEDULE_DELAY) so long runs cannot overflow maxQueueSize, and registers/finalizes the per-trace context as in #214. Streaming is opt-in via VERCEL_OTEL_STREAM_SPANS=1 and not recommended yet: parts of the platform-side delivery pipeline currently only reliably persist batches delivered near the end of the request, so the default remains accumulate-and-ship-once. Once that is addressed, flipping the flag gives mid-run streaming without loss; spans then leave the process every flush interval, bounding client memory on long runs.
lucleray
added a commit
that referenced
this pull request
Jul 8, 2026
…ontexts Variation of the accumulate-and-ship-once approach (#214) that also supports exporting spans DURING the invocation, not only at its end. Empirical finding behind the design: payloads composed of a SINGLE trace that belongs to an active request are the shape the delivery pipeline handles reliably. What poisoned mid-run delivery before was payload mixing: the BatchSpanProcessor's queue is shared by all concurrent requests, so its flushes shipped mixed-trace payloads whose delivery was non-deterministic, and stale retained spans leaking into active payloads recreated the same poisoning even with client-side routing. The exporter now enforces strict per-trace payload discipline: - Every flush groups incoming spans by their own trace id and ships each trace's spans as their own single-trace payload, through the telemetry channel captured for that trace at root-span start (ambient fallback for untracked traces). Traces never share a payload. - Spans with no reachable channel are buffered per trace (bounded, oldest dropped with a warning) and retried on the next flush; the trace's finalize (from the owning request's waitUntil) ships whatever is left. - The composite processor still drains the BSP queue in-context on span end (debounced on OTEL_BSP_MAX_EXPORT_BATCH_SIZE / OTEL_BSP_SCHEDULE_DELAY) so long runs cannot overflow maxQueueSize, and registers/finalizes the per-trace context as in #214. Streaming is opt-in via VERCEL_OTEL_STREAM_SPANS=1 and not recommended yet: parts of the platform-side delivery pipeline currently only reliably persist batches delivered near the end of the request, so the default remains accumulate-and-ship-once. Once that is addressed, flipping the flag gives mid-run streaming without loss; spans then leave the process every flush interval, bounding client memory on long runs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Traces from long-running invocations (e.g. Vercel Workflows) are missing most spans from the first part of the run: the trace looks blank early on, with all step spans bunched into the last few seconds. The work happens the whole time; the spans are genuinely lost.
Root cause (client side)
BatchSpanProcessorschedules flushes with a baresetTimeout, which runs outside the request'sAsyncLocalStorage. The runtime exporter then resolves no request context and silently acked + dropped every mid-run batch. Only the final flush, triggered viawaitUntil(in-context), survived.BatchSpanProcessorqueue is shared by all concurrent requests on an instance, so a flush ships other invocations' spans through the wrong request's telemetry channel, where they are not reliably persisted.Fix
waitUntil. Untracked traces keep shipping immediately through the ambient context.BatchSpanProcessorqueue in-context on span end (debounced onOTEL_BSP_MAX_EXPORT_BATCH_SIZE/OTEL_BSP_SCHEDULE_DELAY) so long runs cannot overflowmaxQueueSize. Off-Vercel behavior is unchanged.Verified on a long-running workflow reproduction: with this change all step spans consistently survive across repeated runs (previously only the last few seconds' spans did).
Relationship to #213
Same problem; this PR keeps its in-context flush and retain-instead-of-drop ideas, and adds per-trace attribution + single-report delivery, which testing showed is required for reliability on warm instances.
Testing
type-check,eslint, full unit suite pass. Bundle size limits bumped slightly (main was within ~2KB / ~100 bytes of them).