fix(otel): strict per-trace span payloads with opt-in mid-run streaming#215
Draft
lucleray wants to merge 2 commits into
Draft
fix(otel): strict per-trace span payloads with opt-in mid-run streaming#215lucleray wants to merge 2 commits into
lucleray wants to merge 2 commits into
Conversation
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
Comment on lines
+115
to
+117
| for (const finalizer of traceFinalizers) { | ||
| finalizer(traceId); | ||
| } |
Contributor
There was a problem hiding this comment.
Direct for...of iteration over a Set<TraceFinalizer> breaks the build-types step with TS2802 because tsc ignores tsconfig.json's target: ES2021 when the entry file is passed as a positional argument.
…equest context Long-running invocations (e.g. Vercel Workflows) lose most spans from the first part of the run: the trace looks blank early on, with all step spans bunched into the last few seconds. Root cause, in layers: 1. The BatchSpanProcessor's scheduled flush runs in a bare setTimeout, outside the request's AsyncLocalStorage, so the runtime exporter resolved no request context and silently dropped every mid-run batch. 2. The BSP queue is shared by all concurrent requests on the instance, so even an in-context flush ships other invocations' spans through a foreign telemetry channel, where they are not reliably persisted (verified empirically: per-trace re-routing alone still lost most spans). 3. Only a single report delivered near the end of the request is reliably persisted end to end (verified empirically: mid-run reports through the correct owning context still did not land; a single report at the end always does). Fix: - Capture the request context per trace id at root-span start (always in-request), bounded by a hard cap against never-ending traces. - The exporter ACCUMULATES spans of registered traces into per-trace buffers instead of reporting mid-run; untracked traces keep shipping immediately through the ambient context; unattributable spans are retained and re-attempted instead of silently acked. - When the trace's final waitUntil flush runs (inside the owning request), finalizeTrace ships the whole buffer in a single report through the owning context and drops the registration. - The composite processor drains the BSP queue in-context on span end (debounced on OTEL_BSP_MAX_EXPORT_BATCH_SIZE / OTEL_BSP_SCHEDULE_DELAY) so long runs cannot overflow its maxQueueSize. Off-Vercel behavior is unchanged. Verified on a long-running workflow reproduction: with this change all step spans consistently survive across repeated runs (previously only the last stretch of the trace did).
lucleray
force-pushed
the
luc/stream-spans-per-trace
branch
from
8510c50 to
b56e730
Compare
lucleray
force-pushed
the
luc/stream-spans-per-trace
branch
from
b56e730 to
cd8cbf2
Compare
…ontexts Variation of the accumulate-and-ship-once approach (#214) that also supports exporting spans DURING the invocation, not only at its end. Empirical finding behind the design: payloads composed of a SINGLE trace that belongs to an active request are the shape the delivery pipeline handles reliably. What poisoned mid-run delivery before was payload mixing: the BatchSpanProcessor's queue is shared by all concurrent requests, so its flushes shipped mixed-trace payloads whose delivery was non-deterministic, and stale retained spans leaking into active payloads recreated the same poisoning even with client-side routing. The exporter now enforces strict per-trace payload discipline: - Every flush groups incoming spans by their own trace id and ships each trace's spans as their own single-trace payload, through the telemetry channel captured for that trace at root-span start (ambient fallback for untracked traces). Traces never share a payload. - Spans with no reachable channel are buffered per trace (bounded, oldest dropped with a warning) and retried on the next flush; the trace's finalize (from the owning request's waitUntil) ships whatever is left. - The composite processor still drains the BSP queue in-context on span end (debounced on OTEL_BSP_MAX_EXPORT_BATCH_SIZE / OTEL_BSP_SCHEDULE_DELAY) so long runs cannot overflow maxQueueSize, and registers/finalizes the per-trace context as in #214. Streaming is opt-in via VERCEL_OTEL_STREAM_SPANS=1 and not recommended yet: parts of the platform-side delivery pipeline currently only reliably persist batches delivered near the end of the request, so the default remains accumulate-and-ship-once. Once that is addressed, flipping the flag gives mid-run streaming without loss; spans then leave the process every flush interval, bounding client memory on long runs.
lucleray
force-pushed
the
luc/stream-spans-per-trace
branch
from
cd8cbf2 to
f711405
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Variation of #214 that structures the exporter for streaming spans during the invocation, while keeping the reliable end-of-trace delivery as the default. Same foundation (per-trace context capture, in-context queue drain, retain-instead-of-drop), plus strict per-trace payload discipline and an opt-in streaming mode.
Behavior
waitUntilflush, through the context captured at root-span start. Untracked traces ship immediately via the ambient context. Unattributable spans are retained and retried instead of silently acked.VERCEL_OTEL_STREAM_SPANS=1): every flush ships each trace's spans as their own single-trace payload immediately, bounding client memory on long runs. Opt-in and not recommended yet: parts of the platform-side delivery pipeline currently only reliably persist batches delivered near the end of the request, so streamed spans of long traces can still be lost. Once that is addressed, flipping the flag gives streaming without loss; the client side is ready.Verification
Long-running workflow reproduction, default env:
Relationship to #213 / #214
Testing
type-check,eslint, full unit suite pass. Bundle size limits bumped slightly (main was within ~2KB / ~100 bytes of them).