feat(core): side-effect-free serialization of workflow VM values - #3257
Conversation
Serialization runs on the host but inspects values constructed inside the
node:vm sandbox, so ordinary dynamic operations dispatch into the sandbox
realm and execute workflow code: `value.toISOString()`, `Array.from(map)`,
`Object.prototype.toString` (via Symbol.toStringTag), `.source`/`.flags`,
`.href`, view `.buffer`/`.byteOffset`/`.byteLength`, and error
`.message`/`.stack`/`.cause` reads.
That is a determinism hazard. A payload is serialized exactly once and is
never re-serialized on replay, so any workflow-visible side effect it
triggers exists only on the live path — a patched `Date.prototype.toISOString`
that consumes a seeded `Math.random()` draw, for example, shifts every
subsequent draw and diverges from replay.
This makes serialization side-effect free where the data allows it, and
observable where it does not:
- Classification uses engine brand checks (node:util types, internal-slot
probes) instead of `instanceof global.X` and Object.prototype.toString, so
it is immune to Symbol.hasInstance, reassigned sandbox globals, and
Symbol.toStringTag spoofs. An unbranded value claiming a brand-decided tag
is now classified as a plain object instead of being routed into an
extractor that requires the real internal slot (unhardened devalue crashes
on that input).
- Extraction goes through intrinsics captured at module load — host boot,
before any workflow bundle runs — invoked with explicit receivers.
Internal slots are realm-agnostic, so host intrinsics read VM-realm
objects without touching the sandbox's patchable prototypes.
- Property access reads through descriptors, so plain data never invokes
anything.
Where workflow code must run because the data lives behind it — getters,
proxies, custom [WORKFLOW_SERIALIZE] methods, toString() on
toStringTag-branded objects like Temporal polyfills — the execution is
preserved for compatibility and recorded in a new `CodecOptions.guestCodeStats`
sink, surfaced as workflow.serialization.guest_code_{executions,details} span
attributes. Consumers that retain a VM across steps can treat a non-empty
report as "serialization may have perturbed VM state".
Engine-provided accessors are deliberately not reported: V8 defines `stack`
as an own accessor on every Error instance, so reporting it would flag every
serialized error. Nativeness is decided with the captured host
Function.prototype.toString; the bound-function caveat is documented in
hardened.ts.
Requires devalue 5.9.0 for the pluggable `operations` option.
🦋 Changeset detectedLatest commit: 7b5a085 The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
📊 Workflow Benchmarkscommit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 382767ms → this run 402865ms (Δ +20098ms, +5%) 1020 steps (queue-hop) Cumulative STSO time: main 6857ms → this run 8217ms (Δ +1360ms, +20%) 📜 Previous results (5)5653844Sat, 01 Aug 2026 06:46:38 GMT · run logs
423983dSat, 01 Aug 2026 00:42:35 GMT · run logs
84c59b7Fri, 31 Jul 2026 22:31:19 GMT · run logs
d738ff4Fri, 31 Jul 2026 22:07:57 GMT · run logs
32cac72Fri, 31 Jul 2026 17:01:46 GMT · run logs
ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
🧪 E2E Test Results✅ All tests passed E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
✅ vercel-multi-region
|
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
|
All of the nextjs tests seem to be failing consistently 🤔 |
Every Next.js e2e job failed on the two webhook tests: the hook POST
returned 404 because `resumeWebhook` could not serialize its step return
value ("Cannot stringify arbitrary non-POJOs"), so no hook was ever
registered.
The value was a `NextRequest`, which Next.js hands over as a **Proxy**.
`isInstanceOfPrototype` rejected proxies outright, so the Request reducer
answered "not a Request" and devalue fell through to the POJO check. The
reasoning behind rejecting them — that proxied built-ins were never
serializable, because internal-slot reads throw on a proxy receiver — is
true for `Map`/`Date`/`URL`, whose reducers read internal slots, but not
for `Request`/`Response`/streams, whose reducers read ordinary
properties. Next's proxy forwards those with the target as receiver, so
they serialized fine before this PR.
Identification now walks through proxies, matching `instanceof`, and
records the traps rather than suppressing the answer. The three reducers
that do read internal slots (URL, URLSearchParams, Headers) fall back to
the dynamic read when the value is a proxy, so their behavior is exactly
what it was before — including throwing for a bare proxy over a built-in,
which threw before too.
Verified against the real thing: the full nextjs-turbopack e2e suite
(135 tests) passes locally, having reproduced the failure first and
confirmed a reverted `serialization.ts` fixed it.
The regression test uses a receiver-correcting proxy, which is what makes
NextRequest work in practice; a comment records that a bare
`new Proxy(request, {})` throws on undici's private slots with or without
this change.
…on' into nrajlich/hardened-vm-serialization
|
Fixed in 8bc462f — thanks, this was a real bug in the PR, not flake. Cause. Both failures were the webhook tests, and the 404 was downstream: Why I got it wrong. I had reasoned (and asserted in a review reply) that proxied built-ins were never serializable, since internal-slot reads throw on a proxy receiver. That holds for Fix. Identification now walks through proxies, matching Verification. Reproduced locally first, then confirmed by reverting Also merged |
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
VaguelySerious
left a comment
VaguelySerious
left a comment
There was a problem hiding this comment.
AI review: no blocking issues
- `isInstanceOfPrototype`'s JSDoc still described the behavior removed in 8bc462f (proxies rejected without firing traps), which is the opposite of what it now does. - The `__closureVarsFn` provenance check proves the function was passed to `useStep`, not that this package generated it: `useStep` is published on the sandbox global, so workflow code can call it with a function of its own and have it marked. Renamed `registerTrustedFunction` / `isTrustedFunction` to `markUseStepClosureFn` / `isUseStepClosureFn` so the name states the boundary, and documented the laundering caveat alongside the existing ones. Marking still earns its keep — reporting every step that captures a variable would bury the signal — and closing the gap properly needs a compiler-emitted marker, which is a compiler change. - Added the missing coverage for both sides of that check: an unmarked `__closureVarsFn` is invoked and reported, a marked one is invoked and not. - `guestCodeStats` was documented as something a retained-VM gate consumes, but no runtime caller passes a sink; the executions reach telemetry from every dehydrate path regardless. Reworded both docs to say that, so the out-param is not mistaken for wiring that already exists.
|
No backport to This is a large To override, re-run the Backport to stable workflow manually via |
Summary
Serialization runs on the host, but the values it inspects were constructed inside the
node:vmsandbox by workflow code. Every ordinary dynamic operation therefore dispatches into the sandbox realm and executes workflow code:Datevalue.getDate(),value.toISOString()Map/SetArray.from(value)→Symbol.iteratorHeadersArray.from(value)→Symbol.iteratorRegExp.source/.flagsprototype gettersURL/URLSearchParams.href,.size,String(value)DataView.buffer/.byteOffset/.byteLengthgetters.name/.message/.stack/.causereadsInstance/Class.constructor,.classIdreadsinstanceof global.X→Symbol.hasInstance; devalue'sObject.prototype.toString→Symbol.toStringTagWhy that's a correctness problem, not just hygiene. A payload is serialized exactly once, when its
step_startedevent is prepared, and is never re-serialized on replay. So any workflow-visible side effect serialization triggers exists only on the live execution path. ADate.prototype.toISOStringpatched to consume one seededMath.random()draw shifts every subsequent draw — step correlation IDs included — and the next cold start replays a different sequence than the log records. Today that survives only because the VM is destroyed immediately after serializing; it becomes load-bearing the moment a VM is retained across steps.This PR is standalone and independent of retention — it removes the hazard at the source, and the retention work inherits it.
Approach
node:utiltypes+ internal-slot probes replaceinstanceof global.XandObject.prototype.toString. Immune toSymbol.hasInstance, reassigned sandbox globals, andSymbol.toStringTag. A value with no engine brand that claims a brand-decided tag is now classified as a plain object rather than routed into an extractor that requires the real slot — unhardened devalue crashes on that input, so this is strictly better.Map.prototype.entries/Set.prototype.values, so the iterator object — and itsnext— are host-realm too.[WORKFLOW_SERIALIZE]methods, andtoString()on toStringTag-branded objects (Temporal polyfills) still run — full compatibility — and land in a newCodecOptions.guestCodeStatssink, surfaced asworkflow.serialization.guest_code_executions/…guest_code_detailsspan attributes. A retention gate can treat a non-empty report as "serialization may have perturbed VM state".Wired in at the two call sites that serialize sandbox values:
dehydrateStepArgumentsanddehydrateWorkflowReturnValue(both passsuspension.globalThis). Host-realm paths — step returns, step errors, client args — are unaffected apart from also being hardened.Two findings worth calling out
stackas an own accessor on every Error instance, with a shared native getter per realm. Reading it always invokes an engine getter, so naively reporting accessor invocations flags every serialized error. Engine-provided accessors are therefore excluded, decided with the captured hostFunction.prototype.toString. Documented caveats inhardened.ts: a bound function also reports as native code (missed report, never wrong output), and V8's nativestackgetter can itself call a workflow-definedError.prepareStackTrace.getCommonReducers(global)no longer uses itsglobalargument forinstanceof. The parameter is kept for API compatibility (revivers still need it) and now documented as such — brand checks are realm-agnostic, which incidentally fixes the cross-realm misclassification the existingRetryableError.retryAftercomment works around.Tests
42 new tests in
serialization/hardened.test.ts, all fixtures minted inside a realcreateContext()VM and serialized with the VM'sglobalThis, mirroringdehydrateStepArguments:Date.prototype.toISOString/getDate,Map/SetSymbol.iterator(plusentries/valuesrigged to throw),RegExp.prototype.source/flags,URL.prototype.href,URLSearchParams.prototype.toString,Error.prototype.message, andObject.prototype.toString: zero invocations, correct output.Symbol.toStringTag: 'Date', hostileSymbol.hasInstance, reassignedglobalThis.Date/Map, and shadowedbyteLength/byteOffseton a typed array (the shadowed view still serializes the correct byte range).[WORKFLOW_SERIALIZE]recorded with its classId; a broad mixed value reports nothing.packages/coreunit suite: 1695 passing, 3 pre-existing expected failures. Build clean. (The 9 failing e2e files in this worktree are missing workbench builds, unrelated.)Dependency
devalue
5.8.1→5.9.0for the pluggableoperationsoption (sveltejs/devalue#172, #173). The workspaceoverridespin moves with it.