Skip to content

fix(core): fail runs delivered to the wrong deployment#2944

Closed
alangenfeld wants to merge 1 commit into
vercel:mainfrom
alangenfeld:alangenfeld/fail-cross-deployment-runs
Closed

fix(core): fail runs delivered to the wrong deployment#2944
alangenfeld wants to merge 1 commit into
vercel:mainfrom
alangenfeld:alangenfeld/fail-cross-deployment-runs

Conversation

@alangenfeld

@alangenfeld alangenfeld commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

A run created on one deployment whose queue callbacks route to another (common on preview branches, where a superseded deployment is deactivated) fails cryptically: the step handler resolves the per-run encryption key from the bare runId, derives it from the wrong deployment's master key, and throws a RuntimeDecryptionError before user code runs. The queue retry callback swallows it, so the run dies as a blank "exceeded max retries."

A run may only execute on the deployment that created it (continuing elsewhere risks code skew). This detects delivery to the wrong deployment and fails the run with the new DEPLOYMENT_MISMATCH error code.

  • Guards every entrypoint that executes a run — main workflow replay, background step, and the queue step handler — so the run is stopped before any workflow code, inline step, or queue step runs.
  • Records the failure without the origin deployment's key. That key is fetched from the origin's API, which is often gone once a run outlives its deployment — depending on it would throw and drop us back into silent retry-exhaustion. The error is written unencrypted (deployment ids only) and the plaintext errorCode is what observability and the UI key off (describeRunError maps it to an actionable hint).
  • Adds RUN_ERROR_CODES.DEPLOYMENT_MISMATCH; WorkflowDeploymentMismatchError classifies to it.

Closes #2816.

Follow-ups (not in this PR)

  • UI: link DEPLOYMENT_MISMATCH to a docs page.
  • Log the swallowed error in world-vercel's queue retry callback (the issue's second problem — general diagnosability, orthogonal to this fix).

Testing

  • pnpm --filter @workflow/core... build
  • pnpm --filter @workflow/core exec vitest run src --exclude src/vm/uint8array-base64.test.ts (1,430 tests)
  • pnpm --filter @workflow/errors test (38 tests)
  • core and errors typechecks

The excluded VM test has existing cross-realm assertion failures on local Node 26; the repository supports through Node 24.

@changeset-bot

changeset-bot Bot commented Jul 15, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: c0b46d5

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 20 packages
Name Type
@workflow/core Patch
@workflow/errors Patch
@workflow/builders Patch
@workflow/cli Patch
@workflow/next Patch
@workflow/nitro Patch
@workflow/vitest Patch
@workflow/web-shared Patch
@workflow/web Patch
workflow Patch
@workflow/world-testing Patch
@workflow/world-local Patch
@workflow/world-postgres Patch
@workflow/world-vercel Patch
@workflow/astro Patch
@workflow/nest Patch
@workflow/nuxt Patch
@workflow/rollup Patch
@workflow/sveltekit Patch
@workflow/vite Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Someone is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

@alangenfeld
alangenfeld force-pushed the alangenfeld/fail-cross-deployment-runs branch 3 times, most recently from c4770b1 to c3b4b0b Compare July 15, 2026 23:15
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
@alangenfeld
alangenfeld force-pushed the alangenfeld/fail-cross-deployment-runs branch from c3b4b0b to c0b46d5 Compare July 16, 2026 00:17
@alangenfeld alangenfeld closed this by deleting the head repository Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant