Skip to content

fix(cli): locate pyodide vendor site-packages by discovery, not a guessed OS-conditional path - #211

Closed
mittalpk wants to merge 1 commit into
cloudflare:mainfrom
mittalpk:fix/pyodide-vendor-site-packages-path
Closed

fix(cli): locate pyodide vendor site-packages by discovery, not a guessed OS-conditional path#211
mittalpk wants to merge 1 commit into
cloudflare:mainfrom
mittalpk:fix/pyodide-vendor-site-packages-path

Conversation

@mittalpk

Copy link
Copy Markdown

Fixes #212.

What's wrong

_install_requirements_to_vendor() guessed the pyodide venv's site-packages location from os.name (Lib/site-packages on Windows, lib/pythonX.Y/site-packages elsewhere). That venv always targets a foreign interpreter (cpython-*-emscripten-wasm32-musl, see get_uv_pyodide_interp_name()), never the host's own — unlike venv_workers_path (a real native venv, which does correctly follow the host OS's convention) — so there's no single OS-conditional formula that's safe to assume holds for it.

When the guess was wrong, uv pip install still installed the real packages correctly (it doesn't use this guessed path at all), but they were never copied into python_modules — so the deployed Worker's python_modules bundle ended up missing the real workers package. At runtime, workerd's Python-Workers helper (src/pyodide/python-entrypoint-helper.ts in cloudflare/workerd) deliberately injects an always-raising stub workers module as a safety net for exactly this situation, which is what surfaced as the reported ModuleNotFoundError: No module named 'workers' at Cloudflare's deploy-time validation step — not a bug in workerd or in workers-runtime-sdk itself.

Traced this the whole way from the original issue on workers-sdk: the traceback's pyodideRuntime-internal/pyodide:python-entrypoint-helper frames pointed at workerd's own runtime code (public source), which pointed back at this repo's vendoring step as the thing responsible for actually getting the real SDK bundled.

Fix

Adds _find_pyodide_site_packages(), which discovers the real site-packages directory by globbing for it (taking the shallowest match) instead of constructing a guessed path. This works regardless of what layout convention uv actually uses for this cross-target venv on any given host OS or uv version, so it doesn't just trade one unverified guess for another.

Testing

  • Built and ran the fixed CLI against a scaffold matching the reporter's exact setup (workers-runtime-sdk as an editable dependency, wrangler.jsonc with python_workers, an entry.py importing Response/WorkerEntrypoint). Confirmed the real workers package now correctly lands in python_modules after pywrangler sync.
  • Added test_find_pyodide_site_packages_discovers_nested_layout (proves discovery works even at a deliberately non-standard nesting depth, not just one that happens to match a real convention) and test_find_pyodide_site_packages_raises_when_missing (a clear error instead of a shutil.copytree crash when the venv has no site-packages at all). Confirmed both fail with AttributeError/uncovered-path against the unpatched code.
  • Full suite: 56 passed (54 existing + 2 new), 0 regressions.
  • ruff format, ruff check, and mypy (via uvx pre-commit run, matching the repo's pinned tool versions) all clean.

…ssed OS-conditional path

_install_requirements_to_vendor guessed the pyodide venv's
site-packages location from os.name (Lib/site-packages on Windows,
lib/pythonX.Y/site-packages elsewhere). That venv always targets a
foreign interpreter (cpython-*-emscripten-wasm32-musl), never the
host's own, so there's no OS-conditional formula guaranteed to hold
for it. When it guessed wrong, uv still installed packages
correctly, but they were never copied into python_modules -- so the
deployed Worker fell back to workerd's always-raising stub "workers"
module and failed at deploy time with ModuleNotFoundError.

Adds _find_pyodide_site_packages(), which discovers the real
directory by globbing for it instead of assuming a layout,
regardless of host OS or uv version.

Fixes cloudflare/workers-sdk#15208
@github-actions

github-actions Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@mittalpk

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

github-actions Bot added a commit that referenced this pull request Aug 18, 2026
@ryanking13

Copy link
Copy Markdown
Contributor

I don't think this is a right fix. The problem is that uv is not calculating the site-packages directory correctly in windows.

@mittalpk

Copy link
Copy Markdown
Author

You were right, and I found the concrete evidence — thank you for pushing on this.

Tested directly on Windows (uv 0.12.5): uv venv --python cpython-3.13.2-emscripten-wasm32-musl creates a venv with the Windows-native layout (Lib\site-packages, Scripts\python.exe), but uv pip install --python <that venv> crashes outright — it doesn't install to the wrong place, it fails trying to create a malformed path:

Using Python 3.13.2 environment at: /
error: Failed to install: click-8.4.2-py3-none-any.whl (click==8.4.2)
  Caused by: Failed to create directory `//lib/python3.13/site-packages\`
  Caused by: failed to create directory `//lib/python3.13/site-packages\`: The network path was not found. (os error 53)

Same result whether --python points at the venv directory or directly at Scripts\python.exe. So this PR's fix doesn't actually solve the reported bug — _find_pyodide_site_packages() can't discover packages that were never installed in the first place.

Filed this against uv directly: astral-sh/uv#21199. Given the real fix lives there, want me to close this PR, or is there appetite for a workaround here in the meantime (e.g. falling back to --prefix instead of --python, which a related closed uv issue confirms does work — just with host-OS layout, which the discovery approach in this PR would still handle correctly)? Happy to go either way.

@ryanking13

Copy link
Copy Markdown
Contributor

#214 fixed it for now, so closing.

@ryanking13 ryanking13 closed this Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python Workers: ModuleNotFoundError: No module named 'workers' on deploy, even with official pywrangler init scaffold

2 participants