fix(cli): locate pyodide vendor site-packages by discovery, not a guessed OS-conditional path - #211
Conversation
…ssed OS-conditional path _install_requirements_to_vendor guessed the pyodide venv's site-packages location from os.name (Lib/site-packages on Windows, lib/pythonX.Y/site-packages elsewhere). That venv always targets a foreign interpreter (cpython-*-emscripten-wasm32-musl), never the host's own, so there's no OS-conditional formula guaranteed to hold for it. When it guessed wrong, uv still installed packages correctly, but they were never copied into python_modules -- so the deployed Worker fell back to workerd's always-raising stub "workers" module and failed at deploy time with ModuleNotFoundError. Adds _find_pyodide_site_packages(), which discovers the real directory by globbing for it instead of assuming a layout, regardless of host OS or uv version. Fixes cloudflare/workers-sdk#15208
|
All contributors have signed the CLA ✍️ ✅ |
|
I have read the CLA Document and I hereby sign the CLA |
|
I don't think this is a right fix. The problem is that uv is not calculating the site-packages directory correctly in windows. |
|
You were right, and I found the concrete evidence — thank you for pushing on this. Tested directly on Windows (uv 0.12.5): Same result whether Filed this against uv directly: astral-sh/uv#21199. Given the real fix lives there, want me to close this PR, or is there appetite for a workaround here in the meantime (e.g. falling back to |
|
#214 fixed it for now, so closing. |
Fixes #212.
What's wrong
_install_requirements_to_vendor()guessed the pyodide venv's site-packages location fromos.name(Lib/site-packageson Windows,lib/pythonX.Y/site-packageselsewhere). That venv always targets a foreign interpreter (cpython-*-emscripten-wasm32-musl, seeget_uv_pyodide_interp_name()), never the host's own — unlikevenv_workers_path(a real native venv, which does correctly follow the host OS's convention) — so there's no single OS-conditional formula that's safe to assume holds for it.When the guess was wrong,
uv pip installstill installed the real packages correctly (it doesn't use this guessed path at all), but they were never copied intopython_modules— so the deployed Worker'spython_modulesbundle ended up missing the realworkerspackage. At runtime,workerd's Python-Workers helper (src/pyodide/python-entrypoint-helper.tsincloudflare/workerd) deliberately injects an always-raising stubworkersmodule as a safety net for exactly this situation, which is what surfaced as the reportedModuleNotFoundError: No module named 'workers'at Cloudflare's deploy-time validation step — not a bug inworkerdor inworkers-runtime-sdkitself.Traced this the whole way from the original issue on
workers-sdk: the traceback'spyodideRuntime-internal/pyodide:python-entrypoint-helperframes pointed atworkerd's own runtime code (public source), which pointed back at this repo's vendoring step as the thing responsible for actually getting the real SDK bundled.Fix
Adds
_find_pyodide_site_packages(), which discovers the real site-packages directory by globbing for it (taking the shallowest match) instead of constructing a guessed path. This works regardless of what layout conventionuvactually uses for this cross-target venv on any given host OS oruvversion, so it doesn't just trade one unverified guess for another.Testing
workers-runtime-sdkas an editable dependency,wrangler.jsoncwithpython_workers, anentry.pyimportingResponse/WorkerEntrypoint). Confirmed the realworkerspackage now correctly lands inpython_modulesafterpywrangler sync.test_find_pyodide_site_packages_discovers_nested_layout(proves discovery works even at a deliberately non-standard nesting depth, not just one that happens to match a real convention) andtest_find_pyodide_site_packages_raises_when_missing(a clear error instead of ashutil.copytreecrash when the venv has no site-packages at all). Confirmed both fail withAttributeError/uncovered-path against the unpatched code.ruff format,ruff check, andmypy(viauvx pre-commit run, matching the repo's pinned tool versions) all clean.