0.3.4: pytest is failing because missing `node_modules/y-websocket/package.json` file

[kloczek]

I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

• python3 -sBm build -w --no-isolation
• because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
• install .whl file in </install/prefix>
• run pytest with $PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>
• build is performed in env which is cut off from access to the public network (pytest is executed with -m "not network")

On first look looks like some files are missing in git repo.
Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-jupyter-ydoc-0.3.4-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-jupyter-ydoc-0.3.4-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -m 'not network'
ImportError while loading conftest '/home/tkloczko/rpmbuild/BUILD/jupyter_ydoc-0.3.4/tests/conftest.py'.
tests/conftest.py:26: in <module>
    update_json_file(here.parent / "node_modules/y-websocket/package.json", d)
tests/conftest.py:17: in update_json_file
    with open(path, "rb") as f:
E   FileNotFoundError: [Errno 2] No such file or directory: '/home/tkloczko/rpmbuild/BUILD/jupyter_ydoc-0.3.4/node_modules/y-websocket/package.json'

Here is list of installed modules in build env

Package                       Version
----------------------------- -----------------
aiofiles                      23.1.0
alabaster                     0.7.13
attrs                         22.2.0
Babel                         2.12.1
build                         0.10.0
charset-normalizer            3.1.0
distro                        1.8.0
docutils                      0.19
editables                     0.3
exceptiongroup                1.0.0
gpg                           1.18.0-unknown
hatchling                     1.13.0
idna                          3.4
imagesize                     1.4.1
importlib-metadata            6.1.0
iniconfig                     2.0.0
Jinja2                        3.1.2
libcomps                      0.1.19
markdown-it-py                2.2.0
MarkupSafe                    2.1.2
mdit-py-plugins               0.3.5
mdurl                         0.1.2
myst-parser                   1.0.0
packaging                     23.0
pathspec                      0.11.0
pip                           23.0.1
pluggy                        1.0.0
Pygments                      2.14.0
pyproject_hooks               1.0.0
pytest                        7.2.2
python-dateutil               2.8.2
pytz                          2023.2
PyYAML                        6.0
requests                      2.28.2
rpm                           4.17.0
six                           1.16.0
snowballstemmer               2.2.0
Sphinx                        6.1.3
sphinxcontrib-applehelp       1.0.4
sphinxcontrib-devhelp         1.0.2.dev20230202
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-jsmath          1.0.1.dev20230128
sphinxcontrib-qthelp          1.0.3.dev20230128
sphinxcontrib-serializinghtml 1.1.5
tomli                         2.0.1
typing_extensions             4.5.0
urllib3                       1.26.15
websockets                    10.4
wheel                         0.38.4
y-py                          0.6.1
ypy-websocket                 0.8.4
zipp                          3.15.0

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[davidbrochart]

Did you install the node modules?

[kloczek]

Here is build output:

+ /usr/bin/python3 -sBm build -w --no-isolation
* Getting build dependencies for wheel...
* Building wheel...
Successfully built jupyter_ydoc-0.3.4-py3-none-any.whl

[davidbrochart]

If you want to run the tests you need to do the same as in the CI.

[kloczek]

So why all necessary steps are not part of the begin of the pytest session and some steps are coded outside pytest framework? 🤔

[kloczek]

BTW line

jupyter_ydoc/.github/workflows/test.yml

Line 88 in eea2058

python -m pytest -v

is wring.
It was many times told by pytest told by pytest developers to not use pytest by python -m pytest because if python is executed by python -m foo it adds current directory to sys.path whcih usually mesess test suite.
This is why pytest provides pytest script wrapper.

[kloczek]

Just tested 1.0.0 and looks like on generate documentation is jet another issue

+ /usr/bin/sphinx-build -n -T -b man docs/source build/sphinx/man
Running Sphinx v6.1.3
making output directory... done
npm ERR! code EUNSUPPORTEDPROTOCOL
npm ERR! Unsupported URL Type "workspace:": workspace:javascript

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/tkloczko/.npm/_logs/2023-03-29T13_27_10_716Z-debug-0.log

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/sphinx/cmd/build.py", line 279, in build_main
    app = Sphinx(args.sourcedir, args.confdir, args.outputdir,
  File "/usr/lib/python3.8/site-packages/sphinx/application.py", line 237, in __init__
    self.config.setup(self)
  File "/home/tkloczko/rpmbuild/BUILD/jupyter_ydoc-1.0.0/docs/source/conf.py", line 96, in setup
    check_call(["npm", "install"], cwd=str(js))
  File "/usr/lib64/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['npm', 'install']' returned non-zero exit status 1.

Exception occurred:
  File "/usr/lib64/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['npm', 'install']' returned non-zero exit status 1.
The full traceback has been saved in /tmp/sphinx-err-3sxomjlk.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
Building jupyterlab API docs

[davidbrochart]

So why all necessary steps are not part of the begin of the pytest session and some steps are coded outside pytest framework?

That would be a nice improvement. Do you want to open a PR?
At the same time, I don't think that asking pytest to run yarn to install node modules would solve your issue, since you work in an environment cut off from access to the public network, right?

[kloczek]

Most of the time I'm spending on packaging.
I'm not bred and butter python develoer however in may jupyter modules already are implemrntd callibg yarn.
Just checked my spec fiels:

[tkloczko@pers-jacek SPECS]$ grep yarn python-*
python-ipyleaflet.spec:BuildRequires:   yarnpkg
python-jupyterlab.spec:BuildRequires:   yarnpkg

[davidbrochart]

But how would you get the node modules without access to the Internet?

[kloczek]

I've tested that currently even with opened access to the public network nothing tries to download sometbing.

Generally it is alway possible to provide necessaty nodejs mdules installed locally.

[davidbrochart]

Currently you have to invoke yarn just like in the CI.