Python issues on Ubuntu 24.04 - PyHSS requirements cannot be installed

[dchard]

root@testhss:/etc/pyhss# pip3 install -r requirements.txt
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.

If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.

If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.

See /usr/share/doc/python3.12/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
root@testhss:/etc/pyhss# pip3 install -r requirements.txt --break-system-packages
Collecting aiohttp==3.8.5 (from -r requirements.txt (line 1))
Using cached aiohttp-3.8.5.tar.gz (7.4 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting Flask==2.2.3 (from -r requirements.txt (line 2))
Using cached Flask-2.2.3-py3-none-any.whl.metadata (3.9 kB)
Collecting flask_restx==1.1.0 (from -r requirements.txt (line 3))
Using cached flask_restx-1.1.0-py2.py3-none-any.whl.metadata (9.0 kB)
Collecting Jinja2==3.1.2 (from -r requirements.txt (line 4))
Using cached Jinja2-3.1.2-py3-none-any.whl.metadata (3.5 kB)
Collecting mongo==0.2.0 (from -r requirements.txt (line 5))
Using cached mongo-0.2.0.tar.gz (4.6 kB)
Preparing metadata (setup.py) ... done
Collecting prometheus_client==0.16.0 (from -r requirements.txt (line 6))
Using cached prometheus_client-0.16.0-py3-none-any.whl.metadata (22 kB)
Collecting pycryptodome==3.17 (from -r requirements.txt (line 7))
Using cached pycryptodome-3.17-cp35-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.3 kB)
Collecting pymongo==4.3.3 (from -r requirements.txt (line 8))
Using cached pymongo-4.3.3.tar.gz (814 kB)
Preparing metadata (setup.py) ... done
Collecting pysctp==0.7.2 (from -r requirements.txt (line 9))
Using cached pysctp-0.7.2.tar.gz (32 kB)
Preparing metadata (setup.py) ... done
Collecting pysnmp==4.4.12 (from -r requirements.txt (line 10))
Using cached pysnmp-4.4.12-py2.py3-none-any.whl.metadata (1.8 kB)
Collecting PyYAML==6.0 (from -r requirements.txt (line 11))
Using cached PyYAML-6.0.tar.gz (124 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [54 lines of output]
running egg_info
writing lib/PyYAML.egg-info/PKG-INFO
writing dependency_links to lib/PyYAML.egg-info/dependency_links.txt
writing top-level names to lib/PyYAML.egg-info/top_level.txt
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/pip/vendor/pyproject_hooks/in_process/in_process.py", line 353, in
main()
File "/usr/lib/python3/dist-packages/pip/vendor/pyproject_hooks/in_process/in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/pip/vendor/pyproject_hooks/in_process/in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-m2udz7l/overlay/local/lib/python3.12/dist-packages/setuptools/build_meta.py", line 327, in get_requires_for_build_wheel
return self.get_build_requires(config_settings, requirements=[])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-m2udz7l/overlay/local/lib/python3.12/dist-packages/setuptools/build_meta.py", line 297, in get_build_requires
self.run_setup()
File "/tmp/pip-build-env-m2udz7l/overlay/local/lib/python3.12/dist-packages/setuptools/build_meta.py", line 313, in run_setup
exec(code, locals())
File "", line 288, in
File "/tmp/pip-build-env-m2udz7l/overlay/local/lib/python3.12/dist-packages/setuptools/init.py", line 108, in setup
return distutils.core.setup(**attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-m2udz7l/overlay/local/lib/python3.12/dist-packages/setuptools/distutils/core.py", line 184, in setup
return run_commands(dist)
^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-m2udz7l/overlay/local/lib/python3.12/dist-packages/setuptools/distutils/core.py", line 200, in run_commands
dist.run_commands()
File "/tmp/pip-build-env-m2udz7l/overlay/local/lib/python3.12/dist-packages/setuptools/distutils/dist.py", line 964, in run_commands
self.run_command(cmd)
File "/tmp/pip-build-env-m2udz7l/overlay/local/lib/python3.12/dist-packages/setuptools/dist.py", line 945, in run_command
super().run_command(command)
File "/tmp/pip-build-env-m2udz7l/overlay/local/lib/python3.12/dist-packages/setuptools/distutils/dist.py", line 983, in run_command
cmd_obj.run()
File "/tmp/pip-build-env-m2udz7l/overlay/local/lib/python3.12/dist-packages/setuptools/command/egg_info.py", line 310, in run
self.find_sources()
File "/tmp/pip-build-env-m2udz7l/overlay/local/lib/python3.12/dist-packages/setuptools/command/egg_info.py", line 318, in find_sources
mm.run()
File "/tmp/pip-build-env-m2udz7l/overlay/local/lib/python3.12/dist-packages/setuptools/command/egg_info.py", line 539, in run
self.add_defaults()
File "/tmp/pip-build-env-m2udz7l/overlay/local/lib/python3.12/dist-packages/setuptools/command/egg_info.py", line 577, in add_defaults
sdist.add_defaults(self)
File "/tmp/pip-build-env-m2udz7l/overlay/local/lib/python3.12/dist-packages/setuptools/command/sdist.py", line 102, in add_defaults
super().add_defaults()
File "/tmp/pip-build-env-m2udz7l/overlay/local/lib/python3.12/dist-packages/setuptools/_distutils/command/sdist.py", line 250, in add_defaults
self.add_defaults_ext()
File "/tmp/pip-build-env-m2udz7l/overlay/local/lib/python3.12/dist-packages/setuptools/_distutils/command/sdist.py", line 335, in add_defaults_ext
self.filelist.extend(build_ext.get_source_files())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 204, in get_source_files
File "/tmp/pip-build-env-m2udz7l/overlay/local/lib/python3.12/dist-packages/setuptools/_distutils/cmd.py", line 107, in getattr
raise AttributeError(attr)
AttributeError: cython_sources
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[dchard]

Going back to 1.0.2 from master does resolve this issue.

[9600]

I'm getting the same error on Ubuntu 24.04 with release tag 1.0.2, when installing the dependencies inside a venv created with python3 -m venv venv.

[9600]

So I resolved the PyYAML build error on Ubuntu 24.04 / Python 3.12 by bumping the version in requirements.txt to PyYAML==6.0.1.

I then had a similar failure with aiohttp, which was resolved by bumping this to aiohttp==3.9.3. Appears that the relevant fix went into 3.9.0, but figured to go with the last 3.9.x release (it may work with current, which is 3.10.10, but I haven't tested).

Looks as though these issues stem from Python moving to Cython 3.0 and some modules needing to be updated to build with this.

[ashokhs77]

I was using Ubuntu 20.04 with python 3.12 and I had issues with Redis while running services (getting error for messagingAsynch on Redis) so upgraded to Ubuntu 24.04 and with python 3.12 getting the same errors with requirements.txt.

Have tried all that is mentioned here (bumping the versions of certain packages, moving to pyhss ver 1.0.2 etc)

But nothing has helped in in resolving the errors with requirements installation. Any pointers here..?

Defaulting to user installation because normal site-packages is not writeable
Collecting aiohttp==3.9.3 (from -r requirements.txt (line 1))
Using cached aiohttp-3.9.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.4 kB)
Collecting Flask==2.2.3 (from -r requirements.txt (line 2))
Using cached Flask-2.2.3-py3-none-any.whl.metadata (3.9 kB)
Collecting flask_restx==1.1.0 (from -r requirements.txt (line 3))
Using cached flask_restx-1.1.0-py2.py3-none-any.whl.metadata (9.0 kB)
Requirement already satisfied: Jinja2==3.1.2 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 4)) (3.1.2)
Collecting mongo==0.2.0 (from -r requirements.txt (line 5))
Using cached mongo-0.2.0.tar.gz (4.6 kB)
Preparing metadata (setup.py) ... done
Collecting prometheus_client==0.16.0 (from -r requirements.txt (line 6))
Using cached prometheus_client-0.16.0-py3-none-any.whl.metadata (22 kB)
Collecting pycryptodome==3.17 (from -r requirements.txt (line 7))
Using cached pycryptodome-3.17-cp35-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.3 kB)
Collecting pymongo==4.3.3 (from -r requirements.txt (line 8))
Using cached pymongo-4.3.3.tar.gz (814 kB)
Preparing metadata (setup.py) ... done
Collecting pysctp==0.7.2 (from -r requirements.txt (line 9))
Using cached pysctp-0.7.2.tar.gz (32 kB)
Preparing metadata (setup.py) ... done
Collecting pysnmp==4.4.12 (from -r requirements.txt (line 10))
Using cached pysnmp-4.4.12-py2.py3-none-any.whl.metadata (1.8 kB)
Requirement already satisfied: PyYAML==6.0.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 11)) (6.0.1)
Collecting redis==5.0.0 (from -r requirements.txt (line 12))
Using cached redis-5.0.0-py3-none-any.whl.metadata (8.8 kB)
Requirement already satisfied: Requests==2.31.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 13)) (2.31.0)
Collecting SQLAlchemy==2.0.9 (from -r requirements.txt (line 14))
Using cached SQLAlchemy-2.0.9-py3-none-any.whl.metadata (9.3 kB)
Collecting SQLAlchemy_Utils==0.41.1 (from -r requirements.txt (line 15))
Using cached SQLAlchemy_Utils-0.41.1-py3-none-any.whl.metadata (4.3 kB)
Collecting Werkzeug==2.2.3 (from -r requirements.txt (line 16))
Using cached Werkzeug-2.2.3-py3-none-any.whl.metadata (4.4 kB)
Collecting mysqlclient (from -r requirements.txt (line 17))
Using cached mysqlclient-2.2.7.tar.gz (91 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [29 lines of output]
Trying pkg-config --exists mysqlclient
Command 'pkg-config --exists mysqlclient' returned non-zero exit status 1.
Trying pkg-config --exists mariadb
Command 'pkg-config --exists mariadb' returned non-zero exit status 1.
Trying pkg-config --exists libmariadb
Command 'pkg-config --exists libmariadb' returned non-zero exit status 1.
Trying pkg-config --exists perconaserverclient
Command 'pkg-config --exists perconaserverclient' returned non-zero exit status 1.
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in
main()
File "/usr/lib/python3/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-j6vtu9f0/overlay/local/lib/python3.12/dist-packages/setuptools/build_meta.py", line 334, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=[])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-j6vtu9f0/overlay/local/lib/python3.12/dist-packages/setuptools/build_meta.py", line 304, in _get_build_requires
self.run_setup()
File "/tmp/pip-build-env-j6vtu9f0/overlay/local/lib/python3.12/dist-packages/setuptools/build_meta.py", line 320, in run_setup
exec(code, locals())
File "", line 156, in
File "", line 49, in get_config_posix
File "", line 28, in find_package_name
Exception: Can not find valid pkg-config name.
Specify MYSQLCLIENT_CFLAGS and MYSQLCLIENT_LDFLAGS env vars manually
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[ashokhs77]

Got this issue resolved by installing libmysqlclient-dev (by running sudo apt-get install libmysqlclient-dev). I tried this on Ubuntu 22.04.