BUG: Pyzmq , _zmq.cpython-313.so does not contain a CELQSTRT CSECT.

[AnkitAhlawat7742]

This is a pyzmq bug

• This is a pyzmq-specific bug, not an issue of zmq socket behavior. Don't worry if you're not sure! We'll figure it out together.

What pyzmq version?

26.2.1

What libzmq version?

4.3.5

Python version (and how it was installed)

Python 3.13.0

OS

zOS

What happened?

Hi Everyone,

What pyzmq version?
26.2.1

What libzmq version?
4.3.5

Python version (and how it was installed)
3.13.0

OS
zOS

I am in the process of building the PyZMQ package on the z/OS platform. The libzmq with version 4.3.5 library has been built ,it is a static lib and added to the correct path. The PyZMQ version being used is 26.2.1.

So far, I have been able to successfully build and install the package:

(jenkins-build-env-313) AQ /u/pyzoda/share/jenkins/jenkins-build-env-313/pyzmq> pwd
/u/pyzoda/share/jenkins/jenkins-build-env-313/pyzmq
(jenkins-build-env-313) AQ /u/pyzoda/share/jenkins/jenkins-build-env-313/pyzmq> pip install -e .
Obtaining file:///u/pyzoda/share/jenkins/jenkins-build-env-313/pyzmq
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build editable ... done
Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: pyzmq
Building editable for pyzmq (pyproject.toml) ... done
Created wheel for pyzmq: filename=pyzmq-26.2.1-cp313-cp313-os390_29_00_3931.whl size=8702932 sha256=9e92877a79401ea381fe5fa0f6c96ea2c8a0d317e18c1c86e1bc2718be5fb383
Stored in directory: /AQFT/tmp/pip-ephem-wheel-cache-ig5ho4sr/wheels/93/c7/4d/45e4bb5077d26369c0d66fe6154fda537f714c6516ad203e71
Successfully built pyzmq
Installing collected packages: pyzmq
Attempting uninstall: pyzmq
Found existing installation: pyzmq 26.2.1
Uninstalling pyzmq-26.2.1:
Successfully uninstalled pyzmq-26.2.1
Successfully installed pyzmq-26.2.1

However, when I run the following command:
pytest tests/

I encounter the following error:

(jenkins-build-env-313) AQ /u/pyzoda/share/jenkins/jenkins-build-env-313/pyzmq> pytest tests/
ImportError while loading conftest '/u/pyzoda/share/jenkins/jenkins-build-env-313/pyzmq/tests/conftest.py'.
tests/conftest.py:23: in
import zmq
zmq/init.py:109: in
from zmq import backend
zmq/backend/init.py:30: in
raise original_error from None
zmq/backend/init.py:25: in
_ns = select_backend(first)
zmq/backend/select.py:31: in select_backend
mod = import_module(name)
zmq/backend/cython/init.py:6: in
from . import _zmq
E ImportError: CEE3595S DLL /u/pyzoda/share/jenkins/jenkins-build-env-313/lib/python3.13/site-packages/zmq/backend/cython/_zmq.cpython-313.so does not contain a CELQSTRT CSECT.
I see we have already install cython in my venv

I'm not sure why the _zmq.cpython-313.so file is causing this error. Does anyone have insights on why this might be happening?

Code to reproduce bug

Traceback, if applicable

More info

No response

[AnkitAhlawat7742]

we can see when i am importing i am getting below error
(venv13_ankit) /u/pyzoda/ankit/venv13_ankit/pyzmq> python3.13 -c "import zmq; print(zmq.version)"
Traceback (most recent call last):
File "", line 1, in
import zmq; print(zmq.version)
^^^^^^^^^^
File "/u/pyzoda/ankit/venv13_ankit/pyzmq/zmq/init.py", line 109, in
from zmq import backend
File "/u/pyzoda/ankit/venv13_ankit/pyzmq/zmq/backend/init.py", line 30, in
raise original_error from None
File "/u/pyzoda/ankit/venv13_ankit/pyzmq/zmq/backend/init.py", line 25, in
_ns = select_backend(first)
File "/u/pyzoda/ankit/venv13_ankit/pyzmq/zmq/backend/select.py", line 31, in select_backend
mod = import_module(name)
File "/u/pyzoda/share/sdk/usr/lpp/IBM/cyp/v3r13/pyz/lib/python3.13/importlib/init.py", line 88, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/u/pyzoda/ankit/venv13_ankit/pyzmq/zmq/backend/cython/init.py", line 6, in
from . import _zmq
ImportError: CEE3595S DLL /u/pyzoda/ankit/venv13_ankit/lib/python3.13/site-packages/zmq/backend/cython/_zmq.cpython-313.so does not contain a CELQSTRT CSECT.

[AnkitAhlawat7742]

@minrk
can you please help me here

[minrk]

I have no idea what CELQSTRT CSECT is, it seems to be something about the z/OS compilers, which I've never encountered before. Try making sure to do verbose compilation:

pip install -v pyzmq -C build.verbose=true

and see if you can build other python extensions (numpy, astyle etc.)

[AnkitAhlawat7742]

Yes i am using zOS
with option -v i am getting below output of build command

End of search list.
[3/3] Linking C shared module _zmq.cpython-313.so
IBM Open XL C/C++ 1.1 for z/OS, (5650-ZOS), version 1.1.0.3, clang version 14.0.0 (build 37a9321)
Target: s390x-ibm-zos
Thread model: posix
InstalledDir: /usr/lpp/IBM/cnw/v1r1/openxl/bin
Customization file: /u/pyzoda/ankit/profiles/clang_v2r4.cfg
Configuration file: /u/pyzoda/ankit/profiles/clang_v2r4.cfg
"/bin/ld" -o _zmq.cpython-313.so -b AMODE=64,LIST,DYNAM=DLL,MSGLEVEL=4,CASE=MIXED,REUS=RENT -e CELQSTRT -O CELQSTRT -u CELQMAIN -x _zmq.cpython-313.x -L/usr/lpp/IBM/cnw/v1r1/openxl/lib/s390x-ibm-zos/ -L/u/pyzoda/share/sdk/usr/lpp/IBM/cyp/v3r13/pyz/lib/libpython3.13.so -L/PRIPKC/usr/lpp/IBM/cnw/v1r1/openxl/bin/../lib/s390x-ibm-zos -lc++ -lc++_a -lc++_exception -lc++abi -lc++abixl -lunwind CMakeFiles/_zmq.dir/_src/_zmq.c.o -L /usr/lpp/IBM/cnw/v1r1/openxl/bin/../lib/64 -lc++ -lc++_a -lc++_exception -lc++abi -lc++abixl -lunwind /u/pyzoda/share/sdk/usr/lpp/IBM/cyp/v3r13/pyz/lib/libpython3.13.so /u/pyzoda/share/jenkins/workspace/ies_libzmq_zos-dependency_v4.3.4/fromtarball/zeromq-4.3.4/lib/libzmq.a -S "//'SYS1.CEE.SCEEBND2'" -S "//'SYS1.CSSLIB'" "//'SYS1.CEE.SCEELIB(CELQS001)'" "//'SYS1.CEE.SCEELIB(CELQS003)'" /PRIPKC/usr/lpp/IBM/cnw/v1r1/openxl/lib/clang/14.0.0/lib/s390x-ibm-zos/libclang_rt.builtins.a
*** Installing project into wheel...
-- Install configuration: "Release"
-- Installing: /tmp/tmpo6r6yh9k/wheel/platlib/zmq/backend/cython/_zmq.cpython-313.so
*** Making wheel...
*** Created pyzmq-26.2.1-cp313-cp313-os390_29_00_3931.whl
Successfully built pyzmq-26.2.1-cp313-cp313-os390_29_00_3931.whl

any idea why we are able to build _zmq.cpython-313.so ? if we have any issue in building or linking

[AnkitAhlawat7742]

@minrk
Now i ma getting this error
(venv13_ankit) /u/pyzoda/ankit/venv13_ankit/pyzmq> pytest --pyargs zmq.tests
/u/pyzoda/ankit/venv13_ankit/lib/python3.13/site-packages/_pytest/config/init.py:324: PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
Plugin: helpconfig, Hook: pytest_cmdline_parse
ConftestImportFailure: ImportError: Importing zmq.backend.cython failed with initialization of _zmq did not return an extension module (from /u/pyzoda/ankit/venv13_ankit/pyzmq/tests/conftest.py)
For more information see https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning
config = pluginmanager.hook.pytest_cmdline_parse(
ImportError while loading conftest '/u/pyzoda/ankit/venv13_ankit/pyzmq/tests/conftest.py'.
tests/conftest.py:23: in
import zmq
zmq/init.py:109: in
from zmq import backend
zmq/backend/init.py:15: in
_ns = select_backend(backend)
zmq/backend/select.py:35: in select_backend
raise ImportError(f"Importing {name} failed with {e}") from e
E ImportError: Importing zmq.backend.cython failed with initialization of _zmq did not return an extension module

Something is wrong which i am not getting

[minrk]

that last error suggests the extension is not built or not installed, and the original source file is imported. That would happen e.g. if you did a standard install and then ran tests from within the source directory.

You can avoid this by not running tests from within the source directory.

[AnkitAhlawat7742]

@minrk
I downgraded the pyzmq version to 24.0.1, and I was able to successfully build the package on zOS with python 3.13 and install the wheel file. After reviewing the pytest strategy mentioned in this issue, I ran pytest --pyargs zmq.tests after exiting the pyzmq folder.
but now somehow pytest got stuck into this file test
(venv13_ankit) /u/pyzoda/ankit/venv13_ankit> pytest --pyargs zmq.tests
=========================================================================== test session starts ===========================================================================
platform zos -- Python 3.13.0, pytest-7.1.2, pluggy-1.5.0
rootdir: /u/pyzoda/ankit, configfile: pytest.ini
plugins: anyio-4.6.0.post0, mock-3.14.1.dev75+g9c18793, flaky-3.8.1, cov-6.0.0
collected 357 items / 1 skipped

lib/python3.13/site-packages/zmq/tests/test_asyncio.py sssssss.........F............sssssss [ 10%]
lib/python3.13/site-packages/zmq/tests/test_auth.py sssssssssssssssss [ 14%]
lib/python3.13/site-packages/zmq/tests/test_cffi_backend.py ssssssssssssss [ 18%]
lib/python3.13/site-packages/zmq/tests/test_constants.py .......................................... [ 30%]
lib/python3.13/site-packages/zmq/tests/test_context.py ....................s...... [ 38%]
lib/python3.13/site-packages/zmq/tests/test_cython.py FF [ 38%]
lib/python3.13/site-packages/zmq/tests/test_decorators.py ........................... [ 46%]
lib/python3.13/site-packages/zmq/tests/test_device.py ........ [ 48%]
lib/python3.13/site-packages/zmq/tests/test_draft.py ss [ 49%]
lib/python3.13/site-packages/zmq/tests/test_error.py ... [ 49%]
lib/python3.13/site-packages/zmq/tests/test_etc.py .ss [ 50%]
lib/python3.13/site-packages/zmq/tests/test_ext.py . [ 50%]
lib/python3.13/site-packages/zmq/tests/test_future.py .................... [ 56%]
lib/python3.13/site-packages/zmq/tests/test_imports.py ..........s.s.s.s.s.s.s [ 63%]
lib/python3.13/site-packages/zmq/tests/test_includes.py .. [ 63%]
ib/python3.13/site-packages/zmq/tests/test_future.py::TestFutureSocket::test_send_noblock PASSED [ 56%]
lib/python3.13/site-packages/zmq/tests/test_future.py::TestFutureSocket::test_send_timeout PASSED [ 56%]
lib/python3.13/site-packages/zmq/tests/test_future.py::TestFutureSocket::test_socket_class PASSED [ 56%]
lib/python3.13/site-packages/zmq/tests/test_imports.py::test_toplevel PASSED [ 56%]
lib/python3.13/site-packages/zmq/tests/test_imports.py::test_core PASSED [ 57%]
lib/python3.13/site-packages/zmq/tests/test_imports.py::test_devices PASSED [ 57%]
lib/python3.13/site-packages/zmq/tests/test_imports.py::test_log PASSED [ 57%]
lib/python3.13/site-packages/zmq/tests/test_imports.py::test_eventloop PASSED [ 57%]
lib/python3.13/site-packages/zmq/tests/test_imports.py::test_utils PASSED [ 58%]
lib/python3.13/site-packages/zmq/tests/test_imports.py::test_ssh PASSED [ 58%]
lib/python3.13/site-packages/zmq/tests/test_imports.py::test_decorators PASSED [ 58%]
lib/python3.13/site-packages/zmq/tests/test_imports.py::test_zmq_all PASSED [ 59%]
lib/python3.13/site-packages/zmq/tests/test_imports.py::test_all_exports[RCVTIMEO-zmq] PASSED [ 59%]
lib/python3.13/site-packages/zmq/tests/test_imports.py::test_all_exports[RCVTIMEO-zmq.green] SKIPPED (could not import 'zmq.green': No module named 'gevent') [ 59%]
lib/python3.13/site-packages/zmq/tests/test_imports.py::test_all_exports[PUSH-zmq] PASSED [ 59%]
lib/python3.13/site-packages/zmq/tests/test_imports.py::test_all_exports[PUSH-zmq.green] SKIPPED (could not import 'zmq.green': No module named 'gevent') [ 60%]
lib/python3.13/site-packages/zmq/tests/test_imports.py::test_all_exports[zmq_version_info-zmq] PASSED [ 60%]
lib/python3.13/site-packages/zmq/tests/test_imports.py::test_all_exports[zmq_version_info-zmq.green] SKIPPED (could not import 'zmq.green': No module named 'ge...) [ 60%]
lib/python3.13/site-packages/zmq/tests/test_imports.py::test_all_exports[SocketOption-zmq] PASSED [ 61%]
lib/python3.13/site-packages/zmq/tests/test_imports.py::test_all_exports[SocketOption-zmq.green] SKIPPED (could not import 'zmq.green': No module named 'gevent') [ 61%]
lib/python3.13/site-packages/zmq/tests/test_imports.py::test_all_exports[device-zmq] PASSED [ 61%]
lib/python3.13/site-packages/zmq/tests/test_imports.py::test_all_exports[device-zmq.green] SKIPPED (could not import 'zmq.green': No module named 'gevent') [ 61%]
lib/python3.13/site-packages/zmq/tests/test_imports.py::test_all_exports[Socket-zmq] PASSED [ 62%]
lib/python3.13/site-packages/zmq/tests/test_imports.py::test_all_exports[Socket-zmq.green] SKIPPED (could not import 'zmq.green': No module named 'gevent') [ 62%]
lib/python3.13/site-packages/zmq/tests/test_imports.py::test_all_exports[Context-zmq] PASSED [ 62%]
lib/python3.13/site-packages/zmq/tests/test_imports.py::test_all_exports[Context-zmq.green] SKIPPED (could not import 'zmq.green': No module named 'gevent') [ 63%]
lib/python3.13/site-packages/zmq/tests/test_includes.py::TestIncludes::test_get_includes PASSED [ 63%]
lib/python3.13/site-packages/zmq/tests/test_includes.py::TestIncludes::test_get_library_dirs PASSED [ 63%]
lib/python3.13/site-packages/zmq/tests/test_ioloop.py::TestIOLoop::test_close_all PASSED [ 63%]
lib/python3.13/site-packages/zmq/tests/test_ioloop.py::TestIOLoop::test_current PASSED [ 64%]
lib/python3.13/site-packages/zmq/tests/test_ioloop.py::TestIOLoop::test_instance PASSED [ 64%]
lib/python3.13/site-packages/zmq/tests/test_ioloop.py::TestIOLoop::test_simple PASSED [ 64%]
lib/python3.13/site-packages/zmq/tests/test_log.py::TestPubLog::test_blank_root_topic. ---->Got stuck here

what could be the cause of this hang ?

[AnkitAhlawat7742]

and To test , why we can not do simple pytest . for this package pyzmq ?

[AnkitAhlawat7742]

Please help me here @minrk and team

[minrk]

I'm afraid there's not too much I can help with. You can skip the test, if it lets you move on. But since z/os is proprietary, I have no real way of investigating the problem.

I wouldn't generally expect outdated pyzmq to work on the very latest Python 3.13, but it appears there is a problem with the new build system introduced in pyzmq 26. Does pyzmq 25.1.2 work? pyzmq 25 is the first version to support Python 3.12. That would help narrow down the relevant change.

[AnkitAhlawat7742]

@minrk I am currently using Python 3.13.0. I'll check with version 25.1.2 and
hope that this package supports Python 3.13.0.?

[AnkitAhlawat7742]

Yeah..!! after commenting out the specific test case (only one) i am able to run my test case's