unable to install splinecalib

[SimpleCoder35]

I am unable to install splinecalib. There was a major update to numPy yesterday, and the package is attempting to install it as it is specified in the pyproject.toml

[build-system]
requires = ["setuptools", "setuptools-scm", "numpy"]
build-backend = "setuptools.build_meta"

!pip install splinecalib --verbose --no-deps

Using pip 24.0 from /usr/local/lib/python3.10/dist-packages/pip (python 3.10)
Collecting splinecalib
Downloading splinecalib-0.0.7.tar.gz (1.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 10.6 MB/s eta 0:00:00
Running command pip subprocess to install build dependencies
Using pip 24.0 from /usr/local/lib/python3.10/dist-packages/pip (python 3.10)
Collecting setuptools
Obtaining dependency information for setuptools from https://files.pythonhosted.org/packages/de/88/70c5767a0e43eb4451c2200f07d042a4bcd7639276003a9c54a68cfcc1f8/setuptools-70.0.0-py3-none-any.whl.metadata
Using cached setuptools-70.0.0-py3-none-any.whl.metadata (5.9 kB)
Collecting setuptools-scm
Obtaining dependency information for setuptools-scm from https://files.pythonhosted.org/packages/a0/b9/1906bfeb30f2fc13bb39bf7ddb8749784c05faadbd18a21cf141ba37bff2/setuptools_scm-8.1.0-py3-none-any.whl.metadata
Downloading setuptools_scm-8.1.0-py3-none-any.whl.metadata (6.6 kB)
Collecting numpy
Obtaining dependency information for numpy from https://files.pythonhosted.org/packages/d6/a8/6a2419c40c7b6f7cb4ef52c532c88e55490c4fa92885964757d507adddce/numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
Downloading numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.9/60.9 kB 2.7 MB/s eta 0:00:00
Collecting packaging>=20 (from setuptools-scm)
Obtaining dependency information for packaging>=20 from https://files.pythonhosted.org/packages/08/aa/cc0199a5f0ad350994d660967a8efb233fe0416e4639146c089643407ce6/packaging-24.1-py3-none-any.whl.metadata
Using cached packaging-24.1-py3-none-any.whl.metadata (3.2 kB)
Collecting tomli>=1 (from setuptools-scm)
Obtaining dependency information for tomli>=1 from https://files.pythonhosted.org/packages/97/75/10a9ebee3fd790d20926a90a2547f0bf78f371b2f13aa822c759680ca7b9/tomli-2.0.1-py3-none-any.whl.metadata
Using cached tomli-2.0.1-py3-none-any.whl.metadata (8.9 kB)
Using cached setuptools-70.0.0-py3-none-any.whl (863 kB)
Downloading setuptools_scm-8.1.0-py3-none-any.whl (43 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.7/43.7 kB 3.1 MB/s eta 0:00:00
Downloading numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (19.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.3/19.3 MB 74.5 MB/s eta 0:00:00
Using cached packaging-24.1-py3-none-any.whl (53 kB)
Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
Installing collected packages: tomli, setuptools, packaging, numpy, setuptools-scm
Creating /tmp/pip-build-env-13p0zqnj/overlay/local/bin
changing mode of /tmp/pip-build-env-13p0zqnj/overlay/local/bin/f2py to 755
changing mode of /tmp/pip-build-env-13p0zqnj/overlay/local/bin/numpy-config to 755
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
ipython 7.34.0 requires jedi>=0.16, which is not installed.
astropy 5.3.4 requires numpy<2,>=1.21, but you have numpy 2.0.0 which is incompatible.
cudf-cu12 24.4.1 requires numpy<2.0a0,>=1.23, but you have numpy 2.0.0 which is incompatible.
cupy-cuda12x 12.2.0 requires numpy<1.27,>=1.20, but you have numpy 2.0.0 which is incompatible.
ibis-framework 8.0.0 requires numpy<2,>=1, but you have numpy 2.0.0 which is incompatible.
numba 0.58.1 requires numpy<1.27,>=1.22, but you have numpy 2.0.0 which is incompatible.
rmm-cu12 24.4.0 requires numpy<2.0a0,>=1.23, but you have numpy 2.0.0 which is incompatible.
scipy 1.11.4 requires numpy<1.28.0,>=1.21.6, but you have numpy 2.0.0 which is incompatible.
tensorflow 2.15.0 requires numpy<2.0.0,>=1.23.5, but you have numpy 2.0.0 which is incompatible.
Successfully installed numpy-2.0.0 packaging-24.1 setuptools-70.0.0 setuptools-scm-8.1.0 tomli-2.0.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Installing build dependencies ... done
Running command Getting requirements to build wheel
running egg_info
writing splinecalib.egg-info/PKG-INFO
writing dependency_links to splinecalib.egg-info/dependency_links.txt
writing requirements to splinecalib.egg-info/requires.txt
writing top-level names to splinecalib.egg-info/top_level.txt
ERROR setuptools_scm._file_finders.git listing git files failed - pretending there aren't any
reading manifest file 'splinecalib.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'splinecalib.egg-info/SOURCES.txt'
Getting requirements to build wheel ... done
Running command pip subprocess to install backend dependencies
Using pip 24.0 from /usr/local/lib/python3.10/dist-packages/pip (python 3.10)
Collecting wheel
Obtaining dependency information for wheel from https://files.pythonhosted.org/packages/7d/cd/d7460c9a869b16c3dd4e1e403cce337df165368c71d6af229a74699622ce/wheel-0.43.0-py3-none-any.whl.metadata
Using cached wheel-0.43.0-py3-none-any.whl.metadata (2.2 kB)
Using cached wheel-0.43.0-py3-none-any.whl (65 kB)
Installing collected packages: wheel
Creating /tmp/pip-build-env-13p0zqnj/normal/local/bin
changing mode of /tmp/pip-build-env-13p0zqnj/normal/local/bin/wheel to 755
Successfully installed wheel-0.43.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Installing backend dependencies ... done
Running command Preparing metadata (pyproject.toml)
running dist_info
creating /tmp/pip-modern-metadata-n6nkp1n9/splinecalib.egg-info
writing /tmp/pip-modern-metadata-n6nkp1n9/splinecalib.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-modern-metadata-n6nkp1n9/splinecalib.egg-info/dependency_links.txt
writing requirements to /tmp/pip-modern-metadata-n6nkp1n9/splinecalib.egg-info/requires.txt
writing top-level names to /tmp/pip-modern-metadata-n6nkp1n9/splinecalib.egg-info/top_level.txt
writing manifest file '/tmp/pip-modern-metadata-n6nkp1n9/splinecalib.egg-info/SOURCES.txt'
ERROR setuptools_scm._file_finders.git listing git files failed - pretending there aren't any
reading manifest file '/tmp/pip-modern-metadata-n6nkp1n9/splinecalib.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file '/tmp/pip-modern-metadata-n6nkp1n9/splinecalib.egg-info/SOURCES.txt'
creating '/tmp/pip-modern-metadata-n6nkp1n9/splinecalib-0.0.7.dist-info'
Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: splinecalib
Running command Building wheel for splinecalib (pyproject.toml)
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-310
creating build/lib.linux-x86_64-cpython-310/splinecalib
copying splinecalib/init.py -> build/lib.linux-x86_64-cpython-310/splinecalib
copying splinecalib/calib_utils.py -> build/lib.linux-x86_64-cpython-310/splinecalib
copying splinecalib/splinecalib.py -> build/lib.linux-x86_64-cpython-310/splinecalib
running build_ext
building 'loss_fun_c' extension
creating build/temp.linux-x86_64-cpython-310
creating build/temp.linux-x86_64-cpython-310/splinecalib
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/tmp/pip-build-env-13p0zqnj/overlay/local/lib/python3.10/dist-packages/numpy/_core/include -I/usr/include/python3.10 -c splinecalib/loss_fun_c.c -o build/temp.linux-x86_64-cpython-310/splinecalib/loss_fun_c.o
In file included from /tmp/pip-build-env-13p0zqnj/overlay/local/lib/python3.10/dist-packages/numpy/_core/include/numpy/ndarraytypes.h:1909,
from /tmp/pip-build-env-13p0zqnj/overlay/local/lib/python3.10/dist-packages/numpy/_core/include/numpy/ndarrayobject.h:12,
from /tmp/pip-build-env-13p0zqnj/overlay/local/lib/python3.10/dist-packages/numpy/_core/include/numpy/arrayobject.h:5,
from splinecalib/loss_fun_c.c:1154:
/tmp/pip-build-env-13p0zqnj/overlay/local/lib/python3.10/dist-packages/numpy/_core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
17 | #warning "Using deprecated NumPy API, disable it with "
| ^~~~~~~
In file included from /usr/include/python3.10/Python.h:74,
from splinecalib/loss_fun_c.c:16:
splinecalib/loss_fun_c.c: In function ‘__pyx_f_5numpy_PyDataType_SHAPE’:
splinecalib/loss_fun_c.c:3539:39: error: ‘PyArray_Descr’ {aka ‘struct _PyArray_Descr’} has no member named ‘subarray’
3539 | __Pyx_INCREF(((PyObject*)__pyx_v_d->subarray->shape));
| ^~
/usr/include/python3.10/object.h:112:41: note: in definition of macro ‘_PyObject_CAST’
112 | #define _PyObject_CAST(op) ((PyObject*)(op))
| ^~
splinecalib/loss_fun_c.c:1796:27: note: in expansion of macro ‘Py_INCREF’
1796 | #define __Pyx_INCREF(r) Py_INCREF(r)
| ^~~~~~~~~
splinecalib/loss_fun_c.c:3539:5: note: in expansion of macro ‘__Pyx_INCREF’
3539 | __Pyx_INCREF(((PyObject*)__pyx_v_d->subarray->shape));
| ^~~~~~~~~~~~
splinecalib/loss_fun_c.c:3540:36: error: ‘PyArray_Descr’ {aka ‘struct _PyArray_Descr’} has no member named ‘subarray’
3540 | __pyx_r = ((PyObject*)__pyx_v_d->subarray->shape);
| ^~
error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
error: subprocess-exited-with-error

× Building wheel for splinecalib (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /usr/bin/python3 /usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmplmwqmkp1
cwd: /tmp/pip-install-ncs4wl02/splinecalib_cede067df01b4261b1685661973e52d3
Building wheel for splinecalib (pyproject.toml) ... error
ERROR: Failed building wheel for splinecalib
Failed to build splinecalib
ERROR: Could not build wheels for splinecalib, which is required to install pyproject.toml-based projects

[blucena]

@SimpleCoder35 Thanks for alerting me to this. From what I can tell, the problem lies in finding the right version of numpy to fit not just splinecalib, but all the other packages you already have installed. A couple of ideas to try:

1. Try installing a specific version of numpy first. Based on the error message you might try a version >=1.23.5 but <2. That should be compatible with everything. Then when you install splinecalib, it will see the version of numpy you already have, and not attempt to install a new one.

2. Another thing you might try is installing without the --no-deps flag. Just pip install splinecalib. I'm not sure how the dependency resolver in pip works when you have the --no-deps flag, so that might be causing problems as well.

Let me know if this helps you get it installed.

[SimpleCoder35]

@blucena , Thanks for the response.

I was trying without any dependencies, This is what I am trying

!pip install splinecalib --verbose --no-deps

but for some reason looks like pip is collecting the numpy 2.0 version.(from the log)

I have Numpy already installed(1.24.4), I froze the environment exactly to the previous working version. Still unable to install the package.

[SimpleCoder35]

@blucena ,
update: I was able to install this package by modifying the setup.py
added <2.0

install_requires=[
    "numpy>=1.16,<2.0",
    "scipy>=1.3"],

[dcturner45]

I am seeing this issue as well. On macOS Sonoma 14.5 (if that matters), in a fresh conda environment with Python 3.9.19:

shell> conda create -y --name sctest python=3.9
shell> conda activate sctest
(sctest) shell> python --version
Python 3.9.19
(sctest) shell> pip install numpy==1.26.4 splinecalib
...
      splinecalib/loss_fun_c.c:3539:41: error: no member named 'subarray' in 'struct _PyArray_Descr'
          __Pyx_INCREF(((PyObject*)__pyx_v_d->subarray->shape));
                                   ~~~~~~~~~  ^
      splinecalib/loss_fun_c.c:1796:37: note: expanded from macro '__Pyx_INCREF'
        #define __Pyx_INCREF(r) Py_INCREF(r)
                                          ^
      /opt/homebrew/Caskroom/miniconda/base/envs/sctest/include/python3.9/object.h:411:49: note: expanded from macro 'Py_INCREF'
      #define Py_INCREF(op) _Py_INCREF(_PyObject_CAST(op))
                                                      ^~
      /opt/homebrew/Caskroom/miniconda/base/envs/sctest/include/python3.9/object.h:112:41: note: expanded from macro '_PyObject_CAST'
      #define _PyObject_CAST(op) ((PyObject*)(op))
                                              ^~
      splinecalib/loss_fun_c.c:3540:38: error: no member named 'subarray' in 'struct _PyArray_Descr'
          __pyx_r = ((PyObject*)__pyx_v_d->subarray->shape);
                                ~~~~~~~~~  ^
      1 warning and 2 errors generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

If explicitly setting an upper bound on numpy in install_requires fixes the issue (as mentioned here by @SimpleCoder35), is setuptools somehow pulling a different version of numpy than the one that's installed in the environment?

[blucena]

@dcturner45 Thanks for posting this. Not sure right now whether this is the same issue that @SimpleCoder35 faced or not. Yours seems to be related to the C-compilation in the Cython code and not necessarily the new version of numpy (esp since you pin the old version of numpy)

Let me do some investigation now and see if I can find a solution

[Jennie-Myhren]

I've also been seeing this issue (running a docker container w/ a Python v3.12 image). I've tried pinning the numpy version and setting an upper bound (project uses pyproject.toml, but I've also tried the solution above with install_requires in a setup.py file and tried installing numpy and/or splinecalib directly in the Dockerfile.)

creating build/temp.linux-x86_64-cpython-312/splinecalib
#13 307.8   gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -fPIC -I/tmp/tmpl3xgatdv/.venv/lib/python3.12/site-packages/numpy/_core/include -I/tmp/tmpl3xgatdv/.venv/include -I/usr/local/include/python3.12 -c splinecalib/loss_fun_c.c -o build/temp.linux-x86_64-cpython-312/splinecalib/loss_fun_c.o
#13 307.8   In file included from /tmp/tmpl3xgatdv/.venv/lib/python3.12/site-packages/numpy/_core/include/numpy/ndarraytypes.h:1909,
#13 307.8                    from /tmp/tmpl3xgatdv/.venv/lib/python3.12/site-packages/numpy/_core/include/numpy/ndarrayobject.h:12,
#13 307.8                    from /tmp/tmpl3xgatdv/.venv/lib/python3.12/site-packages/numpy/_core/include/numpy/arrayobject.h:5,
#13 307.8                    from splinecalib/loss_fun_c.c:1154:
#13 307.8   /tmp/tmpl3xgatdv/.venv/lib/python3.12/site-packages/numpy/_core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#13 307.8      17 | #warning "Using deprecated NumPy API, disable it with " \
#13 307.8         |  ^~~~~~~
#13 307.8   In file included from /usr/local/include/python3.12/Python.h:38,
#13 307.8                    from splinecalib/loss_fun_c.c:16:
#13 307.8   splinecalib/loss_fun_c.c: In function ‘__pyx_f_5numpy_PyDataType_SHAPE’:
#13 307.8   splinecalib/loss_fun_c.c:3539:39: error: ‘PyArray_Descr’ {aka ‘struct _PyArray_Descr’} has no member named ‘subarray’
#13 307.8    3539 |     __Pyx_INCREF(((PyObject*)__pyx_v_d->subarray->shape));
#13 307.8         |                                       ^~
#13 307.8   /usr/local/include/python3.12/pyport.h:24:38: note: in definition of macro ‘_Py_CAST’
#13 307.8      24 | #define _Py_CAST(type, expr) ((type)(expr))
#13 307.8         |                                      ^~~~
#13 307.8   /usr/local/include/python3.12/object.h:661:35: note: in expansion of macro ‘_PyObject_CAST’
#13 307.8     661 | #  define Py_INCREF(op) Py_INCREF(_PyObject_CAST(op))
#13 307.8         |                                   ^~~~~~~~~~~~~~
#13 307.8   splinecalib/loss_fun_c.c:1796:27: note: in expansion of macro ‘Py_INCREF’
#13 307.8    1796 |   #define __Pyx_INCREF(r) Py_INCREF(r)
#13 307.8         |                           ^~~~~~~~~
#13 307.8   splinecalib/loss_fun_c.c:3539:5: note: in expansion of macro ‘__Pyx_INCREF’
#13 307.8    3539 |     __Pyx_INCREF(((PyObject*)__pyx_v_d->subarray->shape));
#13 307.8         |     ^~~~~~~~~~~~
#13 307.8   splinecalib/loss_fun_c.c:3540:36: error: ‘PyArray_Descr’ {aka ‘struct _PyArray_Descr’} has no member named ‘subarray’
#13 307.8    3540 |     __pyx_r = ((PyObject*)__pyx_v_d->subarray->shape);
#13 307.8         |                                    ^~
#13 307.8   error: command '/usr/bin/gcc' failed with exit code 1
#13 307.8   
#13 307.8 
#13 307.8   at /app/poetry/venv/lib/python3.12/site-packages/poetry/installation/chef.py:164 in _prepare
#13 307.8       160│ 
#13 307.8       161│                 error = ChefBuildError("\n\n".join(message_parts))
#13 307.8       162│ 
#13 307.8       163│             if error is not None:
#13 307.8     → 164│                 raise error from None
#13 307.8       165│ 
#13 307.8       166│             return path
#13 307.8       167│ 
#13 307.8       168│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:
#13 307.8 
#13 307.8 Note: This error originates from the build backend, and is likely not a problem with poetry but with splinecalib (0.0.7) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "splinecalib (==0.0.7)"'.

[blucena]

Thanks @Jennie-Myhren.

On my end I've started investigating and have found the following. I have two Macs, one older (2019) and one newer (Mac M2). On both I tried creating a new 3.9.19 environment (with pyenv) and installing splinecalib. In both cases it installed numpy 2.0.0 and scipy 1.3.1. On the older Mac everything installed fine. On the newer Mac I got the same error as @dcturner45 .

As far as I can tell, the problem is due to some issue with Cython's interface with Numpy. Cython creates C code that interacts with numpy and on some configurations it is not working. May or may not be related to the
"#warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]".

I'll kepp digging. Please continue to comment with any relevant findings.

[dcturner45]

@blucena When attempting to build from source locally in a fresh environment + pip install numpy==1.26.4, I get the same failure as from my earlier comment. However, pinning numpy == 1.26.4 in pyproject.toml allows the build to successfully complete. Not sure if pyproject is somehow pulling/caching its own numpy independent of the version that's installed in the environment?

The following is on an AWS EC2 instance running Amazon Linux (python3/pip3 are used because my EC2 shell aliases python/pip to the system python and the conda environment doesn't overwrite them):

[computer@ip ~]# conda create -y --name sctest python=3.9
[computer@ip ~]# conda activate sctest
(sctest) [computer@ip ~]# python3 --version
Python 3.9.19
(sctest) [computer@ip ~]# pip3 --version
pip 24.0 from /opt/conda/envs/sctest/lib/python3.9/site-packages/pip (python 3.9)
(sctest) [computer@ip ~]# git clone https://github.com/numeristical/splinecalib.git
(sctest) [computer@ip ~]# cd splinecalib/
(sctest) [computer@ip splinecalib]# pip3 install numpy==1.26.4
...
Installing collected packages: numpy
Successfully installed numpy-1.26.4

(sctest) [computer@ip splinecalib]# pip3 install .
...
      /mnt/tmp/pip-build-env-vzi0ze44/overlay/lib/python3.9/site-packages/numpy/_core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
       #warning "Using deprecated NumPy API, disable it with " \
        ^~~~~~~
      In file included from /opt/conda/envs/sctest/include/python3.9/pytime.h:6:0,
                       from /opt/conda/envs/sctest/include/python3.9/Python.h:81,
                       from splinecalib/loss_fun_c.c:16:
      splinecalib/loss_fun_c.c: In function ‘__pyx_f_5numpy_PyDataType_SHAPE’:
      splinecalib/loss_fun_c.c:3539:39: error: ‘PyArray_Descr {aka struct _PyArray_Descr}’ has no member named ‘subarray’
           __Pyx_INCREF(((PyObject*)__pyx_v_d->subarray->shape));
                                             ^
      /opt/conda/envs/sctest/include/python3.9/object.h:112:41: note: in definition of macro ‘_PyObject_CAST’
       #define _PyObject_CAST(op) ((PyObject*)(op))
                                               ^~
      splinecalib/loss_fun_c.c:1796:27: note: in expansion of macro ‘Py_INCREF’
         #define __Pyx_INCREF(r) Py_INCREF(r)
                                 ^~~~~~~~~
      splinecalib/loss_fun_c.c:3539:5: note: in expansion of macro ‘__Pyx_INCREF’
           __Pyx_INCREF(((PyObject*)__pyx_v_d->subarray->shape));
           ^~~~~~~~~~~~
      splinecalib/loss_fun_c.c:3540:36: error: ‘PyArray_Descr {aka struct _PyArray_Descr}’ has no member named ‘subarray’
           __pyx_r = ((PyObject*)__pyx_v_d->subarray->shape);
                                          ^~
      error: command '/bin/gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for splinecalib
Failed to build splinecalib
ERROR: Could not build wheels for splinecalib, which is required to install pyproject.toml-based projects

After pinning numpy == 1.26.4 in pyproject.toml:

(sctest) [computer@ip splinecalib]# cat pyproject.toml
[build-system]
requires = ["setuptools", "setuptools-scm", "numpy == 1.26.4"]
build-backend = "setuptools.build_meta"

(sctest) [computer@ip splinecalib]# pip install .
...
Requirement already satisfied: numpy>=1.16 in /opt/conda/envs/sctest/lib/python3.9/site-packages (from splinecalib==0.0.7) (1.26.4)
...
Building wheels for collected packages: splinecalib
  Building wheel for splinecalib (pyproject.toml) ... done
  Created wheel for splinecalib: filename=splinecalib-0.0.7-cp39-cp39-linux_x86_64.whl size=50883 sha256=dad4e860f00466bebfed736432ba7bcc1ae441c4ae40a2229b964d1433f1a0fe
  Stored in directory: /mnt/tmp/pip-ephem-wheel-cache-54fgf97z/wheels/41/91/3f/444f0a9bbacae2fdcca70969469ff7bac4cf22f94478188029
Successfully built splinecalib
Installing collected packages: scipy, splinecalib
Successfully installed scipy-1.13.1 splinecalib-0.0.7

Identical behavior is observed on my Mac (M1).

[blucena]

Thanks @dcturner45 . I also replicated the same behavior you show. I am mystified as to what is going on. I'm sure I have some weirdness in how my package is configured. For example, I notice that I am using both setup.py and pyproject.toml to specify requirements, which is probably wrong.

In the meantime, I have changed pyproject.toml to read "numpy<2" and pushed out a new version of splinecalib (0.0.8). From my local testing this seems to work (and avoids pinning a version). Interestingly, if you have numpy 2.0.0 installed already, this doesn't cause any problem - it seems to let you continue with that version.

If anyone is strong at the nitty-gritty on packaging, I'd love some help advice on how to clean this stuff up...

In the meantime, please let me know if this is now working reasonably well with version 0.0.8 (@SimpleCoder35 @Jennie-Myhren)

[Jennie-Myhren]

@blucena - Thanks for the fix! splinecalib v0.0.8 installs without issue when I run poetry install inside my docker container.

My latest projects only use pyproject.toml, but I found this guide helpful when I was playing around with different potential fixes for the numpy dependency issue: https://packaging.python.org/en/latest/guides/modernize-setup-py-project/.

[dcturner45]

@blucena I am able to build successfully with the 0.0.8 release. Thanks so much for the quick fix! One last small issue - if I don't explicitly specify numpy<2.0.0 in my own package's requirements, I get the following fatal runtime error when trying to run my package:

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.0 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

I'm able to resolve this by specifying numpy<2.0.0 in my package's requirements, which is easy to do, but you may want to also specify numpy<2.0.0 in splinecalib's setup.py's install_requires to ensure the right numpy version is present at runtime. I'm not familiar with pyproject, but it seems in this case the dependencies listed in pyproject.toml are "build-time", while setup.py's dependencies are "runtime". They are apparently resolved independently of each other, so it looks like version-gating needs to be explicitly specified in both.

[blucena]

@Jennie-Myhren Thanks for confirming it works in your setup.
@dcturner45 Also thanks for confirming. I did notice in my environment that it allowed me to work with numpy 2.0 despite the restriction in pyproject.toml and that it was likely because of the install_requires piece in setup.py. i'm going to experiment a little bit, but if it seems the right choice I'll add the "numpy<2" to setup.py also and push out a new version.

[blucena]

@dcturner45 @Jennie-Myhren @SimpleCoder35

Added "numpy<2" to the setup.py and pushed out version 0.0.9

Thanks for all your help and let me know if you have any other feedback / suggestions.

[MushroomHunting]

hi @blucena and all. I've tried installing a git clone of the latest repo (0.0.9) with in a python 3.10 env that has numpy 2.2.3 installed. From my understanding above this should be possible so I fear I must be doing something wrong.

pip install .

gives the following successful installation:

Processing /home/user1/Downloads/splinecalib
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: numpy<2,>=1.16 in /home/user1/.conda/envs/py310/lib/python3.10/site-packages (from splinecalib==0.0.9) (1.26.4)
Requirement already satisfied: scipy>=1.3 in /home/user1/.conda/envs/py310/lib/python3.10/site-packages (from splinecalib==0.0.9) (1.14.1)
Building wheels for collected packages: splinecalib
  Building wheel for splinecalib (pyproject.toml) ... done
  Created wheel for splinecalib: filename=splinecalib-0.0.9-cp310-cp310-linux_x86_64.whl size=49032 sha256=cb58e8e416460a6c45db0e26cf190d4acd96c3dd7591a3d52d13a54b1faed8b4
  Stored in directory: /tmp/pip-ephem-wheel-cache-cz9dxf2v/wheels/5b/ad/9f/91ba314345bde44e4e701db6baffd344a110dbcc1943904faf
Successfully built splinecalib
Installing collected packages: splinecalib
Successfully installed splinecalib-0.0.9

but when i go to import i get the following

>>> import ml_insights
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user1/.conda/envs/py310/lib/python3.10/site-packages/ml_insights/__init__.py", line 7, in <module>
    from splinecalib import SplineCalib
  File "/home/user1/Downloads/splinecalib/splinecalib/__init__.py", line 4, in <module>
    from .splinecalib import SplineCalib
  File "/home/user1/Downloads/splinecalib/splinecalib/splinecalib.py", line 2, in <module>
    from .calib_utils import _natural_cubic_spline_basis_expansion
  File "/home/user1/Downloads/splinecalib/splinecalib/calib_utils.py", line 9, in <module>
    from loss_fun_c import pen_ll_fun, pen_ll_fun_grad
  File "loss_fun_c.pyx", line 1, in init splinecalib.loss_fun_c
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

I would prefer not to install an older version of numpy if possible

appreciate any advice on how to get this working

cheers!

[blucena]

The quick way to get it working is to downgrade numpy to <2. Cython 3 has incompatibilities with numpy>=2. They are being resolved in the next Cython version. The latest release candidate of Cython seems to resolve it but I have been reluctant to update splinecalib using that version of Cython since it is not an "official" version. Lmk if downgrading numpy works for you, if not, I can try to push out a new version. Thanks!