Skip to content

Build of SciPy-bundle v2024.05 failing due to new search-path-cpp-headers option #1216

@bedroge

Description

@bedroge

The build of the the scipy extension of SciPy-bundle v2024.05 failed in #1210 with:

In file included from ../scipy/scipy-1.13.1/scipy/io/_fast_matrix_market/src/_fmm_core_read_coo.cpp:5:
../scipy/scipy-1.13.1/scipy/io/_fast_matrix_market/src/_fmm_core.hpp:14:10: fatal error: pybind11/pybind11.h: No such file or directory
   14 | #include <pybind11/pybind11.h>
      |          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.

This looks rather strange and I started wondering if it could be related to switching to search-path-cpp-headers = include_paths for 2025.06. While inspecting the log, I noticed that right before the build started the following C_INCLUDE_PATH was used.

== 2025-10-01 08:55:18,072 environment.py:95 INFO Environment variable CPLUS_INCLUDE_PATH set to /cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FFTW/3.3.10-GCC-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FlexiBLAS/3.4.4-GCC-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FlexiBLAS/3.4.4-GCC-13.3.0/include/flexiblas
(previous value: '/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FFTW/3.3.10-GCC-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FlexiBLAS/3.4.4-GCC-13.3.0/include/flexiblas:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FlexiBLAS/3.4.4-GCC-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/OpenBLAS/0.3.27-GCC-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/pybind11/2.12.0-GCC-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/Boost/1.85.0-GCC-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/ICU/75.1-GCCcore-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/zstd/1.5.6-GCCcore-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/lz4/1.9.4-GCCcore-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/pkgconf/2.2.0-GCCcore-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/Python/3.12.3-GCCcore-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/OpenSSL/3/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/libffi/3.4.5-GCCcore-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/SQLite/3.45.3-GCCcore-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/Tcl/8.6.14-GCCcore-13.3.0/include')

So it somehow overwrote the existing C_INCLUDE_PATH (which already included paths to FFTW and FlexiBLAS, but also to all other deps incl pybind11) by paths to only FFTW and FlexiBLAS. That definitely doesn't look right.

To confirm that it's really related to this setting, I did another build without this new search-path-cpp-headers setting (unset EASYBUILD_SEARCH_PATH_CPP_HEADERS), and that worked fine. Furthermore, jumping into the EB interactive shell and manually adding the path to the pybind11 header files indeed also solves the issue. So, why is it overwriting the correct CPLUS_INCLUDE_PATH by one that, apparently (?), only contains paths to the toolchain components?

Some more relevant lines from the log file, all just before the build starts:

os.environ["CPLUS_INCLUDE_PATH"] = "/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FFTW/3.3.10-GCC-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FlexiBLAS/3.4.4-GCC-13.3.0/include/flexiblas:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FlexiBLAS/3.4.4-GCC-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/OpenBLAS/0.3.27-GCC-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/pybind11/2.12.0-GCC-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/Boost/1.85.0-GCC-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/ICU/75.1-GCCcore-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/zstd/1.5.6-GCCcore-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/lz4/1.9.4-GCCcore-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/pkgconf/2.2.0-GCCcore-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/Python/3.12.3-GCCcore-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/OpenSSL/3/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/libffi/3.4.5-GCCcore-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/SQLite/3.45.3-GCCcore-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/Tcl/8.6.14-GCCcore-13.3.0/include";

This is still correct. But then toolchain.py is changing the build environment:

== 2025-10-01 08:55:18,060 toolchain.py:1170 DEBUG Adding header paths to toolchain variable 'CPLUS_INCLUDE_PATH': /cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FlexiBLAS/3.4.4-GCC-13.3.0

== 2025-10-01 08:55:18,064 toolchain.py:1170 DEBUG Adding header paths to toolchain variable 'CPLUS_INCLUDE_PATH': /cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FlexiBLAS/3.4.4-GCC-13.3.0

== 2025-10-01 08:55:18,066 toolchain.py:1170 DEBUG Adding header paths to toolchain variable 'CPLUS_INCLUDE_PATH': /cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FFTW/3.3.10-GCC-13.3.0

== 2025-10-01 08:55:18,072 toolchain.py:1206 DEBUG _setenv_variables: setting environment variable CPLUS_INCLUDE_PATH to /cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FFTW/3.3.10-GCC-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FlexiBLAS/3.4.4-GCC-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FlexiBLAS/3.4.4-GCC-13.3.0/include/flexiblas

== 2025-10-01 08:55:18,072 environment.py:95 INFO Environment variable CPLUS_INCLUDE_PATH set to /cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FFTW/3.3.10-GCC-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FlexiBLAS/3.4.4-GCC-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FlexiBLAS/3.4.4-GCC-13.3.0/include/flexiblas (previous value: '/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FFTW/3.3.10-GCC-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FlexiBLAS/3.4.4-GCC-13.3.0/include/flexiblas:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FlexiBLAS/3.4.4-GCC-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/OpenBLAS/0.3.27-GCC-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/pybind11/2.12.0-GCC-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/Boost/1.85.0-GCC-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/ICU/75.1-GCCcore-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/zstd/1.5.6-GCCcore-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/lz4/1.9.4-GCCcore-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/pkgconf/2.2.0-GCCcore-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/Python/3.12.3-GCCcore-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/OpenSSL/3/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/libffi/3.4.5-GCCcore-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/SQLite/3.45.3-GCCcore-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/Tcl/8.6.14-GCCcore-13.3.0/include')

== 2025-10-01 08:55:18,072 environment.py:95 INFO Environment variable EBVARCPLUS_INCLUDE_PATH set to /cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FFTW/3.3.10-GCC-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FlexiBLAS/3.4.4-GCC-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FlexiBLAS/3.4.4-GCC-13.3.0/include/flexiblas (previously undefined)

Right after the build failure (between 8:56 and 9:00), EB also detects that the variable was changed:

== 2025-10-01 09:00:23,541 environment.py:168 DEBUG Key in new environment found that is not in old one: CPLUS_INCLUDE_PATH (/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/Python/3.12.3-GCCcore-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/pybind11/2.12.0-GCC-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/pkgconf/2.2.0-GCCcore-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FFTW/3.3.10-GCC-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FlexiBLAS/3.4.4-GCC-13.3.0/include:/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/FlexiBLAS/3.4.4-GCC-13.3.0/include/flexiblas)

I suspect that the environment gets changed by https://github.com/easybuilders/easybuild-framework/blob/easybuild-framework-v5.1.2/easybuild/tools/toolchain/toolchain.py#L901, but I'm not sure why it doesn't take the existing environment variables into account here, and simply overwrites them.

Metadata

Metadata

Assignees

No one assigned

    Labels

    2025.06-software.eessi.io2025.06 version of software.eessi.iobugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions