Skip to content

add arrow to contrib, move to sha256 #157

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 26 commits into
base: master
Choose a base branch
from
+247 −62
Open

add arrow to contrib, move to sha256 #157

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
4728615
add arrow to contrib, move to sha256
poshul Dec 1, 2025
22c2300
use newer boost ver
poshul Dec 1, 2025
56fffd5
correct arrow file
poshul Dec 1, 2025
e505b42
check success of arrow compilation
poshul Dec 1, 2025
2f16e61
clean up comments
poshul Dec 1, 2025
5b96aeb
correct boost again
poshul Dec 1, 2025
6c11379
boost?
poshul Dec 1, 2025
5ba9230
use correct repo version
poshul Dec 1, 2025
f942b8e
f
poshul Dec 1, 2025
a551a9c
correct zip extract command
poshul Dec 1, 2025
a04ed85
Add thrift to main.yml
poshul Dec 2, 2025
a01a7aa
Refactor CMake setup conditions for ARM architecture
poshul Dec 2, 2025
08a054d
download whole apache arrow zip
poshul Dec 2, 2025
bd334dc
Address review comments
poshul Dec 2, 2025
78ea780
clean up comments
poshul Dec 2, 2025
cea7770
fix readme
poshul Dec 2, 2025
4900c9f
Improve error handling in arrow build process
poshul Dec 2, 2025
3df30b0
don't use relative path in dockerfile
poshul Dec 2, 2025
e20c134
Fix error message for arrow library build failure
poshul Dec 2, 2025
b382dbc
Allow parallel processing (including in coin-or) fix boost headers fo…
poshul Dec 2, 2025
c846eed
fix boost parallelism in ci address review comment
poshul Dec 2, 2025
8924ef1
fix windows
poshul Dec 2, 2025
278a705
Remove NUMBER_OF_JOBS configuration from CMake
poshul Dec 2, 2025
b26db63
fix windows part 2
poshul Dec 2, 2025
50580f3
address review
poshul Dec 2, 2025
32e568a
windows parallelism
poshul Dec 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 23 additions & 9 deletions .github/workflows/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ jobs:
CMAKE_ARGS: -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_C_COMPILER=clang -DCMAKE_OSX_DEPLOYMENT_TARGET=10.9
- os: windows-latest
CMAKE_ARGS: -G"Visual Studio 17 2022" -A"x64"
- os: ubuntu-22.04-arm
- os: ubuntu-24.04-arm
CMAKE_ARGS: ""

steps:
Expand All @@ -44,13 +44,13 @@ jobs:
path: contrib

- name: Setup cmake
if: matrix.os != 'ubuntu-22.04-arm'
if: ${{ !endsWith(matrix.os, '-arm') }}
uses: jwlawson/actions-setup-cmake@v2
with:
cmake-version: '3.29.x'

- name: Install CMake (Ubuntu arm64)
if: matrix.os == 'ubuntu-22.04-arm'
if: ${{ endsWith(matrix.os, '-arm') }}
run: |
sudo apt-get update
sudo apt-get install -y cmake
Expand All @@ -59,12 +59,12 @@ jobs:
shell: bash
run: |
if [ "$RUNNER_OS" == "Linux" ]; then
sudo apt-get -y install make autoconf automake tar patch libtool gcc
sudo apt-get -y install make autoconf automake tar patch libtool gcc libthrift-dev
elif [ "$RUNNER_OS" == "Windows" ]; then
# choco install important_windows_software
echo "Nothing to install"
elif [ "$RUNNER_OS" == "macOS" ]; then
brew install autoconf automake libtool
brew install autoconf automake libtool thrift
else
echo "$RUNNER_OS not supported"
exit 1
Expand All @@ -85,11 +85,25 @@ jobs:
arch: x64

- name: Build contrib
shell: bash
run: |
mkdir "${{ github.workspace }}/contrib-build"
cd "${{ github.workspace }}/contrib-build"
cmake ${{ matrix.CMAKE_ARGS }} -DBUILD_TYPE=ALL -DNUMBER_OF_JOBS=4 ${{ github.workspace }}/contrib
cmake ${{ matrix.CMAKE_ARGS }} -DBUILD_TYPE=OPENMP -DNUMBER_OF_JOBS=4 ${{ github.workspace }}/contrib
if [ "$RUNNER_OS" == "Windows" ]; then
NPROC=$NUMBER_OF_PROCESSORS
WORKSPACE="${GITHUB_WORKSPACE//\\//}"
elif [ "$RUNNER_OS" == "macOS" ]; then
NPROC=$(sysctl -n hw.ncpu)
WORKSPACE="$GITHUB_WORKSPACE"
else
NPROC=$(nproc)
WORKSPACE="$GITHUB_WORKSPACE"
fi
export CMAKE_BUILD_PARALLEL_LEVEL=${NPROC:-4}
Comment on lines +88 to +100
Copy link
Contributor

@coderabbitai coderabbitai bot Dec 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🌐 Web query:

GitHub Actions Windows runner NUMBER_OF_PROCESSORS bash environment variable

💡 Result:

Short answer

  • GitHub-hosted Windows runners have the standard Windows environment variable %NUMBER_OF_PROCESSORS% available (it reflects the processor count visible to the process). Use Windows-style access in PowerShell/CMD or query it from bash via cmd/powershell. (docs.github.com)

How to read it in a workflow step

  • PowerShell step: echo $env:NUMBER_OF_PROCESSORS. (docs.github.com)
  • CMD step: echo %NUMBER_OF_PROCESSORS%. (docs.github.com)
  • Bash step on a Windows runner (Git Bash / msys2): call through cmd or PowerShell, e.g.:
    • cmd.exe /C "echo %NUMBER_OF_PROCESSORS%"
    • powershell -Command '$env:NUMBER_OF_PROCESSORS'
      These reliably return the Windows value when running bash on a Windows runner. (docs.github.com)

Notes / caveats

  • %NUMBER_OF_PROCESSORS% reports processors visible to the process (can differ on systems with >64 logical processors or when process affinity is set). For GitHub-hosted Windows runners the documented CPU count is in the runners reference (typically 2–4 cores depending on runner type). (learn.microsoft.com)

If you want, I can give an exact workflow snippet showing each method.

Citations:

🏁 Script executed:

# Check if there are other workflows in the repo that handle CPU detection
find .github/workflows -type f -name "*.yml" -o -name "*.yaml" | head -20

Repository: OpenMS/contrib

Length of output: 165

🏁 Script executed:

# Check the current workflow file to understand context better
head -100 .github/workflows/main.yml | tail -50

Repository: OpenMS/contrib

Length of output: 1752

Fix Windows CPU detection in bash shell.

Line 91 uses $NUMBER_OF_PROCESSORS directly in bash, but this Windows environment variable is not available to the bash shell on GitHub Actions Windows runners (Git Bash/msys2). The code silently falls back to 4 cores (line 100), masking the availability of the actual CPU count.

On Windows runners with bash, NUMBER_OF_PROCESSORS must be accessed via cmd.exe or PowerShell. Use one of these approaches:

# Option 1: Query via cmd.exe
NPROC=$(cmd.exe /C "echo %NUMBER_OF_PROCESSORS%" 2>/dev/null)

# Option 2: Query via PowerShell
NPROC=$(powershell -Command '$env:NUMBER_OF_PROCESSORS' 2>/dev/null)

# Option 3: Use wmic (Windows-only, more reliable)
NPROC=$(wmic os get processorcount 2>/dev/null | grep -oE '[0-9]+' | head -1)

Any of these will correctly detect the actual Windows CPU count instead of defaulting to 4.

🤖 Prompt for AI Agents 
.github/workflows/main.yml around lines 88 to 100: Windows CPU detection
currently uses $NUMBER_OF_PROCESSORS directly in the bash shell which is not
visible there; change the Windows branch to query the Windows environment from
bash via one of the supported methods (invoke cmd.exe to echo
%NUMBER_OF_PROCESSORS%, or call PowerShell to read $env:NUMBER_OF_PROCESSORS, or
use wmic to get processor count), assign that result to NPROC, ensure stderr is
ignored and fall back to a safe default if the command fails, then export
CMAKE_BUILD_PARALLEL_LEVEL using the detected NPROC (or the default).

mkdir "$WORKSPACE/contrib-build"
cd "$WORKSPACE/contrib-build"
cmake ${{ matrix.CMAKE_ARGS }} -DBUILD_TYPE=ALL -DNUMBER_OF_JOBS=$CMAKE_BUILD_PARALLEL_LEVEL "$WORKSPACE/contrib"
mkdir -p "$WORKSPACE/contrib-build-openmp"
cd "$WORKSPACE/contrib-build-openmp"
cmake ${{ matrix.CMAKE_ARGS }} -DBUILD_TYPE=OPENMP -DNUMBER_OF_JOBS=$CMAKE_BUILD_PARALLEL_LEVEL "$WORKSPACE/contrib"

- name: Configure Git for Tagging
if: github.event_name == 'workflow_dispatch'
Expand Down
67 changes: 39 additions & 28 deletions CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,8 +91,8 @@
## - however: exec_process seems to handle our arguments incorrectly (at least on Windows). Check functionality before you commit!

# Determine what to build (all or single library)
set( BUILD_TYPE LIST CACHE STRING "Can be used to restrict building to a single library: ALL,LIBSVM,XERCESC,BOOST,COINOR,BZIP2,ZLIB,GLPK,KISSFFT,OPENMP")
set( VALID_BUILD_TYPES "ALL" "LIBSVM" "XERCESC" "BOOST" "COINOR" "BZIP2" "ZLIB" "GLPK" "EIGEN" "KISSFFT" "HDF5" "OPENMP")
set( BUILD_TYPE LIST CACHE STRING "Can be used to restrict building to a single library: ALL,LIBSVM,XERCESC,BOOST,COINOR,BZIP2,ZLIB,GLPK,KISSFFT,OPENMP,ARROW")
set( VALID_BUILD_TYPES "ALL" "LIBSVM" "XERCESC" "BOOST" "COINOR" "BZIP2" "ZLIB" "GLPK" "EIGEN" "KISSFFT" "HDF5" "OPENMP" "ARROW")

# check build type
if (BUILD_TYPE STREQUAL "HELP" OR BUILD_TYPE STREQUAL "LIST")
Expand Down Expand Up @@ -187,7 +187,7 @@ set (PATCH_DIR "${PROJECT_SOURCE_DIR}/patches/" CACHE STRING "Directory containi
## Warning: ensure these are all upper-case names!
set(BZIP2_DIR ${CONTRIB_BIN_SOURCE_DIR}/bzip2-1.0.5)
set(ZLIB_DIR ${CONTRIB_BIN_SOURCE_DIR}/zlib-1.3.1)
set(BOOST_DIR ${CONTRIB_BIN_SOURCE_DIR}/boost_1_78_0)
set(BOOST_DIR ${CONTRIB_BIN_SOURCE_DIR}/boost-1.81.0)
set(XERCES_DIR ${CONTRIB_BIN_SOURCE_DIR}/Xerces-C_3_2_0)
set(LIBSVM_DIR ${CONTRIB_BIN_SOURCE_DIR}/libsvm-3.12)
set(GLPK_DIR ${CONTRIB_BIN_SOURCE_DIR}/glpk-4.46)
Expand All @@ -196,64 +196,70 @@ set(EIGEN_DIR ${CONTRIB_BIN_SOURCE_DIR}/eigen-3.4.0)
set(KISSFFT_DIR ${CONTRIB_BIN_SOURCE_DIR}/kissfft)
set(HDF5_DIR ${CONTRIB_BIN_SOURCE_DIR}/hdfsrc)
set(OPENMP_DIR ${CONTRIB_BIN_SOURCE_DIR}/openmp-12.0.1.src)
set(ARROW_DIR ${CONTRIB_BIN_SOURCE_DIR}/arrow-apache-arrow-22.0.0/cpp)

## source(archive) paths
## PLEASE upload all source archives to
## https://abibuilder.cs.uni-tuebingen.de/archive/openms/contrib/source_packages/
## and provide the necessary information below
## https://github.com/OpenMS/contrib-sources/
## and provide the necessary information below
## ARCHIVE_<LIBNAME> the name of the archive as it was uploaded
## ARCHIVE_<LIBNAME>_TAR the name of the tar in the tar.gz
## ARCHIVE_<LIBNAME>_SHA1 the sha1 of the file uploaded to sourceforge
## ARCHIVE_<LIBNAME>_SHA256 the sha256 of the file uploaded
## e.g.
## set(ARCHIVE_ZLIB zlib.tar.gz)
## set(ARCHIVE_ZLIB_TAR zlib.tar)
## set(ARCHIVE_ZLIB_SHA1 "8e8b93fa5eb80df1afe5422309dca42964562d7e")
## set(ARCHIVE_ZLIB_SHA256 "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855")
## NOTE: for local development you can simply place the *.tar.gz in the binary
## directory to prevent the download and sha1 verification
## directory to prevent the download and sha256 verification
## LIBRARIES: BZIP2, ZLIB, BOOST, XERCES, LIBSVM, GLPK, COINOR, EIGEN, KISSFFT, HDF5, OPENMP, ARROW

set(ARCHIVE_BZIP2 bzip2-1.0.5.tar.gz)
set(ARCHIVE_BZIP2_TAR bzip2-1.0.5.tar)
set(ARCHIVE_BZIP2_SHA1 "8b02713ea0c8eb501cc8bdac4432d7f29700f767")
set(ARCHIVE_BZIP2_SHA256 "f7bf5368309d76e5daf3a89d4d1bea688dac7780742e7a0ae1af19be9316fe22")

set(ARCHIVE_ZLIB zlib-1.3.1.tar.gz)
set(ARCHIVE_ZLIB_TAR zlib-1.3.1.tar)
set(ARCHIVE_ZLIB_SHA1 "f535367b1a11e2f9ac3bec723fb007fbc0d189e5")
set(ARCHIVE_ZLIB_SHA256 "9a93b2b7dfdac77ceba5a558a580e74667dd6fede4585b91eefb60f03b72df23")

set(ARCHIVE_BOOST boost_1_78_0.tar.gz)
set(ARCHIVE_BOOST_TAR boost_1_78_0.tar)
set(ARCHIVE_BOOST_SHA1 "ff717ea23af2900e0ad3cae616b3e4ae43e68ad7")
set(ARCHIVE_BOOST boost-1.81.0.tar.gz)
set(ARCHIVE_BOOST_TAR boost-1.81.0.tar)
set(ARCHIVE_BOOST_SHA256 "121da556b718fd7bd700b5f2e734f8004f1cfa78b7d30145471c526ba75a151c")

set(ARCHIVE_XERCES Xerces-C_3_2_0.tar.gz)
set(ARCHIVE_XERCES_TAR Xerces-C_3_2_0.tar)
set(ARCHIVE_XERCES_SHA1 "da64ed4065694a88378cad340c8382d15a614837")
set(ARCHIVE_XERCES_SHA256 "68cca38aa0db168fa8ccce1f470a89d91104c4586e2635e0cd6602baf55e4eb8")

set(ARCHIVE_LIBSVM libsvm-3.12.tar.gz)
set(ARCHIVE_LIBSVM_TAR libsvm-3.12.tar)
set(ARCHIVE_LIBSVM_SHA1 "6ee77eb3d89af07fe48058dba0dcc24c85020b85")
set(ARCHIVE_LIBSVM_SHA256 "9dcd7a301665d2cbd8c109bba0e8cc2d47b5cc2a9c46aeb359f2dab67be69f74")

set(ARCHIVE_GLPK glpk-4.46.tar.gz)
set(ARCHIVE_GLPK_TAR glpk-4.46.tar)
set(ARCHIVE_GLPK_SHA1 "73d589452dc1d8d6a784b3f86f4268cae37420d7")
set(ARCHIVE_GLPK_SHA256 "fbbe438262f06813850837afd0cd55ed66f17f2004e2ce4f0078754f089e3242")

set(ARCHIVE_COINOR CoinMP-1.8.3-vs22.tar.gz)
set(ARCHIVE_COINOR_TAR CoinMP-1.8.3-vs22.tar)
set(ARCHIVE_COINOR_SHA1 "252aabfa824a2656e7abbc14278a61e02fe63e48")
set(ARCHIVE_COINOR_SHA256 "75eefc0b0e06dc11a1e5004222f69766366c1e08d3b739f25f53c756ae28a309")

set(ARCHIVE_EIGEN eigen-3.4.0.tar.gz)
set(ARCHIVE_EIGEN_TAR eigen-3.4.0.tar)
set(ARCHIVE_EIGEN_SHA1 "d222db69a9e87d9006608e029d1039039f360b52")
set(ARCHIVE_EIGEN_SHA256 "8586084f71f9bde545ee7fa6d00288b264a2b7ac3607b974e54d13e7162c1c72")

set(ARCHIVE_KISSFFT kissfft-130.tar.gz)
set(ARCHIVE_KISSFFT_TAR kissfft-130.tar)
set(ARCHIVE_KISSFFT_SHA1 "14ee3bd4a6480dc7fbec9a6d3a16d5d9fbbfc0fb")
set(ARCHIVE_KISSFFT_SHA256 "d846005b3a4b45202f3b288b2d6b5b1df544526ae6aaffe8707770bd09093064")

set(ARCHIVE_HDF5 hdf5-1_14_3.tar.gz)
set(ARCHIVE_HDF5_TAR hdf5-1_14_3.tar)
set(ARCHIVE_HDF5_SHA1 "0e464802c2f9185f8b8db6e04149ca1d3eaf27c5")
set(ARCHIVE_HDF5_SHA256 "126b800e9b87f65e7e7ffa58ee006607bab03a8e11e5209ea3d5c0c331321d43")

set(ARCHIVE_OPENMP openmp-12.0.1.src.tar.xz)
set(ARCHIVE_OPENMP_tar openmp-12.0.1.src.tar)
set(ARCHIVE_OPENMP_SHA1 "7c051f6050ff23763b253a757927774b5eeaf644")
set(ARCHIVE_OPENMP_TAR openmp-12.0.1.src.tar)
set(ARCHIVE_OPENMP_SHA256 "60fe79440eaa9ebf583a6ea7f81501310388c02754dbe7dc210776014d06b091")

set(ARCHIVE_ARROW arrow-apache-arrow-22.0.0.tar.gz )
set(ARCHIVE_ARROW_TAR arrow-apache-arrow-22.0.0.tar )
set(ARCHIVE_ARROW_SHA256 "8a95e6c7b9bec2bc0058feb73efe38ad6cfd49a0c7094db29b37ecaa8ab16051")

## necessary for clean up .. change if install process of library changes
set(INCLUDE_DIR_BOOST ${CONTRIB_BIN_INCLUDE_DIR}/boost)
Expand All @@ -263,6 +269,7 @@ set(INCLUDE_DIR_COINOR ${CONTRIB_BIN_INCLUDE_DIR}/coin)
set(INCLUDE_DIR_EIGEN ${CONTRIB_BIN_INCLUDE_DIR}/eigen3)
set(INCLUDE_DIR_KISSFFT ${CONTRIB_BIN_INCLUDE_DIR}/kissfft)
set(INCLUDE_DIR_OPENMP ${CONTRIB_BIN_INCLUDE_DIR}/openmp.src)
set(INCLUDE_DIR_ARROW ${CONTRIB_BIN_INCLUDE_DIR}/arrow)

## hack for simple libs that do not have include directories
set(INCLUDE_FILES_LIBSVM ${CONTRIB_BIN_INCLUDE_DIR}/svm.h)
Expand Down Expand Up @@ -476,6 +483,7 @@ include ("${CMAKE_SOURCE_DIR}/libraries.cmake/eigen.cmake")
include ("${CMAKE_SOURCE_DIR}/libraries.cmake/kissfft.cmake")
include ("${CMAKE_SOURCE_DIR}/libraries.cmake/hdf5.cmake")
include ("${CMAKE_SOURCE_DIR}/libraries.cmake/openmp.cmake")
include ("${CMAKE_SOURCE_DIR}/libraries.cmake/arrow.cmake")

## build mac os x specific C/C++ flags
set( OPENMS_CONTRIB_MACOSX_MIXED_MODE 0 CACHE INTERNAL "building multiple architectures on MacOSX" FORCE)
Expand All @@ -500,12 +508,6 @@ endforeach()
set( FORCE_REBUILD OFF CACHE BOOL "All installation and build traces of the contrib packages will be removed before building.")
message(STATUS "FORCE_REBUILD: ${FORCE_REBUILD}")

set( NUMBER_OF_JOBS 2 CACHE STRING "Number of jobs executed in parallel during the build process if the build system supports multi-job builds (e.g., make, bjam)")
if("${NUMBER_OF_JOBS}" STREQUAL "")
set( NUMBER_OF_JOBS 1 )
endif("${NUMBER_OF_JOBS}" STREQUAL "")
message(STATUS "NUMBER_OF_JOBS: ${NUMBER_OF_JOBS} (maximal number of concurrent compile jobs)")

## define building of shared or unshared versions of the contrib libraries
set( BUILD_SHARED_LIBRARIES Off CACHE BOOL "Builds all contrib libraries as shared libraries.")

Expand Down Expand Up @@ -614,6 +616,15 @@ if ("OPENMP" IN_LIST BUILD_TYPE)
OPENMS_COPY_LIBS("OPENMP")
endif()

##################################################
### ARROW ###
##################################################
if (BUILD_TYPE STREQUAL "ALL" OR "ARROW" IN_LIST BUILD_TYPE)
OPENMS_CLEAN_LIB("ARROW")
OPENMS_CONTRIB_BUILD_ARROW()
OPENMS_COPY_LIBS("ARROW")
endif()

## finally copy README.txt to project-binary directory to mark process complete
configure_file("${PROJECT_SOURCE_DIR}/README.md" "${PROJECT_BINARY_DIR}/README_contrib.txt" COPYONLY)

Expand Down
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Please note that an internet connection is required to
build the contrib since the actual source packages are
only downloaded when built.

You can find the sources on [(https://abibuilder.cs.uni-tuebingen.de/archive/openms/contrib)](https://github.com/OpenMS/contrib-sources).
You can find the sources on [https://github.com/OpenMS/contrib-sources](https://github.com/OpenMS/contrib-sources).

Build
============
Expand Down
2 changes: 1 addition & 1 deletion dockerfiles/pyopenms/manylinux/ARM64_Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ RUN mkdir contrib-build
RUN ls -l
WORKDIR /contrib-build
RUN ls -l
RUN cmake -DBUILD_TYPE=ALL ../contrib && rm -rf archives src
RUN CMAKE_BUILD_PARALLEL_LEVEL=$(nproc) cmake -DBUILD_TYPE=ALL /contrib && rm -rf archives src

WORKDIR /

Expand Down
2 changes: 1 addition & 1 deletion dockerfiles/pyopenms/manylinux/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ RUN mkdir contrib-build
RUN ls -l
WORKDIR /contrib-build
RUN ls -l
RUN cmake -DBUILD_TYPE=ALL ../contrib && rm -rf archives src
RUN CMAKE_BUILD_PARALLEL_LEVEL=$(nproc) cmake -DBUILD_TYPE=ALL /contrib && rm -rf archives src

WORKDIR /

Expand Down
Loading