Skip to content

Add dat bypass, build-cluster-db CLI, and incremental improvements #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ypriverol merged 15 commits into from
Apr 18, 2026
Merged

Add dat bypass, build-cluster-db CLI, and incremental improvements #10

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
2995518
Add QPX parquet schema adapter and cluster2parquet CLI command
ypriverol Apr 11, 2026
797a3ef
Vectorize consensus strategy: replace groupby.apply with idxmin/sort+…
ypriverol Apr 11, 2026
2fb34ef
Support multiple parquet files in inject_cluster_info
ypriverol Apr 11, 2026
caf9b26
Add incremental clustering module with representative retention
ypriverol Apr 11, 2026
3116dcc
Add dat bypass, build-cluster-db CLI, and incremental merge improvements
ypriverol Apr 13, 2026
ffc337c
Update README to reflect dat bypass as primary conversion path
ypriverol Apr 13, 2026
cc175b5
Remove MGF/bypass/legacy terminology from README
ypriverol Apr 13, 2026
d6a9112
Fix test, switch to hatchling+uv, bump to 0.0.4
ypriverol Apr 13, 2026
a2fa10a
Vectorize dat conversion: batch I/O, columnar access
ypriverol Apr 13, 2026
3229e22
Add DuckDB-backed cluster DB, rewrite MSP for scan_titles, add increm…
ypriverol Apr 17, 2026
4628df8
Update CI workflows for dat-bypass path
ypriverol Apr 17, 2026
d74e4b6
Unify build-cluster-db, drop legacy MGF path, add container build scr…
ypriverol Apr 18, 2026
3442870
Update CLI tests after merge-clusters removal
ypriverol Apr 18, 2026
63d793b
Enable loading images on pull request events
ypriverol Apr 18, 2026
8dcaf3e
Preserve None in parse_all_scan_titles object columns
ypriverol Apr 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 13 additions & 44 deletions .github/workflows/pyspectrafuse-containers.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@ name: pyspectrafuse Containers Build and Sync

on:
push:
branches: [ "master" ]
branches: [master, main]
pull_request:
branches: [ "master" ]
branches: [master, main]
paths:
- 'Dockerfile'
- '.github/workflows/**'
Expand All @@ -17,11 +17,6 @@ on:
required: false
default: true
type: 'boolean'
push_release:
description: 'Push as release version'
required: false
default: false
type: boolean

jobs:
build-pyspectrafuse:
Expand All @@ -32,8 +27,6 @@ jobs:
packages: write

env:
RETRY_TIMES: 3
RETRY_DELAY: 30
SINGULARITY_TMPDIR: /tmp/singularity-tmp
SINGULARITY_CACHEDIR: /tmp/singularity-cache
SINGULARITY_LIBRARYDIR: /tmp/singularity-lib
Expand All @@ -50,10 +43,10 @@ jobs:
uses: actions/checkout@v4

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f
uses: docker/setup-buildx-action@v3

- name: Log in to GitHub Container Registry
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef
uses: docker/login-action@v3
Comment on lines +46 to +49

@coderabbitai coderabbitai Bot Apr 17, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🌐 Web query:

Latest stable major versions of docker/setup-buildx-action, docker/login-action, and docker/build-push-action in 2026

💡 Result:

Latest stable major versions as of 2026-04-17: - docker/setup-buildx-action: v4 (released 2026-03-05) - docker/login-action: v4 (released 2026-04-02) - docker/build-push-action: v7 (released 2026-03-05)

Citations:

Update Docker action versions to latest stable releases. The current versions are outdated: docker/setup-buildx-action@v4, docker/login-action@v4, and docker/build-push-action@v7 are available as of April 2026.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/pyspectrafuse-containers.yml around lines 46 - 49, Update
the GitHub Actions steps that use docker actions to their latest stable tags:
replace docker/setup-buildx-action@v3 with docker/setup-buildx-action@v4,
docker/login-action@v3 with docker/login-action@v4, and any
docker/build-push-action@v? with docker/build-push-action@v7; update the action
references in the workflow to those exact tags (e.g., in the steps using
docker/setup-buildx-action, docker/login-action, and docker/build-push-action)
and run a quick workflow lint or dry-run to confirm no breaking input changes
are required by the newer major versions.

with:
registry: ghcr.io
username: ${{ github.actor }}
Expand All @@ -65,18 +58,13 @@ jobs:
VERSION=$(grep '^version = ' pyproject.toml | cut -d'"' -f2)
echo "VERSION=$VERSION" >> $GITHUB_OUTPUT
echo "Extracted version: $VERSION"
# Extract major and minor version
MAJOR=$(echo $VERSION | cut -d'.' -f1)
MINOR=$(echo $VERSION | cut -d'.' -f2)
echo "MAJOR=$MAJOR" >> $GITHUB_OUTPUT
echo "MINOR=$MINOR" >> $GITHUB_OUTPUT
echo "Major.Minor: $MAJOR.$MINOR"

- name: Build and Push pyspectrafuse image
uses: docker/build-push-action@ca052bb54ab0790a636c9b5f226502c73d547a25
uses: docker/build-push-action@v6
with:
context: .
push: ${{ (github.event.inputs.push_images == true || github.event.inputs.push_images == '') && (github.event_name != 'pull_request') }}
load: ${{ github.event_name == 'pull_request' }}
tags: |
ghcr.io/bigbio/pyspectrafuse:latest
ghcr.io/bigbio/pyspectrafuse:${{ steps.version.outputs.VERSION }}
Expand All @@ -85,31 +73,26 @@ jobs:
build-args: |
BUILDKIT_INLINE_CACHE=1
provenance: false
# Note: Only building for linux/amd64
platforms: linux/amd64

- name: Test container functionality
run: |
echo "Testing container import and CLI..."
docker run --rm ghcr.io/bigbio/pyspectrafuse:${{ steps.version.outputs.VERSION }} --help || docker run --rm ghcr.io/bigbio/pyspectrafuse:latest --help
echo "Testing CLI..."
docker run --rm ghcr.io/bigbio/pyspectrafuse:${{ steps.version.outputs.VERSION }} --help
echo "Testing Python imports..."
docker run --rm --entrypoint python ghcr.io/bigbio/pyspectrafuse:${{ steps.version.outputs.VERSION }} -c "from pyspectrafuse.pyspectrafuse import main; print('CLI import successful')" || docker run --rm --entrypoint python ghcr.io/bigbio/pyspectrafuse:latest -c "from pyspectrafuse.pyspectrafuse import main; print('CLI import successful')"
echo "Testing consensus strategy imports..."
docker run --rm --entrypoint python ghcr.io/bigbio/pyspectrafuse:${{ steps.version.outputs.VERSION }} -c "from pyspectrafuse.consensus_strategy.best_spetrum_strategy import BestSpectrumStrategy; print('BestSpectrumStrategy import successful')" || docker run --rm --entrypoint python ghcr.io/bigbio/pyspectrafuse:latest -c "from pyspectrafuse.consensus_strategy.best_spetrum_strategy import BestSpectrumStrategy; print('BestSpectrumStrategy import successful')"
echo "Testing spectrum_utils import (numba caching test)..."
docker run --rm --entrypoint python ghcr.io/bigbio/pyspectrafuse:${{ steps.version.outputs.VERSION }} -c "import spectrum_utils.spectrum as sus; print('spectrum_utils import successful')" || docker run --rm --entrypoint python ghcr.io/bigbio/pyspectrafuse:latest -c "import spectrum_utils.spectrum as sus; print('spectrum_utils import successful')"
docker run --rm --entrypoint python ghcr.io/bigbio/pyspectrafuse:${{ steps.version.outputs.VERSION }} -c "from pyspectrafuse.pyspectrafuse import main; print('CLI import OK')"
echo "Testing dat conversion import..."
docker run --rm --entrypoint python ghcr.io/bigbio/pyspectrafuse:${{ steps.version.outputs.VERSION }} -c "from pyspectrafuse.maracluster_dat import parquet_to_dat; print('dat conversion import OK')"
echo "All container tests passed!"
Comment on lines +80 to 86

@coderabbitai coderabbitai Bot Apr 17, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🌐 Web query:

docker/build-push-action v6 load image into local docker daemon when push is false

💡 Result:

To load the built image into the local Docker daemon when using docker/build-push-action v6 with push: false, set the load input to true. This corresponds to the Buildx --output=type=docker option, which exports the image to the Docker image store, making it available for commands like docker run. Example workflow step: - name: Build image uses: docker/build-push-action@v6 with: context: . push: false load: true tags: myimage:latest After this step, the image with the specified tags will be listed in docker images and usable locally. Note that load: true works for single-platform builds; for multi-platform, it loads only the current node platform's image. This behavior is consistent across v6 releases (up to v6.19.0 as of 2026-02) and is recommended in official Docker docs for testing before push workflows.

Citations:

🏁 Script executed:

cat .github/workflows/pyspectrafuse-containers.yml

Repository: bigbio/pyspectrafuse

Length of output: 4512

Add load: true to the docker/build-push-action configuration to enable local image testing on pull requests.

The workflow will fail the "Test container functionality" step on pull_request events because the image is never loaded into the local Docker daemon. The docker/build-push-action@v6 step sets push: false on PRs but lacks the required load: true parameter. With load: true, the image will be available locally for the subsequent docker run tests (the single-platform build with platforms: linux/amd64 supports this without issue).

Fix: Add load parameter 
    - name: Build and Push pyspectrafuse image
      uses: docker/build-push-action@v6
      with:
        context: .
        push: ${{ (github.event.inputs.push_images == true || github.event.inputs.push_images == '') && (github.event_name != 'pull_request') }}
        load: ${{ github.event_name == 'pull_request' }}
        tags: |
          ghcr.io/bigbio/pyspectrafuse:latest
          ghcr.io/bigbio/pyspectrafuse:${{ steps.version.outputs.VERSION }}
        cache-from: type=gha
        cache-to: type=gha,mode=min
        build-args: |
          BUILDKIT_INLINE_CACHE=1
        provenance: false
        platforms: linux/amd64
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/pyspectrafuse-containers.yml around lines 79 - 85, The
build step using docker/build-push-action@v6 currently sets push: false for PRs
but doesn't set load, so the image isn't loaded into the local Docker daemon and
subsequent docker run tests fail; update the "Build and Push pyspectrafuse
image" action to add a load parameter (e.g., load: ${{ github.event_name ==
'pull_request' }} or load: true for PRs) so that when push is false on
pull_request the built image is loaded locally, ensuring the later docker run
tests (the commands invoking ghcr.io/bigbio/pyspectrafuse:${{
steps.version.outputs.VERSION }} and the Python import checks) can run against
the locally loaded image while keeping platforms: linux/amd64 and the existing
push conditional logic unchanged.


- name: Free up disk space after build
run: |
docker image rm ghcr.io/bigbio/pyspectrafuse:latest || true
docker system prune -af || true
sudo apt-get clean
sudo rm -rf /var/lib/apt/lists/*

- name: Set up Singularity
if: ${{ (github.event.inputs.push_images == true || github.event.inputs.push_images == '') && (github.event_name != 'pull_request') }}
uses: eWaterCycle/setup-singularity@931d4e31109e875b13309ae1d07c70ca8fbc8537
uses: eWaterCycle/setup-singularity@v7
with:
singularity-version: 3.8.7
Comment on lines +95 to 97

@coderabbitai coderabbitai Bot Apr 17, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🌐 Web query:

eWaterCycle/setup-singularity latest version 2026 Apptainer

💡 Result:

The latest version of the eWaterCycle/setup-singularity GitHub Action is v7, released on 2021-10-20. The repository explicitly recommends using the companion eWaterCycle/setup-apptainer action instead, as it supports newer versions of the open-source fork of Singularity (Apptainer). The setup-apptainer action's latest release is v2.0.0 (2022-10-25), with its last code push on 2025-01-07. For eWaterCycle workflows in 2026, official documentation prefers Apptainer over Singularity. Use uses: eWaterCycle/setup-apptainer@v2 with apptainer-version input (e.g., 1.3.6 or latest compatible). Example: - uses: eWaterCycle/setup-apptainer@v2 with: apptainer-version: 1.3.6 Apptainer is backward-compatible with Singularity images.

Citations:

Migrate from eWaterCycle/setup-singularity@v7 to eWaterCycle/setup-apptainer@v2.

The setup-singularity action (v7, released 2021-10-20) is outdated and explicitly deprecated by the maintainers. Since the Singularity/Apptainer split, the eWaterCycle repository recommends using setup-apptainer@v2 (last updated 2025-01-07) instead. Apptainer is the actively maintained fork of Singularity and is backward-compatible with Singularity images. Update to uses: eWaterCycle/setup-apptainer@v2 and use apptainer-version instead of singularity-version (e.g., apptainer-version: 1.3.6).

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/pyspectrafuse-containers.yml around lines 94 - 96, The
workflow uses the deprecated action uses: eWaterCycle/setup-singularity@v7 and
the singularity-version input; replace that with the maintained action uses:
eWaterCycle/setup-apptainer@v2 and swap the singularity-version key for
apptainer-version (for example apptainer-version: 1.3.6) so the job uses
Apptainer instead of the deprecated Singularity action.


Expand All @@ -125,23 +108,9 @@ jobs:

- name: Login and Deploy Container
if: (github.event_name != 'pull_request')
env:
IS_RELEASE: ${{ github.event_name == 'release' }}
IS_MASTER: ${{ github.ref == 'refs/heads/master' && github.event_name == 'push' }}
run: |
echo ${{ secrets.GHCR_TOKEN }} | singularity remote login -u ${{ secrets.GHCR_USERNAME }} --password-stdin oras://ghcr.io
singularity push pyspectrafuse.sif oras://ghcr.io/bigbio/pyspectrafuse-sif:${{ steps.version.outputs.VERSION }}
if [[ "${{ env.IS_RELEASE }}" == "true" || "${{ env.IS_MASTER }}" == "true" ]]; then
if [[ "${{ github.event_name }}" == "release" || "${{ github.ref }}" == "refs/heads/master" ]]; then
singularity push pyspectrafuse.sif oras://ghcr.io/bigbio/pyspectrafuse-sif:latest
fi

- name: Notify on success
if: success()
run: |
echo "::notice::Successfully built and pushed pyspectrafuse Docker and Singularity images to ghcr.io/bigbio"

- name: Notify on failure
if: failure()
run: |
echo "::error::Failed to build or push pyspectrafuse images. Check the logs for details."

35 changes: 10 additions & 25 deletions .github/workflows/tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@ name: Tests

on:
push:
branches: [ main, master, develop ]
branches: [main, master, dev]
pull_request:
branches: [ main, master, develop ]
branches: [main, master, dev]

jobs:
test:
Expand All @@ -13,43 +13,28 @@ jobs:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest]
python-version: ["3.8", "3.9", "3.10", "3.11"]
python-version: ["3.10", "3.11", "3.12"]

steps:
- uses: actions/checkout@v4

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install uv
uses: astral-sh/setup-uv@v4

- name: Cache pip packages
uses: actions/cache@v4
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/pyproject.toml') }}
restore-keys: |
${{ runner.os }}-pip-
- name: Set up Python ${{ matrix.python-version }}
run: uv python install ${{ matrix.python-version }}

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -e ".[test]"
run: uv sync --extra test

- name: Run tests with pytest
run: |
pytest --cov=pyspectrafuse --cov-report=xml --cov-report=term-missing
run: uv run pytest --cov=pyspectrafuse --cov-report=xml --cov-report=term-missing

- name: Upload coverage to Codecov
uses: codecov/codecov-action@v4
if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.10'
if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.11'
with:
file: ./coverage.xml
flags: unittests
name: codecov-umbrella
fail_ci_if_error: false





23 changes: 11 additions & 12 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ FROM python:3.10-slim
LABEL base_image="python:3.10-slim"
LABEL version="1"
LABEL software="pyspectrafuse"
LABEL software.version="0.0.2"
LABEL about.summary="pyspectrafuse - Command-line utilities for spectrum clustering and conversion"
LABEL software.version="0.0.4"
LABEL about.summary="pyspectrafuse - CLI for spectral clustering and consensus library generation"
LABEL about.home="https://github.com/bigbio/pyspectrafuse"
LABEL about.documentation="https://github.com/bigbio/pyspectrafuse"
LABEL about.license_file="https://github.com/bigbio/pyspectrafuse/blob/master/LICENSE"
Expand All @@ -21,7 +21,6 @@ MAINTAINER Yasset Perez-Riverol <ypriverol@gmail.com>
ENV DEBIAN_FRONTEND=noninteractive

# Disable numba caching to avoid issues in containerized environments
# Numba caching fails in containers because it can't locate source files in site-packages
ENV NUMBA_DISABLE_CACHING=1

## Update and install packages
Expand All @@ -31,20 +30,20 @@ RUN apt-get update -y && \
procps \
&& rm -rf /var/lib/apt/lists/*

## Install uv
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/

@coderabbitai coderabbitai Bot Apr 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🌐 Web query:

What is the recommended secure Docker pattern for ghcr.io/astral-sh/uv (pinning by explicit version and sha256 digest)?

💡 Result:

The recommended secure Docker pattern for ghcr.io/astral-sh/uv is to pin to an explicit version tag and further pin by SHA256 digest for reproducibility and security against tag movement. Official uv documentation provides this example in a multi-stage Dockerfile: FROM python:3.12-slim-trixie COPY --from=ghcr.io/astral-sh/uv:0.11.6 /uv /uvx /bin/ For maximum security, replace the tag with @sha256:: COPY --from=ghcr.io/astral-sh/uv@sha256:2381d6aa60c326b71fd40023f921a0a3b8f91b14d5db6b90402e65a635053709 /uv /uvx /bin/ To obtain the current digest for a specific tag (e.g., :0.4.17), run: docker manifest inspect ghcr.io/astral-sh/uv:0.4.17 (look for "digest" under the platform config). Available tags include :latest, :{version} (e.g., 0.11.6), and variants like :debian, :python3.12-trixie. View all at https://ghcr.io/astral-sh/uv. Images are signed with GitHub attestations; verify with gh attestation verify oci://ghcr.io/astral-sh/uv:0.11.6. This ensures supply chain security by preventing image substitution.

Citations:

Pin the uv image by explicit version tag and SHA256 digest.

The ghcr.io/astral-sh/uv:latest tag is non-reproducible and weakens supply-chain control. Use an explicit version tag combined with SHA256 digest pinning to ensure build reproducibility and prevent image substitution attacks.

Proposed change 
-COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
+COPY --from=ghcr.io/astral-sh/uv:<pinned-version>@sha256:<pinned-digest> /uv /uvx /bin/

Consult the official uv Docker integration guide for available version tags and instructions to retrieve the corresponding digest using docker manifest inspect.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@Dockerfile` at line 34, Replace the non-reproducible image reference used as
a build stage source in the COPY statement (currently
"ghcr.io/astral-sh/uv:latest" in the COPY --from=... line) with an explicit
version tag and its SHA256 digest (e.g.,
ghcr.io/astral-sh/uv:<version>@sha256:<digest>); obtain the correct tag and
digest using docker manifest inspect (or the uv Docker integration docs) and
update the COPY --from=... source to that pinned reference to ensure
reproducible, verifiable builds.


## Set working directory
WORKDIR /data/

## Copy requirements first for better caching
COPY requirements.txt .

## Install Python dependencies
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir -r requirements.txt

## Copy project files
COPY pyproject.toml .
COPY README.md .
COPY pyspectrafuse/ ./pyspectrafuse/

## Install the package
RUN pip install --no-cache-dir -e .
## Install the package with uv
RUN uv pip install --system --no-cache .
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

## No ENTRYPOINT — Nextflow runs commands via /bin/bash
## Use `pyspectrafuse` as the CLI command in scripts
CMD ["pyspectrafuse", "--help"]
Loading
Loading