Added automatic build pipelines#3
Draft
florian-asche wants to merge 58 commits into
Draft
Conversation
This adds the docker/setup-qemu-action to enable QEMU emulation, allowing the kernel to be built for multiple target architectures in the CI pipeline.
- Switch to ubuntu-24.04-arm runner for native ARM builds - Add APT-cacher-ng proxy to cache package downloads - Implement cache restoration and saving between builds - Make QEMU setup conditional (only for 32bit builds) - Add debug logging for cache operations These changes significantly reduce build times by avoiding repeated package downloads and leveraging cached packages across workflow runs.
The build workflow for satellite1-rpi-setup now uses ubuntu-24.04-arm runner to natively build for ARM64 architecture, avoiding x86_64 emulation issues. APT caching is implemented using apt-cacher-ng to cache downloaded packages between runs, significantly reducing build time. The cache is persisted via actions/cache. QEMU is set up for 32-bit builds when matrix.name is '32bit'. These changes align with the kernel build workflow updates and improve overall build efficiency for ARM targets.
Add GitHub Release creation to kernel and setup build workflows. Releases are created as full releases for tags and pre-releases for branch builds. Added new workflows for release management and changelog generation.
Introduce a new GitHub Actions workflow for building satellite1 RPi Python components. Also disable automatic release note generation in existing kernel and RPi setup workflows.
Remove the upload-artifact step from the satellite1 RPi Python build workflow to streamline the CI process.
Initialize the docker directory for the satellite1 RPi component to support containerized builds.
Move debian packaging files from `satellite1-rpi/build/` to `satellite1-rpi/debian/` to follow standard project structure and improve organization.
Update the build process and project structure by: - Adjusting `pyproject.toml` to use the current directory for package discovery. - Modifying `Makefile` to correctly mount and set the working directory for Docker-based builds. - Updating `.gitignore` to exclude Python build artifacts like `dist/`, `build/`, and wheels.
- Move all core logic from `satellite1-rpi/satellite1/` to `satellite1-rpi/src/satellite1/`.
Update pyproject.toml to support the new src-based directory structure by adjusting setuptools package discovery. Additionally, define project scripts to expose CLI entry points for satellite control and audio routing.
Inject SETUPTOOLS_SCM_PRETEND_VERSION into the docker build and dpkg-buildpackage commands to ensure consistent versioning during the build process.
Remove manual version passing from the Makefile and instead derive the version directly from the Debian changelog within the debian/rules file. This ensures that the build process uses the official Debian versioning automatically.
Update the Makefile to copy the generated .deb files directly to the output directory and ensure the wheelhouse files are correctly positioned for the build process.
Refactor the root README.md to provide a high-level overview and package status table, and add individual README.md files to each sub-package to improve documentation discoverability and organization.
florian-asche
added
documentation
Upgrade the entire software stack from Debian Bookworm to Debian Trixie. This includes updating the custom Raspberry Pi kernel to the 6.18.y branch, updating Dockerfiles for image building and package construction, and updating all documentation and configuration files to reflect the new OS version and kernel release. - Update `image-builder` to use `debian:trixie` and `pi-gen` trixie branch - Update `rpi-kernel-fusb302` to use `debian:trixie` and `rpi-6.18.y` kernel - Update `satellite1-rpi-setup` with new Trixie-based Dockerfile - Update kernel version references in `Makefile` and `README.md` files - Update `README.md` to reflect Trixie as the target OS
…ncies Add libdw-dev and python3 to the Dockerfile to support kernel build processes and debugging symbol generation.
Update the Debian build process to abort immediately on recipe errors and implement a fallback mechanism in the SDK post-installation script. If the initial offline pip installation from the local wheelhouse fails, the script will now attempt to resolve missing dependencies via the online index.
Fix the spidev C-extension cross-compilation which was silently failing in the x86_64 → arm64 Docker build, leaving the wheelhouse without spidev and causing dpkg --install to fail with: ERROR: No matching distribution found for spidev>=3.6 Root causes: 1. .SHELLFLAGS := set -e; in debian/rules was broken syntax (replaces the -c flag, breaking multi-line recipe execution entirely). Using the standard POSIX form -e -c. 2. The Docker builder images were missing linux-libc-dev:arm64, which provides the linux/spi/spidev.h kernel-userspace header that spidev's C extension needs when cross-compiling from x86_64 to arm64. 3. python3-dev:arm64 and SETUPTOOLS_USE_DISTUTILS=stdlib are added to ensure the Python headers are available and spidev's legacy setup.py (which uses the deprecated install_layout) works with modern setuptools. 4. postinst: the fallback from wheelhouse install (--no-index) to online pip install was already present from c8cef18; that path is retained as a secondary safety net. Signed-off-by: Florian Asche <github@florian-asche.de>
Update spidev to version 3.8 and remove platform constraints from RPi.GPIO to ensure compatibility with the updated build environment.
Update `requires-python` to specify a range compatible with the recent OS upgrade, ensuring compatibility with Python 3.12 while preventing potential issues with Python 3.13.
Update the base images for both bookworm and trixie Dockerfiles from python:3.11-slim to python:3.12-slim to align with the updated python version requirements.
Expand the supported python version range to include 3.11, ensuring compatibility with environments that have not yet transitioned to 3.12.
Remove python3-venv, python3-pip, and related packaging tools from the debian build Dockerfiles as they are no longer required for the current build process.
Add python3-all, python3-setuptools, python3-wheel, and python3-setuptools-scm to the packaging stack in the Bookworm and Trixie Dockerfiles to ensure a complete build environment for python-based packages.
Ensure compatible versions of setuptools and setuptools-scm are installed within the build virtual environment to support modern packaging features.
Split setuptools-scm installation and add --no-build-isolation to pip wheel command to ensure consistent builds using the venv environment. Added PYTHONPATH to include venv site-packages and dist-packages during the wheelhouse creation process.
Update debian build rules to replace direct pip wheel execution with the `python3 -m build` module. This simplifies the build process and removes the need for manual PYTHONPATH manipulation. Added `python3-setuptools-scm` to Build-Depends and included `build` in the venv installation to support this transition.
Pass SETUPTOOLS_USE_DISTUTILS=stdlib to dh_auto_install to ensure compatibility with the existing installation process during the debian build lifecycle.
Introduce a dedicated build.env file to manage environment variables used during the Debian packaging process.
Modify the build target to ensure both the generated .deb files and the wheelhouse files are correctly copied to the output directory.
Include python3-setuptools-scm in the Dockerfiles for both bookworm and trixie arm64 images to support version discovery during the build process.
Update pyproject.toml to allow python versions greater than or equal to 3.11, removing the upper bound constraint of 3.13.
Update pydantic and pydantic-settings to newer versions to ensure compatibility and access to recent features/fixes.
Change pydantic dependency from a minimum version requirement to an exact version match to ensure environment consistency.
Update pydantic and pydantic-settings dependencies to allow for minor and patch version updates within the major version 2 range. This prevents dependency conflicts while maintaining compatibility.
Add missing commas to the pydantic dependency declarations to ensure valid TOML parsing.
Unset PIP_NO_INDEX during the retry attempt to allow the installer to fetch missing dependencies from the online index if the local wheelhouse installation fails.
…inst Update the postinst script to not only reload the systemd daemon but also enable and start the satellite1-init.service immediately.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces a complete automated CI/CD pipeline for the Satellite1-RPi project and restructures the repository to improve maintainability, packaging consistency, and release automation.
The changes focus on:
Main Changes
CI/CD & Build Automation
rpi-kernel-fusb302satellite1-rpi-setupsatellite1-rpiPython packageubuntu-24.04-armrunnersPackaging & Build System Improvements
src/layoutdebian/directories.debartifact handlingout/docker buildxRepository Cleanup & Refactoring
.gitignorefor Python build artifactsDocumentation
README.mdBenefits
Notes