feat: manifest generation tooling, workflow optimization, and Python 3.13/3.14 support #10
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- dotnet-install.py: Add retry logic (8 attempts) for JSON fetching to handle network flakes. - Makefile: Upgrade Trivy to v0.68.2 and enforce build failure on High/Critical vulnerabilities. Signed-off-by: Adilhusain Shaikh <[email protected]>
adilhusain-s
requested review from
anup-kodlekere,
mtarsel and
rahulssv-ibm
as code owners
- Add 'generate_partial_manifest.py' and 'apply_partial_manifests.py' scripts. - Add 'backfill-manifests.yml' workflow to process partial manifests. - Add unit tests for manifest generation and application logic. Signed-off-by: Adilhusain Shaikh <[email protected]> fix(tests): update error message assertion for invalid JSON handling Signed-off-by: Adilhusain Shaikh <[email protected]>
- release-matching-python-tags: Target Python 3.13.* and implement concurrency groups. - reusable-release-python-tar: Remove direct Git push logic; generate partial manifest artifacts instead. - release-matching-python-tags: Add 'update-manifests' job to aggregate partials and commit atomically. - Optimize 'max-parallel' and disable 'fail-fast' for better resilience. Signed-off-by: Adilhusain Shaikh <[email protected]>
- Drop legacy manifest files for Python 3.9, 3.10, 3.11, and 3.12. - Add and update manifest definitions for Python 3.13.x and 3.14.x on ppc64le and s390x architectures. Signed-off-by: Adilhusain Shaikh <[email protected]>
adilhusain-s
force-pushed
the
release-python-3.13.x
branch
from
299e57f to
cd1088f
Compare
Collaborator
Author
|
please review. |
Signed-off-by: Adilhusain Shaikh <[email protected]>
…gged URLs Signed-off-by: Adilhusain Shaikh <[email protected]>
Signed-off-by: Adilhusain Shaikh <[email protected]>
Signed-off-by: Adilhusain Shaikh <[email protected]>
…nd improve descriptions Signed-off-by: Adilhusain Shaikh <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
This PR introduces a robust set of tooling to automate partial manifest generation, optimizes CI/CD workflows to prevent race conditions during releases, and rotates the version manifest data to support Python 3.13 and 3.14.
It addresses flaky builds caused by network transient errors and decouples the artifact build process from the Git push operations.
Key Changes
Infrastructure & Security
• Reliability: Added configurable retry logic (8 attempts, 5s delay) to dotnet-install.py to handle transient network errors during dependency fetching.
• Security: Upgraded Trivy to v0.68.2 and enabled strict build failures (FAIL_ON_HIGH=1, FAIL_ON_CRITICAL=1) to ensure security standards are met before release.
• Cleanup: Simplified the Makefile by removing unnecessary sudo calls and streamlining the build commands.
New Tooling (Backfill & Manifests)
Introduced a new Python-based toolchain to handle manifest operations programmatically:
• generate_partial_manifest.py: Generates architecture-specific manifest JSONs using assets from GitHub Releases.
• apply_partial_manifests.py: Merges partial manifests into the main version files.
• backfill-manifests.yml: A new workflow to manually or conceptually trigger manifest updates for existing tags without rebuilding binaries.
• Testing: Added unit tests in tests/ to verify manifest generation logic.
CI/CD Architecture Refactor
Major refactoring of release-matching-python-tags.yml and reusable-release-python-tar.yml:
• Atomic Updates: Removed "git push" logic from the individual build matrix jobs. Instead, jobs now upload partial manifest artifacts.
• Aggregation: Added a new update-manifests job that runs after the build matrix completes. It downloads all partial artifacts, merges them, and commits them in a single atomic operation.
• Concurrency: Implemented concurrency groups (release-matching-${{ github.ref }}) to prevent overlapping runs from corrupting the git history.
• Resilience: Disabled fail-fast and tuned max-parallel to ensure temporary failures in one architecture (e.g., s390x) do not cancel builds for others.
Data Rotation
• Removed: Legacy manifest files for Python 3.9, 3.10, 3.11, and 3.12.
• Added/Updated: Full manifest definitions for Python 3.13.x and 3.14.x for ppc64le and s390x.
Technical Context
Why the workflow change? Previously, the reusable release workflow attempted to push to main from within the matrix strategy. If multiple architectures finished simultaneously, they would trigger race conditions, causing git push failures or merge conflicts. By moving to an Artifact -> Aggregate -> Commit pattern, we eliminate these race conditions and ensure a clean git history.
Verification
• [x] Unit Tests: Verified tests/test_generate_partial_manifest.py and tests/test_apply_partial_manifests.py.
• [x] Infrastructure: Validated make commands with the new Trivy version.
• [x] Workflows: Verified the Backfill workflow parses tags and generates artifacts correctly.