Skip to content

fix: prevent release channel publication starvation #4806

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Danathar wants to merge 2 commits into from
Closed

fix: prevent release channel publication starvation #4806

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0081c23
fix: prevent release channel publication starvation
Danathar Aug 26, 2026
0450e81
🐛 fix(release): bound SHA tag retention and publish idempotently
clubanderson Aug 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
184 changes: 36 additions & 148 deletions .github/workflows/docker.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,10 @@ on:
# cancel in-progress builds: on a fast-merging release branch (v2 saw ~34 merges
# in 2h), cancel-in-progress meant every build was killed by the next merge and
# NONE ever completed, freezing <branch>-latest and blocking all deploys. With
# cancel-in-progress:false the builds queue and each runs to completion, so
# <branch>-latest always advances to the newest merged commit.
# cancel-in-progress:false the builds queue and each runs to completion. Each
# completed build publishes an immutable SHA tag; publish-image-tags.sh uses a
# per-tag workflow generation to advance moving tags without either starving
# during a merge burst or regressing when queued runs execute out of order.
concurrency:
group: docker-build-${{ github.ref_name }}
cancel-in-progress: false
Expand Down Expand Up @@ -143,6 +145,11 @@ jobs:
build-args: |
GIT_HASH=${{ github.sha }}
GIT_BRANCH=${{ github.ref_name }}
# Monotonic moving-tag guard metadata (#4804). This config label is
# inherited by each platform manifest without changing its media type.
labels: |
io.kubestellar.hive.github-actions-run-number=${{ github.run_number }}
org.opencontainers.image.revision=${{ github.sha }}
# STALE-BINARY FIX (unblocks the #3760 fix in the SHIPPED image).
# `no-cache-filters: builder` alone rebuilt
# a fresh /hive in the `builder` stage but let BuildKit serve the FINAL
Expand Down Expand Up @@ -269,6 +276,9 @@ jobs:
runs-on: ubuntu-latest
needs: [gate, build]
steps:
- name: Checkout tag publisher
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- name: Download digests
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
Expand All @@ -286,56 +296,12 @@ jobs:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Verify build commit is still HEAD of branch
id: head-check
run: |
BRANCH="${{ github.ref_name }}"
CURRENT_HEAD=$(git ls-remote https://github.com/${{ github.repository }} "$BRANCH" | awk '{print $1}')
if [ "${{ github.sha }}" != "$CURRENT_HEAD" ]; then
echo "Build is stale (built ${{ github.sha }} but $BRANCH HEAD is $CURRENT_HEAD) — skipping tag"
echo "stale=true" >> "$GITHUB_OUTPUT"
else
echo "stale=false" >> "$GITHUB_OUTPUT"
fi

- name: Extract git metadata
if: steps.head-check.outputs.stale != 'true'
id: meta
run: |
echo "git_short=${GITHUB_SHA::7}" >> "$GITHUB_OUTPUT"
# Docker tags can't contain '/', but branch names can (feat/x).
# Use the github.ref_name expression (reliably populated) and
# sanitize '/' -> '-' so feat/x becomes tag feat-x-latest.
REF_NAME='${{ github.ref_name }}'
echo "branch_tag=${REF_NAME//\//-}" >> "$GITHUB_OUTPUT"
# RELEASE CHANNELS. stable/candidate/edge are moving pointers that must
# always resolve to the newest good build of the RELEASE branch (v4).
# They are emitted here, in the same imagetools create that publishes
# <branch>-latest, so a channel is a RETAG of the digest we just built —
# never a rebuild, and never a manual promotion step that can be
# forgotten. Empty on every other branch, so a feature branch can never
# move a channel out from under production.
if [ "$REF_NAME" = "v4" ]; then
echo "release_channels=stable candidate edge" >> "$GITHUB_OUTPUT"
else
echo "release_channels=" >> "$GITHUB_OUTPUT"
fi

- name: Create manifest list and push
if: steps.head-check.outputs.stale != 'true'
working-directory: /tmp/digests
run: |
shopt -s nullglob
digests=(*)
if [ ${#digests[@]} -eq 0 ]; then
echo "No digests to merge (per-arch builds published none) — skipping."
exit 0
fi
docker buildx imagetools create \
-t ghcr.io/kubestellar/hive:${{ steps.meta.outputs.branch_tag }}-latest \
$(for c in ${{ steps.meta.outputs.release_channels }}; do printf -- '-t ghcr.io/kubestellar/hive:%s ' "$c"; done) \
-t ghcr.io/kubestellar/hive:${{ steps.meta.outputs.git_short }} \
$(printf 'ghcr.io/kubestellar/hive@sha256:%s ' "${digests[@]}")
src/scripts/publish-image-tags.sh \
ghcr.io/kubestellar/hive /tmp/digests \
'${{ github.ref_name }}' '${{ github.sha }}' '${{ github.run_number }}' \
v4 false

build-contributor:
needs: gate
Expand Down Expand Up @@ -374,6 +340,9 @@ jobs:
context: .
file: src/Dockerfile.contributor
platforms: ${{ matrix.platform }}
labels: |
io.kubestellar.hive.github-actions-run-number=${{ github.run_number }}
org.opencontainers.image.revision=${{ github.sha }}
# #3760: no default provenance/SBOM attestations → plain image
# manifests, not an OCI index. See the note on the main build above.
provenance: false
Expand Down Expand Up @@ -403,6 +372,9 @@ jobs:
runs-on: ubuntu-latest
needs: [gate, build-contributor]
steps:
- name: Checkout tag publisher
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- name: Download digests
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
Expand All @@ -420,57 +392,12 @@ jobs:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Verify build commit is still HEAD of branch
id: head-check
run: |
BRANCH="${{ github.ref_name }}"
CURRENT_HEAD=$(git ls-remote https://github.com/${{ github.repository }} "$BRANCH" | awk '{print $1}')
if [ "${{ github.sha }}" != "$CURRENT_HEAD" ]; then
echo "Build is stale (built ${{ github.sha }} but $BRANCH HEAD is $CURRENT_HEAD) — skipping tag"
echo "stale=true" >> "$GITHUB_OUTPUT"
else
echo "stale=false" >> "$GITHUB_OUTPUT"
fi

- name: Extract git metadata
if: steps.head-check.outputs.stale != 'true'
id: meta
run: |
echo "git_short=${GITHUB_SHA::7}" >> "$GITHUB_OUTPUT"
# Docker tags can't contain '/', but branch names can (feat/x).
# Use the github.ref_name expression (reliably populated) and
# sanitize '/' -> '-' so feat/x becomes tag feat-x-latest.
REF_NAME='${{ github.ref_name }}'
echo "branch_tag=${REF_NAME//\//-}" >> "$GITHUB_OUTPUT"
# RELEASE CHANNELS. stable/candidate/edge are moving pointers that must
# always resolve to the newest good build of the RELEASE branch (v4).
# They are emitted here, in the same imagetools create that publishes
# <branch>-latest, so a channel is a RETAG of the digest we just built —
# never a rebuild, and never a manual promotion step that can be
# forgotten. Empty on every other branch, so a feature branch can never
# move a channel out from under production.
if [ "$REF_NAME" = "v4" ]; then
echo "release_channels=stable candidate edge" >> "$GITHUB_OUTPUT"
else
echo "release_channels=" >> "$GITHUB_OUTPUT"
fi

- name: Create contributor manifest list and push
if: steps.head-check.outputs.stale != 'true'
working-directory: /tmp/contrib-digests
run: |
shopt -s nullglob
digests=(*)
if [ ${#digests[@]} -eq 0 ]; then
echo "No digests to merge (per-arch builds published none) — skipping."
exit 0
fi
docker buildx imagetools create \
-t ghcr.io/kubestellar/hive-contributor:latest \
-t ghcr.io/kubestellar/hive-contributor:${{ steps.meta.outputs.branch_tag }}-latest \
$(for c in ${{ steps.meta.outputs.release_channels }}; do printf -- '-t ghcr.io/kubestellar/hive-contributor:%s ' "$c"; done) \
-t ghcr.io/kubestellar/hive-contributor:${{ steps.meta.outputs.git_short }} \
$(printf 'ghcr.io/kubestellar/hive-contributor@sha256:%s ' "${digests[@]}")
src/scripts/publish-image-tags.sh \
ghcr.io/kubestellar/hive-contributor /tmp/contrib-digests \
'${{ github.ref_name }}' '${{ github.sha }}' '${{ github.run_number }}' \
v4 true

build-hub:
needs: gate
Expand Down Expand Up @@ -512,6 +439,9 @@ jobs:
build-args: |
GIT_HASH=${{ github.sha }}
GIT_BRANCH=${{ github.ref_name }}
labels: |
io.kubestellar.hive.github-actions-run-number=${{ github.run_number }}
org.opencontainers.image.revision=${{ github.sha }}
# STALE-BINARY FIX (see the main build step). The hub image also does
# `COPY --from=builder /hive` into a `runtime` final stage, and its GHA
# cache scope (hub-<slug>) is likewise branch-persistent — so without
Expand Down Expand Up @@ -547,6 +477,9 @@ jobs:
runs-on: ubuntu-latest
needs: [gate, build-hub]
steps:
- name: Checkout tag publisher
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- name: Download digests
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
Expand All @@ -564,54 +497,9 @@ jobs:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Verify build commit is still HEAD of branch
id: head-check
run: |
BRANCH="${{ github.ref_name }}"
CURRENT_HEAD=$(git ls-remote https://github.com/${{ github.repository }} "$BRANCH" | awk '{print $1}')
if [ "${{ github.sha }}" != "$CURRENT_HEAD" ]; then
echo "Build is stale (built ${{ github.sha }} but $BRANCH HEAD is $CURRENT_HEAD) — skipping tag"
echo "stale=true" >> "$GITHUB_OUTPUT"
else
echo "stale=false" >> "$GITHUB_OUTPUT"
fi

- name: Extract git metadata
if: steps.head-check.outputs.stale != 'true'
id: meta
run: |
echo "git_short=${GITHUB_SHA::7}" >> "$GITHUB_OUTPUT"
# Docker tags can't contain '/', but branch names can (feat/x).
# Use the github.ref_name expression (reliably populated) and
# sanitize '/' -> '-' so feat/x becomes tag feat-x-latest.
REF_NAME='${{ github.ref_name }}'
echo "branch_tag=${REF_NAME//\//-}" >> "$GITHUB_OUTPUT"
# RELEASE CHANNELS. stable/candidate/edge are moving pointers that must
# always resolve to the newest good build of the RELEASE branch (v4).
# They are emitted here, in the same imagetools create that publishes
# <branch>-latest, so a channel is a RETAG of the digest we just built —
# never a rebuild, and never a manual promotion step that can be
# forgotten. Empty on every other branch, so a feature branch can never
# move a channel out from under production.
if [ "$REF_NAME" = "v4" ]; then
echo "release_channels=stable candidate edge" >> "$GITHUB_OUTPUT"
else
echo "release_channels=" >> "$GITHUB_OUTPUT"
fi

- name: Create hub manifest list and push
if: steps.head-check.outputs.stale != 'true'
working-directory: /tmp/hub-digests
run: |
shopt -s nullglob
digests=(*)
if [ ${#digests[@]} -eq 0 ]; then
echo "No digests to merge (per-arch builds published none) — skipping."
exit 0
fi
docker buildx imagetools create \
-t ghcr.io/kubestellar/hive-hub:latest \
-t ghcr.io/kubestellar/hive-hub:${{ steps.meta.outputs.branch_tag }}-latest \
$(for c in ${{ steps.meta.outputs.release_channels }}; do printf -- '-t ghcr.io/kubestellar/hive-hub:%s ' "$c"; done) \
-t ghcr.io/kubestellar/hive-hub:${{ steps.meta.outputs.git_short }} \
$(printf 'ghcr.io/kubestellar/hive-hub@sha256:%s ' "${digests[@]}")
src/scripts/publish-image-tags.sh \
ghcr.io/kubestellar/hive-hub /tmp/hub-digests \
'${{ github.ref_name }}' '${{ github.sha }}' '${{ github.run_number }}' \
v4 true
53 changes: 53 additions & 0 deletions .github/workflows/prune-ghcr.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,3 +116,56 @@ jobs:
done
done
[ "$DRY_RUN" = "true" ] && echo "(dry-run: nothing deleted above)" || true

# The docker workflow publishes an immutable short-SHA tag for every
# successful long-lived-branch build so merge bursts cannot leave completed
# commits without an addressable image. Keep that history bounded: once all
# moving tags have advanced away from an old build, the remaining pure
# 7-hex-SHA version is only a rollback/debug affordance and is pruned after
# the retention window below. Versions that still carry any non-SHA tag
# (v4-latest, stable, candidate, edge, latest, or future channel names) are
# never deleted by this job.
prune-old-sha-tags:
runs-on: ubuntu-latest
steps:
- name: Delete old pure short-SHA image tags
env:
GH_TOKEN: ${{ secrets.GHCR_PRUNE_TOKEN }}
DRY_RUN: ${{ github.event_name == 'schedule' && 'false' || inputs.dry_run }}
RETENTION_DAYS: "90"
run: |
set -euo pipefail
cutoff=$(date -u -d "$RETENTION_DAYS days ago" +%s)
for image in hive hive-hub hive-contributor; do
echo "=== image: $image (pure short-SHA tags older than ${RETENTION_DAYS}d) ==="
gh api --paginate \
"/orgs/kubestellar/packages/container/$image/versions" \
--jq '.[] | {id: .id, created_at: .created_at, tags: .metadata.container.tags}' \
| jq -c '.' | while read -r row; do
id=$(echo "$row" | jq -r '.id')
created_at=$(echo "$row" | jq -r '.created_at')
mapfile -t tags < <(echo "$row" | jq -r '.tags[]?')
[ "${#tags[@]}" -gt 0 ] || continue

all_short_sha=true
for t in "${tags[@]}"; do
if ! [[ $t =~ ^[0-9a-f]{7}$ ]]; then
all_short_sha=false
break
fi
done
[ "$all_short_sha" = true ] || continue

created_epoch=$(date -u -d "$created_at" +%s)
[ "$created_epoch" -lt "$cutoff" ] || continue

joined=$(printf '%s,' "${tags[@]}")
echo "old pure-SHA version $id created=$created_at tags=[${joined%,}]"
if [ "$DRY_RUN" != "true" ]; then
gh api -X DELETE \
"/orgs/kubestellar/packages/container/$image/versions/$id" \
&& echo " deleted $id" || echo " FAILED $id"
fi
done
done
[ "$DRY_RUN" = "true" ] && echo "(dry-run: nothing deleted above)" || true
11 changes: 9 additions & 2 deletions .github/workflows/v2-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ on:
# not run on the one change that can break it — which is how the quick start
# shipped broken twice over. src/docs/** already triggers this workflow via
# src/**, so this is the same rule applied to the docs outside src/.
paths: ['src/**', 'bin/**', 'README.md', 'docs/**', '.github/workflows/v2-ci.yml', '.github/workflows/changelog-reminder.yml']
paths: ['src/**', 'bin/**', 'README.md', 'docs/**', '.github/workflows/v2-ci.yml', '.github/workflows/changelog-reminder.yml', '.github/workflows/docker.yml']
pull_request:
branches: [v2, v4]
# bin/** is in scope because the gh wrapper is a SECURITY boundary (it is the
Expand All @@ -26,7 +26,7 @@ on:
# not run on the one change that can break it — which is how the quick start
# shipped broken twice over. src/docs/** already triggers this workflow via
# src/**, so this is the same rule applied to the docs outside src/.
paths: ['src/**', 'bin/**', 'README.md', 'docs/**', '.github/workflows/v2-ci.yml', '.github/workflows/changelog-reminder.yml']
paths: ['src/**', 'bin/**', 'README.md', 'docs/**', '.github/workflows/v2-ci.yml', '.github/workflows/changelog-reminder.yml', '.github/workflows/docker.yml']

permissions:
contents: read
Expand Down Expand Up @@ -100,6 +100,13 @@ jobs:
- name: Supply-chain pin guard
run: bash deploy/test_supply_chain_pins.sh

# #4804: HEAD-only tagging made every completed build in a merge burst
# exit green without publishing anything. Exercise the real publisher
# against a fake registry and pin both halves of the replacement policy:
# immutable SHA tags are unconditional, while moving tags are monotonic.
- name: Docker release-tag publication policy (#4804)
run: bash scripts/test-publish-image-tags.sh

# #4206: the standalone stack's image references live in exactly one
# place, src/deploy/standalone-images.sh, so the Docker Compose assets
# and the Podman assets that land later cannot drift onto different
Expand Down
4 changes: 4 additions & 0 deletions src/docs/release-channels.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,10 @@ Channels are **retags, not rebuilds**. The `docker.yml` workflow adds `stable`,

Only builds of branch `v4` publish channels — a feature-branch build can never move a production channel.

Publishing is monotonic by workflow run number. Every successful multi-arch build receives its immutable short-SHA tag even if a newer merge has already reached the branch. If that exact short-SHA tag already exists, a re-run leaves it untouched. Moving tags (`v4-latest` and the three channels) advance only when that build is newer than the generation currently published; an older workflow that runs out of queue order publishes only any missing immutable tag. This avoids both failure modes of a HEAD-only guard: a merge burst cannot starve all tags, and an old queued build cannot move a channel backwards. Registry inspection failures fail the publish job instead of producing a silent green skip.

Short-SHA tags are retained as a bounded rollback/debug window, not forever. The scheduled GHCR pruning workflow deletes only old package versions whose complete tag set is one or more 7-hex short-SHA tags, after 90 days. Versions still carrying any moving tag (`v4-latest`, `latest`, `stable`, `candidate`, `edge`, or future channel names) are never deleted by that cleanup.

## Switching a hive to a channel

From the hub dashboard's **My Hives** list, click the blue version pill on a hive row. The menu lists branches first, then a **Channels** section with the three channels (most stable first). Only the hive's **owner** can switch.
Expand Down
Loading
Loading