Skip to content

fix: prevent release channel publication starvation - #4806

Closed
Danathar wants to merge 2 commits into
kubestellar:v4from
Danathar:fix/4804-release-channel-starvation
Closed

fix: prevent release channel publication starvation#4806
Danathar wants to merge 2 commits into
kubestellar:v4from
Danathar:fix/4804-release-channel-starvation

Conversation

@Danathar

Copy link
Copy Markdown
Contributor

Summary

  • replace the HEAD-only release guard that silently skipped every completed build in a merge burst
  • publish an immutable short-SHA tag for every successful multi-arch build
  • advance each moving tag independently using a monotonic GitHub workflow run-number label
  • fail red on registry inspection errors instead of reporting a green no-publish run
  • apply the policy consistently to hive, contributor, and hub images

Why this cannot regress tags

Each platform image carries io.kubestellar.hive.github-actions-run-number. Before moving a mutable tag, the publisher reads the generation already attached to that tag. A queued run may advance tags whose published generation is older, but it leaves independently newer tags untouched. Re-runs of the same workflow generation remain idempotent.

Existing unlabeled tags are generation 0, so the first post-merge run migrates them without manual registry work. The exact Buildx query was verified against the current public stable image, which correctly returned an empty legacy label set.

Validation

  • shellcheck src/scripts/publish-image-tags.sh src/scripts/test-publish-image-tags.sh
  • bash src/scripts/test-publish-image-tags.sh
  • bash src/scripts/check-release-lines.sh
  • bash src/scripts/test-release-lines-guard.sh
  • parsed both changed workflow files with yq
  • git diff --check

The publisher tests cover missing and legacy tags, forward progress, stale out-of-order runs, per-tag independence, feature-branch channel isolation, and fail-closed registry errors. The secondary dashboard ? behind presentation noted in the issue is intentionally left for a separate UI change.

Fixes #4804

— hive: backend=codex

@Danathar
Danathar requested a review from clubanderson as a code owner August 26, 2026 15:46
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Aug 26, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign clubanderson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Aug 26, 2026
@clubanderson
clubanderson force-pushed the fix/4804-release-channel-starvation branch from 746050b to 674274d Compare August 26, 2026 16:38
@clubanderson

Copy link
Copy Markdown
Contributor

Reopening to refresh CI after a reviewer fix-forward commit on the PR branch.

Danathar and others added 2 commits August 26, 2026 13:29
Signed-off-by: Danathar <doug.baggett@gmail.com>
Keep immutable short-SHA image tags immutable across workflow reruns by leaving an existing SHA tag untouched, and only advance moving tags when the current workflow generation is strictly newer.

Add a scheduled GHCR cleanup path for old pure short-SHA versions so per-build tags remain bounded while moving tags and channels stay protected.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@clubanderson
clubanderson force-pushed the fix/4804-release-channel-starvation branch from f9979f5 to 0450e81 Compare August 26, 2026 17:30
@clubanderson

Copy link
Copy Markdown
Contributor

Thanks, Danathar, for identifying and fixing the release-channel publication starvation path. I carried the core fix forward through #4811, including idempotent immutable short-SHA publishing for successful builds and a 90-day cleanup workflow for old pure short-SHA image tags to keep growth bounded.

#4811 has now merged as f992bfa, so I am closing this PR as superseded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Version badge shows '✓' and '? behind' at the same time when the branch tip has no image

2 participants