Repository navigation
Add rerun-action-download-failures composite action #15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
ffb7cde
Add rerun-action-download-failures composite action
oshadmi ee83309
rerun-action-download-failures: quote the github-token description
oshadmi e5b79a4
rerun-action-download-failures: skip unless the run is confirmed latest
oshadmi 2aa378b
rerun-action-download-failures: re-check the latest run before retries
oshadmi File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| name: Re-run action download failures | ||
| description: | | ||
| Re-runs the failed jobs of a failed pull-request run, once, when every failure | ||
| is the runner failing to download an action archive in "Set up job". | ||
|
|
||
| The runner downloads every action a job references before any step runs, | ||
| including ones whose step `if:` later skips. actions/runner treats a 404 from | ||
| codeload.github.com as permanent and gives up after one attempt, so a | ||
| transient codeload error ("An action could not be found at the URI ...", | ||
| "Failed to download archive ... after 1 attempts") fails the job, and the | ||
| PR's gate job with it, even though the pinned SHA exists. No step-level retry | ||
| can help, because no step has started. | ||
|
|
||
| Call it from a job triggered by `workflow_run: types: [completed]` on the | ||
| repo's PR workflow. The job needs `actions: write` (read the run's jobs, call | ||
| rerun-failed-jobs) and `checks: read` (the job annotations carry the runner's | ||
| error), plus `python3` and `gh` on the runner. It needs no checkout. | ||
|
|
||
| It re-runs only when all of these hold, and otherwise exits successfully | ||
| without re-running: | ||
| - The run is for a pull request from a branch of the calling repository (so | ||
| no bot-initiated re-run of fork code), and it failed on its first attempt. | ||
| A persistent failure therefore stays red after one retry instead of looping. | ||
| Merge-queue runs are skipped: a failed one has already removed the PR from | ||
| the queue, and re-running it would not put the PR back. | ||
| - Every failed job other than `gate-job` carries the runner's download-failure | ||
| annotation. Any other failure means a human looks first. The gate fails | ||
| whenever anything it needs fails, and re-running failed jobs also re-runs | ||
| their dependents, so the gate is re-evaluated on the same run. | ||
| - Right before the re-run request, the run is still on attempt 1 and is | ||
| confirmed as the latest run of its workflow for the PR. Re-running a | ||
| superseded run would join the PR's concurrency group and cancel the newer | ||
| run, so a run that cannot be confirmed as the latest is not re-run. | ||
|
|
||
| inputs: | ||
| gate-job: | ||
| description: > | ||
| Name of the PR's gate job, which fails whenever the jobs it needs fail. | ||
| Its own failure is not required to carry the download error. | ||
| required: false | ||
| default: pr-validation | ||
| github-token: | ||
| description: > | ||
| Token with `actions: write` and `checks: read` on the calling repository. | ||
| required: false | ||
| default: ${{ github.token }} | ||
|
|
||
| runs: | ||
| using: composite | ||
| steps: | ||
| - name: Re-run failed jobs if all failed downloading an action | ||
| shell: bash | ||
| env: | ||
| GH_TOKEN: ${{ inputs.github-token }} | ||
| GATE_JOB: ${{ inputs.gate-job }} | ||
| run: python3 "$GITHUB_ACTION_PATH/rerun.py" |
125 changes: 125 additions & 0 deletions
125
.github/actions/rerun-action-download-failures/rerun.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,125 @@ | ||
| """Re-run a pull-request run's failed jobs when all of them failed downloading | ||
| an action. The conditions are documented in this directory's action.yml.""" | ||
|
|
||
| import json | ||
| import os | ||
| import re | ||
| import subprocess | ||
| import sys | ||
| import time | ||
| from urllib.parse import quote | ||
|
|
||
| # Messages the runner emits when an action archive download fails, | ||
| # whether on a 404 (one attempt) or after exhausting its retries. | ||
| DOWNLOAD_FAILURE = re.compile( | ||
| r"Failed to download archive '[^']+' after \d+ attempts" | ||
| r"|An action could not be found at the URI '" | ||
| ) | ||
| RERUN_ATTEMPTS = 3 | ||
|
|
||
| repo = os.environ["GITHUB_REPOSITORY"] | ||
| gate_job = os.environ["GATE_JOB"] | ||
|
|
||
| if os.environ["GITHUB_EVENT_NAME"] != "workflow_run": | ||
| sys.exit("rerun-action-download-failures must run in a workflow_run job") | ||
| with open(os.environ["GITHUB_EVENT_PATH"]) as f: | ||
| workflow_run = json.load(f)["workflow_run"] | ||
| run_id = workflow_run["id"] | ||
|
|
||
|
|
||
| def try_gh_api(path, *args): | ||
| # gh authenticates with GH_TOKEN, which action.yml sets from the github-token input. | ||
| result = subprocess.run( | ||
| ["gh", "api", *args, path], capture_output=True, text=True, | ||
| ) | ||
| if result.returncode != 0: | ||
| print(f"gh api {path} failed: {result.stderr.strip()}") | ||
| return False, None | ||
| return True, json.loads(result.stdout) if result.stdout.strip() else None | ||
|
|
||
|
|
||
| def gh_api(path, *args, attempts=5): | ||
| for attempt in range(1, attempts + 1): | ||
| ok, data = try_gh_api(path, *args) | ||
| if ok: | ||
| return data | ||
| if attempt < attempts: | ||
| time.sleep(2 * attempt) | ||
| sys.exit(f"gh api {path} failed after {attempts} attempts") | ||
|
|
||
|
|
||
| def skip(reason): | ||
| print(f"Not re-running run {run_id}: {reason}") | ||
| sys.exit(0) | ||
|
|
||
|
|
||
| def require_rerunnable(): | ||
| """Skip unless the run is still on attempt 1 and is the PR's latest run. | ||
|
|
||
| The event payload is a snapshot: a manual re-run or a new push can change | ||
| either since, and a stale re-run would cancel a newer run of the PR. | ||
| """ | ||
| run = gh_api(f"repos/{repo}/actions/runs/{run_id}") | ||
| if run["run_attempt"] != 1 or run["status"] != "completed": | ||
| skip(f"run is now attempt {run['run_attempt']}, status {run['status']}") | ||
| runs = gh_api( | ||
| f"repos/{repo}/actions/workflows/{workflow_run['workflow_id']}/runs" | ||
| f"?branch={quote(workflow_run['head_branch'], safe='')}&event=pull_request&per_page=20" | ||
| )["workflow_runs"] | ||
| pr_runs = sorted( | ||
| (r for r in runs | ||
| if any(p["number"] == pr_number for p in r.get("pull_requests") or [])), | ||
| key=lambda r: r["run_number"], reverse=True, | ||
| ) | ||
| # Fail closed: an empty or incomplete list cannot confirm this run is the newest. | ||
| if not pr_runs or pr_runs[0]["id"] != run_id: | ||
| newest = pr_runs[0]["id"] if pr_runs else "unknown" | ||
| skip(f"not confirmed as the latest run for PR #{pr_number} (newest: {newest})") | ||
|
|
||
|
|
||
| if workflow_run["conclusion"] != "failure": | ||
| skip(f"run concluded {workflow_run['conclusion']}") | ||
| if workflow_run["run_attempt"] != 1: | ||
| skip(f"run is on attempt {workflow_run['run_attempt']}") | ||
| if workflow_run["event"] != "pull_request": | ||
| skip(f"run was triggered by {workflow_run['event']}, not pull_request") | ||
| head_repo = (workflow_run.get("head_repository") or {}).get("full_name") | ||
| if head_repo != repo: | ||
| skip(f"head repository {head_repo} is not {repo}") | ||
| pull_requests = workflow_run.get("pull_requests") or [] | ||
| if not pull_requests: | ||
| skip("no pull request attached to the run") | ||
| pr_number = pull_requests[0]["number"] | ||
|
|
||
| jobs = gh_api( | ||
| f"repos/{repo}/actions/runs/{run_id}/attempts/1/jobs?per_page=100", | ||
| "--paginate", "--slurp", | ||
| ) | ||
| jobs = [j for page in jobs for j in page["jobs"]] | ||
| failed = [ | ||
| j for j in jobs | ||
| if j["conclusion"] in ("failure", "timed_out") and j["name"] != gate_job | ||
| ] | ||
| if not failed: | ||
| skip(f"no failed jobs besides {gate_job}") | ||
| for job in failed: | ||
| annotations = gh_api(f"repos/{repo}/check-runs/{job['id']}/annotations") or [] | ||
| if not any(DOWNLOAD_FAILURE.search(a.get("message") or "") for a in annotations): | ||
| skip(f"job '{job['name']}' failed for another reason") | ||
| print(f"Job '{job['name']}' failed downloading an action in setup.") | ||
|
|
||
| # Checked last, right before the POST, to keep the window for a manual re-run | ||
| # or a new push as short as the API allows. | ||
| require_rerunnable() | ||
|
|
||
| print(f"Re-running {len(failed)} failed job(s) of run {run_id} (PR #{pr_number}).") | ||
| for attempt in range(1, RERUN_ATTEMPTS + 1): | ||
| ok, _ = try_gh_api(f"repos/{repo}/actions/runs/{run_id}/rerun-failed-jobs", "-X", "POST") | ||
| if ok: | ||
| sys.exit(0) | ||
| if attempt < RERUN_ATTEMPTS: | ||
| time.sleep(2 * attempt) | ||
| # An error does not prove the re-run did not start, and the PR may have | ||
| # a newer run by now: a blind retry could start an attempt 3 or cancel it. | ||
| require_rerunnable() | ||
| sys.exit(f"rerun-failed-jobs failed after {RERUN_ATTEMPTS} attempts") | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.