Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 56 additions & 0 deletions .github/actions/rerun-action-download-failures/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
name: Re-run action download failures
description: |
Re-runs the failed jobs of a failed pull-request run, once, when every failure
is the runner failing to download an action archive in "Set up job".

The runner downloads every action a job references before any step runs,
including ones whose step `if:` later skips. actions/runner treats a 404 from
codeload.github.com as permanent and gives up after one attempt, so a
transient codeload error ("An action could not be found at the URI ...",
"Failed to download archive ... after 1 attempts") fails the job, and the
PR's gate job with it, even though the pinned SHA exists. No step-level retry
can help, because no step has started.

Call it from a job triggered by `workflow_run: types: [completed]` on the
repo's PR workflow. The job needs `actions: write` (read the run's jobs, call
rerun-failed-jobs) and `checks: read` (the job annotations carry the runner's
error), plus `python3` and `gh` on the runner. It needs no checkout.

It re-runs only when all of these hold, and otherwise exits successfully
without re-running:
- The run is for a pull request from a branch of the calling repository (so
no bot-initiated re-run of fork code), and it failed on its first attempt.
A persistent failure therefore stays red after one retry instead of looping.
Merge-queue runs are skipped: a failed one has already removed the PR from
the queue, and re-running it would not put the PR back.
- Every failed job other than `gate-job` carries the runner's download-failure
annotation. Any other failure means a human looks first. The gate fails
whenever anything it needs fails, and re-running failed jobs also re-runs
their dependents, so the gate is re-evaluated on the same run.
- Right before the re-run request, the run is still on attempt 1 and is
confirmed as the latest run of its workflow for the PR. Re-running a
superseded run would join the PR's concurrency group and cancel the newer
run, so a run that cannot be confirmed as the latest is not re-run.

inputs:
gate-job:
description: >
Name of the PR's gate job, which fails whenever the jobs it needs fail.
Its own failure is not required to carry the download error.
required: false
default: pr-validation
github-token:
description: >
Token with `actions: write` and `checks: read` on the calling repository.
required: false
default: ${{ github.token }}

runs:
using: composite
steps:
- name: Re-run failed jobs if all failed downloading an action
shell: bash
env:
GH_TOKEN: ${{ inputs.github-token }}
GATE_JOB: ${{ inputs.gate-job }}
run: python3 "$GITHUB_ACTION_PATH/rerun.py"
125 changes: 125 additions & 0 deletions .github/actions/rerun-action-download-failures/rerun.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
"""Re-run a pull-request run's failed jobs when all of them failed downloading
an action. The conditions are documented in this directory's action.yml."""

import json
import os
import re
import subprocess
import sys
import time
from urllib.parse import quote

# Messages the runner emits when an action archive download fails,
# whether on a 404 (one attempt) or after exhausting its retries.
DOWNLOAD_FAILURE = re.compile(
r"Failed to download archive '[^']+' after \d+ attempts"
r"|An action could not be found at the URI '"
)
RERUN_ATTEMPTS = 3

repo = os.environ["GITHUB_REPOSITORY"]
gate_job = os.environ["GATE_JOB"]

if os.environ["GITHUB_EVENT_NAME"] != "workflow_run":
sys.exit("rerun-action-download-failures must run in a workflow_run job")
with open(os.environ["GITHUB_EVENT_PATH"]) as f:
workflow_run = json.load(f)["workflow_run"]
run_id = workflow_run["id"]


def try_gh_api(path, *args):
# gh authenticates with GH_TOKEN, which action.yml sets from the github-token input.
result = subprocess.run(
["gh", "api", *args, path], capture_output=True, text=True,
Comment thread
oshadmi marked this conversation as resolved.
Resolved
)
if result.returncode != 0:
print(f"gh api {path} failed: {result.stderr.strip()}")
return False, None
return True, json.loads(result.stdout) if result.stdout.strip() else None


def gh_api(path, *args, attempts=5):
for attempt in range(1, attempts + 1):
ok, data = try_gh_api(path, *args)
if ok:
return data
if attempt < attempts:
time.sleep(2 * attempt)
sys.exit(f"gh api {path} failed after {attempts} attempts")


def skip(reason):
print(f"Not re-running run {run_id}: {reason}")
sys.exit(0)


def require_rerunnable():
"""Skip unless the run is still on attempt 1 and is the PR's latest run.

The event payload is a snapshot: a manual re-run or a new push can change
either since, and a stale re-run would cancel a newer run of the PR.
"""
run = gh_api(f"repos/{repo}/actions/runs/{run_id}")
if run["run_attempt"] != 1 or run["status"] != "completed":
skip(f"run is now attempt {run['run_attempt']}, status {run['status']}")
runs = gh_api(
f"repos/{repo}/actions/workflows/{workflow_run['workflow_id']}/runs"
f"?branch={quote(workflow_run['head_branch'], safe='')}&event=pull_request&per_page=20"
)["workflow_runs"]
pr_runs = sorted(
(r for r in runs
if any(p["number"] == pr_number for p in r.get("pull_requests") or [])),
key=lambda r: r["run_number"], reverse=True,
)
# Fail closed: an empty or incomplete list cannot confirm this run is the newest.
if not pr_runs or pr_runs[0]["id"] != run_id:
newest = pr_runs[0]["id"] if pr_runs else "unknown"
skip(f"not confirmed as the latest run for PR #{pr_number} (newest: {newest})")


if workflow_run["conclusion"] != "failure":
skip(f"run concluded {workflow_run['conclusion']}")
if workflow_run["run_attempt"] != 1:
skip(f"run is on attempt {workflow_run['run_attempt']}")
if workflow_run["event"] != "pull_request":
skip(f"run was triggered by {workflow_run['event']}, not pull_request")
head_repo = (workflow_run.get("head_repository") or {}).get("full_name")
if head_repo != repo:
skip(f"head repository {head_repo} is not {repo}")
pull_requests = workflow_run.get("pull_requests") or []
if not pull_requests:
skip("no pull request attached to the run")
pr_number = pull_requests[0]["number"]

jobs = gh_api(
f"repos/{repo}/actions/runs/{run_id}/attempts/1/jobs?per_page=100",
"--paginate", "--slurp",
)
jobs = [j for page in jobs for j in page["jobs"]]
failed = [
j for j in jobs
if j["conclusion"] in ("failure", "timed_out") and j["name"] != gate_job
]
if not failed:
skip(f"no failed jobs besides {gate_job}")
for job in failed:
annotations = gh_api(f"repos/{repo}/check-runs/{job['id']}/annotations") or []
if not any(DOWNLOAD_FAILURE.search(a.get("message") or "") for a in annotations):
skip(f"job '{job['name']}' failed for another reason")
print(f"Job '{job['name']}' failed downloading an action in setup.")

# Checked last, right before the POST, to keep the window for a manual re-run
# or a new push as short as the API allows.
require_rerunnable()

print(f"Re-running {len(failed)} failed job(s) of run {run_id} (PR #{pr_number}).")
for attempt in range(1, RERUN_ATTEMPTS + 1):
ok, _ = try_gh_api(f"repos/{repo}/actions/runs/{run_id}/rerun-failed-jobs", "-X", "POST")
if ok:
sys.exit(0)
if attempt < RERUN_ATTEMPTS:
time.sleep(2 * attempt)
# An error does not prove the re-run did not start, and the PR may have
# a newer run by now: a blind retry could start an attempt 3 or cancel it.
require_rerunnable()
sys.exit(f"rerun-failed-jobs failed after {RERUN_ATTEMPTS} attempts")
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ jobs:
python-version: '3.x'

- name: Compile Python helpers
run: python -m compileall -q scripts
run: python -m compileall -q scripts .github/actions

- name: Validate YAML (parse + reject duplicate keys)
run: |
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ plumbing.
| `setup-sccache` composite action | [`.github/actions/setup-sccache`](.github/actions/setup-sccache/action.yml) | Install sccache and, best-effort, point it at an S3 remote cache via AWS OIDC. The job must grant `id-token: write`; role/bucket/region/prefix are caller inputs. Fork PRs (no OIDC token) and missing inputs fall back to a local-only cache — never fails the job. `rustc-wrapper: 'true'` also exports `RUSTC_WRAPPER=sccache` / `CARGO_INCREMENTAL=0`. |
| `build-rejson` composite action | [`.github/actions/build-rejson`](.github/actions/build-rejson/action.yml) | Build RedisJSON for flow tests via RediSearch's `tests/deps/setup_rejson.sh` and cache `rejson.so`, keyed on RedisJSON SHA, platform, arch, sanitizer, coverage, C compiler + build flags and the pinned nightly. `redisearch-root` points at the RediSearch checkout (`.` or a submodule path). Returns `rejson-path`; also exports `REJSON_PATH` unless `export-rejson-path: 'false'`. Container-safe. |
| `changed-paths` composite action | [`.github/actions/changed-paths`](.github/actions/changed-paths/action.yml) | Gate CI on changed paths: tj-actions/changed-files' `any_modified` (same `files` / `files_ignore` inputs) plus submodule pointer moves, which changed-files drops when submodules aren't checked out. Read `changed`. Needs `fetch-depth: 0` and bash 4+. **Fails closed** (`changed=true` + warning) on an unresolvable range, forced push, unknown event or untranslatable glob, unlike the best-effort units here that never fail the job: for a skip-CI gate, running CI is the safe failure. |
| `rerun-action-download-failures` composite action | [`.github/actions/rerun-action-download-failures`](.github/actions/rerun-action-download-failures/action.yml) | Re-run a failed PR run's failed jobs, once, when every one of them failed downloading an action in "Set up job" (a transient codeload error the runner does not retry). Call it from a thin `on: workflow_run` caller of the repo's PR workflow with `actions: write` and `checks: read`; no checkout. Skips fork PRs, runs already re-run, superseded runs, and any run with another kind of failure. |
| `fast-track` reusable workflow | [`.github/workflows/fast-track.yml`](.github/workflows/fast-track.yml) | Approve an author's eligible PR based on its Jira `ai-fast-track` label. Uses a dedicated caller-scoped App; [policy, setup, and rollout](scripts/fast_track/README.md). |
| `pr-size-label` reusable workflow | [`.github/workflows/pr-size-label.yml`](.github/workflows/pr-size-label.yml) | Label a PR by diff size. Add a thin `on: pull_request` caller. |
| `spellcheck` reusable workflow | [`.github/workflows/spellcheck.yml`](.github/workflows/spellcheck.yml) | codespell over a PR's changed files. The repo supplies its own `.codespell/` config. |
Expand Down
Loading