Skip to content

Commit 8a6416f

Browse files
Avinash-RajAvinash-Raj
authored
CI workflow refactoring (#277)
**Refactor CI Images workflow:** - Splitted up `build.yml` and `test.yml` into `velox-build.yml`, `presto-build.yml`, `velox-test.yml`, `presto-test.yml` - Renamed the existing `presto-test.yml`, `velox-test.yml` into `presto-standalone-test.yml`, `velox-standalone-test.yml` (which could be removed later)
1 parent cc2774b commit 8a6416f

18 files changed

+1837
-1456
lines changed

‎.github/actions/resolve-commits/action.yml‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
name: Resolve Commits
2+
description: >
3+
Resolves Velox and Presto commit SHAs from refs/branches, including
4+
"presto-pinned" which reads the Velox submodule pin from Presto.
5+
Also sets the build date (YYYYMMDD UTC).
6+
7+
inputs:
8+
velox_repository:
9+
description: 'Velox repository (e.g. facebookincubator/velox)'
10+
required: true
11+
velox_commit:
12+
description: 'Velox commit SHA, branch, or "presto-pinned"'
13+
required: true
14+
presto_repository:
15+
description: 'Presto repository (e.g. prestodb/presto)'
16+
required: true
17+
presto_commit:
18+
description: 'Presto commit SHA or branch'
19+
required: true
20+
github_token:
21+
description: 'GitHub token for API calls'
22+
required: true
23+
24+
outputs:
25+
velox_sha:
26+
description: 'Resolved full Velox commit SHA'
27+
value: ${{ steps.velox.outputs.sha }}
28+
velox_short_sha:
29+
description: 'Resolved short Velox commit SHA (7 chars)'
30+
value: ${{ steps.velox.outputs.short_sha }}
31+
presto_sha:
32+
description: 'Resolved full Presto commit SHA'
33+
value: ${{ steps.presto.outputs.sha }}
34+
presto_short_sha:
35+
description: 'Resolved short Presto commit SHA (7 chars)'
36+
value: ${{ steps.presto.outputs.short_sha }}
37+
date:
38+
description: 'Build date (YYYYMMDD UTC)'
39+
value: ${{ steps.date.outputs.date }}
40+
41+
runs:
42+
using: composite
43+
steps:
44+
- name: Resolve Presto commit
45+
id: presto
46+
shell: bash
47+
env:
48+
GH_TOKEN: ${{ inputs.github_token }}
49+
PRESTO_REF: ${{ inputs.presto_commit }}
50+
PRESTO_REPO: ${{ inputs.presto_repository }}
51+
run: |
52+
SHA=$(git ls-remote "https://github.com/${PRESTO_REPO}.git" "${PRESTO_REF}" | head -1 | cut -f1)
53+
if [[ -z "$SHA" ]]; then
54+
SHA=$(gh api "repos/${PRESTO_REPO}/commits/${PRESTO_REF}" --jq '.sha')
55+
fi
56+
echo "sha=${SHA}" >> "$GITHUB_OUTPUT"
57+
echo "short_sha=${SHA:0:7}" >> "$GITHUB_OUTPUT"
58+
echo "Presto SHA: ${SHA} (short: ${SHA:0:7})"
59+
60+
- name: Resolve Velox commit
61+
id: velox
62+
shell: bash
63+
env:
64+
GH_TOKEN: ${{ inputs.github_token }}
65+
VELOX_REF: ${{ inputs.velox_commit }}
66+
VELOX_REPO: ${{ inputs.velox_repository }}
67+
PRESTO_REPO: ${{ inputs.presto_repository }}
68+
PRESTO_SHA: ${{ steps.presto.outputs.sha }}
69+
run: |
70+
if [[ "${VELOX_REF}" == "presto-pinned" ]]; then
71+
SHA=$(gh api "repos/${PRESTO_REPO}/contents/presto-native-execution/velox?ref=${PRESTO_SHA}" --jq '.sha')
72+
echo "Resolved Velox SHA from Presto submodule pin: ${SHA}"
73+
else
74+
SHA=$(git ls-remote "https://github.com/${VELOX_REPO}.git" "${VELOX_REF}" | head -1 | cut -f1)
75+
if [[ -z "$SHA" ]]; then
76+
SHA=$(gh api "repos/${VELOX_REPO}/commits/${VELOX_REF}" --jq '.sha')
77+
fi
78+
fi
79+
echo "sha=${SHA}" >> "$GITHUB_OUTPUT"
80+
echo "short_sha=${SHA:0:7}" >> "$GITHUB_OUTPUT"
81+
echo "Velox SHA: ${SHA} (short: ${SHA:0:7})"
82+
83+
- name: Set build date
84+
id: date
85+
shell: bash
86+
run: echo "date=$(date -u +%Y%m%d)" >> "$GITHUB_OUTPUT"

‎.github/actions/resolve-inputs/action.yml‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
name: Resolve Test Inputs
2+
description: >
3+
Parses a full image tag to extract SHAs and date, or passes through
4+
directly provided values. Supports both presto format
5+
(presto-{SHA}-velox-{SHA}-...-{DATE}) and velox format
6+
(velox-{SHA}-...-{DATE}).
7+
8+
inputs:
9+
image_tag:
10+
description: 'Full image tag to parse (optional). If empty, direct inputs are used.'
11+
required: false
12+
default: ''
13+
velox_short_sha:
14+
description: 'Short Velox commit SHA (7 chars)'
15+
required: false
16+
default: ''
17+
presto_short_sha:
18+
description: 'Short Presto commit SHA (7 chars)'
19+
required: false
20+
default: ''
21+
date:
22+
description: 'Build date (YYYYMMDD)'
23+
required: false
24+
default: ''
25+
26+
outputs:
27+
velox_short_sha:
28+
description: 'Resolved short Velox commit SHA'
29+
value: ${{ steps.resolve.outputs.velox_short_sha }}
30+
presto_short_sha:
31+
description: 'Resolved short Presto commit SHA'
32+
value: ${{ steps.resolve.outputs.presto_short_sha }}
33+
date:
34+
description: 'Resolved build date'
35+
value: ${{ steps.resolve.outputs.date }}
36+
37+
runs:
38+
using: composite
39+
steps:
40+
- name: Resolve test inputs
41+
id: resolve
42+
shell: bash
43+
env:
44+
IMAGE_TAG: ${{ inputs.image_tag }}
45+
VELOX_SHORT_SHA: ${{ inputs.velox_short_sha }}
46+
PRESTO_SHORT_SHA: ${{ inputs.presto_short_sha }}
47+
DATE: ${{ inputs.date }}
48+
run: |
49+
if [[ -n "${IMAGE_TAG}" ]]; then
50+
TAG="${IMAGE_TAG##*:}"
51+
echo "Parsing tag: ${TAG}"
52+
53+
# Presto format: presto-{SHA}-velox-{SHA}-...-{DATE}[-arch]
54+
if [[ "$TAG" =~ ^presto-([a-f0-9]+)-velox-([a-f0-9]+)-.*-([0-9]{8})(-[a-z0-9]+)?$ ]]; then
55+
echo "presto_short_sha=${BASH_REMATCH[1]}" >> "$GITHUB_OUTPUT"
56+
echo "velox_short_sha=${BASH_REMATCH[2]}" >> "$GITHUB_OUTPUT"
57+
echo "date=${BASH_REMATCH[3]}" >> "$GITHUB_OUTPUT"
58+
echo "Parsed presto image: presto=${BASH_REMATCH[1]}, velox=${BASH_REMATCH[2]}, date=${BASH_REMATCH[3]}"
59+
60+
# Velox format: velox-{SHA}-...-{DATE}[-arch]
61+
elif [[ "$TAG" =~ ^velox-([a-f0-9]+)-.*-([0-9]{8})(-[a-z0-9]+)?$ ]]; then
62+
echo "velox_short_sha=${BASH_REMATCH[1]}" >> "$GITHUB_OUTPUT"
63+
echo "presto_short_sha=" >> "$GITHUB_OUTPUT"
64+
echo "date=${BASH_REMATCH[2]}" >> "$GITHUB_OUTPUT"
65+
echo "Parsed velox image: velox=${BASH_REMATCH[1]}, date=${BASH_REMATCH[2]}"
66+
67+
else
68+
echo "::error::Cannot parse image tag '${TAG}'. Expected presto-{SHA}-velox-{SHA}-...-{DATE} or velox-{SHA}-...-{DATE}"
69+
exit 1
70+
fi
71+
else
72+
if [[ -z "${VELOX_SHORT_SHA}" || -z "${DATE}" ]]; then
73+
echo "::error::Either image_tag or at minimum velox_short_sha and date must be provided"
74+
exit 1
75+
fi
76+
echo "velox_short_sha=${VELOX_SHORT_SHA}" >> "$GITHUB_OUTPUT"
77+
echo "presto_short_sha=${PRESTO_SHORT_SHA}" >> "$GITHUB_OUTPUT"
78+
echo "date=${DATE}" >> "$GITHUB_OUTPUT"
79+
fi

‎.github/workflows/README.md‎

Lines changed: 49 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ This directory contains GitHub Actions workflows for automated testing, benchmar
1111
| `velox-create-staging.yml` | Creates Velox staging branch by merging cuDF PRs | Auto-fetches PRs with `cudf` label by default |
1212
| `presto-create-staging.yml` | Creates Presto staging branch by merging specified PRs | Requires manual PR numbers (no auto-fetch) |
1313
| **Velox Testing** |||
14-
| `velox-test.yml` | Builds and tests Velox (CPU and/or GPU) | Foundation workflow called by nightly jobs |
14+
| `velox-standalone-test.yml` | Builds and tests Velox (CPU and/or GPU) | Foundation workflow called by nightly jobs |
1515
| `velox-nightly-upstream.yml` | Nightly tests against upstream `facebookincubator/velox:main` | Early detection of upstream regressions |
1616
| `velox-nightly-staging.yml` | Nightly tests against the staging branch | Validates staging before cuDF merges |
1717
| **Velox Benchmarking** |||
@@ -20,19 +20,21 @@ This directory contains GitHub Actions workflows for automated testing, benchmar
2020
| **Velox Dependencies** |||
2121
| `velox-deps-upload.yml` | Builds and uploads Velox dependencies image to S3 | Run when base dependencies change |
2222
| **Presto Testing** |||
23-
| `presto-test.yml` | Tests Presto with Java, Native CPU, or Native GPU workers | Supports multiple worker configurations |
24-
| `presto-test-composite.yml` | Reusable workflow implementing Presto test logic | Called by presto-test.yml |
23+
| `presto-standalone-test.yml` | Tests Presto with Java, Native CPU, or Native GPU workers | Supports multiple worker configurations |
24+
| `presto-test-composite.yml` | Reusable workflow implementing Presto test logic | Called by presto-standalone-test.yml |
2525
| `presto-nightly-upstream.yml` | Nightly tests against upstream Presto and Velox | Tests latest upstream compatibility |
2626
| `presto-nightly-staging.yml` | Nightly tests against staging/stable versions | Validates known-good configurations |
2727
| `presto-nightly-pinned.yml` | Nightly tests using Presto's pinned Velox version | Tests exact Velox commit Presto depends on |
2828
| **CI Images** |||
2929
| `ci-images-nightly.yml` | Nightly builds of CI images for upstream, pinned, and staging | Schedule only (5am UTC) |
3030
| `ci-images-manual.yml` | Manual builds of CI images with user-specified inputs | `workflow_dispatch` only |
31-
| `build.yml` | Reusable workflow implementing CI image build logic | Called by ci-images workflows |
32-
| `test.yml` | Reusable workflow implementing CI image test logic | Called by ci-images workflows; also supports `workflow_dispatch` for test-only runs |
31+
| `actions/resolve-commits/` | Composite action to resolve Velox/Presto commit SHAs | Used by ci-images workflows |
32+
| `actions/resolve-inputs/` | Composite action to parse image tags into SHAs + date | Used by test workflows for `workflow_dispatch` |
33+
| `velox-build.yml` | Reusable workflow for Velox CI image builds + merge | Builds deps + build images, creates multi-arch manifests |
34+
| `presto-build.yml` | Reusable workflow for Presto CI image builds + merge | Builds deps + build + coordinator, creates multi-arch manifests |
35+
| `velox-test.yml` | Reusable workflow for Velox CI image tests | CPU + GPU tests; supports `workflow_dispatch` for test-only runs |
36+
| `presto-test.yml` | Reusable workflow for Presto CI image tests | Smoke + integration tests; supports `workflow_dispatch` for test-only runs |
3337
| `ci-image-cleanup.yml` | Deletes CI images older than 30 days from GHCR | Weekly (Tuesdays) + manual dispatch |
34-
| **Preliminary Checks** |||
35-
| `preliminary-checks.yml` | Runs tests when specific directories change | Triggers on `benchmark_data_tools/` or `presto/` changes |
3638

3739
---
3840

@@ -103,7 +105,7 @@ The additional merge happens **after** the base reset but **before** PR merging.
103105

104106
## Velox Testing Workflows
105107

106-
### `velox-test.yml`
108+
### `velox-standalone-test.yml`
107109
Core build and test workflow for Velox. Supports CPU-only, GPU-only, or both targets. Used as a foundation by nightly workflows.
108110

109111
### `velox-nightly-upstream.yml`
@@ -132,16 +134,38 @@ Multi-arch (amd64/arm64) Docker images for Velox and Presto are built and publis
132134

133135
- **`ci-images-nightly.yml`** — Nightly schedule (5am UTC). Builds images for three source combinations in parallel:
134136

135-
| Job | Velox | Presto |
136-
|-----|-------|--------|
137-
| `nightly-upstream` | `facebookincubator/velox:main` | `prestodb/presto:master` |
138-
| `nightly-pinned` | Presto's pinned Velox submodule | `prestodb/presto:master` |
139-
| `nightly-staging` | `$STABLE_VELOX_REPO:$STABLE_VELOX_COMMIT` | `$STABLE_PRESTO_REPO:$STABLE_PRESTO_COMMIT` |
137+
| Variant | Velox | Presto |
138+
|---------|-------|--------|
139+
| `upstream` | `facebookincubator/velox:main` | `prestodb/presto:master` |
140+
| `pinned` | Presto's pinned Velox submodule | `prestodb/presto:master` |
141+
| `staging` | `$STABLE_VELOX_REPO:$STABLE_VELOX_COMMIT` | `$STABLE_PRESTO_REPO:$STABLE_PRESTO_COMMIT` |
140142

141-
A new nightly run cancels any in-progress nightly run.
143+
Per variant, velox and presto builds run in parallel. A new nightly run cancels any in-progress nightly run.
142144

143145
- **`ci-images-manual.yml`** — Manual dispatch only. Builds a single image set with user-specified repository/commit inputs. Runs never cancel each other.
144146

147+
### CI Image Pipeline
148+
149+
Each variant follows this dependency graph:
150+
151+
```
152+
resolve-commits ─┬─► velox-build ──► velox-test
153+
└─► presto-build ─► presto-test
154+
```
155+
156+
The pipeline is split into focused reusable workflows:
157+
158+
| Workflow | Purpose |
159+
|----------|---------|
160+
| `actions/resolve-commits/` | Composite action: resolves Velox/Presto commit SHAs (including `presto-pinned` logic) and sets the build date |
161+
| `velox-build.yml` | Builds velox-deps and velox images, creates multi-arch manifests |
162+
| `presto-build.yml` | Builds presto-deps, presto native worker, and coordinator images, creates multi-arch manifests |
163+
| `velox-test.yml` | Runs Velox CPU and GPU tests against built images |
164+
| `presto-test.yml` | Runs Presto smoke tests and TPC-H/TPC-DS integration tests against built images |
165+
| `actions/resolve-inputs/` | Composite action: parses image tags into SHAs + date for `workflow_dispatch` test-only runs |
166+
167+
Both `velox-test.yml` and `presto-test.yml` support `workflow_dispatch` for test-only runs against pre-built images.
168+
145169
### Image Tags
146170

147171
Images are tagged with commit SHAs, CUDA version, and build date:
@@ -152,6 +176,7 @@ Images are tagged with commit SHAs, CUDA version, and build date:
152176
- **Presto deps:** `presto-deps-${PRESTO_SHA}-velox-${VELOX_SHA}-cuda${CUDA_VERSION}-${DATE}`
153177
- **Presto build (GPU):** `presto-${PRESTO_SHA}-velox-${VELOX_SHA}-gpu-cuda${CUDA_VERSION}-${DATE}`
154178
- **Presto build (CPU):** `presto-${PRESTO_SHA}-velox-${VELOX_SHA}-cpu-${DATE}`
179+
- **Presto coordinator:** `presto-coordinator-${PRESTO_SHA}-${DATE}`
155180

156181
Images are purged after 30 days by the `ci-image-cleanup.yml` workflow.
157182

@@ -182,7 +207,7 @@ Browse available tags at [ghcr.io/rapidsai/velox-testing-images](https://github.
182207

183208
## Presto Testing Workflows
184209

185-
### `presto-test.yml`
210+
### `presto-standalone-test.yml`
186211
Main Presto testing workflow supporting three worker types:
187212
- **Java Worker:** Traditional Presto Java execution
188213
- **Native CPU Worker:** Prestissimo with Velox CPU
@@ -231,7 +256,7 @@ VELOX WORKFLOWS
231256
───────────────
232257
velox-create-staging.yml ──► create-staging-composite.yml ──► [staging branch]
233258
234-
velox-nightly-upstream.yml ─┬──► velox-test.yml
259+
velox-nightly-upstream.yml ─┬──► velox-standalone-test.yml
235260
velox-nightly-staging.yml ──┘
236261
237262
velox-benchmark-nightly-staging.yml ──► velox-benchmark-sanity-test.yml
@@ -242,14 +267,18 @@ PRESTO WORKFLOWS
242267
presto-create-staging.yml ──► create-staging-composite.yml ──► [staging branch]
243268
244269
presto-nightly-upstream.yml ─┬
245-
presto-nightly-staging.yml ──┼──► presto-test.yml ──► presto-test-composite.yml
270+
presto-nightly-staging.yml ──┼──► presto-standalone-test.yml ──► presto-test-composite.yml
246271
presto-nightly-pinned.yml ───┘
247272
248273
249274
CI IMAGES
250275
─────────
251-
ci-images-nightly.yml ─┬──► build.yml ──► [GHCR images] ──► test.yml
252-
ci-images-manual.yml ──┘
253-
test.yml (workflow_dispatch) ──► [test pre-built images]
276+
┌──► velox-build.yml ──► velox-test.yml
277+
ci-images-nightly.yml ─┬──► [resolve-commits action] ─┤
278+
ci-images-manual.yml ──┘ └──► presto-build.yml ─► presto-test.yml
279+
280+
velox-test.yml (workflow_dispatch) ──► [resolve-inputs action] ──► [test pre-built velox images]
281+
presto-test.yml (workflow_dispatch) ──► [resolve-inputs action] ──► [test pre-built presto images]
282+
254283
ci-image-cleanup.yml ──► [delete old images]
255284
```

0 commit comments

Comments
 (0)