Add automated Kernel CI workflow #675

shreeya-patel98 · 2025-11-07T11:05:49Z

Implements a 5-stage GitHub Actions pipeline for automated kernel testing and PR creation.
Uses kernel-container-build automated-testing-v1 branch for build/test tooling.

Stage 1: Build (15-30 min)

Checkout kernel source + kernel-container-build repo (automated-testing-v1)
Build kernel in CIQ builder container with kABI checking
Convert built container to QCOW2 VM image
Upload: kernel-build.log, QCOW2 image

Stage 2: Boot Verification (2-5 min)

Download QCOW2 image
Boot kernel in QEMU (KVM or TCG) and validate login prompt appears
Upload: boot logs

Stage 3: Kernel Selftests (40-60 min)

Download QCOW2 image
Execute comprehensive kselftests in QEMU with dual serial consoles
Upload: kselftest TAP logs, dmesg output

Stage 4: Compare Results (1-2 min)
Purpose: Detect test regressions by comparing against base branch

Steps:

Checkout with full history (fetch-depth: 0)
Download current kselftest logs
Fetching base branch:
- For pushes: Fetches the base branch where the patches needs to be
  merged using the user branch name.
  Example :- If user branch pushed is {shreeya}_ciqlts9_2 then
  ciqlts9_2 is the base branch.
- For force-pushes: Doesn't do the base branch fetching again, instead uses the
  same base branch as detected during the first PR creation and also
  compares the result against that base branch.
- Outputs: base_branch (reused by PR stage)
Download baseline logs from base branch (searches last 5 successful runs)
Compare results:
- Counts passing/failing tests (before/after)
- Fails if >±3 tests changed
- Outputs: comparison_status, comparison_message

Stage 5: Create Pull Request (1-2 min)
Purpose: Auto-create/update PR after all tests pass

Prerequisites: Only runs if build + boot + kselftest passed, no regressions detected

Steps:

Check all stages passed and comparison_status != failed
Checkout (shallow: fetch-depth: 100) for commit messages
Download all artifacts (build/boot/test logs)
Extract statistics (pass/fail counts, build times)
Get commit info:
- Single commit: Use commit message
- Multiple commits: Create summary
Create/Update PR:
- Reuses base_branch from compare-results (no duplication!)
- Generate PR body with test results via create-pr-body.sh
- Creates new PR or updates existing one in case of force pushes

Note :-

To skip this github action workflow, add [ci skip] or [skip ci]
into the head commit when pushing the patches.
For the CI to generate automatic PRs against a valid branch,
one needs to have the user branch name in the format of
{any_name}_basebranch.
Braces and base branch name in the user branch is necessary or else
PR creation stage will fai}_basebranch.
Braces and base branch name in the user branch is necessary or else
PR creation stage will fail.

Signed-off-by: Shreeya Patel [email protected]

.github/workflows/kernel-build-and-test-x86_64.yml

.github/scripts/create-pr-body.sh

.github/workflows/kernel-build-and-test-x86_64.yml

.github/scripts/create-pr-body.sh

.github/workflows/kernel-build-and-test-x86_64.yml

roxanan1996

I requires a fix to trigger the pull request workflow when the pull request is created

.github/workflows/kernel-build-and-test-x86_64.yml

.github/scripts/create-pr-body.sh

.github/workflows/kernel-build-and-test-x86_64.yml

Implements a 5-stage GitHub Actions pipeline for automated kernel testing and PR creation. Uses kernel-container-build automated-testing-v1 branch for build/test tooling. Stage 1: Build (15-30 min) - Checkout kernel source + kernel-container-build repo (automated-testing-v1) - Build kernel in CIQ builder container with kABI checking - Convert built container to QCOW2 VM image - Upload: kernel-build.log, QCOW2 image Stage 2: Boot Verification (2-5 min) - Download QCOW2 image - Boot kernel in QEMU (KVM or TCG) and validate login prompt appears - Upload: boot logs Stage 3: Kernel Selftests (40-60 min) - Download QCOW2 image - Execute comprehensive kselftests in QEMU with dual serial consoles - Upload: kselftest TAP logs, dmesg output Stage 4: Compare Results (1-2 min) Purpose: Detect test regressions by comparing against base branch Steps: 1. Checkout with full history (fetch-depth: 0) 2. Download current kselftest logs 3. Fetching base branch: - For pushes: Fetches the base branch where the patches needs to be merged using the user branch name. Example :- If user branch pushed is {shreeya}_ciqlts9_2 then ciqlts9_2 is the base branch. - For force-pushes: Doesn't do the base branch fetching again, instead uses the same base branch as detected during the first PR creation and also compares the result against that base branch. - Outputs: base_branch (reused by PR stage) 4. Download baseline logs from base branch (searches last 5 successful runs) 5. Compare results: - Counts passing/failing tests (before/after) - Fails if >±3 tests changed - Outputs: comparison_status, comparison_message Stage 5: Create Pull Request (1-2 min) Purpose: Auto-create/update PR after all tests pass Prerequisites: Only runs if build + boot + kselftest passed, no regressions detected Steps: 1. Check all stages passed and comparison_status != failed 2. Checkout (shallow: fetch-depth: 100) for commit messages 3. Download all artifacts (build/boot/test logs) 4. Extract statistics (pass/fail counts, build times) 5. Get commit info: - Single commit: Use commit message - Multiple commits: Create summary 6. Create/Update PR: - Reuses base_branch from compare-results (no duplication!) - Generate PR body with test results via create-pr-body.sh - Creates new PR or updates existing one in case of force pushes Note :- 1. To skip this github action workflow, add [ci skip] or [skip ci] into the head commit when pushing the patches. 2. For the CI to generate automatic PRs against a valid branch, one needs to have the user branch name in the format of {any_name}_basebranch. 3. Braces and base branch name in the user branch is necessary or else PR creation stage will fai}_basebranch. 3. Braces and base branch name in the user branch is necessary or else PR creation stage will fail. Signed-off-by: Shreeya Patel <[email protected]>

Script to generate detailed PR descriptions with kselftest results. Signed-off-by: Shreeya Patel <[email protected]>

roxanan1996

Thanks for your work!

Copilot

Pull request overview

This PR implements a comprehensive 5-stage GitHub Actions CI pipeline for automated kernel testing with automated PR creation. The workflow triggers on pushes to branches matching *_ciqlts9_2 pattern and can be skipped with [ci skip] or [skip ci] tags.

Key changes:

Adds automated kernel build, boot verification, and kselftest execution in QEMU
Implements regression detection by comparing test results against base branch
Auto-creates/updates PRs only when all tests pass and no regressions are detected

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 8 comments.

File	Description
.github/workflows/kernel-build-and-test-x86_64.yml	Main CI workflow with 5 stages: build, boot verification, kselftest execution, result comparison, and PR creation
.github/scripts/create-pr-body.sh	Helper script to generate formatted PR body with test results and commit messages

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-24T13:10:31Z

.github/workflows/kernel-build-and-test-x86_64.yml

+    if: success() || failure()
+    outputs:
+      base_branch: ${{ steps.base_branch.outputs.base_branch }}
+      comparison_status: ${{ steps.comparison.outputs.comparison_status }}


The compare-results job outputs comparison_status but the downstream create-pr job doesn't utilize the comparison_message output that is set in the comparison step (lines 314, 355, 360, 367). This message contains useful diagnostic information. Consider adding comparison_message to the outputs so it can be displayed in PR creation logs or included in the PR body for better traceability.

Suggested change

comparison_status: ${{ steps.comparison.outputs.comparison_status }}

comparison_status: ${{ steps.comparison.outputs.comparison_status }}

comparison_message: ${{ steps.comparison.outputs.comparison_message }}

Copilot · 2025-11-24T13:10:32Z

.github/workflows/kernel-build-and-test-x86_64.yml

+        # TODO: Use a centralized place to get the base branches
+        VALID_BASES="ciqlts9_2 ciqlts9_4 ciqlts8_6"


The hardcoded whitelist of valid base branches creates a maintenance burden. Each time a new base branch needs to be supported, this workflow file must be updated. Consider moving this to a configuration file (e.g., .github/config/valid-branches.txt) or repository variable that can be updated independently.

Suggested change

# TODO: Use a centralized place to get the base branches

VALID_BASES="ciqlts9_2 ciqlts9_4 ciqlts8_6"

# Read valid base branches from config file

VALID_BASES="$(cat .github/config/valid-branches.txt | tr '\n' ' ')"

Copilot · 2025-11-24T13:10:32Z

.github/workflows/kernel-build-and-test-x86_64.yml

+        skip_unpack: false
+        if_no_artifact_found: warn
+        # Only search the last 5 successful runs for better performance
+        run_number: ${{ github.run_number }}


The run_number parameter appears to be misused. According to dawidd6/action-download-artifact documentation, this parameter specifies which run number to download artifacts from, not the current run. Passing github.run_number (the current run) doesn't make sense for downloading baseline artifacts from the base branch. This parameter should likely be removed to let the action find the appropriate successful run automatically.

Suggested change

run_number: ${{ github.run_number }}

Copilot · 2025-11-24T13:10:32Z

.github/workflows/kernel-build-and-test-x86_64.yml

+        BUILD_TIME=$(grep -oP '\[TIMER\]\{BUILD\}:\s*\K[0-9]+' artifacts/build/kernel-build.log | head -1 || echo "N/A")
+        TOTAL_TIME=$(grep -oP '\[TIMER\]\{TOTAL\}\s*\K[0-9]+' artifacts/build/kernel-build.log | head -1 || echo "N/A")
+        echo "build_time=${BUILD_TIME}s" >> $GITHUB_OUTPUT
+        echo "total_time=${TOTAL_TIME}s" >> $GITHUB_OUTPUT


The regex patterns for extracting timer values use Perl-style regex with grep -oP which may not be portable across all systems. The patterns \[TIMER\]\{BUILD\}: and \[TIMER\]\{TOTAL\} appear to have escaped braces that should match literal braces. However, the inconsistency in spacing (colon in BUILD but no colon in TOTAL) suggests potential issues. Verify these patterns match the actual log format and consider adding validation that extracted values are numeric before appending 's' suffix.

Suggested change

BUILD_TIME=$(grep -oP '\[TIMER\]\{BUILD\}:\s*\K[0-9]+' artifacts/build/kernel-build.log | head -1 || echo "N/A")

TOTAL_TIME=$(grep -oP '\[TIMER\]\{TOTAL\}\s*\K[0-9]+' artifacts/build/kernel-build.log | head -1 || echo "N/A")

echo "build_time=${BUILD_TIME}s" >> $GITHUB_OUTPUT

echo "total_time=${TOTAL_TIME}s" >> $GITHUB_OUTPUT

BUILD_TIME=$(awk -F': ' '/\[TIMER\]\{BUILD\}:/{print $2}' artifacts/build/kernel-build.log | grep -E '^[0-9]+$' | head -1)

TOTAL_TIME=$(awk -F': ' '/\[TIMER\]\{TOTAL\}:/{print $2}' artifacts/build/kernel-build.log | grep -E '^[0-9]+$' | head -1)

if [[ "$BUILD_TIME" =~ ^[0-9]+$ ]]; then

echo "build_time=${BUILD_TIME}s" >> $GITHUB_OUTPUT

else

echo "build_time=N/A" >> $GITHUB_OUTPUT

fi

if [[ "$TOTAL_TIME" =~ ^[0-9]+$ ]]; then

echo "total_time=${TOTAL_TIME}s" >> $GITHUB_OUTPUT

else

echo "total_time=N/A" >> $GITHUB_OUTPUT

fi

Copilot · 2025-11-24T13:10:33Z

.github/workflows/kernel-build-and-test-x86_64.yml

+        for commit in $(git log origin/$BASE_BRANCH..HEAD --format=%h | tac); do
+          git log -1 $commit --format=%B | awk 'BEGIN{print "```"} /^$/{empty++} empty==2{exit} {print} END{print "```"}' >> /tmp/commit_message.txt
+        done


The commit variable in line 482 should be quoted to handle commit hashes safely. Additionally, the awk script wraps each commit message in markdown code blocks (```), but with multiple commits, this creates multiple separate code blocks appended to the file. This may not render as intended in the PR body. Consider wrapping all commits in a single code block or using a different format for multiple commits.

Suggested change

for commit in $(git log origin/$BASE_BRANCH..HEAD --format=%h | tac); do

git log -1 $commit --format=%B | awk 'BEGIN{print "```"} /^$/{empty++} empty==2{exit} {print} END{print "```"}' >> /tmp/commit_message.txt

done

echo '```' > /tmp/commit_message.txt

for commit in $(git log origin/$BASE_BRANCH..HEAD --format=%h | tac); do

git log -1 "$commit" --format=%B >> /tmp/commit_message.txt

echo "" >> /tmp/commit_message.txt

done

echo '```' >> /tmp/commit_message.txt

Tja, this is exactly what we want dear Copilot.

Copilot · 2025-11-24T13:10:33Z

.github/workflows/kernel-build-and-test-x86_64.yml

+          COMPARISON_SECTION="### ✅ Test Comparison
+        - Status: Passed - Within acceptable threshold (±3 tests)
+        - Compared against: $BASE_BRANCH"
+        elif [ "$COMPARISON_STATUS" = "skipped" ]; then
+          COMPARISON_SECTION="### ⚠️ Test Comparison
+        - Status: Skipped
+        - Reason: No baseline test results available from $BASE_BRANCH
+        - **Note:** Manual review recommended to ensure no regressions"
+        else
+          COMPARISON_SECTION="### ❌ Test Comparison
+        - Status: Failed - Regression detected
+        - Compared against: $BASE_BRANCH
+        - **Action Required:** Review test differences before merging"


The multi-line string assignment using double quotes in a shell script (lines 502-504, 506-509, 511-515) can be fragile with indentation and special characters. The indentation shown here will be preserved in the output, potentially breaking markdown formatting. Consider using a here-document (<<EOF) or ensuring the continuation lines are not indented, or explicitly handling whitespace.

Suggested change

COMPARISON_SECTION="### ✅ Test Comparison

- Status: Passed - Within acceptable threshold (±3 tests)

- Compared against: $BASE_BRANCH"

elif [ "$COMPARISON_STATUS" = "skipped" ]; then

COMPARISON_SECTION="### ⚠️ Test Comparison

- Status: Skipped

- Reason: No baseline test results available from $BASE_BRANCH

- **Note:** Manual review recommended to ensure no regressions"

else

COMPARISON_SECTION="### ❌ Test Comparison

- Status: Failed - Regression detected

- Compared against: $BASE_BRANCH

- **Action Required:** Review test differences before merging"

COMPARISON_SECTION=$(cat <<EOF

### ✅ Test Comparison

- Status: Passed - Within acceptable threshold (±3 tests)

- Compared against: $BASE_BRANCH

EOF

)

elif [ "$COMPARISON_STATUS" = "skipped" ]; then

COMPARISON_SECTION=$(cat <<EOF

### ⚠️ Test Comparison

- Status: Skipped

- Reason: No baseline test results available from $BASE_BRANCH

- **Note:** Manual review recommended to ensure no regressions

EOF

)

else

COMPARISON_SECTION=$(cat <<EOF

### ❌ Test Comparison

- Status: Failed - Regression detected

- Compared against: $BASE_BRANCH

- **Action Required:** Review test differences before merging

EOF

)

Copilot · 2025-11-24T13:10:33Z

.github/scripts/create-pr-body.sh

+convert_time() {
+  local seconds="${1%s}"  # Remove 's' suffix if present
+  local minutes=$((seconds / 60))
+  local remaining_seconds=$((seconds % 60))
+  echo "${minutes}m ${remaining_seconds}s"
+}


The convert_time function assumes the input is numeric after removing the 's' suffix, but doesn't validate this. If BUILD_TIME or TOTAL_TIME are 'N/A' (as set in the workflow on line 447-448), the arithmetic operations on lines 40-41 will fail. Add validation to check if the input is 'N/A' or non-numeric before attempting conversion.

Copilot · 2025-11-24T13:10:34Z

.github/scripts/create-pr-body.sh

+if [ $# -lt 7 ] || [ $# -gt 8 ]; then
+  echo "Error: Expected 7 or 8 arguments, got $#" >&2


[nitpick] The argument validation checks for 7 or 8 arguments, but the 8th argument has a default value on line 22, making it effectively always present when accessed. This creates confusion about whether the argument is optional or required. Consider either removing the upper bound check (allow any number >= 7) or documenting more clearly that the 8th argument is optional with a default.

Suggested change

if [ $# -lt 7 ] || [ $# -gt 8 ]; then

echo "Error: Expected 7 or 8 arguments, got $#" >&2

if [ $# -lt 7 ]; then

echo "Error: Expected at least 7 arguments, got $#" >&2

PlaidCat

I didn't review the the copilot recommendations. The glance shows it could be useful to do them in a follow up change set.

bmastbergen

🥌

shreeya-patel98 requested a review from a team November 7, 2025 11:05

shreeya-patel98 self-assigned this Nov 7, 2025

shreeya-patel98 requested review from PlaidCat, bmastbergen and kerneltoast as code owners November 7, 2025 11:05

shreeya-patel98 force-pushed the shreeya_kernelci_main branch from 53a82b2 to 965ef63 Compare November 11, 2025 14:23

roxanan1996 reviewed Nov 11, 2025

View reviewed changes

.github/workflows/kernel-build-and-test-x86_64.yml Show resolved Hide resolved

roxanan1996 reviewed Nov 12, 2025

View reviewed changes

.github/scripts/create-pr-body.sh Show resolved Hide resolved

.github/workflows/kernel-build-and-test-x86_64.yml Outdated Show resolved Hide resolved

.github/workflows/kernel-build-and-test-x86_64.yml Outdated Show resolved Hide resolved

PlaidCat reviewed Nov 12, 2025

View reviewed changes

.github/scripts/create-pr-body.sh Outdated Show resolved Hide resolved

PlaidCat reviewed Nov 12, 2025

View reviewed changes

.github/scripts/create-pr-body.sh Show resolved Hide resolved

PlaidCat reviewed Nov 12, 2025

View reviewed changes

.github/workflows/kernel-build-and-test-x86_64.yml Show resolved Hide resolved

PlaidCat reviewed Nov 12, 2025

View reviewed changes

.github/workflows/kernel-build-and-test-x86_64.yml Show resolved Hide resolved

PlaidCat reviewed Nov 12, 2025

View reviewed changes

.github/workflows/kernel-build-and-test-x86_64.yml Outdated Show resolved Hide resolved

shreeya-patel98 force-pushed the shreeya_kernelci_main branch 2 times, most recently from f8d9d1c to 9982c5c Compare November 20, 2025 12:13

roxanan1996 reviewed Nov 21, 2025

View reviewed changes

.github/workflows/kernel-build-and-test-x86_64.yml Outdated Show resolved Hide resolved

roxanan1996 requested changes Nov 21, 2025

View reviewed changes

roxanan1996 reviewed Nov 21, 2025

View reviewed changes

.github/workflows/kernel-build-and-test-x86_64.yml Show resolved Hide resolved

.github/scripts/create-pr-body.sh Outdated Show resolved Hide resolved

.github/workflows/kernel-build-and-test-x86_64.yml Outdated Show resolved Hide resolved

shreeya-patel98 added 2 commits November 24, 2025 11:02

Add PR body generation script

e48c890

Script to generate detailed PR descriptions with kselftest results. Signed-off-by: Shreeya Patel <[email protected]>

Copilot AI review requested due to automatic review settings November 24, 2025 13:06

shreeya-patel98 force-pushed the shreeya_kernelci_main branch from 9982c5c to e48c890 Compare November 24, 2025 13:06

Copilot started reviewing on behalf of shreeya-patel98 November 24, 2025 13:07 View session

shreeya-patel98 requested a review from PlaidCat November 24, 2025 13:08

roxanan1996 approved these changes Nov 24, 2025

View reviewed changes

Copilot finished reviewing on behalf of shreeya-patel98 November 24, 2025 13:08

shreeya-patel98 requested a review from roxanan1996 November 24, 2025 13:08

roxanan1996 approved these changes Nov 24, 2025

View reviewed changes

Copilot AI reviewed Nov 24, 2025

View reviewed changes

PlaidCat approved these changes Nov 24, 2025

View reviewed changes

bmastbergen approved these changes Nov 24, 2025

View reviewed changes

shreeya-patel98 merged commit 8419d3d into main Nov 25, 2025
6 checks passed

	comparison_status: ${{ steps.comparison.outputs.comparison_status }}
	comparison_status: ${{ steps.comparison.outputs.comparison_status }}
	comparison_message: ${{ steps.comparison.outputs.comparison_message }}

		# TODO: Use a centralized place to get the base branches
		VALID_BASES="ciqlts9_2 ciqlts9_4 ciqlts8_6"

-        for commit in $(git log origin/$BASE_BRANCH..HEAD --format=%h | tac); do
-          git log -1 $commit --format=%B | awk 'BEGIN{print "```"} /^$/{empty++} empty==2{exit} {print} END{print "```"}' >> /tmp/commit_message.txt
-        done
+        echo '```' > /tmp/commit_message.txt
+        for commit in $(git log origin/$BASE_BRANCH..HEAD --format=%h | tac); do
+          git log -1 "$commit" --format=%B >> /tmp/commit_message.txt
+          echo "" >> /tmp/commit_message.txt
+        done
+        echo '```' >> /tmp/commit_message.txt

		if [ $# -lt 7 ] \|\| [ $# -gt 8 ]; then
		echo "Error: Expected 7 or 8 arguments, got $#" >&2

Add automated Kernel CI workflow #675

Add automated Kernel CI workflow #675

Uh oh!

Conversation

shreeya-patel98 commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

roxanan1996 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

roxanan1996 left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

roxanan1996 Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

PlaidCat left a comment

Choose a reason for hiding this comment

Uh oh!

bmastbergen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

shreeya-patel98 commented Nov 7, 2025 •

edited

Loading