Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
b20af67
feat(ci): Add Go build cache warming for integration tests
the-mann Feb 13, 2026
1c1d8cf
feat(ci): Add Go build cache warming to PR-test workflow
the-mann Feb 16, 2026
37d97a2
fix(ci): Only pass cache_key to terraform when non-empty
the-mann Feb 16, 2026
2cde20a
fix(ci): Check terraform config declares cache_key before passing it
the-mann Feb 16, 2026
6333a71
feat(ci): Add pre-compiled test binary build and distribution (Phase 2)
the-mann Feb 17, 2026
c4baf26
fix(ci): Build test binaries for both amd64 and arm64
the-mann Feb 17, 2026
9b7b015
Merge branch 'main' into feat/go-build-cache-s3
the-mann Feb 18, 2026
49a8e7d
refactor(ci): Remove Go cache warming, keep only pre-compiled binaries
the-mann Feb 19, 2026
aeda280
feat: Upload test binaries to ITAR and CN S3 buckets
the-mann Feb 20, 2026
0947704
fix: Use separate jobs with OIDC auth for ITAR/CN uploads
the-mann Feb 20, 2026
1cc298c
feat: Pass test_func to terraform for per-function test execution
the-mann Feb 20, 2026
307372c
feat: Parallelize test binary and Mac package builds
the-mann Feb 20, 2026
1cbe231
fix: Skip matrix jobs when array is empty
the-mann Feb 20, 2026
0a43483
fix: Use /tmp/ directly for MakeMacPkg since each arch runs on separa…
the-mann Feb 20, 2026
9c15a56
fix: Use separate go caches for mac amd64/arm64
the-mann Feb 20, 2026
556c92f
Merge branch 'main' into feat/go-build-cache-s3
the-mann Feb 26, 2026
d956795
fix: avoid 1MB job output limit by writing ec2_linux_matrix to file
the-mann Feb 26, 2026
88210e6
fix: strip empty fields from test matrix to avoid 1MB output limit
the-mann Feb 26, 2026
76312d4
Revert "fix: strip empty fields from test matrix to avoid 1MB output …
the-mann Feb 26, 2026
bc99e3f
fix: strip empty fields from matrix JSON to stay under 1MB limit
the-mann Feb 26, 2026
df53dc8
fix: use artifacts for ec2_linux matrices to avoid 1MB output limit
the-mann Feb 27, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 27 additions & 7 deletions .github/workflows/PR-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -187,10 +187,24 @@ jobs:
echo "ec2_linux_matrix pages: ${{ steps.paginate-matrix.outputs.ec2_linux_matrix_page_count }}"
echo "ec2_selinux_matrix: ${{ steps.set-matrix.outputs.ec2_selinux_matrix }}"

BuildTestBinaries:
name: 'BuildTestBinaries'
needs: [CheckLabel, OutputEnvVariables]
if: needs.CheckLabel.outputs.should_run == 'true'
uses: ./.github/workflows/build-test-binaries.yml
secrets: inherit
permissions:
id-token: write
contents: read
with:
test_repo_name: ${{ needs.OutputEnvVariables.outputs.CWA_GITHUB_TEST_REPO_NAME }}
test_repo_branch: ${{ needs.OutputEnvVariables.outputs.CWA_GITHUB_TEST_REPO_BRANCH }}
s3_integration_bucket: ${{ vars.S3_INTEGRATION_BUCKET }}
terraform_assume_role: ${{ vars.TERRAFORM_AWS_ASSUME_ROLE }}

EC2LinuxIntegrationTest-0:
name: 'EC2LinuxTests-0'
needs: [ CheckLabel, GenerateTestMatrix, OutputEnvVariables, StartLocalStack, BuildAndUpload ]
needs: [ CheckLabel, GenerateTestMatrix, OutputEnvVariables, StartLocalStack, BuildAndUpload, BuildTestBinaries ]
if: needs.CheckLabel.outputs.should_run == 'true'
uses: ./.github/workflows/ec2-integration-test.yml
permissions:
Expand All @@ -208,12 +222,13 @@ jobs:
region: us-west-2
terraform_assume_role: ${{ vars.TERRAFORM_AWS_ASSUME_ROLE }}
s3_integration_bucket: ${{ vars.S3_INTEGRATION_BUCKET }}
test_binaries_prefix: ${{ needs.BuildTestBinaries.outputs.test_binaries_prefix }}
is_selinux_test: false
secrets: inherit

EC2LinuxIntegrationTest-1:
name: 'EC2LinuxTests-1'
needs: [ CheckLabel, GenerateTestMatrix, OutputEnvVariables, StartLocalStack, BuildAndUpload ]
needs: [ CheckLabel, GenerateTestMatrix, OutputEnvVariables, StartLocalStack, BuildAndUpload, BuildTestBinaries ]
if: ${{ needs.CheckLabel.outputs.should_run == 'true' && needs.GenerateTestMatrix.outputs.ec2_linux_matrix_page_count > 1 }}
uses: ./.github/workflows/ec2-integration-test.yml
permissions:
Expand All @@ -231,12 +246,13 @@ jobs:
region: us-west-2
terraform_assume_role: ${{ vars.TERRAFORM_AWS_ASSUME_ROLE }}
s3_integration_bucket: ${{ vars.S3_INTEGRATION_BUCKET }}
test_binaries_prefix: ${{ needs.BuildTestBinaries.outputs.test_binaries_prefix }}
is_selinux_test: false
secrets: inherit

EC2LinuxIntegrationTest-2:
name: 'EC2LinuxTests-2'
needs: [ CheckLabel, GenerateTestMatrix, OutputEnvVariables, StartLocalStack, BuildAndUpload ]
needs: [ CheckLabel, GenerateTestMatrix, OutputEnvVariables, StartLocalStack, BuildAndUpload, BuildTestBinaries ]
if: ${{ needs.CheckLabel.outputs.should_run == 'true' && needs.GenerateTestMatrix.outputs.ec2_linux_matrix_page_count > 2 }}
uses: ./.github/workflows/ec2-integration-test.yml
permissions:
Expand All @@ -254,12 +270,13 @@ jobs:
region: us-west-2
terraform_assume_role: ${{ vars.TERRAFORM_AWS_ASSUME_ROLE }}
s3_integration_bucket: ${{ vars.S3_INTEGRATION_BUCKET }}
test_binaries_prefix: ${{ needs.BuildTestBinaries.outputs.test_binaries_prefix }}
is_selinux_test: false
secrets: inherit

EC2LinuxIntegrationTest-3:
name: 'EC2LinuxTests-3'
needs: [ CheckLabel, GenerateTestMatrix, OutputEnvVariables, StartLocalStack, BuildAndUpload ]
needs: [ CheckLabel, GenerateTestMatrix, OutputEnvVariables, StartLocalStack, BuildAndUpload, BuildTestBinaries ]
if: ${{ needs.CheckLabel.outputs.should_run == 'true' && needs.GenerateTestMatrix.outputs.ec2_linux_matrix_page_count > 3 }}
uses: ./.github/workflows/ec2-integration-test.yml
permissions:
Expand All @@ -277,12 +294,13 @@ jobs:
region: us-west-2
terraform_assume_role: ${{ vars.TERRAFORM_AWS_ASSUME_ROLE }}
s3_integration_bucket: ${{ vars.S3_INTEGRATION_BUCKET }}
test_binaries_prefix: ${{ needs.BuildTestBinaries.outputs.test_binaries_prefix }}
is_selinux_test: false
secrets: inherit

EC2LinuxIntegrationTest-4:
name: 'EC2LinuxTests-4'
needs: [ CheckLabel, GenerateTestMatrix, OutputEnvVariables, StartLocalStack, BuildAndUpload ]
needs: [ CheckLabel, GenerateTestMatrix, OutputEnvVariables, StartLocalStack, BuildAndUpload, BuildTestBinaries ]
if: ${{ needs.CheckLabel.outputs.should_run == 'true' && needs.GenerateTestMatrix.outputs.ec2_linux_matrix_page_count > 4 }}
uses: ./.github/workflows/ec2-integration-test.yml
permissions:
Expand All @@ -300,12 +318,13 @@ jobs:
region: us-west-2
terraform_assume_role: ${{ vars.TERRAFORM_AWS_ASSUME_ROLE }}
s3_integration_bucket: ${{ vars.S3_INTEGRATION_BUCKET }}
test_binaries_prefix: ${{ needs.BuildTestBinaries.outputs.test_binaries_prefix }}
is_selinux_test: false
secrets: inherit

EC2SELinuxIntegrationTest:
name: 'EC2SELinuxTests'
needs: [ CheckLabel, GenerateTestMatrix, OutputEnvVariables, StartLocalStack, BuildAndUpload ]
needs: [ CheckLabel, GenerateTestMatrix, OutputEnvVariables, StartLocalStack, BuildAndUpload, BuildTestBinaries ]
if: needs.CheckLabel.outputs.should_run == 'true'
uses: ./.github/workflows/ec2-integration-test.yml
permissions:
Expand All @@ -323,6 +342,7 @@ jobs:
region: us-west-2
terraform_assume_role: ${{ vars.TERRAFORM_AWS_ASSUME_ROLE }}
s3_integration_bucket: ${{ vars.S3_INTEGRATION_BUCKET }}
test_binaries_prefix: ${{ needs.BuildTestBinaries.outputs.test_binaries_prefix }}
is_selinux_test: true
secrets: inherit

Expand Down Expand Up @@ -364,7 +384,7 @@ jobs:
verify-all:
name: Verify All PR Test Jobs
needs: [IntegrationTestGate, CheckLabel, BuildAndUpload, OutputEnvVariables, StartLocalStack,
GenerateTestMatrix, EC2LinuxIntegrationTest-0, EC2LinuxIntegrationTest-1,
GenerateTestMatrix, BuildTestBinaries, EC2LinuxIntegrationTest-0, EC2LinuxIntegrationTest-1,
EC2LinuxIntegrationTest-2, EC2LinuxIntegrationTest-3, EC2LinuxIntegrationTest-4,
EC2SELinuxIntegrationTest, StopLocalStack]
runs-on: ubuntu-latest
Expand Down
197 changes: 197 additions & 0 deletions .github/workflows/build-test-binaries.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,197 @@
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: MIT

name: Build Test Binaries

on:
workflow_call:
inputs:
test_repo_name:
required: true
type: string
test_repo_branch:
required: true
type: string
s3_integration_bucket:
required: true
type: string
terraform_assume_role:
required: true
type: string
s3_integration_bucket_itar:
required: false
type: string
terraform_assume_role_itar:
required: false
type: string
s3_integration_bucket_cn:
required: false
type: string
terraform_assume_role_cn:
required: false
type: string
outputs:
test_binaries_prefix:
description: "S3 prefix for pre-compiled test binaries"
value: ${{ jobs.Complete.outputs.test_binaries_prefix }}

jobs:
BuildTestBinaries:
name: 'BuildTestBinaries-${{ matrix.arch }}'
runs-on: ubuntu-latest
strategy:
matrix:
arch: [amd64, arm64]
outputs:
test_binaries_prefix: ${{ steps.set-prefix.outputs.test_binaries_prefix }}
permissions:
id-token: write
contents: read
steps:
- uses: actions/checkout@v3
with:
repository: ${{ inputs.test_repo_name }}
ref: ${{ inputs.test_repo_branch }}

- name: Set prefix
id: set-prefix
run: |
COMMIT_SHA=$(git rev-parse --short HEAD)
echo "test_binaries_prefix=integration-test/test-binaries/${COMMIT_SHA}" >> "$GITHUB_OUTPUT"

- name: Set up Go 1.x
uses: actions/setup-go@v4
with:
go-version: ~1.25
cache: false

- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ inputs.terraform_assume_role }}
aws-region: us-west-2

- name: Build and upload test binaries
env:
S3_BUCKET: ${{ inputs.s3_integration_bucket }}
GOARCH: ${{ matrix.arch }}
run: |
COMMIT_SHA=$(git rev-parse --short HEAD)
PREFIX="integration-test/test-binaries/${COMMIT_SHA}/linux/${GOARCH}"
mkdir -p "build/bin/${GOARCH}"

go mod download

echo "=== Building for linux/${GOARCH} ==="
for pkg in $(go list ./test/...); do
name=$(basename "$pkg")
if CGO_ENABLED=0 GOOS=linux GOARCH=$GOARCH go test -c -o "build/bin/${GOARCH}/${name}.test" "$pkg" 2>&1; then
echo " ✓ ${name}.test"
else
echo " ✗ ${name}.test (skipped)"
fi
done

BUILT=$(ls "build/bin/${GOARCH}/"*.test 2>/dev/null | wc -l)
echo "Built ${BUILT} binaries for linux/${GOARCH}"

aws s3 cp "build/bin/${GOARCH}/" "s3://${S3_BUCKET}/${PREFIX}/" --recursive --quiet
echo "Upload complete to commercial bucket"

- name: Upload binaries artifact
uses: actions/upload-artifact@v4
with:
name: test-binaries-${{ matrix.arch }}
path: build/bin/${{ matrix.arch }}/
retention-days: 1

UploadTestBinariesITAR:
name: 'UploadTestBinariesITAR'
needs: [BuildTestBinaries]
if: ${{ inputs.s3_integration_bucket_itar != '' && inputs.terraform_assume_role_itar != '' }}
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read
steps:
- name: Download amd64 binaries
uses: actions/download-artifact@v4
with:
name: test-binaries-amd64
path: build/bin/amd64/

- name: Download arm64 binaries
uses: actions/download-artifact@v4
with:
name: test-binaries-arm64
path: build/bin/arm64/

- name: Configure AWS Credentials for ITAR
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ inputs.terraform_assume_role_itar }}
aws-region: us-gov-east-1

- name: Upload to ITAR bucket
env:
S3_BUCKET: ${{ inputs.s3_integration_bucket_itar }}
PREFIX: ${{ needs.BuildTestBinaries.outputs.test_binaries_prefix }}
run: |
for GOARCH in amd64 arm64; do
aws s3 cp "build/bin/${GOARCH}/" "s3://${S3_BUCKET}/${PREFIX}/linux/${GOARCH}/" --recursive --quiet
done
echo "Upload complete to ITAR bucket"

UploadTestBinariesCN:
name: 'UploadTestBinariesCN'
needs: [BuildTestBinaries]
if: ${{ inputs.s3_integration_bucket_cn != '' && inputs.terraform_assume_role_cn != '' }}
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read
steps:
- name: Download amd64 binaries
uses: actions/download-artifact@v4
with:
name: test-binaries-amd64
path: build/bin/amd64/

- name: Download arm64 binaries
uses: actions/download-artifact@v4
with:
name: test-binaries-arm64
path: build/bin/arm64/

- name: Configure AWS Credentials for CN
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ inputs.terraform_assume_role_cn }}
aws-region: cn-north-1

- name: Upload to CN bucket
env:
S3_BUCKET: ${{ inputs.s3_integration_bucket_cn }}
PREFIX: ${{ needs.BuildTestBinaries.outputs.test_binaries_prefix }}
run: |
for GOARCH in amd64 arm64; do
aws s3 cp "build/bin/${GOARCH}/" "s3://${S3_BUCKET}/${PREFIX}/linux/${GOARCH}/" --recursive --quiet
done
echo "Upload complete to CN bucket"

Complete:
name: 'Complete'
needs: [BuildTestBinaries, UploadTestBinariesITAR, UploadTestBinariesCN]
if: ${{ always() }}
runs-on: ubuntu-latest
outputs:
test_binaries_prefix: ${{ needs.BuildTestBinaries.outputs.test_binaries_prefix }}
steps:
- name: Check results
run: |
echo "BuildTestBinaries: ${{ needs.BuildTestBinaries.result }}"
echo "UploadTestBinariesITAR: ${{ needs.UploadTestBinariesITAR.result }}"
echo "UploadTestBinariesCN: ${{ needs.UploadTestBinariesCN.result }}"
if [[ "${{ needs.BuildTestBinaries.result }}" != "success" ]]; then
exit 1
fi
Comment on lines +183 to +197

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {}

Copilot Autofix

AI about 1 month ago

To fix the problem, we should explicitly define a permissions block for the Complete job so it no longer relies on repository/organization defaults. Since the Complete job only echoes results and evaluates conditions without interacting with GitHub APIs or repository contents, it does not need any token permissions at all, so the least‑privilege configuration is to disable the GITHUB_TOKEN for this job using permissions: {}.

Concretely, in .github/workflows/build-test-binaries.yml, in the Complete job definition starting at line 182, add a permissions: {} entry alongside runs-on and outputs. For example, immediately after runs-on: ubuntu-latest insert a line permissions: {} with the same indentation as runs-on. No imports or additional methods are needed, as this is purely a YAML configuration change.

Suggested changeset 1
.github/workflows/build-test-binaries.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/build-test-binaries.yml b/.github/workflows/build-test-binaries.yml
--- a/.github/workflows/build-test-binaries.yml
+++ b/.github/workflows/build-test-binaries.yml
@@ -184,6 +184,7 @@
     needs: [BuildTestBinaries, UploadTestBinariesITAR, UploadTestBinariesCN]
     if: ${{ always() }}
     runs-on: ubuntu-latest
+    permissions: {}
     outputs:
       test_binaries_prefix: ${{ needs.BuildTestBinaries.outputs.test_binaries_prefix }}
     steps:
@@ -194,4 +195,3 @@
           echo "UploadTestBinariesCN: ${{ needs.UploadTestBinariesCN.result }}"
           if [[ "${{ needs.BuildTestBinaries.result }}" != "success" ]]; then
             exit 1
-          fi
EOF
@@ -184,6 +184,7 @@
needs: [BuildTestBinaries, UploadTestBinariesITAR, UploadTestBinariesCN]
if: ${{ always() }}
runs-on: ubuntu-latest
permissions: {}
outputs:
test_binaries_prefix: ${{ needs.BuildTestBinaries.outputs.test_binaries_prefix }}
steps:
@@ -194,4 +195,3 @@
echo "UploadTestBinariesCN: ${{ needs.UploadTestBinariesCN.result }}"
if [[ "${{ needs.BuildTestBinaries.result }}" != "success" ]]; then
exit 1
fi
Copilot is powered by AI and may make mistakes. Always verify output.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this warning still valid, or did you resolve it?

11 changes: 11 additions & 0 deletions .github/workflows/ec2-integration-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,9 @@ on:
type: boolean
s3_integration_bucket:
type: string
test_binaries_prefix:
type: string
required: false
secrets:
AWS_PRIVATE_KEY:
required: false
Expand All @@ -52,6 +55,7 @@ on:

jobs:
EC2IntegrationTest:
if: ${{ inputs.test_props != '[]' }}
name: ${{matrix.arrays.testName}}
runs-on: ubuntu-latest
strategy:
Expand Down Expand Up @@ -103,6 +107,11 @@ jobs:
cd ${{inputs.test_dir}}
fi

TEST_BINARIES_VAR=""
if [ -n "${{ inputs.test_binaries_prefix }}" ] && grep -q 'variable "test_binaries_prefix"' variables.tf 2>/dev/null; then
TEST_BINARIES_VAR="-var=test_binaries_prefix=${{ inputs.test_binaries_prefix }}"
fi

terraform init
if terraform apply --auto-approve \
-var="agent_start=${{ matrix.arrays.agentStartCommand }}" \
Expand All @@ -127,7 +136,9 @@ jobs:
-var="ssh_key_value=${{env.PRIVATE_KEY}}" \
-var="test_dir=${{ matrix.arrays.test_dir }}" \
-var="test_name=${{ matrix.arrays.os }}" \
-var="test_run_filter=${{ matrix.arrays.testRunFilter }}" \
-var="is_onprem=${{ inputs.is_onprem_test }}" \
$TEST_BINARIES_VAR \
-var="user=${{ matrix.arrays.username }}"; then
terraform destroy -var="region=${{ inputs.region }}" -var="ami=${{ matrix.arrays.ami }}" -auto-approve
else
Expand Down
Loading
Loading