Skip to content

Add GitHub Actions workflow to verify flaky test #712

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions .github/workflows/verify-flaky-test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
name: Verify Flaky Test

on:
push:
branches: [ verify-flaky-test ]
workflow_dispatch:

jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3

- name: Set up JDK 11
uses: actions/setup-java@v3
with:
java-version: '11'
distribution: 'adopt'

- name: Build with Maven (Skip Tests)
run: mvn clean install -DskipTests

- name: Run Test Normally (Expected to Pass)
run: |
echo "Running test without NonDex - should PASS"
mvn test -pl core -Dtest=org.jsmart.zerocode.core.kafka.DeliveryDetailsTest#testSerDeser
echo "Test passed without NonDex as expected"

- name: Run Test with NonDex (Expected to Fail)
continue-on-error: true
id: nondex-run
run: |
echo "Running test with NonDex - should FAIL"
mvn edu.illinois:nondex-maven-plugin:2.1.1:nondex -pl core -Dtest=org.jsmart.zerocode.core.kafka.DeliveryDetailsTest#testSerDeser
# Store the exit code
echo "nondex_exit_code=$?" >> $GITHUB_OUTPUT

- name: Display NonDex Results
run: |
echo "Checking NonDex results"
ls -la core/.nondex/ || echo "No .nondex directory found"
for dir in core/.nondex/*/; do
if [ -d "$dir" ]; then
echo "Contents of $dir:"
ls -la "$dir"
if [ -f "$dir/failure" ]; then
echo "Found failure file in $dir:"
cat "$dir/failure"
fi
fi
done

- name: Verify Test Flakiness
run: |
if [ "${{ steps.nondex-run.outputs.nondex_exit_code }}" != "0" ]; then
echo "✅ Test is flaky: Passes without NonDex but fails with NonDex"
else
echo "❌ Test is not flaky: Passes both with and without NonDex"
exit 1
fi