-
Notifications
You must be signed in to change notification settings - Fork 21
test: Add pytest-rerunfailures for integration tests #746
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
Add `pytest-rerunfailures` as a dev dependency and configure all integration tests to rerun up to 3 times on failure automatically. This helps handle occasional platform-related instabilities without manual test reruns. Uses `pytest_collection_modifyitems` hook to apply the flaky marker to all integration tests, avoiding the need to decorate each test individually. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #746 +/- ##
==========================================
- Coverage 81.98% 81.90% -0.08%
==========================================
Files 46 46
Lines 2681 2681
==========================================
- Hits 2198 2196 -2
- Misses 483 485 +2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Pijukatel
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would prefer to apply any retries only on per-test basis. Most often I feel that the flakiness is due to specific test design or sometimes it actually exposes real issue. Such blanket retry could lead to hiding issues.
Can you please link some example failures you want to deal with this?
All tests can fail if the platform is overloaded (or due to similar transient issues). For example: This run failed because the Actor just timed out (rather than because of a test-specific issue): When this happens, the only option is to restart the entire test suite, not just the single test that failed due to the timeout (which means 10+ minutes). |
Add
pytest-rerunfailuresas a dev dependency and configure all integration tests to rerun up to 3 times on failure automatically. This helps handle occasional platform-related instabilities without manual test reruns.Uses
pytest_collection_modifyitemshook to apply the flaky marker to all integration tests, avoiding the need to decorate each test individually.