Problem
The evaluator installs pytest-rerunfailures without a version constraint. Version 16.6.1 changed its JUnit reporting behavior so that every rerun attempt produces a <testcase> record.
For a test that still fails after two reruns, the XML contains two empty records followed by one failure record for the same classname and name:
<testcase classname="test_rerun" name="test_always_fails" />
<testcase classname="test_rerun" name="test_always_fails" />
<testcase classname="test_rerun" name="test_always_fails">
<failure message="assert False">AssertionError</failure>
</testcase>
parse_test_results currently processes each record independently and treats a record without a result child as passed. The single failing logical test is therefore scored as two passes and one failure, producing a score of 2/3 instead of 0/1.
Reproduction
- Install
pytest-rerunfailures==16.6.1.
- Run an always-failing test with
pytest --reruns=2 --junitxml=results.xml.
- Pass the resulting XML to
parse_test_results.
With 16.6, the XML contains only the final failure record. With 16.6.1, it contains the three records shown above.
Expected behavior
Each logical test should contribute exactly once to the score, using the final attempt's status. Retry-selection metrics should also count logical tests rather than raw JUnit records. The installed rerun plugin version should be reproducible.
A fix is available in #63.
Problem
The evaluator installs
pytest-rerunfailureswithout a version constraint. Version 16.6.1 changed its JUnit reporting behavior so that every rerun attempt produces a<testcase>record.For a test that still fails after two reruns, the XML contains two empty records followed by one failure record for the same
classnameandname:parse_test_resultscurrently processes each record independently and treats a record without a result child as passed. The single failing logical test is therefore scored as two passes and one failure, producing a score of 2/3 instead of 0/1.Reproduction
pytest-rerunfailures==16.6.1.pytest --reruns=2 --junitxml=results.xml.parse_test_results.With 16.6, the XML contains only the final failure record. With 16.6.1, it contains the three records shown above.
Expected behavior
Each logical test should contribute exactly once to the score, using the final attempt's status. Retry-selection metrics should also count logical tests rather than raw JUnit records. The installed rerun plugin version should be reproducible.
A fix is available in #63.