Skip to content

pytest-rerunfailures 16.6.1 duplicates JUnit records and inflates scores #64

Description

@zwzhang0107

Problem

The evaluator installs pytest-rerunfailures without a version constraint. Version 16.6.1 changed its JUnit reporting behavior so that every rerun attempt produces a <testcase> record.

For a test that still fails after two reruns, the XML contains two empty records followed by one failure record for the same classname and name:

<testcase classname="test_rerun" name="test_always_fails" />
<testcase classname="test_rerun" name="test_always_fails" />
<testcase classname="test_rerun" name="test_always_fails">
  <failure message="assert False">AssertionError</failure>
</testcase>

parse_test_results currently processes each record independently and treats a record without a result child as passed. The single failing logical test is therefore scored as two passes and one failure, producing a score of 2/3 instead of 0/1.

Reproduction

  1. Install pytest-rerunfailures==16.6.1.
  2. Run an always-failing test with pytest --reruns=2 --junitxml=results.xml.
  3. Pass the resulting XML to parse_test_results.

With 16.6, the XML contains only the final failure record. With 16.6.1, it contains the three records shown above.

Expected behavior

Each logical test should contribute exactly once to the score, using the final attempt's status. Retry-selection metrics should also count logical tests rather than raw JUnit records. The installed rerun plugin version should be reproducible.

A fix is available in #63.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions