Skip to content

Enhancement: present final rerun test summary info section in the same way as the test session starts section #191

@jeffwright13

Description

@jeffwright13

The test session starts section is really nice in how it presents a running test suite's test outcomes: each test is shown in succession, is color-coded according to outcome, and the whole flow is easy to follow because you can follow along as individual tests get run, and either pass or fail, thereby triggering the plugin's rerun mechanism.

For example, it's easy to see in the following run:

  • that the first test passed without a rerun
  • the second test ran twice (1 rerun) but failed both times, leading to an overall failed test result
  • the third test ran three times (2 reruns) and eventually passed
  • the fourth test ran four times (3 reruns) and eventually failed
  • the fifth test ran three times (2 reruns) and eventually failed
  • the last test ran once and passed

Screen Shot 2022-05-17 at 11 49 05 PM

However, the rerun_test_summary_info section (which is obtained by invoking command line option -raR) leaves a bit to be desired:

  • not color-coded
  • only states which tests required a rerun (regardless of final outcome) instead of giving a blow-by-blow account of each test as reruns occur

Is it possible to make the final section more like the first?

Here is my testcase for above screenshot. I ran the test session as:
pytest -v -raR -k flaky

import pytest
import random

def flakey_pass_fail():
    random.seed(random.randint(0,100))
    return random.randint(0, 1) == 1

@pytest.mark.flaky(reruns=0)
def test_flaky_0_reruns():
    result = bool(flakey_pass_fail())
    assert random.randint(0,1)

@pytest.mark.flaky(reruns=1)
def test_flaky_1_reruns():
    result = bool(flakey_pass_fail())
    assert random.randint(0,1)

@pytest.mark.flaky(reruns=2)
def test_flaky_2_reruns():
    result = bool(flakey_pass_fail())
    assert random.randint(0,1)

@pytest.mark.flaky(reruns=3)
def test_flaky_3_reruns():
    result = bool(flakey_pass_fail())
    assert random.randint(0,1)

@pytest.mark.flaky(reruns=2)
def test_flaky_2_reruns_but_always_fail():
    assert False

@pytest.mark.flaky(reruns=2)
def test_flaky_2_reruns_and_always_pass():
    assert True

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions