-
Notifications
You must be signed in to change notification settings - Fork 91
Open
Description
The test session starts
section is really nice in how it presents a running test suite's test outcomes: each test is shown in succession, is color-coded according to outcome, and the whole flow is easy to follow because you can follow along as individual tests get run, and either pass or fail, thereby triggering the plugin's rerun mechanism.
For example, it's easy to see in the following run:
- that the first test passed without a rerun
- the second test ran twice (1 rerun) but failed both times, leading to an overall failed test result
- the third test ran three times (2 reruns) and eventually passed
- the fourth test ran four times (3 reruns) and eventually failed
- the fifth test ran three times (2 reruns) and eventually failed
- the last test ran once and passed
However, the rerun_test_summary_info
section (which is obtained by invoking command line option -raR
) leaves a bit to be desired:
- not color-coded
- only states which tests required a rerun (regardless of final outcome) instead of giving a blow-by-blow account of each test as reruns occur
Is it possible to make the final section more like the first?
Here is my testcase for above screenshot. I ran the test session as:
pytest -v -raR -k flaky
import pytest
import random
def flakey_pass_fail():
random.seed(random.randint(0,100))
return random.randint(0, 1) == 1
@pytest.mark.flaky(reruns=0)
def test_flaky_0_reruns():
result = bool(flakey_pass_fail())
assert random.randint(0,1)
@pytest.mark.flaky(reruns=1)
def test_flaky_1_reruns():
result = bool(flakey_pass_fail())
assert random.randint(0,1)
@pytest.mark.flaky(reruns=2)
def test_flaky_2_reruns():
result = bool(flakey_pass_fail())
assert random.randint(0,1)
@pytest.mark.flaky(reruns=3)
def test_flaky_3_reruns():
result = bool(flakey_pass_fail())
assert random.randint(0,1)
@pytest.mark.flaky(reruns=2)
def test_flaky_2_reruns_but_always_fail():
assert False
@pytest.mark.flaky(reruns=2)
def test_flaky_2_reruns_and_always_pass():
assert True
Metadata
Metadata
Assignees
Labels
No labels