feature: each test starts with a unique seed. --randomly-use-same-seed-per-test for old behavior

[brycedrennan]

Resolves #600

I'm tempted to make it the default behavior but this is safer.

[adamchainz]

Thanks for this.

I think we can make it the default behaviour, no option. Otherwise it’s just another thing other users have to learn about and consider activating.

[brycedrennan]

It's done but there is an inconsistency with how seeding works with faker. I'm not sure what would be least surprising for someone given the docs from faker.

pytest-randomly/tests/test_pytest_randomly.py

Lines 674 to 692 in 8b80740

	assert fake.name() == 'Justin Richard'
	def test_two():
	assert fake.name() == 'Tiffany Williams'
	"""
	)
	out = ourtester.runpytest("--randomly-seed=1")
	out.assert_outcomes(passed=2)
	def test_faker_fixture(ourtester):
	ourtester.makepyfile(
	test_one="""
	def test_one(faker):
	assert faker.name() == 'Ryan Gallagher'
	def test_two(faker):
	assert faker.name() == 'Ryan Gallagher'

https://faker.readthedocs.io/en/master/pytest-fixtures.html#seeding-configuration

[alexandrupatrascu]

hi folks, any chance this gets merged and released soon?
reason why I'm asking is that I'm running pytest-rerunfailures alongside pytest-randomly and the re-runs use the same seed ID, causing the tests to re-use the same test data and not generate new inputs, hence they keep on failing.

thank you for the help!

[brycedrennan]

Sounds like you should change your test to either work with the same seed of data or change the data generator to not generate invalid data? I don't believe the changes proposed here would even help with your situation either way.

[tamird]

This would be very useful indeed.

In the meantime I worked around this in my own project using

@pytest.fixture(autouse=True)
# autouse fixtures with leading underscores are ignored. See https://github.com/pytest-dev/pytest/issues/12404.
def emulate_pytest_randomly_pull_request_617(request: pytest.FixtureRequest) -> None:  # noqa: PT004
    # pytest-randomly sets `randomly_seed` to an integer at pytest_configure time. See
    # https://github.com/pytest-dev/pytest-randomly/blob/8a3a241/src/pytest_randomly/__init__.py#L138.
    #
    # We include that integer to preserve the behavior of an explicit `--randomly-seed <int>` flag
    # passed on the command line while also giving each test a unique seed.
    request.config.option.randomly_seed += int(hashlib.sha512(request.node.nodeid.encode()).hexdigest(), base=16)

[mjpieters]

nitpick: The actual switch implemented is called --randomly-dont-seed-per-test.

[brycedrennan]

nice catch. that's confusing

[adamchainz]

Thank you for this @brycedrennan . I have eventually got around to looking at it, and I decided it would be easiest to redo the PR, with you recorded as coauthor, in #687. I went for avoiding an option and updating the tests to expect certain fixed random values.

I think this is a great change, and will be combining it with #686 for a new major release! 🥳

[brycedrennan]

thanks! glad it made it in!