Integrate pytest-subtests #13738

nicoddemus · 2025-09-20T11:59:58Z

This PR copies the files from pytest-subtests and performs minimal integration.
I'm opening this to gauge whether everyone is on board with integrating this feature into the core.

Why?

Pros

subtests is a standard unittest feature, so it makes sense for pytest to support it as well.
Provides a simple alternative to parametrization.
Adds the ability to generate new test cases at runtime during the test execution, which is not possible with parametrization.
While it can exist as an external plugin, it requires many hacks, and better report integration is not easily achievable without core integration (off the top of my head: issues with terminal reporting, last failed and stepwise support, among others).

Cons

Adds another maintenance burden to the core.

TODO ✅

If everyone is on board, I will take the time this week to polish it and get it ready to merge ASAP:

✅ Cleanup the implementation: Currently it relies on monkey-patching, which should no longer be needed since we can modify the core code directly.
✅ Documentation: The feature should be documented as experimental -- meaning that while the feature itself is solid, we are still working out details such as how to best report subtest failures, integration with other plugins, etc.
✅ Fix lint and testing failures (obviously).

Continuing from the comment above, I can see two ways we can approach this:

1 - Subtests are for "runtime parametrization"

Per comment above, subtests are useful when you have some data dynamically fetched in the test and want individualized reporting for each data value.

The idea here is that parametrization should be the go-to tool, but we offer this subtest tool for this particular scenario.

2 - Subtests are for "sub-testing"

By which I mean, subtests are for when you have one conceptual test, i.e. consider it a complete whole, but just want to break down its reporting to parts.

How do you see it? The reason I'm asking is that it can affect how we document the feature, what we recommend, etc.

I think of subtests specially for the first case, but the 2nd case is also useful. Say you test 3 different objects in the test, and would like to see the test results for all the objects, even if the very first fails. Similar use case for pytest-check.

How do you suggest we approach those cases here?

pytest-check is deferred assertions with a worse api
it would be neat to have subtests.deferred_assertions() and allowing people to use the assert statement there

as for the real case of action grouping - whene doing something like a larger acceptance test(as for example done in moinmoin wiki) the goal is to report sections

a way for a section failure to directly bubble trough to fail the full test would be helpful

I actually didn't think of the "want to see all failures" use case. So it would be nice to document :)

I think my preferred approach to documenting subtests is not as alternative to parametrization, but as useful for a bunch of use cases which the reader can read and add to their "tool set". That's just my idea.

RonnyPfannschmidt · 2025-10-12T10:20:10Z

we might want to bikeshed a api around sub-section, subtests and parameter loops a little - not necessarily for doing right now - but for setting up a roadmap

nicoddemus · 2025-10-12T12:33:02Z

I was hoping to get this feature into 9.0, if possible.

RonnyPfannschmidt · 2025-10-12T13:13:51Z

we absolutely want this in 9.0

we shouldnt change the api to ensure compat with existing users
we shoul dplan for mkaing some things nicer

we should try to ensure lastfailed does rerun tests with failed subtests before releasing 9.0

nicoddemus · 2025-10-12T21:56:00Z

we should try to ensure lastfailed does rerun tests with failed subtests before releasing 9.0

I will add a test, good call.

bluetech

Unfortunately ran out of time to do a full review again, but left some comments.

I also noticed two things while I was testing it:

If some or all subtests fail, the parent test is still PASSED, is this intentional?

For some reason the errors are shown as "Captured log call", seems wrong as there are not log calls in my test.

Regarding the name SUBPASS etc., I saw that regular pass is PASSED so I wonder it shouldn't be SUBPASSED etc? (I can check the code later).

changelog/1367.feature.rst

doc/en/how-to/subtests.rst

bluetech · 2025-10-14T06:58:12Z

doc/en/how-to/subtests.rst

+---------------------------
+
+While :ref:`traditional pytest parametrization <parametrize>` and ``subtests`` are similar, they have important differences and use cases.
+


I actually didn't think of the "want to see all failures" use case. So it would be nice to document :)

I think my preferred approach to documenting subtests is not as alternative to parametrization, but as useful for a bunch of use cases which the reader can read and add to their "tool set". That's just my idea.

bluetech · 2025-10-14T07:08:46Z

src/_pytest/subtests.py

+    from typing_extensions import Self
+
+
+def pytest_addoption(parser: Parser) -> None:


Can you explain the rationale for these options -- did someone ask for them in pytest-subtests?

I admit the difference between the options is a bit obscure, I wonder if they can't be folded into a single option?

I wonder if this shouldn't be an ini option rather than a cli option? I imagine someone would want to set this if it's too noisy, in which case it more of a project setting. But maybe it's more of a user-preference thing, then it's a CLI flag...

Maybe it should be configurable at the test level, using the fixture itself? (Probably not)

Should we document the options in the tutorial?

These were external contributions: pytest-dev/pytest-subtests#81 and pytest-dev/pytest-subtests#198.

I'm OK with changing them somehow into a single option, perhaps an option to disable all subtest progress unless it is failure?

src/_pytest/subtests.py

nicoddemus · 2025-10-18T14:02:12Z

Now the top-level test will fail with a message in case it contains failed subtests but otherwise does not contain failed assertions on its own:

__________________________________ test_foo ___________________________________
Contains 2 failed subtests

In addition, enable the plugin in `pytest/__init__.py` and `config/__init__.py`.

nicoddemus changed the title ~~[DRAFT] Integrate pytest subtests~~ [DRAFT] Integrate pytest-subtests Sep 20, 2025

nicoddemus commented Sep 20, 2025

View reviewed changes

src/_pytest/subtests.py Outdated Show resolved Hide resolved

nicoddemus force-pushed the integrate-pytest-subtests branch from b43ab38 to 97ee032 Compare September 22, 2025 23:07

nicoddemus mentioned this pull request Sep 23, 2025

minor: do not import pytest in unittest.py #13748

Merged

nicoddemus force-pushed the integrate-pytest-subtests branch from 97ee032 to 6b5831f Compare September 26, 2025 13:34

psf-chronographer bot added the bot:chronographer:provided (automation) changelog entry is part of PR label Sep 26, 2025

nicoddemus force-pushed the integrate-pytest-subtests branch 2 times, most recently from 5f56d81 to c93c0e0 Compare September 26, 2025 14:00

nicoddemus marked this pull request as ready for review September 26, 2025 14:00

nicoddemus changed the title ~~[DRAFT] Integrate pytest-subtests~~ Integrate pytest-subtests Sep 26, 2025

nicoddemus force-pushed the integrate-pytest-subtests branch from c93c0e0 to b569c93 Compare September 29, 2025 22:17

bluetech reviewed Sep 30, 2025

View reviewed changes

nicoddemus force-pushed the integrate-pytest-subtests branch from b569c93 to 506c75b Compare October 11, 2025 21:23

nicoddemus requested a review from bluetech October 11, 2025 21:23

nicoddemus force-pushed the integrate-pytest-subtests branch 2 times, most recently from 42f910d to ec3144f Compare October 11, 2025 21:32

RonnyPfannschmidt added this to the 9.0 milestone Oct 12, 2025

bluetech mentioned this pull request Oct 14, 2025

Set __module__ = "pytest" for all public class types? #13812

Open

bluetech reviewed Oct 14, 2025

View reviewed changes

Add pytest-subtests files changes

2172c09

In addition, enable the plugin in `pytest/__init__.py` and `config/__init__.py`.

nicoddemus added 8 commits October 18, 2025 11:02

subtests: remove direct pytest import

f2f066d

Force using xdist plugin and fix linting

2f5cf34

Replace attr by dataclass

159217d

Add docs

016c7c3

Cleanup internal hacks

7b09c6c

Code review

cf908c1

Code review

66313f2

Docs

a63b021

nicoddemus force-pushed the integrate-pytest-subtests branch from d8c7655 to 1c31ff1 Compare October 18, 2025 14:02

Make top-level tests fail when there are failing subtests

ee3d5db

nicoddemus force-pushed the integrate-pytest-subtests branch from 1c31ff1 to ee3d5db Compare October 18, 2025 14:13

nicoddemus requested review from RonnyPfannschmidt and bluetech October 18, 2025 14:22

		---------------------------

		While :ref:`traditional pytest parametrization <parametrize>` and ``subtests`` are similar, they have important differences and use cases.

		from typing_extensions import Self


		def pytest_addoption(parser: Parser) -> None:

Uh oh!

Integrate pytest-subtests #13738

Are you sure you want to change the base?

Integrate pytest-subtests #13738

Conversation

nicoddemus commented Sep 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why?

Pros

Cons

TODO ✅

Related

Uh oh!

Uh oh!

Pierre-Sassoulas commented Sep 20, 2025

Uh oh!

RonnyPfannschmidt commented Sep 20, 2025

Uh oh!

RonnyPfannschmidt commented Sep 20, 2025

Uh oh!

webknjaz commented Sep 20, 2025

Uh oh!

nicoddemus commented Sep 26, 2025

Uh oh!

bluetech left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bluetech Sep 30, 2025

Choose a reason for hiding this comment

1 - Subtests are for "runtime parametrization"

2 - Subtests are for "sub-testing"

Uh oh!

nicoddemus Oct 11, 2025

Choose a reason for hiding this comment

Uh oh!

RonnyPfannschmidt Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

bluetech Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

RonnyPfannschmidt commented Oct 12, 2025

Uh oh!

nicoddemus commented Oct 12, 2025

Uh oh!

RonnyPfannschmidt commented Oct 12, 2025

Uh oh!

nicoddemus commented Oct 12, 2025

Uh oh!

bluetech left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bluetech Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

bluetech Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

nicoddemus Oct 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nicoddemus commented Oct 18, 2025

Uh oh!

nicoddemus commented Sep 20, 2025 •

edited

Loading