Experiment: take 864 of 1145 tests in 13 files off pytester - #140
Draft
RonnyPfannschmidt wants to merge 30 commits into
Draft
Experiment: take 864 of 1145 tests in 13 files off pytester#140RonnyPfannschmidt wants to merge 30 commits into
RonnyPfannschmidt wants to merge 30 commits into
Conversation
Reviewer's GuideThis PR ports a large set of pytest internal tests from the older pytester-based runner to the new _pytest.ensemble API, updating fixtures, xunit setup/teardown expectations, marker semantics, and parametrization behavior while preserving test outcomes. It also adds a small codespell ignore entry and leaves a few tests intentionally on pytester as canaries or due to current ensemble limitations. Sequence diagram for a ported test using run_testssequenceDiagram
participant TestCode as test_function
participant build_module
participant ConfigSpec
participant run_tests
participant RunRecord
TestCode->>build_module: build_module("test_name", test_function,...)
build_module-->>TestCode: module
TestCode->>ConfigSpec: ConfigSpec(rootpath=tmp_path, extra_plugins,...)
ConfigSpec-->>TestCode: spec
TestCode->>run_tests: run_tests(module_or_fixtures..., spec=spec, capture_output=True)
run_tests-->>RunRecord: record
TestCode->>RunRecord: assert_outcomes(passed=..., errors=..., xfailed=..., xpassed=...)
TestCode->>RunRecord: access record["test_name"], stdout, warnings
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
RonnyPfannschmidt
changed the title
RonnyPfannschmidt
force-pushed
the
ensemble-migration-experiment
branch
3 times, most recently
from
August 14, 2026 06:02
18edf5e to
02bcb13
Compare
RonnyPfannschmidt
changed the title
RonnyPfannschmidt
force-pushed
the
ensemble-migration-experiment
branch
from
August 14, 2026 06:48
02bcb13 to
a39aa5b
Compare
158 of 205 pytester-based tests now build their sources as real python objects and assert on typed records. The largest single unlock was that a rootdir conftest is exactly reproducible as a plugin object in extra_plugins - fixture visibility, override chains, getfixturevalue super-lookup and autouse all behave identically - which carried ~30 tests on its own. 47 tests stay on pytester, each with an `# ensemble:` reason: conftest directory scoping (10), --fixtures/--markers rendering which is served from pytest_cmdline_main (13), assertions on a fixture's file:line, which is host-anchored (8), package layout and scope="package" (7), errors raised at module import (4), real plugin modules and example trees (2), subprocess (2). Two outcome assertions were wrong in opposite directions and are now corrected with a note at each site: a failing setup is `errors`, not `failed`, and one test asserting only `ret == 0` was really `passed=1, xfailed=2`.
62 of 67 pytester-driven tests ported; the ~54 that already drive Metafunc and IdMaker directly are untouched. Module- and class-level pytest_generate_tests work natively because the synthesized module is a real module object, and conftest-level hooks come over as plugin objects in extra_plugins. Most -v/fnmatch_lines assertions here only checked parametrization ids and pass/fail, so they became exact collect_tests() name lists and record[...] lookups - stronger than the globs they replace. Four rendering-centric tests keep their fnmatch_lines verbatim via capture_output=True. The xfail tests asserted `skipped` because HookRecorder lumps xfails in with skips; they now assert `xfailed`, with a note at each site so the change does not read as a mistake. Four stay on pytester: conftest directory scoping, a misspelled marker that raises against the host config at decoration time, a package-scope reordering test that would go green while exercising a different path, and a subprocess test. Also fixes a trap this port exposed: Pytester.genitems() does not reorder, but the full collection protocol does, so a test unpacking items positionally silently got the wrong ones. It now looks them up by name. `hellow` joins the codespell ignore list: it is a deliberate parametrize id that predates this change, and codespell only scans touched files.
59 of 70 tests ported. Setup/teardown ordering tests record into closure lists asserted directly, which removes the smuggling of state through module attributes and through monkeypatched sys attributes. Outcome categories are corrected where HookRecorder's counting differed from the terminal's, each with a note: xunit-style setup failures are `errors`, trial todo and module-level pytestmark are `xfailed`, and one test was silently ignoring a teardown error. unittest's own setUp/tearDown failures stay call-phase `failed` - the two mappings genuinely differ. Moving sources out of makepyfile strings put them under ruff and mypy for the first time, which exposed two undefined names that had been sitting in the strings. The one in test_unittest_skip_issue148 is now a closure-list assertion that actually proves setUpClass/tearDownClass did not run. 11 stay on pytester: example-script driven (5), subprocess (3), pexpect (1), --fixtures output (1), and one deliberate file-based canary so real module import and path-based nodeids remain covered here.
44 of 53 pytester-based tests ported, ~100 parametrized cases now running in memory. Marker names are the awkward part: pytest's own suite runs with strict = true, and a decorator in the enclosing test body is resolved against the *host* config long before the ensemble exists. Rather than rename the markers to ones the host happens to register - which would weaken the tests - an `ensemble_mark()` helper builds the MarkDecorator directly, keeping every original name and kwargs shape. Marks resolved *inside* an ensemble are unaffected: MARK_GEN._config is swapped to the nested config while it is configured. Two tests got stronger rather than merely equivalent: one asserting only `ret == 0` now asserts the outcome counts, and the -k selection test gains an unmarked sibling so it proves selection rather than "everything ran". 9 stay on pytester: --markers rendering served from pytest_cmdline_main (2), path arguments and arg-level collection (2), real files and conftest scoping (1), decoration-time marker resolution (1), string conditions resolved in host globals (1), absence of Directory nodes (1), and one asserting a host-anchored line number.
47 tests ported, and the file is 78 lines shorter for it. Rendering globs became exact nodeid lists in several places - test_ordered_by_definition_order and test_abstract_class_is_not_collected now assert the whole collected sequence rather than fnmatch-ing a few lines of --collect-only output. The build_module naming rule caught two sources: @pytest.fixture returns a FixtureFunctionDefinition whose __name__ comes from the wrapped function, so a decorated fixture passed positionally would have registered under the wrong name and left the test green with the fixture simply missing. Both are keyword members now, with a note saying why. 29 stay on pytester: real module import (10, the whole TestModule class is about the import chokepoint an ensemble bypasses), real files, packages and duplicate arguments (9), conftest scoping and hooks that never fire (4), host-anchored reportinfo (3), rendering anchored at conftest line numbers (2), and python_files discovery (1).
10 of 12 tests ported. These are setup/teardown ordering tests, so they
lose the module-global smuggling and the getcalls("pytest_runtest_setup")
archaeology entirely: events go into a closure list that the assertion
reads directly.
Every xunit setup failure moves from assertoutcome(failed=N) to
assert_outcomes(errors=N) - setup_module/setup_class/setup_method failures
are setup-phase - with a note at each site. One test previously asserted
no outcome at all and now asserts errors=1, so it can no longer pass
vacuously.
2 stay on pytester as canaries, each with a reason: one whose subject is a
genuine module global shared between hooks and test bodies, and one
covering path-argument collection with real file nodeids.
64 of 79 pytester-based tests ported - the highest yield in the migration so far, because this file is dominated by outcome assertions. The decisive rule for this file: short-summary lines keyed on the *nodeid* (-rx/-rX/-rE/-ra, FAILED) transfer verbatim once the synthesized module is named deliberately, while "SKIPPED [1] <path>:<line>: reason" renders item.location, which for an ensemble source is the host file. That single line is why the seven skip-reporting tests stayed. Two outcome assertions were hiding something. test_failing_teardown_issue9 asserted "*1 xfail*", which happily matched the "1 xpassed, 1 xfailed" summary and concealed the xpass; it is now assert_outcomes(xpassed=1, xfailed=1). Four tests failing in pytest_runtest_setup are errors, not failures. 15 stay on pytester: host-anchored skip locations (7), module-import-time behaviour (3), custom Items via pytest_collect_file (2), conftest directory scoping (1), --markers (1), and one whose subject is a module-global reachable from a string condition - evaluate_condition reads item.obj.__globals__, which for an ensemble source is this file.
34 of 53 pytester-based tests ported. Report-shape assertions map onto RunRecord.reports and the ItemRecord phases directly, and ret == 1 has a faithful stand-in in session.testsfailed - which is exactly what wrap_session turns into the exit code. Two tests carried a conftest defining `class Function(pytest.Function)`. Node-class customization by name has not been consulted in years - python.py instantiates Function directly - and every assertion that would have noticed was already commented out. Verified empirically with a real conftest on disk before dropping it. 19 stay on pytester: import-time behaviour of a real module (7), wrap_session/main() (4), capture (2), session bail-out mid-run (2), custom Collector via pytest_collect_file (1), module-level hooks (1), subprocess (1), real stream encoding (1). The two session bail-out tests are worth noting: they look like plain assert_outcomes ports, but run_items has no shouldstop handling, so a port would have quietly asserted a fully-run session instead.
32 of 51 pytester-based tests ported, asserting on typed WarningMessage objects instead of scraping the -rw summary block. The expected obstacle did not materialize. Config._catch_configured_warnings prepends `always` filters for (Pending)DeprecationWarning, so they beat the host suite's inherited `filterwarnings = error` - which means the whole TestDeprecationWarningsByDefault class, whose docstring explains that all its runs are in a subprocess precisely so they do not inherit filters, ports with no ini override at all. Six subprocess runs became in-process. Several assertions got stronger rather than merely equivalent: "the warnings summary is absent from stdout" became warnings=0 plus an empty record.warnings, and the collection warning test now asserts it was reported with when="collect". 19 stay on pytester: warning file/line/func or a summary keyed on real paths (8), real importable plugin modules (4), a fresh interpreter (2), Config.parse phases an ensemble skips (2), and three one-offs.
27 of 31 pytester-based tests ported. This file taught us how subtest outcomes actually read: a failed subtest is plain "failed" and the top-level report is also failed, so one test with one failing subtest is failed=2; a passed subtest is "subtests passed", a key assert_outcomes() silently ignores, and one that only exists when verbosity_subtests > 0. Ported tests assert it explicitly. The --last-failed test was a silently green no-op before it was made to load the terminal plugin: subtests marks the top-level test failed as a side effect *inside* pytest_report_teststatus, so with nothing rendering, LFPlugin saw the report still passed and popped the nodeid the failing subtest had just added. cache/lastfailed was never written and the rerun was indistinguishable from a fresh run. 4 stay on pytester: three need capture around the whole item, and one is xdist over real files.
10 of 24 pytester-based tests ported - a lower yield than the file's pytester count suggests, because much of it is bound to rootdir discovery and path-argument collection. --exitfirst and --maxfail were the interesting ones. run_items runs every item unconditionally; the early exit lives in pytest_runtestloop, which ensembles bypass, so a naive port would have run everything and asserted the opposite of the subject. They now drive the real hook and assert the exact surviving nodeid list plus the shouldfail message. The sticky-flag tests hit a related edge: RunRecord is built before pytest_sessionfinish, and the warnings those tests are about are emitted inside it, so assert_outcomes(warnings=1) would have been a silent false-green. Both use a pytest_warning_recorded impl on their plugin object instead. 14 stay on pytester: filesystem and path-argument collection (6), real module import (2), -p (2), rootdir discovery (2), pytest_collect_file (1), wrap_session cwd restore (1).
33 of 54 pytester-based tests ported; the 98 that already unit-test the assertion internals directly are untouched. A rewritten assertion renders essentially byte-identical to a real run - same >/E prefixes, same explanation, same Full diff, same short-summary echo. The traceback formatter dedents the source, so exact-indent patterns transfer unchanged however deeply the source function is nested in the host test body. The decisive finding is that an ensemble needs the `assertion` plugin loaded, and it is not in DEFAULT_PLUGINS. Without it util._reprcompare stays bound to whatever the *host* run installed: an explanation is still produced, so tests go green, but the ensemble's own -vv, ini values and pytest_assertrepr_compare implementations are silently ignored. Swapping "assertion" for "terminal" in the local spec helper makes 16 of the 33 ports fail, so they are load-bearing rather than vacuous. 21 stay on pytester: rewriting of files on import (7), assertions on the failing source's file:line (3), rewrite-time warnings (2), module-file semantics of the rewriter (2), and seven one-offs.
39 of 146 pytester-based tests ported - deliberately conservative. This
file's subject is rendering, so a port buys speed and hermeticity but no
assertion strength, and relaxing a pattern to make one pass would destroy
what the test is for. Everything left carries an `# ensemble:` reason.
What blocks the rest, measured rather than assumed:
- the progress column, ~40 tests, and it is not the item-location problem.
TerminalReporter._determine_show_progress_info() returns False whenever
getoption("capture", "no") == "no", and an ensemble does not load the
capture plugin, so the column is off at every verbosity whatever
console_output_style says. The final [100%] additionally comes from a
pytest_runtestloop hookwrapper that run_items never reaches.
- --collect-only rendering, ~16, a pytest_cmdline_main concern.
- captured-output sections, 10.
- host-anchored file:line, 9, plus 4 asserting the `<- file>` suffix.
Note _locationline only appends `<- ...` at verbosity >= 2, so plain -v
progress lines already match upstream verbatim - which is what carried
test_verbose_reporting and TestClassicOutputStyle.test_verbose.
One hazard recorded for future ports: assertion-explanation verbosity is
host-global, because _pytest.assertion.util._reprcompare is bound at host
configure time. -vv inside an ensemble does not un-truncate an assertion.
Follow-on to the base branch containing configure-time warnings. test_pytest_configure_warning had been catching the warning through the host's recwarn fixture - it was depending on the leak. It now asserts on record.warnings, and declares the ini filter that makes the warning a warning rather than an error: absent one, an ensemble inherits this suite's filterwarnings = error.
These five patterns had to drop their trailing "[100%]" when the file was ported, because the terminal reporter defers that final fill to pytest_runtestloop and an ensemble drove the items directly. The ensemble now goes through that hook, so the fill is rendered and the patterns match upstream again - one less place where a port asserted less than the original.
The example-driven tests were either left on pytester or ported by
inlining their source - and inlining orphaned the example files while
moving every reported location into the host test file.
module_from_path() plus EnsembleModule keeping a real __file__ makes the
third option work: run the example from where it lives. Eight tests in
fixtures.py move off inlined copies and back onto the real scripts, and
four more in test_unittest.py come off pytester entirely.
The locations are the point. test_funcarg_lookupfails can now assert
"file *test_funcarg_lookupfails.py, line 12"
which named this file when the source was inlined, so the pattern could
not be carried over at all. Same for test_fixture_named_request's
"*test_fixture_named_request.py:8" and the nodeid assertions.
test_extend_fixture_conftest_module is the shape worth noting: the
example's conftest.py sits at the example's own root, which is exactly
what an ensemble plugin object stands for, so both files now run as
themselves.
Every example script under fixtures/ and unittest/ is referenced again -
the eight that inlining had orphaned are back in use.
17 more tests move to _pytest.ensemble now that a private output stream counts as captured and the run goes through pytest_runtestloop: TestProgressOutputStyle (6), TestProgressWithTeardown (4) and the non-collect-only half of TestFineGrainedTestCase (7). Every pattern is kept verbatim - the column renders at the same character positions, needs no console_output_style ini, and ends on the same [100%]/[20/20] fill - and each port gains an assert_outcomes() the pytester original never had. The ported fine-grained tests pin COLUMNS=80, which pytester used to supply implicitly. Two tests are now unportable *because* of the fix and say so: test_capture_no and test_capture_no_progress_enabled assert that --capture=no suppresses the column, and an ensemble's private stream always counts as captured, so porting them would assert nothing. Running total: 70 of 159 tests in this file are off pytester.
The existing benchmark contrasts pytester and _pytest.ensemble on generated sources. That measures the harness but not the usual shape of a pytester test, which is: take a script that already exists under testing/example_scripts, put it somewhere, and run it. This one runs the *same real files* through both routes - copy into the pytester tmpdir and run it there, versus module_from_path and run_tests where it lives - so the only difference is the harness. It also isolates the copy and the import, which turn out to be noise (0.1-0.2ms) next to the session bootstrap either side of them. The import-failure example is deliberately not a subject: the two harnesses would not be doing the same work.
27 of 86 pytester tests move to _pytest.ensemble. The yield is deliberately low: this file is the one place where the *program* is the system under test, and roughly 40 of the 59 that stay assert something with no meaning inside a nested config - an exit code, a stderr line, what -p/--pyargs do to argv, which conftest is visible from which directory, what happens when a module fails to import. TestInvocationVariants (24 tests) should never move; its entire content is "invoked this way, pytest still works". What did move was mostly misplaced rather than slow: parametrize ids, fixture scope ordering, async rejection, StopIteration handling - unit-ish behaviour that had accreted into an acceptance file because pytester was the only tool available. Several gained real assertions on the way, with `assert result.ret == 0` becoming exact outcome counts, and test_with_deselected/test_with_not now asserting the deselection they were named for. TestDurations turns out not to be about duration measurement at all - with mock_timing the clock is fake - so it is a terminal rendering test, and ports onto capture_output=True keeping its lines verbatim. test_fixture_mock_integration runs the real example script via module_from_path rather than inlining it, so the example stays in use. Wall clock 7.82s -> 7.32s: 30% of the tests bought 6% of the time. The value here is placement, not speed.
44 of 50 pytester tests. cacheprovider works in an ensemble with nothing
special: with_plugins("cacheprovider") against a real tmp_path rootdir
gives a real config.cache, and two ensembles sharing a rootpath share the
cache - which is exactly the shape --lf/--ff/--nf tests need.
The one real design decision is which sources exist on disk. --lf's
file-level skipping and --nf's mtime ordering both need path.exists(), so
those use a written file plus module_from_path; everything else uses a
synthesized module, where the last-failed wrapper ignores non-existent
paths and so behaves like a file named on the command line - which is what
preserves the original "N deselected" assertions. Both helpers are
documented at module level.
Assertions that had to change are commented at each site: --co trees lose
their <Dir> lines, one summary reads "- AssertionError" rather than
"- assert 0" because ensemble sources are not rewritten, and
NO_TESTS_COLLECTED becomes an empty outcomes() dict. A print from
pytest_sessionfinish became a direct assertion on the plugin object, which
is stronger. Seven of the riskiest new assertions were mutation-tested.
6 stay on pytester: real packages and nested trees (3), modules that raise
at import (2), and --help (1).
Wall clock 4.51s -> 3.84s. Modest, because every test here was already
in-process - see the commit adding the autoload benchmark for why that is
not the interesting number.
47 of 67 pytester functions (75 of 140 instances). junitxml works in an ensemble unchanged: the plugin loads, pytest_configure builds the LogXML, and the file is written from pytest_sessionfinish as the context exits, so it exists by the time run_tests returns. The classname hazard I warned about turned out not to exist: junitxml derives classname from report.nodeid via mangle_test_address, not from the code object, and the synthesized module name *is* the nodeid's first segment. Only the xunit1 file/line attributes follow the code object, and nothing here asserts on them. The two tests that do need real paths use module_from_path. run_and_parse kept its shape and became ensemble-backed; the original is preserved as run_and_parse_pytester for the tests that stayed. Every port asserts more than before, `assert result.ret` becoming assert_outcomes. test_record_fixtures_xunit2 was rebinding expected_lines and silently dropping its first expected line - both warnings are asserted now. 20 stay on pytester, 13 of them for item-level capture, which is the single largest remaining blocker in this file. Wall clock 6.47s -> 6.23s: this file's pytester runs were already inline, so the saving is only pytester's fixture setup, and the residual cost is in the 56 instances that could not move.
RonnyPfannschmidt
force-pushed
the
ensemble-migration-experiment
branch
from
August 14, 2026 08:46
a39aa5b to
e65f3e9
Compare
The rewriter's output is a pure function of source and config: the message an ``assert`` produces is fully determined by ``rewrite_asserts`` before any reporter sees it -- ``_format_explanation`` runs inside the rewritten ``raise``, so ``str(exc)`` is already the rendered message, escaping and all. Tests that fnmatched ``*AssertionError: ...*`` on terminal output were paying for a session to read a string the rewriter had already decided. Convert 34 such tests to the file's existing direct path, and add ``getmsg_src`` for sources that need module-level statements -- imports, class definitions, a ``PYTEST_DONT_REWRITE`` docstring, a module-level ``assert``. It execs the module and then calls every ``test_*`` it defines, in definition order, in the module's own namespace, which is what the walrus-leak regressions need: they are about state leaking through module-level ``@py_assert`` temporaries from one test function to the next. The 45 remaining pytester tests are tagged with why they stay. The line falls in three places, and none of them are the rewriter's output: * who decides to rewrite -- the import hook, ``register_assert_rewrite``, ``python_files``, ``--assert=plain``. Note ``PYTEST_DONT_REWRITE`` is checked in both ``AssertionRewriter.run`` and the hook's ``_should_rewrite``; the direct path covers only the former. * where the result is stored -- pyc headers, cache dirs, marshalling. * what config the rewriter reads -- ``enable_assertion_pass_hook`` at rewrite time, ``util._config`` at run time, both session-installed. Verbosity-threshold tests are left alone for that last reason: the message is rewriter output, but the threshold arrives via ``util._config`` and stubbing it would leave the wiring untested. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An ensemble item's code object belongs to the *host* test file, so a failing item's `repr_failure` runs `getstatementrange_ast` over all 4000-odd lines of `test_terminal.py` -- `inspect.findsource` hands back whole-file lines, and the AST cache lives on `FormattedExcinfo`, i.e. per failure, so it never hits. That is ~40ms per failing item. A pytester temp file is ten lines, so the same repr costs ~0.2ms there. This is why the ported tests came out *slower* than the pytester originals they replaced: the port trades ~10ms of config build for ~37ms of host-file reparse on every failure. `Function.repr_failure` only reaches `_getentrysource` for tbstyle `short`/`long`, and `--tb=no` still produces `longrepr.reprcrash`, so short summary lines (`FAILED ... - assert 0`, `-rfE`, `-rx`) render unchanged. Pass it at the 16 call sites -- 20 test functions -- whose assertions never touch traceback text, each annotated with what it does assert on instead. Host-file parses drop from 109 to 18, and the file from 20.15s to 16.70s. The remaining 18 are tests that genuinely assert on traceback output and must keep paying. Levers that did not pay, measured rather than assumed: dropping `capture_output` is worth 0.9ms and only 7 of 74 ported tests can (the rest assert on `record.output`); `collect_tests` over `run_tests` is worth 0.5ms, since collection is nearly the whole config cost; batching rounds into one `Ensemble` saves only config builds, and the cost here is per failing *item* -- `test_teardown_many` did 20 reparses inside a single run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Locating the failing statement of a traceback entry means parsing source into an AST, and `getsource` handed `getstatementrange_ast` the entire file. The AST cache lives on `FormattedExcinfo`, i.e. one per rendered failure, so a large test module reparsed all of itself for every entry of every failure -- ~40ms per entry for a 4000-line file, against ~0.2ms for the ten-line temp file the same test would use under pytester. Narrow the parse to the block the frame belongs to. Full suite goes from 304.45s to 249.88s. `inspect.getblock` rather than the code object's own line extent: `co_lines()` can undershoot, because a trailing `), "msg"` line may carry no bytecode, and cutting it would change the rendered source. Two guards, both needed and both pinned by tests: * `getblock` only walks an indented suite for def/class/decorated code; for anything else `BlockFinder` stops at the first logical line. A module-level frame therefore gets a block that is not the module -- which rendered `???` in place of the conftest line in `test_better_reporting_on_conftest_load_failure`. Falling back whenever the reported line is outside the block covers that, and covers exec'd or generated code whose lines do not match the file. * tokenizing can fail on sources that do not correspond to the frame. The astcache key carries the parse offset rather than the frame's first line: whether narrowing applied depends on the line being reported, so two entries in the same function can disagree, and a cached tree must never be paired with linenos it was not parsed from. Not used here, for the record: `co_positions()` gives the failing *expression* (lines 9-11 of a five-line assert whose statement is 8-12), and `ast.stmt.end_lineno` spans a compound statement's whole suite where pytest shows only its header -- across `src/_pytest` it differs from the current result on 30.6% of lines, 99.5% of those compound. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A `TempPathFactory` built from a config allocates a numbered directory under the global `pytest-of-<user>` root. That scans the root and every sibling run's leftovers, takes a cleanup lock, and leaves a directory behind -- per ensemble. Ensembles are built in the hundreds, so the `tmpdir` plugin cannot be a default. Drop it from DEFAULT_PLUGINS and load it only when `ConfigSpec.tmp_path_factory` supplies a factory, bound over the one the plugin builds by a trylast `pytest_configure`, so the fixtures, retention handling and sessionfinish cleanup all stay the plugin's own. `make_tmp_path_factory(basetemp)` builds one that takes the `--basetemp` path rather than allocating; the `ensemble_tmp_path_factory` fixture roots it in the host test's own `tmp_path`, so it needs no allocation and is cleaned up with the host. Five tests in testing/python/fixtures.py genuinely use `tmp_path` inside the ensemble and now opt in; nothing else in the suite did, which is the point -- ~1000 ensemble configs were registering a plugin they never used. Note this does not move `find_prefixed`: `getbasetemp()` is lazy, so those ensembles never allocated a base temp to begin with. The 19s of directory scanning is pytester's own `tmp_path`, not the ensemble's. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`unraisableexception` was already absent from DEFAULT_PLUGINS, but an ensemble that opted into it would inherit the default gc policy and run a full `gc.collect()` at both the cleanup and the unconfigure site. The heap that walks is the *host* process's - measured at ~243ms a pass in pytest's own suite - so it costs whatever the host happens to be holding rather than anything the ensemble owns. `ConfigSpec.gc_collect_iterations` seeds the stash key the plugin reads, defaulting to none: opting the plugin in gets the unraisable *reporting* without the collection, and a test that needs finalizers flushed for an exception to surface raises it deliberately. pytester keeps its own policy unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`pytester.getitems` builds a config through `parseconfigure`, which does not get pytester's `gc_collect_iterations = 0` the way `inline_run` does -- so each of these three paid two full `gc.collect()` passes over the host suite's heap, ~243ms each, for two lines of assertion. `warn` needs a live config for the hook call, so the parametrized one collects inside an `Ensemble` block; the type-enforcement one only needs the item, so `collect_tests` is enough. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`test_check_equality` and `test_unpacked_marks_added_to_keywords` are about node identity and mark unpacking, not about collecting a filesystem -- which is what the rest of this file tests and what an ensemble deliberately does not do. Note `Node` defines no `__eq__`: equality is identity, and the original passed because `collect_by_name` returned pytester's cached node both times. The port looks the name up twice in one collection rather than collecting twice, which would have compared two distinct objects with equal node ids and failed. `build_module` carries the module-level `pytestmark` as a keyword member, so the class and method marks nest exactly as in the source file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`configured()` runs configure and unconfigure around the block, which is what the original drove by hand. The terminal plugin is opted in so that `-p terminal` names one that is loaded already, and given a private buffer -- an ensemble's terminal reporter must never bind the outer test's stdout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The subject is that `TopRequest.path` and `.fspath` agree with the module's, so it needs `legacypath` opted in but nothing from the filesystem. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Experiment, stacked on top of #14809. Base branch is
scenario-api-poc, so this diff shows only the migration. It exists to answer what a proof of concept cannot: what actually happens when you point the new API at real tests, in bulk.What this is
864 of 1145 tests across 13 files no longer use
pytester. The suite is unchanged where it counts: 4452 passed, 47 skipped, 13 xfailed, 7 xpassed. No test deleted, renamed or reordered.mypysits at its baseline error count and the fullpre-commit run --all-filespasses.testing/python/fixtures.pytesting/test_assertion.pytesting/python/metafunc.pytesting/test_terminal.pytesting/test_skipping.pytesting/test_unittest.pytesting/test_mark.pytesting/python/collect.pytesting/test_runner.pytesting/test_warnings.pytesting/test_subtests.pytesting/test_session.pytesting/test_runner_xunit.pyOne commit per file, so any single port can be dropped. Every test still on
pytestercarries an# ensemble: <reason>comment.Speed — and where it stops paying
Same machine, best of runs, single process.
mainruns the original pytester versions;ensembleruns the ported ones.testing/complete (-n 8)The headline number got worse as more tests were ported, and that is the most useful result here. An earlier state of this branch measured 1.44x / 1.41x with ~34 fewer tests ported. Per file:
test_subtests.pytest_unittest.pypython/fixtures.pytest_assertion.pytest_terminal.pyThe two files that go backwards are exactly the two whose ports lean on
capture_output=True. Loading the terminal plugin costs ~29% per ensemble run (seebench/ensemble_vs_pytester.pyin the base PR), and now that the progress column renders, a rendered ensemble also does more work per test than it used to.So the rule, with numbers behind it:
If suite time were the only goal, the
test_terminal.pyports should be dropped and the branch would sit near 1.4x. They are kept because the point of the exercise is to find out what the API can express, and because a rendered ensemble is still hermetic where a pytester run is not.What the port surfaced
Porting was not a neutral rewrite. It found real bugs — in the tests, in the ensemble API, and in pytest itself.
In pytest.
subtestsrewritesreport.outcome/report.longreprinsidepytest_report_teststatus, a reporting hook. So a top-level test containing a failed subtest is only marked failed if something renders it — any consumer reading reports without a terminal sees un-rewritten outcomes. This made--last-faileda silent no-op under an ensemble:LFPluginsaw the report stillpassedand popped the nodeid the failing subtest had just added. Doing the rewrite inpytest_runtest_logreportwould make it robust. Separately, two tests intest_runner.pycarried a conftest customizing node classes by name — a mechanismpython.pyhasn't consulted in years, with every assertion that would have noticed already commented out.In the tests.
test_failing_teardown_issue9asserted"*1 xfail*", which happily matched the1 xpassed, 1 xfailedsummary and concealed an xpass. Two undefined names had been sitting insidemakepyfilestrings, invisible to ruff and mypy because they were string literals. Several outcome assertions were simply wrong:HookRecorder.assertoutcome()counts any failed report, so setup/teardown failures were recorded asfailedand xfails asskipped, and one test was silently ignoring a teardown error.In the ensemble API — eight defects, every one of the "green test that asserts nothing" kind. All are now fixed on the base PR, found only because real tests were pointed at the API:
Ensemble.run()bypassedpytest_runtestloop, so--maxfail/-x/shouldstopwere silently inert (hit independently by three ports).RunRecordwas built beforepytest_sessionfinish, so anything emitted there — warnings especially — was invisible.assertionplugin was not inDEFAULT_PLUGINS, soutil._reprcomparestayed bound to the host run: an explanation was still produced and tests went green, but the ensemble's own-vv, ini values andpytest_assertrepr_compareimpls were ignored. Swapping the plugin out makes 16 of 33 ports intest_assertion.pyfail, so they are load-bearing.filterwarnings = error. One ported test was depending on the leak and had to be rewritten when it was closed.--strict-markers,-o name=valueandaddoptswere parsed and then dropped.--setup-plan/--setup-only/--setup-showwere inert, because their normalization lived inpytest_cmdline_main.collect_tests()returned[]on a collection failure, making "collects nothing" and "collection exploded" indistinguishable.Ensemble.collect()was not idempotent.What the enablers bought
Three capabilities were added mid-experiment, each in response to something the ports measured:
_determine_show_progress_info()suppressed the progress column whenevergetoption("capture","no") == "no", always true for an ensemble — but the concern behind Use classic console output when -s is used pytest-dev/pytest#3038 is progress interleaving with test output, and a reporter writing to its own buffer has none. This alone unlocked 17 tests intest_terminal.py.run_itemsgoes throughpytest_runtestloop. That is where the terminal defers its final[100%]fill, so 14 of those 17 would otherwise have needed a relaxed final pattern — and five patterns intest_subtests.pythat the first pass had to weaken were restored.module_from_path()which imports a file without touchingsys.modulesor writing bytecode. This is what lets an example script run as itself.That last one changed how example-driven tests are handled. They had been ported by inlining their source, which orphaned eight files under
testing/example_scripts/and moved every reported location into the host test file. They now run the real scripts in place, sotest_funcarg_lookupfailscan assert— a pattern that was unusable while the source was inlined. Every example script under
fixtures/andunittest/is referenced again.What blocks the remaining 281
Measured, and it twice corrected my own ranking. I first expected host-anchored item locations to dominate;
test_terminal.pyquantified it, and the progress column turned out to be larger. With the column fixed, locations are now the top blocker:test_terminal.pyalone): a source written inline in a test body reports the host file, sofile:lineassertions, the<- file>suffix at-vv, andconsole_output_style=times— which groups byreport.location[0]and therefore sees every item as one module — all fail.module_from_pathsidesteps this for scripts that exist on disk; making a synthesized item report its collector's path would close the rest.pytest_cmdline_mainoptions (~30):--collect-only,--fixtures,--markers,--help.extra_pluginsreproduces a rootdir conftest exactly, but nothing below it.wrap_session-only output and exit codes (~15).Two are design findings rather than gaps.
scope="package"degrades silently with noPackagenode, so a test can go green while exercising a different path. And two tests became unportable because of the progress fix — they assert that--capture=nosuppresses the column, which an ensemble's private stream can never do; porting them would assert nothing.Caveats
pytestercoverage, one or two tests were kept as deliberate file-based canaries so real module import and path-based nodeids stay exercised.Per our AI contribution policy: this was researched, designed and implemented with Claude Code under my direction and review.