Race conditions when using 'tox --parallel'

[kenahoo]

Describe the bug
I've been getting a lot of errors about "no such table" or "table already exists" when trying to run tox with coverage reporting. It looks a fair amount like #1303, but I'm opening a new issue because it looks like at least one problem was already solved in that ticket, and the existing issue has gotten kind of complicated over time.

This means that users can't use parallel testing at the same time as coverage reporting.

To Reproduce

1. Check out the minimal project I created to show the bug:
   git clone git@github.com:kenahoo/pkg-test-coverage-bug.git
2. Have a working version of python 3.8 and 3.9 available to tox.
3. Run python -m tox and (hopefully) observe that everything passes.
4. Run python -m tox --parallel auto and (hopefully) observe that things fail with the above errors.

If the above doesn't demonstrate the problem, I can try to provide additional context, let me know.

Notes
This is important in my organization, because we need to assure compatibility with several dependency chains, and the tests take a while, so parallel testing is very important to get timely feedback.

[nedbat]

I don't see the errors you mention:

% python -m tox --parallel auto
⠙ [6/6] py38-more_itertools812 | py38-more_itertools813 | py38-more_itertools814 | py39-more_itertools812 |.../usr/local/virtualenvs/tmp-46d8ab1a6519ed3/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
py39-more_itertools814: OK ✔ in 3.93 seconds
py39-more_itertools812: OK ✔ in 3.96 seconds
py38-more_itertools813: OK ✔ in 3.96 seconds
py38-more_itertools814: OK ✔ in 3.99 seconds
py39-more_itertools813: OK ✔ in 3.99 seconds
  py38-more_itertools812: OK (4.05=setup[3.03]+cmd[0.60,0.21,0.21] seconds)
  py38-more_itertools813: OK (3.96=setup[2.92]+cmd[0.61,0.22,0.22] seconds)
  py38-more_itertools814: OK (3.99=setup[2.96]+cmd[0.60,0.21,0.21] seconds)
  py39-more_itertools812: OK (3.96=setup[2.93]+cmd[0.57,0.22,0.23] seconds)
  py39-more_itertools813: OK (3.99=setup[2.97]+cmd[0.57,0.22,0.22] seconds)
  py39-more_itertools814: OK (3.93=setup[2.90]+cmd[0.58,0.22,0.23] seconds)
  congratulations :) (4.30 seconds)

But I wouldn't expect one .coverage file to be used by multiple processes simultaneously anyway. You can use the --parallel (or [run] parallel=true setting) to have a different data file for each process, then use coverage combine to combine them together before reporting.

I'll close this issue, but please do re-open if I've misunderstood something.

[kenahoo]

Hmm, I'm curious why we're seeing different results. I just did a fresh clone of the repo and ran the parallel instantiation, here's the output:

% python -m tox --parallel auto
py38-more_itertools814: FAIL ✖ in 1.68 seconds
.pkg: _optional_hooks> python /Users/kwilliams/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_sdist> python /Users/kwilliams/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: prepare_metadata_for_build_wheel> python /Users/kwilliams/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_sdist> python /Users/kwilliams/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
py38-more_itertools814: install_package> python -I -m pip install --force-reinstall --no-deps /private/tmp/pkg-test-coverage-bug/.tox/.tmp/package/7/test_pkg-0.1.tar.gz
py38-more_itertools814: commands[0]> python -m coverage run -m pytest
====================================================================================== test session starts =======================================================================================
platform darwin -- Python 3.8.12, pytest-7.2.0, pluggy-1.0.0
cachedir: .tox/py38-more_itertools814/.pytest_cache
rootdir: /private/tmp/pkg-test-coverage-bug
collected 1 item                                                                                                                                                                                 

test/test_code.py .                                                                                                                                                                        [100%]

======================================================================================= 1 passed in 0.01s ========================================================================================
py38-more_itertools814: exit 1 (0.30 seconds) /private/tmp/pkg-test-coverage-bug> python -m coverage run -m pytest pid=31356
py38-more_itertools812: FAIL ✖ in 1.69 seconds
py38-more_itertools812: install_package> python -I -m pip install --force-reinstall --no-deps /private/tmp/pkg-test-coverage-bug/.tox/.tmp/package/12/test_pkg-0.1.tar.gz
py38-more_itertools812: commands[0]> python -m coverage run -m pytest
====================================================================================== test session starts =======================================================================================
platform darwin -- Python 3.8.12, pytest-7.2.0, pluggy-1.0.0
cachedir: .tox/py38-more_itertools812/.pytest_cache
rootdir: /private/tmp/pkg-test-coverage-bug
collected 1 item                                                                                                                                                                                 

test/test_code.py .                                                                                                                                                                        [100%]

======================================================================================= 1 passed in 0.02s ========================================================================================
py38-more_itertools812: exit 1 (0.31 seconds) /private/tmp/pkg-test-coverage-bug> python -m coverage run -m pytest pid=31354
py39-more_itertools812: OK ✔ in 1.77 seconds
py39-more_itertools814: OK ✔ in 1.81 seconds
py39-more_itertools813: OK ✔ in 1.83 seconds
  py38-more_itertools812: FAIL code 1 (1.69=setup[1.38]+cmd[0.31] seconds)
  py38-more_itertools813: OK (1.88=setup[1.40]+cmd[0.30,0.09,0.08] seconds)
  py38-more_itertools814: FAIL code 1 (1.68=setup[1.38]+cmd[0.30] seconds)
  py39-more_itertools812: OK (1.77=setup[1.31]+cmd[0.26,0.10,0.10] seconds)
  py39-more_itertools813: OK (1.83=setup[1.39]+cmd[0.26,0.10,0.09] seconds)
  py39-more_itertools814: OK (1.81=setup[1.36]+cmd[0.26,0.10,0.09] seconds)
  evaluation failed :( (1.93 seconds)
Traceback (most recent call last):
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/cmdline.py", line 837, in do_run
    runner.run()
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/execfile.py", line 198, in run
    exec(code, main_mod.__dict__)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/pytest/__main__.py", line 5, in <module>
    raise SystemExit(pytest.console_main())
SystemExit: ExitCode.OK

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 1121, in _execute
    return self.con.execute(sql, parameters)
sqlite3.OperationalError: no such table: coverage_schema

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 1126, in _execute
    return self.con.execute(sql, parameters)
sqlite3.OperationalError: no such table: coverage_schema

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 273, in _read_db
    schema_version, = db.execute_one("select version from coverage_schema")
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 1178, in execute_one
    with self.execute(sql, parameters) as cur:
  File "/Users/kwilliams/.pyenv/versions/3.8.12/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 1151, in execute
    cur = self._execute(sql, parameters)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 1143, in _execute
    raise DataError(f"Couldn't use data file {self.filename!r}: {msg}") from exc
coverage.exceptions.DataError: Couldn't use data file '/private/tmp/pkg-test-coverage-bug/.coverage': no such table: coverage_schema

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/kwilliams/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/kwilliams/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/__main__.py", line 8, in <module>
    sys.exit(main())
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/cmdline.py", line 950, in main
    status = CoverageScript().command_line(argv)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/cmdline.py", line 665, in command_line
    return self.do_run(options, args)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/cmdline.py", line 844, in do_run
    self.coverage.save()
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/control.py", line 733, in save
    data = self.get_data()
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/control.py", line 806, in get_data
    if self._collector and self._collector.flush_data():
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/collector.py", line 475, in flush_data
    self.covdata.add_lines(self.mapped_file_dict(self.data))
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 239, in _wrapped
    return method(self, *args, **kwargs)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 472, in add_lines
    self._choose_lines_or_arcs(lines=True)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 536, in _choose_lines_or_arcs
    with self._connect() as con:
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 322, in _connect
    self._open_db()
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 267, in _open_db
    self._read_db()
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 276, in _read_db
    self._init_db(db)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 304, in _init_db
    db.executescript(SCHEMA)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 1226, in executescript
    self.con.executescript(script).close()
sqlite3.OperationalError: table coverage_schema already exists
Traceback (most recent call last):
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/cmdline.py", line 837, in do_run
    runner.run()
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/execfile.py", line 198, in run
    exec(code, main_mod.__dict__)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/pytest/__main__.py", line 5, in <module>
    raise SystemExit(pytest.console_main())
SystemExit: ExitCode.OK

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 1121, in _execute
    return self.con.execute(sql, parameters)
sqlite3.OperationalError: no such table: coverage_schema

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 1126, in _execute
    return self.con.execute(sql, parameters)
sqlite3.OperationalError: no such table: coverage_schema

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 273, in _read_db
    schema_version, = db.execute_one("select version from coverage_schema")
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 1178, in execute_one
    with self.execute(sql, parameters) as cur:
  File "/Users/kwilliams/.pyenv/versions/3.8.12/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 1151, in execute
    cur = self._execute(sql, parameters)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 1143, in _execute
    raise DataError(f"Couldn't use data file {self.filename!r}: {msg}") from exc
coverage.exceptions.DataError: Couldn't use data file '/private/tmp/pkg-test-coverage-bug/.coverage': no such table: coverage_schema

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/kwilliams/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/kwilliams/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/__main__.py", line 8, in <module>
    sys.exit(main())
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/cmdline.py", line 950, in main
    status = CoverageScript().command_line(argv)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/cmdline.py", line 665, in command_line
    return self.do_run(options, args)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/cmdline.py", line 844, in do_run
    self.coverage.save()
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/control.py", line 733, in save
    data = self.get_data()
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/control.py", line 806, in get_data
    if self._collector and self._collector.flush_data():
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/collector.py", line 475, in flush_data
    self.covdata.add_lines(self.mapped_file_dict(self.data))
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 239, in _wrapped
    return method(self, *args, **kwargs)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 472, in add_lines
    self._choose_lines_or_arcs(lines=True)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 536, in _choose_lines_or_arcs
    with self._connect() as con:
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 322, in _connect
    self._open_db()
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 267, in _open_db
    self._read_db()
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 276, in _read_db
    self._init_db(db)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 304, in _init_db
    db.executescript(SCHEMA)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 1226, in executescript
    self.con.executescript(script).close()
sqlite3.OperationalError: table coverage_schema already exists

As far as your suggestion to use --parallel, have I understood it correctly to mean applying a patch like the following to the tox.ini?

diff --git a/tox.ini b/tox.ini
index d6910e7..c9fbab8 100644
--- a/tox.ini
+++ b/tox.ini
@@ -7,6 +7,7 @@ deps = pytest
        more_itertools812: more_itertools>=8.12,<8.13
        more_itertools813: more_itertools>=8.13,<8.14
        more_itertools814: more_itertools>=8.14,<8.15
-commands = python -m coverage run -m pytest
+commands = python -m coverage run --parallel -m pytest
+           python -m coverage combine
            python -m coverage xml
            python -m coverage html

After I make that change, the error changes to a different error, but it still doesn't succeed:

% python -m tox --parallel auto
py39-more_itertools813: FAIL ✖ in 1.67 seconds
.pkg: _optional_hooks> python /Users/kwilliams/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_sdist> python /Users/kwilliams/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: prepare_metadata_for_build_wheel> python /Users/kwilliams/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_sdist> python /Users/kwilliams/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
py39-more_itertools813: install_package> python -I -m pip install --force-reinstall --no-deps /private/tmp/pkg-test-coverage-bug/.tox/.tmp/package/16/test_pkg-0.1.tar.gz
py39-more_itertools813: commands[0]> python -m coverage run --parallel -m pytest
====================================================================================== test session starts =======================================================================================
platform darwin -- Python 3.9.13, pytest-7.2.0, pluggy-1.0.0
cachedir: .tox/py39-more_itertools813/.pytest_cache
rootdir: /private/tmp/pkg-test-coverage-bug
collected 1 item                                                                                                                                                                                 

test/test_code.py .                                                                                                                                                                        [100%]

======================================================================================= 1 passed in 0.01s ========================================================================================
py39-more_itertools813: commands[1]> python -m coverage combine
No data to combine
py39-more_itertools813: exit 1 (0.09 seconds) /private/tmp/pkg-test-coverage-bug> python -m coverage combine pid=31541
py38-more_itertools814: FAIL ✖ in 1.7 seconds
py38-more_itertools814: install_package> python -I -m pip install --force-reinstall --no-deps /private/tmp/pkg-test-coverage-bug/.tox/.tmp/package/13/test_pkg-0.1.tar.gz
py38-more_itertools814: commands[0]> python -m coverage run --parallel -m pytest
====================================================================================== test session starts =======================================================================================
platform darwin -- Python 3.8.12, pytest-7.2.0, pluggy-1.0.0
cachedir: .tox/py38-more_itertools814/.pytest_cache
rootdir: /private/tmp/pkg-test-coverage-bug
collected 0 items / 1 error                                                                                                                                                                      

============================================================================================= ERRORS =============================================================================================
_________________________________________________________________________________ ERROR collecting test session __________________________________________________________________________________
.tox/py38-more_itertools814/lib/python3.8/site-packages/_pytest/runner.py:339: in from_call
    result: Optional[TResult] = func()
.tox/py38-more_itertools814/lib/python3.8/site-packages/_pytest/runner.py:370: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
.tox/py38-more_itertools814/lib/python3.8/site-packages/_pytest/main.py:738: in collect
    for x in self._collectfile(path):
.tox/py38-more_itertools814/lib/python3.8/site-packages/_pytest/main.py:572: in _collectfile
    assert (
E   AssertionError: PosixPath('/private/tmp/pkg-test-coverage-bug/.coverage.spmacpro140.31539.834052-journal') is not a file (isdir=False, exists=False, islink=False)
==================================================================================== short test summary info =====================================================================================
ERROR  - AssertionError: PosixPath('/private/tmp/pkg-test-coverage-bug/.coverage.spmacpro140.31539.834052-journal') is not a file (isdir=False, exists=False, islink=False)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================================================== 1 error in 0.08s ========================================================================================
py38-more_itertools814: exit 2 (0.36 seconds) /private/tmp/pkg-test-coverage-bug> python -m coverage run --parallel -m pytest pid=31536
py38-more_itertools813: FAIL ✖ in 1.7 seconds
py38-more_itertools813: install_package> python -I -m pip install --force-reinstall --no-deps /private/tmp/pkg-test-coverage-bug/.tox/.tmp/package/17/test_pkg-0.1.tar.gz
py38-more_itertools813: commands[0]> python -m coverage run --parallel -m pytest
====================================================================================== test session starts =======================================================================================
platform darwin -- Python 3.8.12, pytest-7.2.0, pluggy-1.0.0
cachedir: .tox/py38-more_itertools813/.pytest_cache
rootdir: /private/tmp/pkg-test-coverage-bug
collected 0 items / 1 error                                                                                                                                                                      

============================================================================================= ERRORS =============================================================================================
_________________________________________________________________________________ ERROR collecting test session __________________________________________________________________________________
.tox/py38-more_itertools813/lib/python3.8/site-packages/_pytest/runner.py:339: in from_call
    result: Optional[TResult] = func()
.tox/py38-more_itertools813/lib/python3.8/site-packages/_pytest/runner.py:370: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
.tox/py38-more_itertools813/lib/python3.8/site-packages/_pytest/main.py:738: in collect
    for x in self._collectfile(path):
.tox/py38-more_itertools813/lib/python3.8/site-packages/_pytest/main.py:572: in _collectfile
    assert (
E   AssertionError: PosixPath('/private/tmp/pkg-test-coverage-bug/.coverage.spmacpro140.31539.834052-journal') is not a file (isdir=False, exists=False, islink=False)
==================================================================================== short test summary info =====================================================================================
ERROR  - AssertionError: PosixPath('/private/tmp/pkg-test-coverage-bug/.coverage.spmacpro140.31539.834052-journal') is not a file (isdir=False, exists=False, islink=False)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================================================== 1 error in 0.08s ========================================================================================
py38-more_itertools813: exit 2 (0.36 seconds) /private/tmp/pkg-test-coverage-bug> python -m coverage run --parallel -m pytest pid=31537
py38-more_itertools812: FAIL ✖ in 1.7 seconds
py38-more_itertools812: install_package> python -I -m pip install --force-reinstall --no-deps /private/tmp/pkg-test-coverage-bug/.tox/.tmp/package/15/test_pkg-0.1.tar.gz
py38-more_itertools812: commands[0]> python -m coverage run --parallel -m pytest
====================================================================================== test session starts =======================================================================================
platform darwin -- Python 3.8.12, pytest-7.2.0, pluggy-1.0.0
cachedir: .tox/py38-more_itertools812/.pytest_cache
rootdir: /private/tmp/pkg-test-coverage-bug
collected 0 items / 1 error                                                                                                                                                                      

============================================================================================= ERRORS =============================================================================================
_________________________________________________________________________________ ERROR collecting test session __________________________________________________________________________________
.tox/py38-more_itertools812/lib/python3.8/site-packages/_pytest/runner.py:339: in from_call
    result: Optional[TResult] = func()
.tox/py38-more_itertools812/lib/python3.8/site-packages/_pytest/runner.py:370: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
.tox/py38-more_itertools812/lib/python3.8/site-packages/_pytest/main.py:738: in collect
    for x in self._collectfile(path):
.tox/py38-more_itertools812/lib/python3.8/site-packages/_pytest/main.py:572: in _collectfile
    assert (
E   AssertionError: PosixPath('/private/tmp/pkg-test-coverage-bug/.coverage.spmacpro140.31539.834052-journal') is not a file (isdir=False, exists=False, islink=False)
==================================================================================== short test summary info =====================================================================================
ERROR  - AssertionError: PosixPath('/private/tmp/pkg-test-coverage-bug/.coverage.spmacpro140.31539.834052-journal') is not a file (isdir=False, exists=False, islink=False)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================================================== 1 error in 0.08s ========================================================================================
py38-more_itertools812: exit 2 (0.36 seconds) /private/tmp/pkg-test-coverage-bug> python -m coverage run --parallel -m pytest pid=31538
py39-more_itertools814: OK ✔ in 1.84 seconds
  py38-more_itertools812: FAIL code 2 (1.70=setup[1.34]+cmd[0.36] seconds)
  py38-more_itertools813: FAIL code 2 (1.70=setup[1.34]+cmd[0.36] seconds)
  py38-more_itertools814: FAIL code 2 (1.70=setup[1.34]+cmd[0.36] seconds)
  py39-more_itertools812: OK (1.88=setup[1.36]+cmd[0.26,0.10,0.08,0.08] seconds)
  py39-more_itertools813: FAIL code 1 (1.67=setup[1.33]+cmd[0.25,0.09] seconds)
  py39-more_itertools814: OK (1.84=setup[1.31]+cmd[0.26,0.10,0.09,0.08] seconds)
  evaluation failed :( (1.94 seconds)

[kenahoo]

I'll close this issue, but please do re-open if I've misunderstood something.

@nedbat I don't seem to have the ability to re-open the issue, but I've posted the comments above.

[kenahoo]

Trying to make a reproducible environment - here's a Dockerfile that I used, instead of relying on my local MacOS environment:

FROM ubuntu:22.04

# Install preliminaries
RUN DEBIAN_FRONTEND=noninteractive apt-get -y update && apt-get -y install git
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev


# Install pyenv
RUN git clone https://github.com/pyenv/pyenv.git ~/.pyenv
RUN echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
RUN echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
RUN echo 'if command -v pyenv 1>/dev/null 2>&1; then\n eval "$(pyenv init -)"\nfi' >> ~/.bashrc
ENV HOME="/root"
ENV PYENV_ROOT $HOME/.pyenv
ENV PATH $PYENV_ROOT/shims:$PYENV_ROOT/bin:$PATH

# Install some python versions
RUN pyenv install 3.8.16 3.9.16
RUN pyenv install 3.10.9
RUN pyenv global 3.10.9

RUN git clone https://github.com/kenahoo/pkg-test-coverage-bug.git /tmp/pkg-test-coverage-bug
WORKDIR /tmp/pkg-test-coverage-bug
RUN pyenv local 3.8.16 3.9.16
RUN pip install tox

# Now ready to run 'while true; do python -m tox --parallel=auto || break; done'


CMD /bin/bash

After that is built (using docker build -t pkg-test-coverage-bug .), run (using docker run -it pkg-test-coverage-bug) and run repeatedly until failure (using while true; do python -m tox --parallel=auto || break; done).

For me, in this Docker environment, it took a few iterations to see the failure:

# !8
while true; do python -m tox --parallel=auto || break; done
py39-more_itertools812: OK ✔ in 1.73 seconds
py38-more_itertools813: OK ✔ in 2 seconds
py39-more_itertools813: OK ✔ in 2.17 seconds
py38-more_itertools814: OK ✔ in 2.21 seconds
py38-more_itertools812: OK ✔ in 2.26 seconds
  py38-more_itertools812: OK (2.25=setup[1.42]+cmd[0.46,0.28,0.08] seconds)
  py38-more_itertools813: OK (2.00=setup[1.17]+cmd[0.42,0.25,0.15] seconds)
  py38-more_itertools814: OK (2.21=setup[1.26]+cmd[0.65,0.13,0.17] seconds)
  py39-more_itertools812: OK (1.72=setup[1.04]+cmd[0.28,0.16,0.25] seconds)
  py39-more_itertools813: OK (2.17=setup[1.13]+cmd[0.67,0.25,0.11] seconds)
  py39-more_itertools814: OK (1.41=setup[1.05]+cmd[0.22,0.07,0.07] seconds)
  congratulations :) (3.18 seconds)
py38-more_itertools812: OK ✔ in 1.59 seconds
py39-more_itertools813: OK ✔ in 1.66 seconds
py38-more_itertools813: OK ✔ in 1.85 seconds
py38-more_itertools814: OK ✔ in 2.14 seconds
py39-more_itertools812: OK ✔ in 2.21 seconds
  py38-more_itertools812: OK (1.57=setup[0.97]+cmd[0.25,0.18,0.17] seconds)
  py38-more_itertools813: OK (1.85=setup[1.03]+cmd[0.44,0.20,0.18] seconds)
  py38-more_itertools814: OK (2.13=setup[1.20]+cmd[0.60,0.17,0.16] seconds)
  py39-more_itertools812: OK (2.21=setup[1.29]+cmd[0.59,0.18,0.15] seconds)
  py39-more_itertools813: OK (1.66=setup[1.16]+cmd[0.24,0.14,0.12] seconds)
  py39-more_itertools814: OK (1.44=setup[1.04]+cmd[0.24,0.07,0.08] seconds)
  congratulations :) (3.05 seconds)
py39-more_itertools813: OK ✔ in 1.58 seconds
py38-more_itertools814: OK ✔ in 1.86 seconds
py39-more_itertools812: OK ✔ in 1.9 seconds
py38-more_itertools813: OK ✔ in 2.1 seconds
py38-more_itertools812: OK ✔ in 2.19 seconds
  py38-more_itertools812: OK (2.18=setup[1.29]+cmd[0.62,0.11,0.16] seconds)
  py38-more_itertools813: OK (2.10=setup[1.22]+cmd[0.40,0.32,0.16] seconds)
  py38-more_itertools814: OK (1.86=setup[1.32]+cmd[0.37,0.07,0.10] seconds)
  py39-more_itertools812: OK (1.90=setup[1.02]+cmd[0.33,0.23,0.32] seconds)
  py39-more_itertools813: OK (1.55=setup[1.07]+cmd[0.26,0.09,0.12] seconds)
  py39-more_itertools814: OK (1.49=setup[1.11]+cmd[0.21,0.07,0.09] seconds)
  congratulations :) (3.09 seconds)
py38-more_itertools813: OK ✔ in 1.51 seconds
py38-more_itertools812: OK ✔ in 1.52 seconds
py39-more_itertools813: OK ✔ in 1.51 seconds
py39-more_itertools812: OK ✔ in 1.54 seconds
py38-more_itertools814: OK ✔ in 1.58 seconds
  py38-more_itertools812: OK (1.52=setup[1.13]+cmd[0.24,0.08,0.08] seconds)
  py38-more_itertools813: OK (1.51=setup[1.11]+cmd[0.23,0.08,0.09] seconds)
  py38-more_itertools814: OK (1.58=setup[1.19]+cmd[0.24,0.07,0.07] seconds)
  py39-more_itertools812: OK (1.54=setup[1.15]+cmd[0.21,0.10,0.08] seconds)
  py39-more_itertools813: OK (1.51=setup[1.15]+cmd[0.22,0.08,0.07] seconds)
  py39-more_itertools814: OK (0.97=setup[0.66]+cmd[0.18,0.06,0.07] seconds)
  congratulations :) (2.52 seconds)
py39-more_itertools813: OK ✔ in 1.34 seconds
py39-more_itertools812: OK ✔ in 1.43 seconds
py38-more_itertools812: OK ✔ in 1.48 seconds
py38-more_itertools813: OK ✔ in 1.5 seconds
py38-more_itertools814: OK ✔ in 1.54 seconds
  py38-more_itertools812: OK (1.48=setup[1.09]+cmd[0.25,0.07,0.07] seconds)
  py38-more_itertools813: OK (1.50=setup[1.08]+cmd[0.23,0.07,0.11] seconds)
  py38-more_itertools814: OK (1.54=setup[1.12]+cmd[0.27,0.07,0.07] seconds)
  py39-more_itertools812: OK (1.42=setup[1.04]+cmd[0.23,0.08,0.08] seconds)
  py39-more_itertools813: OK (1.34=setup[0.98]+cmd[0.21,0.07,0.08] seconds)
  py39-more_itertools814: OK (1.02=setup[0.70]+cmd[0.19,0.07,0.07] seconds)
  congratulations :) (2.39 seconds)
py39-more_itertools813: OK ✔ in 1.3 seconds
py39-more_itertools812: OK ✔ in 1.32 seconds
py38-more_itertools813: OK ✔ in 1.38 seconds
py38-more_itertools812: OK ✔ in 1.38 seconds
py38-more_itertools814: OK ✔ in 1.41 seconds
  py38-more_itertools812: OK (1.38=setup[1.00]+cmd[0.23,0.08,0.07] seconds)
  py38-more_itertools813: OK (1.38=setup[0.98]+cmd[0.24,0.08,0.07] seconds)
  py38-more_itertools814: OK (1.41=setup[0.99]+cmd[0.28,0.07,0.07] seconds)
  py39-more_itertools812: OK (1.32=setup[0.95]+cmd[0.20,0.08,0.08] seconds)
  py39-more_itertools813: OK (1.29=setup[0.90]+cmd[0.24,0.07,0.08] seconds)
  py39-more_itertools814: OK (0.99=setup[0.66]+cmd[0.19,0.07,0.07] seconds)
  congratulations :) (2.33 seconds)
py39-more_itertools812: FAIL ✖ in 1.27 seconds
.pkg: _optional_hooks> python /root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_sdist> python /root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: prepare_metadata_for_build_wheel> python /root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_sdist> python /root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
py39-more_itertools812: install_package> python -I -m pip install --force-reinstall --no-deps /tmp/pkg-test-coverage-bug/.tox/.tmp/package/134/test_pkg-0.1.tar.gz
py39-more_itertools812: commands[0]> python -m coverage run -m pytest
===================================================================================================== test session starts =====================================================================================================
platform linux -- Python 3.9.16, pytest-7.2.0, pluggy-1.0.0
cachedir: .tox/py39-more_itertools812/.pytest_cache
rootdir: /tmp/pkg-test-coverage-bug
collected 1 item                                                                                                                                                                                                              

test/test_code.py .                                                                                                                                                                                                     [100%]

====================================================================================================== 1 passed in 0.01s ======================================================================================================
py39-more_itertools812: commands[1]> python -m coverage xml
No data to report.
py39-more_itertools812: exit 1 (0.06 seconds) /tmp/pkg-test-coverage-bug> python -m coverage xml pid=5280
py38-more_itertools812: OK ✔ in 1.47 seconds
py38-more_itertools813: OK ✔ in 1.48 seconds
py39-more_itertools813: OK ✔ in 1.48 seconds
py38-more_itertools814: OK ✔ in 1.52 seconds
  py38-more_itertools812: OK (1.47=setup[1.06]+cmd[0.24,0.10,0.07] seconds)
  py38-more_itertools813: OK (1.48=setup[1.08]+cmd[0.24,0.08,0.08] seconds)
  py38-more_itertools814: OK (1.52=setup[1.12]+cmd[0.25,0.07,0.07] seconds)
  py39-more_itertools812: FAIL code 1 (1.27=setup[0.99]+cmd[0.22,0.06] seconds)
  py39-more_itertools813: OK (1.48=setup[1.12]+cmd[0.21,0.08,0.08] seconds)
  py39-more_itertools814: OK (0.97=setup[0.66]+cmd[0.18,0.06,0.07] seconds)
  evaluation failed :( (2.29 seconds)

[nedbat]

If this is still a problem for you, can you try adding this line to your coverage configuration?

[run]
data_file = .coverage-${TOX_ENV_NAME}

[hneiva]

Interesting, didn't think about that. So every thread will produce a different coverage report, then aggregate in the end?