concurrent.interpreters.list_all() fails if an interpreter is finalized during the call

[serhiy-storchaka]

Bug report

concurrent.interpreters.list_all() lists the interpreter ids and then creates an Interpreter object for every id:

def list_all():
    """Return all existing interpreters."""
    return [Interpreter(id, _whence=whence)
            for id, whence in _interpreters.list_all(require_ready=True)]

Every allocation in that list comprehension can start a garbage collection. If it finalizes an Interpreter object which owns a reference (_ownsref) and is only kept alive by a reference cycle, __del__() destroys the interpreter, and creating the object for the id which was just listed fails:

import gc
from concurrent import interpreters

interp = interpreters.create()
cycle = []
cycle.append(cycle)
cycle.append(interp)

gc.disable()
del interp, cycle
gc.enable()
gc.set_threshold(1)

interpreters.list_all()

  File "Lib/concurrent/interpreters/__init__.py", line 126, in __new__
    self = _known[id]
KeyError: 1

During handling of the above exception, another exception occurred:

  File "Lib/concurrent/interpreters/__init__.py", line 71, in list_all
    return [Interpreter(id, _whence=whence)
  File "Lib/concurrent/interpreters/__init__.py", line 136, in __new__
    _interpreters.incref(id)
concurrent.interpreters.InterpreterNotFoundError: unrecognized interpreter ID 1

gc.set_threshold(1) only makes it reliable; a collection at the wrong moment is enough.

Observed in the wild in the tearDown() of test.test_interpreters.test_api.TestInterpreterCall.test_call_in_thread, which calls list_all() in clean_up_interpreters(). It failed in every run of the whole test suite on the CI of one pull request (six times, in two different jobs), and never when test_interpreters was run alone: https://github.com/python/cpython/actions/runs/32110741697/job/95647559014

The interpreter listed but already gone should probably be skipped, rather than making the whole call fail.

Linked PRs

• gh-155997: Fix list_all() if an interpreter is destroyed during the call #155998
• [3.15] gh-155997: Fix list_all() if an interpreter is destroyed during the call (GH-155998) #156004
• [3.14] gh-155997: Fix list_all() if an interpreter is destroyed during the call (GH-155998) #156005
• [3.13] gh-155997: Fix list_all() if an interpreter is destroyed during the call (GH-155998) #156006