Support for Python limited API extensions assumes there is only one valid extension suffix

[ngoldbaum]

I am working on ecosystem support for the free-threaded build of Python. Lately I've been thinking about PEP 803, which proposes a new version of the stable ABI that can support the free-threaded build.

Yesterday, while working on a test to validate the possibility of distributing .abi3t.so extensions that are loadable on both builds, I realized that currently Meson hard-codes the index into importlib.machinery.EXTENSION_SUFFIXES to find the suffix to use at build time:

meson/mesonbuild/scripts/python_info.py

Lines 95 to 101 in fb12c9c

	limited_api_suffix = None
	if sys.version_info >= (3, 2):
	try:
	from importlib.machinery import EXTENSION_SUFFIXES
	limited_api_suffix = EXTENSION_SUFFIXES[1]
	except Exception:
	pass

For reference, on my Mac:

goldbaum at Nathans-MBP in ~
○  python
Python 3.14.3 free-threading build (main, Feb 10 2026, 08:41:08) [Clang 17.0.0 (clang-1700.6.3.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib.machinery
>>> importlib.machinery.EXTENSION_SUFFIXES
['.cpython-314t-darwin.so', '.abi3.so', '.so']

Even if PEP 803 doesn't go into CPython for the 3.15 release cycle, IMO it's still worth adding some way for a user to supply the extension name and override that, but still verify that the provided extension suffix is present in the EXTENSION_SUFFIXES list provided by the interpreter. This also has bearing on other proposals from Debian to modify this list: python/cpython#122917. I'm not sure offhand if Debian carries a patch for this.

Once there's a UI in meson, then meson-python can be updated to select the correct extension name depending on the project's build configuration.

Ping @mgorny who I chatted with a little about this.

[eli-schwartz]

I strongly oppose adding a user build option for this. There is only ever one valid extension to use, if you interpret "valid" as meaning "semantically correct" rather than "accepted by all interpreters that should load it".¹

If users were to configure it, that would mean Meson didn't try to do the right thing. Instead Meson should just do the right thing.

I realized that currently Meson hard-codes the index into importlib.machinery.EXTENSION_SUFFIXES to find the suffix to use at build time:

The original design of this was because CPython doesn't actually provide an official API to find out this information, but the ordering of EXTENSION_SUFFIXES is semantically based on "what is the best extension that most fits the current interpreter".

• the first and most preferred extension suffix is the unlimited API / unstable ABI, that is to say, the one with the major-minor version encoded
• after that comes abi3 extensions. They're slower than version specific ones but they match the interpreter
• plain .so is least preferred. It is a legacy holdover from the python 2 era, which never had a stable ABI anyway and only loads undistinguished extensions. It is also "necessary" to support these because ancient python2 projects that don't see much development other than life support, may still be using custom GNU Automake build systems to build "generic dlopen plugins" linking to CPython, i.e. they use the bare name "foo.so" with no real awareness of the CPython ecosystem. They may or may not use the limited API, build against python2 headers instead of python3 headers, etc. If it is possible to be more specific in the extension name, do so with all urgency... but if not, it will probably load. Maybe.

Since current and historical versions of CPython only supported a single stable ABI at a time, it is a "reasonable" assumption to make that indexing like this will get you the "current stable ABI name" even if/when CPython moves to a new revision of the stable ABI. Remember that the primary purpose of the stable ABI is to allow things that are source compatible already, to keep working without "expensive" rebuilds (7+ versions of python can coalesce to currently one or potentially two if lying across a transition point). It was an understandable assumption that any given interpreter would likely only support one of them, and if the transition point hadn't been freethreading I still think that it would have been more likely to be a hard cutover.

Now the question becomes, what ABIs are actually there and what is the best one to use? We are adding one ABI, without removing any old ones. So, should abi4 (which appears to be getting called "the freethreading only ABI, which is a terrible name but people are very attached to it so whatever), be in between the unstable ABI and abi3, or in between abi3 and ".so"?

Specifically, if both abi3 and abi4 are supported, which one is "better"?

I'd say the newer one that you can do more stuff with. Nobody can build abi4 wheels for python 3.8, so abi4 wheels can utilize APIs specific to 3.14+, and should not ever under any circumstances use APIs that were deprecated in 3.14. The only remaining question is: does abi4 provide worse utility than abi3? That is, people should stay on it (abi3) for as long as they can until they are forced off? If so, it would be bad to prefer it when abi3 exists. But I have in no way gotten such an impression.

tl;dr CPython should list EXTENSION_SUFFIXES = ['.cpython-314-archtuple.so', '.abi4.so', '.abi3.so', '.so'] because users will prefer importing abi4 if both exist, and meson will transparently detect the output artifact name to be abi4.

Footnotes

1. and also accepted by interpreters that should not, and crash because they did anyway ↩

[rgommers]

I strongly oppose adding a user build option for this.

I agree that a build option for this is undesirable. I am looking at the PEP design discussion from the perspective of whether it works for Meson and meson-python. I don't really care that much what the filename extension will be, as long as it's unambiguous and build systems can make the right decision without end user input.

It's probably not very useful to try to decide on something in this issue either way before there's an approved PEP.

There is already a way to override this by the way, namely passing a build-details.json with the abi.stable-abi-suffix one wants: https://peps.python.org/pep-0739/#abi-stable-abi-suffix. This is still very new, but it's much better structured than grabbing an extension from EXTENSION_SUFFIXES, so I hope over time we get to a place where everything uses this metadata file.

[ngoldbaum]

does abi4 provide worse utility than abi3? That is, people should stay on it (abi3) for as long as they can until they are forced off? If so, it would be bad to prefer it when abi3 exists. But I have in no way gotten such an impression.

Looking at this again, I think the big issue people will run into is that extensions can't work with the new ABI without updates to any types or modules that are defined statically. Projects generating extensions via bindings generators will have an easier time since the bindings generator handles those details, but projects like SciPy, NumPy, and astropy will all need some amount of work to migrate to heap types and new module initialization APIs before they can ship binaries using the new ABI.

That means I don't think it makes sense for Meson to default to abi3t when a user does a limited API build on the GIL-enabled build and IMO it's correct for the GIL-enabled build to prefer abi3 extensions over abi3t.

Since the free-threaded build has never supported the limited API, it's OK to prefer the new ABI extension there.

[eli-schwartz]

That means I don't think it makes sense for Meson to default to abi3t when a user does a limited API build on the GIL-enabled build

If it will fail to compile then that's a good reason to not default to building them. It's also a good reason to not allow them at all, full stop. No options.

Projects that support the freethreading API and therefore can build for the freethreading ABI need to say so by passing a kwarg in meson.build -- and a kwarg is the opposite of a build option. The kwarg informs Meson that the module is source compatible with the freethreading API even though it's compiling for the GIL-enabled build, and Meson hooks that up.

I still don't see the benefit of building an abi3-cp314+ extension when the extension is tagged as source compatible with abi4 (abi3t).

• When compiling abi3 extensions for python 3.8, abi3 is the only ABI and we need to build them to ship them for cpython interpreters versions 3.8 -- 3.13,
• but an abi3 extension built on python 3.14 GIL-enabled cannot be used on older cpython releases, which means all releases it can be used on, can also load abi4 (abi3t) which, once successfully built, are just as good as abi3.

and IMO it's correct for the GIL-enabled build to prefer abi3 extensions over abi3t.

The argument you offer for not building them is that it isn't source compatible and will therefore not build.

I don't see how this translates to not loading them if they already built and will work. That being said I suppose that cpython can construct its load order based on what is more convenient for Meson (i.e. GIL-enabled build lists abi3 first just so that Meson detects the correct default extension).

[rgommers]

Projects that support the freethreading API and therefore can build for the freethreading ABI need to say so by passing a kwarg in meson.build

I'll need to think about whether we want to allow that for a bit. Because a kwarg in meson.build isn't useful to meson-python, which won't know what wheel tag to produce.

I'd start in 3.15 with:

• Not touching the GIL-enabled behavior at all -> always produce .abi3.so and abi3 wheel tags
• For free-threaded, if limited_api is requested, change from the current error to always producing .abi3t.so and abi3.abi3t wheel tags (assuming the PEP keeps saying to produce that combined tag; alternatively installers can be taught that GIL-enabled interpreters understand plain abi3t).

This is unambiguous and requires zero new kwargs/options/UX in either meson or meson-python.

---

What's being discussed above is if/how to allow producing .abi3t.so and abi3.abi3t wheel tags also when using a GIL-enabled interpreter. That would require both a new config setting in addition to tool.meson-python.limited-api and a new kwarg for Meson's py.extension_module. Or equivalently, letting the existing limited_api kwarg accept '3.15t' as the value.

Adding that is not too difficult, but I'd rather wait until the dust settles before doing so.