Should PyFloat_Pack8/PyFloat_Unpack8 be infallible?

[encukou]

Since 3.11, building CPython requires support for “IEEE” floating-point format and NaN. The discussion's consensus seems to be “CPython requires IEEE 754 floats, but the Python language does not.” The requirement was added to simplify internals and avoid untested code paths, but, as Greg says on the thread:

Should a meaningful large-os platform come along that promotes the use of a
different format available from C we could make use of that and loosen the
policy as needed.

---

A 3.15 change made PyFloat_Pack8 and PyFloat_Unpack8 always succeed on CPython. I think that goes against the decision: error-less conversion means a 1:1 mapping, i.e. it effectively requires that double is IEEE binary64. Or at least that NaNs and infinities are available. Changing this (“loosening the policy”) would be painful: we'd probably need to remove the now-fallible functions, with a suitable deprecation period.

So, it would make sense to me to keep PyFloat_Pack* and PyFloat_Unpack8 fallible; infallibility should be an internal implementation detail, which we can drop in the next version.

What's your opinion?

[skirpichev]

python/cpython#145845 made PyFloat_Pack8 and PyFloat_Unpack8 always succeed on CPython.

Minor correction: all PyFloat_Pack* and PyFloat_Unpack8 always succeed.

Major correction: it's a CPython implementation detail. The documentation explicitly says that these function may fail:

Return value: 0 if all is OK, -1 if error (and an exception is set, most likely OverflowError).

or

Return value: The unpacked double. On error, this is -1.0 and PyErr_Occurred() is true (and an exception is set, most likely OverflowError).

I think that goes against the decision: error-less conversion means a 1:1 mapping, i.e. it effectively requires that double is IEEE binary64.

Yes, of course. But it's already a decision made for CPython.

Are you trying to take into account possible changing of this in future? That would be a big and painful change, anyway. I believe that removal of impl-detail blocks (following PEP 387) for PyFloat_* API will be an easiest part of this.

So, it would make sense to me to keep PyFloat_Pack* and PyFloat_Unpack8 fallible

They are.

infallibility should be an internal implementation detail

Why, how this restricts alternative Python implementations, using Python C-API? They are free to raise exceptions. People just can't rely on CPython implementation details, if they are using, say, PyPy or GraalPy.

[vstinner]

Yes, IEEE 754 floating point numbers and Not-a-Number (NaN) are required to build Python. So it sounds reasonable to me to expect that C double is IEEE 754 binary64 and that infinities and NaN are available. And it sounds reasonable to me to mark PyFloat_Pack8 and PyFloat_Unpack8 as "cannot fail" in the doc, for the CPython implementation (which is already done).

[Yhg1s]

FYI, Petr discussed this with the SC today, and we're in favour of leaving the APIs documented as fallible even if they can't fail on any currently supported (or known) platforms.

[skirpichev]

we're in favour of leaving the APIs documented as fallible even if they can't fail on any currently supported (or known) platforms.

This is unclear. As I already noticed: API already documented as fallible. The only difference that we document also: PyFloat_Pack* and PyFloat_Unpack8 can't fail in CPython.

Should we keep things as is?

Or should we remove impl-detail blocks?

[encukou]

Yes, remove the impl-detail blocks. I'll do it later this week.

The infallibility should not be a guaranteed CPython implementation detail. PyFloat_Pack* and PyFloat_Unpack8 can't fail in current CPython. In the future (when CPython is ported to a non-IEEE platform and the build requirement is loosened), they can start failing again.

removal of impl-detail blocks (following PEP 387) for PyFloat_* API will be an easiest part of [supporting non-IEEE platforms]

We can't just make functions fail and remove the docs that allow people to skip error checking. We'd likely need to remove the functions and add new fallible ones under new names.
(Of course, some people will skip the error checking anyway -- but that's technical debt & possible future bug in their code.)

[skirpichev]

Yes, remove the impl-detail blocks. I'll do it later this week.

I think there is not need to rush. Why not wait SC reply?

The infallibility should not be a guaranteed CPython implementation detail.

Why not? This is something, that was asked before.

PyFloat_Pack* and PyFloat_Unpack8 can't fail in current CPython.

But can in PyPy or GraalPy. So, why it's an issue?

[Yhg1s]

Yes, remove the impl-detail blocks. I'll do it later this week.

I think there is not need to rush. Why not wait SC reply?

We already replied. We're with Petr on this.

The infallibility should not be a guaranteed CPython implementation detail.

Why not? This is something, that was asked before.

Because we don't want third-party module code to need to require an IEEE 754 platform, and we don't want to close the door on porting CPython to a non-IEEE 754 platform.

[skirpichev]

We already replied. We're with Petr on this.

Sorry, but this wasn't clear before.

(1) Because we don't want third-party module code to need to require an IEEE 754 platform, (2) and we don't want to close the door on porting CPython to a non-IEEE 754 platform.

(1) - in first, there is no way to require IEEE 754 platform from Python implementation :-)
(2) - this is not something, that CPython implementation details could constraint, unless you think that Python language specification do this.

[Yhg1s]

(1) Because we don't want third-party module code to need to require an IEEE 754 platform, (2) and we don't want to close the door on porting CPython to a non-IEEE 754 platform.

(1) - in first, there is no way to require IEEE 754 platform from Python implementation :-)

There is? We do? I'm not sure what point you're trying to make here. If the result of running Python on a non-IEEE 754 platform is incorrect, then we require an IEEE 754 platform. And we already do, currently, both at runtime and internally, in CPython's code. The question is whether we extent that expectation to third-party code, meaning code written by others, which, though written for CPython and against the CPython C API, may or may not end up being used with CPython, and may of course end up being used with future versions of CPython. We don't want to leak the current requirement of an IEEE 754 platform into the C API. We want to try and limit the changes necessary to support a non-IEEE 754 platform to CPython itself, not the C API we expose.

(2) - this is not something, that CPython implementation details could constraint, unless you think that Python language specification do this.

We're not talking about adding a constraint here. We're talking about not having a constraint in the C API design.

[skirpichev]

whether we extent that expectation to third-party code, meaning code written by others

I don't think that currently we do this: documentation explicitly says, that our API calls might fail.

though written for CPython and against the CPython C API, may or may not end up being used with CPython

Well, if code written with the CPython C API ended up being used with other Python implementation, it must avoid using implementation details of the CPython. From docs it's obvious how to write such code. So, what real constraint imposes this on third-party code?

We're talking about not having a constraint in the C API design.

Should we then remove other impl-block's in the C API? For example, in PyLong_Export(). They impose essentially same "constraint" on third-party code.

BTW, what we should do with our internal calls to PyFloat_Pack* and PyFloat_Unpack8 after the proposed change? Handle errors? If not, this API should be documented as infallible somewhere. Where?

[encukou]

PR: python/cpython#153440

Well, if code written with the CPython C API ended up being used with other Python implementation

You have ignored the relevant part of Thomas's sentence.
This is about future versions of CPython, not (that much) about alternate implementations.

Should we then remove other impl-block's in the C API? For example, in PyLong_Export(). They impose essentially same "constraint" on third-party code.

Once they've been in a release, removing them would be a backwards-incompatible change. We shoudn't do that, in general.

The note on PyLong_Export() constrains CPython's int internals -- if we'd want to change the internal layout so that PyLongLayout no longer fits, we'd need to deprecate and remove PyLong_Export() first. That's a reasonable compromise.
Keep in mind one use of PyLong_Export a supportable replacement for private API. We want people to be able to switch, even if they used the old internals in code without exception paths.

BTW, what we should do with our internal calls to PyFloat_Pack* and PyFloat_Unpack8 after the proposed change? Handle errors? If not, this API should be documented as infallible somewhere. Where?

Internal code is free to rely on version-specific internals, so using use more portable API is a judgement call. (My call would be to only skip error checking in the code that touches PyFloatObject internals direcly -- which is of course code that doesn't need PyFloat_Pack.)

[vstinner]

The Steering Council made a decision, and there is now a PR implementing this decision. So this issue can now be closed.