stubtest: get better signatures for __init__ of C classes

[tungol]

When an __init__ method has the generic C-class signature, check the underlying class for a better signature.

I was looking at asyncio.futures.Future.__init__ and wanted to take advantage of the better __text_signature__ on asyncio.futures.Future.

The upside is that this clears several allowlist entries and surfaced several other cases where typeshed is currently incorrect, with no false positives.

[tungol]

Now handles when runtime.__objclass__ is object instead of skipping those, which clears a couple more allowlist entries in typeshed but didn't surface any new problems to fix.

Technically the implementation of the lookup makes the assumption that the class name is only one element in the object_path list, and will fail for a nested class like modulename.class1.class2.__init__. Those are tricky to look up, and unlikely for something that looks like a C implementation anyway.

[tungol]

While investigating what to do about __new__, I found this issue: python/typeshed#8632

The meaning of which is that this check should not attempt to get a better signature by working around getting passed object.__init__ or object.__new__. Having __init__ in the stubs when we should have __new__ results in this passing type check but being incorrect at runtime:

from __future__ import annotations
from zoneinfo import ZoneInfo

KEY = "America/Los_Angeles"

class Foo(ZoneInfo):
    def __init__(self) -> None:
        super().__init__(KEY)

    def __new__(cls) -> Foo:
        return super().__new__(cls)

Foo()

Traceback (most recent call last):
  File "temp.py", line 13, in <module>
    Foo()
    ~~~^^
  File "temp.py", line 11, in __new__
    return super().__new__(cls)
           ~~~~~~~~~~~~~~~^^^^^
TypeError: ZoneInfo() missing required argument 'key' (pos 1)

While this is correct at runtime but fails type check:

from __future__ import annotations
from zoneinfo import ZoneInfo

KEY = "America/Los_Angeles"

class Foo(ZoneInfo):
    def __init__(self) -> None:
        super().__init__()  # error: Missing positional argument "key" in call to "__init__" of "ZoneInfo"  [call-arg]

    def __new__(cls) -> Foo:
        return super().__new__(cls, KEY)  # error: Too many arguments for "__new__" of "object"  [call-arg]

Foo()

This is good, actually, because it means that this check is a lot simpler, and I don't have to worry about duplicating __init__ and __new__; I just need to follow the runtime.

[tungol]

I'm working through the errors flagged this way. A couple things that are coming up:

• Some classes have both a __init__ and a __new__ that are custom to that class. Without additional work, this check will prompt both to be created in the stubs - I think that's probably okay.

• Overloads and positionals

a function like def (arg, /) -> None errors if you try to add self to the start and it's POSITIONAL_OR_KEYWORD, so I check for "first argument is positional only") and add self as positional only if that's the case. However, a bunch of these have overload stubs like:

@overload
def __init__(self) -> None: ...
@overload
def __init__(self, arg, /) -> None: ...

And it's causing stubtest to prompt to make the first overload positional-only, and normally typeshed wouldn't do that.

• More complicated overloads

We also have overloads where the runtime definition is def __init__(self, *args) -> None: ...
and the stub makes an overload that breaks this out into a bunch individually typed parameters. itertools.product and builtins.zip are the gnarliest ones. These want a positional-only self or cls as well, since the stubs end up positional-only. Those two are also producing overload stub signatures that are also just hot nonsense.

This is from builtins.zip.__new__:

Inferred signature: def (cls: type[builtins.zip[_T_co`1]] = ..., cls: type[builtins.zip[_T_co`1]] = ..., iter1: Union[typing.Iterable[_T1`-1], typing.Iterable[_T1`-1], typing.Iterable[_T1`-1], typing.Iterable[_T1`-1], typing.Iterable[_T1`-1], typing.Iterable[Any]] = ..., iter2: Union[typing.Iterable[_T2`-2], typing.Iterable[_T2`-2], typing.Iterable[_T2`-2], typing.Iterable[_T2`-2], typing.Iterable[Any]] = ..., iter3: Union[typing.Iterable[_T3`-3], typing.Iterable[_T3`-3], typing.Iterable[_T3`-3], typing.Iterable[Any]] = ..., iter4: Union[typing.Iterable[_T4`-4], typing.Iterable[_T4`-4], typing.Iterable[Any]] = ..., iter5: Union[typing.Iterable[_T5`-5], typing.Iterable[Any]] = ..., iter6: typing.Iterable[Any] = ..., *iterables, strict: builtins.bool = ...)
Runtime:
def (cls, /, *iterables, strict=False)

Inferred signature with multiple copies of cls, cls with default values, etc. It's real bad, and that's not code that I've touched in this MR, it's just that we never actually ran a comparison between it and runtime before.

[tungol]

I worked out the issues. "self" was special cased in two places, and I needed to add special-casing in the same way for "cls" to deal with the sudden glut of complicated overloaded __new__ methods. Once I did that, all the troublesome bits went away.

I pushed up the fixes that this MR surfaced to python/typeshed#13211 but pyright is unhappy with some of them, so I need to work through that still.

[tungol]

I think this is basically good now. See python/typeshed#13211 for the changes that this suggests for the stdlib stubs. Some tests would be good still.

[tungol]

Having looked at how stubtest tests are run, I'm not sure if I can create a test for this? I can't see a way to mimic a C-implemented class as a test case in teststubtest.py.

[JelleZijlstra]

Hm this seems to be causing some issues in typeshed, I got some new errors in python/typeshed#14599 when I pinned mypy to the commit from this PR. I pinned it to an earlier commit for now so that PR can focus on disjoint base changes, but we'll have to fix this.

[JelleZijlstra]

The runs with problems were in https://github.com/python/typeshed/actions/runs/17144606835/job/48638239781, e.g. the complaint about _random.Random.__init__.

[tungol]

error: _random.Random.__init__ is inconsistent, runtime does not have argument "seed"

This shows on 3.9 and 3.10, where _random.Random uses __new__, not init`. There should be a version branch in the stubs there. Changed in python/cpython#26455 . Stubtest doesn't flag this, but on 3.11+ it should be a positional only argument, a keyword isn't accepted.

error: select.epoll.__init__ is inconsistent, runtime does not have argument "sizehint"
error: select.epoll.__init__ is inconsistent, runtime does not have argument "flags"

I don't have a linux machine handy (which is also why I didn't catch this on my earlier typeshed MR), but it looks like select.epoll also uses __new__, not __init__. See https://github.com/python/cpython/blob/main/Modules/selectmodule.c#L2527

[tungol]

error: _blake2.blake2s.__new__ is inconsistent, stub does not have argument "string"
error: _blake2.blake2b.__new__ is inconsistent, stub does not have argument "string"

These appear on 3.13 and 3.14; string was added by python/cpython@c6e63d9#diff-c7440bf2e8c27b17a856c469b7839a7c0bfd93008e3c4f553616df6f1c33d854L678

There's a corresponding error for hashlib.blake2b.__new__ on 3.13 but not 3.14; I'm not sure why it's not happening on 3.14.

[tungol]

on 3.14:

error: _io.BufferedRWPair.__init__ is inconsistent, runtime argument "buffer_size" has a default value of 131072, which is different from stub argument default 8192
error: _io.BufferedRandom.__init__ is inconsistent, runtime argument "buffer_size" has a default value of 131072, which is different from stub argument default 8192
error: io.BufferedReader.__init__ is inconsistent, runtime argument "buffer_size" has a default value of 131072, which is different from stub argument default 8192
error: io.BufferedWriter.__init__ is inconsistent, runtime argument "buffer_size" has a default value of 131072, which is different from stub argument default 8192

_io.DEFAULT_BUFFER_SIZE changed in python/cpython#118144