Copying interned object via `copy`

[Red4Ru]

Bug report

Bug description:

Just found a weird behavior with objects interned via caching __new__ result: __new__ is called with no additional arguments during copying which leads to unexpected behavior (all copies become the same object)

import copy
from functools import cache

class Dummy:
    def __init__(self, field):
        self.field = field
    @cache
    def __new__(cls, *args, **kwargs):
        print(f'__new__ is called with {args=} {kwargs=}')
        return super(Dummy, cls).__new__(cls)

a, b, c = Dummy('a'), Dummy('b'), Dummy('c')
# __new__ is called with args=('a',) kwargs={}
# __new__ is called with args=('b',) kwargs={}
# __new__ is called with args=('c',) kwargs={}
print(vars(a), vars(b), vars(c))
# {'field': 'a'} {'field': 'b'} {'field': 'c'}
dc_a = copy.deepcopy(a)
# __new__ is called with args=() kwargs={}
print(vars(dc_a))
# {'field': 'a'}
dc_b = copy.deepcopy(b)
print(vars(dc_a), vars(dc_b))
# {'field': 'b'} {'field': 'b'}
dc_a is dc_b
# True
dc_c = copy.deepcopy(c)
print(vars(dc_a), vars(dc_b), vars(dc_c))
# {'field': 'c'} {'field': 'c'} {'field': 'c'}
dc_c is dc_b
# True

# with copy.copy works too
c_a = copy.copy(a)
c_b = copy.copy(b)
c_c = copy.copy(c)
print(vars(c_a), vars(c_b), vars(c_c))
# {'field': 'c'} {'field': 'c'} {'field': 'c'}
c_a is dc_a
# True
c_c is c
# False

It works with frozen sloted dataclasses as well

@dataclass(slots=True, frozen=True)
class Dummy:
    str_field: str
    int_field: int
    @cache
    def __new__(cls, *args, **kwargs):
        print(f'__new__ called with {args=} {kwargs=}')
        return super(Dummy, cls).__new__(cls)
Dummy('a', 1) is Dummy('a', 1)
# __new__ called with args=('a', 1) kwargs={}
# True
Dummy('a', 1) is not Dummy('b', 1)
# __new__ called with args=('b', 1) kwargs={}
# True
Dummy('a', 1) is copy.deepcopy(Dummy('a', 1))
# __new__ called with args=() kwargs={}
# False
copy.deepcopy(Dummy('a', 1)) is copy.deepcopy(Dummy('b', 1))
# True

Yes, there is a workaround requiring to define something like this in the class

def __deepcopy__(self, memo=None):
    return self

But it still looks like a bug.
If it is somehow an intended behavior it would be useful to point that out here: https://docs.python.org/3/library/copy.html

CPython versions tested on:

3.14

Operating systems tested on:

macOS

[abdoulrasheed]

the issue originates in _reconstruct() in Lib/copy.py. When __reduce_ex__(4) returns (__newobj__, (cls,), state) without arguments for __new__, _reconstruct ends up calling cls.__new__(cls). Because of @cache, this returns the same interned instance each time. The state is then applied to that shared instance, which corrupts all references to it.

A fix is to detect when func is copyreg.__newobj__ or __newobj_ex__ and the returned object already has instance state (non-empty __dict__ or populated slots). In that case, a new instance should be created using object.__new__(type(y)) instead of reusing the cached one.

I'd like to submit a pr.

[Red4Ru]

So anyway interning won't apply during object copying? Without special measures like defining __reduce__ or __copy__+_deepcopy__ or using copyreg.pickle etc.
It might be a little confusing considering that a singleton object can still be "copied" without any problems (being independent from __new__ args besides cls) despite using pretty same mechanism. So I wonder maybe there is something else we can do with this 🤔

[picnixz]

cc @serhiy-storchaka

[serhiy-storchaka]

This is how it works. If you have custom __new__, you should also define __getnewargs__ or __reduce__, or __copy__ and __deepcopy__. Python does not know what arguments to pass to __new__ if you don't say this to it.

[Red4Ru]

This is how it works. If you have custom __new__, you should also define __getnewargs__ or __reduce__, or __copy__ and __deepcopy__. Python does not know what arguments to pass to __new__ if you don't say this to it.

Oh, then it's fine, thank you for the answer!

Maybe it is worth it to add a note mentioning this feature to copy docs? Considering that both __new__ and __copy__ methods are quite popular in contrast to rarely mentioned __getnewargs__ method and now it is easy to make the same mistake as I did.
(I admit that this may be unnecessary and is solved by "just read the docs", so this is just a suggestion).

[picnixz]

There is a mention about copy/deepcopy but nothing about getnewargs/reduce so I don't mind having a mention to getnewargs but I don't know if we want to advertise this or rather __copy__/__deepcopy__.

@serhiy-storchaka What do you suggest? do you want to advertise __getnewargs__ in copy docs or do you think it's better to redefine the other dunders?