[HyukjinKwon]

What

A follow-up of #3 and #4. I compared the Rust implementation line by line with upstream's cloudpickle/cloudpickle.py again (4af5936, still the head of upstream's master) and reviewed the commits since 0.2.0. This found one regression and one undocumented difference with upstream, plus an upstream bug and a few smaller differences, all fixed here.

import cloudpickle_rust fails with python -OO (regression in 0.3.0)

$ python -OO -c "import cloudpickle_rust"
Traceback (most recent call last):
  ...
TypeError: 'None' is not an instance of 'str'

Since #3, the docstrings of the Pickler methods implemented in Rust (reducer_override) are compiled when the module is imported (functions::compiled_doc), so that they match upstream's on Python 3.13+. With -OO (or PYTHONOPTIMIZE=2), the compiler removes them, and compiled_doc failed to convert the resulting None. 0.2.0 imports fine.

compiled_doc now returns None in that case, and the method descriptor then gets no docstring. As with upstream under -OO, the functions and methods have no docstring. Neither do the module (whose docstring otherwise also states the license of cloudpickle-rust) and _DispatchTable.

Subinterpreters

Upstream cloudpickle can be imported in subinterpreters, for instance the ones of concurrent.interpreters (new in Python 3.14). There, cloudpickle_rust raises ImportError: module cloudpickle_rust.cloudpickle does not support loading in subinterpreters, like every PyO3 extension module (PyO3/pyo3#576; PyO3 0.28 has no opt-in). The README now lists this among the differences with upstream, and a test checks it on Python 3.14 and later.

Pickling a class while other threads load it (free-threaded builds; also an upstream bug)

Loading a payload of a tracked class sets the attributes of the class again (_class_setstate). On Python 3.13+, setting __module__ also deletes __firstlineno__ until it is set again. _extract_class_dict read the __dict__ of the class several times, like upstream's. So pickling the class meanwhile in another thread can raise KeyError: '__firstlineno__' or RuntimeError: dictionary changed size during iteration on free-threaded builds, and upstream cloudpickle does too. It now reads a snapshot (mappingproxy.copy(), atomic also on free-threaded builds), with the same result when the class does not change. The new test fails in 10 runs out of 10 with upstream on 3.14t, and passes here.

Smaller differences

• On Python 3.13+, Pickler gets the __firstlineno__ and __static_attributes__ that the compiler sets for upstream's class statement. _empty_cell_value is defined by upstream's code itself, taken from the bundled cloudpickle.py and run at the same line, so its class also has upstream's __firstlineno__.
• The type stub of _extract_code_globals_cache is the weakref.WeakKeyDictionary that it has been since 0.3.0 (mypy.stubtest now only reports upstream's leaked loop variables k and v).
• The README says that the pickles of tests/old_pickles were generated with cloudpickle-rust 0.1.0.
• test_payloads_identical_to_upstream covers all the pickle protocols. Protocols 0 and 1 cannot pickle some of the objects (the InitVar of dataclasses), and the exception must then be upstream's. It also covers more kinds of objects: enums with __init__, IntFlag, StrEnum, NamedTuple, TypedDict, Protocol, metaclasses, namedtuple, dataclasses with slots and KW_ONLY, PEP 695 generics, positional-only parameters, generators and coroutines.

Verification

• New tests in tests/cloudpickle_rust_test.py:
  • test_docstrings_with_optimizations: run in a subprocess with no flag, with -O and with -OO, cloudpickle_rust imports and pickles. The docstrings of the module, its functions and the Pickler methods are upstream's (all None with -OO).
  • test_subinterpreters_are_not_supported (Python 3.14 and later).
  • test_pickling_a_class_while_other_threads_load_it, and the extended test_payloads_identical_to_upstream, test_module_namespace_matches_upstream and test_empty_cell_value_is_an_instance_of_a_python_class.
• The full suite, with the new tests, passes on macOS arm64 with CPython 3.8 to 3.14, 3.13t and 3.14t (3.13: 328 passed, 12 skipped, 2 xfailed) and with PyPy 7.3.23 (268 passed, 72 skipped, 2 xfailed). With python -OO, cloudpickle_rust imports and round-trips a lambda on all of them.
• With -OO on CPython 3.13, upstream's suite run against upstream cloudpickle and this suite give the same outcome for every test. 5 tests fail in both, because they check docstrings.
• Parity checks against upstream on the same interpreter that go beyond the suite (CPython 3.8 to 3.14, 3.13t and 3.14t; ad-hoc scripts, not committed):
  • For up to 113 kinds of objects (depending on the Python version) and protocols 0 to 5, the normalized opcodes are identical, and so is whether the payload loads. The objects include closures, recursive and nested functions, classes with slots, ABCs, metaclasses or Generic, PEP 695 generics and type aliases, dataclasses (including slots and KW_ONLY), Enum, Flag, StrEnum and enums with __init__, modules, files, loggers, dict views, cells and code objects.
  • 70 behaviors are identical. They cover error types, messages, __cause__ and __suppress_context__ for unpicklable objects and invalid arguments; Pickler subclasses that override reducer_override, _function_reduce, _function_getnewargs, persistent_id or dispatch_table; the registry; and the private helpers.
  • The per-test outcomes of upstream's suite against upstream cloudpickle and of this suite are identical on 3.8, 3.13 and 3.14.
  • Memory does not grow over 30,000 round trips of dynamic classes, enums, dataclasses and functions.
• cargo fmt --check, cargo clippy --all-targets -- -D warnings, black 25.9.0 and ruff 0.14.3 pass.

This pull request and its description were written by Isaac.