[ty] Infer definite equality comparison results#26337
Conversation
Typing conformance resultsNo changes detected ✅Current numbersThe percentage of diagnostics emitted that were expected errors held steady at 94.47%. The percentage of expected errors that received a diagnostic held steady at 89.19%. The number of fully passing files held steady at 95/134. |
Memory usage reportSummary
Significant changesClick to expand detailed breakdownprefect
sphinx
trio
flake8
|
|
| Lint rule | Added | Removed | Changed |
|---|---|---|---|
invalid-argument-type |
0 | 13 | 0 |
invalid-key |
0 | 7 | 0 |
unresolved-attribute |
0 | 5 | 0 |
invalid-assignment |
0 | 2 | 0 |
possibly-unresolved-reference |
0 | 1 | 0 |
unsupported-operator |
0 | 1 | 0 |
| Total | 0 | 29 | 0 |
Flaky changes detected. This PR summary excludes flaky changes; see the HTML report for details.
Raw diff (29 changes)
cloud-init (https://github.com/canonical/cloud-init)
- tests/unittests/distros/test_opensuse.py:287:16 error[unresolved-attribute] Object of type `Distro` has no attribute `read_only_root`
- tests/unittests/distros/test_opensuse.py:308:16 error[unresolved-attribute] Object of type `Distro` has no attribute `read_only_root`
- tests/unittests/distros/test_opensuse.py:329:16 error[unresolved-attribute] Object of type `Distro` has no attribute `read_only_root`
freqtrade (https://github.com/freqtrade/freqtrade)
- freqtrade/rpc/telegram.py:579:59 error[invalid-key] Unknown key "status" for TypedDict `RPCAnalyzedDFMsg` (subscripted object has type `RPCStatusMsg | RPCStrategyMsg | RPCProtectionMsg | ... omitted 5 union elements`)
- freqtrade/rpc/telegram.py:579:59 error[invalid-key] Unknown key "status" for TypedDict `RPCEntryMsg` (subscripted object has type `RPCStatusMsg | RPCStrategyMsg | RPCProtectionMsg | ... omitted 5 union elements`)
- freqtrade/rpc/telegram.py:579:59 error[invalid-key] Unknown key "status" for TypedDict `RPCExitMsg` (subscripted object has type `RPCStatusMsg | RPCStrategyMsg | RPCProtectionMsg | ... omitted 5 union elements`)
- freqtrade/rpc/telegram.py:579:59 error[invalid-key] Unknown key "status" for TypedDict `RPCNewCandleMsg` (subscripted object has type `RPCStatusMsg | RPCStrategyMsg | RPCProtectionMsg | ... omitted 5 union elements`)
- freqtrade/rpc/telegram.py:579:59 error[invalid-key] Unknown key "status" for TypedDict `RPCProtectionMsg` (subscripted object has type `RPCStatusMsg | RPCStrategyMsg | RPCProtectionMsg | ... omitted 5 union elements`)
- freqtrade/rpc/telegram.py:579:59 error[invalid-key] Unknown key "status" for TypedDict `RPCStrategyMsg` (subscripted object has type `RPCStatusMsg | RPCStrategyMsg | RPCProtectionMsg | ... omitted 5 union elements`)
- freqtrade/rpc/telegram.py:579:59 error[invalid-key] Unknown key "status" for TypedDict `RPCWhitelistMsg` (subscripted object has type `RPCStatusMsg | RPCStrategyMsg | RPCProtectionMsg | ... omitted 5 union elements`)
psycopg (https://github.com/psycopg/psycopg)
- tests/test_cursor_common.py:159:24 error[unresolved-attribute] Module `psycopg` has no member `ProgrammingError`
- tests/test_cursor_common_async.py:157:24 error[unresolved-attribute] Module `psycopg` has no member `ProgrammingError`
pycryptodome (https://github.com/Legrandin/pycryptodome)
- lib/Crypto/IO/PKCS8.py:210:33 error[invalid-argument-type] Argument to bound method `DerSequence.decode` is incorrect: Expected `bytes`, found `None`
- lib/Crypto/IO/PKCS8.py:210:45 error[invalid-argument-type] Argument to bound method `DerSequence.decode` is incorrect: Expected `int | None`, found `tuple[Literal[1], Literal[2]]`
- lib/Crypto/IO/PKCS8.py:211:37 error[invalid-argument-type] Argument to bound method `DerObjectId.decode` is incorrect: Expected `bytes`, found `None`
- lib/Crypto/IO/PKCS8.py:216:30 error[invalid-argument-type] Argument to bound method `DerObject.decode` is incorrect: Expected `bytes`, found `None`
- lib/Crypto/IO/PKCS8.py:222:43 error[invalid-argument-type] Argument to bound method `DerObject.decode` is incorrect: Expected `bytes`, found `None`
- lib/Crypto/PublicKey/ECC.py:833:44 error[invalid-argument-type] Argument to bound method `DerObject.decode` is incorrect: Expected `bytes`, found `None`
- lib/Crypto/PublicKey/ECC.py:840:57 error[invalid-argument-type] Argument to bound method `DerObjectId.decode` is incorrect: Expected `bytes`, found `None`
- lib/Crypto/PublicKey/ECC.py:865:62 error[invalid-argument-type] Argument to bound method `DerBitString.decode` is incorrect: Expected `bytes`, found `None`
- lib/Crypto/PublicKey/RSA.py:700:22 error[invalid-argument-type] Method `__getitem__` of type `bound method DerSequence.__getitem__(n: int) -> None` cannot be called with key of type `slice[Literal[1], Literal[6], None]` on object of type `DerSequence`
- lib/Crypto/PublicKey/RSA.py:700:22 error[unsupported-operator] Operator `+` is not supported between objects of type `None` and `list[IntegerBase]`
- lib/Crypto/PublicKey/RSA.py:700:42 error[invalid-argument-type] Argument to `IntegerBase.__init__` is incorrect: Expected `IntegerBase | int`, found `None`
- lib/Crypto/PublicKey/RSA.py:700:58 error[invalid-argument-type] Argument to bound method `IntegerBase.inverse` is incorrect: Expected `IntegerBase | int`, found `None`
trio (https://github.com/python-trio/trio)
- src/trio/_core/_tests/test_run.py:2440:24 error[invalid-assignment] Object of type `Value[object] | Error` is not assignable to `Outcome[str] | None`
- src/trio/_core/_tests/test_run.py:2457:34 error[invalid-argument-type] Argument to bound method `CoroutineType.send` is incorrect: Expected `Outcome[object]`, found `None`
- src/trio/_core/_tests/test_run.py:2470:24 error[invalid-argument-type] Argument to bound method `CoroutineType.send` is incorrect: Expected `Outcome[object]`, found `None`
werkzeug (https://github.com/pallets/werkzeug)
- tests/test_test.py:150:9 error[invalid-assignment] Invalid subscript assignment with key of type `Literal["test_int"]` and value of type `Literal[1]` on object of type `MultiDict[str, str]`
xarray (https://github.com/pydata/xarray)
- xarray/tests/test_cftime_offsets.py:326:22 warning[possibly-unresolved-reference] Name `expected` used when possibly not defined|
@AlexWaygood -- this currently contains a bunch of false negatives because we don't seem to consider mutable fields. An example would be like: class Response: ...
class HtmlResponse(Response): ...
class TestResponse:
response_class = Response
def check(self) -> None:
if self.response_class == Response:
print("plain response")
else:
print("specialized response")
class TestHtmlResponse(TestResponse):
response_class = HtmlResponseOn this PR, we consider the |
Merging this PR will not alter performance
Comparing Footnotes
|
|
Codex is trying to fix that by promoting some top-level class literals, e.g.: + /// Promote top-level class literals to the class objects represented by `type[...]`.
+ ///
+ /// This is intentionally separate from regular promotion. Applying it during collection
+ /// inference would lose useful precision for local and module-level collections of class
+ /// objects.
+ pub(crate) fn promote_class_literals(self, db: &'db dyn Db) -> Type<'db> {
+ match self {
+ Type::ClassLiteral(class) => {
+ SubclassOfType::from(db, class.default_specialization(db))
+ }
+ Type::Union(union) => union.map_leave_aliases(db, |element| {
+ element.promote_class_literals(db)
+ }),
+ _ => self,
+ }
+ } |
I think we'll have to stop inferring We've discussed before about whether we need to "promote" class-literal types to their supertypes in more contexts when they're unannotated. I think this is another piece of evidence that we do need to do that. |
|
Okay, I will try that in a separate PR first. |
|
yeah, I think codex is on the right track there, but we'd definitely want to do that as a standalone change first rather than here |
a708685 to
e8d9d71
Compare
dd25afa to
c2b82e8
Compare
e8d9d71 to
e923e73
Compare
3947599 to
1bbd9a8
Compare
e923e73 to
faa199e
Compare
|
I think the ecosystem results here are now generally acceptable. There's one diagnostic we're now discarding due to incorrect |
1bbd9a8 to
0d56725
Compare
28c3746 to
82e9e58
Compare
82e9e58 to
9fb38ce
Compare
AlexWaygood
left a comment
There was a problem hiding this comment.
I think there's even more we could do to unify the two modules in the future, but this is a great improvement on its own for now, thanks!
| /// Selects how recursive comparison results are combined. | ||
| /// | ||
| /// The goal is only an optimization; both modes use the same comparison semantics and agree on | ||
| /// which results are definite. [`Constraint`](Self::Constraint) preserves branch-specific narrowing | ||
| /// for the left operand. [`Truthiness`](Self::Truthiness) can discard those constraints because its | ||
| /// caller only needs to know whether every expanded alternative agrees, and can stop as soon as the | ||
| /// comparison cannot be definite. | ||
| /// | ||
| /// For example, truthiness evaluation proves that this comparison is always false by checking the | ||
| /// finite alternatives on both sides, without constructing a narrowing constraint: | ||
| /// | ||
| /// ```python | ||
| /// from enum import Enum | ||
| /// from typing import Literal | ||
| /// | ||
| /// class Choice(Enum): | ||
| /// A = 1 | ||
| /// B = 2 | ||
| /// C = 3 | ||
| /// D = 4 | ||
| /// | ||
| /// def compare(left: Literal[Choice.A, Choice.B], right: Literal[Choice.C, Choice.D]): | ||
| /// reveal_type(left == right) # Literal[False] | ||
| /// ``` |
There was a problem hiding this comment.
this is clever. Thanks for documenting it so clearly!
## Summary Since #25955 and #26337, equality narrowing, match reachability, membership compatibility, and ordinary `==` and `!=` result inference all use the same comparison evaluator, within which comparing large enums ends up performing a large member cross-product operation. This PR adds a fast path for enums, whereby same-class operands use compact member sets and cross-class / multi-class unions project members onto cached runtime comparison keys. The cache is per enum class and comparison operator, so repeated comparisons reuse the class-wide scan instead of repeating it for every type pair or expression. Because it's in the shared evaluator, this optimization applies consistently to equality and inequality narrowing, definite comparison results and resulting reachability, value-pattern narrowing, and membership compatibility. (Note: `SameEnumComparison` is a fast path within the fast path for enums of the same class. Within the same class, we can just compare member-name sets; for different classes, we need project down to a value.) In quick local benchmarks against current `main`, the original astral-sh/ty#3830 reproducer drops from roughly 14.4 s to 9.3 ms, a comparison between two 256-member `StrEnum`s drops from roughly 242 ms to 10.8 ms, and a comparison between unions spanning 16 enum classes drops from roughly 296 ms to 10.9 ms. Closes astral-sh/ty#3830.
Summary
The equality rewrite in #25788 centralized runtime
==and!=reasoning for narrowing and match reachability, but ordinary comparison inference still uses separate literal and dunder logic. As a result, two disjoint enum domains can narrow correctly while the comparison expression itself remainsbool, keeping an impossible branch reachable.This routes definite equality and inequality truthiness through the shared comparison evaluator before falling back to existing inference. Ambiguous comparisons still use normal dunder inference, preserving custom non-boolean return types and the independence of
__eq__and__ne__.Because result inference only needs definite truthiness, the evaluator uses a truthiness-only aggregation mode for this path. Union and intersection evaluation stops as soon as an arm is ambiguous, only provides a narrowing constraint, or disagrees with a previous definite result; the full aggregation path remains unchanged for narrowing callers that need to construct constraints.
This PR is stacked on #26338, which widens inferred class-valued attributes accessed through an instance. The regression coverage here verifies the interaction: a subclass-overridable
self.response_class == Responseremainsbool, aFinalattribute can still produceLiteral[True], and module-level collections of class objects retain enough precision for exhaustive equality checks.