[mypyc] feat: specialize `any` and `all` using for loop helpers if possible [1/2] #19948

BobTheBuidler · 2025-09-28T21:56:38Z

This PR modifies the any and all call specializers to use our for loop helpers in cases where they can be used.

This is only marginally helpful now but will become more impactful with the addition of ForMap and ForFilter helpers, as well as any future helpers.

While the private helpers might seem unnecessary at this point, I intend to use them for future specialization use cases such as list(iterable) and tuple(iterable) as discussed in #19649 , and sum(iterable) which will come shortly after this PR.

This PR is ready for review

Update: The sum(iterable) is now ready in #19950. It's a bit more involved than this one so I'm going to leave it separate.

Update: The helpers are no longer private

mypyc/test-data/irbuild-basic.test

BobTheBuidler · 2025-09-28T22:14:20Z

mypyc/test-data/irbuild-basic.test

+    r12 = list_get_item_unsafe r2, r9
+    r13 = cast(str, r12)
+    __mypyc_all_item__ = r13
+    r14 = CPyStr_IsTrue(__mypyc_all_item__)


The boolean check for the all call is now specialized per the dtype of the iterable, and we no longer have to use python's iterator protocol for the input

On master this seemed to just call the stdlib all using a vector call? So the change is that we now generate a for loop for this, right?

BobTheBuidler · 2025-09-29T06:53:51Z

mypyc/irbuild/specialize.py

+                builder.goto(loop_exit)
+                builder.activate_block(true_block)
+
+            index_type = builder._analyze_iterable_item_type(arg)


should I deduplicate this block or is this fine?

It might be useful to deduplicate, since the only difference seems to be true/false switch, so a helper would just need a bool flag that indicates whether it's any or all. Not a big deal though.

lets just keep it as-is, the intent of the code is explicit and clear

BobTheBuidler · 2025-09-29T16:15:35Z

mypyc/irbuild/for_helpers.py

+
+def create_synthetic_nameexpr(index_name: str, index_type: Type) -> NameExpr:
+    """This helper spoofs a NameExpr to use as the lvalue in one of the for loop helpers."""
+    index = NameExpr(index_name)


I probably need something here to ensure no name collisions globally, but not sure what

In my other comment I suggested adding _<line>_column suffix. Another thing we use is to use the temp_counter defined in IRBuilder to get an increasing suffix.

should create_synthetic_nameexpr be an IRBuilder method instead of going here then?

I think it could be used to refactor a few other places within the codebase where NameExpr are artificially created

I've implemented the temp_counter but left this function where it is for now

mypyc/irbuild/specialize.py

JukkaL · 2025-10-13T16:09:33Z

mypyc/irbuild/for_helpers.py

+
+def create_synthetic_nameexpr(index_name: str, index_type: Type) -> NameExpr:
+    """This helper spoofs a NameExpr to use as the lvalue in one of the for loop helpers."""
+    index = NameExpr(index_name)


In my other comment I suggested adding _<line>_column suffix. Another thing we use is to use the temp_counter defined in IRBuilder to get an increasing suffix.

JukkaL · 2025-10-13T16:12:20Z

mypyc/irbuild/specialize.py

+        arg = expr.args[0]
+        if isinstance(arg, GeneratorExpr):
+            return any_all_helper(builder, arg, builder.false, lambda x: x, builder.true)
+        elif expr_has_specialized_for_helper(builder, arg):


It took me a while to understand why this check is useful. The idea is that if we'd have a generic for loop (PyObject_GetIter etc.) we can as well call the stdlib function?

What would happen if we'd always created the for loop here, since it would likely avoid the overhead of calling any using generic call op, and the namespace dict lookup? If that sounds feasible, can you run some microbenchmarks to check if it's useful? I think it might be worth it especially if the iterable is small (0 or 1 items), which is quite common in practice.

"It took me a while to understand why this check is useful. The idea is that if we'd have a generic for loop (PyObject_GetIter etc.) we can as well call the stdlib function?"

Uh. No. While I appreciate your assessment, it wasn't exactly so thought-out. My only intent was to fall back to the existing logic, whatever that may be, if there is no supported ForHelper.

I need to think about how to implement this

Wait I want to confirm, you're saying if we just remove the if check, we're good to go without any other changes? I suppose that makes sense since the for_loop_helper does have the generic ForGenerator class but I wonder if we're missing any edge cases where that might cause failure

JukkaL · 2025-10-13T16:16:00Z

mypyc/irbuild/specialize.py

+                builder.goto(loop_exit)
+                builder.activate_block(true_block)
+
+            index_type = builder._analyze_iterable_item_type(arg)


It might be useful to deduplicate, since the only difference seems to be true/false switch, so a helper would just need a bool flag that indicates whether it's any or all. Not a big deal though.

mypyc/test-data/irbuild-basic.test

JukkaL · 2025-10-13T16:20:58Z

mypyc/test-data/irbuild-basic.test

+    r12 = list_get_item_unsafe r2, r9
+    r13 = cast(str, r12)
+    __mypyc_all_item__ = r13
+    r14 = CPyStr_IsTrue(__mypyc_all_item__)


On master this seemed to just call the stdlib all using a vector call? So the change is that we now generate a for loop for this, right?

for more information, see https://pre-commit.ci

BobTheBuidler · 2025-10-14T00:45:10Z

mypyc/irbuild/specialize.py

-            if desc:
-                obj = builder.accept(obj_expr)
-                return builder.primitive_op(desc, [obj], expr.line)
+        if node := type_expr.node:


This was definitely not necessary but it bothered me how ugly that snippet was

BobTheBuidler commented Sep 28, 2025

View reviewed changes

mypyc/test-data/irbuild-basic.test Outdated Show resolved Hide resolved

BobTheBuidler commented Sep 28, 2025

View reviewed changes

This was referenced Sep 29, 2025

[mypyc] feat: ForMap generator helper for builtins.map #19649

Open

[mypyc] feat: specialize sum using for loop helpers if possible [2/2] #19950

Open

BobTheBuidler commented Sep 29, 2025

View reviewed changes

BobTheBuidler changed the title ~~[mypyc] feat: specialize any and all using for loop helpers if possible~~ [mypyc] feat: specialize any and all using for loop helpers if possible [1/2] Oct 1, 2025

BobTheBuidler changed the title ~~[mypyc] feat: specialize any and all using for loop helpers if possible [1/2]~~ [mypyc] feat: specialize any and all using for loop helpers if possible [1/2] Oct 1, 2025

BobTheBuidler force-pushed the any-all-for-helper branch from 40f0ddd to 92818b2 Compare October 7, 2025 12:11

[mypyc] feat: specialize any and all using for loop helpers if possible

41c7c9b

BobTheBuidler force-pushed the any-all-for-helper branch from 667e30b to 41c7c9b Compare October 7, 2025 12:30

Merge branch 'master' into any-all-for-helper

50bd3ce

JukkaL reviewed Oct 13, 2025

View reviewed changes

BobTheBuidler and others added 10 commits October 13, 2025 15:50

Update specialize.py

f636c95

Update specialize.py

1af0a05

unique names for index

df669fe

Update specialize.py

2b43f62

[pre-commit.ci] auto fixes from pre-commit.com hooks

e0dc704

for more information, see https://pre-commit.ci

Update irbuild-basic.test

1028f8c

Update irbuild-basic.test

70fb118

Update irbuild-statements.test

1e22947

Merge branch 'master' into any-all-for-helper

6a2c8a0

Update irbuild-basic.test

a2ab6ba

BobTheBuidler commented Oct 14, 2025

View reviewed changes

Merge branch 'master' into any-all-for-helper

f4c1c77

Uh oh!

[mypyc] feat: specialize any and all using for loop helpers if possible [1/2] #19948

Are you sure you want to change the base?

[mypyc] feat: specialize any and all using for loop helpers if possible [1/2] #19948

Uh oh!

Conversation

BobTheBuidler commented Sep 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

BobTheBuidler Sep 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BobTheBuidler Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[mypyc] feat: specialize `any` and `all` using for loop helpers if possible [1/2] #19948

[mypyc] feat: specialize `any` and `all` using for loop helpers if possible [1/2] #19948

BobTheBuidler commented Sep 28, 2025 •

edited

Loading

BobTheBuidler Sep 28, 2025 •

edited

Loading

BobTheBuidler Oct 13, 2025 •

edited

Loading