RFC: Add iter! macro

[eholk]

Tracking issue: rust-lang/rust#142269

Summary

Add an iter! macro to provide a better way to create iterators.

Implementing the Iterator trait directly can be tedious. Generators (see RFC 3513) are available on nightly but there are enough open design questions that generators are unlikely to be stabilized in the foreseeable future.

On the other hand, we have an iter! macro available on nightly that provides a subset of the full generator functionality. Stabilizing this version now would have several benefits:

• Users immediately gain a more ergonomic way to write many iterators.
• We can better inform the design of generators by getting more real-world usage experience.
• Saves the gen { ... } and gen || { ... } syntax for a more complete feature in the future.

Rendered

[kennytm]

so std::iter::from_coroutine should be deprecated?

[BurntSushi]

This looks amazing! This does look like a nice incremental step forward.

Has any thought been given to how this might be used in library APIs? One could of course return an impl IntoIterator, but this is usually sub-optimal for various reasons. I guess this would be blocked on being able to name the type returned by iter!(...). This sort of thing is viral, so it means it would be difficult to use iter!(...) to implement any publicly exported iterators unless you're okay with returning impl Trait. Relatedly, there is the question of how to make iterators created by iter!(...) implement the various other iterator traits (e.g., ExactSizeIterator or FusedIterator) where applicable.

[bluebear94]

so std::iter::from_coroutine should be deprecated?

from_coroutine would be useful if you wanted to convert an existing Coroutine from elsewhere to an iterator, so I don’t think so.

[sanbox-irl]

Can you go into more detail on why this couldn't be a library?

[Lokathor]

My biggest concern is that this is supposed to be about a macro for iterator creation, but it can use yield too, which isn't available in Stable, so it's really making the subset of generators that fit the iterator pattern.

It's mildly confusing, and not what I expected from the RFC title, or what I'd usually expect from finding iter! in a project. My concern is purely a naming/organization concern though. If it was called generator! instead, then it would "make sense" that this macro allows use of the yield keyword.

EDIT: also, personally I've never had a problem making custom iterators once I learned about using core::iter::from_fn with a move closure. Maybe we could give people better messaging about that if it's such a concern.

[sanbox-irl]

EDIT: also, personally I've never had a problem making custom iterators once I learned about using core::iter::from_fn with a move closure. Maybe we could give people better messaging about that if it's such a concern.

To me, this is very motivating -- core::iter::from_fn is strictly more powerful than this macro, as it can return borrowed data.

[eholk]

Has any thought been given to how this might be used in library APIs? One could of course return an impl IntoIterator, but this is usually sub-optimal for various reasons. I guess this would be blocked on being able to name the type returned by iter!(...). This sort of thing is viral, so it means it would be difficult to use iter!(...) to implement any publicly exported iterators unless you're okay with returning impl Trait. Relatedly, there is the question of how to make iterators created by iter!(...) implement the various other iterator traits (e.g., ExactSizeIterator or FusedIterator) where applicable.

I could think of a few future things that could help. For example, we could add something like #[iter] fn foo() -> i32 { yield 42; }, and adding the IterFn* trait family could help too.

That doesn't help with the fundamental problem of wanting to name a concrete type though, or implement other traits.

A while back I was musing some about anonymous impls. I could see riffing on that with something like:

iter!(|| { yield 1; yield 2; yield 3; } with fn size_hint(&self) { (3, Some(3)) });

or maybe

iter!(|| { yield 1; yield 2; yield 3; } with impl DoubleEndedIterator {
    fn next_back(&self) -> Option<Self::Item> { ... }
});

So I think there are possibilities, but it's definitely future work. Even being able to write one-off iterators inline seems like an improvement though!

[clarfonthey]

I definitely think that the inability to pin these iterators and use them for references is more limiting than helpful. Perhaps it's worth investigating whether something like #3851 could be done to allow a supertrait of Iterator which takes Pin<&mut self> instead of &mut self.

[eholk]

That doesn't help with the fundamental problem of wanting to name a concrete type though, or implement other traits.

You might be able to work around the concrete type issue with TAIT.

type MapIter = impl Iterator;

struct Map(MapIter);

fn map<I: Iterator, T, F: Fn(I::Item) -> T>(iter: I, f: F) -> Map {
    Map(iter!(|| {
        for i in iter {
            yield f(i);
        }
    })())
}

I don't think this example will work exactly, since Map and MapIter will need some type parameters, but maybe some more exploration here will yield something?

[eholk]

My biggest concern is that this is supposed to be about a macro for iterator creation, but it can use yield too, which isn't available in Stable, so it's really making the subset of generators that fit the iterator pattern.

This RFC also includes stabilizing yield expressions, so they would be available on stable after stabilizing the iter! macro.

It's mildly confusing, and not what I expected from the RFC title, or what I'd usually expect from finding iter! in a project. My concern is purely a naming/organization concern though. If it was called generator! instead, then it would "make sense" that this macro allows use of the yield keyword.

In my mind I've kind of been saving the generator name for the more powerful version that allows self-borrows and borrowing across yield. The iter! name corresponds better to something that returns an object that implements Iterator. I'd expect generator! to return an impl Generator or something like that.

EDIT: also, personally I've never had a problem making custom iterators once I learned about using core::iter::from_fn with a move closure. Maybe we could give people better messaging about that if it's such a concern.

I think there generators or iter! shine are where you have different phases or states your iterator needs to move through, because you can encode this state in the control flow instead of having to be explicit about it. For example, with from_fn to concatenate two iterators you have to do something like:

fn concat(mut a: impl Iterator<Item = i32>, mut b: impl Iterator<Item = i32>) -> impl Iterator<Item = i32> {
    let mut first_done = false;
    core::iter::from_fn(move || {
        if !first_done {
            match a.next() {
                Some(i) => return Some(i),
                None => {
                    first_done = true;
                    return b.next();
                }
            }
        }
        b.next()
    })
}

On the other hand, with iter! it's:

fn concat(a: impl Iterator<Item = i32>, b: impl Iterator<Item = i32>) -> impl Iterator<Item = i32> {
    iter!(move || {
        for i in a {
            yield i;
        }
        for i in b {
            yield i;
       }
    })()
}

I think there's room for both approaches. If from_fn works, then great! But for some patterns, letting the compiler generate the state machine is going to be much nicer.

[Lokathor]

Well if this is a macro that expanded to a new "iterator closure" type category (which I've never really heard of as being its own "thing" before, and which the RFC does not particularly define despite having a section header about it), then why not call the macro iter_closure!.

I think there generators or iter! shine are where you have different phases or states your iterator needs to move through, because you can encode this state in the control flow instead of having to be explicit about it.

Yes, using yield makes for better code. I don't dispute that at all. However, using from_fn with a fn that uses yield would give equally clear code I think. So the big "win" is the yield keyword, I would say.

Why can't the yield keyword be stabilized on its own, without this macro. That doesn't seem to be discussed in Rationale and alternatives, though the RFC is so long perhaps I missed it.

[Jules-Bertholet]

As you note, Rust generally uses postfix operators only in situations where chaining is natural. You mention the potential case of full coroutines which might benefit from that. But iter! as specified in this RFC is narrowly tailored toward iterators, and doesn’t aim to have anything to do with full coroutines.

So, AFAICT, there is no reason to support postfix yield here; all it does is break the “there should be one—and preferably only one—obvious way to do it” rule, and introduce confusion. We can always reevaluate when we do get full coroutines.

[Jules-Bertholet]

One alternative would be to avoid committing to any syntax, and instead offer some sort of yield!() macro (name would likely have to be different).

[Jules-Bertholet]

I don’t think this should be supported at all. An iterator that yields () is unusual enough that it should be indicated explicitly. And we can always add it later.

[Jules-Bertholet]

Relatedly, there is the question of how to make iterators created by iter!(...) implement the various other iterator traits (e.g., ExactSizeIterator or FusedIterator) where applicable.

Perhaps FusedIterator should be implemented for you? After all, there is no way to use this feature to implement an iterator that isn’t fused. And, due to the type being unnameable except via impl Trait, there is no semver hazard…

[ds84182]

There's a rather annoying issue with coroutines where taking a shared reference to a !Sync value owned by the coroutine across a yield point results in a non-Send coroutine. I worry that increasing coroutine surface area to include iterators will make this issue more prevalent. Are there any plans to address this?

[BurntSushi]

@Jules-Bertholet That's just one example. It doesn't really address the grander point. There are other traits you might want to implement too for an iterator exported in a library API. Notably Debug and Clone. And ExactSizeIterator.

[Jules-Bertholet]

@Jules-Bertholet That's just one example. It doesn't really address the grander point.

And it did not intend to.

[eholk]

Why can't the yield keyword be stabilized on its own, without this macro. That doesn't seem to be discussed in Rationale and alternatives, though the RFC is so long perhaps I missed it.

One of the things the macro does is it creates a context where yield is allowed, similar to how await operators are only allowed in async {} contexts. That's somewhat a design decision for Rust, as I think other languages let you yield anywhere. Early versions of generators in nightly Rust let you do this; a closure was a generator if it contained the yield keyword. More recent versions need a #[coroutine] attribute to make a closure into a coroutine. Besides being clearer (at least in my opinion), it also lets you do things like have a coroutine (or iterator, in the case of this RFC) that yield no items.

[eholk]

There's a rather annoying issue with coroutines where taking a shared reference to a !Sync value owned by the coroutine across a yield point results in a non-Send coroutine. I worry that increasing coroutine surface area to include iterators will make this issue more prevalent. Are there any plans to address this?

This is one of the reasons iter! makes a closure instead of an iterator directly, since the closure can still be Send even if the iterator borrows a !Sync value across a yield point. It doesn't totally alleviate the problem, but it does give some ways around it.