Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Async Streams #606

Open
wants to merge 9 commits into
base: draft-v8
Choose a base branch
from
Open

Add support for Async Streams #606

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
f4689c6
add support for async streams
RexJaeschke Aug 8, 2022
5fa2054
add support for async streams
RexJaeschke Aug 8, 2022
c889f5c
add support for async streams
RexJaeschke Aug 8, 2022
a1d3d6c
add example annotation
RexJaeschke Jan 22, 2023
250f15d
Create Support1AsyncStreams.cs
RexJaeschke Feb 7, 2023
5b5d590
Create Support2AsyncStreams.cs
RexJaeschke Feb 7, 2023
fe01c0d
Update standard/classes.md
jskeet May 15, 2024
1354a0e
Merge branch 'draft-v8' into async-streams
RexJaeschke Feb 4, 2025
23e4ec5
Add new types to C.5
RexJaeschke Feb 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 63 additions & 12 deletions standard/classes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5326,6 +5326,8 @@ For a discussion of the behavior when an exception is thrown from a finalizer, s

### 15.14.1 General

(Almost all of the text in this subclause and its siblings was written prior to the addition of async streams; the text applies to async streams as well, unless stated otherwise, usually in parenthetic text. Asynchronous stream creation and usage is discussed in §asynchronous-streams.)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the intention to keep this part? It looks a little odd at the moment. I suspect it should at least be a note instead of just in parentheses.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does feel a little odd. As an off the cuff suggestion:

  • swap the order of §15.14 Iterators and §15.15 Async Functions
  • rename "Iterators” to “Iterators and Async Streams”
  • rewrite this new intro paragraph to say the clause covers synchronous and async iterators, the latter usually termed async streams.
  • remove the parentheses and merge the content into the surrounding text

Of course this isn't the only “odd” thing we’ve done, e.g. property & indexer accessors with the “read exchanging terms” instruction.


A function member ([§12.6](expressions.md#126-function-members)) implemented using an iterator block ([§13.3](statements.md#133-blocks)) is called an ***iterator***.

An iterator block may be used as the body of a function member as long as the return type of the corresponding function member is one of the enumerator interfaces ([§15.14.2](classes.md#15142-enumerator-interfaces)) or one of the enumerable interfaces ([§15.14.3](classes.md#15143-enumerable-interfaces)). It may occur as a *method_body*, *operator_body* or *accessor_body*, whereas events, instance constructors, static constructors and finalizer shall not be implemented as iterators.
Expand All @@ -5334,41 +5336,41 @@ When a function member is implemented using an iterator block, it is a compile-t

### 15.14.2 Enumerator interfaces

The ***enumerator interfaces*** are the non-generic interface `System.Collections.IEnumerator` and all instantiations of the generic interface `System.Collections.Generic.IEnumerator<T>`. For the sake of brevity, in this subclause and its siblings these interfaces are referenced as `IEnumerator` and `IEnumerator<T>`, respectively.
The ***enumerator interfaces*** are the non-generic interface `System.Collections.IEnumerator` and all instantiations of the generic interface `System.Collections.Generic.IEnumerator<T>` (`System.Collections.Generic.IAsyncEnumerator<T>` for async streams). For the sake of brevity, in this subclause and its siblings these interfaces are referenced as `IEnumerator` and `IEnumerator<T>` (`IAsyncEnumerator<T>` for async streams), respectively.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm personally finding this style really hard to read. I'd be happy to try to get this PR into a shape we want, merge it, and then I can create a new PR to change the style - assuming others feel the same way.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to introduce the term "stream" here? I would prefer us to be consistent between synchronous and asynchronous terminology unless there's an inherent reason for them to differ.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think changing the name is a non-starter (flashbacks to NNRT here ;-)) and there is some convention in using the term “stream”. If my first suggestion above is followed the stream/async iterator stuff could be covered in §asynchronous-streams which will occur before the §15.15 Iterators and Async Streams clause.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also think we should continue to use the term "async stream". It fits well with the feature, and "stream" is the conventional term.

I also like the idea of some reordering. That may be a follow-up PR?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As an example of what my opening off the cuff suggestion would produce this para would become:

Suggested change
The ***enumerator interfaces*** are the non-generic interface `System.Collections.IEnumerator` and all instantiations of the generic interface `System.Collections.Generic.IEnumerator<T>` (`System.Collections.Generic.IAsyncEnumerator<T>` for async streams). For the sake of brevity, in this subclause and its siblings these interfaces are referenced as `IEnumerator` and `IEnumerator<T>` (`IAsyncEnumerator<T>` for async streams), respectively.
The ***enumerator interfaces*** are the non-generic interface `System.Collections.IEnumerator` and all instantiations of the generic interfaces `System.Collections.Generic.IEnumerator<T>` and `System.Collections.Generic.IAsyncEnumerator<T>`. For the sake of brevity, in this subclause and its siblings these interfaces are referenced as `IEnumerator`, `IEnumerator<T>` and `IAsyncEnumerator<T>` respectively.


### 15.14.3 Enumerable interfaces

The ***enumerable interfaces*** are the non-generic interface `System.Collections.IEnumerable` and all instantiations of the generic interface `System.Collections.Generic.IEnumerable<T>`. For the sake of brevity, in this subclause and its siblings these interfaces are referenced as `IEnumerable` and `IEnumerable<T>`, respectively.
The ***enumerable interfaces*** are the non-generic interface `System.Collections.IEnumerable` and all instantiations of the generic interface `System.Collections.Generic.IEnumerable<T>` (`System.Collections.Generic.IAsyncEnumerable<T>` for async streams). For the sake of brevity, in this subclause and its siblings these interfaces are referenced as `IEnumerable` and `IEnumerable<T>` (`IAsyncEnumerable<T>` for async streams), respectively.

### 15.14.4 Yield type

An iterator produces a sequence of values, all of the same type. This type is called the ***yield type*** of the iterator.

- The yield type of an iterator that returns `IEnumerator` or `IEnumerable` is `object`.
- The yield type of an iterator that returns `IEnumerator<T>` or `IEnumerable<T>` is `T`.
- The yield type of an iterator that returns `IEnumerator<T>` (`IAsyncEnumerator<T>` for async streams) or `IEnumerable<T>` (`IAsyncEnumerable<T>` for async streams) is `T`.

### 15.14.5 Enumerator objects

#### 15.14.5.1 General

When a function member returning an enumerator interface type is implemented using an iterator block, invoking the function member does not immediately execute the code in the iterator block. Instead, an ***enumerator object*** is created and returned. This object encapsulates the code specified in the iterator block, and execution of the code in the iterator block occurs when the enumerator object’s `MoveNext` method is invoked. An enumerator object has the following characteristics:

- It implements `IEnumerator` and `IEnumerator<T>`, where `T` is the yield type of the iterator.
- It implements `System.IDisposable`.
- It implements `IEnumerator` and `IEnumerator<T>` (`IAsyncEnumerator<T>` for async streams), where `T` is the yield type of the iterator.
- It implements `System.IDisposable` (`IAsyncDisposable` for async streams).
- It is initialized with a copy of the argument values (if any) and instance value passed to the function member.
- It has four potential states, **before**, **running**, **suspended**, and **after**, and is initially in the **before** state.

An enumerator object is typically an instance of a compiler-generated enumerator class that encapsulates the code in the iterator block and implements the enumerator interfaces, but other methods of implementation are possible. If an enumerator class is generated by the compiler, that class will be nested, directly or indirectly, in the class containing the function member, it will have private accessibility, and it will have a name reserved for compiler use ([§6.4.3](lexical-structure.md#643-identifiers)).

An enumerator object may implement more interfaces than those specified above.

The following subclauses describe the required behavior of the `MoveNext`, `Current`, and `Dispose` members of the `IEnumerator` and `IEnumerator<T>` interface implementations provided by an enumerator object.
The following subclauses describe the required behavior of the `MoveNext`, `Current`, and `Dispose` members of the `IEnumerator` and `IEnumerator<T>` (`IAsyncEnumerator<T>` for async streams) interface implementations provided by an enumerator object.

Enumerator objects do not support the `IEnumerator.Reset` method. Invoking this method causes a `System.NotSupportedException` to be thrown.

#### 15.14.5.2 The MoveNext method

The `MoveNext` method of an enumerator object encapsulates the code of an iterator block. Invoking the `MoveNext` method executes code in the iterator block and sets the `Current` property of the enumerator object as appropriate. The precise action performed by `MoveNext` depends on the state of the enumerator object when `MoveNext` is invoked:
The `MoveNext` (`MoveNextAsync` for async streams) method of an enumerator object encapsulates the code of an iterator block. Invoking the `MoveNext` method executes code in the iterator block and sets the `Current` property of the enumerator object as appropriate. The precise action performed by `MoveNext` depends on the state of the enumerator object when `MoveNext` is invoked:

- If the state of the enumerator object is **before**, invoking `MoveNext`:
- Changes the state to **running**.
Expand Down Expand Up @@ -5407,11 +5409,11 @@ An enumerator object’s `Current` property is affected by `yield return` statem

When an enumerator object is in the **suspended** state, the value of `Current` is the value set by the previous call to `MoveNext`. When an enumerator object is in the **before**, **running**, or **after** states, the result of accessing `Current` is unspecified.

For an iterator with a yield type other than `object`, the result of accessing `Current` through the enumerator object’s `IEnumerable` implementation corresponds to accessing `Current` through the enumerator object’s `IEnumerator<T>` implementation and casting the result to `object`.
For an iterator with a yield type other than `object`, the result of accessing `Current` through the enumerator object’s `IEnumerable` implementation corresponds to accessing `Current` through the enumerator object’s `IEnumerator<T>` (`IAsyncEnumerator<T>` for async streams) implementation and casting the result to `object`.

#### 15.14.5.4 The Dispose method

The `Dispose` method is used to clean up the iteration by bringing the enumerator object to the **after** state.
The `Dispose` (`DisposeAsync` for async streams) method is used to clean up the iteration by bringing the enumerator object to the **after** state.

- If the state of the enumerator object is **before**, invoking `Dispose` changes the state to **after**.
- If the state of the enumerator object is **running**, the result of invoking `Dispose` is unspecified.
Expand All @@ -5425,9 +5427,9 @@ The `Dispose` method is used to clean up the iteration by bringing the enumerato

#### 15.14.6.1 General

When a function member returning an enumerable interface type is implemented using an iterator block, invoking the function member does not immediately execute the code in the iterator block. Instead, an ***enumerable object*** is created and returned. The enumerable object’s `GetEnumerator` method returns an enumerator object that encapsulates the code specified in the iterator block, and execution of the code in the iterator block occurs when the enumerator object’s `MoveNext` method is invoked. An enumerable object has the following characteristics:
When a function member returning an enumerable interface type is implemented using an iterator block, invoking the function member does not immediately execute the code in the iterator block. Instead, an ***enumerable object*** is created and returned. The enumerable object’s `GetEnumerator` (`GetAsyncEnumerator` for async streams) method returns an enumerator object that encapsulates the code specified in the iterator block, and execution of the code in the iterator block occurs when the enumerator object’s `MoveNext` (`MoveNextAsync` for async streams) method is invoked. An enumerable object has the following characteristics:

- It implements `IEnumerable` and `IEnumerable<T>`, where `T` is the yield type of the iterator.
- It implements `IEnumerable` and `IEnumerable<T>` (`IAsyncEnumerable<T>` for async streams), where `T` is the yield type of the iterator.
- It is initialized with a copy of the argument values (if any) and instance value passed to the function member.

An enumerable object is typically an instance of a compiler-generated enumerable class that encapsulates the code in the iterator block and implements the enumerable interfaces, but other methods of implementation are possible. If an enumerable class is generated by the compiler, that class will be nested, directly or indirectly, in the class containing the function member, it will have private accessibility, and it will have a name reserved for compiler use ([§6.4.3](lexical-structure.md#643-identifiers)).
Expand All @@ -5438,7 +5440,7 @@ An enumerable object may implement more interfaces than those specified above.

#### 15.14.6.2 The GetEnumerator method

An enumerable object provides an implementation of the `GetEnumerator` methods of the `IEnumerable` and `IEnumerable<T>` interfaces. The two `GetEnumerator` methods share a common implementation that acquires and returns an available enumerator object. The enumerator object is initialized with the argument values and instance value saved when the enumerable object was initialized, but otherwise the enumerator object functions as described in [§15.14.5](classes.md#15145-enumerator-objects).
An enumerable object provides an implementation of the `GetEnumerator` (`GetAsyncEnumerator` for async streams) methods of the `IEnumerable` and `IEnumerable<T>` (`IAsyncEnumerable<T>` for async streams) interfaces. The two `GetEnumerator` methods share a common implementation that acquires and returns an available enumerator object. The enumerator object is initialized with the argument values and instance value saved when the enumerable object was initialized, but otherwise the enumerator object functions as described in [§15.14.5](classes.md#15145-enumerator-objects).

## 15.15 Async Functions

Expand Down Expand Up @@ -5542,3 +5544,52 @@ When the body of the async function terminates, the return task is moved out of
If the return type of the async function is `void`, evaluation differs from the above in the following way: Because no task is returned, the function instead communicates completion and exceptions to the current thread’s ***synchronization context***. The exact definition of synchronization context is implementation-dependent, but is a representation of “where” the current thread is running. The synchronization context is notified when evaluation of a `void`-returning async function commences, completes successfully, or causes an uncaught exception to be thrown.

This allows the context to keep track of how many `void`-returning async functions are running under it, and to decide how to propagate exceptions coming out of them.

### §asynchronous-streams Asynchronous stream creation and usage
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this section at all? (Really not needed in normative text?)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(All of the bits in the bullet points can be included in the iterator section.)


A stream can be created and consumed asynchronously.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The definition of “stream” needs to be added here, elsewhere in the Standard we have streams of Unicode characters and tokens, uses of StreamReader etc.

See previous comments for how this might be done; it may even be decided to move this section into the (now following) §15.15 Iterators and Async Streams clause.


A method that creates an asynchronous stream has the following characteristics:

- It's declared with the `async` modifier.
- It returns an `IAsyncEnumerable<T>`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or IAsyncEnumerator<T>.

- It contains a `yield return` statement to return successive elements in the asynchronous stream.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to add that the argument to yield return must be implicitly convertible to the type argument for the return.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't combine this section with the enumerator section, we should add that yield break is can be used.


> *Example*:
>
> <!-- Example: {template:"code-in-partial-class", name:"AsyncStreamCandU1", additionalFiles:["Support1AsyncStreams.cs"], ignoreOutput:true} -->
> <!-- Maintenance Note: A version of this method exists in additional-files as "Support2AsyncStreams.cs". As such, certain changes to this method definition might need to be reflected in that file, in which case, *all* examples using that file should be tested. -->
> <!-- Note: The output is ignored, as it is asynchronous, and might not be complete. -->
> ```csharp
> static async IAsyncEnumerable<int> GenerateSequence()
> {
> for (int i = 0; i <= 5; i++)
> {
> await Task.Delay(100);
> yield return i;
> }
> }
> ```
>
> *end example*

Consumption of an asynchronous stream requires an `await foreach` to enumerate the elements of the stream. Adding the `await` keyword requires the method that enumerates the asynchronous stream to be declared with the `async` modifier and to return a type allowed for an async method.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"requires" seems too strong; it is certainly possible to enumerate the items of an asynchronous stream by calling IAsyncEnumerator<T>.MoveNextAsync explicitly, without await foreach. I thought perhaps "consumption" is a term that means only foreach loops, but I didn't find the term used with such a meaning anywhere else in this repo.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels like an odd paragraph in general. In particular, the "Adding the await keyword" part feels ambiguous - adding it to what? I think we can make that a lot clearer, probably taking language from earlier in 15.15. (The program text for a method can include await without it being asynchronous, due to local functions of course, so some care is required.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More later, but I think a good way to address this is to look at the statements section, and add a new subclause for "The await foreach" statement, rather than modifying the foreach statement section.


> *Example*:
>
> <!-- Example: {template:"code-in-partial-class", name:"AsyncStreamCandU2", additionalFiles:["Support2AsyncStreams.cs"], ignoreOutput:true} -->
> <!-- Maintenance Note: A version of this method exists in additional-files as "Support1AsyncStreams.cs". As such, certain changes to this method definition might need to be reflected in that file, in which case, *all* examples using that file should be tested. -->
> <!-- Note: The output is ignored, as it is asynchronous, and might not be complete. -->
> ```csharp
> static async Task M()
> {
> await foreach (var number in GenerateSequence())
> {
> Console.WriteLine(number);
> }
> }
> ```
>
> *end example*

A method can both consume and produce an asynchronous stream.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this sentence? I don't see the equivalent for synchronous iterators.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's needed.

30 changes: 30 additions & 0 deletions standard/standard-library.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -379,9 +379,22 @@ The following types, including the members listed, shall be defined in a conform

A conforming implementation may provide `Task.GetAwaiter()` and `Task<TResult>.GetAwaiter()` as extension methods.

> **Note to TG2 reviewers:** The .NET library actually declares `GetAsyncEnumerator` to take one argument, as follows:

```csharp
IAsyncEnumerator<T> GetAsyncEnumerator(System.Threading.CancellationToken cancellationToken = default);
```

> However, as we have not mentioned type `System.Threading.CancellationToken` in any other threading contexts in the spec, it has been omitted from the requirements, as if there was no argument expected.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my opinion, System.Threading.CancellationToken and System.Runtime.CompilerServices.EnumeratorCancellationAttribute should be added to the standard.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the C# standard defines IAsyncEnumerable<T> with IAsyncEnumerator<T> GetAsyncEnumerator();, and a user-defined class implements IAsyncEnumerable<T> by explicit interface implementation:

using System;
using System.Collections.Generic;
class Class1 : IAsyncEnumerable<int> {
    IAsyncEnumerator<int> IAsyncEnumerable<int>.GetAsyncEnumerator() {
        throw new NotImplementedException();
    }
}

then Class1 will not be portable between the standard and the .NET implementation, as the method signature will not match.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that we should add this to the standard and make everything match up. We could probably do it in a second PR, if that's easier.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let’s do it in this PR


```csharp
namespace System
{
public interface IAsyncDisposable
{
ValueTask DisposeAsync();
jskeet marked this conversation as resolved.
Show resolved Hide resolved
}

public class FormattableString : IFormattable { }

public class OperationCanceledException : Exception
Expand Down Expand Up @@ -564,6 +577,20 @@ namespace System.Diagnostics.CodeAnalysis
}
}

namespace System.Collections.Generic
{
public interface IAsyncEnumerable<out T>
{
IAsyncEnumerator<T> GetAsyncEnumerator();
}

public interface IAsyncEnumerator<out T> : IAsyncDisposable
{
ValueTask<bool> MoveNextAsync();
T Current { get; }
}
}

namespace System.Linq.Expressions
{
public sealed class Expression<TDelegate>
Expand Down Expand Up @@ -1075,6 +1102,7 @@ The following library types are referenced in this specification. The full names
- `global::System.Exception`
- `global::System.FormattableString`
- `global::System.GC`
- `global::System.IAsyncDisposable`
- `global::System.IDisposable`
- `global::System.IFormattable`
- `global::System.IndexOutOfRangeException`
Expand Down Expand Up @@ -1118,6 +1146,8 @@ The following library types are referenced in this specification. The full names
- `global::System.Collections.IEnumerable`
- `global::System.Collections.IEnumerator`
- `global::System.Collections.IList`
- `global::System.Collections.Generic.IAsyncEnumerable<out T>`
- `global::System.Collections.Generic.IAsyncEnumerator<out T>`
- `global::System.Collections.Generic.ICollection<T>`
- `global::System.Collections.Generic.IEnumerable<T>`
- `global::System.Collections.Generic.IEnumerator<T>`
Expand Down
Loading
Loading