JIT: Add an opportunistic optimization to reuse continuation instances

[jakobbotsch]

This adds an optimization that tries to reuse the continuation instance that was used to resume a runtime async function if it suspends again.

This optimization is not reliable. It only kicks in if the same suspension point resumes and then suspends again. However, it is useful to get an idea of how costly the continuation allocation part is when looking at the gap between
async1 and runtime async in raw suspension/resumption performance.

The optimization is controlled by DOTNET_JitOpportunisticContinuationReuse and is disabled by default.

Despite the above this is not a very costly transformation, so we may consider enabling it even if it does not kick in very reliably.

Example benchmark that measures raw suspension/resumption performance (best case for async1, worst case for runtime async):

using System;
using System.Diagnostics;
using System.Runtime.CompilerServices;
using System.Threading.Tasks;

public class Program
{
    static void Main()
    {
        NullAwaiter na = new NullAwaiter();

        Task t = Foo(100_000_000, na);
        while (!t.IsCompleted)
        {
            na.Continue();
        }
        Console.WriteLine("Finished");
    }

    static async Task Foo(int n, NullAwaiter na)
    {
        Stopwatch timer = Stopwatch.StartNew();
        for (int i = 0; i < n; i++)
        {
            await na;
        }
        Console.WriteLine("Took {0:F1} ms", timer.Elapsed.TotalMilliseconds);
    }

    private class NullAwaiter : ICriticalNotifyCompletion
    {
        public Action Continue;

        public NullAwaiter GetAwaiter() => this;

        public bool IsCompleted => false;

        public void GetResult()
        {
        }

        public void UnsafeOnCompleted(Action continuation)
        {
            Continue = continuation;
        }

        public void OnCompleted(Action continuation)
        {
            throw new NotImplementedException();
        }
    }
}

With DOTNET_JitOpportunisticContinuationReuse=0: Took 5788.9 ms
With DOTNET_JitOpportunisticContinuationReuse=1: Took 4611.5 ms
Async1: Took 3433.7 ms

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[jakobbotsch]

I plan to also build a "mega continuation" mode, just so that we have the ability to try it out on scenarios. Mega continuations would be able to be reused always, so it would make the optimization reliable, but it comes with larger continuations which is a more tangible downside.