Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TaskEx: parallelLimit #143

Open
bartelink opened this issue Jan 5, 2023 · 0 comments
Open

TaskEx: parallelLimit #143

bartelink opened this issue Jan 5, 2023 · 0 comments
Labels
feature request New feature or enhancement request topic: task-ex Related to the proposed new TaskEx library, which should get its own oss haven
Milestone

Comments

@bartelink
Copy link
Member

bartelink commented Jan 5, 2023

Replaces #129. TaskEx top level issue: #139

Async.Parallel's optional degree of parallelism parameter was added late in the game, but is critical - dumping an arbitrary unbounded number of work items onto the threadpool is not something that should be easy and/or the default thing to do without due consideration for how that will work under stess.

There are some other shortcomings, which frequently lead to various bespoke helpers proliferating:

  • pipelining is painful, necessitating an explicit argument name (e.g. fun computations -> Async.Parallel(computations, maxDegreeOfParallelism=dop) etc) (note this is not the case for Async.Sequential)
  • before v FSharp.Core v 6.0.6, [there was a stack overflow bug that can tear down the process](// Async.Parallel stack overflow on cancellation of ~2000 uncompleted computations dotnet/fsharp#13165) if >1200 items are started with a throttle and cancellation is triggered quickly (so having a layer between Async.Parallel and direct consumption within an app might be useful)

Current proposed APIs (will be updated inline based on any discussion below):

module Async =
    let parallelLimit maxDegreeOfParallelism computations =
        Async.Parallel(computations, maxDegreeOfParallelism = maxDegreeOfParallelism)

NOTES:

@bartelink bartelink changed the title TaskShims: parallelThrottled TaskEx: parallelThrottled Jan 10, 2023
@bartelink bartelink changed the title TaskEx: parallelThrottled TaskEx: parallelLimit Jan 18, 2023
bartelink added a commit to jet/propulsion that referenced this issue Jan 23, 2023
bartelink added a commit to jet/propulsion that referenced this issue Jan 23, 2023
@abelbraaksma abelbraaksma added feature request New feature or enhancement request topic: task-ex Related to the proposed new TaskEx library, which should get its own oss haven labels Oct 29, 2023
@abelbraaksma abelbraaksma added this to the vFuture milestone Mar 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or enhancement request topic: task-ex Related to the proposed new TaskEx library, which should get its own oss haven
Projects
None yet
Development

No branches or pull requests

2 participants