A future created inside of a parallel collection map / foreach sometimes blocks the fork join pool with JDK 11

[OndrejSpanel]

reproduction steps

(using Scala 2.12.12 or 2.13.3)

• clone a project from https://github.com/OndrejSpanel/ParLoopAndFuture
• run the Main class using JDK 11.0.4

problem

Observe the message "Suspicious background duration ..." being printed multiple times. This messages shows that a problem with a parallel foreach scheduling has been detected, namely that a long-duration future created from the parallel foreach has blocked the foreach executtion. This happens both with Scala 2.12.12 and Scala 2.13.3 running on JDK 11.0.4. It does not happen when running on JDK / JRE 8. It seems to happen intermittently, probably depending on exact timing of the Future call relative the to parallel collection tasks execution.

The foreach should normally terminate in much less than 200 ms, as 50 tasks are executed, each of them taking 2 ms. Once it takes more than 200 ms, it shows there is a scheduling problem with the slow future blocking the parallel collection tasks.

Following JFR screenshot shows the thread scheduling once the issue is seen (you can also see normals iterations without the issue):

My configuration is Windows 10 x64 build 2004, Intel i7 / 4x core.

The code demonstrating the issue is https://github.com/OndrejSpanel/ParLoopAndFuture/blob/c2f4d1b6b155217d8f8111f58e6a7c9337f36619/src/main/scala/com/github/ondrejspanel/parFuture/Main.scala#L1-L45

package com.github.ondrejspanel.parFuture

import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global

object Main {


  def hotSleep(ms: Int): Unit = {
    val now = System.currentTimeMillis()
    while (System.currentTimeMillis() < now + ms) {}

  }

  def main(args: Array[String]): Unit = {
    val stateCount = 50
    val state = (0 until stateCount).par

    val triggerIssue = true

    (0 until 1000).foreach { i =>
      hotSleep(25)
      val innerScopeBeg = System.currentTimeMillis()
      if (!triggerIssue) {
        Future {
          hotSleep(205)
        }
      }
      state.foreach { x =>
        if (triggerIssue && x == 0) {
          Future {
            hotSleep(205)
          }
        }
        hotSleep(2)
      }
      val innerScopeEnd = System.currentTimeMillis()
      val duration = innerScopeEnd - innerScopeBeg
      if (duration >= 200) {
        println(s"Suspicious background duration $duration")
      }
    }
  }

}

[SethTisue]

does using blocking help? see #12089

cc @viktorklang

[viktorklang]

@SethTisue blocking could help, but ultimately, if it manifests on 2.12 it is likely a FJP thing.

[OndrejSpanel]

@SethTisue Where should I use blocking? There is no blocking or sleeping anywhere in the code. The hotSleep function is just used to simulate a CPU intensive code, no real sleeping there.

[OndrejSpanel]

it is likely a FJP thing

I think this is very likely, as the thread which is waiting is inside of the java.util.concurrent.ForkJoinTask#internalWait function.

[viktorklang]

@OndrejSpanel blocking means "blocking progress for other tasks"—whish seems to be what hotSleep does.

[OndrejSpanel]

I have tested it and it does not help, at least when used as this:

            Future {
              blocking {
                hotSleep(205)
              }
            }

[OndrejSpanel]

blocking means "blocking progress for other tasks"

@viktorklang I find such definition unusual. Compare with https://en.wikipedia.org/wiki/Blocking_(computing):

A process that is blocked is one that is waiting for some event, such as a resource becoming available or the completion of an I/O operation.

[viktorklang]

@OndrejSpanel Remember that when you execute on a thread pool you have cooperative scheduling for the tasks, not preemptive, so in this case, no matter if you are busy-spinning or sleeping you are preventing progress for other tasks.

[OndrejSpanel]

"busy-spinning" is just a simple replacement of a CPU intensive computation found in the real application where I see the issue. I could use a recursive Fibonacci computation instead, or something like that, but hotSleep is much eaiser to implement and understand.

When I execute one long task on a pool of 8 threads, I do not expect it to be blocking the whole process. The same code works on Java 8 without any issues and I do not see any reason why it should not.

There is some problem with Java 11 scheduling (most likely caused by some FJP implementation changes), which perhaps causes some short task to be scheduled on the thread which is busy with the long task, or something like that. Spawning more threads (which is the result of using blocking) would not help.

By posting the issue here I expect to achieve two things:

• the issue to be acknowledged and documented
• some reasonable workaround to be found

Fixing the issue either in Scala or in Java would be nice, but I do not expect for this to happen soon enough to help me in any way.

[lrytz]

Putting the blocking discussion aside, it would be good to keep investigating.

Since the same thread pool is used, maybe the surface area of the reproduction can be reduced? Avoid using parallel collections, or maybe even write a java-only reproducer? @OndrejSpanel do you have time to give this a try? Or even in the current state, it might be worth reporting it to Java concurrency experts. http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest for example be active.

For the record, I can reproduce the issue as described in the ticket on my macOS.

[OndrejSpanel]

Avoid using parallel collections

I did not see the issue when not using parallel collections. It is possible it could be reproduced by expanding the parallel collections to the plain Java executor calls, I am not sure how much work this would be.

@OndrejSpanel do you have time to give this a try?

I am not sure I want to dedicate much time to this. Given the issue shows with Java 11 which is available to general public, I guess a priority for me will be finding a suitable workaround, as I cannot expect the users of my application will avoid this particular runtime.

[lrytz]

Understood, thanks.

I also looked at it in JMC. Here's the stack trace of the waiting executor thread:

void java.lang.Object.wait(long)	1	100 %
void java.util.concurrent.ForkJoinTask.internalWait(long)	1	100 %
int java.util.concurrent.ForkJoinPool.awaitJoin(ForkJoinPool$WorkQueue, ForkJoinTask, long)	1	100 %
int java.util.concurrent.ForkJoinTask.doJoin()	1	100 %
Object java.util.concurrent.ForkJoinTask.join()	1	100 %
void scala.collection.parallel.ForkJoinTasks$WrappedTask.sync()	1	100 %
void scala.collection.parallel.ForkJoinTasks$WrappedTask.sync$(ForkJoinTasks$WrappedTask)	1	100 %
void scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.sync()	1	100 %
void scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.internal()	1	100 %
...

So the wait starts here: https://github.com/scala/scala/blob/v2.12.12/src/library/scala/collection/parallel/Tasks.scala#L174

It seems that the last subtask within the parallel collections task happens to be the spawned future, if I understand correctly?

I'll try to dig a little further in the next few days.

[OndrejSpanel]

the last subtask within the parallel collections task happens to be the spawned future

I am not sure I understand you correctly. If you check the source code of the repro, you will see the first iteration of the parallel collection spawns a future, but this does not mean the future becomes the subtask of the collection - the subtasks finishes normally without any further interaction with the spawned future. I repeat the corresponding source lines:

      state.foreach { x =>
        if (triggerIssue && x == 0) {
          Future {
            hotSleep(205)
          }
        }
        hotSleep(2)
      }

[lrytz]

Yes, the future task should be independent of the parallel collections task, but what seems to happen is that the parallel collections task waits for the future to complete at the code location I pointed to. I'll try to figure out if that's really it, and why.