8373468: Shenandoah: Mutator may block at _gc_waiters_lock after allocation failure even block parameter is false

[pengxiaolong]

There is behavior discrepancy between Shenandoah generational mode and non-generational when it handles mutator allocation failure, as stated in the description of JDK-8373468, in non-generational, the block parameter may not always work.

Further looking into ShenandoahGenerationalControlThread and ShenandoahControlThread, they handle the _requested_gc_cause and ShenandoahHeap::cancelled_cause differently in other places as well, the change in this PR will minimize the gap to unify the behavior, and also fix potentially missed allocation failure notifications in some rare cases for both control threads.

Test

• hotspot_gc_shenandoah
• GHA (includes tier1 but no unit unit with gtest)
• gtest

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8373468: Shenandoah: Mutator may block at _gc_waiters_lock after allocation failure even block parameter is false (Bug - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/28758/head:pull/28758
$ git checkout pull/28758

Update a local copy of the PR:
$ git checkout pull/28758
$ git pull https://git.openjdk.org/jdk.git pull/28758/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 28758

View PR using the GUI difftool:
$ git pr show -t 28758

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/28758.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back xpeng! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@pengxiaolong The following labels will be automatically applied to this pull request:

• hotspot-gc
• shenandoah

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 04: Full - Incremental (085a0497)
• 03: Full - Incremental (469dc1fc)
• 02: Full - Incremental (467b2746)
• 01: Full - Incremental (b3e18e9f)
• 00: Full (e60e71da)

[earthling-amzn]

_requested_gc_cause should always be accessed when holding the _control_lock. Making this atomic should not be unnecessary.

[pengxiaolong]

Yes, the change to use Atomic for _requested_gc_cause is unnecessary, I have reverted it with a minor change for ShenandoahControlThread to always access _requested_gc_cause with _control_lock held.

[earthling-amzn]

Can you describe the scenario you are trying to fix here? The two control threads have different idle/wakeup conditions.

[pengxiaolong]

Can you describe the scenario you are trying to fix here? The two control threads have different idle/wakeup conditions.

The major thing I am fixing in the PR is to make ShenandoahControlThread always honor the block parameter. In current impl ShenandoahControlThread may still block the mutator thread even block parameter is false, because the ShenandoahControlThread::request_gc blocks the mutator use _gc_waiters_lock for allocation failure; while ShenandoahGenerationalControlThread::request_gc doesn't, it simplify call notify_cancellation(cause) if cause is allocation failure.
Yes the two control threads have different idle/wakeup conditions, I have more changes in ShenandoahControlThread to make sure they have same/much closer behavior when handle alloc failure.