8373366: HandshakeState should disallow suspend ops for disabler threads

[sspitsyn]

An asynchronous handshake operation (ThreadSelfSuspensionHandshakeClosure) can be installed when the target thread is not in a MountUnmountDisabler scope. But the target thread can enter such scope by the time the operation is self-processed by the target thread.

This is fixed by a small tweak in the function
HandshakeOperation* HandshakeState::get_op_for_self(bool allow_suspend, bool check_async_exception).
The tweak is to skip a HandshakeOperation if _handshakee->is_vthread_transition_disabler() == true, so the same temporary suspension disabling mechanism would be used as for _handshakee->is_disable_suspend() == true.

All other changes are to move the is_vthread_transition_disabler() out of DEBUG to product.

Testing:

• In progress: mach5 tiers 1-6

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8373366: HandshakeState should disallow suspend ops for disabler threads (Bug - P3)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/28740/head:pull/28740
$ git checkout pull/28740

Update a local copy of the PR:
$ git checkout pull/28740
$ git pull https://git.openjdk.org/jdk.git pull/28740/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 28740

View PR using the GUI difftool:
$ git pr show -t 28740

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/28740.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back sspitsyn! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@sspitsyn The following label will be automatically applied to this pull request:

• hotspot-runtime

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 01: Full - Incremental (b51267f3)
• 00: Full (8c25f690)

[lmesnik]

The fix looks good. However, I think the it makes a sense to add regression test that tries to suspend thread while it processing event callback.

Can you please show where mounting/unmounting is disabled for jvmti events. i was unable to fines this place.

[lmesnik]

/cc serviceability

[openjdk]

@lmesnik
The serviceability label was successfully added.

[pchilano]

Hi Serguei, some comments below, thanks.

[sspitsyn]

However, I think the it makes a sense to add regression test that tries to suspend thread while it processing event callback.

Right question, thanks. This issue came from testing of the PR #28407 .
The issue appeared on the updated test: test/hotspot/jtreg/serviceability/jvmti/vthread/ThreadStateTest.
Initially, I wanted to move the test update to this PR but encountered one more issue which needs to be addressed separately. So, I gave up on moving the test update.

Can you please show where mounting/unmounting is disabled for jvmti events. i was unable to fines this place.

I might incorrectly understand your question. Let's discuss it offline.