7904061: Enable JMH integration tests to be executed against other impls

[galderz]

NOTE: This PR should be marked as draft because not all tests have been migrated to this setup. I've migrated the ones in the root test package to get a feel on how things would work and get feedback on whether this approach is fine for the JMH community.

This is the last in a series of PRs extending JMH to benchmark Java code running as GraalVM native images more easily.

In this PR:

• We enable jmh-core-it test jar to be created so that JMH extensions can consume it.
• We create a test RunnerFactory interface with a method called createRunner that by default creates JMH's Runner instance. JMH extensions that want to run the same test with a different runner would apply changes like these. Here the JMH extension extends the default RunnerFactory interface, and extends individual test classes to make them implement the extended runner factory interface.

The result of both these changes enables JMH tests to be executed with the JMH extension:

[INFO] --- failsafe:3.5.2:integration-test (default) @ fibula-jmh-it ---
[INFO] Using auto detected provider org.apache.maven.surefire.junit4.JUnit4Provider
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.mendrugo.fibula.jmh.it.SingleShotTest
# JMH version: fibula:999-SNAPSHOT
# VM version: JDK 24.0.1, Substrate VM, GraalVM CE 24.0.1+9.1
# *** WARNING: This VM is not supported by JMH. The produced benchmark data can be completely wrong.
# VM invoker: target/benchmarks
...
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.536 s -- in org.mendrugo.fibula.jmh.it.SingleShotTest
[INFO] Running org.mendrugo.fibula.jmh.it.IterationCountCmdTest
# JMH version: fibula:999-SNAPSHOT
# VM version: JDK 24.0.1, Substrate VM, GraalVM CE 24.0.1+9.1
# *** WARNING: This VM is not supported by JMH. The produced benchmark data can be completely wrong.
# VM invoker: target/benchmarks
...
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.712 s -- in org.mendrugo.fibula.jmh.it.IterationCountCmdTest
[INFO] Running org.mendrugo.fibula.jmh.it.IterationCountAnnTest
# JMH version: fibula:999-SNAPSHOT
# VM version: JDK 24.0.1, Substrate VM, GraalVM CE 24.0.1+9.1
# *** WARNING: This VM is not supported by JMH. The produced benchmark data can be completely wrong.
# VM invoker: target/benchmarks
...
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.345 s -- in org.mendrugo.fibula.jmh.it.IterationCountAnnTest
[INFO] Running org.mendrugo.fibula.jmh.it.WarmupIterationCountAnnTest
# JMH version: fibula:999-SNAPSHOT
# VM version: JDK 24.0.1, Substrate VM, GraalVM CE 24.0.1+9.1
# *** WARNING: This VM is not supported by JMH. The produced benchmark data can be completely wrong.
# VM invoker: target/benchmarks
...
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.751 s -- in org.mendrugo.fibula.jmh.it.WarmupIterationCountAnnTest
[INFO] Running org.mendrugo.fibula.jmh.it.WarmupIterationCountCmdTest
# JMH version: fibula:999-SNAPSHOT
# VM version: JDK 24.0.1, Substrate VM, GraalVM CE 24.0.1+9.1
# *** WARNING: This VM is not supported by JMH. The produced benchmark data can be completely wrong.
# VM invoker: target/benchmarks
...
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.756 s -- in org.mendrugo.fibula.jmh.it.WarmupIterationCountCmdTest
[INFO] Running org.mendrugo.fibula.jmh.it.StackTraceInThrowableTest
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0 s -- in org.mendrugo.fibula.jmh.it.StackTraceInThrowableTest
[INFO] 
[INFO] Results:
[INFO] 
[WARNING] Tests run: 7, Failures: 0, Errors: 0, Skipped: 1

Note that above we can observe that one of the JMH tests is disabled. It could happen that several JMH tests are not applicable to native image, e.g. perfasm tests. In the usage changes above we can observe how JUnit's @Ignore annotation can be easily used to disable particular tests as needed. In the particular case of PrintFlags, the JMH extension could filter this flag and not pass on (there are already such cases) so with that modification the test could pass but I left it as is so that test exclusion can be demonstrated for this PR.

Here is the PR list for reference:

1. Make OutputFormatAdapter public
2. Enable BenchmarkParams construction to be overriden
3. Enable alternative Runner instantiation and Main reuse
4. Enable profiler classes to be reused outside of JMH
5. Enable JMH integration tests to be executed against other impls

---

Progress

• Change must not contain extraneous whitespace
• Change must be properly reviewed (1 review required, with at least 1 Committer)

Issue

• CODETOOLS-7904061: Enable JMH integration tests to be executed against other impls (Enhancement - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jmh.git pull/163/head:pull/163
$ git checkout pull/163

Update a local copy of the PR:
$ git checkout pull/163
$ git pull https://git.openjdk.org/jmh.git pull/163/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 163

View PR using the GUI difftool:
$ git pr show -t 163

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jmh/pull/163.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back galder! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[mlbridge]

Webrevs

• 00: Full (4b871393)

[bridgekeeper]

@galderz This pull request has been inactive for more than 4 weeks and will be automatically closed if another 4 weeks passes without any activity. To avoid this, simply issue a /touch or /keepalive command to the pull request. Feel free to ask for assistance if you need help with progressing this pull request towards integration!

[galderz]

/touch

[openjdk]

@galderz The pull request is being re-evaluated and the inactivity timeout has been reset.

[bridgekeeper]

@galderz This pull request has been inactive for more than 4 weeks and will be automatically closed if another 4 weeks passes without any activity. To avoid this, simply issue a /touch or /keepalive command to the pull request. Feel free to ask for assistance if you need help with progressing this pull request towards integration!

[galderz]

/touch

[openjdk]

@galderz The pull request is being re-evaluated and the inactivity timeout has been reset.

[bridgekeeper]

@galderz This pull request has been inactive for more than 4 weeks and will be automatically closed if another 4 weeks passes without any activity. To avoid this, simply issue a /touch or /keepalive command to the pull request. Feel free to ask for assistance if you need help with progressing this pull request towards integration!

[galderz]

/touch

[openjdk]

@galderz The pull request is being re-evaluated and the inactivity timeout has been reset.

[bridgekeeper]

@galderz This pull request has been inactive for more than 4 weeks and will be automatically closed if another 4 weeks passes without any activity. To avoid this, simply issue a /touch or /keepalive command to the pull request. Feel free to ask for assistance if you need help with progressing this pull request towards integration!

[galderz]

/touch

[openjdk]

@galderz The pull request is being re-evaluated and the inactivity timeout has been reset.