TestRunListenerTest5.testThatLauncherLibGetsAdded fails since I20251026-0750

[iloveeclipse]

TestRunListenerTest5.testThatLauncherLibGetsAdded fails since I20251026-0750 build (but actually before, just the builds were broken, see eclipse-platform/eclipse.platform.releng.aggregator#3430). Las worked in I20251024-0020.

Most likely combined result of eclipse-jdt/eclipse.jdt#146 / eclipse-jdt/eclipse.jdt@e5b600e and #2536 / 28fa254 (as the only test case is failing that is supposed to check whether specific junit-platform-launcher library is added if JUNIT5_CONTAINER_PATH is removed but the rest of JUnit 5 libraries are on classpath). Most likely test needs to add version ranges to JUnit libraries.

Note: the test works just fine in IDE / build I20251026-1800 is started with default settings, means something (most likely the installed bundles in platform) is different at tests runtime.

https://download.eclipse.org/eclipse/downloads/drops4/I20251026-1800/testresults/html/org.eclipse.jdt.ui.tests_ep438I-unit-linux-x86_64-java21_linux.gtk.x86_64_21.html

expected:<[sessionStarted-sessionName: ATestCase state: Running result: Undefined testCaseStarted-testCaseMethod: testSucceed class: pack.ATestCase state: Running result: Undefined testCaseFinished-testCaseMethod: testSucceed class: pack.ATestCase state: Completed result: OK sessionFinished-sessionName: ATestCase state: Completed result: OK ] > but was:<[AbstractTestRunListenerTest#launchJUnit(IJavaElement, TestRunLog) timed out] >

org.junit.ComparisonFailure: expected:<[sessionStarted-sessionName: ATestCase
state: Running
result: Undefined

testCaseStarted-testCaseMethod: testSucceed
class: pack.ATestCase
state: Running
result: Undefined

testCaseFinished-testCaseMethod: testSucceed
class: pack.ATestCase
state: Completed
result: OK

sessionFinished-sessionName: ATestCase
state: Completed
result: OK
]
> but was:<[AbstractTestRunListenerTest#launchJUnit(IJavaElement, TestRunLog) timed out]
>
at org.junit.Assert.assertEquals(Assert.java:117)
at org.junit.Assert.assertEquals(Assert.java:146)
at org.eclipse.jdt.junit.tests.AbstractTestRunListenerTest.assertEqualLog(AbstractTestRunListenerTest.java:249)
at org.eclipse.jdt.junit.tests.TestRunListenerTest5.testThatLauncherLibGetsAdded(TestRunListenerTest5.java:173)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)

[iloveeclipse]

@trancexpress : could you please check?

[trancexpress]

I cant reproduce neither with the latest SDK running the test manually, nor on command line from latest eclipse.jdt.ui state.

The message is printed here: org.eclipse.jdt.junit.tests.AbstractTestRunListenerTest.launchJUnit(IJavaElement, String, String, TestRunLog)

		if (! success)
			log.add("AbstractTestRunListenerTest#launchJUnit(IJavaElement, TestRunLog) timed out");

Since there is nothing else in this log, I guess something must be failing before the test session can even start... I wonder how to reproduce this though. Running the whole aggregator build would be taking quite a while...

Nothing useful on standard out: https://download.eclipse.org/eclipse/downloads/drops4/I20251026-1800/testresults/ep438I-unit-linux-x86_64-java21_linux.gtk.x86_64_21/org.eclipse.jdt.ui.tests.AutomatedSuite.txt

The code that is being tested is:

"main" #1 [203456] prio=6 os_prio=0 cpu=3821.42ms elapsed=34.43s tid=0x00007f7ee001b300 nid=203456 at breakpoint [0x00007f7ee5dfb000]
   java.lang.Thread.State: RUNNABLE
        at org.eclipse.jdt.junit.launcher.JUnitLaunchConfigurationDelegate.getVMRunnerConfiguration(JUnitLaunchConfigurationDelegate.java:200)
        at org.eclipse.jdt.junit.launcher.JUnitLaunchConfigurationDelegate.launch(JUnitLaunchConfigurationDelegate.java:275)
        - locked <0x000000071be00400> (a org.eclipse.jdt.junit.launcher.AdvancedJUnitLaunchConfigurationDelegate)
        at org.eclipse.debug.internal.core.LaunchConfiguration.launch(LaunchConfiguration.java:778)
        at org.eclipse.debug.internal.core.LaunchConfiguration.launch(LaunchConfiguration.java:689)
        at org.eclipse.debug.internal.core.LaunchConfiguration.launch(LaunchConfiguration.java:684)
        at org.eclipse.jdt.junit.tests.AbstractTestRunListenerTest.launchJUnit(AbstractTestRunListenerTest.java:190)
        at org.eclipse.jdt.junit.tests.AbstractTestRunListenerTest.launchJUnit(AbstractTestRunListenerTest.java:214)
        at org.eclipse.jdt.junit.tests.AbstractTestRunListenerTest.launchJUnit(AbstractTestRunListenerTest.java:210)
        at org.eclipse.jdt.junit.tests.AbstractTestRunListenerTest.launchJUnit(AbstractTestRunListenerTest.java:206)
        at org.eclipse.jdt.junit.tests.TestRunListenerTest5.runSequenceTest(TestRunListenerTest5.java:40)
        at org.eclipse.jdt.junit.tests.TestRunListenerTest5.testThatLauncherLibGetsAdded(TestRunListenerTest5.java:172)

Maybe this is not added with a specific version and sometimes it works, sometimes it doesn't...

					if (!Arrays.stream(classpath).anyMatch(s -> s.contains("junit-platform-launcher") || s.contains("org.junit.platform.launcher"))) { //$NON-NLS-1$ //$NON-NLS-2$
						try {
							JUnitRuntimeClasspathEntry x= new JUnitRuntimeClasspathEntry("junit-platform-launcher", null); //$NON-NLS-1$
							String entryString= new ClasspathLocalizer(Platform.inDevelopmentMode()).entryString(x);
							int length= classpath.length;
							System.arraycopy(classpath, 0, classpath= new String[length + 1], 0, length);
							classpath[length]= entryString;
						} catch (IOException | URISyntaxException e) {
							throw new CoreException(new Status(IStatus.ERROR, JUnitCorePlugin.CORE_PLUGIN_ID, IStatus.ERROR, "", e)); //$NON-NLS-1$
						}
					}

I get the same fail with:

diff --git a/org.eclipse.jdt.junit.core/src/org/eclipse/jdt/junit/launcher/JUnitLaunchConfigurationDelegate.java b/org.eclipse.jdt.junit.core/src/org/eclipse/jdt/junit/launcher/JUnitLaunchConfigurationDelegate.java
index e97ac908a3..1297d43cb3 100644
--- a/org.eclipse.jdt.junit.core/src/org/eclipse/jdt/junit/launcher/JUnitLaunchConfigurationDelegate.java
+++ b/org.eclipse.jdt.junit.core/src/org/eclipse/jdt/junit/launcher/JUnitLaunchConfigurationDelegate.java
@@ -200,8 +200,8 @@ public class JUnitLaunchConfigurationDelegate extends AbstractJavaLaunchConfigur
                                                        JUnitRuntimeClasspathEntry x= new JUnitRuntimeClasspathEntry("junit-platform-launcher", null); //$NON-NLS-1$
                                                        String entryString= new ClasspathLocalizer(Platform.inDevelopmentMode()).entryString(x);
                                                        int length= classpath.length;
-                                                       System.arraycopy(classpath, 0, classpath= new String[length + 1], 0, length);
-                                                       classpath[length]= entryString;
+//                                                     System.arraycopy(classpath, 0, classpath= new String[length + 1], 0, length);
+//                                                     classpath[length]= entryString;
                                                } catch (IOException | URISyntaxException e) {
                                                        throw new CoreException(new Status(IStatus.ERROR, JUnitCorePlugin.CORE_PLUGIN_ID, IStatus.ERROR, "", e)); //$NON-NLS-1$
                                                }

So it must be something like that, bad version.

[trancexpress]

Lets see if this helps: #2574

It includes the change from: #2565

[iloveeclipse]

Note: the test works just fine in IDE / build I20251026-1800 is started with default settings

I can now reproduce the fail in the IDE if following same "steps" as in eclipse-platform/eclipse.platform#2213.

All what is needed is to select "Restore defaults" in the launch config, which will add both junit 5 and 6 to the target platform. Default launch config that only uses required dependencies doesn't fail because it uses only junit 5.

This PR fixes the test failure, but PDE fix eclipse-pde/eclipse.pde#2070 is needed to properly run test.