Add note about duplicate test execution with @suite (#5080)

Users are surprised to see that their tests are executed twice[1]. This
is the natural consequence of using a test engine to define a suite.

We can at least let them know in the documentation.

1. #4327

Co-authored-by: Marc Philipp <[email protected]>

Author: mpkorstanje

CONTRIBUTING.md

@@ -177,8 +177,8 @@ To deprecate an item:
 
 ## Building the Project
 
-Please refer to [the readme](README.md#building-from-source) for the most common
-build commands.
+Please refer to [the readme](README.md#building-from-source) and [the documentation readme](documentation/README.md) for
+the most common build commands.
 
 ### Build Cache
 

documentation/README.md

@@ -18,7 +18,7 @@ This following Gradle command generates the HTML version of the User Guide as
 `build/docs/asciidoc/user-guide/index.html`.
 
 ```
-gradlew asciidoctor
+./gradlew asciidoctor
 ```
 
 On Linux operating systems, the `graphviz` package providing `/usr/bin/dot` must be
@@ -30,5 +30,5 @@ This following Gradle command generates the PDF version of the User Guide to
 `build/docs/asciidocPdf/user-guide/index.pdf`.
 
 ```
-gradlew asciidoctorPdf
+./gradlew asciidoctorPdf
 ```

documentation/documentation.gradle.kts

@@ -268,7 +268,7 @@ tasks {
 		outputs.cacheIf { true }
 	}
 
-	val componentDiagram = plantUml.flatMap { it.outputDirectory.file("component-diagram.svg") }
+	val plantUmlOutputDirectory = plantUml.flatMap { it.outputDirectory }
 
 	withType<AbstractAsciidoctorTask>().configureEach {
 		inputs.files(
@@ -278,15 +278,15 @@ tasks {
 			generateConsoleLauncherEnginesOptions,
 			generateApiTables,
 			generateStandaloneConsoleLauncherShadowedArtifactsFile,
-			componentDiagram
+			plantUmlOutputDirectory
 		)
 
 		resources {
 			from(sourceDir) {
 				include("**/images/**/*.png")
 				include("**/images/**/*.svg")
 			}
-			from(componentDiagram) {
+			from(plantUmlOutputDirectory) {
 				into("user-guide/images")
 			}
 		}

documentation/src/docs/asciidoc/user-guide/advanced-topics/junit-platform-suite-engine.adoc

@@ -1,8 +1,10 @@
 [[junit-platform-suite-engine]]
 === JUnit Platform Suite Engine
 
-The JUnit Platform supports the declarative definition and execution of suites of tests
-from _any_ test engine using the JUnit Platform.
+The Suite Engine supports the declarative selection and execution of tests from
+_any_ test engine on the JUnit Platform using the <<launcher-api>>.
+
+image::junit-platform-suite-engine-diagram.svg[role=text-center]
 
 [[junit-platform-suite-engine-setup]]
 ==== Setup
@@ -58,3 +60,19 @@ all tests of the test suite.
 ----
 include::{testDir}/example/BeforeAndAfterSuiteDemo.java[tags=user_guide]
 ----
+
+[[junit-platform-suite-engine-duplicate-test-execution]]
+==== Duplicate test execution
+
+Depending on the declared test selection, different suites may contain the same tests,
+potentially with different configurations.
+Moreover, tests in a suite are executed in addition to the tests executed by every other
+test engine. This can result in the same tests being executed twice.
+
+image::junit-platform-suite-engine-duplicate-test-execution-diagram.svg[role=text-center]
+
+This can be prevented by configuring your build tool to only include the
+`junit-platform-suite` engine. Or by using a naming pattern. For example, name all suites
+`*Suite` and all tests `*Test` and configure your build tool to only include the former.
+
+Alternatively, consider <<running-tests-tags, using tags>> to select specific groups of tests.

documentation/src/docs/asciidoc/user-guide/advanced-topics/launcher-api.adoc

@@ -14,6 +14,8 @@ tests. Moreover, third party test libraries – like Spock or Cucumber – can p
 JUnit Platform's launching infrastructure by providing a custom
 <<test-engines,TestEngine>>.
 
+image::launcher-api-diagram.svg[role=text-center]
+
 The launcher API is in the `{junit-platform-launcher}` module.
 
 An example consumer of the launcher API is the `{ConsoleLauncher}` in the

documentation/src/plantuml/junit-platform-suite-engine-diagram.puml

@@ -0,0 +1,32 @@
+@startuml
+object "IDEs"
+object "Build Tools"
+object "Console Launcher"
+
+object "JUnit Platform"
+object "Suite Test Engine"
+
+object "@Suite annotated class A"
+object "@Suite annotated class B"
+object "JUnit Platform (A)"
+object "JUnit Platform (B)"
+object "Jupiter Test Engine (A)"
+object "Jupiter Test Engine (B)"
+object "Tests in package A"
+object "Tests in package B"
+
+"IDEs" --> "JUnit Platform"
+"Build Tools" --> "JUnit Platform"
+"Console Launcher" --> "JUnit Platform" : requests discovery and execution
+"JUnit Platform" --> "Suite Test Engine": forwards request
+
+"Suite Test Engine" --> "@Suite annotated class A"
+"@Suite annotated class A" --> "JUnit Platform (A)" 
+"JUnit Platform (A)" --> "Jupiter Test Engine (A)"
+"Jupiter Test Engine (A)"  --> "Tests in package A"
+
+"Suite Test Engine" --> "@Suite annotated class B" : discovers and executes
+"@Suite annotated class B" --> "JUnit Platform (B)" : requests discovery and execution
+"JUnit Platform (B)" --> "Jupiter Test Engine (B)" : forwards request
+"Jupiter Test Engine (B)" --> "Tests in package B" : discovers and executes
+@enduml

documentation/src/plantuml/junit-platform-suite-engine-duplicate-test-execution-diagram.puml

@@ -0,0 +1,30 @@
+@startuml
+object "IDEs"
+object "Build Tools"
+object "Console Launcher"
+object "JUnit Platform"
+together {
+    object "Suite Test Engine"
+    object "Jupiter Test Engine"
+}
+object "@Suite annotated class"
+object "JUnit Platform (@Suite)"
+object "Jupiter Test Engine (@Suite)"
+together {
+    object "Example Test A"
+    object "Example Test A (@Suite)"
+}
+
+"IDEs" --> "JUnit Platform"
+"Build Tools" --> "JUnit Platform"
+"Console Launcher" --> "JUnit Platform" : requests discovery and execution
+
+"JUnit Platform" --> "Suite Test Engine"
+"Suite Test Engine" --> "@Suite annotated class" : discovers and executes
+"@Suite annotated class" --> "JUnit Platform (@Suite)" : requests discovery and execution
+"JUnit Platform (@Suite)" --> "Jupiter Test Engine (@Suite)" : forwards request
+"Jupiter Test Engine (@Suite)"  --> "Example Test A (@Suite)" : discovers and executes
+
+"JUnit Platform" --> "Jupiter Test Engine": forwards request
+"Jupiter Test Engine" --> "Example Test A" #line:red;line.bold;text:red : also discover and executes!
+@enduml

documentation/src/plantuml/launcher-api-diagram.puml

@@ -0,0 +1,24 @@
+@startuml
+object "IDEs"
+object "Build Tools"
+object "Console Launcher"
+object "JUnit Platform"
+object "Jupiter Test Engine"
+object "Cucumber Test Engine"
+object "Spock Test Engine"
+object "Test Classes"
+object "Feature Files"
+object "Specifications"
+
+"IDEs" --> "JUnit Platform"
+"Build Tools" --> "JUnit Platform"
+"Console Launcher" --> "JUnit Platform" : requests discovery and execution
+"JUnit Platform" --> "Jupiter Test Engine"
+"Jupiter Test Engine" --> "Test Classes"
+
+"JUnit Platform" --> "Cucumber Test Engine"
+"Cucumber Test Engine" --> "Feature Files"
+
+"JUnit Platform" --> "Spock Test Engine": forwards request
+"Spock Test Engine" --> "Specifications" : discovers and executes
+@enduml

junit-platform-suite-api/src/main/java/org/junit/platform/suite/api/Suite.java

@@ -26,7 +26,7 @@
  * {@code @Suite} marks a class as a test suite on the JUnit Platform.
  *
  * <p>Selector and filter annotations are used to control the contents of the
- * suite. Additionally configuration can be passed to the suite via the
+ * suite. Additionally, configuration can be passed to the suite via the
  * configuration annotations.
  *
  * <p>When the {@link IncludeClassNamePatterns @IncludeClassNamePatterns}
@@ -44,6 +44,17 @@
  * annotation disables the latter as a source of parameters so that only explicit
  * configuration parameters are taken into account.
  *
+ * <p>Note: Depending on the declared test selection, different suites may contain the
+ * same tests, potentially with different configurations.
+ * Moreover, tests in a suite are executed in addition to the tests executed by every
+ * other test engine. This can result in the same tests being executed twice and can be
+ * prevented by configuring your build tool to only include the
+ * {@code junit-platform-suite} engine. Or by using a naming pattern. For example, name
+ * all suites {@code *Suite} and all tests {@code *Test} and configure your build tool
+ * to only include the former.
+ *
+ * <p>Alternatively, consider using tags to select specific groups of tests.
+ *
  * @since 1.8
  * @see Select
  * @see SelectClasses