Re-run Maven Netty comparison using -Pfast flag, fix SBT compile al…

…l benchmark (#3918) Fixes #3871. I also moved the `./mvnw clean` into a separate command to avoid counting that time spent towards the Maven benchmark The differences between Maven and Mill are no longer quite as stark, but they are still very significant. The Mill benchmarks are about the same, as expected, so I skipped running the last few For the Gatling/SBT comparison, SBT `compile` does not compile test classes, while `test:compile` does. So Using SBT's `test:compile` is necessary for an apples-to-apples comparison with Mill's `__.compile` which does compile test classes
com-lihaoyi · Nov 7, 2024 · 2c2a779 · 2c2a779
1 parent 5e51651
commit 2c2a779
Show file tree

Hide file tree

Showing 6 changed files with 55 additions and 87 deletions.
diff --git a/docs/modules/ROOT/pages/comparisons/gradle.adoc b/docs/modules/ROOT/pages/comparisons/gradle.adoc
@@ -77,7 +77,7 @@ Mill's performance compares to Gradle:
 |===
 
 The column on the right shows the speedups of how much faster Mill is compared to the
-equivalent Gradle workflow. In most cases,  Mill is 5-10x faster than Gradle. Below, we
+equivalent Gradle workflow. In most cases,  Mill is 2-4x faster than Gradle. Below, we
 will go into more detail of each benchmark: how they were run, what they mean, and how
 we can explain the difference in performing the same task with the two different build tools.
 

diff --git a/docs/modules/ROOT/pages/comparisons/maven.adoc b/docs/modules/ROOT/pages/comparisons/maven.adoc
@@ -11,7 +11,7 @@ Compared to Maven:
 projects require minimal effort to get started, and larger Mill projects have a consistent
 structure building on these defaults.
 
-* **Mill automatically caches and parallelizes your build, offering 3-10x speedups**:
+* **Mill automatically caches and parallelizes your build, offering 4-10x speedups**:
 Not just the built-in tasks that Mill ships with, but also any custom tasks or modules.
 This maximizes the snappiness of your command-line build workflows to keep you productive,
 which especially matters in larger codebases where builds tend to get slow:
@@ -73,31 +73,30 @@ Mill's performance compares to Maven:
 |===
 | Benchmark | Maven | Mill | Speedup
 
-| <<Sequential Clean Compile All>> | 2m 31.12s | 0m 22.19s | 6.8x
-
-| <<Parallel Clean Compile All>> | 1m 16.45s | 0m 09.95s | 7.7x
-| <<Clean Compile Single-Module>> | 0m 19.62s | 0m 02.17s | 9.0x
-| <<Incremental Compile Single-Module>> | 0m 21.10s | 0m 00.54s | 39.1x
-| <<No-Op Compile Single-Module>> | 0m 17.34s | 0m 00.47s | 36.9x
+| <<Sequential Clean Compile All>> | 1m 38.80s | 23.14s | 4.3x
+| <<Parallel Clean Compile All>> | 0m 48.92s | 0m 08.79s | 5.6x
+| <<Clean Compile Single-Module>> | 0m 08.46s | 0m 01.94s | 9.0x
+| <<Incremental Compile Single-Module>> | 0m 06.82s | 0m 00.54s | 12.6x
+| <<No-Op Compile Single-Module>> | 0m 05.25s | 0m 00.47s  | 11.2x
 |===
 
 The column on the right shows the speedups of how much faster Mill is compared to the
-equivalent Maven workflow. In most cases,  Mill is 5-10x faster than Maven. Below, we
+equivalent Maven workflow. In most cases,  Mill is 4-10x faster than Maven. Below, we
 will go into more detail of each benchmark: how they were run, what they mean, and how
 we can explain the difference in performing the same task with the two different build tools.
 
 === Sequential Clean Compile All
 
 ```bash
-$ time ./mvnw -DskipTests  -Dcheckstyle.skip -Denforcer.skip=true clean install
-2m 42.96s
-2m 27.58s
-2m 31.12s
+$ ./mvnw clean; time ./mvnw -Pfast  -Dcheckstyle.skip -Denforcer.skip=true -DskipTests install
+1m 38.80s
+1m 36.14s
+1m 39.95s
 
 $ ./mill clean; time ./mill -j1 __.compile
-0m 29.14s
-0m 22.19s
-0m 20.79s
+0m 23.99s
+0m 23.14s
+0m 22.68s
 ```
 
 This benchmark exercises the simple "build everything from scratch" workflow, with all remote
@@ -113,29 +112,6 @@ to take 10-50 seconds without parallelism.
 The 20-30s taken by Mill seems about what you would expect for a codebase of this size,
 and the ~150s taken by Maven is far beyond what you would expect from simple Java compilation.
 
-==== Where is Maven spending its time?
-From eyeballing the logs, the added overhead comes from things like:
-
-_Downloading Metadata from Maven Central_
-
-```text
-Downloading from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/io/netty/netty-transport-native-unix-common/maven-metadata.xml
-Downloading from central: https://repo.maven.apache.org/maven2/io/netty/netty-transport-native-unix-common/maven-metadata.xml
-Downloaded from central: https://repo.maven.apache.org/maven2/io/netty/netty-transport-native-unix-common/maven-metadata.xml (4.3 kB at 391 kB/s)
-Downloaded from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/io/netty/netty-transport-native-unix-common/maven-metadata.xml (2.7 kB at 7.4 kB/s)
-```
-
-_Comparing Jars_
-
-```text
-Comparing [io.netty:netty-transport-sctp:jar:4.1.112.Final] against [io.netty:netty-transport-sctp:jar:4.1.113.Final-SNAPSHOT] (including their transitive dependencies).
-```
-
-In general, Maven spends much of time working with Jar files: packing them, unpacking them,
-comparing them, etc. None of this is strictly necessary for compiling Java source files to
-classfiles! But if they are not necessary, then why is Maven doing it? It turns out the
-reason comes down to the difference of `mvn compile` vs `mvn install`
-
 ==== Maven Compile vs Install
 
 In general, the reason we have to use `./mvnw install` rather than `./mvnw compile` is that
@@ -166,16 +142,6 @@ overhead of doing so is just a few seconds, far less than the two entire minutes
 Maven's overhead adds to the clean build:
 
 ```bash
-$ time ./mvnw -DskipTests  -Dcheckstyle.skip -Denforcer.skip=true clean install
-2m 42.96s
-2m 27.58s
-2m 31.12s
-
-$ ./mill clean; time ./mill -j1 __.compile
-0m 29.14s
-0m 22.19s
-0m 20.79s
-
 $ ./mill clean; time ./mill -j1 __.jar
 0m 32.58s
 0m 24.90s
@@ -191,15 +157,15 @@ whereas Mill directly uses the classfiles generated on disk to bypass all that w
 === Parallel Clean Compile All
 
 ```bash
-$ time ./mvnw -T 10 -DskipTests  -Dcheckstyle.skip -Denforcer.skip=true clean install
-1m 19.58s
-1m 16.34s
-1m 16.45s
+$ ./mvnw clean; time ./mvnw -T 10 -Pfast -DskipTests  -Dcheckstyle.skip -Denforcer.skip=true install
+0m 48.92s
+0m 48.41s
+0m 49.50s
 
 $ ./mill clean; time ./mill __.compile
-0m 14.80s
-0m 09.95s
-0m 08.83s
+0m 09.07s
+0m 08.79s
+0m 07.93s
 ```
 
 This example compares Maven v.s. Mill, when performing the clean build on 10 threads.
@@ -214,15 +180,15 @@ when performing a clean build of the Netty repository.
 === Clean Compile Single-Module
 
 ```bash
-$ time ./mvnw -pl common -DskipTests  -Dcheckstyle.skip -Denforcer.skip=true clean install
-0m 19.62s
-0m 20.52s
-0m 19.50s
+$ ./mvnw clean; time ./mvnw -pl common -Pfast -DskipTests  -Dcheckstyle.skip -Denforcer.skip=true install
+0m 08.46s
+0m 08.90s
+0m 08.30s
 
 $ ./mill clean common; time ./mill common.test.compile
-0m 04.94s
-0m 02.17s
-0m 01.95s
+0m 01.99s
+0m 01.81s
+0m 01.94s
 ```
 
 This exercise limits the comparison to compiling a single module, in this case `common/`.
@@ -242,13 +208,13 @@ at what you would expect, whereas Maven's has a significant overhead.
 
 ```bash
 $ echo "" >> common/src/main/java/io/netty/util/AbstractConstant.java
-$ time ./mvnw -pl common -DskipTests  -Dcheckstyle.skip -Denforcer.skip=true install
+$ time ./mvnw -pl common -Pfast -DskipTests  -Dcheckstyle.skip -Denforcer.skip=true install
 Compiling 174 source files to /Users/lihaoyi/Github/netty/common/target/classes
 Compiling 60 source files to /Users/lihaoyi/Github/netty/common/target/test-classes
 
-0m 21.10s
-0m 19.64s
-0:21:29
+0m 06.89s
+0m 06.34s
+0m 06.82s
 
 
 $ echo "" >> common/src/main/java/io/netty/util/AbstractConstant.java
@@ -341,10 +307,10 @@ the same thing in Maven
 === No-Op Compile Single-Module
 
 ```bash
-$ time ./mvnw -pl common -DskipTests  -Dcheckstyle.skip -Denforcer.skip=true install
-0m 16.34s
-0m 17.34s
-0m 18.28s
+$ time ./mvnw -pl common -Pfast -DskipTests  -Dcheckstyle.skip -Denforcer.skip=true install
+0m 05.08s
+0m 05.25s
+0m 05.26s
 
 $ time ./mill common.test.compile
 0m 00.49s

diff --git a/docs/modules/ROOT/pages/comparisons/sbt.adoc b/docs/modules/ROOT/pages/comparisons/sbt.adoc
@@ -69,7 +69,7 @@ real-world project compares to using SBT.
 |===
 | Benchmark | Cold SBT | Hot SBT (rounded) | Mill
 
-| <<Parallel Clean Compile All>> | 28.7s | ≈ 12s | 10.4s
+| <<Parallel Clean Compile All>> | 34.28s | ≈ 14s | 10.4s
 | <<Clean Compile Single-Module>> | 10.1s | ≈ 1s | 0.96s
 | <<Incremental Compile Single-Module>> | 6.2s | ≈ 0s | 0.48s
 | <<No-Op Compile Single-Module>> | 4.2s | ≈ 0s | 0.40s
@@ -82,9 +82,10 @@ distinction, and can only be run directly from the command line. The `Hot SBT` m
 reports timings to the nearest second, so that is the number used in this comparison.
 
 The Mill build benchmarks for Gatling is generally much snappier than the `Cold SBT` benchmark,
-and comparable to that `Hot SBT` benchmark. Mill does not have the same _Cold vs Hot_
-distinction that SBT has, as Mill is always run "cold" from the command line and keeps the
-process around to provide "hot" performance automatically.
+and comparable to that `Hot SBT` benchmark. Mill is marginally faster in the
+`Parallel Clean Compile All` benchmark (10s vs 14s), but more importantly does not have the same
+_Cold vs Hot_ distinction that SBT has: as Mill is always run "cold" from the command line and
+keeps the process around to provide "hot" performance automatically.
 
 For the benchmarks above, each provided number is the median wall time of three consecutive runs
 on my M1 Macbook Pro. While ad-hoc, these benchmarks are enough to give you a flavor of how
@@ -95,17 +96,17 @@ since Mill doesn't bundle them and instead expects them to be invoked separately
 === Parallel Clean Compile All
 
 ```bash
-$ sbt clean; time sbt compile
-28.7s
-27.6s
-31.2s
+$ sbt clean; time sbt test:compile
+34.28s
+32.84s
+34.55s
 
 $ sbt
 
-sbt> clean; compile
-12s
-12s
-10s
+sbt> clean; test:compile
+15s
+13s
+14s
 
 $ ./mill clean; time ./mill -j 10 __.compile
 10.7s

diff --git a/docs/modules/ROOT/pages/comparisons/unique.adoc b/docs/modules/ROOT/pages/comparisons/unique.adoc
@@ -389,8 +389,9 @@ ecosystems rely on tools like Maven or Gradle to build their code, and I believe
 provides a better alternative. Even today, there are already many advantages of
 using Mill over the incumbent build tools:
 
-1. Mill today runs the equivalent local workflows 2-10x faster than Maven
-   or Gradle, with automatic parallelization and caching for every part of your build
+1. Mill today runs the equivalent local workflows xref:comparisons/maven.adoc[4-10x faster than Maven]
+   and xref:comparisons/gradle.adoc[2-4x faster than Gradle], with automatic parallelization and caching for
+   every part of your build
 
 2. Mill today provides better ease of use than Maven or Gradle, with IDE support for
    navigating your build graph and visualizing what your build is doing

diff --git a/docs/modules/ROOT/pages/index.adoc b/docs/modules/ROOT/pages/index.adoc
@@ -33,7 +33,7 @@ digraph G {
 Mill is a fast, scalable, multi-language build tool that supports Java, Scala,
 and Kotlin:
 
-* Mill can build the same Java codebase xref:comparisons/maven.adoc[5-10x faster than Maven],
+* Mill can build the same Java codebase xref:comparisons/maven.adoc[4-10x faster than Maven],
 or xref:comparisons/gradle.adoc[2-4x faster than Gradle]
 
 * Mill's typed config language and immutable xref:depth/design-principles.adoc[task graph]

diff --git a/docs/modules/ROOT/pages/why-mill.adoc b/docs/modules/ROOT/pages/why-mill.adoc
@@ -57,7 +57,7 @@ We will discuss each one in turn.
 
 ### Maven
 
-Overall across our benchmarks, Mill is 5-10x faster than Maven for clean compiles,
+Overall across our benchmarks, Mill is 4-10x faster than Maven for clean compiles,
 both parallel and sequential, and for many modules or for a single module:
 
 |===