Support for parallel loading of subprojects via Isolated Projects

[jjohannes]

Is your feature request related to a problem? Please describe.

Large projects with hundreds of subprojects take minutes on import/refresh.

Describe the solution you'd like

This is a general problem of Gradle's configuration phase. Traditionally, all (sub)projects are configured in sequence. Hence, the more projects you have, the longer the configuration takes.

To improve this situation, Gradle is working on parallel configuration of projects for some time already. The feature that should finally solve it is now called Isolated Projects. The main change is that projects are no longer allowed to access each others state during configuration. With this, the order in which projects are configured no longer influences the result.

To use the feature, you first need to make sure your own build configuration and the third-party plugins you use do no longer utilise cross-project configuration constructs. Then you can turn it on in the project's gradle.properties via:

org.gradle.unsafe.isolated-projects=true

This works in my projects when building from command line or importing in recent IntelliJ versions. Importing in VS Code fails though, because the configuration code added via init script to build the IDE Model is using APIs that access cross-project state.

You get these errors (can be reproduced with any project):

- Initialization script '...': line 8:  Project ':' cannot access 'Project.apply' functionality on subprojects via 'allprojects'
 
- Plugin class 'com.microsoft.gradle.GradlePlugin': Project ':' cannot access 'Project.buildscript' functionality on child projects
 
- Plugin class 'com.microsoft.gradle.GradlePlugin': Project ':' cannot access 'Project.configurations' functionality on child projects

I did some initial analysis and I think the two things that need to be changed is:

• No longer use allprojects { ... } in the init script here.
  Instead, use the new isolated API gradle.lifecycle.beforeProject { ... }
• No longer access child projects through their parent here to later access their buildscript and configurations containers inside GradleProjectModelBuilder. How exactly this should be done would need to be researched. But it must be possible to avoid this.

Describe alternatives you've considered

No alternatives.

[github-actions]

Hi @jjohannes, I'm an AI Support assistant here to help with your issue. While the team reviews your request, I wanted to provide some possible tips and documentation that might help you in the meantime.

Suggestions

• Update the init script in PluginUtils.java:
  • Replace allprojects { … } with the isolated API hook:

  gradle.lifecycle.beforeProject { project ->
    // your project-specific configuration
  }

  • See Gradle's docs on isolated projects: https://docs.gradle.org/current/userguide/isolated_projects.html

• Refactor GradleProjectModelBuilder:
  • Avoid iterating child projects via parent (Project.getChildProjects()).
  • Instead, register a model builder per project and collect buildscript and configurations within that project’s context.
  • Review Gradle’s Tooling API samples for project-scoped model builders: https://docs.gradle.org/current/javadoc/org/gradle/tooling/ProjectConnection.html

References

• Gradle Isolated Projects user guide (enables parallel configuration):
  https://docs.gradle.org/current/userguide/isolated_projects.html

Other references with low confidence

• Issue Parallel runnning of gradle task #1045: requests parallel execution of tasks rather than parallel project configuration. Not directly applicable.
  Parallel runnning of gradle task #1045

The team will respond to your issue shortly. I hope these suggestions are helpful in the meantime. If this comment helped you, please give it a 👍. If the suggestion was not helpful or incorrect, please give it a 👎. Your feedback helps us improve!

[jjohannes]

I got this input from @alllex for how to solve the issue in GradleProjectModelBuilder:

The solution is to split the “main” model builder and the per-project model builders. The aggregation is then done by running nested build actions via BuildController.run — it’s IP safe and can even run the nested actions in parallel.

[chagong]

@jjohannes do you have an isolated projects repo for testing?

[jjohannes]

@chagong this can be tested with any project that has org.gradle.unsafe.isolated-projects=true and at least one subproject.

You can use this repo if you like:

https://github.com/jjohannes/gradle-project-setup-howto/tree/isolated-projects

Note that you need to switch to the isolated-projects branch as main currently uses two plugins that are also not yet compatible.