On big parallel machines you can interactively notice which ones do not enable parallel builds and take more than a few seconds to build (especially during bootstrap).
Let's enable parallel builds by default and disable parallelism of select packages that break.
Many build systems already enable parallel builds. To name a few:
--- a/pkgs/stdenv/generic/make-derivation.nix
+++ b/pkgs/stdenv/generic/make-derivation.nix
@@ -60,6 +60,8 @@ in
(stdenv.hostPlatform != stdenv.buildPlatform)
[ "build" "host" ]
+, enableParallelBuilding ? true
+
# TODO(@Ericson2314): Make unconditional / resolve #33599
# Check phase
, doCheck ? config.doCheckByDefault or false
@@ -310,7 +312,8 @@ else let
llvm-config = 'llvm-config-native'
'';
in [ "--cross-file=${crossFile}" ] ++ mesonFlags;
- } // lib.optionalAttrs (attrs.enableParallelBuilding or false) {
+ } // lib.optionalAttrs enableParallelBuilding {
+ inherit enableParallelBuilding;
enableParallelChecking = attrs.enableParallelChecking or true;
} // lib.optionalAttrs (hardeningDisable != [] || hardeningEnable != [] || stdenv.hostPlatform.isMusl) {
NIX_HARDENING_ENABLE = enabledHardeningOptions;
Example run of this PoC on set of packages from my NixOS desktop reveals at least ~20 failures (of 2100 packages), ~1% of failures. For all of nixpkgs if we extrapolate to 40000 packages that should be about 400 failures (very rough estimate). Does not sound too bad, but needs a hydra run to get most of them before the default flip.
Some examples (to be fixed / worked around).
I keep stumbling on packages that use sequential build for seemingly no reason:
On big parallel machines you can interactively notice which ones do not enable parallel builds and take more than a few seconds to build (especially during bootstrap).
Let's enable parallel builds by default and disable parallelism of select packages that break.
Many build systems already enable parallel builds. To name a few:
About 1300 packages set
enableParallelBuilding = true;.PoC implementation for
stdenv+makepackages:Example run of this PoC on set of packages from my NixOS desktop reveals at least ~20 failures (of 2100 packages), ~1% of failures. For all of
nixpkgsif we extrapolate to 40000 packages that should be about 400 failures (very rough estimate). Does not sound too bad, but needs a hydra run to get most of them before the default flip.Some examples (to be fixed / worked around).
xfce.garcon:fails occasionally regardless of build parallelism: autoconf: pull upstream fix for autoreconf race #144998