Spurious timeout in the asmjs builder

[alexcrichton]

The asm.js builder has been creeping up in time it takes to run ever since it was originally added, and clocking in at around 2.5 hrs now this builder is ripe for timing out on occasion. The suspected cause here is the fact that the run-pass test suite takes >1hr to run and the suspected cause there is that we're effectively compiling every test with LTO which can take quite some time.

This issue is intended to track possible solutions to bring the builder back under control. Today in a discussion a few possible avenues arose:

• Shard the asm.js builder into two builders, each running half the run-pass tests
• Stop running the full test suite for asm.js altogether, we seem to regress this very rarely
• Stop optimizing tests, debug mode may be much faster.

cc @kennytm

[kennytm]

For reference, the rough timing breakdown according to the log in the recent successful builds are:

(Unit = seconds)

	69261	69207	69163	69145
make-prepare	51.80	51.39	44.70	51.33
stage0-test	11.46	11.49	9.95	11.95
stage0-rustc	837.85	894.54	736.97	891.64
llvm	334.30	331.78	299.45	320.13
stage0-trans-llvm	61.98	66.88	56.99	65.78
llvm-emscripten	155.00	154.79	148.31	157.17
stage0-trans-emscripten	55.33	56.83	49.81	57.00
stage1-std	91.74	90.80	74.97	88.20
stage1-test	17.12	17.73	14.51	17.77
stage1-rustc	1306.78	1379.93	1115.28	1350.69
stage1-trans-llvm	93.22	99.94	81.47	96.71
stage1-trans-emscripten	81.14	86.71	73.54	84.38
stage2-std	58.86	57.58	47.54	56.76
stage2-test	12.49	12.86	10.79	12.88
stage0-compiletest	45.64	47.33	39.31	46.43
test/ui	237.82	219.62	197.82	218.42
test/run-pass	3697.79	3738.31	3276.24	3717.92
test/compile-fail	114.26	110.62	102.75	110.47
test/parse-fail	3.86	3.88	3.55	3.80
test/run-fail	167.54	172.85	148.10	170.07
test/run-pass-valgrind	14.39	14.91	12.64	14.69
test/mir-opt	67.39	69.09	59.82	69.13
test/codegen	3.23	3.24	2.88	3.23
test/codegen-units	18.60	19.27	16.86	19.12
test/incremental	119.36	121.20	109.48	121.45
test/debuginfo	153.45	167.77	135.39	162.18

We see that finishing the test/run-pass tests take roughly 1 hour.

We could compare with those of

android...

make-prepare                	  54.11
stage0-test                 	  12.21
stage0-rustc                	 772.94
llvm                        	 295.78
stage0-trans-llvm           	  56.48
stage1-std                  	  81.96
stage1-test                 	  14.97
stage1-rustc                	1152.85
stage1-trans-llvm           	  87.40
stage2-std                  	  63.00
stage2-test                 	  14.74
stage2-remote-test-server   	   2.61
stage0-remote-test-client   	   2.42
stage0-compiletest          	  45.28
test/ui                     	 101.67
test/run-pass               	 660.46
test/compile-fail           	 135.16
test/parse-fail             	   5.25
test/run-fail               	  19.38
test/run-pass-valgrind      	   3.89
test/mir-opt                	  20.29
test/codegen                	   3.49
test/codegen-units          	   2.16
test/incremental            	  17.71
test/debuginfo              	 111.61

x86_64...

make-prepare                	  52.89
stage0-tidy                 	  54.54
bootstrap                   	  11.89
stage0-rustc                	 883.23
llvm                        	 333.87
stage0-trans-llvm           	  64.13
stage1-std                  	 145.13
stage1-test                 	  17.89
stage1-rustc                	1369.94
stage1-trans-llvm           	  97.95
stage0-compiletest          	  47.20
test/ui                     	 101.08
test/run-pass               	 614.08
test/compile-fail           	 135.41
test/parse-fail             	   5.07
test/run-fail               	  14.34
test/run-pass-valgrind      	   3.50
test/mir-opt                	  20.33
test/codegen                	   4.22
test/codegen-units          	   2.40
test/incremental            	  14.98
test/debuginfo              	  13.54
test/ui-fulldeps            	  39.77
test/run-pass-fulldeps      	 330.72
test/run-fail-fulldeps      	   4.92
test/compile-fail-fulldeps  	  86.63
test/incremental-fulldeps   	   3.24
stage2-rustdoc              	 160.96
test/rustdoc                	 125.74
test/run-make               	 311.71
...

wasm32...

make-prepare                	  50.83
stage0-test                 	  11.30
stage0-rustc                	 841.60
llvm                        	 307.38
stage0-trans-llvm           	  61.75
lld                      	  19.76
stage1-std                  	  88.76
stage1-test                 	  16.50
stage1-rustc                	1263.30
stage1-trans-llvm           	  89.47
stage2-std                  	  54.54
stage2-test                 	  13.24
stage0-compiletest          	  45.60
test/ui                     	  90.01
test/run-pass               	 865.87
test/compile-fail           	 122.83
test/parse-fail             	   4.91
test/mir-opt                	  19.47
test/codegen-units          	   1.97
stage2-rustdoc              	 150.73
stage2-test-core            	  82.39
...

and we see that the run time of the run-pass suite typically take 10 to 15 minutes to complete, so the running time of asmjs's run-pass is 5 times longer than other builders.

[alexcrichton]

FWIW pre-LLD the wasm32-unknown run-pass test suite took 2700s, wheras after LLD it took 850s, a transition from 45m to ~15m. That's I think definitely confirming that Emscripten is so slow because of how it's linker works, it's just LLVM's LTO linker.

Once my machine frees up from timing other things I'll run some tests to see how much optimizing tests costs us here.

[alexcrichton]

Disabling optimizations in tests unfortunately broke a bunch of tests

failures:
    [run-pass] run-pass/deep.rs
    [run-pass] run-pass/intrinsics-integer.rs
    [run-pass] run-pass/intrinsics-math.rs
    [run-pass] run-pass/issue-32947.rs
    [run-pass] run-pass/issue-38074.rs
    [run-pass] run-pass/issue-39720.rs
    [run-pass] run-pass/mir_heavy_promoted.rs
    [run-pass] run-pass/next-power-of-two-overflow-debug.rs
    [run-pass] run-pass/next-power-of-two-overflow-ndebug.rs
    [run-pass] run-pass/packed-struct-borrow-element.rs
    [run-pass] run-pass/simd-intrinsic-generic-arithmetic.rs
    [run-pass] run-pass/simd-intrinsic-generic-comparison.rs
    [run-pass] run-pass/simd-intrinsic-generic-elements.rs

and shaved 5 minutes off the total build time locally. I'm not sure if that'll actually translate to wins upstream...

[alexcrichton]

Looks like with #49095 our builds have stabilized at from 2.5 hours to 2 hours on the asmjs builder. That should hopefully buy us at least a few months!