potrf_dtd_mpi and gemm_dtd_mpi produce wrong results with 1 GPU

[abouteiller]

Describe the bug

MPI POTRF DTD with 1 GPU produces wrong results. The 1-node variant is correct.

Not clear ATM if issue is in the DTD testers for GEMM and POTRF, or in PaRSEC.

Important note

After #114 this error will not manifest in normal ctest/CI (because test is forced to run on CPU only), but can still be reproduced by hand. The fix PR should add a specific test for DTD+GPU to explicitly test for this case.

Buggy output

❯ SLURM_TIMELIMIT=1 PARSEC_MCA_device_cuda_enabled=0 PARSEC_MCA_device_cuda_memory_use=10 OMPI_MCA_rmaps_base_oversubscribe=true salloc -wleconte  ctest  -R spotrf_dtd_mpi --verbose
salloc: Granted job allocation 5295
UpdateCTestConfiguration  from :/home/bouteill/parsec/dplasma/build.cuda/DartConfiguration.tcl
Parse Config file:/home/bouteill/parsec/dplasma/build.cuda/DartConfiguration.tcl
UpdateCTestConfiguration  from :/home/bouteill/parsec/dplasma/build.cuda/DartConfiguration.tcl
Parse Config file:/home/bouteill/parsec/dplasma/build.cuda/DartConfiguration.tcl
Test project /home/bouteill/parsec/dplasma/build.cuda
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 287
    Start 287: dplasma_spotrf_dtd_mpi

287: Test command: /apps/spacks/2023-08-14/opt/spack/linux-rocky9-x86_64/gcc-11.3.1/openmpi-4.1.5-2rgaqk2wseegpmbdbbygvwrljccjaqsk/bin/mpiexec "-n" "4" "./testing_spotrf_dtd" "-N" "378" "-t" "19" "-x" "-v=5"
287: Working Directory: /home/bouteill/parsec/dplasma/build.cuda/tests
287: Test timeout computed to be: 1500
287: i@00003 CPU Device: Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz
287:    Parsec Streams     : 40
287:    Frequency (GHz)    : 2.20
287:    Peak Tflop/s       : 0.3520 fp64,       0.7040 fp32
287: i@00001 CPU Device: Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz
287:    Parsec Streams     : 40
287:    Frequency (GHz)    : 2.20
287:    Peak Tflop/s       : 0.3520 fp64,       0.7040 fp32
287: W@00000 /!\ DEBUG LEVEL WILL PROBABLY REDUCE THE PERFORMANCE OF THIS RUN /!\.
287: i@00000 CPU Device: Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz
287:    Parsec Streams     : 40
287:    Frequency (GHz)    : 2.20
287:    Peak Tflop/s       : 0.3520 fp64,       0.7040 fp32
287: i@00002 CPU Device: Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz
287:    Parsec Streams     : 40
287:    Frequency (GHz)    : 2.20
287:    Peak Tflop/s       : 0.3520 fp64,       0.7040 fp32
287: [   3] TIME(s)      0.04076 : PaRSEC initialized
287: [   1] TIME(s)      0.04855 : PaRSEC initialized
287: [   2] TIME(s)      0.04899 : PaRSEC initialized
287: #+++++ cores detected       : 40
287: #+++++ nodes x cores + gpu  : 4 x 40 + 0 (160+0)
287: #+++++ thread mode          : THREAD_SERIALIZED
287: #+++++ P x Q                : 4 x 1 (4/4)
287: #+++++ M x N x K|NRHS       : 378 x 378 x 1
287: #+++++ LDA                  : 378
287: #+++++ MB x NB              : 19 x 19
287: [   0] TIME(s)      0.06141 : PaRSEC initialized
287: W@00000 /!\ PERFORMANCE MIGHT BE REDUCED /!\: Multiple PaRSEC processes on the same node may share the same physical core(s);
287:    This is often unintentional, and will perform poorly.
287:    Note that in managed environments (e.g., ALPS, jsrun), the launcher may set `cgroups`
287:    and hide the real binding from PaRSEC; if you verified that the binding is correct,
287:    this message can be silenced using the MCA argument `runtime_warn_slow_binding`.
287: +++ warm up ... Done
287: +++ warm up ... Done
287: +++ warm up ... Done
287: +++ Generate matrices ... Done
287: +++ warm up ... Done
287: +++ Generate matrices ... Done
287: +++ Generate matrices ... Done
287: +++ Generate matrices ... Done
287: [****] TIME(s)      0.13299 :      PxQ=   4 1   NB=   19 N=     378 :       0.135908 gflops
287: ============
287: Checking the Cholesky factorization
287: -- ||A||_oo = 4.826267e+02, ||L'L-A||_oo = 1.256157e-04
287: -- ||L'L-A||_oo/(||A||_oo.N.eps) = 1.155209e-02
287: -- Factorization is CORRECT !
287: ============
287: Checking the Residual of the solution
287: -- ||A||_oo = 4.826267e+02, ||X||_oo = 1.350270e-03, ||B||_oo= 5.000000e-01, ||A X - B||_oo = 4.668881e-07
287: -- ||Ax-B||_oo/((||A||_oo||x||_oo+||B||_oo).N.eps) = 1.799328e-02
287: -- Solution is CORRECT !
287: +----------------------------------------------------------------------------------------------------------------------------+
287: |         |                    |                       Data In                              |         Data Out               |
287: |Rank   0 |  # KERNEL |    %   |  Required  |   Transfered H2D(%)   |   Transfered D2D(%)   |  Required  |   Transfered(%)   |
287: |---------|-----------|--------|------------|-----------------------|-----------------------|------------|-------------------|
287: |  Dev  0 |      2938 | 100.00 |     0.00 B |       0.00 B(  nan)   |       0.00 B(  nan)   |     0.00 B |     0.00 B(  nan) | cpu-cores
287: |---------|-----------|--------|------------|-----------------------|-----------------------|------------|-------------------|
287: |All Devs |      2938 | 100.00 |     0.00 B |       0.00 B(  nan)   |       0.00 B(  nan)   |     0.00 B |     0.00 B(  nan) |
287: +----------------------------------------------------------------------------------------------------------------------------+
287:
287: Full transfer matrix:
287: dst\src          0
287:    0        -
287: +----------------------------------------------------------------------------------------------------------------------------+
287: |         |                    |                       Data In                              |         Data Out               |
287: |Rank   2 |  # KERNEL |    %   |  Required  |   Transfered H2D(%)   |   Transfered D2D(%)   |  Required  |   Transfered(%)   |
287: |---------|-----------|--------|------------|-----------------------|-----------------------|------------|-------------------|
287: |  Dev  0 |      2978 | 100.00 |     0.00 B |       0.00 B(  nan)   |       0.00 B(  nan)   |     0.00 B |     0.00 B(  nan) | cpu-cores
287: |---------|-----------|--------|------------|-----------------------|-----------------------|------------|-------------------|
287: |All Devs |      2978 | 100.00 |     0.00 B |       0.00 B(  nan)   |       0.00 B(  nan)   |     0.00 B |     0.00 B(  nan) |
287: +----------------------------------------------------------------------------------------------------------------------------+
287:
287: Full transfer matrix:
287: dst\src          0
287:    0        -
287: +----------------------------------------------------------------------------------------------------------------------------+
287: |         |                    |                       Data In                              |         Data Out               |
287: |Rank   1 |  # KERNEL |    %   |  Required  |   Transfered H2D(%)   |   Transfered D2D(%)   |  Required  |   Transfered(%)   |
287: |---------|-----------|--------|------------|-----------------------|-----------------------|------------|-------------------|
287: |  Dev  0 |      2963 | 100.00 |     0.00 B |       0.00 B(  nan)   |       0.00 B(  nan)   |     0.00 B |     0.00 B(  nan) | cpu-cores
287: |---------|-----------|--------|------------|-----------------------|-----------------------|------------|-------------------|
287: |All Devs |      2963 | 100.00 |     0.00 B |       0.00 B(  nan)   |       0.00 B(  nan)   |     0.00 B |     0.00 B(  nan) |
287: +----------------------------------------------------------------------------------------------------------------------------+
287:
287: Full transfer matrix:
287: dst\src          0
287:    0        -
287: +----------------------------------------------------------------------------------------------------------------------------+
287: |         |                    |                       Data In                              |         Data Out               |
287: |Rank   3 |  # KERNEL |    %   |  Required  |   Transfered H2D(%)   |   Transfered D2D(%)   |  Required  |   Transfered(%)   |
287: |---------|-----------|--------|------------|-----------------------|-----------------------|------------|-------------------|
287: |  Dev  0 |      2963 | 100.00 |     0.00 B |       0.00 B(  nan)   |       0.00 B(  nan)   |     0.00 B |     0.00 B(  nan) | cpu-cores
287: |---------|-----------|--------|------------|-----------------------|-----------------------|------------|-------------------|
287: |All Devs |      2963 | 100.00 |     0.00 B |       0.00 B(  nan)   |       0.00 B(  nan)   |     0.00 B |     0.00 B(  nan) |
287: +----------------------------------------------------------------------------------------------------------------------------+
287:
287: Full transfer matrix:
287: dst\src          0
287:    0        -
1/1 Test #287: dplasma_spotrf_dtd_mpi ...........   Passed    1.79 sec

The following tests passed:
        dplasma_spotrf_dtd_mpi

100% tests passed, 0 tests failed out of 1

Label Time Summary:
dplasma    =   1.79 sec*proc (1 test)
mpi        =   1.79 sec*proc (1 test)

Total Test time (real) =   1.81 sec
salloc: Relinquishing job allocation 5295

     ~/parsec/dplasma/build.cuda     cleanup/ngpus-match-g *1 !2 ?13 ··············································· bouteill@methane  16:32:00   ─╮
❯ SLURM_TIMELIMIT=1 PARSEC_MCA_device_cuda_enabled=1 PARSEC_MCA_device_cuda_memory_use=10 OMPI_MCA_rmaps_base_oversubscribe=true salloc -wleconte  ctest  -R spotrf_dtd_mpi --verbose
salloc: Granted job allocation 5296
UpdateCTestConfiguration  from :/home/bouteill/parsec/dplasma/build.cuda/DartConfiguration.tcl
Parse Config file:/home/bouteill/parsec/dplasma/build.cuda/DartConfiguration.tcl
UpdateCTestConfiguration  from :/home/bouteill/parsec/dplasma/build.cuda/DartConfiguration.tcl
Parse Config file:/home/bouteill/parsec/dplasma/build.cuda/DartConfiguration.tcl
Test project /home/bouteill/parsec/dplasma/build.cuda
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 287
    Start 287: dplasma_spotrf_dtd_mpi

287: Test command: /apps/spacks/2023-08-14/opt/spack/linux-rocky9-x86_64/gcc-11.3.1/openmpi-4.1.5-2rgaqk2wseegpmbdbbygvwrljccjaqsk/bin/mpiexec "-n" "4" "./testing_spotrf_dtd" "-N" "378" "-t" "19" "-x" "-v=5"
287: Working Directory: /home/bouteill/parsec/dplasma/build.cuda/tests
287: Test timeout computed to be: 1500
287: W@00000 /!\ DEBUG LEVEL WILL PROBABLY REDUCE THE PERFORMANCE OF THIS RUN /!\.
287: i@00002 GPU Device cuda(0) : Tesla V100-SXM2-32GB [capability 7.0]
287:    Location (PCI Bus/Device/Domain): 6:0.0
287:    SM                 : 80
287:    Frequency (GHz)    : 1.530000
287:    peak Tflop/s       : 7.83 fp64, 15.67 fp32,     125.34 tf32,    31.33 fp16
287:    Peak Mem Bw (GB/s) : 898.05 [Clock Rate (Ghz) 0.88 | Bus Width (bits) 4096]
287:    concurrency        : yes
287:    computeMode        : 0
287: i@00002 CPU Device: Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz
287:    Parsec Streams     : 40
287:    Frequency (GHz)    : 2.20
287:    Peak Tflop/s       : 0.3520 fp64,       0.7040 fp32
287: i@00001 GPU Device cuda(0) : Tesla V100-SXM2-32GB [capability 7.0]
287:    Location (PCI Bus/Device/Domain): 6:0.0
287:    SM                 : 80
287:    Frequency (GHz)    : 1.530000
287:    peak Tflop/s       : 7.83 fp64, 15.67 fp32,     125.34 tf32,    31.33 fp16
287:    Peak Mem Bw (GB/s) : 898.05 [Clock Rate (Ghz) 0.88 | Bus Width (bits) 4096]
287:    concurrency        : yes
287:    computeMode        : 0
287: i@00000 GPU Device cuda(0) : Tesla V100-SXM2-32GB [capability 7.0]
287:    Location (PCI Bus/Device/Domain): 6:0.0
287:    SM                 : 80
287:    Frequency (GHz)    : 1.530000
287:    peak Tflop/s       : 7.83 fp64, 15.67 fp32,     125.34 tf32,    31.33 fp16
287:    Peak Mem Bw (GB/s) : 898.05 [Clock Rate (Ghz) 0.88 | Bus Width (bits) 4096]
287:    concurrency        : yes
287:    computeMode        : 0
287: i@00000 CPU Device: Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz
287:    Parsec Streams     : 40
287:    Frequency (GHz)    : 2.20
287:    Peak Tflop/s       : 0.3520 fp64,       0.7040 fp32
287: i@00001 CPU Device: Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz
287:    Parsec Streams     : 40
287:    Frequency (GHz)    : 2.20
287:    Peak Tflop/s       : 0.3520 fp64,       0.7040 fp32
287: i@00003 GPU Device cuda(0) : Tesla V100-SXM2-32GB [capability 7.0]
287:    Location (PCI Bus/Device/Domain): 6:0.0
287:    SM                 : 80
287:    Frequency (GHz)    : 1.530000
287:    peak Tflop/s       : 7.83 fp64, 15.67 fp32,     125.34 tf32,    31.33 fp16
287:    Peak Mem Bw (GB/s) : 898.05 [Clock Rate (Ghz) 0.88 | Bus Width (bits) 4096]
287:    concurrency        : yes
287:    computeMode        : 0
287: i@00003 CPU Device: Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz
287:    Parsec Streams     : 40
287:    Frequency (GHz)    : 2.20
287:    Peak Tflop/s       : 0.3520 fp64,       0.7040 fp32
287: [   2] TIME(s)      0.80787 : PaRSEC initialized
287: [   1] TIME(s)      0.81451 : PaRSEC initialized
287: #+++++ cores detected       : 40
287: #+++++ nodes x cores + gpu  : 4 x 40 + 1 (160+4)
287: #+++++ thread mode          : THREAD_SERIALIZED
287: #+++++ P x Q                : 4 x 1 (4/4)
287: #+++++ M x N x K|NRHS       : 378 x 378 x 1
287: #+++++ LDA                  : 378
287: #+++++ MB x NB              : 19 x 19
287: [   0] TIME(s)      0.81472 : PaRSEC initialized
287: [   3] TIME(s)      0.82741 : PaRSEC initialized
287: W@00000 /!\ PERFORMANCE MIGHT BE REDUCED /!\: Multiple PaRSEC processes on the same node may share the same physical core(s);
287:    This is often unintentional, and will perform poorly.
287:    Note that in managed environments (e.g., ALPS, jsrun), the launcher may set `cgroups`
287:    and hide the real binding from PaRSEC; if you verified that the binding is correct,
287:    this message can be silenced using the MCA argument `runtime_warn_slow_binding`.
287: +++ warm up ... Done
287: +++ Generate matrices ... Done
287: +++ warm up ... Done
287: +++ warm up ... Done
287: +++ warm up ... Done
287: +++ Generate matrices ... Done
287: +++ Generate matrices ... Done
287: +++ Generate matrices ... Done
287: [****] TIME(s)      0.30603 :      PxQ=   4 1   NB=   19 N=     378 :       0.059063 gflops
287: ============
287: Checking the Cholesky factorization
287: -- ||A||_oo = 4.826267e+02, ||L'L-A||_oo = 8.671714e-01
287: -- ||L'L-A||_oo/(||A||_oo.N.eps) = 7.974834e+01
287: -- Factorization is suspicious !
287: ============
287: Checking the Residual of the solution
287: -- ||A||_oo = 4.826267e+02, ||X||_oo = 1.349938e-03, ||B||_oo= 5.000000e-01, ||A X - B||_oo = 1.503229e-04
287: -- ||Ax-B||_oo/((||A||_oo||x||_oo+||B||_oo).N.eps) = 5.794063e+00
287: -- Solution is CORRECT !
287: +----------------------------------------------------------------------------------------------------------------------------+
287: |         |                    |                       Data In                              |         Data Out               |
287: |Rank   0 |  # KERNEL |    %   |  Required  |   Transfered H2D(%)   |   Transfered D2D(%)   |  Required  |   Transfered(%)   |
287: |---------|-----------|--------|------------|-----------------------|-----------------------|------------|-------------------|
287: |  Dev  0 |      2463 |  83.83 |     0.00 B |       0.00 B(  nan)   |       0.00 B(  nan)   |     0.00 B |     0.00 B(  nan) | cpu-cores
287: |  Dev  1 |       475 |  16.17 |     1.82MB |     274.98KB(14.77)   |       0.00 B( 0.00)   |   669.82KB |    97.30KB(14.53) | cuda(0)
287: |---------|-----------|--------|------------|-----------------------|-----------------------|------------|-------------------|
287: |All Devs |      2938 | 100.00 |     1.82MB |     274.98KB(14.77)   |       0.00 B( 0.00)   |   669.82KB |    97.30KB(14.53) |
287: +----------------------------------------------------------------------------------------------------------------------------+
287:
287: Full transfer matrix:
287: dst\src          0          1
287:    0        -         97.30KB
287:    1      274.98KB     -
287: +----------------------------------------------------------------------------------------------------------------------------+
287: |         |                    |                       Data In                              |         Data Out               |
287: |Rank   3 |  # KERNEL |    %   |  Required  |   Transfered H2D(%)   |   Transfered D2D(%)   |  Required  |   Transfered(%)   |
287: |---------|-----------|--------|------------|-----------------------|-----------------------|------------|-------------------|
287: |  Dev  0 |      2488 |  83.97 |     0.00 B |       0.00 B(  nan)   |       0.00 B(  nan)   |     0.00 B |     0.00 B(  nan) | cpu-cores
287: |  Dev  1 |       475 |  16.03 |     1.82MB |     253.83KB(13.64)   |       0.00 B( 0.00)   |   669.82KB |    76.15KB(11.37) | cuda(0)
287: |---------|-----------|--------|------------|-----------------------|-----------------------|------------|-------------------|
287: |All Devs |      2963 | 100.00 |     1.82MB |     253.83KB(13.64)   |       0.00 B( 0.00)   |   669.82KB |    76.15KB(11.37) |
287: +----------------------------------------------------------------------------------------------------------------------------+
287:
287: Full transfer matrix:
287: dst\src          0          1
287:    0        -         76.15KB
287:    1      253.83KB     -
287: +----------------------------------------------------------------------------------------------------------------------------+
287: |         |                    |                       Data In                              |         Data Out               |
287: |Rank   1 |  # KERNEL |    %   |  Required  |   Transfered H2D(%)   |   Transfered D2D(%)   |  Required  |   Transfered(%)   |
287: |---------|-----------|--------|------------|-----------------------|-----------------------|------------|-------------------|
287: |  Dev  0 |      2478 |  83.63 |     0.00 B |       0.00 B(  nan)   |       0.00 B(  nan)   |     0.00 B |     0.00 B(  nan) | cpu-cores
287: |  Dev  1 |       485 |  16.37 |     1.86MB |     269.34KB(14.15)   |       0.00 B( 0.00)   |   683.93KB |    91.66KB(13.40) | cuda(0)
287: |---------|-----------|--------|------------|-----------------------|-----------------------|------------|-------------------|
287: |All Devs |      2963 | 100.00 |     1.86MB |     269.34KB(14.15)   |       0.00 B( 0.00)   |   683.93KB |    91.66KB(13.40) |
287: +----------------------------------------------------------------------------------------------------------------------------+
287:
287: Full transfer matrix:
287: dst\src          0          1
287:    0        -         91.66KB
287:    1      269.34KB     -
287: +----------------------------------------------------------------------------------------------------------------------------+
287: |         |                    |                       Data In                              |         Data Out               |
287: |Rank   2 |  # KERNEL |    %   |  Required  |   Transfered H2D(%)   |   Transfered D2D(%)   |  Required  |   Transfered(%)   |
287: |---------|-----------|--------|------------|-----------------------|-----------------------|------------|-------------------|
287: |  Dev  0 |      2493 |  83.71 |     0.00 B |       0.00 B(  nan)   |       0.00 B(  nan)   |     0.00 B |     0.00 B(  nan) | cpu-cores
287: |  Dev  1 |       485 |  16.29 |     1.86MB |     262.29KB(13.78)   |       0.00 B( 0.00)   |   683.93KB |    84.61KB(12.37) | cuda(0)
287: |---------|-----------|--------|------------|-----------------------|-----------------------|------------|-------------------|
287: |All Devs |      2978 | 100.00 |     1.86MB |     262.29KB(13.78)   |       0.00 B( 0.00)   |   683.93KB |    84.61KB(12.37) |
287: +----------------------------------------------------------------------------------------------------------------------------+
287:
287: Full transfer matrix:
287: dst\src          0          1
287:    0        -         84.61KB
287:    1      262.29KB     -
287: --------------------------------------------------------------------------
287: Primary job  terminated normally, but 1 process returned
287: a non-zero exit code. Per user-direction, the job has been aborted.
287: --------------------------------------------------------------------------
287: --------------------------------------------------------------------------
287: mpiexec detected that one or more processes exited with non-zero status, thus causing
287: the job to be terminated. The first process to do so was:
287:
287:   Process name: [[42679,1],1]
287:   Exit code:    1
287: --------------------------------------------------------------------------
1/1 Test #287: dplasma_spotrf_dtd_mpi ...........***Failed    3.08 sec

0% tests passed, 1 tests failed out of 1

Label Time Summary:
dplasma    =   3.08 sec*proc (1 test)
mpi        =   3.08 sec*proc (1 test)

Total Test time (real) =   3.09 sec

The following tests FAILED:
        287 - dplasma_spotrf_dtd_mpi (Failed)
Errors while running CTest
Output from these tests are in: /home/bouteill/parsec/dplasma/build.cuda/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
salloc: Relinquishing job allocation 5296

Setup

• 416aec9 (origin/master, origin/HEAD, master) Merge pull request bugfix: we must count the actual number of cuda devices  #109 from abouteiller/bugfix/dtd_gpu
• parsec/4df0d0cb (origin/master, origin/HEAD, master) Merge pull request #631 from abouteiller/cleanup/cosmetics

 module list                                                                                                                                               ─╯
Currently Loaded Modulefiles:
 1) ncurses/6.4/gcc-11.3.1-6rvznd           25) berkeley-db/18.1.40/gcc-11.3.1-yl6wjj                49) libvterm/0.3.1/gcc-11.3.1-we43r4
 2) htop/3.2.2/gcc-11.3.1-xm6i3t            26) readline/8.2/gcc-11.3.1-b26lae                       50) lua-lpeg/1.0.2-1/gcc-11.3.1-6e6xv6
 3) nghttp2/1.52.0/gcc-11.3.1-yzhzx5        27) gdbm/1.23/gcc-11.3.1-6u5vme                          51) msgpack-c/3.1.1/gcc-11.3.1-pzscaq
 4) zlib/1.2.13/gcc-11.3.1-uhneca           28) perl/5.38.0/gcc-11.3.1-r63sx3                        52) lua-mpack/1.0.9/gcc-11.3.1-z26msa
 5) openssl/3.1.2/gcc-11.3.1-w3u2b2         29) git/2.41.0/gcc-11.3.1-tx4xbg                         53) tree-sitter/0.20.8/gcc-11.3.1-pgy6wn
 6) curl/8.1.2/gcc-11.3.1-dhcq4d            30) cuda/11.8.0/gcc-11.3.1-vltbfy                        54) neovim/0.9.1/gcc-11.3.1-aro6rp
 7) libmd/1.0.4/gcc-11.3.1-yl2qth           31) libpciaccess/0.17/gcc-11.3.1-qp6jxc                  55) cmake/3.26.3/gcc-11.3.1-6bgawm
 8) libbsd/0.11.7/gcc-11.3.1-rxtb5h         32) hwloc/2.9.1/gcc-11.3.1-hvnu6p                        56) ninja/1.11.1/gcc-11.3.1-qf72ao
 9) expat/2.5.0/gcc-11.3.1-z3mywy           33) numactl/2.0.14/gcc-11.3.1-x35xlq                     57) gmp/6.2.1/gcc-11.3.1-c5vz5h
10) bzip2/1.0.8/gcc-11.3.1-g7buii           34) pmix/3.2.3/gcc-11.3.1-b6ek7p                         58) libffi/3.4.4/gcc-11.3.1-suq3vd
11) libiconv/1.17/gcc-11.3.1-h5tewp         35) slurm/22.05.9/gcc-11.3.1-yqiafz                      59) sqlite/3.42.0/gcc-11.3.1-trzf26
12) xz/5.4.1/gcc-11.3.1-ybherp              36) gdrcopy/2.3/gcc-11.3.1-zm6nhb                        60) util-linux-uuid/2.38.1/gcc-11.3.1-h4vnny
13) libxml2/2.10.3/gcc-11.3.1-jijod2        37) libnl/3.3.0/gcc-11.3.1-s2rfpt                        61) python/3.10.12/gcc-11.3.1-msankb
14) pigz/2.7/gcc-11.3.1-2ysjo2              38) rdma-core/41.0/gcc-11.3.1-zlh7l5                     62) gdb/13.1/gcc-11.3.1-awps3c
15) zstd/1.5.5/gcc-11.3.1-maqtnh            39) ucx/1.14.0/gcc-11.3.1-6ffd5t                         63) libevent/2.1.12/gcc-11.3.1-iqf4hw
16) tar/1.34/gcc-11.3.1-jl543d              40) openmpi/4.1.5/gcc-11.3.1-2rgaqk                      64) tmux/3.3a/gcc-11.3.1-nt2vwg
17) gettext/0.21.1/gcc-11.3.1-sgm6rr        41) gperf/3.1/gcc-11.3.1-lq7yw2                          65) cscope/15.9/gcc-11.3.1-4duk6k
18) libunistring/1.1/gcc-11.3.1-mswbrm      42) jemalloc/5.3.0/gcc-11.3.1-gnjgyl                     66) exuberant-ctags/5.8/gcc-11.3.1-f56ide
19) libidn2/2.3.4/gcc-11.3.1-kp77oe         43) libuv/1.44.1/gcc-11.3.1-ikknoi                       67) intel-oneapi-tbb/2021.10.0/gcc-11.3.1-ptv4p2
20) krb5/1.20.1/gcc-11.3.1-hb7cxy           44) unzip/6.0/gcc-11.3.1-xm5nhk                          68) intel-oneapi-mkl/2023.2.0/gcc-11.3.1-d5uffv
21) libedit/3.1-20210216/gcc-11.3.1-b2res4  45) lua-luajit-openresty/2.1-20230410/gcc-11.3.1-lgkuf6  69) mpfr/4.2.0/gcc-11.3.1-n3mu53
22) libxcrypt/4.4.35/gcc-11.3.1-v7ot4t      46) libluv/1.44.2-1/gcc-11.3.1-pyqvat                    70) mpc/1.3.1/gcc-11.3.1-2x6jci
23) openssh/9.3p1/gcc-11.3.1-jo2led         47) unibilium/2.0.0/gcc-11.3.1-az5pko                    71) gcc/13.2.0/gcc-11.3.1-ir6jns
24) pcre2/10.42/gcc-11.3.1-bk6jhf           48) libtermkey/0.22/gcc-11.3.1-gwvd67

[abouteiller]

Results still incorrect after #133 😠
PMIX_MCA_psec='' SLURM_TIMELIMIT=1 PARSEC_MCA_device_cuda_enabled=1 PARSEC_MCA_device_cuda_memory_use=10 OMPI_MCA_rmaps_base_oversubscribe=true salloc -wleconte -n 8 --gpus-per-task=1 /usr/bin/srun "-n" "4" "tests/testing_spotrf_dtd" "-N" "378" "-t" "19" "-x" "-v=5"

[abouteiller]

with ICLDisco/parsec@064918a3 we get a clean crash, maybe easier to track .

PMIX_MCA_psec='' SLURM_TIMELIMIT=1 PARSEC_MCA_device_cuda_enabled=1 PARSEC_MCA_device_cuda_memory_use=10 OMPI_MCA_rmaps_base_oversubscribe=true salloc -wleconte -n 8 --gpus-per-task=1 /usr/bin/srun "-n" "4" "tests/testing_spotrf_dtd" "-N" "378" "-t" "19" "-x" "-v=5"
salloc: Pending job allocation 799
salloc: job 799 queued and waiting for resources
salloc: job 799 has been allocated resources
salloc: Granted job allocation 799
salloc: Waiting for resource configuration

salloc: Nodes leconte are ready for job
W@00000 /!\ DEBUG LEVEL WILL PROBABLY REDUCE THE PERFORMANCE OF THIS RUN /!\.
#+++++ cores detected       : 40
#+++++ nodes x cores + gpu  : 4 x 40 + 1 (160+4)
#+++++ thread mode          : THREAD_SERIALIZED
#+++++ P x Q                : 4 x 1 (4/4)
#+++++ M x N x K|NRHS       : 378 x 378 x 1
#+++++ LDA                  : 378
#+++++ MB x NB              : 19 x 19
W@00000 /!\ PERFORMANCE MIGHT BE REDUCED /!\: Multiple PaRSEC processes on the same node may share the same physical core(s);
        This is often unintentional, and will perform poorly.
        Note that in managed environments (e.g., ALPS, jsrun), the launcher may set `cgroups`
        and hide the real binding from PaRSEC; if you verified that the binding is correct,
        this message can be silenced using the MCA argument `runtime_warn_slow_binding`.
testing_spotrf_dtd: /home/bouteill/parsec/dplasma-master/parsec/parsec/class/info.c:263: void parsec_ioa_resize_and_rdlock(parsec_info_object_array_t *, parsec_info_id_t): Assertion `oa->infos->max_id >= iid' failed.
[leconte:2608290] *** Process received signal ***
[leconte:2608290] Signal: Aborted (6)
[leconte:2608290] Signal code:  (-6)
[leconte:2608290] [ 0] /lib64/libc.so.6(+0x3e730)[0x7f973a761730]
[leconte:2608290] [ 1] /lib64/libc.so.6(+0x8ba6c)[0x7f973a7aea6c]
[leconte:2608290] [ 2] /lib64/libc.so.6(raise+0x16)[0x7f973a761686]
[leconte:2608290] [ 3] /lib64/libc.so.6(abort+0xd3)[0x7f973a74b833]
[leconte:2608290] [ 4] /lib64/libc.so.6(+0x2875b)[0x7f973a74b75b]
[leconte:2608290] [ 5] /lib64/libc.so.6(+0x373c6)[0x7f973a75a3c6]
[leconte:2608290] [ 6] /home/bouteill/parsec/dplasma-master/build.cuda/parsec/parsec/libparsec.so.4(+0x2e3c1)[0x7f978758d3c1]
[leconte:2608290] [ 7] /home/bouteill/parsec/dplasma-master/build.cuda/parsec/parsec/libparsec.so.4(parsec_info_get+0x32)[0x7f978758d582]
[leconte:2608290] [ 8] /home/bouteill/parsec/dplasma-master/build.cuda/src/libdplasma.so.2(parsec_core_spotrf_cuda+0xc3)[0x7f9787bf5f53]
[leconte:2608290] [ 9] /home/bouteill/parsec/dplasma-master/build.cuda/parsec/parsec/libparsec.so.4(+0x94684)[0x7f97875f3684]
[leconte:2608290] [10] /home/bouteill/parsec/dplasma-master/build.cuda/parsec/parsec/libparsec.so.4(+0x93846)[0x7f97875f2846]
[leconte:2608290] [11] /home/bouteill/parsec/dplasma-master/build.cuda/parsec/parsec/libparsec.so.4(parsec_device_kernel_scheduler+0x52f)[0x7f97875f19bf]
[leconte:2608290] [12] /home/bouteill/parsec/dplasma-master/build.cuda/parsec/parsec/libparsec.so.4(+0x84c1d)[0x7f97875e3c1d]
[leconte:2608290] [13] /home/bouteill/parsec/dplasma-master/build.cuda/parsec/parsec/libparsec.so.4(__parsec_execute+0x269)[0x7f97875cad29]
[leconte:2608290] [14] /home/bouteill/parsec/dplasma-master/build.cuda/parsec/parsec/libparsec.so.4(__parsec_task_progress+0x10e)[0x7f97875cbc0e]
[leconte:2608290] [15] /home/bouteill/parsec/dplasma-master/build.cuda/parsec/parsec/libparsec.so.4(__parsec_context_wait+0x3d7)[0x7f97875cc2d7]
[leconte:2608290] [16] /home/bouteill/parsec/dplasma-master/build.cuda/parsec/parsec/libparsec.so.4(+0x4b5f7)[0x7f97875aa5f7]