PMIX ERROR: BAD-PARAM in file base/bfrop_base_unpack.c at line 692

[angelinesalberto]

When I run wrf in several nodes and I get the next errors,

PMIX ERROR: BAD-PARAM in file dstore_base.c at line 2225
PMIX ERROR: BAD-PARAM in file base/bfrop_base_unpack.c at line 692

for each node.

I am using:
openmpi-4.1.0
pmix-2.2.4
slurm 18.08.6

I installed openmpi with Spack
spack install openmpi@4.1.0 %intel@19.0.5.281 fabrics=ucx +pmi schedulers=slurm +thread_multiple +vt

Please describe the system on which you are running

• Operating system/version: centos7
• Network type: infiniband mlx5

---

Details of the problem

I run the simulation with
srun --mpi=pmix wrf.exe
I get the problems but the program doesn't stop, It continues the execution.

I get the same errors with pmix3.0.2 and pmix2.2.2

[jsquyres]

Is there any other context given in the PMIX error messages, or just those 2 lines?

[angelinesalberto]

Online those 2 lines but by core. The program not abort but I don’t know if the program can lose efficiency

…

________________________________ De: Jeff Squyres <notifications@github.com> Enviado: Monday, March 8, 2021 10:08:23 PM Para: open-mpi/ompi <ompi@noreply.github.com> Cc: angelinesalberto <angelinesalberto@hotmail.com>; Author <author@noreply.github.com> Asunto: Re: [open-mpi/ompi] PMIX ERROR: BAD-PARAM in file base/bfrop_base_unpack.c at line 692 (#8548) Is there any other context given in the PMIX error messages, or just those 2 lines? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#8548 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJCUO76B32EOB64K434Z6F3TCU4EPANCNFSM4YQ7JZGQ>.

[jsquyres]

The dstore and the SLURM plugins are maintained by @artpol84 -- Artem, can you have a look?

[artpol84]

Thanks, @angelinesalberto

Quick question. Looking into the pmix-2.2.4 codebase, I'm trying to map the output

PMIX ERROR: BAD-PARAM in file dstore_base.c at line 2225

onto the sources and it seems to have a mismatch. i.e.:
line 2225 in the v2.2.4 tag points to a comment /* unset lock */:
https://github.com/openpmix/openpmix/blob/v2.2.4/src/mca/common/dstore/dstore_base.c#L2225

The same thing with the base/bfrop_base_unpack.c:

PMIX ERROR: BAD-PARAM in file base/bfrop_base_unpack.c at line 692

Line 692 contains a comment /* this field is now a pointer, so we must allocate storage for it */ instead of the error logging call:
https://github.com/openpmix/openpmix/blob/v2.2.4/src/mca/bfrops/base/bfrop_base_unpack.c#L692

So it seems like at least dstore_base.c and base/bfrop_base_unpack.c were changed compared to v2.2.4.

My naive guess would be that your Slurm is build with the different PMIx version that is for some reasons incompatible with OMPI counterpart.

Can you run the following commands in the OMPI and Slurm installation directories and post the output:

• OMPI: $ find -name "mca_pmix*" | xargs ldd
• Slurm: $ find -name "mpi_pmix*" | xargs ldd

[artpol84]

srun --mpi=pmix wrf.exe
I get the problems but the program doesn't stop, It continues the execution.

Also, it would be interesting to see if a simpler executable yields the same result:

• First try running something like [ring.c] from OMPI examples to see if the simple MPI app runs fine. If it's not - the issue is not specific to the wrf.exe.
• Please try a bare PMIx program, like pmix_perf [link].
  • Please make sure to change Makefile [here] to point to your PMIx and Slurm installations. If you'll find from the experiment from my previous post that PMIx installations used by Slurm and OMPI are different - try to experiment with both PMIx installations. The expectation is that PMIx with which the Slurm was built should work, the other one should fail (because this is what you observe).
  • After that execute make pmix_intra_perf.
  • Launch it with srun --mpi=pmix ./pmix_intra_perf

[angelinesalberto]

Hi Artem, thanks for your help OMPI: $ find -name "mca_pmix*" | xargs ldd find -name "mca_pmix*" | xargs ldd ./lib/openmpi/mca_pmix_flux.so: linux-vdso.so.1 => (0x00007ffcae4f4000) libopen-pal.so.40 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/openmpi-4.1.0-tjpbvgi3b7imtx2spy7ryygw4ecxvkyw/lib/libopen-pal.so.40 (0x00002b5601872000) librt.so.1 => /lib64/librt.so.1 (0x00002b5601b58000) libutil.so.1 => /lib64/libutil.so.1 (0x00002b5601d60000) libhwloc.so.5 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/hwloc-1.11.9-fo534f7swi4rdu5ng6supwxgsgcxnnet/lib/libhwloc.so.5 (0x00002b5601f63000) libnuma.so.1 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/numactl-2.0.11-6nblprppndv2i3cold2c3f5fwhbdh5so/lib/libnuma.so.1 (0x00002b56021bb000) libpciaccess.so.0 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/libpciaccess-0.13.5-pcvclpjgvwvwhcj7h2lv5rvbdm7k4vlk/lib/libpciaccess.so.0 (0x00002b56023c9000) libxml2.so.2 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/libxml2-2.9.8-rhrn72yfyzknktaybz3xbvq5ud5xe7nt/lib/libxml2.so.2 (0x00002b56025d5000) libz.so.1 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/zlib-1.2.11-kdh3v4z6c36m67oovrg55ba53abj3wev/lib/libz.so.1 (0x00002b5602a9c000) liblzma.so.5 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/xz-5.2.4-mfl2reblod54l227df66ftyacu4s3flj/lib/liblzma.so.5 (0x00002b5602cba000) libm.so.6 => /lib64/libm.so.6 (0x00002b5602ef2000) libimf.so => /mnt/lustre/localsoft/intel/intel19/Fortran/compilers_and_libraries_2019.5.281/linux/compiler/lib/intel64_lin/libimf.so (0x00002b56031f4000) libsvml.so => /mnt/lustre/localsoft/intel/intel19/Fortran/compilers_and_libraries_2019.5.281/linux/compiler/lib/intel64_lin/libsvml.so (0x00002b5603879000) libirng.so => /mnt/lustre/localsoft/intel/intel19/Fortran/compilers_and_libraries_2019.5.281/linux/compiler/lib/intel64_lin/libirng.so (0x00002b5605305000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b5605670000) libintlc.so.5 => /mnt/lustre/localsoft/intel/intel19/Fortran/compilers_and_libraries_2019.5.281/linux/compiler/lib/intel64_lin/libintlc.so.5 (0x00002b5605886000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b5605af8000) libc.so.6 => /lib64/libc.so.6 (0x00002b5605d14000) libdl.so.2 => /lib64/libdl.so.2 (0x00002b56060e1000) /lib64/ld-linux-x86-64.so.2 (0x00002b5601447000) ./lib/openmpi/mca_pmix_pmix3x.so: linux-vdso.so.1 => (0x00007ffc5a1c0000) libopen-pal.so.40 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/openmpi-4.1.0-tjpbvgi3b7imtx2spy7ryygw4ecxvkyw/lib/libopen-pal.so.40 (0x00002abf696fa000) librt.so.1 => /lib64/librt.so.1 (0x00002abf699e0000) libutil.so.1 => /lib64/libutil.so.1 (0x00002abf69be8000) libhwloc.so.5 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/hwloc-1.11.9-fo534f7swi4rdu5ng6supwxgsgcxnnet/lib/libhwloc.so.5 (0x00002abf69deb000) libnuma.so.1 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/numactl-2.0.11-6nblprppndv2i3cold2c3f5fwhbdh5so/lib/libnuma.so.1 (0x00002abf6a043000) libpciaccess.so.0 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/libpciaccess-0.13.5-pcvclpjgvwvwhcj7h2lv5rvbdm7k4vlk/lib/libpciaccess.so.0 (0x00002abf6a251000) libxml2.so.2 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/libxml2-2.9.8-rhrn72yfyzknktaybz3xbvq5ud5xe7nt/lib/libxml2.so.2 (0x00002abf6a45d000) libz.so.1 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/zlib-1.2.11-kdh3v4z6c36m67oovrg55ba53abj3wev/lib/libz.so.1 (0x00002abf6a924000) liblzma.so.5 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/xz-5.2.4-mfl2reblod54l227df66ftyacu4s3flj/lib/liblzma.so.5 (0x00002abf6ab42000) libm.so.6 => /lib64/libm.so.6 (0x00002abf6ad7a000) libimf.so => /mnt/lustre/localsoft/intel/intel19/Fortran/compilers_and_libraries_2019.5.281/linux/compiler/lib/intel64_lin/libimf.so (0x00002abf6b07c000) libsvml.so => /mnt/lustre/localsoft/intel/intel19/Fortran/compilers_and_libraries_2019.5.281/linux/compiler/lib/intel64_lin/libsvml.so (0x00002abf6b701000) libirng.so => /mnt/lustre/localsoft/intel/intel19/Fortran/compilers_and_libraries_2019.5.281/linux/compiler/lib/intel64_lin/libirng.so (0x00002abf6d18d000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002abf6d4f8000) libintlc.so.5 => /mnt/lustre/localsoft/intel/intel19/Fortran/compilers_and_libraries_2019.5.281/linux/compiler/lib/intel64_lin/libintlc.so.5 (0x00002abf6d70e000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00002abf6d980000) libc.so.6 => /lib64/libc.so.6 (0x00002abf6db9c000) libdl.so.2 => /lib64/libdl.so.2 (0x00002abf6df69000) /lib64/ld-linux-x86-64.so.2 (0x00002abf6919d000) ./lib/openmpi/mca_pmix_s1.la: no es un ejecutable dinámico ./lib/openmpi/mca_pmix_pmix3x.la: no es un ejecutable dinámico ./lib/openmpi/mca_pmix_isolated.la: no es un ejecutable dinámico ./lib/openmpi/mca_pmix_flux.la: no es un ejecutable dinámico ./lib/openmpi/mca_pmix_s2.la: no es un ejecutable dinámico ./lib/openmpi/mca_pmix_s2.so: linux-vdso.so.1 => (0x00007ffdc2f95000) libopen-pal.so.40 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/openmpi-4.1.0-tjpbvgi3b7imtx2spy7ryygw4ecxvkyw/lib/libopen-pal.so.40 (0x00002ba2ab35d000) libpmi2.so.0 => /lib64/libpmi2.so.0 (0x00002ba2ab643000) librt.so.1 => /lib64/librt.so.1 (0x00002ba2ab85b000) libutil.so.1 => /lib64/libutil.so.1 (0x00002ba2aba63000) libhwloc.so.5 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/hwloc-1.11.9-fo534f7swi4rdu5ng6supwxgsgcxnnet/lib/libhwloc.so.5 (0x00002ba2abc66000) libnuma.so.1 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/numactl-2.0.11-6nblprppndv2i3cold2c3f5fwhbdh5so/lib/libnuma.so.1 (0x00002ba2abebe000) libpciaccess.so.0 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/libpciaccess-0.13.5-pcvclpjgvwvwhcj7h2lv5rvbdm7k4vlk/lib/libpciaccess.so.0 (0x00002ba2ac0cc000) libxml2.so.2 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/libxml2-2.9.8-rhrn72yfyzknktaybz3xbvq5ud5xe7nt/lib/libxml2.so.2 (0x00002ba2ac2d8000) libz.so.1 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/zlib-1.2.11-kdh3v4z6c36m67oovrg55ba53abj3wev/lib/libz.so.1 (0x00002ba2ac79f000) liblzma.so.5 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/xz-5.2.4-mfl2reblod54l227df66ftyacu4s3flj/lib/liblzma.so.5 (0x00002ba2ac9bd000) libm.so.6 => /lib64/libm.so.6 (0x00002ba2acbf5000) libimf.so => /mnt/lustre/localsoft/intel/intel19/Fortran/compilers_and_libraries_2019.5.281/linux/compiler/lib/intel64_lin/libimf.so (0x00002ba2acef7000) libsvml.so => /mnt/lustre/localsoft/intel/intel19/Fortran/compilers_and_libraries_2019.5.281/linux/compiler/lib/intel64_lin/libsvml.so (0x00002ba2ad57c000) libirng.so => /mnt/lustre/localsoft/intel/intel19/Fortran/compilers_and_libraries_2019.5.281/linux/compiler/lib/intel64_lin/libirng.so (0x00002ba2af008000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002ba2af373000) libintlc.so.5 => /mnt/lustre/localsoft/intel/intel19/Fortran/compilers_and_libraries_2019.5.281/linux/compiler/lib/intel64_lin/libintlc.so.5 (0x00002ba2af589000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00002ba2af7fb000) libc.so.6 => /lib64/libc.so.6 (0x00002ba2afa17000) libdl.so.2 => /lib64/libdl.so.2 (0x00002ba2afde4000) /lib64/ld-linux-x86-64.so.2 (0x00002ba2aaf32000) ./lib/openmpi/mca_pmix_s1.so: linux-vdso.so.1 => (0x00007ffdc99fd000) libopen-pal.so.40 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/openmpi-4.1.0-tjpbvgi3b7imtx2spy7ryygw4ecxvkyw/lib/libopen-pal.so.40 (0x00002b106abca000) libpmi.so.0 => /lib64/libpmi.so.0 (0x00002b106aeb0000) librt.so.1 => /lib64/librt.so.1 (0x00002b106b0b6000) libutil.so.1 => /lib64/libutil.so.1 (0x00002b106b2be000) libhwloc.so.5 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/hwloc-1.11.9-fo534f7swi4rdu5ng6supwxgsgcxnnet/lib/libhwloc.so.5 (0x00002b106b4c1000) libnuma.so.1 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/numactl-2.0.11-6nblprppndv2i3cold2c3f5fwhbdh5so/lib/libnuma.so.1 (0x00002b106b719000) libpciaccess.so.0 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/libpciaccess-0.13.5-pcvclpjgvwvwhcj7h2lv5rvbdm7k4vlk/lib/libpciaccess.so.0 (0x00002b106b927000) libxml2.so.2 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/libxml2-2.9.8-rhrn72yfyzknktaybz3xbvq5ud5xe7nt/lib/libxml2.so.2 (0x00002b106bb33000) libz.so.1 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/zlib-1.2.11-kdh3v4z6c36m67oovrg55ba53abj3wev/lib/libz.so.1 (0x00002b106bffa000) liblzma.so.5 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/xz-5.2.4-mfl2reblod54l227df66ftyacu4s3flj/lib/liblzma.so.5 (0x00002b106c218000) libm.so.6 => /lib64/libm.so.6 (0x00002b106c450000) libimf.so => /mnt/lustre/localsoft/intel/intel19/Fortran/compilers_and_libraries_2019.5.281/linux/compiler/lib/intel64_lin/libimf.so (0x00002b106c752000) libsvml.so => /mnt/lustre/localsoft/intel/intel19/Fortran/compilers_and_libraries_2019.5.281/linux/compiler/lib/intel64_lin/libsvml.so (0x00002b106cdd7000) libirng.so => /mnt/lustre/localsoft/intel/intel19/Fortran/compilers_and_libraries_2019.5.281/linux/compiler/lib/intel64_lin/libirng.so (0x00002b106e863000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b106ebce000) libintlc.so.5 => /mnt/lustre/localsoft/intel/intel19/Fortran/compilers_and_libraries_2019.5.281/linux/compiler/lib/intel64_lin/libintlc.so.5 (0x00002b106ede4000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b106f056000) libc.so.6 => /lib64/libc.so.6 (0x00002b106f272000) libdl.so.2 => /lib64/libdl.so.2 (0x00002b106f63f000) libslurmfull.so => /usr/lib64/slurm/libslurmfull.so (0x00002b106f843000) /lib64/ld-linux-x86-64.so.2 (0x00002b106a79f000) ./lib/openmpi/mca_pmix_isolated.so: linux-vdso.so.1 => (0x00007ffc73ff3000) libopen-pal.so.40 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/openmpi-4.1.0-tjpbvgi3b7imtx2spy7ryygw4ecxvkyw/lib/libopen-pal.so.40 (0x00002b23dda71000) librt.so.1 => /lib64/librt.so.1 (0x00002b23ddd57000) libutil.so.1 => /lib64/libutil.so.1 (0x00002b23ddf5f000) libhwloc.so.5 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/hwloc-1.11.9-fo534f7swi4rdu5ng6supwxgsgcxnnet/lib/libhwloc.so.5 (0x00002b23de162000) libnuma.so.1 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/numactl-2.0.11-6nblprppndv2i3cold2c3f5fwhbdh5so/lib/libnuma.so.1 (0x00002b23de3ba000) libpciaccess.so.0 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/libpciaccess-0.13.5-pcvclpjgvwvwhcj7h2lv5rvbdm7k4vlk/lib/libpciaccess.so.0 (0x00002b23de5c8000) libxml2.so.2 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/libxml2-2.9.8-rhrn72yfyzknktaybz3xbvq5ud5xe7nt/lib/libxml2.so.2 (0x00002b23de7d4000) libz.so.1 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/zlib-1.2.11-kdh3v4z6c36m67oovrg55ba53abj3wev/lib/libz.so.1 (0x00002b23dec9b000) liblzma.so.5 => /mnt/lustre/ohpc/admin/spack/0.12.1/opt/spack/linux-centos7-x86_64/intel-19.0.5.281/xz-5.2.4-mfl2reblod54l227df66ftyacu4s3flj/lib/liblzma.so.5 (0x00002b23deeb9000) libm.so.6 => /lib64/libm.so.6 (0x00002b23df0f1000) libimf.so => /mnt/lustre/localsoft/intel/intel19/Fortran/compilers_and_libraries_2019.5.281/linux/compiler/lib/intel64_lin/libimf.so (0x00002b23df3f3000) libsvml.so => /mnt/lustre/localsoft/intel/intel19/Fortran/compilers_and_libraries_2019.5.281/linux/compiler/lib/intel64_lin/libsvml.so (0x00002b23dfa78000) libirng.so => /mnt/lustre/localsoft/intel/intel19/Fortran/compilers_and_libraries_2019.5.281/linux/compiler/lib/intel64_lin/libirng.so (0x00002b23e1504000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b23e186f000) libintlc.so.5 => /mnt/lustre/localsoft/intel/intel19/Fortran/compilers_and_libraries_2019.5.281/linux/compiler/lib/intel64_lin/libintlc.so.5 (0x00002b23e1a85000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b23e1cf7000) libc.so.6 => /lib64/libc.so.6 (0x00002b23e1f13000) libdl.so.2 => /lib64/libdl.so.2 (0x00002b23e22e0000) /lib64/ld-linux-x86-64.so.2 (0x00002b23dd647000) * Slurm: $ find -name "mpi_pmix*" | xargs ldd find -name "mpi_pmix*" | xargs ldd ./slurm/mpi_pmix.so: linux-vdso.so.1 => (0x00007ffe4f18e000) libhwloc.so.5 => /usr/lib64/libhwloc.so.5 (0x00002b166a5ab000) libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00002b166a7e8000) libc.so.6 => /usr/lib64/libc.so.6 (0x00002b166aa04000) libm.so.6 => /usr/lib64/libm.so.6 (0x00002b166add1000) libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x00002b166b0d3000) libltdl.so.7 => /usr/lib64/libltdl.so.7 (0x00002b166b2df000) /lib64/ld-linux-x86-64.so.2 (0x00002b166a15c000) libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00002b166b4e9000) libdl.so.2 => /usr/lib64/libdl.so.2 (0x00002b166b6ff000) ./slurm/mpi_pmix_v2.so: linux-vdso.so.1 => (0x00007ffff5f80000) libhwloc.so.5 => /usr/lib64/libhwloc.so.5 (0x00002aafd65d0000) libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00002aafd680d000) libc.so.6 => /usr/lib64/libc.so.6 (0x00002aafd6a29000) libm.so.6 => /usr/lib64/libm.so.6 (0x00002aafd6df6000) libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x00002aafd70f8000) libltdl.so.7 => /usr/lib64/libltdl.so.7 (0x00002aafd7304000) /lib64/ld-linux-x86-64.so.2 (0x00002aafd6181000) libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00002aafd750e000) libdl.so.2 => /usr/lib64/libdl.so.2 (0x00002aafd7724000) De: Artem Polyakov ***@***.***> Responder a: open-mpi/ompi ***@***.***> Fecha: martes, 23 de marzo de 2021, 1:08 Para: open-mpi/ompi ***@***.***> CC: ***@***.***" ***@***.***>, Mention ***@***.***> Asunto: Re: [open-mpi/ompi] PMIX ERROR: BAD-PARAM in file base/bfrop_base_unpack.c at line 692 (#8548) Thanks, @angelinesalberto<https://github.com/angelinesalberto> Quick question. Looking into the pmix-2.2.4 codebase, I'm trying to map the output PMIX ERROR: BAD-PARAM in file dstore_base.c at line 2225 onto the sources and it seems to have a mismatch. i.e.: line 2225 in the v2.2.4 tag points to a comment /* unset lock */: https://github.com/openpmix/openpmix/blob/v2.2.4/src/mca/common/dstore/dstore_base.c#L2225 The same thing with the base/bfrop_base_unpack.c: PMIX ERROR: BAD-PARAM in file base/bfrop_base_unpack.c at line 692 Line 692 contains a comment /* this field is now a pointer, so we must allocate storage for it */ instead of the error logging call: https://github.com/openpmix/openpmix/blob/v2.2.4/src/mca/bfrops/base/bfrop_base_unpack.c#L692 So it seems like at least dstore_base.c and base/bfrop_base_unpack.c were changed compared to v2.2.4. My naive guess would be that your Slurm is build with the different PMIx version that is for some reasons incompatible with OMPI counterpart. Can you run the following commands in the OMPI and Slurm installation directories and post the output: * OMPI: $ find -name "mca_pmix*" | xargs ldd * Slurm: $ find -name "mpi_pmix*" | xargs ldd — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#8548 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJCUO72RLWHDSO5RZSB277LTE7LZLANCNFSM4YQ7JZGQ>.

[artpol84]

I forgot that in Slurm we are using a dlopen to load PMIx so the PMIx build used in Slurm is unfortunately not reflected in the corresponding ldd. Do you have Slurm configuration line handy?

But what I see is that you are using OMPI's internal PMIx (mca_pmix_pmix3x.so) while we usually recommend building OMPI

--with-pmix=<Slurm's-PMIx-path>

While PMIx should have backwards compatibility features, I would recommend playing safe and explicitly build OMPI against the same PMIx as in Slurm and try again.
NOTE: This would require you to build and provide libevent and hwloc (also using --with-hwloc/libevent OMPI config parameters). Hopefully they were already built for Slurm's PMIx anyway.

[rhc54]

@angelinesalberto Any update on this?