using PythonCall causes external MPI programs to run in series

[joegilkes]

I have an external MPI-enabled program that I need Julia to call as a Cmd, for example:

c = Cmd(`mpirun -np 32 aims.231208.scalapack.mpi.x`)
run(c)

This launches the program over 32 parallel MPI processes as expected, with the main process taking the lead and communicating with the other processes, as well as doing the logging.

However, if using PythonCall is run before this, I get entirely different behaviour. Instead of distributing computation among the processes, mpirun creates 32 separate processes, which all run the entire program individually and do not communicate, as evidenced by the fact that they all output their own logs to stdout at the same time.

I have no idea why PythonCall could be responsible here as I can't find it ever mentioning MPI, but this is the only thread I've found to fix this behaviour so far. The actual problem involves PythonCall running subprocess.Popen with this program from inside Julia, but it seems like the issue starts with only loading PythonCall and not even interacting with it.

Does anyone have any idea why this would be happening?

[joegilkes]

Problem seems to be solved now - a Python dependency of a package installed with CondaPkg included its own mpirun executable, which was getting added to PATH in front of my system's mpirun and being executed preferentially when PythonCall was loaded. Some kind of weird interplay between the conda version of mpirun and the version that the MPI-aware program was expecting seems to have caused the issue I was experiencing, without complaining of a version mismatch.

[cjdoris]

Yeah this sounds like a side-effect of PythonCall activating the Conda environment "globally" inside Julia. In particular it modifies the PATH to include all everywhere Conda installs things.

Ideally PythonCall wouldn't need to do this but it does seem necessary or else packages don't load properly.