mpirun v4.1.7 prioritizing rank placement based on launch node

[aw-lauria]

Background information

Hi all - we're seeing unexpected rank placement when not launching from the first host in our hostfile. Orte seems to prioritize the launch node when assigning ranks. For example:

mpirun --hostfile ./hosts -N 4 ./echo.sh  | grep computeA
computeA: 0
computeA: 1
computeA: 2
computeA: 3

where the hostfile looks like this:

computeB
computeA

and echo.sh is just:

#!/usr/bin/bash
echo $(hostname): $OMPI_COMM_WORLD_RANK

Basically it is giving priority rank assignment to the launch node. We would expect that computeA would be assigned ranks 4 through 7 based on the hostfile ordering.

What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)

v4.1.4 and v4.1.7

Is this expected behavior? What is the rationale? This is something we've run into occasionally, and it can have an performance impact on certain workloads. We can work-around it of course by always launching from the first node in the hostfile. It just happens that sometimes in our testing, we occasionally launch from the wrong node.

Thanks!

[rhc54]

Hostfile has never been an ordered list - it's just tells us hostnames and slot allocations. There is no attempt to order placements based on it outside of the sequential mapper. Been that way for over 20 years now 😄

We do treat --host as ordered, if that helps.

[rhc54]

Thinking about it - should be pretty simple to make it ordered so it matches “—host”. I’ll take a peek at it next week

[aw-lauria]

That would be awesome @rhc54 . If for whatever reason that can't get into the v4.1.x series, we can probably transition to using --host.

Thanks!

[rhc54]

I very much doubt the release managers would be willing to bring this back to the v4 series - it may be "simple", but the code impact is a source of risk. I'm not entirely sure of the plans for OMPI v5 or v6 runtimes going forward, so I can't say if/when this will appear there either. I'll update PRRTE - kinda up to the OMPI folks from there. 🤷‍♂

Your best bet for OMPI v4 is to use --host.