core assignments with unbalanced tasks per node

[ryanday36]

When one asks for a number of tasks that is evenly divisible by the number of nodes requested, Flux binds all of those tasks to the same number of cores. E.g.

day36@fluke131:~]$ flux run -N2 -n2 --cores-per-task=1 -o mpibind=off -o cpu-affinity=per-task --label numactl -s | grep physcpu
0: physcpubind: 1
1: physcpubind: 1

If the number of tasks is not divisible evenly across the nodes, Flux allows some tasks to access more cpus:

[day36@fluke131:~]$ flux run -N2 -n3 --cores-per-task=1 -o mpibind=off -o cpu-affinity=per-task --label numactl -s | grep physcpu
0: physcpubind: 0
2: physcpubind: 0 1
1: physcpubind: 1

This example uses the built-in cpu-affinity shell option, but this is also affecting mpibind and causing it to improperly bind gpus in cases where the user asks for a number of gpus that isn't evenly divisible by the number of nodes. E.g.

day36@rzadams1041:$ MPIBIND_RESTRICT= flux run -N2 -n7 --gpus-per-task=1 --cores-per-task=24 --label -o mpibind=v bash -c 'echo $ROCR_VISIBLE_DEVICES'
0.072s: flux-shell[1]: mpibind:
mpibind: task 0 nths 96 gpus 0,1 cpus 0-47,96-143
mpibind: task 1 nths 48 gpus 2 cpus 48-71,144-167
mpibind: task 2 nths 48 gpus 3 cpus 72-95,168-191
0.070s: flux-shell[0]: mpibind:
mpibind: task 0 nths 24 gpus 0 cpus 0-23
mpibind: task 1 nths 24 gpus 1 cpus 24-47
mpibind: task 2 nths 24 gpus 2 cpus 48-71
mpibind: task 3 nths 24 gpus 3 cpus 72-95
0: 0
4: 0,1
1: 1
5: 2
6: 3
2: 2
3: 3
day36@rzadams1041:$

[grondo]

This might be due to #3750. IIRC, this is a limitation of Jobspec V1 -- at least presently we can't represent non uniform request across nodes, so uneven distribution results in wasted task slots.

The affinity plugin in the shell is unaware of the originally requested cores-per-task, and all it does is set the affinity of the job to the actual resources assigned to the current node. When per-task is requested, the affinity plugin then just evenly distributes tasks over those resources using hwloc_distribute.

You can see the problem by looking at the jobspec for the second example:

$ flux run -N2 -n3 --cores-per-task=1 -o mpibind=off -o cpu-affinity=per-task --label --dry-run hostname | jq .resources
[
  {
    "type": "node",
    "count": 2,
    "with": [
      {
        "type": "slot",
        "count": 2,
        "with": [
          {
            "type": "core",
            "count": 1
          }
        ],
        "label": "task"
      }
    ]
  }
]

There are a couple things we could do to try to tackle this one:

• See if there is any way JobspecV1 could support or easily be extended to support non-uniform slots per node, or multiple node entries in the resources section
• The shell can get the slot size from jobspec and share that to the cpu-affinity and mpibind plugins so they can assign less than the allocated cores to the job and/or tasks. @sam-maloney may have some insight here since the shell now supports getting the slot counts from R but not the actual slot shape or size.

Ideally we could do the first above. Probably need to consult with @trws on that one.

[sam-maloney]

I agree with @grondo that the first bullet is the more ideal way forward, and would also begin addressing more general needs for MPMD type workflows by allowing heterogeneous slot shapes. To be really ideal, this would require a more general Resource Set specification, since it would start to break the assumptions of uniformity in v1; even with nslots in R it doesn't provide any mapping of resources to slots which would be problematic if they aren't homogeneous (some earlier thoughts on this flux-framework/rfc#466 (comment))

As for what's possible right now, within the assumptions of jobspec v1 it should be trivial to just calculate, i.e. since all slots must currently have the same shape, total_cores/total_slots (or floor(total_cores/total_tasks)) would give you the number of cores requested per task, and similarly for GPUs if present. Presumably one could then just not assign more resources than this to any tasks, even if resources are then left idle. This would be a temporary band-aid fix, but could be quicker to implement while a more complete solution is developed?

[grondo]

Thanks for your great insight @sam-maloney! I was thinking along the same lines for a quick fix.

[trws]

I'm catching up on what I missed last week, so sorry to resurrect. Just to be sure, this doesn't happen (without the patch I see above) unless you're on a system with node exclusive configured right? I ask because I'm pretty sure what we select in the scheduler assumes the user's job gets what we gave them on any given node, and if that's not holding non-exclusive allocations could end up overlapping in corner cases. We do specific allocations on a per-node basis so the information is available, if we decide on a way to expose it in R.