Add group.node_params to partitions/groups.

[kbendl]

Allows Features, etc., to be added to partitions.
Also added #comment lines for readability to highlight partitions/groups sections.

[sjpb]

@jovial this was my suggestion for adding arbitrary node parameters. But having seen your #181 , should we actually be adding node parameters on a groupvar level, rather than in partitions, to avoid the same clash issue with nodes in multiple partitions?

[jovial]

@jovial this was my suggestion for adding arbitrary node parameters. But having seen your #181 , should we actually be adding node parameters on a groupvar level, rather than in partitions, to avoid the same clash issue with nodes in multiple partitions?

I personally think that using group and host vars would be a more natural way of setting per host configuration options. I am now wondering if that is consistent with what we currently do with gres which is a property on the NodeName line, but defined as part of the partition definition. To switch to the host/group vars approach I think we'd need to support something like:

# group_vars/inventory/a100/openhpc

# fallback to value defined on partition if this isn't defined for backwards compatibility
openhpc_gres:
    - conf: "gpu:A100:2"
      file:  /dev/nvidia[0-1]

Are there any other inconsistencies like this?

So you might argue that this PR is consistent with the current approach and I should change mine to work similarly. Like you commented on my PR, I could make mine error out if a inconsistency was detected. On the other hand, this could be a good opportunity to change path before we describe too many properties in the partition definition if we think that using ansible inventory is the better approach.

If using host_vars/groups vars, it could look something like this to the end user:

# group_vars/inventory/a100/openhpc
openhpc_node_params:
    Features: "gpu,a100"

[sjpb]

Using inventory does seem more "correct", TBH. I think the complication is that actually the inventory groups referenced in openhpc_slurm_partitions actually have openhpc_cluster_name + '_' prefix. Which is annoying in the appliance where the openhpc_cluster_name is usually used to distinguish staging/production. Obvs per-node vars could be defined for inventory groups which don't have those prefixes, but currently TF doesn't auto-define those groups (I was wondering if it should for another client where we need to set vars for os-level things on the tf group anyway).

[sjpb]

I don't think I mind about removing gres from the partition config being backward incompatible at least (although I do generally try hard to avoid that). Between the people in this thread we've got awareness of all appliances using it.

[sjpb]

Merging so I can fix CI