Create a standard metadata interface for RM jobid

[mplegendre]

Suggestion, make a different metadata name/value pair for each resource manager that could be present:

slurm_jobid: XXXXX
flux_jobid: YYYYYY
lsf_jobid: ZZZZZZ

We should give some thought to flux subjob metadata. Though it may not be critical if we don't capture that.

[baallan]

WRT slurm, at least, uniqueness requires also collecting (or joining later, which is often troublesome) SLURM_CLUSTER_NAME and job submit time. I suspect similar for LSF, but I know nothing useful about what would make a flux_jobid unique.

[mplegendre]

We've got clustername and launchdate as already collected metadata. Launchdate will be similar to submit time, but not precisely.

Though I recollect that clustername is implemented by removing numbers from the hostname. That may need to change due to systems like Cray. Perhaps it should be changed to using the environment variable SLURM_CLUSTER_NAME, then hostname if that isn't present.

And I don't think there will be many issues if we're not complete in our LSF implementation.

[baallan]

WRT job start time vs slurm submit time, neither are easily/reliably available at runtime afaict, without calling on the slurm accounting service. Also, submit time is a bit more useful when correlating data about the various kinds of slurm array jobs/subjobs. Oy, are RMs a pain...

$(SLURM_TIME_FORMAT=%s sacct -j $JOBID --format=Submit -n -P)

is one way to get the submit time; can be extended to get other fields along with Submit.