Exporter fails due to "AllocGRES is deprecated" fatal error

[tgooding]

The latest prometheus-slurm-exporter runs for a few seconds before terminating with a fatal error:

prometheus-slurm-exporter/bin/prometheus-slurm-exporter   
INFO[0000] Starting Server: :8080                        source="main.go:48"
FATA[0004] exit status 1                                 source="gpus.go:101"

I'm running slurm-20.11.3-1 and a rebuild picked up the new gpus.go module. Digging into it a bit, it appears the Allocgres option to sacct is treated as fatal, which causes the Execute() routine to terminate:

sh-4.4$ sacct -a -X --format=Allocgres --state=RUNNING --noheader --parsable2
sacct: fatal: AllocGRES is deprecated, please use AllocTRES

[mtds]

Interesting. At the moment I do not have a chance to test it, since we are still running with version 18.08.8. I can try to reproduce it on VMs and eventually look for a workaround but I guess the only solution (not particularly nice) would be with an if/else based on the output of sacct.

[biocyberman]

I faced the same problem because I am running SLURM 20.11.3. The problem is at this line: https://github.com/vpenso/prometheus-slurm-exporter/blob/master/gpus.go#L41
I changed it to
args := []string{"-a", "-X", "--format=AllocTRES", "--state=RUNNING", "--noheader", "--parsable2"}
Rebuild, reinstall ant it works.

[mtds]

This can be a thorny issue: we rely on the output provided by the Slurm command lines utilities. Whenever the developers from SchedMD change the format or drop some options, this exporter is not able to cope with.

As it seem also from issue #38 , at the moment it can be guaranteed that this exporter will run on Slurm version 18.x.x. On higher version there may be problems.

[lahwaacz]

I changed it to
args := []string{"-a", "-X", "--format=AllocTRES", "--state=RUNNING", "--noheader", "--parsable2"}
Rebuild, reinstall ant it works.

It's not that simple, the format of the output changed as well. More info is already in #38.