Use Slurm array jobs to limit concurrent extraction jobs

[takluyver]

When reprocessing a whole bunch of runs at once, submit them as a Slurm array job, and ask Slurm to limit how many will run at once. This should mitigate the 'database locked' errors, without requiring people to manually batch their jobs and monitor their progress.

The limit is set at 30 concurrent jobs for now. Obviously we can make that configurable.

I've taken the shortcut of using the array task IDs for the run numbers. This avoids having to define a way to communicate 'array task 1 -> run 53', but it does mean that only the run numbers can vary within an array. So the one job we ask to update the variables table in the DB is submitted separately, and if you do reprocess all on a database spanning multiple proposals, each proposal would be submitted separately, raising the effective limit to N * 30 jobs. That's not ideal, but so far it's largely theoretical.

Closes #96.

[codecov]

Codecov Report

Attention: Patch coverage is 88.46154% with 6 lines in your changes missing coverage. Please review.

Project coverage is 75.27%. Comparing base (c295f8d) to head (579743e).
Report is 74 commits behind head on master.

Files with missing lines	Patch %	Lines
damnit/backend/extraction_control.py	89.79%	5 Missing ⚠️
damnit/gui/main_window.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #335      +/-   ##
==========================================
+ Coverage   74.81%   75.27%   +0.46%     
==========================================
  Files          32       32              
  Lines        4892     4959      +67     
==========================================
+ Hits         3660     3733      +73     
+ Misses       1232     1226       -6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[takluyver]

😮‍💨 no, we can't use array task IDs directly as run numbers, because:

The minimum index value is 0. the maximum value is one less than the configuration parameter MaxArraySize.

MaxArraySize            = 1001

So this will need a rework.

[takluyver]

OK, now it creates one script per run in .tmp under the AMORE directory. The scripts clean themselves up when they run, so we don't get hundreds of tiny files left around. This allows more diverse jobs to be submitted as one array.

[JamesWrigley]

I'm increasingly worried by this swathe of code that's entirely untested 😅 Maybe in the future we can add some tests that'll only run on Maxwell. Anyway, LGTM!

[takluyver]

Good point, I'll try to figure out some ways to test this stuff better. Thanks for the review!