Use Slurm jobs for reprocessing #270
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hcraT
reviewed
Jul 1, 2024
|
LGTM! |
|
Thanks! |
|
I am deploying this to beta. I'm in the midst of processing a whole lot of runs for some old proposals, this should be a big help. |
|
This is pretty nice so far - I can process a whole bunch of runs in parallel. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In #256 we made the listener submit jobs to the Solaris cluster to process data, rather than running subprocesses wherever it's running. This extends that to extraction run by
amore-proto reprocessas well, so restrictions on CPU & memory usage apply consistently, however the processing was launched.Reprocessing jobs now run in parallel by default (superseding #250), limited by how quickly the solaris cluster starts jobs. A new
--watchflag switches back to processing one run at a time and showing live output directly in the terminal, but still uses Slurm with thesruncommand.I see this as a step towards launching reprocessing from the GUI, and having the status of processing jobs visible in the GUI. But I'm leaving those features for a later PR.
I've also refactored a bit. The
extract_datamodule is now only the code that runs inside a Slurm job, whileextraction_controlis the code to submit Slurm jobs.