ModCon Segmentation

[davramov]

For the ModCon Segmentation demo, we will need to update our bl832 NERSC code to incorporate these new features.

In orchestration/flows/bl832/nersc.py, I propose adding a new function to NERSCTomographyHPCController called segmentation().

• This function will include a SLURM SFAPI job with a reference to the Docker image that contains the dependencies for segmentation. We can use an existing SLURM job as a starting point, but we will need to modify the allocation name, the compute type (CPU -> GPU), and number of nodes. I recommend using Shifter instead of Podman.
• We can store the reference to the segmentation image in config.yml, which will be loaded to Config832. I should note it is important to use a specific image tag (not latest) to ensure that it can be cached by shifter and not pulled each time the job is called.

I have an open PR with my Multinode Reconstruction optimizations: #111

• In reconstruct_multinode(), I am using Shifter, so this would be a good starting point for the segmentation SLURM job.
• There is an additional function for pulling the image once with Shifter, so it can be cached.

Also in orchestration/flows/bl832/nersc.py, we need to modify the Prefect flow nersc_reconstruction_flow to trigger segmentation inference (e.g. segmentation_success = controller.segment(...)) after reconstruction completes.

• As Dylan mentioned, we may want to disable Zarr conversion for the demo.