This repository is based on the contribution of Gregor Sturm.
In my case, I need to execute my deep learning codes in our High Performance Computing (HPC) cluster and it is required to
- use singularity to containerize an existing conda environment;
- make the image based on Ubuntu (because the existing conda environment is in my local machine with Ubuntu OS);
- activate automatically the containerized conda environment once the container is started.
I tried "conda-pack" but it doesn't work, therefore I applied conda_to_singularity.py given by Gregor Sturm without modifying anything and edited his singularity.template to fit in my case. The file singularity.template in the current repository is the edited one.
(exactly the same way as in Gregor Sturm's repository.)
usage: conda_to_singularity.py [-h] [--template TEMPLATE] CONDA_ENV OUTPUT_CONTAINER
Convert a conda env to a singularity container.
positional arguments:
CONDA_ENV Absolute path to the conda enviornment. Must be exactely the path as it shows up in `conda env list`, not a symbolic link to it, nor a realpath.
OUTPUT_CONTAINER Output path where the singularity container will be safed.
optional arguments:
-h, --help show this help message and exit
--template TEMPLATE Path to a Singularity template file. Must contain a `{conda_env}` placeholder. If not specified, uses the default template shipped with this script.
For example
conda_to_singularity.py /home/sturm/.conda/envs/whatever whatever.sif
By default, the image will be based on Ubuntu. If you want a different base image,
you can modify Singularity.template
, and specify it with the --template
argument.
- modify the Header of the singularity definition file to
Bootstrap: debootstrap
MirrorURL: http://us.archive.ubuntu.com/ubuntu/
OSVersion: jammy
where
-
the mandatory Bootstrap keyword describes the bootstrap module to use. The debootstrap build agent module allows us to build a Debian/Ubuntu style container from a mirror URI. And it is mandatory to specify the OS version and a URI for the mirror when we use the debootstrap module to specify a base for a Debian-like container.
-
jammy corresponds to Ubuntu 22.04 according to the website of the Ubuntu releases. Similarily you can use other code words like trusty (14.04), xenial (16.04), and yakkety (17.04) for Ubuntu. (More info. can be found here.)
- modify the post section as
%post
apt-get update
apt-get install -y tar curl
The originally repository by Gregor Sturm has the line below in the singularity definition file to append a line to the Singularity environment file in order to activates the conda environment once the container is started
echo "source /opt/conda/bin/activate {conda_env}" >>$SINGULARITY_ENVIRONMENT
However, this doesn't work for me because of two reasons:
- my UNIX shell does not support source. instead, it support .
- activate cannot be executed in my case. And in the updated version, I use the two lines below to replace the line above,
echo ". /opt/conda/etc/profile.d/conda.sh" >>$SINGULARITY_ENVIRONMENT
echo "conda activate {conda_env}" >>$SINGULARITY_ENVIRONMENT
where the renewed version appends the conda command to the singularity environment and then use conda command to do activation. Because conda activate is also written in the singularity environment, this command would be executed once the container is started.
- In the environment section, I added
export LC_ALL=C
- In the post section, I modified miniconda to be the latest version
curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh > /install_conda.sh
- In the post section, I added
chmod -R o+rx /opt/conda