diff --git a/content/notes/containers-for-hpc/_index.md b/content/notes/containers-for-hpc/_index.md index c838cc65..2dcd0a32 100644 --- a/content/notes/containers-for-hpc/_index.md +++ b/content/notes/containers-for-hpc/_index.md @@ -1,14 +1,10 @@ --- title: "Software Containers for HPC" -summary: "An Introduction to using and building software containers." authors: [rs] -categories: ["Containers","HPC"] -tags: [Containers, HPC] toc: true type: docs date: "2025-05-15T00:00:00" weight: 1 - menu: containers-for-hpc: name: Software Containers for HPC diff --git a/content/notes/containers-for-hpc/building-apptainer.md b/content/notes/containers-for-hpc/building-apptainer.md index 0670a480..8a492c02 100644 --- a/content/notes/containers-for-hpc/building-apptainer.md +++ b/content/notes/containers-for-hpc/building-apptainer.md @@ -4,9 +4,10 @@ title: "Building Containers on HPC [Apptainer]" toc: true type: docs weight: 4 + menu: containers-for-hpc: - parent: Software Containers for HPC + parent: Software Containers for HPC --- @@ -114,7 +115,7 @@ This is the very first entry. It defines the bootstrap agent: - `docker` - `library` - `shub` -- and [many more](https://apptainer.org/docs/user/latest/definition_files.html#preferred-bootstrap-agents) +- and [many more (opens in new tab)](https://apptainer.org/docs/user/latest/definition_files.html#preferred-bootstrap-agents) #### `From` (mandatory) Define the base container. @@ -342,7 +343,7 @@ Save this as `lolcow_2.def`. - You may need to specify extra packages - `fortune` itself provides the executable without the message database - `fortunes-min` contains the message database -- See [how Ubuntu reduced image size by 60%](https://ubuntu.com/blog/we-reduced-our-docker-images-by-60-with-no-install-recommends) +- See [how Ubuntu reduced image size by 60% (opens in new tab)](https://ubuntu.com/blog/we-reduced-our-docker-images-by-60-with-no-install-recommends) ### Image size comparison @@ -447,21 +448,21 @@ $ apptainer push lolcow_0.sif oras://myname/lolcow:0 ``` Check: -`https://hub.docker.com/r/myname/lolcow/tags` +[Docker Hub tags for `myname/lolcow` (opens in new tab)](https://hub.docker.com/r/myname/lolcow/tags) -{{< info >}}Best practice: Sign your containers; see [here](https://apptainer.org/docs/user/latest/signNverify.html).{{< /info >}} +{{< info >}}Best practice: Sign your containers; see [Apptainer signing and verification documentation (opens in new tab)](https://apptainer.org/docs/user/latest/signNverify.html).{{< /info >}} {{< warning >}}While Apptainer can convert a Docker image into SIF, you cannot run SIF with Docker. You are simply using Docker Hub to host your SIF - it is not converted into Docker.{{< /warning >}} #### UVA Research Computing container resources -[Docker Hub account](https://hub.docker.com/u/uvarc) +[Docker Hub account (opens in new tab)](https://hub.docker.com/u/uvarc) -[Repository of Dockerfiles and definition files](https://github.com/uvarc/rivanna-docker) +[Repository of Dockerfiles and definition files (opens in new tab)](https://github.com/uvarc/rivanna-docker) ### GitHub Packages -1. [Create a personal access token](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-with-a-personal-access-token-classic) +1. [Create a personal access token (opens in new tab)](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-with-a-personal-access-token-classic) 1. Login ```bash apptainer remote login --username myname docker://ghcr.io @@ -471,7 +472,7 @@ Check: ```bash apptainer push lolcow_0.sif oras://ghcr.io/myname/lolcow ``` -1. Check `https://github.com/users/myname/packages/container/package/lolcow` +1. Check [your GitHub package page (opens in new tab)](https://github.com/users/myname/packages/container/package/lolcow) ## Case Studies @@ -487,16 +488,16 @@ While you can create a conda environment locally, you cannot directly migrate it Your project requires PyTorch, Numpy, Seaborn, and Pandas. Write the corresponding Apptainer definition file. Hints: -- Find the appropriate base image from [here](https://hub.docker.com/r/pytorch/pytorch/tags). +- Find the appropriate base image from [PyTorch Docker image tags on Docker Hub (opens in new tab)](https://hub.docker.com/r/pytorch/pytorch/tags). - Pull the base image and examine it first. Does it already provide some packages? {{< info >}}While PyTorch runs on a GPU, you do not need to build the container on a GPU.{{< /info >}} -{{< info >}}You will likely run out of memory when building large containers (over a few GBs). Request an [interactive job](https://www.rc.virginia.edu/userinfo/rivanna/slurm/#submitting-an-interactive-job) with say `--mem=50G` to build on a compute node with 50 GB memory.{{< /info >}} +{{< info >}}You will likely run out of memory when building large containers (over a few GBs). Request an [interactive job (opens in new tab)](https://www.rc.virginia.edu/userinfo/rivanna/slurm/#submitting-an-interactive-job) with say `--mem=50G` to build on a compute node with 50 GB memory.{{< /info >}} ### R -Rocker provides many base images for all R versions (see [here](https://rocker-project.org/images/)): +Rocker provides many base images for all R versions (see [Rocker image catalog (opens in new tab)](https://rocker-project.org/images/)): - `rocker/r-ver`: basic R installation - `rocker/rstudio`: with RStudio Server - `rocker/tidyverse`: plus tidyverse and dependencies @@ -515,14 +516,14 @@ Hints: ## Multistage Build -By distinguishing between buildtime-dependencies vs runtime-dependencies, it is possible to reduce the image size drastically via a [multistage build](https://apptainer.org/docs/user/latest/definition_files.html#multi-stage-builds). (My experience with some extreme cases is that only 1% is needed at runtime.) +By distinguishing between buildtime-dependencies vs runtime-dependencies, it is possible to reduce the image size drastically via a [multistage build (opens in new tab)](https://apptainer.org/docs/user/latest/definition_files.html#multi-stage-builds). (My experience with some extreme cases is that only 1% is needed at runtime.) This is beyond the scope of the workshop, but you are welcome to browse the Apptainer documentation and Appendix 2 on Minimal Containers. --- ## References -- [Apptainer User Guide](https://apptainer.org/docs/user/latest) -- [Definition File](https://apptainer.org/docs/user/latest/definition_files.html) -- [Definition File vs Dockerfile](https://apptainer.org/docs/user/latest/docker_and_oci.html#apptainer-definition-file-vs-dockerfile) -- [GitHub Packages](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry) +- [Apptainer User Guide (opens in new tab)](https://apptainer.org/docs/user/latest) +- [Definition File (opens in new tab)](https://apptainer.org/docs/user/latest/definition_files.html) +- [Definition File vs Dockerfile (opens in new tab)](https://apptainer.org/docs/user/latest/docker_and_oci.html#apptainer-definition-file-vs-dockerfile) +- [GitHub Packages (opens in new tab)](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry) diff --git a/content/notes/containers-for-hpc/building-docker.md b/content/notes/containers-for-hpc/building-docker.md index ed3554d5..23191b71 100644 --- a/content/notes/containers-for-hpc/building-docker.md +++ b/content/notes/containers-for-hpc/building-docker.md @@ -21,7 +21,7 @@ Starting from a simple Dockerfile, we will adopt best practices sequentially and 2. Install Docker - To follow along on your own computer, please install Docker Desktop and register for a free account on Docker Hub. Both can be found [here](https://www.docker.com/get-started). + To follow along on your own computer, please install Docker Desktop and register for a free account on Docker Hub. Both are available from [Docker Get Started (opens in new tab)](https://www.docker.com/get-started). After the installation, open a terminal ("cmd" on Windows) and make sure you can execute the command `docker run hello-world` successfully. @@ -38,7 +38,7 @@ Starting from a simple Dockerfile, we will adopt best practices sequentially and - Singularity - Users cannot build on Rivanna (needs `sudo` privilege) - Singularity Library/Hub (more limitations) - - Refer to [workshop](https://learning.rc.virginia.edu/workshops/singularity/) in Spring 2020 + - Refer to [workshop (opens in new tab)](https://learning.rc.virginia.edu/workshops/singularity/) in Spring 2020 ## Intro to Dockerfile: lolcow @@ -65,7 +65,7 @@ FROM ubuntu:22.04 - Doesn't have to be a bare OS - `python`, `continuumio/miniconda3`, `node`, `nvidia/cuda`, etc. -[Dockerfile reference](https://docs.docker.com/engine/reference/builder/#format) +[Dockerfile reference (opens in new tab)](https://docs.docker.com/engine/reference/builder/#format) ### Steps 2 & 3: Install software @@ -237,14 +237,14 @@ ENTRYPOINT fortune | cowsay | lolcat - You may need to specify extra packages - `fortune` itself provides the executable without the message database - `fortunes-min` contains the message database -- See [how Ubuntu reduced image size by 60%](https://ubuntu.com/blog/we-reduced-our-docker-images-by-60-with-no-install-recommends) +- See [how Ubuntu reduced image size by 60% (opens in new tab)](https://ubuntu.com/blog/we-reduced-our-docker-images-by-60-with-no-install-recommends) ### 3. Use a smaller base image For installation of common packages, you may consider Alpine. - BusyBox + package manager + musl libc (beware of compatibility issues) -- [Presentation](https://youtu.be/sIG2P9k6EjA) on Alpine Linux from DockerCon EU 17 +- [Presentation (opens in new tab)](https://youtu.be/sIG2P9k6EjA) on Alpine Linux from DockerCon EU 17 Look for `slim` variants (e.g. `debian:buster-slim`) of a base image, if any. @@ -281,7 +281,7 @@ $ docker images | grep lolcow | sort -nk 2 | awk '{print $1, $2, $NF}' |2 |Combination of previous two |53 | 26 | |3 |Alpine base image |161| 78 | -Reference: [_Best practices for writing Dockerfiles_](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) +Reference: [_Best practices for writing Dockerfiles_ (opens in new tab)](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) ### Summary @@ -309,11 +309,11 @@ docker push /lolcow:latest ### Docker Hub interface -In your browser, go to `https://hub.docker.com/r//lolcow`. +In your browser, go to [Docker Hub for `/lolcow` (opens in new tab)](https://hub.docker.com/r//lolcow). - Overview: - Sync with GitHub to update `README.md`; or - - Use [docker-pushrm](https://github.com/christian-korneck/docker-pushrm) + - Use [docker-pushrm (opens in new tab)](https://github.com/christian-korneck/docker-pushrm) - Tags: - List all versions - View image history if Dockerfile not provided @@ -327,23 +327,23 @@ By now, we know how to write a simple Dockerfile to install software using the d ### Compiled language (C++) -https://github.com/lilab-bcb/cumulus_feature_barcoding +[cumulus_feature_barcoding repository (opens in new tab)](https://github.com/lilab-bcb/cumulus_feature_barcoding) Hints: - You do not have to start from a bare OS. Search for `gcc` on Docker Hub. - Install `build-essential` if you are starting from a bare OS. -- Version pinning - to choose a specific version, download from https://github.com/lilab-bcb/cumulus_feature_barcoding/releases (you will need `wget`). +- Version pinning - to choose a specific version, download from [cumulus_feature_barcoding releases (opens in new tab)](https://github.com/lilab-bcb/cumulus_feature_barcoding/releases) (you will need `wget`). ### Interpreted language (Python) -https://docs.qiime2.org/2022.8/install/native/#install-qiime-2-within-a-conda-environment +[QIIME 2 native install guide (opens in new tab)](https://docs.qiime2.org/2022.8/install/native/#install-qiime-2-within-a-conda-environment) Hints: - Click on "Linux" to get the URL for the yaml file. Download the yaml file in the same directory as your Dockerfile. - You do not have to start from a bare OS in your Dockerfile. Search for `miniconda3` on Docker Hub. -- (Recommended) There is a much faster dependency solver than conda - micromamba. See [here](https://micromamba-docker.readthedocs.io/en/latest/quick_start.html) for instructions. +- (Recommended) There is a much faster dependency solver than conda - micromamba. See the [Micromamba Docker quick start guide (opens in new tab)](https://micromamba-docker.readthedocs.io/en/latest/quick_start.html) for instructions. - Use the suggested `COPY` and `ENTRYPOINT` statements. -- After you're done, compare with the [official Dockerfile](https://github.com/qiime2/vm-playbooks/blob/0fda9dce42802596756986e2f80c38437872c66e/docker/Dockerfile) and image size. What is the biggest reason for the difference? +- After you're done, compare with the [official Dockerfile (opens in new tab)](https://github.com/qiime2/vm-playbooks/blob/0fda9dce42802596756986e2f80c38437872c66e/docker/Dockerfile) and image size. What is the biggest reason for the difference? ### General Remarks @@ -351,7 +351,7 @@ Hints: - If you encounter a Docker statement that you have not used before, first check the official documentation for best practices. - A comprehensive list of dependencies may be lacking. Some developers may not specify any at all. You will have to rely on a combination of experience, error message, and web search. (Most likely all of the above.) - Especially for Python packages, versions may be too permissive or too restrictive such that, in either case, future installation of the application will fail. (I have encountered both.) Tweak the versions until it works. -- The next step is "multi-stage build" which is covered in the [Minimal Containers](../minimal) workshop. There you will learn how to distinguish between buildtime versus runtime dependencies and separate them out. +- The next step is "multi-stage build" which is covered in the [Minimal Containers](/notes/containers-for-hpc/minimal) workshop. There you will learn how to distinguish between buildtime versus runtime dependencies and separate them out. ## Clean Up @@ -375,8 +375,8 @@ If you build containers often, you can run out of disk space quickly. To clean u ## References -- [UVA Rivanna-Docker GitHub](https://github.com/uvarc/rivanna-docker) +- [UVA Rivanna-Docker GitHub (opens in new tab)](https://github.com/uvarc/rivanna-docker) - Dockerfiles by UVA Research Computing - - [Tips](https://github.com/uvarc/rivanna-docker/wiki/Tips) -- [_Best practices for writing Dockerfiles_](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) -- [Natanael Copa, _Small, Simple, and Secure: Alpine Linux under the Microscope_, DockerCon EU (2017)](https://youtu.be/sIG2P9k6EjA) + - [Tips (opens in new tab)](https://github.com/uvarc/rivanna-docker/wiki/Tips) +- [_Best practices for writing Dockerfiles_ (opens in new tab)](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) +- [Natanael Copa, _Small, Simple, and Secure: Alpine Linux under the Microscope_, DockerCon EU (2017) (opens in new tab)](https://youtu.be/sIG2P9k6EjA) diff --git a/content/notes/containers-for-hpc/intro.md b/content/notes/containers-for-hpc/intro.md index 78348f66..5c031131 100755 --- a/content/notes/containers-for-hpc/intro.md +++ b/content/notes/containers-for-hpc/intro.md @@ -7,11 +7,10 @@ weight: 2 menu: containers-for-hpc: - parent: Software Containers for HPC - + parent: Software Containers for HPC --- -"A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another." () +"A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another." ([Docker definition of a container (opens in new tab)](https://www.docker.com/resources/what-container)) ## Why use software containers? diff --git a/content/notes/containers-for-hpc/minimal.md b/content/notes/containers-for-hpc/minimal.md index d841d2d1..b9a47a7f 100644 --- a/content/notes/containers-for-hpc/minimal.md +++ b/content/notes/containers-for-hpc/minimal.md @@ -48,17 +48,17 @@ pip install --no-cache-dir ... ``` ### 3. Base image -- For OS and common tools (e.g. python/conda, GCC) start from [official image](https://docs.docker.com/docker-hub/official_images/) +- For OS and common tools (e.g. python/conda, GCC) start from [official image (opens in new tab)](https://docs.docker.com/docker-hub/official_images/) - Do not reinvent the wheel - Know which variant to use (e.g. `devel` vs `runtime`, `slim`) - Read their overview ## Exercise: QIIME 2 -QIIME 2 is a popular bioinformatics software. Can you suggest any potential improvement to the [Dockerfile](https://github.com/qiime2/vm-playbooks/blob/0fda9dce42802596756986e2f80c38437872c66e/docker/Dockerfile)? +QIIME 2 is a popular bioinformatics software. Can you suggest any potential improvement to the [Dockerfile (opens in new tab)](https://github.com/qiime2/vm-playbooks/blob/0fda9dce42802596756986e2f80c38437872c66e/docker/Dockerfile)?
Answer -With our [Dockerfile](https://github.com/uvarc/rivanna-docker/blob/master/qiime2/2020.8/Dockerfile), we managed to reduce the image size by half. +With our [Dockerfile (opens in new tab)](https://github.com/uvarc/rivanna-docker/blob/master/qiime2/2020.8/Dockerfile), we managed to reduce the image size by half.
@@ -84,7 +84,7 @@ The objective is to minimize image size without loss of functionality. What's ne - only install runtime dependencies (e.g. `cmake`, `go` not needed) - copy software from build stage -Reference: [_Use multi-stage builds_](https://docs.docker.com/develop/develop-images/multistage-build/) +Reference: [_Use multi-stage builds_ (opens in new tab)](https://docs.docker.com/develop/develop-images/multistage-build/) ### Multi-stage Dockerfile template @@ -109,11 +109,11 @@ ENTRYPOINT ["binary"] ## Exercise: LightGBM LightGBM is a gradient boosting framework that uses tree based learning algorithms. It is an open source project by Microsoft. -1. Examine the [official Dockerfile](https://github.com/microsoft/LightGBM/blob/master/docker/gpu/dockerfile.gpu). Can you identify any problems? +1. Examine the [official Dockerfile (opens in new tab)](https://github.com/microsoft/LightGBM/blob/master/docker/gpu/dockerfile.gpu). Can you identify any problems?
Answer - - `nvidia/cuda:cudnn-devel` as [base image](https://hub.docker.com/r/nvidia/cuda/tags) (>1 GB) + - `nvidia/cuda:cudnn-devel` as [base image (opens in new tab)](https://hub.docker.com/r/nvidia/cuda/tags) (>1 GB) - Clean up in separate `RUN` statements
@@ -205,7 +205,7 @@ LightGBM is a gradient boosting framework that uses tree based learning algorith
-3. Rewrite the Dockerfile using a multi-stage build based on [OpenCL](https://hub.docker.com/r/nvidia/cuda/tags). +3. Rewrite the Dockerfile using a multi-stage build based on [OpenCL (opens in new tab)](https://hub.docker.com/r/nvidia/cuda/tags). - Use `nvidia/opencl:devel` as the build stage - Remove everything related to CUDA since it is not relevant - Redefine the OpenCL environment variables as: @@ -216,7 +216,7 @@ LightGBM is a gradient boosting framework that uses tree based learning algorith - Keep the same dependencies - Remember to clean up at the right place - Ignore the command under `# Add OpenCL ICD files for LightGBM` - - Instead of `mkdir foo && cd foo` use `WORKDIR foo` (see [documentation](https://docs.docker.com/engine/reference/builder/#workdir)) + - Instead of `mkdir foo && cd foo` use `WORKDIR foo` (see [documentation (opens in new tab)](https://docs.docker.com/engine/reference/builder/#workdir)) - Use the same command to install LightGBM, but replace the paths with the corresponding OpenCL environment variables - Add an `ENTRYPOINT` - Use `nvidia/opencl:runtime` as the production stage @@ -274,7 +274,7 @@ LightGBM is a gradient boosting framework that uses tree based learning algorith
-4. Challenge: Verify that the two containers have the same performance on a GPU node. Follow the [tutorial example](https://lightgbm.readthedocs.io/en/latest/GPU-Tutorial.html#dataset-preparation). Run the same job without using GPU. How much faster is it with GPU? +4. Challenge: Verify that the two containers have the same performance on a GPU node. Follow the [tutorial example (opens in new tab)](https://lightgbm.readthedocs.io/en/latest/GPU-Tutorial.html#dataset-preparation). Run the same job without using GPU. How much faster is it with GPU? --- @@ -294,12 +294,12 @@ LightGBM is a gradient boosting framework that uses tree based learning algorith - Build production ### Example base images -- [Scratch](https://hub.docker.com/_/scratch) +- [Scratch (opens in new tab)](https://hub.docker.com/_/scratch) - Literally start from scratch! - A "no-op" in the Dockerfile, meaning no extra layer in image - Build other base images (beyond scope of this workshop) - Minimal image with a single application -- [Distroless](https://github.com/GoogleContainerTools/distroless) +- [Distroless (opens in new tab)](https://github.com/GoogleContainerTools/distroless) - Derived from Debian - Support for C/C++, Go, Rust, Java, Node.js, Python @@ -330,8 +330,8 @@ This exercise illustrates how we can cherry-pick files from the package manager
For your reference, the content of the packages can be found here: - - fortune-mod [package list](https://packages.ubuntu.com/xenial/all/fortune-mod/filelist) - - fortunes-min [package list](https://packages.ubuntu.com/xenial/all/fortunes-min/filelist) + - fortune-mod [package list (opens in new tab)](https://packages.ubuntu.com/xenial/all/fortune-mod/filelist) + - fortunes-min [package list (opens in new tab)](https://packages.ubuntu.com/xenial/all/fortunes-min/filelist)
@@ -443,21 +443,21 @@ The image size is merely **10.7 MB**, 99.5% smaller than what we started with. T -We submitted a [pull request](https://github.com/microsoft/LightGBM/pull/3408) that has been [merged](https://github.com/microsoft/LightGBM/tree/master/docker/gpu). +We submitted a [LightGBM GPU Docker pull request (opens in new tab)](https://github.com/microsoft/LightGBM/pull/3408) that has been [merged into the LightGBM Docker GPU directory (opens in new tab)](https://github.com/microsoft/LightGBM/tree/master/docker/gpu). --- ## Dynamic vs Static Linking -The above procedure, while impressive, may be tedious for the average user. All the examples so far are based on [dynamic linking](https://en.wikipedia.org/wiki/Dynamic_linker), where the shared libraries of an executable are stored separately. If you are compiling code from source, you may choose to build a static binary (e.g. `-static` in GCC) so that all the necessary libraries are built into the binary. +The above procedure, while impressive, may be tedious for the average user. All the examples so far are based on [dynamic linking (opens in new tab)](https://en.wikipedia.org/wiki/Dynamic_linker), where the shared libraries of an executable are stored separately. If you are compiling code from source, you may choose to build a static binary (e.g. `-static` in GCC) so that all the necessary libraries are built into the binary. ## Exercise: Linking against OpenBLAS -OpenBLAS is a linear algebra library. The code in this exercise is taken from its [user manual](https://github.com/xianyi/OpenBLAS/wiki/User-Manual#code-examples). It is based on `dgemm` which performs the matrix operation: +OpenBLAS is a linear algebra library. The code in this exercise is taken from its [user manual (opens in new tab)](https://github.com/xianyi/OpenBLAS/wiki/User-Manual#code-examples). It is based on `dgemm` which performs the matrix operation: $$ \alpha A B + \beta C, $$ -where $A_{mk}, B_{kn}, C_{mn}$ are matrices and $\alpha, \beta$ are constants. For details please visit the [Intel tutorial page](https://software.intel.com/content/www/us/en/develop/documentation/mkl-tutorial-c/top/multiplying-matrices-using-dgemm.html). +where $A_{mk}, B_{kn}, C_{mn}$ are matrices and $\alpha, \beta$ are constants. For details please visit the [Intel tutorial page (opens in new tab)](https://software.intel.com/content/www/us/en/develop/documentation/mkl-tutorial-c/top/multiplying-matrices-using-dgemm.html). 1. Select an appropriate base image. (Hint: You will be compiling C++ code.) @@ -568,7 +568,7 @@ where $A_{mk}, B_{kn}, C_{mn}$ are matrices and $\alpha, \beta$ are constants. F This exercise illustrates that it is easier to build a minimal container of a static binary. ## Exercise: Linking against LibTorch -LibTorch is the C++ frontend of PyTorch. This exercise is based on the ["Writing a Basic Application"](https://pytorch.org/tutorials/advanced/cpp_frontend.html#writing-a-basic-application) section of the PyTorch tutorial. +LibTorch is the C++ frontend of PyTorch. This exercise is based on the ["Writing a Basic Application" (opens in new tab)](https://pytorch.org/tutorials/advanced/cpp_frontend.html#writing-a-basic-application) section of the PyTorch tutorial. 1. Select an appropriate base image. (Hint: You will be compiling C++ code.) @@ -576,7 +576,7 @@ LibTorch is the C++ frontend of PyTorch. This exercise is based on the ["Writing - Build tools: `build-essential cmake` - Download and decompress: `wget ca-certificates unzip` -1. Find the download link for LibTorch under the "Install PyTorch" section at https://pytorch.org/. Select "None" for CUDA. Download the file to `/opt` in the image. Hints: +1. Find the download link for LibTorch under the "Install PyTorch" section at [pytorch.org (opens in new tab)](https://pytorch.org/). Select "None" for CUDA. Download the file to `/opt` in the image. Hints: - In your `wget` command, you may want to rename the output file using `-O libtorch.zip`. - Remember to decompress. @@ -703,10 +703,10 @@ LibTorch is the C++ frontend of PyTorch. This exercise is based on the ["Writing ## References -- [UVA Rivanna-Docker GitHub](https://github.com/uvarc/rivanna-docker) +- [UVA Rivanna-Docker GitHub (opens in new tab)](https://github.com/uvarc/rivanna-docker) - Dockerfiles by UVA Research Computing - - [Tips](https://github.com/uvarc/rivanna-docker/wiki/Tips) -- [_Best practices for writing Dockerfiles_](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) -- [_Use multi-stage builds_](https://docs.docker.com/develop/develop-images/multistage-build/) -- [Google Distroless GitHub](https://github.com/GoogleContainerTools/distroless) -- [Matthew Moore, _Distroless Docker: Containerizing Apps, not VMs_, swampUP (2017)](https://youtu.be/lviLZFciDv4) + - [Tips (opens in new tab)](https://github.com/uvarc/rivanna-docker/wiki/Tips) +- [_Best practices for writing Dockerfiles_ (opens in new tab)](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) +- [_Use multi-stage builds_ (opens in new tab)](https://docs.docker.com/develop/develop-images/multistage-build/) +- [Google Distroless GitHub (opens in new tab)](https://github.com/GoogleContainerTools/distroless) +- [Matthew Moore, _Distroless Docker: Containerizing Apps, not VMs_, swampUP (2017) (opens in new tab)](https://youtu.be/lviLZFciDv4) diff --git a/content/notes/containers-for-hpc/using.md b/content/notes/containers-for-hpc/using.md index 869a6414..13b73770 100755 --- a/content/notes/containers-for-hpc/using.md +++ b/content/notes/containers-for-hpc/using.md @@ -34,7 +34,7 @@ To download a container hosted on a registry, use the `pull` command. Docker ima - `` (Unified resource identifiers) - `[library|docker|shub]://[/][:] ` - - Default prefix: `library` ([Singularity Library](https://cloud.sylabs.io/library)) + - Default prefix: `library` ([Singularity Library (opens in new tab)](https://cloud.sylabs.io/library)) - `user`: optional; may be empty (e.g. `apptainer pull ubuntu`) - `tag`: optional; default: `latest` - `` (Singularity image format) @@ -264,7 +264,7 @@ sbatch tensorflow-2.17.0.slurm #### What does `--nv` do? -See [Apptainer GPU user guide](https://apptainer.org/user-docs/master/gpu.html#nvidia-gpus-cuda-standard) +See [Apptainer GPU user guide (opens in new tab)](https://apptainer.org/user-docs/master/gpu.html#nvidia-gpus-cuda-standard) ```bash $ apptainer shell $CONTAINERDIR/tensorflow-2.17.0.sif @@ -289,7 +289,7 @@ Suppose you need to use TensorFlow 2.19.0 on JupyterLab. First, note we do not h module spider tensorflow ``` -Go to [TensorFlow's Docker Hub page](https://hub.docker.com/r/tensorflow/tensorflow) and search for the tag (i.e. version). You'll want to use one that has the `-gpu-jupyter` suffix. Pull the container in your account. +Go to [TensorFlow's Docker Hub page (opens in new tab)](https://hub.docker.com/r/tensorflow/tensorflow) and search for the tag (i.e. version). You'll want to use one that has the `-gpu-jupyter` suffix. Pull the container in your account. ### Installation @@ -348,7 +348,7 @@ jkrollout /path/to/sif "Tensorflow 2.19" gpu ### Test your new kernel -- Go to https://ood.hpc.virginia.edu +- Go to [Open OnDemand at UVA (opens in new tab)](https://ood.hpc.virginia.edu) - Select JupyterLab - Partition: GPU - Work Directory: (location of your `mnist_example.ipynb`) @@ -366,6 +366,6 @@ rm -rf ~/.local/share/jupyter/kernels/tensorflow-2.19.0 ## References -- [Apptainer User Guide](https://apptainer.org/docs/user/latest/) - - [Overview](https://apptainer.org/docs/user/latest/quick_start.html) - - [Bind Path and Mounts](https://apptainer.org/docs/user/latest/bind_paths_and_mounts.html) +- [Apptainer User Guide (opens in new tab)](https://apptainer.org/docs/user/latest/) + - [Apptainer quick start overview (opens in new tab)](https://apptainer.org/docs/user/latest/quick_start.html) + - [Bind Path and Mounts (opens in new tab)](https://apptainer.org/docs/user/latest/bind_paths_and_mounts.html) diff --git a/content/notes/containers/_index.md b/content/notes/containers/_index.md deleted file mode 100755 index c5fef2bd..00000000 --- a/content/notes/containers/_index.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -# Course title, summary, and position. -title: "Introduction to Building and Deploying Containers" -authors: [cag] - -# Page metadata. -date: "2023-05-01T00:00:00" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. - -# Add menu entry to sidebar. -menu: - containers: - name: Containers - weight: 5 ---- - -You've developed an app and written the paper. Now it's time to deploy the app so the world (and the reviewers) can see how awesome it is. This is Part 1 of a two-part workshop that will cover how to deploy web apps for publication. In Part 1 we will go over how to containerize our apps with Docker and maintain them with GitHub. - -docker logo img github logo image diff --git a/content/notes/containers/overview-purpose.md b/content/notes/containers/overview-purpose.md deleted file mode 100755 index b007d7f2..00000000 --- a/content/notes/containers/overview-purpose.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -title: Why Use Containers? -date: "2023-05-01:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 20 -date: "2023-05-01T00:00:00Z" -menu: - containers: - parent: Containers ---- - -Have you ever tried using new code or software from an exciting paper you just read, only to end up spending hours figuring out which versions of the dependencies work on your own machine? Containers eliminate that issue altogether! - -A container is a single unit of software that contains all the packages and code you need to run an application. Sometimes that application is as small as a single function (like printing 'Hello World!'), and sometimes that application is an entire web app. A container will always run the same, regardless of the host system it runs on--making it the perfect solution for sharing reproducible code. - -There are several container technologies out there, but the big ones are Docker and Singularity. Docker is what you will encounter most often in the wild. Singularity (now called Apptainer) is used on HPC systems where most users don't have root access. - -docker logo image -apptainer logo image \ No newline at end of file diff --git a/content/notes/containers/overview-services.md b/content/notes/containers/overview-services.md deleted file mode 100755 index dc65195f..00000000 --- a/content/notes/containers/overview-services.md +++ /dev/null @@ -1,20 +0,0 @@ ---- -title: Container Services -date: "2023-05-01:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 40 -date: "2023-05-01T00:00:00Z" -menu: - containers: - parent: Containers ---- - -Container-based architecture, also known as "microservices," is an approach to designing and running applications as a distributed set of components or layers. Such applications are typically run within containers, made popular in the last few years by Docker. - -Containers are portable, efficient, reusable, and contain code and any dependencies in a single package. Containerized services typically run a single process, rather than an entire stack within the same environment. This allows developers to replace, scale, or troubleshoot portions of their entire application at a time. - -Diagram of multiple containers running separate services in a cluster. - -Research Computing runs microservices in a clustered orchestration environment that automates the deployment and management of many containers easy and scalable. This cluster has >1000 cores and ~1TB of memory allocated to running containerized services. It also has over 300TB of cluster storage and can attach to project and value storage. \ No newline at end of file diff --git a/content/notes/containers/overview-vms.md b/content/notes/containers/overview-vms.md deleted file mode 100755 index 23aa4c74..00000000 --- a/content/notes/containers/overview-vms.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -title: Containers vs VMs -date: "2023-05-01:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 30 -date: "2023-05-01T00:00:00Z" -menu: - containers: - parent: Containers ---- - -You may be familiar with the virtual machines (VMs), which accomplish the same goal as containers: to create a reproducible and shareable compute environment. The big difference between VMs and containers is that VMs provide their own guest OS, whereas containers don't require an OS and run on the Docker Engine and share the host's OS kernel. - -The size of Docker images is usually on the order of tens of MBs, while VMs can be several tens of GBs large. - -Side-by-side comparison of containerized applications (Docker on a host OS) and virtual machines (apps running on guest OSs over a hypervisor). \ No newline at end of file