Software Exercise 3.2: Build Your Own Docker Container (Optional)¶
Objective: Build a custom Docker container with
numpy and use it in a job
Why learn this?: Docker containers can be run on both your laptop and OSPool. DockerHub also provides a convenient platform for sharing containers. If you want to use a custom container, run across platforms, and/or share a container amongst a group, building in Docker first is a good approach.
- For this example, create a script called
rand_array.pyon the Access Point.
import numpy as np #numpy array with random values a = np.random.rand(4,2,3) print(a)
To run this script, we will need a copy of Python with the
This exercise will walk you through the steps to build your own Docker container
based on Python, with the
numpy Python library added on.
Getting Set Up¶
Before building your own Docker container, you need to go through the following set up steps:
Install Docker Dekstop on your computer.
You may need to create a Docker Hub user name to download Docker Desktop; if not created at that step, create a user name for Docker Hub now.
(Optional): Once Docker is up and running on your computer, you are welcome to take some time to explore the basics of downloading and running a container, as shown in the initial sections of this Docker lesson:
- Introduction to Docker However, this isn't strictly necessary for building your own container.
Building a Container¶
In order to make our container reproducible, we will be using Docker's capability to build a container image from a specification file.
First, create an empty build directory on your computer, not the Access Points.
In the build directory, create a file called
Dockerfile(no file extension!) with the following contents:
# Start with this image as a "base". # It's as if all the commands that created that image were inserted here. # Always use a specific tag like "4.10.3", never "latest"! # The version referenced by "latest" can change, so the build will be # more stable when building from a specific version tag. FROM continuumio/miniconda3:4.10.3 # Use RUN to execute commands inside the image as it is being built up. RUN conda install --yes numpy # RUN multiple commands together. # Try to always "clean up" after yourself to reduce the final size of your image. RUN apt-get update \ && apt-get --yes install --no-install-recommends graphviz \ && apt-get --yes clean \ && rm -rf /var/lib/apt/lists/*
This is our specification file and provides Docker with the information it needs to build our new container. There are other options besides
RUN; see the Docker documentation for more information.
Note that our container is starting from an existing container
continuumio/miniconda3:4.10.3. This container is produced by the
continuumioorganization; the number
4.10.3indicates the container version. When we create our new container, we will want to use a similar naming scheme of:
In what follows, you will want to replace
USERNAMEwith your DockerHub user name. The
VERSIONTAGare your choice; in what follows, we will use
py3-numpyas the container name and
2021-08as the version tag.
To build and name the new container, open a command line window on your computer where you can run Docker commands. Use the
cdcommand to change your working directory to the build directory with the
$ docker build -t USERNAME/py3-numpy:2021-08 .
.at the end of the command! This indicates that we're using the current directory as our build environment, including the
Upload Container and Submit Job¶
Right now the container image only exists on your computer. To use it in CHTC or elsewhere, it needs to be added to a public registry like Docker Hub.
To put your container image in Docker Hub, use the
docker pushcommand on the command line:
$ docker push USERNAME/py3-numpy:2021-08
If the push doesn't work, you may need to run
docker loginfirst, enter your Docker Hub username and password and then try the push again.
Once your container image is in DockerHub, you can use it in jobs as described in Exercise 1.3.
Thanks to Josh Karpel for providing the original sample