Software Exercise 2.5: Using Conda Environments (Beta!)¶

This exercise covers how to use Python environments managed by miniconda.

Introduction¶

Many Python users manage their Python installation and environments with either the Anaconda or miniconda distributions. These distribution tools are great for creating portable Python installations and can be used on HTC systems with some help from a tool called conda pack.

Sample Script¶

For this example, create a script called rand_array.py on learn.chtc.wisc.edu or login04.osgconnect.net:
```
import numpy as np

#numpy array with random values
a = np.random.rand(4,2,3)

print(a)
```

To run this script, we will need a copy of Python with the numpy library.

Create and Pack a Conda Environment¶

(For a generic version of these instructions, see the CHTC User Guide)

Our first step is to create a miniconda installation on the submit server.
1. You should be logged into whichever server you made the rand_array.py script on.
2. Download the latest Linux miniconda installer
```
user@login $ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
```
3. Run the installer to install miniconda; you'll need to accept the license terms and you can use the default installation location:
```
[user@login]$ sh Miniconda3-latest-Linux-x86_64.sh
```
4. At the end, you can choose whether or not to "initialize Miniconda3 by running conda init?" The default is no; you would then run the eval command listed by the installer to "activate" Miniconda. If you choose "no" you'll want to save this command so that you can reactivate the Miniconda installation when needed in the future.

Next we'll create our conda "environment" with numpy (we've called the environment "py3-numpy"):

(base)[user@login]$ conda create -n py3-numpy
(base)[user@login]$ conda activate py3-numpy
(py3-numpy)[user@login]$ conda install numpy

Once everything is installed, deactivate the environment to go back to the Miniconda "base" environment.
```
(py3-numpy)[user@login]$ conda deactivate
```
We'll now install a tool that will pack up the just created conda environment so we can run it elsewhere. Make sure that your job's Miniconda environment is created, but deactivated, so that you're in the "base" Miniconda environment, then run:
```
(base)[user@login]$ conda install -c conda-forge conda-pack
```
Enter y when it asks you to install.
Finally, we will run the conda pack command, which will automatically create a tar.gz file with our environment:
```
(base)[user@login]$ conda pack -n py3-numpy
```

Submit a Job¶

The executable for this job will need to be a wrapper script. What steps do you think need to be included? Write down a rough draft, then compare with the following script.

Create a wrapper script like the following:

#!/bin/bash

set -e

export PATH
mkdir py3-numpy
tar -xzf py3-numpy.tar.gz -C py3-numpy
. py3-numpy/bin/activate

python3 rand_array.py

What needs to be included in your submit file for the job to run successfully? Try yourself and then check the suggestions in the next point.
In your submit file, make sure to have the following:
- Your executable should be the the bash script you created in the previous step.
- Remember to transfer your Python script and the environment tar.gz file via transfer_input_files.
Submit the job and see what happens!