Software Exercise 5.1: Pre-package a Research Code¶
Objective: Install software (HHMER) to a folder and run it in a job using a wrapper script.
Why learn this?: If not using a container, this is a template for how to create a portable software installation using your own files, especially if the software is not available already compiled for Linux.
Our Software Example¶
For this exercise, we will be using the bioinformatics package HMMER. HMMER is a good example of software that is not compiled to a single executable; it has multiple executables as well as a helper library.
Create a directory for this exercise on the Access Point.
Do an internet search to find the HMMER software downloads page and the installation instructions page. On the installation page, there are short instructions for how to install HMMER. There are two options shown for installation -- which should we use?
For the purposes of this example, we are going to use the instructions under the heading "...to obtain and compile from source." Download the HMMER source as shown in these instructions (command should start with
Go back to the installation documentation page and look at the steps for compiling from source. This process should be similar to what was described in the lecture!
Normally, it is better to install software on a dedicated "build" server, but for this example, we are going to compile directly on the Access Point
Before we follow the installation instructions, we should create a directory to hold our installation. You can create this in the current directory.
username@host $ mkdir hmmer-build
Now run the commands to unpack the source code:
username@host $ tar -zxf hmmer.tar.gz username@host $ cd hmmer-3.3.2
Now we can follow the second set of installation instructions. For the prefix, we'll use the variable
$PWDto capture the name of our current working directory and then a relative path to the
hmmer-builddirectory we created in step 1:
username@host $ ./configure --prefix=$PWD/../hmmer-build username@host $ make username@host $ make install
Go back to the previous working directory:
username@host $ cd ..
and confirm that our installation procedure created
sharedirectories in the
username@host $ ls hmmer-build bin share
Now we want to package up our installation, so we can use it in other jobs. We can do this by compressing any necessary directories into a single gzipped tarball.
username@host $ tar -czf hmmer-build.tar.gz hmmer-build
Note that we now have two tarballs in our directory -- the source tarball (
hmmer.tar.gz), which we will no longer need and our newly built installation (
hmmer-build.tar.gz) which is what we will actually be using to run jobs.
Now that we've created our portable installation, we need to write a script that opens and uses the installation, similar to the process we used in a previous exercise. These steps should be performed back on the submit server (
Create a script called
The script will first need to untar our installation, so the script should start out like this:
#!/bin/bash tar -xzf hmmer-build.tar.gz
We're going to use the same
$PWDtrick from the installation in order to tell the computer how to find HMMER. We will do this by setting the
PATHenvironment variable, to include the directory where HMMER is installed:
Finally, the wrapper script needs to not only setup HMMER, but actually run the program. Add the following lines to your
hmmbuild globins4.hmm globins4.sto hmmsearch -o search-results.txt globins4.hmm globins45.fa
Make sure the wrapper script has executable permissions:
username@ap1 $ chmod u+x run_HMMER.sh
Run a HMMER job¶
We're almost ready! We need two more pieces to run a HMMER job.
We're going to use some of the tutorial files provided with the HMMER download to run the job. You already have these files back in the directory where you unpacked the source code:
username@ap1 $ ls hmmer-3.3.2/tutorial 7LESS_DROME fn3.hmm globins45.fa globins4.sto MADE1.hmm Pkinase.hmm dna_target.fa fn3.sto globins4.hmm HBB_HUMAN MADE1.sto Pkinase.sto
If you don't see these files, you may want to redownload the
hmmer.tar.gzfile and untar it here.
Our last step is to create a submit file for our HMMER job. Think about which lines this submit file will need. Make a copy of a previous submit file (you could use the blast submit file from a previous exercise as a base) and modify it as you think necessary.
The two most important lines to modify for this job are listed below; check them against your own submit file:
executable = run_hmmer.sh transfer_input_files = hmmer-build.tar.gz, hmmer-3.3.2/tutorial/
A wrapper script will always be a job's
executable. When using a wrapper script, you must also always remember to transfer the software/source code using
transfer_input_filesline indicates that we are transferring the contents of that directory (which in this case, is what we want), rather than the directory itself.
Submit the job with
Once the job completes, it should produce a
For a very similar compiling example, see this guide on how to compile
samtools: Example Software Compilation