Data Exercise 1.1: Understanding Data Requirements

Exercise Goal

This exercise's goal is to learn to think critically about an application's data needs, especially before submitting a large batch of jobs or using tools for delivering large data to jobs. In this exercise we will attempt to understand the input and output of the bioinformatics application BLAST.

Setup

For this exercise, we will use the ap40.uw.osg-htc.org access point. Log in:

$ ssh <USERNAME>@ap40.uw.osg-htc.org

Create a directory for this exercise named blast-data and change into it

Copy the Input Files

To run BLAST, we need the executable, input file, and reference database. For this example, we'll use the "pdbaa" database, which contains sequences for the protein structure from the Protein Data Bank. For our input file, we'll use an abbreviated fasta file with mouse genome information.

1. Copy the BLAST executables:

   user@ap40 $ osdf object get /osg-public/school/2026/ncbi-blast-2.16.0+-x64-linux.tar.gz ./
   user@ap40 $ tar -xzvf ncbi-blast-2.16.0+-x64-linux.tar.gz
   user@ap40 $ mv ncbi-blast-2.16.0+/bin/blastx ./
   

2. Download these files to your current directory:

   user@ap40 $ osdf object get /osg-public/school/2026/pdbaa.tar.gz ./
   user@ap40 $ osdf object get /osg-public/school/2026/mouse.fa ./
   

3. Untar the pdbaa database:

   user@ap40 $ tar -xzvf pdbaa.tar.gz
   

Understanding BLAST

Remember that blastx is executed in a command like the following:

user@ap40 $ ./blastx -db <DATABASE ROOTNAME> -query <INPUT FILE> -out <RESULTS FILE>

In the above, the <INPUT FILE> is the name of a file containing a number of genetic sequences (e.g. mouse.fa), and the database that these are compared against is made up of several files that begin with the same <DATABASE ROOTNAME>, (e.g. pdbaa/pdbaa). The output from this analysis will be printed to <RESULTS FILE> which is also indicated in the command.

Calculating Data Needs

Using the files that you prepared in blast-data, we will calculate how much disk space is needed if we were to run a hypothetical BLAST job with a wrapper script, where the job:

• Transfers all of its input files (including the executable) as tarballs
• Untars the input files tarballs on the execute host
• Runs blastx using the untarred input files

Here are some commands that will be useful for calculating your job's storage needs:

• List the size of a specific file:

  user@ap40 $ ls -lh <FILE NAME>
  

• List the sizes of all files in the current directory:

  user@ap40 $ ls -lh
  

• Sum the size of all files in a specific directory:

  user@ap40 $ du -sh <DIRECTORY>
  

Input requirements

Using the commands that you have learned so far, you should be able to get an idea about the input file size. Now, do the following before moving on to the next exercise.

• Total up the amount of data in all of the files necessary to run the blastx wrapper job, including the executable itself.
• Write down this number.
• Also take note of how much total data is in the pdbaa directory.

Remember to Calculate Uncompressed File Sizes

Remember, blastx reads the un-compressed pdbaa files. When calculating the disk space needed on the execute host, you need to include the uncompressed size of the pdbaa files, not the compressed tarball.

Intermediate and Output requirements

You also need to get an idea about the intermediate and output file sizes as well to estimate the total disk requirement. To sum the output file sizes, estimate the file sizes of the following files

• The output that we care about from blastx is saved in the file whose name is indicated after the -out argument to blastx.
• Also, remember that HTCondor also creates the error, output, and log files, which you'll need to add up, too.
• Are there any other files?
• Total all of these together, as well.

Exercise: Where Does the Data Live?

For each file a job uses, sort it into the box where it takes up disk space — on the Access Point (AP), the Execution Point (EP), or both — then work out the total storage each side needs. Some files live in both places; only the untarred data is EP-only.

The activity has two tabs:

• Exercise 1 — One job: a single blastx job (the case you just calculated by hand).
• Exercise 2 — Three jobs: the same workflow submitted as three jobs, each with its own query (mouse.fa, horse.fa, dog.fa) and output. Watch what changes on the AP versus a single EP when you scale up — the shared inputs are stored once, only the per-job files multiply, and each execute node's footprint stays the same.

Exercise 1 — One job Exercise 2 — Three jobs

The submit file below runs a single blastx job. Every file it uses is shown as a card. Drag each card into the box where that file takes up disk space — on the Access Point (AP), the Execution Point (EP), or both — then work out the total storage each side needs. Some files live in both places; only the untarred data is EP‑only. (Prefer clicking? Use the small AP / EP buttons on each card.)

executable            = blast_wrapper.sh
arguments             = mouse.fa results

transfer_input_files  = pdbaa.tar.gz, blast.tar.gz, mouse.fa
transfer_output_files = results

output                = job.out
error                 = job.err
log                   = job.log

request_cpus          = 1
request_memory        = 2 GB
request_disk          = 2 GB

queue 1

Files used by this job:

Storage Needs on the Access Point (AP)

Storage Needs on the Execution Point (EP)

Check placement Show answer Reset

Now total the storage each side needs:

Total storage on the AP: MB Total storage on the EP: MB Check totals

Up next!

Next you will create a HTCondor submit script to transfer the Blast input files in order to run Blast on a worker nodes. Next Exercise