OSG Virtual School Materials

School Overview and Intro to HTC

View the slides (PDF, PowerPoint)

Watch the lecture recording (Intro to the Virtual School, HTC, and OSG)

Intro to HTCondor Job Execution

View the slides (PDF, PowerPoint)

Watch the lecture recording (HTC Job Excecution with HTCondor)

Intro Exercises 1: Running and Viewing Simple Jobs (Strongly Recommended)

• Exercise 1.1: Log in to the local submit machine and look around
• Exercise 1.2: Experiment with HTCondor commands
• Exercise 1.3: Run jobs!
• Exercise 1.4: Read and interpret log files
• Exercise 1.5: Determining Resource Needs
• Exercise 1.6: Remove jobs from the queue
• Bonus Exercise 1.7: Compile and run some C code

Intro Exercises 2: Running Many HTC Jobs (Strongly Recommended)

• Exercise 2.1: Work with input and output files
• Exercise 2.2: Use queue N, $(Cluster), and $(Process)
• Exercise 2.3: Use queue from with custom variables
• Bonus Exercise 2.4: Use queue matching with a custom variable

Bonus Exercises: Job Attributes and Handling

• Bonus Exercise 3.1: Explore condor_q
• Bonus Exercise 3.2: Explore condor_status
• Bonus Exercise 3.3: A job that needs retries

OSG

View the slides (PDF, PowerPoint)

Watch the lecture recording (Introduction to OSG)

OSG Exercises 1: Comparing CHTC and OSG (Strongly Recommended)

• Exercise 1.1: Refresher – submitting multiple jobs
• Exercise 1.2: Log in to the OS Pool Access Point
• Exercise 1.3: Running jobs in OSG
• Exercise 1.4: Hardware differences between CHTC and OSG
• Exercise 1.5: Software differences in OSG

OSG Exercise 2: Troubleshooting (Strongly Recommended)

This one could take longer. It is good to work through, but give yourself some time.

• Exercise 2.1: Troubleshooting HTCondor jobs

Software

View the slides (PDF, PowerPoint)

Watch the lecture recording (Backpacking with Code: Software Portability for DHTC)

Software Exercises 1: Basic Software and Wrapper Script Use (Strongly Recommended)

• Exercise 1.1: Work With Downloaded Software
• Exercise 1.2: Use a Wrapper Script To Run Software
• Exercise 1.3: Using Arguments With Wrapper Scripts

Software Exercises 2: Specific Software Examples (Pick One)

• Exercise 2.1: Compiling and Running a Simple Code
• Exercise 2.2: Compiling a Research Software
• Exercise 2.3: Compiling Python and Running Jobs
• Exercise 2.4: Compiling Matlab and Running Jobs
• Exercise 2.5: Using Conda Environments (Beta)

Software Exercises 3: Using Containers in Jobs (Strongly Recommended)

• Exercise 3.1: Using Software in a Singularity Container

Bonus Exercises: Container Examples

• Exercise 4.1: Singularity Examples on OSG Connect
• Exercise 4.2: Using Software in a Docker Container
• Exercise 4.3: Building Your Own Docker Container (Beta)

Data

View the slides (PDF, PowerPoint)

Watch the lecture recording (Handling Data on OSG)

Data Exercises 1: HTCondor File Transfer (Strongly Recommended)

• Exercise 1.1: Understanding a job's data needs
• Exercise 1.2: Using data compression with HTCondor file transfer
• Exercise 1.3: Splitting input

Data Exercises 2: Using Stash (Strongly Recommended)

• Exercise 2.1: Using a web proxy for shared input
• Exercise 2.2: Stash for shared input
• Exercise 2.3: Stash for shared output

Bonus Exercises: Shared File Systems

• Exercise 3.1: Shared filesystems for large input
• Exercise 3.2: Shared filesystems for large output

Extra Topics

Self-checkpointing for long-running jobs

View the slides (PDF, PowerPoint)

• Exercise 1.1: Trying out self-checkpointing

Containers and GPUs

View the slides (PDF, PowerPoint)

• Exercise 1.1: Containers Overview
• Exercise 1.2: Running a CPU job
• Exercise 1.3: Running a GPU job

Introduction to Research Computing Facilitation

View the slides (PDF)

Workflows with DAGMan

(PDF, PowerPoint)

• Exercise 1.1: Coordinating set of jobs: A simple DAG
• Exercise 1.2: A brief detour through the Mandelbrot set
• Exercise 1.3: A more complex DAG
• Exercise 1.4: Handling jobs that fail with DAGMan
• Bonus Exercise 4.5: HTCondor challenges

Showcase

Below are the OSGVS21 Showcase speakers, presentation titles, and links to slides (etc.).

1. Spencer Ericksen (University of Wisconsin–Madison), “Scaling virtual screening to ultra-large virtual chemical libraries” (PDF).

2. Hannah Moshontz (University of Wisconsin–Madison), “Using high throughput computing for a simulation study on cross-validation for model evaluation in psychological science” (PDF, OSF page with scripts).

3. Anirvan Shukla (University of Hawaiʻi at Mānoa), “Antimatter: Using high throughput computing to study very rare processes” (PDF).

Final “What’s Next?” Information

View the slides (PDF, PowerPoint)