OSG Virtual School Materials¶
Welcome¶
View the slides (PDF)
Intro to HTC and HTCondor Job Execution¶
View the slides (PDF, PowerPoint)
Intro Exercises 1: Running and Viewing Simple Jobs (Strongly Recommended)¶
- Exercise 1.1: Log in to the local submit machine and look around
- Exercise 1.2: Experiment with HTCondor commands
- Exercise 1.3: Run jobs!
- Exercise 1.4: Read and interpret log files
- Exercise 1.5: Determining Resource Needs
- Exercise 1.6: Remove jobs from the queue
Bonus Exercises: Job Attributes and Handling¶
- Bonus Exercise 1.7: Compile and run some C code
- Bonus Exercise 1.8: Explore
condor_q - Bonus Exercise 1.9: Explore
condor_status
Intro to HTCondor Multiple Job Execution¶
View the Slides (PDF)
Intro Exercises 2: Running Many HTC Jobs (Strongly Recommended)¶
- Exercise 2.1: Work with input and output files
- Exercise 2.2: Use
queue N,$(Cluster), and$(Process) - Exercise 2.3: Use
queue fromwith custom variables - Bonus Exercise 2.4: Use
queue matchingwith a custom variable
OSG¶
View the slides (PDF, PowerPoint)
OSG Exercises: Comparing CHTC and OSG (Strongly Recommended)¶
- Exercise 1.1: Refresher – submitting multiple jobs
- Exercise 1.2: Log in to the OS Pool Access Point
- Exercise 1.3: Running jobs in OSG
- Exercise 1.4: Hardware differences between CHTC and OSG
- Exercise 1.5: Software differences in OSG
Troubleshooting¶
(PDF, PowerPoint)
Troubleshooting Exercises:¶
Software¶
View the slides (PDF, PowerPoint)
Software Exercises 1: Basic Software and Wrapper Script Use (Strongly Recommended)¶
- Exercise 1.1: Work With Downloaded Software
- Exercise 1.2: Use a Wrapper Script To Run Software
- Exercise 1.3: Using Arguments With Wrapper Scripts
Software Exercises 2: Specific Software Examples (Pick Two)¶
- Exercise 2.1: Compiling and Running a Simple Code
- Exercise 2.2: Compiling a Research Software
- Exercise 2.3: Compiling Python and Running Jobs
- Exercise 2.4: Compiling Matlab and Running Jobs
- Exercise 2.5: Using Conda Environments
Software Exercises 3: Using Containers in Jobs (Strongly Recommended)¶
- Exercise 3.1: Using Software in a Singularity Container still working on
- Exercise 3.2: Singularity Examples on OSG Connect
Bonus Exercises: More With Containers¶
- Exercise 4.1: Using Software in a Docker Container
- Exercise 4.2: Building Your Own Docker Container (Beta)
Data¶
View the slides (PDF, PowerPoint)
Data Exercises 1: HTCondor File Transfer (Strongly Recommended)¶
- Exercise 1.1: Understanding a job's data needs
- Exercise 1.2: Using data compression with HTCondor file transfer
- Exercise 1.3: Splitting input
Data Exercises 2: Using Stash (Strongly Recommended)¶
- Exercise 2.1: Using a web proxy for shared input
- Exercise 2.2: Stash for shared input
- Exercise 2.3: Stash for shared output
Bonus Exercises: Shared File Systems¶
Showcase, Part 1¶
-
“Leveraging computer vision systems for monitoring animal health and productivity on dairy farms”, Ariana Negreiro, Animal & Dairy Sciences, UW–Madison (PDF)
-
“High throughput computing to study the functions of plant genes”, Edgar Spalding, Botany, UW–Madison (PDF)
Scaling Up¶
View the slides (PDF)
Scaling Up Exercises¶
Extra Topics¶
Workflows with DAGMan¶
View the slides (PDF,PowerPoint)
- Exercise 1.1: Coordinating set of jobs: A simple DAG
- Exercise 1.2: A brief detour through the Mandelbrot set
- Exercise 1.3: A more complex DAG
- Exercise 1.4: Handling jobs that fail with DAGMan
- Bonus Exercise 4.5: HTCondor challenges
Containers and GPUs¶
View the slides (PDF, PowerPoint)
- Exercise 1.1: Building a Singularity Container
- Exercise 1.2: Using Containers in Jobs
- Exercise 1.3: Running a GPU job
Self-checkpointing for long-running jobs¶
View the slides (PDF, PowerPoint)
Showcase, Part 2¶
-
“Investigating the strong nuclear force with the OSG”, Connor Natzke, Physics, Colorado School of Mines (PDF)
-
“Simulating the spread of flying insects using HTC”, Matthew Garcia, Forest & Wildlife Ecology, UW–Madison (PDF)
Closing¶
View the slides (PDF, PowerPoint too large)