OSG School Materials¶
School Overview and Intro¶
View the slides: PDF
Intro to HTC and HTCondor Job Execution¶
Intro to HTC Slides¶
Intro to HTC: PDF
Worksheet: PDF
Intro to HTCondor Slides¶
View the slides: PDF
Intro Exercises 1: Running and Viewing Simple Jobs (Strongly Recommended)¶
- Exercise 1.1: Log in to the local submit machine and look around
- Exercise 1.2: Experiment with HTCondor commands
- Exercise 1.3: Run jobs!
- Exercise 1.4: Read and interpret log files
- Exercise 1.5: Determining Resource Needs
- Exercise 1.6: Remove jobs from the queue
Bonus Exercises: Job Attributes and Handling¶
- Bonus Exercise 1.7: Compile and run some C code
- Bonus Exercise 1.8: Explore
condor_q - Bonus Exercise 1.9: Explore
condor_status
Intro to HTCondor Multiple Job Execution¶
View the Slides (PDF)
Intro Exercises 2: Running Many HTC Jobs (Strongly Recommended)¶
- Exercise 2.1: Work with input and output files
- Exercise 2.2: Use
queue N,$(Cluster), and$(Process) - Exercise 2.3: Use
queue fromwith custom variables - Bonus Exercise 2.4: Use
queue matchingwith a custom variable
OSG¶
View the slides: PDF
OSG Exercises: Comparing PATh and OSG (Strongly Recommended)¶
- Exercise 1.1: Log in to the OSPool Access Point
- Exercise 1.2: Running jobs in the OSPool
- Exercise 1.3: Hardware differences between PATh and OSG
- Exercise 1.4: Software differences in OSPool
Troubleshooting¶
Slides: (PDF, PowerPoint)
Troubleshooting Exercises:¶
Software¶
Slides: PDF
Software Exercises 1: Exploring Containers¶
- Exercise 1.1: Run and Explore Apptainer Containers
- Exercise 1.2: Use Apptainer Containers in OSPool Jobs
- Exercise 1.3: Use Docker Containers in OSPool Jobs
- Exercise 1.4: Build, Test, and Deploy an Apptainer Container
- Exercise 1.5: Choose Software Options
Software Exercises 2: Preparing Scripts¶
Software Exercises 3: Container Examples (Optional)¶
Software Exercises 4: Exploring Compiled Software (Optional)¶
- Exercise 4.1: Download and Use Compiled Software
- Exercise 4.2: Use a Wrapper Script To Run Software
- Exercise 4.3: Using Arguments With Wrapper Scripts
Software Exercises 5: Compiled Software Examples (Optional)¶
- Exercise 5.1: Compiling a Research Software
- Exercise 5.2: Compiling Python and Running Jobs
- Exercise 5.3: Using Conda Environments
- Exercise 5.4: Compiling and Running a Simple Code
Data¶
View the slides (PDF, PowerPoint)
Data Exercises 1: HTCondor File Transfer (Strongly Recommended)¶
- Exercise 1.1: Understanding a job's data needs
- Exercise 1.2: transfer_input_files, transfer_output_files, and remaps
- Exercise 1.3: Splitting input
Data Exercises 2: Using OSDF (Strongly Recommended)¶
Scaling Up¶
View the slides (PDF)
Scaling Up Exercises¶
- Exercise 1.1: Organizing HTC workloads
- Exercise 1.2: Investigating Job Attributes
- Exercise 1.3: Getting Job Information from Log Files
Workflows with DAGMan¶
View the slides (PDF, PowerPoint)
DAGMan Exercises 1¶
- Exercise 1.1: Coordinating set of jobs: A simple DAG
- Exercise 1.2: A brief detour through the Mandelbrot set
- Exercise 1.3: A more complex DAG
- Exercise 1.4: Handling jobs that fail with DAGMan
- Exercise 1.5: Workflow Challenges
Extra Topics¶
Self-checkpointing for long-running jobs¶
Special Environments¶
View the slides (PDF, PowerPoint)
Special Environments Exercises 1¶
Introduction to Research Computing Facilitation¶
View the slides: PDF
Final Talks¶
- Philosophy: (slides coming soon)
- Final thoughts: PDF