OSG School Materials¶
School Overview and Intro¶
View the slides: pdf
Intro to HTC and HTCondor Job Execution¶
Intro to HTC Slides¶
Intro to HTC: pdf
Worksheet: pdf or Google Drive
Intro to HTCondor Slides¶
View the slides: pdf
Intro Exercises 1: Running and Viewing Simple Jobs (Strongly Recommended)¶
- Exercise 1.1: Log in to the local submit machine and look around
- Exercise 1.2: Experiment with HTCondor commands
- Exercise 1.3: Run jobs!
- Exercise 1.4: Read and interpret log files
- Exercise 1.5: Determining Resource Needs
- Exercise 1.6: Remove jobs from the queue
Bonus Exercises: Job Attributes and Handling¶
Intro to HTCondor Multiple Job Execution¶
View the Slides: pdf
Intro Exercises 2: Running Many HTC Jobs (Strongly Recommended)¶
- Exercise 2.1: Work with input and output files
- Exercise 2.2: Use
queue N,$(Cluster), and$(Process) - Exercise 2.3: Use
queue fromwith custom variables - Bonus Exercise 2.4: Use
queue matchingwith a custom variable
The Open Science Pool (OSPool)¶
View the slides: [Coming soon]
OSPool Exercises: Researching the OSPool (Strongly Recommended)¶
- Exercise 1.1: Where Do Jobs Run?
- Exercise 1.2: How Much Can I Get?
- Exercise 1.3: How Does Capacity Change?
- Exercise 1.4: What Is In an Execution Point?
- Bonus Exercise 1.5: Viewing OSPool Information
Troubleshooting¶
Troubleshooting Exercises:¶
Software¶
Software Exercises 1: Exploring Containers¶
- Exercise 1.1: Run and Explore Apptainer Containers
- Exercise 1.2: Use Apptainer Containers in OSPool Jobs
- Exercise 1.3: Use Docker Containers in OSPool Jobs
- Exercise 1.4: Build, Test, and Deploy an Apptainer Container
- Exercise 1.5: Choose Software Options
Software Exercises 2: Preparing Scripts¶
Software Exercises 3: Container Examples (Optional)¶
Software Exercises 4: Exploring Compiled Software (Optional)¶
- Exercise 4.1: Download and Use Compiled Software
- Exercise 4.2: Use a Wrapper Script To Run Software
- Exercise 4.3: Using Arguments With Wrapper Scripts
Software Exercises 5: Compiled Software Examples (Optional)¶
- Exercise 5.1: Compiling a Research Software
- Exercise 5.2: Compiling Python and Running Jobs
- Exercise 5.3: Using Conda Environments
- Exercise 5.4: Compiling and Running a Simple Code
Data¶
View the Slides: [Coming soon]
Data Exercises 1: HTCondor File Transfer (Strongly Recommended)¶
- Exercise 1.1: Understanding a job's data needs
- Exercise 1.2: transfer_input_files, transfer_output_files, and remaps
- Exercise 1.3: Splitting input
Data Exercises 2: Using OSDF (Strongly Recommended)¶
Scaling Up¶
View the Slides: [Coming soon]
Scaling Up Exercises¶
- Exercise 1.1: Organizing HTC workloads
- Exercise 1.2: Composing Your Jobs
- Exercise 1.2: Investigating Job Attributes
- Exercise 1.3: Getting Job Information from Log Files
Workflows with DAGMan¶
View the Slides: [Coming soon]
DAGMan Exercises 1¶
- Exercise 1.1: Coordinating set of jobs: A simple DAG
- Exercise 1.2: A brief detour through the Mandelbrot set
- Exercise 1.3: A more complex DAG
- Exercise 1.4: Handling jobs that fail with DAGMan
- Exercise 1.5: Workflow Challenges
Extra Topics¶
Machine Learning¶
View the Slides: [Coming soon]
Self-checkpointing for long-running jobs¶
View the Slides: [Coming soon]
Special Environments¶
View the slides: [Coming soon]
Special Environments Exercises 1¶
Introduction to Research Computing Facilitation¶
View the slides: [Coming soon]
Final Talks¶
- Philosophy: [Slides coming soon]
- Final thoughts: [Slides coming soon]
- Forward (Tim’s final talk): PDF