2018 OSG User School Materials¶

Monday¶

Monday Morning: Introduction to HTC and HTCondor¶

• Lecture: Introduction to HTC (PDF;PPT)
• Exercise 1.1: Log in to the local submit machine and look around
• Exercise 1.2: Experiment with basic HTCondor commands
• Exercise 1.3: Run jobs!
• Exercise 1.4: Read and interpret log files
• Exercise 1.5: Determining Resource Needs
• Exercise 1.6: Remove jobs from the queue
• Bonus Exercise 1.7: Compile and run some C code

Monday Morning: Running Many HTC Jobs¶

• Lecture: More HTCondor (PDF;PPT)
• Exercise 2.1: Work with input and output files
• Exercise 2.2: Use queue N, $(Cluster), and $(Process)
• Exercise 2.3: Use queue matching with a custom variable
• Exercise 2.4: Use queue from with custom variables

Monday Afternoon: Job Attributes and Handling¶

• Lecture: Intermediate HTCondor: Workflows (PDF;PPT)
• Exercise 3.1: Explore condor_q
• Exercise 3.2: Explore condor_status
• Exercise 3.3: A job that needs retries

Monday Afternoon: Automating Workflows with HTCondor's DAGMan¶

• Lecture: HTCondor: More on Workflows (PDF;PPT)
• Exercise 4.1: Coordinating set of jobs: A simple DAG
• Exercise 4.2: A brief detour through the Mandelbrot set
• Exercise 4.3: A more complex DAG
• Exercise 4.4: Handling jobs that fail with DAGMan
• Bonus Exercise 4.5: HTCondor challenges (If and only if you have time)

Tuesday¶

Tuesday Morning: Introduction to Distributed HTC and Overlay Systems¶

• Lecture: Introduction to DHTC (PDF; PPT)
• Exercise 1.1: Refresher - Submitting multiple jobs
• Exercise 1.2: Log in to the OSG submit machine
• Exercise 1.3: Running jobs in the OSG

Tuesday Morning: Comparing Local and Remote HTC¶

• Lecture: What is different about overlay systems? (PDF; PPT)
• Exercise 2.1: Hardware differences in the OSG
• Exercise 2.2: Software differences in the OSG

Tuesday Afternoon: Security in OSG¶

• Lecture: Security in OSG (PDF; PPT)

Tuesday Afternoon: Troubleshooting jobs¶

• Lecture: Troubleshooting jobs (PDF; PPT)
• Exercise 3.1: Troubleshooting a DAG

Tuesday Afternoon: Connecting to OSG¶

• Lecture: Ways to Connect to OSG (PDF)

Wednesday¶

Wednesday Morning: Software Portability¶

• Lecture: Software Portability for DHTC (PDF; PPT)
• Exercise 1.1: Compiling programs for portability
• Exercise 1.2: Using a pre-compiled binary
• Exercise 1.3: Using a wrapper script
• Exercise 1.4: Pre-packaging code
• Bonus Exercise 1.5: Passing Arguments Through the Wrapper Script

Wednesday Morning: Software Limitations¶

• Lecture: Considerations for licensing and programming packages (PDF; PPT)
• Exercise 2.1: Compile and run Matlab code
• Exercise 2.2: Pre-packaging Python
• Exercise 2.3: In-job installation of Python
• Bonus Exercise 2.4: Using containers

Wednesday Afternoon: On Your Own¶

• Ideas for activities

Thursday¶

Thursday Morning: OSG Connect¶

• Lecture: OSG Connect (PDF)
• Exercise 1.1: Get acquainted with OSG Connect
• Exercise 1.2: Do the OSG Connect "quickstart"
• Exercise 1.3: Try an OSG Connect software module
• Exercise 1.4: Try Singularity Container Job on the OSG (Optional)

Thursday Morning: Data Handling¶

• Lecture: Overall data considerations (PDF)
• Exercise 2.1: Understanding your data requirements
• Exercise 2.2: HTCondor file transfer and compression
• Exercise 2.3: Splitting large input data

Thursday Afternoon: Data Handling (continued)¶

• Lecture: Solutions for large input data (PDF)
• Exercise 3.1: Using a web proxy for large, shared input
• Exercise 3.2: Using StashCache for large, shared input
• Exercise 3.3: Using StashCache for large, unique input

Thursday Afternoon: Data Handling (continued)¶

• Lecture: Large output and shared file systems; Data summary (PDF)
• Exercise 4.1: Using a local shared filesystem for large input files
• Exercise 4.2: Using a local shared filesystem for large output files

The submit host training.osgconnect.net will be active for about two weeks. For long term use, please sign up for OSG Connect. If you have any questions about the signup process, please email user-support@opensciencegrid.org

Friday¶

Friday Morning: From Science to Production Workflows¶

• Lecture: From Science to Real Workflow (PDF, PPT)
• Exercise 1.1: Learn about Joe’s Desired Computing Work
• Exercise 1.2: Plan Overall Workflow

Friday Morning: From Science to Production Workflows¶

• Lecture: From Workflow to Automated Production (PDF,PPT)
• Exercise 1.3: Execute Joe’s Workflow
• Bonus Exercise 1.4: Further Optimization and Scaling

Friday Afternoon: HTC Showcase¶

• Talk: Spencer Ericksen, Small Molecule Facility (Carbone Cancer Center): Exploring Virtual Screening Approaches with HTC (PDF)
• Talk: Josh Karpel, Physics: High-Throughput Computing in Atomic Physics (PPT)
• Talk: Ross Kleiman, Computer Sciences: High-Throughput Machine Learning from Electronic Health Records (PPTX)
• Talk: Dave O'Connor, Pathology: When Low Throughput Biologists Meet High Throughput Computing

Friday Afternoon: Foundations of HTC¶

• Lecture: The Principles of HTC (PDF)

Friday Afternoon: Wrap Up¶

• Lecture: Where to Go and What to Do Next (PDF)