2017 OSG User School Materials¶

Monday¶

Monday Morning: Introduction to HTC¶

• Lecture: Introduction to HTC (PDF)
• Exercise 1.1: Log in to the local submit machine and look around
• Exercise 1.2: Experiment with basic HTCondor commands
• Exercise 1.3: Run jobs!
• Exercise 1.4: Read and interpret log files
• Exercise 1.5: Determining Resource Needs
• Exercise 1.6: Remove jobs from the queue
• Bonus Exercise 1.7: Compile and run some C code

Monday Morning: More HTCondor¶

• Lecture: More HTCondor (PDF)
• Exercise 2.1: Explore condor_q
• Exercise 2.2: Explore condor_status
• Exercise 2.3: Work with input and output files
• Exercise 2.4: Use queue <em>N</em>, $(Cluster), and $(Process)
• Exercise 2.5: Use queue matching, and a custom variable
• Exercise 2.6: Use queue from, and custom variables

Monday Afternoon: Retrying jobs and Workflows with DAGMan¶

• Lecture: Intermediate HTCondor: Workflows (PDF)
• Exercise 3.1: A job that needs retries
• Exercise 3.2: A brief detour through the Mandelbrot set
• Exercise 3.3: Coordinating set of jobs: A simple DAG
• Exercise 3.4: A more complex DAG

Monday Afternoon: Intermediate Workflows with DAGMan¶

• Lecture: HTCondor: More on Workflows (PDF)
• Exercise 4.1: Handling jobs that fail with DAGMan
• Exercise 4.2: Simpler DAGs with variable substitutions
• Exercise 4.3: Using DAG SPLICE for node organization
• Bonus Exercise 4.4: HTCondor challenges (If and only if you have time)

Tuesday¶

Tuesday Morning: Introduction to Distributed HTC and Overlay Systems¶

• Lecture: Introduction to DHTC (PDF)
• Exercise 1.1: Refresher - Submitting multiple jobs
• Exercise 1.2: Log in to the OSG submit machine
• Exercise 1.3: Running jobs in the OSG

Tuesday Morning: Comparing Local and Remote HTC¶

• Lecture: What is different about overlay systems? (PDF)
• Exercise 2.1: Hardware differences in the OSG
• Exercise 2.2: Software differences in the OSG

Tuesday Afternoon: Security in OSG¶

• Lecture: Security in OSG (PDF)

Tuesday Afternoon: Troubleshooting jobs¶

• Lecture: Troubleshooting jobs (PDF)
• Exercise 3.1: Troubleshooting a DAG

Tuesday Afternoon: Connecting to OSG¶

• Lecture: Ways to Connect to OSG (PDF)

Wednesday¶

Wednesday Morning: Software Portability¶

• Lecture: Software Portability for DHTC (PDF)
• Exercise 1.1: Compiling programs for portability
• Exercise 1.2: Using a pre-compiled binary
• Exercise 1.3: Using a wrapper script
• Exercise 1.4: Pre-packaging code
• Bonus Exercise 1.5: Passing Arguments Through the Wrapper Script

Wednesday Morning: Software Limitations¶

• Lecture: Considerations for licensing and programming packages (PDF)
• Exercise 1.6: Compile and run Matlab code
• Exercise 1.7: Pre-packaging Python
• Exercise 1.8: In-job installation of Python
• Bonus Exercise 1.9: Using containers

Wednesday Afternoon: On Your Own¶

• Ideas for activities

Thursday¶

Thursday Morning: OSG Connect¶

• Lecture: OSG Connect (PDF)
• Exercise 1.1: Get acquainted with OSG Connect
• Exercise 1.2: Do the OSG Connect "quickstart"
• Exercise 1.3: Try an OSG Connect software module
• Exercise 1.4: Try Singularity Container Job on the OSG (Optional)

Thursday Morning: Data Handling¶

• Lecture: Overall data considerations (PDF)
• Exercise 2.1: Understanding your data requirements
• Exercise 2.2: HTCondor file transfer and compression
• Exercise 2.3: Splitting large input data

Thursday Afternoon: Data Handling (continued)¶

• Lecture: Solutions for large input data (PDF)
• Exercise 3.1: Using a web proxy for large, shared input
• Exercise 3.2: Using StashCache for large, shared input
• Exercise 3.3: Using StashCache for large, unique input

Thursday Afternoon: Data Handling (continued)¶

• Lecture: Large output and shared file systems; Data summary (PDF)
• Exercise 4.1: Using a local shared filesystem for large input files
• Exercise 4.2: Using a local shared filesystem for large output files

The submit host user-training.osgconnect.net will be active for about two weeks. For long term use, please sign up for OSG Connect. If you have any questions about the signup process, please email user-support@opensciencegrid.org

Friday¶

Friday Morning: From Science to Production Workflows¶

• Lecture: From Science to Real Workflow (PDF)
• Exercise 1.1: Learn about Joe’s Desired Computing Work
• Exercise 1.2: Plan Overall Workflow

Friday Morning: From Science to Production Workflows¶

• Lecture: From Workflow to Automated Production (PDF)
• Exercise 1.3: Execute Joe’s Workflow
• Bonus Exercise 1.4: Further Optimization and Scaling

Friday Afternoon: HTC Showcase¶

• Talk: Dane Morgan, Engineering: HTC for Engineering Better Materials (PDF)
• Talk: Megan Frayer, Genetics: Using HTC in Genomic Ancestry Analysis (PDF)
• Talk: William Cocke, Mathematics: Building a Character Table Database: A Use of Condor in Pure Math (PDF)
• Talk: Edgar Spalding, Botany: HTC As a Tool to Study How Plant Genomes Function (PDF)

Friday Afternoon: Foundations of HTC¶

• Lecture: The Principles of HTC (PDF)

Friday Afternoon: Wrap Up¶

• Lecture: Where to Go and What to Do Next (PDF)