Advancing computational throughput of NSF funded projects with the PATh Facility

Since 2022, the Partnership to Advance Throughput Computing (PATh) Facility has provided dedicated high throughput computing (HTC) capacity to researchers nationwide. Following a year of expansion, here’s a look into the researchers’ work and how it has been enabled by the PATh Facility.

Searching for more computing capacity, Dr. Himadri Chakraborty of Northwest Missouri State University first heard of the PATh Facility, a purpose-built, national-scale distributed high throughput computing (HTC) resource, from his NSF program director. After approaching PATh Research Facilitators to acquire an account and computing “credits,” Chakraborty’s team was able to advance their work in physics using computing resources from the PATh Facility. Christina Koch, Lead Research Facilitator at CHTC guided Chakraborty’s team in transitioning workflows to run within HTCondor.

“As our ambition grew, we were looking out for a larger system, PATh came as a blessing,” Chakraborty reflected. “The ultimate satisfaction is to get some new understanding and learning of the science we are working on. We hope that this will be one of our first major achievements using the PATh Facility.”

The PATh Facility, created through the Partnership to Advance Throughput Computing (PATh; NSF grant #2030508), enables researchers of an NSF-funded project to use its dedicated distributed high-throughput computing (HTC) capacity. PATh is an ongoing collaboration between the Center for High Throughput Computing (CHTC) and the OSG Consortium to support HTC-enabled research by improving national cyberinfrastructure. The PATh Facility acquires capacity from six institutions — the University of Nebraska-Lincoln, Syracuse University, Florida International University, the San Diego Supercomputing Center, the University of Wisconsin-Madison and the Texas Advanced Computing Center — integrated into a single pool of capacity available to users. After becoming fully operational in mid-2022, it saw “the most growth in terms of new projects joining and getting started in 2023,” Koch said.

The PATh Facility guarantees users access through credits. Credits operate as a stand-in unit to ensure no one user is monopolizing the Facility’s capacity. Users receive 1,000 start-up credits to test if the PATh Facility is a good fit for them, available as Central Processing Unit (CPU) or Graphic Processing Unit (GPU) charges. After this initial testing period, they can apply for supplemental credits by contacting PATh Facilitators and their NSF officer. If users run through all their credits, they are still able to keep running and facilitators will work with them to request additional credits.

In comparison to the PATh Facility, the Open Science Pool (OSPool) — another distributed HTC resource created by the OSG Consortium — acquires available capacity from idle resources across 70 institutions nationwide. Projects may be better suited for the PATh Facility than the OSPool if they need additional cores, memory, data or dedicated time. “Since the PATh Facility is hardware owned and operated by PATh, we can make more guarantees about how long individual computations can run, that people will be able to get certain resources and run computations of a certain size,” Koch explained.

Following the PATh Facility’s growth, some OSPool users have begun to use the Facility’s dedicated capacity in tandem. One example is North Carolina State University Ph.D. candidate Matthew Dorsey, who relied on capacity from the OSPool for two years before expanding his research to the newer PATh Facility. In doing so, he was able to run larger jobs without worrying about running out of capacity. “The transition to the PATh Facility was extremely easy,” Dorsey said. “I was pleased with how seamless it was.”

Dorsey became interested in the OSPool after attending OSG School in the summer of 2022. There, he learned the basics of HTCondor and got to know Koch and other facilitators. Dorsey’s research specializes in statistical physics. He uses computational models to study magnetic materials and how magnetic fields can be used to alter properties made from different kinds of magnetic nanoparticles. His work benefits from the consistent access to computing for runs that accumulate over a long period of time.

Dorsey acknowledges that each system has its advantages and disadvantages. For Dorsey, the PATh Facility is better equipped for more complex jobs due to capacity and allocated time, while the OSPool is better for testing out comparatively smaller runs. “It was really easy to translate what I was doing on the OSPool to the PATh Facility and quickly scale it up,” Dorsey said.

A testament to the strength of the PATh Facility, the National Radio Astronomy Observatory (NRAO) used its capacity with the OSPool to develop one of the oldest radio images of a well-studied area in space. Working alongside PATh and CHTC staff, the capacity from the PATh Facility was instrumental in planning when certain jobs would run, without the risk of reduced capacity from the OSPool.

The PATh Facility makes it possible to support projects with larger computational requirements. Dr. Vladan Stevanovic of the Colorado School of Mines is studying computational material science and relies heavily on the PATh Facility to plan and run data workflows. Stevanovic became familiar with the PATh Facility after receiving a Dear Colleague Letter from the NSF.

His work requires more cores than what the OSPool alone could offer, and he was drawn to the PATh Facility due to its specialization in HTC and ability to guarantee availability. Stevanovic and his team hope to develop computational tools to reliably predict the metastable states of solid matter. He describes this work as very computational, and has worked with HTC workflows for over 12 years. “PATh is amazing for my research because its primary purpose is HTC, which I rely on,” he said. “I’m grateful because my project critically depends on PATh,” he said.

Stevanovic also appreciates how easy it was to start using the PATh Facility. Start-up credits are typically granted while or directly after meeting with the Facilitation team, and Koch’s team continues to support researchers as they ask the NSF for more credits. “The onboarding process was great, and the support from Christina was amazing. We were able to get running and get up to speed quickly.”

Chakraborty’s team faced some initial challenges in switching workflows from in-house to distributed, but coming to the PATh Facility nonetheless expanded the capacity available to his team. He recounted that his previous in-house system provided about 28 CPUs per node, while the PATh Facility offers up to 150 CPUs. Overall, Chakraborty is optimistic that the new capacity will improve his findings for the future. “We are happy we took that plunge because we went through some difficult times and got help from the PATh folks,” he said. “We’ve made some pretty good progress in making our codes run on PATh. It was mostly to be able to use a larger pool of computer power.”

His work focuses on the simulation of electronic and phonontonic coupled ultrafast relaxation of photoexcited large molecules. The PATh Facility’s new capacity allowed his team to make new advances “of a polymer functional system that has a lot of applications,” he said. “It’s not the end, it’s still preliminary and intermediate, and they are so exciting. We are looking forward to the final results and finding out new physics.”

Interested PIs can submit an interest form on the PATh website, to then meet a research computing facilitator for a consultation. If the researcher is a good fit, PATh Facilitators help the researcher log in and begin using their start-up credits. If they wish to continue, the researcher begins drafting a proposal letter to the NSF, requesting credits. Koch notes that the credit proposal is simpler than a typical project proposal, and the Facilitation team provides a multitude of resources such as credit calculators and proposal templates. For users who encounter issues, the facilitation team is available through support email address, and weekly support hours, as well as maintaining documentation on the website.