The Big Questions: Lois Curfman McInnes on Software for High-Performance Computers

Scientists recognized by the Department of Energy (DOE) Office of Science Distinguished Scientist Fellow Award are pursuing answers to science’s biggest questions. Lois Curfman McInnes is a senior computational scientist and Argonne distinguished fellow in the Mathematics and Computer Science Division of DOE’s Argonne National Laboratory.  

Anyone in the computer world would recognize a photo of the late Steve Jobs in his distinctive black turtleneck. In contrast, they’ve probably never heard of astrophysicist Sir Arthur Stanley Eddington. But these thinkers both had important ideas that sum up much of my philosophy about computational science. 

Computing is revolutionizing how scientists learn, experiment, and theorize. Computing has become a prevalent means of discovery and innovation in essentially all areas of science, engineering, technology, and society.  High-performance computing systems at the Department of Energy’s (DOE) national laboratories enable scientists to address huge questions in chemistry, materials, physics, biology, national security, and more.

To make this research possible, I lead teams that develop scientific software ecosystems. Like natural ecosystems, software ecosystems support a variety of individuals working together as a whole. These ecosystems allow software applications to connect and build on each other. Software ecosystems are essential for running programs on high-performance computers to advance discovery in the computing sciences. Importantly, the research enabled by computing is only as good as the software that produces it. 

A major advantage of scientific software ecosystems is that software designers don’t need to rewrite the same code over and over again. As Jobs told attendees at the Apple Worldwide Developers conference in 1997, “The way you get programmer productivity is by eliminating lines of code you have to write.” By using scientific software ecosystems, we have replaced much more than 100,000 lines of code with high-quality libraries and tools. 

At the same time, the connections in these ecosystems create immense complexity. Eddington’s quote sums up my thinking about those connections. He said, “We often think that when we have completed our study of one we know all about two because ‘two’ is ‘one and one.’ We forget that we still have to make a study of ‘and.’” While he was referring to science at large, it’s true of the specifics of building a software ecosystem as well. 

Throughout my career, I’ve focused on a big question: How can we develop foundational scientific software ecosystems that enable new applications while also managing their complexity?

For a love of math

I’ve loved math for a long time. After growing up in a small town in Pennsylvania, I went to Muhlenberg College, a small liberal arts college in Pennsylvania. With a double major in mathematics and physics, I moved onto a PhD in applied mathematics at the University of Virginia. As scientific software is fundamentally rooted in math, these studies gave me the foundation for my future work.

My summer activities were what truly set me on my path. One summer, I had the honor of working at DOE’s Argonne National Laboratory. Being a student at Argonne changed my life. I absolutely loved being in the lab environment. 

I’ve been at the lab in one way or another ever since. After starting in 1993 as a post-doc, I worked my way up from assistant computer scientist to software engineer, computational scientist, and eventually to a position as a lab fellow. Along with fulfilling work and a salary, the lab also has provided the work-life balance I need. I am proud to have raised two daughters who are now pursuing their own careers.

One thing I love about Argonne is its leadership in computing. Not long after its founding in 1946, laboratory physicists built the first digital computer at the lab. In the 1970s, Argonne scientists created some of the earliest software libraries. Software libraries are collections of codes for basic mathematical functions and various advanced capabilities. Software developers can use these codes to create their own software. Libraries make it so that everyone doesn’t need to start from scratch. 

By the time I arrived in the 1990s, Argonne’s computing research focused on advanced algorithms and creating software for high-performance computers. A little more than a decade later, the DOE established the Argonne Leadership Computing Facility. This facility now hosts some of the most powerful computers in the world.

An ecosystem for scientific computing

Without software ecosystems, computational scientists wouldn’t be able to harness a fraction of high-performance computers’ potential.  

Software ecosystems are the next step up from software libraries. While software libraries provide reusable code, ecosystems include complementary libraries and other tools. For example, there are algorithms that solve the equations that describe physical movement. They’re essential for modeling the movement of air around an airplane wing or plasma in a fusion device. Other potential tools are ones that analyze a computer’s performance or create visualizations.  On high-performance computers, it’s especially important to have software that can manage both CPUs and GPUs. (CPUs are computing processing units, which your laptop uses. GPUs are graphic processing units, like those used for AI and video gaming systems.) Computers combine CPUs and GPUs for maximum power and efficiency. 

One of my favorite areas within scientific software ecosystems is mathematical libraries. These libraries are used in modeling many areas that range from cancer treatments to earthquakes. Personally, I’ve worked with application teams in accelerator physics, nuclear energy, and fusion.

Building a toolkit 

The main mathematical library I’ve worked on is the Portable, Extensible Toolkit for Scientific Computation or PETSc. DOE and Argonne have supported its development since the 1990s.

Within this toolkit, my focus is on nonlinear algebraic solvers. Solvers are codes or software that solve systems of equations.  Linear algebra is the branch of math that deals with equations similar to those you may remember from middle school algebra, representing lines and planes. Software libraries that solve linear equations are essential for modeling scientific phenomena. 

Nonlinear algebra uses equations to describe systems where variables are not linear (for example, including exponentials, special functions, and so on). These systems are far more complex. A process like combustion has multiple chemical processes at different levels that affect each other. The equations that describe them can have millions of unknowns.

PETSc brings together different kinds of solvers and enables them to build on each other. My work on nonlinear algebraic solvers builds on linear solvers. Other examples of tools include those that simplify staggeringly complex problems and create grids for large-scale models. 

Understanding how these different types of software work by themselves and together depends on long-term research in applied mathematics. There’s no way we could optimize PETSc without understanding the interplay among mathematics, algorithms, physics, and computer hardware infrastructure. 

We built PETSc to be as flexible and adaptable as possible. Users can customize it according to their needs. We provide the code as open source and support an active user community. PETSc can work on a wide range of computers, from laptops all the way up to supercomputers. 

As a result, PETSc is the backbone of numerous scientific applications that address topics from hydrology to manufacturing steel. Advances in mathematics research that go into PETSc flow down into these applications to improve that research.  

PFLOTRAN is one important application that builds on PETSc. The brave PFLOTRAN team volunteered to be early users of the nonlinear solvers in PETSc. PFLOTRAN simulates flow and movement in porous materials – like water through cracks in the Earth. At DOE, it’s been used to deal with nuclear waste, understand how water moves through ecosystems, and improve geothermal energy production. The collaboration between computer scientists and researchers using the applications has helped both sides. Math research has enabled new science research, which has enabled more advances in math and so on.

Currently, there are tens of thousands of researchers who use PETSc directly or the applications that have been built on it. PETSc makes it so that physicists can focus on physics and chemists can focus on chemistry. There are more than 10,000 citations of the user’s manual and website. Applications that use PETSc have won multiple Gordon Bell prizes, considered the “Oscars of computing.” In 2009, our team even received an R&D 100 award.

Tackling exascale

One of our biggest community challenges has been exascale computing. The DOE’s Office of Science and National Nuclear Security Administration started the Exascale Computing Project in 2016. The goal was to develop software and support for computers that would be 50 times more powerful than the best computers at the time. 

I had the privilege of serving as the deputy director of software technology for the Exascale Computing Project from 2020 to 2023. My team had the task of designing, implementing, delivering, and deploying software that could make the most of exascale computers. While previous computers had used both CPUs and GPUs, exascale computers rely on this combination more than ever before. As we designed the software, we had a goal to raise the performance of these components by five to ten times or more. At the same time, we also wanted to ensure that the software could be adapted for smaller systems, like clusters at universities or even laptops.

A team of teams

None of this would have happened without the dedication of the people behind the computers and the software. DOE’s national laboratories enable long-term science. Experienced researchers can teach early career staff, who then become team leaders in their own right. You can consider me a case study of this very process.  For example, PETSc as a human organization is just as important as its software library.

Like the challenges in hardware and software, the Exascale Computing Project put our collaboration skills to the test. Drawing from national labs, universities, and industry, we brought together some of the brightest experts from coast to coast. We worked as a team of teams, including applied mathematicians, applications scientists, computer scientists, data scientists, performance engineers, and more.

The Extreme-scale Scientific Software Stack (E4S) evolved from this effort. The developer community launched E4S to create and support a fully open software ecosystem for science. Hundreds of the world’s most advanced scientific applications are built on its collection of libraries and tools. The team developed policies to prevent conflicts between software, reduce development time, and improve testing. This community effort is one of the Exascale Computing Project’s most important legacies.

AI and the future

Exascale computing has brought us into the era of artificial intelligence. The advances in software we achieved when designing for exascale computers demonstrated the viability of heterogeneous computing systems for science (combining CPUs and GPUs). 

AI brings both possibilities and huge challenges. My team is looking at how we can take a holistic approach that supports both our community and software. 

On the opportunities side, we’re looking at developing AI assistants for PETSc. To augment the capabilities of standard large language models like ChatGPT, we are incorporating fine-tuning training on data specific to our technical area. While our tools aren’t perfect, early experiments have provided good responses to technical questions, similar to those an expert would give. 

One of the biggest issues with AI is that it’s power hungry. To reduce its consumption, my team is working on software that is more computationally efficient than previous versions. We’re aiming to make simulations based on PETSc 100 times more computationally efficient than they are now.

Looking ahead, we need to have an increasing focus on science to improve research software. Just like any other form of science, we can create and test hypotheses to move our work forward. We need all sorts of viewpoints to contribute – not just physical scientists, but even social scientists to understand how people use our software products. 

Investments in software are investments in science. DOE’s sustained investment, from the earliest years of computing through exascale and beyond, has been essential to our progress. My career has been a fulfilling journey in advancing scientific discovery. I look forward to balancing the needs and complexity of the next generation of scientific computing ecosystems.