The E3SM project recently completed the development of SCREAM, the Simple Cloud Resolving E3SM Atmosphere model. This is the E3SM’s first performance portable global atmospheric model capable of running at nonhydrostatic cloud-resolving resolutions. In 2023, the SCREAM model completed some of the world’s first year-long climate simulations at cloud-resolving resolution, making use of Frontier, the world’s fastest supercomputer (according to TOP500 list). A group of SCREAM computational scientists was awarded the inaugural 2023 Gordon Bell Prize for Climate Modeling, recognizing SCREAM’s “Innovative parallel computing contributions toward solving the global climate crisis”.

The SCREAM model assembles a long list of innovations developed by E3SM team members over the last few years. One of the most unique innovations in SCREAM is its programming model: Team members rewrote the entire model in C++ from Fortran, making use of the Kokkos library to obtain record-setting performance on state-of-the-art GPU and CPU based supercomputers. Team members also developed many new algorithmic innovations, including a discrete Hamiltonian formulation of the nonhydrostatic equations of motion and a highly efficient horizontally explicit vertically implicit timestepping method. One of SCREAM’s biggest performance improvements comes from the team’s work to develop its interpolatory semi-Lagrangian method for monotone and conservative transport of water vapor, liquid and ice, and other hydrometers. Team members also reformulated all of the required physical parameterizations into C++, including turbulence, boundary layer, microphysics, and radiative transfer parameterizations. 

Team Members

Winning this prestigious award was a true team effort, so we want to highlight the individuals who contributed to the success. We asked each team member to tell us a little about themselves and their role on the team. Responses have been edited for length and clarity.

Mark Taylor, the E3SM’s Chief Computational Scientist, is a mathematician by training who has worked on numerical methods for global atmospheric models and parallel computing for most of his career. He thus has a good understanding of the differential equations underlying global atmospheric dynamics and the challenges to solving them efficiently on parallel computers but enjoys the many challenging numerical issues that come up during model development that further improve his knowledge. His favorite aspect of the team is its unique collaboration of atmospheric and computational scientists.

Peter Caldwell, the SCREAM team lead, received a Ph.D. in atmospheric science from the University of Washington, and an MS and BS in mathematics from Western Washington University. He coordinates tasks, provides strategic guidance, and does bureaucratic tasks like writing proposals. His goal is to produce more trustworthy climate predictions by explicitly resolving relevant processes rather than applying heuristic relationships for sub-grid scale processes, a role well suited to computational scientists. He appreciates the team culture where people feel comfortable making mistakes and asking naive questions, which helps the team correct mistakes and make good design choices.

Luca Bertagna has a BS/MS in math/engineering and a Ph.D. in applied math. His passion for well-organized, well-designed, and well-tested code helps to bring structure to the large E3SM code base. He likes to design high-quality software, meaning performant, portable, well-tested, readable, well-organized, maintainable, and possibly extensible. He likes working with the other team members, all of whom are top-notch in their diverse fields. This allows others to learn something new almost by “osmosis”. It also teaches members how to interact with others with different backgrounds.

Thomas Clevenger has a Ph.D. in Mathematics from Clemson University and focuses on large-scale, parallel computation. He is skilled in writing structured, highly performant, and well-tested software, as well as debugging various different types of computer architectures. He enjoys porting features from the F90 code to C++, improving code organization, structure, and performance. He likes the facts that the team collaborates very well and scientists help software engineers understand the application (and visa versa understanding the code).

Aaron Donahue has a BA/MS in Mathematics from Sonoma State and San Diego State respectively, and earned a Ph.D. in Civil Engineering from the University of Notre Dame. He is a computational scientist by training, mostly focusing on model development for large-scale phenomena. He is experienced in altering and designing big model codebases for use on parallel computer systems. He likes working on big problems – he is continually impressed by the big science that can be solved by the proper connections between math, physics, and computers. He is glad to write code for a model that will be used by a huge customer base. He has appreciated growing as a scientist by learning from other team members, who are often enthusiastic and patient about sharing that knowledge with their colleagues.

James Foucar has a bachelor’s degree from University of Texas, Austin; and a master’s from University of New Mexico. He has a broad set of computer science and software engineering skills including DevOps, python scripting, and writing Kokkos kernels. He likes the challenging coding tasks, the people, and the mission of the team.

Oksana Guba has a background in computer sciences and applied mathematics and likes learning how things work in order to fix them.

Ben Hillman has a background in physics and atmospheric science, with a focus on understanding and evaluating cloud-radiative effects in large-scale atmosphere models. He has contributed to many aspects of the project, including the development, configuration, running, and evaluation of the model. He particularly enjoys doing technical work and coming up with new solutions to problems. He loves learning from other team members, the supportive and collaborative nature of the team, and the attention to correctness and understanding that everyone on the team brings.

Noel Keen has BS/MS in Nuclear Engineering. He has been involved with HPC for over 30 years and enjoys the challenges of using the big machines. He also enjoys contributing to research/science of importance to humans. He believes working in team environment is critical to making progress on this ongoing work.

Jayesh Krishna has a MS in Computer Science and extensive experience working as a software developer in the software industry and research labs. He has a lot of experience developing and debugging parallel programs on supercomputers, especially at large scale. He is one of the principal developers of SCORPIO (Software for Caching Output and Reads for Parallel I/O) and is currently working on analyzing and improving the I/O performance of the model. He likes to work on challenging HPC problems, especially at large scales and he enjoys that the team is very collaborative and open to innovative and transformational ideas.

Matt Norman has a background in PDE discretizations, refactoring for GPUs, portable C++ libraries, and tightly integrated Machine Learning surrogates. He helped rewrite the RRTMGP (Rapid Radiative Transfer Model for GCMs in Parallel) radiation code used in SCREAM. He likes working on PDEs algorithms that are well-suited to GPUs by increasing arithmetic intensity and decreasing data movement, as well as improving readability for portable C++ code. He likes that the team is diverse and complementary in terms of areas of expertise.

Sarat Sreepathi has a Ph.D. in Computer Science from North Carolina State University. His research interests include High Performance Computing, Computational Climate Science, Performance Analytics, Exascale Co-design, Optimization Algorithms, Computational Intelligence, Parallel I/O, Performance Analysis and Optimization. He is the Performance Coordinator for the E3SM project.

Christopher Terai’s background is in Physics and Atmospheric Sciences. He now works at the intersection of models and observations, evaluating how well SCREAM simulates our atmosphere. He loves digging into the output from SCREAM and figuring out how SCREAM simulates various atmospheric phenomena. He is excited thinking of the science questions SCREAM’s high spatial resolution can help tackle. He enjoys absorbing and learning new things from his teammates, as well as the supportive culture where people are encouraged to ask questions and make mistakes along the way to improving things.

Trey White has an MS in Physics and a Ph.D. in Computer Science and has worked on a broad array of HPC applications. He has a good intuition regarding speeding up software and finding bugs and enjoys making important computations run faster. He likes how the team prioritizes big science goals, and how they work cooperatively to achieve those goals.

Andrew Salinger has a Ph.D. in Chemical Engineering and has spent much of his career as a computational scientist, focusing on algorithms and software for solving PDEs with high-performance computing. He articulated the vision for rewriting the atmosphere dynamics in C++/Kokkos for scalability on exascale machines in the CMDV Software Modernization project proposal and he supports the SCREAM leadership in building a healthy multidisciplinary team. He likes bringing computational science expertise to the climate model development world, to deliver a high-fidelity model with sustainable software. He likes that the team is a true multi-disciplinary partnership where each person is trusted to own the part in which they are an expert.

Danqing Wu holds two master’s degrees in computer science and has played an active role in the development of SCORPIO (Software for Caching Output and Reads for Parallel I/O). He enjoys specializing in open-source research software products, particularly focusing on parallel I/O for exascale supercomputers. He likes that the team fosters collaborative efforts across different groups, between scientists and software developers.

Renata McCoy has a Ph.D. in statistical physics and an MS in nonlinear optics. She enjoys her role as E3SM’s Chief Operating Officer, which involves tracking the whole project and connecting efforts, people, and tasks. She likes working on bringing order and streamlining processes, making them simple and transparent. She likes that the unique team blends both scientists and computer engineers into a very cohesive group, with very close comradery. She also appreciates their thorough meeting notes.

Ruby Leung was trained in physics and atmospheric science. As the chief scientist of the E3SM project, she enjoys working with the team to develop a vision for the model and the project. Having worked on both regional and global models, she is thrilled that the team has developed a global climate model that can resolve storms and other weather phenomena, giving scientists the ability to project how extreme weather events may change in the future. She appreciates being part of the team that works well and learns from one another.

David Bader has undergraduate training in engineering and graduate degrees in atmospheric science. His job as the E3SM Principle Investigator was to help the team leader maintain overall project support and eliminate as many distractions as he could for the team. He enjoys being able to do technical work rather than management and budgets. He likes the team’s unique and extraordinary dedication, excellence, persistence, and focus.

# # #

This article is a part of the E3SM “Floating Points” Newsletter. Read the full February 2024 edition.