Biological and Environmental Research - Earth and Environmental System Sciences
Earth and Environmental System Modeling

A Petascale Non-Hydrostatic Atmospheric Dynamical Core in the Homme Framework

Historically, the development of weather and climate models has been considered two separate disciplines. Climate modelers have traditionally focused on global-scale modeling on long time- scales (decades, centuries to millennia) and weather modelers have focused on the regional/local- scale atmospheric modeling mainly in the context of short-term weather forecasts (days to weeks). The distinction was primarily motivated by the large differences in the target grid resolutions and science missions that necessitated substantially different physical parameterizations and numerical approximations for the equations of motion. This distinction may no longer be obvious. With the ever-increasing computing resources available to modelers, the difference in the resolutions is rapidly narrowing, bringing the two groups closer together than ever before. Therefore, many climate research and weather forecasting institutions are considering, or have already initiated the development, of unified weather and climate models. It is this demand that has created renewed interest in the numerical methods required for next-generation modeling systems. Recently, the issue has become even more urgent by the arrival of petascale computing resources with core counts in the tens of thousands to hundreds of thousands range, which requires that the numerical schemes exhibit unprecedented levels of scalability on these modern highly parallel computer architectures.

To address this challenge, the team assembled here will build a non-hydrostatic solver in the High-Order Method Modeling Environment (HOMME). More specifically, the team will develop the fully compressible 3D Navier-Stokes equations governing atmospheric flows in conservative form and solve them using a hybrid formulation where the momentum equations in the vector invariant-form are discretized with the spectral element method and where the continuity equation and tracer equations (advection dominated flows) are discretized with the discontinuous Galerkin method, thereby exploiting the merits of both methods. To control oscillations, the Discontinuous Gerkin method is combined with a Hermite WENO (H-WENO) limiter. Though a variety of time stepping approaches will be explored, the primary focus is a split-explicit method. The additional parallelism required is exposed through space-filling curve improvements and identification and exploitation of thread-level parallelism. Providing non-hydrostatic capability within HOMME has the potential for significant impact to the climate modeling community. HOMME was recently integrated into Community Earth System Model (CAM), the atmospheric component of the Community Earth System Model (CESM), and has been included in the CAM 5.0 release. Moreover, the spectral element option will be the default dynamical core in CAM 5.2. Recent results indicate that the CAM/HOMME atmospheric component in CESM has excellent scalability on the petascale platforms of DOE, achieving 4.6 simulated-years-per-day on 172,800 cores of the JaguarPF at the Oak Ridge Laboratory on a 0.125-degree resolution simulation. The proposed work directly addresses the following item in the "SciDAC: Earth System Model Development" call to which we are responding to: "numerical formulations for high-resolution modeling."

This is tightly integrated project involving three researchers from the Computational Science Center at the University of Colorado at Boulder, National Center for Atmospheric Research (NCAR), Institute for Mathematics Applied to Geosciences (IMAGe), and NCAR's Computer Science Section (CSS). Collectively, the members of this team have over thirty years of experience employing high-order methods to simulate atmospheric flows and are long-time members of the HOMME development effort. (Co-I Dennis and Drs. Loft and Thomas are the original authors of HOMME.) PI Tufo and Co-I Nair have been close collaborators since 2002 and together DOE has continuously funded them since 2004 (see project reports for DE-SC0001658, DE-FG02-07ER64464, and DE-FG02-04ER63870). The work proposed here builds off of the extensive set of accomplishments that are the direct result of this collaboration and funding. The team is augmented by a series of non-funded collaborators (see letters of collaboration from Drs. Taylor, Evans, and Cai) and partners with current DOE climate and solver efforts (see letters of support from Drs. Jones and Keyes).

Project Term: 
2011 to 2014
Project Type: 
University Project