Introduction

The Energy Exascale Earth System Model (E3SM) is a fully integrated ESM that employs code optimized for high-performance computers to address Earth system science questions. Within the E3SM framework, the E3SM Land Model (ELM) models the exchanges between terrestrial land surfaces and other Earth system components, facilitating our understanding of hydrologic cycles, biogeophysics, and terrestrial ecosystem dynamics. The advent of state-of-the-art datasets and exascale computers has made km-scale, ultra-high-resolution ELM (uELM) simulations feasible. Significant strides have been made in developing a uELM simulation using Exascale computers for km-scale land simulation at continental and global scales.

This talk updates our research on uELM code development, input data preparation, and early applications with several EESM programs and projects. We also show how uELM opens up new collaboration opportunities across projects, and among ESMD, RGMA, and MSD program areas. The uELM is tested on various computational platforms and is being evaluated across multiple areas/regions in North America.

Method and Development

uELM model and toolkit: Currently, the uELM model supports km-scale simulations with user-defined data, grid resolution, forcing and surface properties, and areas of interest. Technically, the km-scale uELM adopts the CIME infrastructure with fine-resolution grids, data atmosphere, MCT coupler, and PIO. uELM also uses observation-based daily and subdaily atmospheric forcing. We have ported the uELM code onto GPUs and are in the stage of performance tuning and scalability evaluation using the Summit supercomputer.

We have developed a data generation tool, called kiloCraft, to generate km-scale ELM simulation input data, which includes domain, forcing, and surface properties data. KiloCraft also has several functions to support data projection and transformation among different geographic coordinates (such as GWS84 and Lambert Conformal Conic). It also contains a series of functions for data sanity checking (such as the heterogeneity of landscape surface and distributions of plant functional types).

Computational platforms: The study is conducted on various computational platforms including the Summit supercomputer (No.9 in the Top500 list), a baseline Linux cluster (2560 cores), a CADES-CCSI HPC cluster(1024 cores), an Nvidia DGX station(64 cores and 4 GPUs), and virtual machines in the ORNL cloud.

Simulation areas and forcing data: We are conducting uELM simulations across two major domains: North America as a whole, and a sub-region focused on peatlands in the northern US and Canada. The North American domain is defined using Lambert Conformal Conic coordinates, adopting a grid solution of 1km by 1km, and encompasses 21.5 million land grid cells. The 35 years of forcing data (3 hourly) occupies approximately 40 TB, while the surface properties dataset requires 3.7 GB of disk space (using netcdf4 with level 5 compression). The regional peatland domain comprises 0.6 million land grid cells, using GWS84 coordinates at a resolution of 1/8 of a degree. The total forcing data consumes 1.1 TB, and the surface properties dataset requires roughly 0.3 GB (using netcdf4 with level 5 compression). We are also developing new Daymet-ERA5 datasets for the period of 1980- 2024.

Applications and Future Directions

The uELM capability is being used now across multiple projects spanning EESM and ESS programs in BER. These include applications to simulate portions of Alaska at high resolution for the NGEE Arctic project, peatland regional simulations for the ORNL TES SFA, and urban regional simulations for UIFL locations in Baltimore, Chicago, Knoxville, Atlanta, and New Orleans. We are also exploring applications outside the BER space, such as regional simulations over the Tennessee Valley supported in part by the Tennessee Valley Authority.

There are multiple motivations for having an efficient km-scale land modeling capability within E3SM. As the atmosphere and ocean components of E3SM move toward higher resolution with SCREAM and OMEGA, it is useful to capture land processes at similar resolution to avoid aggregation and averaging over heterogeneous land surfaces. The remote sensing datasets used to parameterize, initialize, and evaluate land model predictions generally have a spatial resolution of 1 km or finer, and so a km-scale land capability allows us to take full advantage of the available observational constraints. Most importantly, land processes that humans care about the most, like agricultural and forest productivity, evaporation, surface heating, movement of surface and subsurface water, and land uses including urban and suburban development all have the inherent spatial resolution at the km-scale and finer. By beginning to resolve these scales we are setting up the conditions for future model applications that focus on detailed processes among coastal and inland ecosystems, and among natural, managed, and developed land covers and land uses.

Acknowledgments

This research leverages previous efforts that were supported as part of the Energy Exascale Earth System Model (E3SM) project and the Next Generation Ecosystem Experiment-Arctic (NGEE-Arctic) project, Terrestrial Ecosystem Science (TES-SFA), funded by DOE’s Office of Science, Office of Biological and Environmental Research. This research used resources of the Oak Ridge Leadership Computing Facility and Computing and Data Environment for Science at the Oak Ridge National Laboratory, supported by the DOE’s Office of Science under Contract No. DE-AC05-00OR22725.