Energy Exascale Earth System Model

Energy Exascale Earth System Model

Vision

The Energy Exascale Earth System Model project is an ongoing, state-of-the-science Earth system modeling, simulation, and prediction project that optimizes the use of DOE laboratory resources to meet the science needs of the nation and the mission needs of DOE.

E3SM Overview

E3SM Project Plan

E3SM Collaboration Policy

Funding Program: 

The Energy Exascale Earth System Model (E3SM) project, previously known as ACME, is central to ESM as well as many of the Climate and Environmental Sciences Division activities, as it is developing a computationally advanced coupled climate-energy model to investigate the challenges posed by the interactions of weather-climate scale variability with energy and related sectors. The E3SM model simulates the fully coupled Earth system at high-resolution (15-25km, including higher resolution within regionally refined areas) and is incorporating coupling with energy, water, land-use and related energy-relevant activities, with a focus on near-term hind-casts (1970-2015) for model validation and a near-term projection (2015-2050) as needed for energy sector planning. The model further employs regional-refinement using variable mesh methodologies designed to provide high resolution in regions where the complex physical and dynamical processes require it, or where more detailed information is desired. The project is led by a collaboration of several DOE-National Laboratories and includes several academic and private partners. While E3SM’s primary purpose is for scientific research, it will be available to support planning for National energy and related sectoral needs, for example by indicating the probability for regional changes in extreme temperature and precipitation, water availability, sea-level change and coastal impacts, Arctic ocean accessibility, and carbon exchange across atmosphere, land and ocean systems.

E3SM Science

E3SM’s scientific goals address three areas of importance to both climate and earth system research:

  1. Water Cycle. The key water cycle question is: “How do the hydrological cycle and water resources interact with the climate system on local to global scales?” Understanding and developing the capability to project the evolution of water in the Earth’s systems is of fundamental importance both to climate-science and to societal and many energy-related processes, including coal-, nuclear-, biofuel-, and hydro-power potentials. Using river flow as a key indicator of hydrological changes from natural and human systems, E3SM is testing the hypothesis that changes in river flow have been historically dominated by land management, water management, and aerosol forcing, but will shift to be increasingly dominated by greenhouse gas changes in coming decades.  The initial phase of the project focuses on simulation of precipitation and surface water in orographically complex regions, including the western United States and Southeastern Asia. The longer-term goal is to understand how the hydrological cycle in the fully coupled climate system will evolve with climate change and the expected effect on local, regional, and national supplies of fresh water.
  2. Biogeochemistry. The key biogeochemistry question is: “How do biogeochemical cycles interact with global climate change?” The degree of carbon exchange among components is important for investigating human influences on atmospheric carbon dioxide, methane and elemental carbon particle concentrations, yet this exchange is in turn affected by climate change and nutrient availability. The early phase of E3SM is examining how more complete treatments of nutrient cycles affect carbon–climate system feedbacks. E3SM is adding phosphorus to its below-ground carbon-nitrogen nutrient system, since availability may limit, e.g., tropical ecosystem production, and may play an important role in regulating global-scale feedbacks. Experiments will investigate the nutrient and climate interactions for the preindustrial through the 21st century. A longer-term goal is to study interactions between land and coastal ecosystems, combining coastal-zone biogeochemical cycling and its interaction with the silt, nutrients, and other substances transported by rivers and runoff.
  3. Cryosphere-Ocean System. The key cryosphere-ocean question is: “How do rapid changes in cryosphere-ocean systems interact with the climate system?” As E3SM builds and couples new dynamic ice sheet and ocean components, it will simulate the potential for ice sheet melt, destabilization and sea-level rise. Simulations will utilize E3SM’s variable-mesh capabilities to enhance resolution in the ocean near the ice sheet and in active regions of the ice sheets, with particular focus on Antarctica. The Model Prediction Across Scales project, or MPAS-Ocean, will provide a new capability to dramatically influence the ability to resolve eddies to better represent the circumpolar deep water and dynamics associated with bringing this water onto the continental shelf under the ice sheet, with ocean model resolution attaining 5 km or less near the ice sheets, and the ice sheet resolution up to 500 m near the margins. Sea ice modeling is also crucial to capture the processes of buttressing at the ice shelf-sea ice boundary, including the development of ice calving dynamics and iceberg models. In the fully coupled system, climatic changes that influence the atmospheric general circulation will also influence the behavior of the Southern Ocean and sea ice. In the long-term, E3SM will include components required to simulate impacts of sea-level change and storm surge on coastal regions, including wave models and focusing resolution in coastal and storm-track regions.

E3SM Computation

A major motivation for the E3SM project is the paradigm shift in computing architectures and their related programming models as computational capabilities move towards the exascale era. DOE, through its science programs and early adoption of new computing architectures, traditionally leads many scientific communities, including Earth system simulation, through these disruptive changes in computing.

E3SM is optimizing Earth system code performance for current and next-generation DOE computer facilities, particularly those at the Argonne Leadership Computing Facility, the Oak Ridge Leadership Computing Facility, and the Lawrence Berkeley National Laboratory National Energy Research Scientific Computing Center. To use these machines, the climate codes must support stricter memory management and more complex thread management. The E3SM performance “gold-standard” is to maintain a coupled–model speed of five simulated years per wall-clock day, even while moving to higher resolution. E3SM focuses on exposing increased concurrency throughout the model and on increasing the on-core performance of key computational kernels. Initially the project is implementing conventional approaches, such as threading and message-passing while increasingly employing the use of on-processor accelerators added in the latest machine designs. Redesigning code for better concurrency through the use of modularized kernels for accelerators will be beneficial for most envisioned exascale architectures. In the longer-term, E3SM will explore dynamic auto-tuning and load balancing to minimize latency and make the model resilient to system disruptions anticipated on exascale architectures.

An important aspect of adaptation to new architectures is a substantial effort to improve software design and practice. Early priorities for E3SM software engineering include maintaining build, test, and performance tools for the relevant computer platforms, and providing rapid development and debugging capabilities to the team. The E3SM code repository both expedites the merging and testing of the fully coupled system and supports a distributed development environment where separate features are being co-developed at different sites. In the longer-term, E3SM will expand use of regression testing, tools for code coverage, correctness analysis, debugging at scale, and traceability of code back to scientific requirements. Productivity will be enhanced by greater use of libraries, frameworks, and tools.

E3SM also has a substantial workflow effort, to enable and automate model simulation, post-processing, analysis and validation. Building from the Ultrascale Visualization Climate Data Analysis Tools (UV-CDAT) software, E3SM component and coupled simulation output will be processed on a single workflow platform. Importantly, the workflow software can accommodate the very large data sets from the E3SM high-resolution simulations and it will enable “server-side” analysis of output rather than requiring porting of output to local machines. The analysis provenance will be captured, to enable replication of the process. Model output will be hosted and shared through the Earth System Grid Federation, using a Climate Model Intercomparison -friendly format. Model evaluation is initially based on well-established metrics developed by leading climate modeling centers. Availability of new observations, a focus on the E3SM driving questions, and emphasis on high-resolution require development of new diagnostics and metrics. Metrics are being established that will track model improvement and realism of the coupled system.

Phase 1: Project term 2014-2017

Phase 2: Plan and proposal are under development

Energy Exascale Earth System Model (E3SM) Project

Project Term: 
2014 to 2017
Project Type: 
Laboratory Funded Research

Publications:

Large Contribution of Coarse Mode to Aerosol Microphysical and Optical Properties: Evidence from Ground-based Observations of a Trans-Pacific Dust Outbreak at a High-Elevation North American Site
Quantifying Residual, Eddy, and Mean Flow Effects on Mixing in an Idealized Circumpolar Current
2014 AGU ACME Town Hall
A Case Study of CUDA FORTRAN and OpenACC for an Atmospheric Climate Kernel
A Generic Law-of- the-Minimum Flux Limiter for Simulating Substrate Limitation in Biogeochemical Models
A marine biogenic source of atmospheric ice nucleating particles
A Modeling Study of Irrigation Effects on Global Surface Water and Groundwater Resources Under a Changing Climate
A new and inexpensive non-bit-for-bit solution reproducibility test based on time step convergence (TSC1.0)
A Thickness-Weighted Average Perspective of Force Balance in an Idealized Circumpolar Current
Accelerated Climate Modeling for Energy (ACME) Project Strategy and Initial Implementation Plan
Accelerated Climate Modeling for Energy Overview
ACME Collaboration Policy
An assessment of global and regional sea level for years 1993–2007 in a suite of interannual CORE-II simulations
Assessment of Simulated Water Balance from Noah, Noah-MP, CLM, and VIC over CONUS using the NLDAS Test Bed
Biogeochemical Modeling of CO2 and CH4 Production in Anoxic Arctic Soil Microcosms
Biospheric Feedback Effects in a Synchronously Coupled Model of Human and Earth Systems
Bit Grooming: Statistically accurate precision-preserving quantization with compression, evaluated in the netCDF Operators (NCO, v4.4.8+
Bit Grooming: Statistically accurate precision-preserving quantization with compression, evaluated in the netCDF Operators (NCO, v4.4.8+)
Bulletin of the American Meteorological Society
Classification of hydrological parameter sensitivity and evaluation of parameter transferability across 431 US MOPEX basins
Climate Mitigation from Vegetation Biophysical Feedbacks during the Past Three Decades
CMIP5 Multi-model Initialized Decadal Hindcasts for the Mid-1970s Shift and Early-2000s Hiatus and Predictions for 2016-2035
Compiler Technologies for Understanding Legacy Scientific Code
Constraining the Instantaneous Aerosol Influence on Cloud Albedo
Contribution of feldspar and marine organic aerosols to global ice nucleating particle concentrations
Description and evaluation of a new four-mode version of the Modal Aerosol Module (MAM4) within version 5.3 of the Community Atmosphere Model
Diagnosing Isopycnal Diffusivity in an Eddying, Idealized Midlatitude Ocean Basin via Lagrangian, in Situ, Global, High-Performance Particle Tracking (LIGHT)
Direct Comparisons of Ice Cloud Macro- and Microphysical Properties Simulated by the Community Atmosphere Model Version 5 with HIPPO Aircraft Observations
Effects of Pre-Existing Ice Crystals on Cirrus Clouds And Comparison Between Different Ice Nucleation Parameterizations with the Community Atmosphere Model (CAM5)
Effects of Spatially Distributed Sectoral Water Management on the Redistribution of Water Resources in an Integrated Water Model
Evaluating Global Streamflow Simulations by a Physically Based Routing Model Coupled with the Community Land Model
Evaluating the Community Land Model in a Pine Stand with Shading Manipulations and 13CO2 Labeling
Evaluation of observed and modelled aerosol lifetimes using radioactive tracers of opportunity and an ensemble of 19 global models
Evaluation of the Arbitrary Lagrangian–Eulerian Vertical Coordinate Method in the MPAS-Ocean Model
Exploring new topography-based subgrid spatial structures for improving land surface modeling
Global distribution and surface activity of macromolecules in offline simulations of marine organic chemistry
Global Distribution and Surface Activity of Macromolecules in Simulations of Marine Organic Chemistry
How Does Increasing Horizontal Resolution in a Global Climate Model Improve the Simulation of Aerosol-Cloud Interactions?
Human-Induced Greening of the Northern Extratropical Land Surface
Implementing and Evaluating Variable Soil Thickness in the Community Land Model, Version 4.5 (CLM4.5)
Increasing water cycle extremes in California and in relation to ENSO cycle under global warming
Influence of Phaeocystis Paramterizations on Global Distributions of Marine Dimethyl Sulfide
Informing Climate Models with Rapid Chamber Measurements of Forest Carbon Uptake
Isopycnal eddy mixing across the Kuroshio Extension: Stable versus unstable states in an eddying model
Journal of Physical Oceanography
Modeling surface water dynamics in the Amazon Basin using MOSART-Inundation v1.0: impacts of geomorphological parameters and river flow representation
Modeling the winter-spring transition of first-year ice in the western Weddell Sea
Multiple Soil Nutrient Competition between Plants, Microbes, and Mineral Surfaces: Model development, parameterization, and example applications in several tropical forests
Natural Aerosols Explain Seasonal and Spatial Patterns of Southern Ocean Cloud Albedo
Natural Aerosols Explain Seasonal and Spatial Patterns of Southern Ocean Cloud Albedo
On the Characteristics of Aerosol Indirect Effect Based on Dynamic Regimes in Global Climate Models
Parametric sensitivity analysis of precipitation at global and local scales in the Community Atmosphere Model CAM5
Phosphorus Feedbacks Constraining Tropical Ecosystem Responses to Changes in Atmospheric CO2 and Climate
Quantifying Sources of Black Carbon in Western North America using Observationally Based Analysis and an Emission Tagging Technique in the Community Atmosphere Model
Quantifying Sources, Deposition, Transport and Radiative Forcing of Black Carbon Over the Himalayas and Tibetan Plateau
Representing Leaf and Root Physiological Traits in CLM Improves Global Carbon and Nitrogen Cycling Predictions
Representing northern peatland microtopography and hydrology within the Community Land Model
Seasonality of global and Arctic black carbon processes in the Arctic Monitoring and Assessment Programme models
Significant Impacts of Irrigation Water Sources and Methods on Modeling Irrigation Effects in the ACME Land Model
The Atmospheric Hydrologic Cycle in the ACME v0.3 Model
The Low-Level Jet over the Southern Great Plains Determined from Observations and Reanalyses and Its Impact on Moisture Transport
The North American Winter ‘Dipole’ and Extremes Activity: A CMIP5 Assessment
Uncertainty Quantification in Climate Modeling and Projection
Using an Explicit Emission Tagging Method in Global Modeling of Source-receptor Relationships for Black Carbon in the Arctic: Variations, Sources, and Transport Ways
Web-based Visual Analytics for Extreme Scale Climate Science

Research Highlights:

A Burning Issue: Following Soot to the Arctic Highlight Presentation
A Case Study of CUDA FORTRAN and OpenACC for an Atmospheric Climate Kernel Highlight Presentation
A Generic Law-of-the-Minimum Flux Limiter for Simulating Substrate Limitation in Biogeochemical Models Highlight Presentation
A High Order Characteristic Discontinuous Galerkin Scheme for Advection on Unstructured Meshes Highlight Presentation
A Marine Biogenic Source of Atmospheric Ice Nucleating Particles Highlight Presentation
A new conceptual model of short-term photosynthate storage and transport based on experimental observations Highlight Presentation
Accounting for Groundwater Use and Return Flow Improves Modeling of Water Management Highlight Presentation
Aerosol Effects on Different Types of Clouds Highlight Presentation
An Improved Minimal Representation of Aerosols for Climate Models Highlight Presentation
Assessment of Simulated Water Balance from Four Land Surface Models using the NLDAS Test Bed Highlight
Biospheric Feedback Effects in a Synchronously Coupled Model of Human and Earth Systems Highlight Presentation
Bit Grooming Improves Precision/Compression Ratio Highlight Presentation
Bit Grooming Improves Precision/Storage Ratio Highlight Presentation
Born by Bubbles, Destined for Clouds Highlight
Classification of Hydrological Parameter Sensitivity and Evaluation of Parameter Transferability across 431 US MOPEX Basins Highlight Presentation
Climate Mitigation from Vegetation Biophysical Feedbacks during the Past Three Decades Highlight Presentation
CMIP5 Climate Model Analyses: Climate extremes in the United States Highlight Presentation
Compiler Technologies for Understanding Legacy Scientific Code Highlight
Detecting Modeling Problems Early and Quickly Highlight Presentation
Diagnosing Ocean Mixing via Lagrangian Particle Tracking (LIGHT) Highlight Presentation
Dust and Organic Sea Spray Contribute to Cloud Freezing Highlight Presentation
Effects of Pre-Existing Ice Crystals on Cirrus Clouds in the Community Atmosphere Model Highlight
Evaluating Climate Simulations against Aircraft Measurements Highlight Presentation
Evaluating Global Streamflow Simulations by a Physically-based Routing Model Coupled with the Community Land Model Highlight Presentation
Explicit Aqueous Phase Redox and pH Dynamics Added to the Converging Trophic Cascade (CTC) Decomposition Model Highlight Presentation
Explicit Consideration of Irrigation Source and Method Matters for Modeling Irrigation Effects Highlight Presentation
Exploring an Ensemble-Based Approach to Atmospheric Climate Modeling and Testing at Scale Highlight Presentation
Fidelity of Climate Extremes in High Resolution Climate Models Highlight Presentation
Global Model Improved by Incorporating New Hypothesis for Vegetation Nutrient Limitation, Supported by Field Experiment Highlight Presentation
Global Ocean Organosulfur Cycling: DMS in ACME Highlight Presentation
Global Ocean Organosulfur Cycling: DMS in ACME Highlight Presentation
Gone with the Wind: The Trans-Pacific Journey of Atmospheric Particles Highlight Presentation
Hope for Constraining Atmospheric Particles’ Effects on Clouds Highlight Presentation
How Does Increasing Horizontal Resolution in a Global Climate Model Improve the Simulation of Aerosol-Cloud Interactions? Highlight Presentation
Human-Induced Greening of the Northern Extratropical Land Surface Highlight Presentation
Implementing and Evaluating Variable Soil Thickness in the Community Land Model, Version 4.5 (CLM4.5) Highlight Presentation
Improved Numerical Methods Reduce Spurious Mixing in MPAS-Ocean Highlight Presentation
Irrigation Methods Drain Water Availability Highlight Presentation
Isopycnal Eddy Mixing across the Kuroshio Extension: Stable versus unstable states in an eddying model Highlight Presentation
Marine Macromolecules: Planetary Scale 2D Chemistry Highlight Presentation
Marine Macromolecules: Planetary Scale 2D Chemistry Highlight Presentation
Marine Macromolecules: Planetary Scale 2D Chemistry Highlight Presentation
Marine Macromolecules: Planetary Scale 2D Chemistry Highlight Presentation
Marine Organic Chemistry: Global Distribution and Surface Activity of Macromolecules in Offline Simulations Highlight
Measuring the Impact of Mesoscale Eddies in the Ocean’s Climate Highlight Presentation
Multiple Soil Nutrient Competition between Plants, Microbes, and Mineral Surfaces Highlight Presentation
New Topography-Based Subgrid Units Improve Land-Surface Modeling Highlight Presentation
Pacific Ocean patterns fueled by climate warming may launch more extreme weather Highlight Presentation
Parametric Sensitivity and Uncertainty Quantification of Precipitation at Global and Local Scales in CAM5 Highlight Presentation
Particle's Warming Impact Brought to Light: Measuring the impact of light-absorbing particles on Earth's snowpack, glaciers and ice cover Highlight Presentation
Phosphorus Feedbacks Constraining Tropical Ecosystem Responses to Changes in Atmospheric CO2 and Climate Highlight Presentation
Representing Floodplain Inundation in an Earth System Model Highlight Presentation
Representing Leaf and Root Physiological Traits in CLM Improves Global Carbon and Nitrogen Cycling Predictions Highlight Presentation
Representing Northern Peatland Microtopography and Hydrology within the Community Land Model Highlight Presentation
Researchers use risk analysis and modeled climate change to assess fire risk in The Golden State's future Highlight
Simulating the Global Water Cycle with A High Spatial-Resolution Climate Model Highlight Presentation
Soot Sources, a Savvy Study, and the Tibetan Plateau Highlight Presentation
The Down and Dirty About North American Snowpack Highlight Presentation
The North American Winter ‘Dipole’ and Extremes Activity: A CMIP5 Assessment Highlight
Tracking Aerosol Lifetimes using Radioactive Tracers Highlight Presentation
Uncertainty Quantification in Climate Modeling and Projection Highlight Presentation
Understanding Current and Eddy Contributions to Ocean Mixing Highlight Presentation
Validating Simulations of Sea Ice Brine Dynamics with Field Observations Highlight Presentation
Web-based Visual Analytics for Extreme Scale Climate Science Highlight Presentation
When the Wind Comes Sweeping Down the Plain Highlight Presentation