Through the Exascale Computing Project (ECP), significant work was invested in porting the Model for Prediction Across Scales-Ocean (MPAS-Ocean) to GPU via use of openACC pragmas. This effort made clear that modification of MPAS-Ocean loops with these pragmas would not be sufficient to achieve sufficient performance gains on GPU. The performance gains required to dramatically increase horizontal resolution for long model integrations are not possible with openACC pragmas. Further, a number of choices in the design of the MPAS-Ocean framework have made performance gains difficult. This led to the creation of a new exascale ready ocean model, the Ocean Model for E3SM Global Applications (OMEGA).

OMEGA will be the new ocean component of E3SM, replacing MPAS-Ocean. Similar to MPAS-Ocean, OMEGA is an unstructured mesh (Fig. 1) model based on the TRiSK numerical methods (Thuburn et al. 2009). Most algorithms (e.g. advection, pressure gradient) will be similar to MPAS-Ocean. Unlike MPAS-Ocean, OMEGA is designed to enable the full/efficient use of modern exascale supercomputers. Emerging exascale resources include accelerators (e.g., GPU) and require use of new programming models. For OMEGA, the Kokkos library (Trott et al. 2022) will be used to enable performance portability across current architectures. This approach addresses the shortcomings of using directives-based in Fortran codes that often lack full compiler support on new systems.

The MPAS-Ocean framework is opaque, making efficient development of the model difficult. In OMEGA, we have sought to radically streamline the framework. To this end, the OMEGA team has moved from a namelist and XML based configuration to the simpler and more robust YAML approach. The team has also implemented a thin layer on top of Kokkos to simplify the syntax of parallel constructs within OMEGA. There will be times in development where simplicity will have to be sacrificed to accelerate performance, but increases in the opaqueness of code will not be pursued for a small performance increase.

With the new model, a new set of stringent requirements for each new feature/development has been set. Documentation for all new features is required in markdown format for ease of integration into user and development guides. The OMEGA team has also dramatically increased verification and validation (V&V) testing of the ocean model, including unit tests for all model infrastructure. Previously, MPAS-Ocean had a basic set of tests, including restart, decomposition, and threading tests, but this resulted in a very small fraction of code covered by testing. The OMEGA group is also spending significant effort developing Polaris, which replaces COMPASS, for automated, task parallel V&V with full documentation required for all new tests. A number of tests from the literature (e.g., Williamson et al 1992) have already been implemented, and an extensive new suite of tests for ocean model dynamical cores (Bishnu et al 2024) has been developed. The team is continuing to work on developing tests for OMEGA features as they become available.

With the introduction of OMEGA, the team is now planning for the end of support for MPAS-Ocean. The E3SM-Project will continue to actively develop MPAS-Ocean through the end of this summer. After this point, the E3SM team will continue to maintain the code but will not add new features. Ecosystem projects planning significant additions to the ocean model should work with the OMEGA leads to discuss inclusion in MPAS-Ocean and how it may be transitioned to OMEGA. Full details of the MPAS-Ocean to OMEGA policy can be found at https://acme-climate.atlassian.net/wiki/spaces/ECM/pages/3825041502

The development of OMEGA will be done in stages with OMEGA-v0 supporting stacked shallow water configurations. The next stage, OMEGA-v1, will add tracers, linear equation of state, and simplified vertical mixing. Finally, OMEGA-v2 will add more advanced parameterizations and coupling terms. OMEGA-v2 will then replace MPAS-Ocean in E3SM as a high-resolution, mesoscale eddy resolving, ocean model for E3SM version 4 (scheduled for 2027). An eddy parameterized version will be developed after validation of the eddy resolving OMEGA-v2.

Once completed, the performance, testing, and support for new high-performance computing architectures will allow OMEGA to be applied to a much broader set of scientific questions, unlocking new areas of exploration. The new capability, coupled with SCREAM will place DOE at the forefront of high resolution coupled earth system modeling. The rigorous documentation and streamlined model configuration and framework will also better serve the broader community by expanding the E3SM user community.

# # #

This article is a part of the E3SM “Floating Points” Newsletter. Read the full May 2024 edition.