Why Do Models Disagree? An Example from the Electric Sector
Researchers often use models to understand how the electric system might change over time and under scenarios with different technology innovations, resource availability, and policies. Some studies compare results across multiple models using common assumptions, but it can be difficult to determine why results vary across the models. This study tested four alternative ways to increase consistency of the data and assumptions used in two models and then compared the model outputs across six electric grid scenarios. Despite the attempts to harmonize the input assumptions, the consistency between model output varied widely across scenarios due to fundamental structural differences between the models.
This work uses electric sector scenario modeling as a case study to better understand why multiple models have results that are sometimes similar and sometimes different. It demonstrates a quantitative method for comparing results across models and identifies specific reasons why two models (ReEDS and GCAM-USA) can disagree even after harmonizing several key assumptions. The results are important for performing multi-model comparisons or using multi-model results to make inferences and decisions. They show the importance of understanding the potential limitations of harmonization efforts and clarify the importance of understanding structural differences between models.
Differences in data inputs, parameter settings, and underlying model structures often confound model intercomparison studies, leading to difficulties in interpreting results. A typical approach is to harmonize as many assumptions as possible between the models being compared. This allows researchers to differentiate between the relative impacts of the harmonized assumptions and underlying structural differences between models.
This study compares cross-model consistency between two electric sector capacity expansion models (GCAM-USA and ReEDS) for six electric sector scenarios under four harmonization configurations that vary model representations of electricity demand, fuel prices, renewable resources, and power plant retirements. These comparisons show that cross-model consistency can vary widely across future electricity scenarios for a given set of harmonized inputs, suggesting that harmonization might need to be scenario-specific to achieve broad consistency between models. Differences in key model structures can hinder consistency despite harmonization efforts.