In this paper, we introduce a testbed for evaluating and comparing climate modeling systems at cloud resolving scales using hindcasts of the June 2012 North American derecho. To demonstrate its utility for model intercomparison, the testbed is applied to two models: the regionally‐refined Simple Cloud‐Resolving E3SM Atmosphere Model (SCREAM) at 6.5, 3.25 and 1.625 km grid spacing and the Weather Research and Forecasting (WRF) model with 3.2 and 1.6 km grid spacing. We find the simulation results to be highly sensitive to the initial conditions (ICs), initialization time, and model configurations, with ICs from the Rapid Refresh producing the best simulation. Significant improvement is identified in both models as horizontal grid spacing is refined. While a propagation delay of approximately 2 hr is found in both models, SCREAM at 1.625 km simulates the observed bow echo structure of the derecho well and predicts strong surface gusts that exceed 30 m/s. In comparison, WRF has difficulty producing surface wind over 25 m/s, with wind gusts in WRF 42%–46% lower than in SCREAM. However, WRF has a lower bias in simulating cloud top temperature and extent, but overestimates precipitation intensity. Both models reproduce the observed outgoing longwave radiation spatial patterns well (Pearson correlation >0.88), but, compared with NEXRAD observations, simulate generally larger areas of composite radar reflectivity >40 dBZ and underestimate the precipitating area by ∼47%.