Sediment yield (SY) plays an important role in the global carbon cycle for carrying particulate carbon into rivers and oceans, but it is rarely represented in Earth system models (ESMs). Existing SY models have mostly been tested over a few small catchments in specific regions or in large river basins globally. By comparing the performance of eight well‐known SY models in 454 small catchments with various land covers and uses across the United States, Canada, Puerto Rico, U.S. Virgin Islands, and Guam, we identified the simple Morgan model for its better performance in representing the spatial variability of continental scale SY at spatial scales relevant to ESMs (several to hundreds of square kilometers) than other models because of a more realistic representation of runoff‐driven erosion and sediment transport capacity in the context of current data availability. The results also indicated that runoff‐driven erosion should be formulated using a power function of runoff, shear stress, or stream power to better represent the total effect of concentrated flow if gully erosion and channel erosion are not explicitly modeled. We also demonstrated that the Morgan model can be further improved by removing snowmelt‐driven runoff in modeling runoff‐driven erosion and to a minor degree by integrating a landslide model. The improved Morgan model explains 57% of the spatial variability of the measured SY. The new model also demonstrated the capability to simulate SY in cross‐validation catchments at fine temporal scales, which is important for coupling SY with other biogeochemistry processes in ESMs.