Two-stage estimation of hourly diffuse solar radiation across China using end-to-end gradient boosting with sequentially boosted features

Diffuse solar radiation (DR) constitutes a vital component of solar energy reaching the surface of the Earth. The demand for extensive temporal and spatial coverage of DR data has intensified in the realms of solar energy harvesting, agriculture, and climate change. However, until now, long-term DR observations have only been available from 17 stations across mainland China. Consequently, there is a pressing need to estimate spatially continuous, high-temporal-resolution DR for large-scale regions in China. The current hindrance to DR estimations stems from the scarcity of stations equipped with DR observations. This study proposes a two-stage strategy to efficiently estimate seamless DR in 2019 at a national scale, leveraging both DR and total solar radiation (TR) observations from numerous stations. In the first stage, the approach generates virtual DR at TR stations by establishing a learned relationship between DR and TR observations. Subsequently, in the second stage, these virtual DR data, in conjunction with satellite and reanalysis datasets, are utilized to estimate national-scale DR. Additionally, a novel model, End-to-end Gradient Boosting with Shortcuts and Feature selection (EGB-SF), is introduced to estimate DR over China. One advantage of this model is its consideration of the impact of sequentially boosted features and their interactions. Embedded shortcut connections fully exploit the influence of existing features on newly introduced ones during the learning process. Beyond enhancing the accuracy of DR estimation, the EGB-SF algorithm can also elucidate the relative importance levels of input features to the model. Moreover, the two-stage strategy outperforms the method of estimating national DR using only DR observations, as evidenced by its superior spatial generalization abilities. Statistical evaluation, collaborative analysis with influencing factors, and comparisons with related products confirm the accuracy and spatial continuity of the DR estimations in this study. These results furnish reliable DR data across China for research in agriculture, climate, solar radiation, and related fields. •Hourly diffuse solar radiation estimation using a two-stage strategy.•Two-stage strategy adds estimated DR with TR observations to expand available DR.•Features are sequentially selected according to their importance for DR estimation.•Interactions among multiple features are realized with shortcuts between neurons.•Superior performance in DR es