Explicit and stepwise models for spatiotemporal fusion of remote sensing images with deep neural networks

•Differences of space, sensor, and time lie in the images for spatiotemporal fusion.•To pursue higher fusion quality, complete modeling of the three differences is necessary.•Diverse deep and shallow neural networks can be deliberately designed to model these differences.•Performance of existing fusion algorithms can be improved by adding the proposed models for spatial and sensor differences. The spatial, sensor, and temporal differences can be observed in the process of spatiotemporal fusion because source images are from different sensors or moments. The existing spatiotemporal fusion methods have modelled the temporal difference, but they did not solve the spatial difference or the sensor difference to build complete models. In this paper, a step-by-step modelling framework is proposed, and three models are designed based on deep neural networks to model the spatial difference, sensor difference, and temporal difference in a separate and explicit way. The spatial difference is modelled with cascaded dual regression networks. The sensor difference is simulated with a four-layer convolutional neural network. The temporal difference is predicted with a generative adversarial network. The proposed method is compared with six algorithms for the reconstruction of Landsat-7 and Landsat-5 which validates the effectiveness of the spatial fusion strategy. The digital evaluation on radiometric, structural, and spectral loss illustrates that the proposed method can give the optimal performance steadily. The necessity of complete modelling is also tested by connecting the spatial and sensor models of the proposed method with one-pair fusion methods, and the steadily improved performance shows that all the difference models contribute to performance improvement.