Deep spatial-spectral fusion transformer for remote sensing pansharpening

Pansharpening is the task which reconstructs spatial-spectral properties during the fusion of high-resolution panchromatic (PAN) with low-resolution multi-spectral (LR-MS) images, to generate a high-resolution multi-spectral (HR-MS) image. Recent approaches typically model spatial and spectral properties and fuse them using end-to-end deep learning networks, which fail to take their crucial task-related relationship priors into consideration. In this paper, we propose a novel deep spatial-spectral fusion Transformer (SSFT) inspired by two crucial task-related findings: (i) spatial property-related prior, i.e., PAN image itself can provide enough spatial property to reconstruct the required spatial property of target HR-MS image; and (ii) spectral property-related prior, i.e., both LR-MS and PAN should be involved in the process of modeling spectral property for target HR-MS image. Specifically, our approach consists of three novel blocks: the Fourier-guided spectral reconstruction (FGSR) block innovatively applies complex feature interaction strategy to Fourier representations11The generated representation in the process of Fourier-domain learning. of LR-MS and PAN images, for reconstructing the amplitude manifesting the spectral property for the target HR-MS; the Top-kspatial reconstruction (TKSR) block exploits the Top-k selection strategy to select the most relevant spatial regions from PAN image, for modeling the required spatial property of the target HR-MS; and the spatial–spectral fusion (SSF) block re-weights value (V) matrix of FGSR’s output feature according to the TKSR’s output feature, thus achieving a seamless integration of spatial and spectral properties. Extensive experiments show that our SSTL significantly outperforms state-of-the-art (SOTA) methods on widely-used WorldView-3, QuickBird and GaoFen-2 datasets. Our code is available at https://github.com/Florina2333/SSFT. •Combine the task-related relationship priors to enhance the generalization.•Exploit the complex feature interaction strategy to model spectral property.•Introduce the Top-k selection strategy to model spatial property.•Design the modulate mechanism to achieve a spatial-spectral properties fusion.