CMNet: Classification-oriented multi-task network for hyperspectral pansharpening

Hyperspectral pansharpening (HP) can fuse different kinds of remote sensing images for the downstream classification or detection task. Deep learning methods have made rapid development in HP. However, most deep learning-based HP methods design the network architecture without considering the specific downstream task. When the pansharpening image is transferred to the downstream tasks, it may lead to poor performance on the downstream tasks. To solve this problem, a novel classification-oriented multi-task network (CMNet) is proposed for HP. CMNet combines the classification and HP networks in a multi-task learning way. In the classification network, class-specific and spatial-difference contrastive constraints are defined to enforce the extraction module to extract discriminative features for classification. In the HP network, a fusion module with an identical structure to the extraction module is constructed to extract fused features for reconstruction. Then, multi-gradient learning is designed to optimize the HP network under the guidance of the classification network. It adopts gradient mapping to consider the conflicting or collaborative situations between these different tasks. In this way, the fusion module in the HP network learns discriminative feature extraction ability from the classification network, which makes the HP network realize classification-oriented HP. Experimental results demonstrate that the proposed method markedly improves the pansharpening performance and the pansharpening is also able to elevate the classification performance.