Cloud-EGAN: Rethinking CycleGAN From a Feature Enhancement Perspective for Cloud Removal by Combining CNN and Transformer

Cloud cover presents a major challenge for geoscience research of remote sensing images with thick clouds causing complete obstruction with information loss while thin clouds blurring the ground objects. Deep learning (DL) methods based on convolutional neural networks (CNNs) have recently introduced to the cloud removal task. However, their performance is hindered by weak contextual information extraction and aggregation capabilities. Unfortunately, such capabilities play a vital role in characterizing remote sensing images with complex ground objects. Therefore, it remains challenging to eliminate clouds thoroughly while maintaining non-cloudy areas intact. To circumvent this obstacle, the conventional cycle-consistent generative adversarial network (CycleGAN) framework is revitalized for cloud removal from a feature enhancement perspective in this work. More specifically, a saliency enhancement (SE) module is first designed to replace the original CNN module in CycleGAN before multi-level feature maps are enhanced by re-calibrating channel attention weights to capture detailed information about the ground surface. Furthermore, a high-level feature enhancement (HFE) module is developed to generate contextualized cloud-free features while suppressing cloud components from remote sensing images. In particular, HFE is composed of a CNN-based and a transformer-based module. The former enhances the local high-level features by residual learning and multi-scale context modeling, while the latter captures the long-range contextual dependencies by the classical Swin transformer module to exploit high-level information from a global perspective. Capitalizing on the SE and HFE modules, an effective Cloud-Enhancement GAN, namely Cloud-EGAN, is proposed to accomplish thin and thick cloud removal tasks. Extensive experiments on the RICE dataset and the WHUS2-CR dataset confirm the impressive performance of Cloud-EGAN as compared to several existing DL-based methods through visual and quantitative evaluations.