Effective Light Field De-Occlusion Network Based on Swin Transformer

Existing CNN-based light field de-occlusion (LF-DeOcc) methods suffer from occlusion removal performance degradation in the presence of large-size occlusions. In this paper, we infer that it is possibly caused by the limited receptive field of CNN and experimentally demonstrate that the de-occlusion performance is high-related to the receptive field. Therefore, a novel LF-DeOcc network based on Swin Transformer and CNN, which aims to exploit both global and local receptive fields, is firstly proposed for light field de-occlusion task. CNNs are employed at shallow layers to compensate for the deficiency of Transformers in extracting local features, while Transformers are employed at deep layers to capture the global patterns of large size occlusions. Hence, by integrating the global and local features, one could restore occlusion-free images effectively. For performance evaluation, a large dataset with mild-to-severe occlusions is developed and tested, with average occlusion rates of 19.82%, 32.13% and 40.56%, respectively. Experimental results show that the proposed network is superior to state-of-the-art methods, achieving 27.87 dB and 29.46 dB on the public dataset and our developed dataset, respectively. Finally, a new evaluation method has been presented in our work, i.e., by utilizing the real target detection task to evaluate the performance of LF de-occlusion algorithms. The practicability of our algorithm is validated using the new evaluation method.