Improving vegetation segmentation with shadow effects based on double input networks using polarization images

•Vegetation under shadow conditions was extracted accurately.•Polarization information was added to the light intensity information during vegetation segmentation.•A new double input network DIR_DeepLabv3plus with three different fusion strategies is proposed. Fractional vegetation cover (FVC) plays an important role in the study of vegetation growth state, and the key issue is accurately segmenting and extracting green vegetation from the background. However, the shadows generated by natural lights produce extreme illuminance differences in images, which greatly reduces the vegetation extraction accuracy. The polarization information for ground objects is independent of the physical state of the reflectivity of ground objects, and it can be used to eliminate the influence of strong reflections in images to a certain extent, reduce the illuminance differences under extreme sunlight conditions, and help improve the vegetation recognition effect under shadow conditions. To improve the accuracy of vegetation segmentation under shadow conditions, this study introduces polarized reflection information for vegetation and an improved semantic segmentation network, notably a double input residual network based on DeepLabv3plus (DIR_DeepLabv3plus), with fusion strategies based on concatenation and addition is proposed. The network extracts low-level features and high-level features at different spatial scales from both light intensity (red-greenblue (RGB)) images and degree of linear polarization (DoLP) images independently by a deep residual network and atrous spatial pyramid pooling (ASPP) structure, effectively improving the accuracy of vegetation segmentation in shadow situations. The results show that the mean intersection over union (mIoU) values of vegetation without shadows, with light shadows and with shadows are 94.01%, 92.508% and 90.969%, respectively. Compared with the color index method and green fractional vegetation cover extraction from digital images using a shadow-resistant algorithm (SHAR-LABFVC), the proposed method provides a greatly improved extraction accuracy, and it has 0.18%, 1.00% and 1.49% higher mIoU values for vegetation under different shadow conditions than the method without polarization information. This study provides a new approach for vegetation segmentation and improves the accuracy of FVC calculations under shadow conditions.