A brief survey on RGB-D semantic segmentation using deep learning

•In this research, we make an in-depth analysis and summary of the research progress of RGB-D semantic segmentation in recent years.•We provide an extensive survey of the latest available data sets, which may be helpful to RGB-D semantic segmentation project based on deep learning.•We classified the related work in this field, and introduced the related key work emphatically. Then a comprehensive performance evaluation of these works is carried out.•Finally, we summarize the progress in this field, and discuss the existing problems and a possible development in the future. Semantic segmentation is referred to as a process of linking each pixel in an image to a class label. With this pragmatic technique, it is possible to recognize different objects in an RGB image based on the color and texture, and hence it becomes easier to evaluate. Recently, researchers could perform semantic segmentation pretty well in RGB images. However, the methods based on RGB image lack enough information to realize semantic segmentation of complex scenes. RGB-D semantic segmentation with depth information has been proved to achieve better segmentation results bya lotofexperiments, but there is a lack of a comprehensive survey. In this paper, the main purpose is to offer a detailed review of RGB-D semantic segmentation according to the research progress in recent years. Specifically, recently updated RGB-D datasets will be focused on first, and problems on RGB-D semantic segmentation will be discussed. In the end, a comprehensive analysis is carried out on recent methods and their analysis of the semantic segmentation.