Multimodal deep learning for point cloud panoptic segmentation of railway environments

The demand for transportation asset digitalisation has significantly increased over the years. For this purpose, mobile mapping systems (MMSs) are among the most popular technologies that allow capturing high precision three-dimensional point clouds of the infrastructure. In this paper, a multimodal deep learning methodology is presented for panoptic segmentation of the railway infrastructure. The methodology takes advantage of image rasterisation of the point clouds to perform a rough segmentation and discard more than 80% of points that are not relevant to the infrastructure. With this approach, the computational requirements for processing the remaining point cloud are highly reduced, allowing the process of dense point clouds in short periods of time. A 90 km-long railway scenario was used for training and testing. The proposed methodology is two times faster than the current state-of-the-art for the same point cloud density, and pole-like object segmentation metrics are improved. •Railway point clouds panoptic segmentation based on deep learning.•High performance based on multimodal approach.•Trained and tested on 90 km long railway.•Application on large-scale high-density point clouds.