Learning to see through the haze: Multi-sensor learning-fusion System for Vulnerable Traffic Participant Detection in Fog

We present an experimental investigation of a multi-sensor fusion-learning system for detecting pedestrians in foggy weather conditions. The method combines two pipelines for people detection running on two different sensors commonly found on moving vehicles: lidar and radar. The two pipelines are not only combined by sensor fusion, but information from one pipeline is used to train the other. We build upon our previous work, where we showed that a lidar pipeline can be used to train a Support Vector Machine (SVM)-based pipeline to interpret radar data, which is useful when conditions then become unfavourable to the original lidar pipeline. In this paper, we test the method on a wider range of conditions, such as from a moving vehicle, and with multiple people present. Additionally, we also compare how the traditional SVM performs interpreting the radar data versus a modern deep neural network on these experiments. Our experiments indicate that either of the approaches results in progressive improvement in the performance during normal operation. Further, our experiments indicate that in the event of the loss of information from a sensor, pedestrian detection and position estimation is still effective.