Aggregating physiological and eye tracking signals to predict perception in the absence of ground truth

Today's driving assistance systems build on numerous sensors to provide assistance for specific tasks. In order to not patronize the driver, intensity and timing of critical responses by such systems is determined based on parameters derived from vehicle dynamics and scene recognition. However, to date, information on object perception by the driver is not considered by such systems. With advances in eye-tracking technology, a powerful tool to assess the driver's visual perception has become available, which, in many studies, has been integrated with physiological signals, i.e., galvanic skin response and EEG, for reliable prediction of object perception. We address the problem of aggregating binary signals from physiological sensors and eye tracking to predict a driver's visual perception of scene hazards. In the absence of ground truth, it is crucial to use an aggregation scheme that estimates the reliability of each signal source and thus reliably aggregates signals to predict whether an object has been perceived. To this end, we apply state-of-the-art methods for response aggregation on data obtained from simulated driving sessions with 30 subjects. Our results show that a probabilistic aggregation scheme on top of an Expectation-Maximization-based estimation of source reliabilities can predict hazard perception at a recall and precision of 96% in real-time. •Aggregation of eye tracking with physiological sensors to predict visual perception.•Probabilistic response aggregation methods for data obtained from driving sessions.•High prediction quality, i.e. 96% precision and recall for hazard perception.•Real-time applicability in the absence of ground truth.