Sound-based multiple-equipment activity recognition using convolutional neural networks

Automatically recognizing activities of heavy construction equipment using sound data has recently received considerable attention as a promising research area in construction. Although existing methods are effective, they only focus on tracking the activities of one single piece of equipment. On construction job sites, multiple equipment sound signals are mixed in the environment; Thus, there is a need for a robust method to recognize these activities that are taking place simultaneously. To address this shortcoming, we proposed a multi-label multi-level sound classification method based on Short-Time Fourier Transform (STFT) and Convolutional Neural Network (CNN) that only requires a single-channel off-the-shelf microphone. In addition, we developed a data augmentation method to simulate real-world equipment sound mixtures. We tested the proposed method on both synthetic and real-world equipment sound mixtures. The results of our study showed that this method was effective in identifying activities of multiple pieces of equipment on real construction job sites without the need for separating sound signals in advance. Future studies can focus on other potential applications of sound signal processing in the construction domain, including analyzing engine abnormalities and monitoring environmental performance of the equipment. •This study introduces a novel method for multiple construction equipment activity recognition using off-the-shelf single-channel microphone.•This method utilizes sound-based image signatures to extract specific features for equipment activities.•The method identifies the activities directly without the need for separating sound signals.•Also, a novel data augmentation method is introduced to simulate the real-world sound mixtures when multiple machines performing activities simultaneously.•This method is tested on both synthetic and real-world mixed data and the results are promising.