Action Recognition of Excavator with Data Augmentation of Simulator-Generated Training Data

In construction sites, construction machinery such as excavators plays a critical role. The management of such equipment, notably the monitoring of actions conducted by each construction machinery, is, therefore, key to high productivity and efficiency. This time-consuming and laborious task is currently conducted manually by humans and thus, its automation is highly sought after. Previous works on this issue have achieved high performance using deep learning-based approaches and cameras. However, the investments needed to obtain the training data critical to such approaches are often prohibitive. Using a simulator to generate the training data appears therefore as an alternative to allow fast and easy gathering of training data. However, models trained using such training data perform poorly on real data. The purpose of this study is therefore to increase the performance of action recognition of construction machinery such as excavators using simulator-generated training data. A data augmentation process using background images gathered from actual construction sites is used to reduce the gap between simulator-generated data and real-world data. Experiments with data collected in an actual construction site showed the effectiveness of the proposed method.