Automated content-based filtering for enhanced vision-based documentation in construction toward exploiting big visual data from drones

In recent years, emerging mobile devices and camera-equipped platforms have offered a great convenience to visually capture and constantly document the as-is status of construction sites. In this regard, visual data are regularly collected in the form of numerous photos or lengthy videos. However, massive amounts of visual data that are being collected from jobsites (e.g., data collection on daily or weekly bases by Unmanned Aerial Vehicles, UAVs) has provoked visual data overload as an inevitable problem to face. To address such data overload issue in the construction domain, this paper aims at proposing a new method to automatically retrieve photo-worthy frames containing construction-related contents that are scattered in collected video footages or consecutive images. In the proposed method, the presence of objects of interest (i.e., construction-related contents) in given image frames are recognized by the semantic segmentation, and then scores of the image frames are computed based on the spatial composition of the identified objects. To improve the filtering performance, high-score image frames are further analyzed to estimate their likelihood to be intentionally taken. Case studies in two construction sites have revealed that the accuracy of the proposed method is close-to-human judgment in filtering visual data to retrieve photo-worthy image frames containing construction-related contents. The performance metrics demonstrate around 91% of accuracy in the semantic segmentation, and we observed enhanced human-like judgment in filtering construction visual data comparing to prior works. It is expected that the proposed automated method enables practitioners to assess the as-is status of construction sites efficiently through selective visual data, thereby facilitating data-driven decision making at the right time. •A method to automatically filter large-scale visual data captured via UAVs for construction sites.•Visual data is semantically segmented to investigate the presence of objects of interest and their spatial composition.•The likelihood of image to be intentionally taken is estimated based on the compositional similarity with the SUN dataset.•Two case studies were conducted to evaluate the performance of the proposed method.•Results show the potential for exploiting big visual data in construction.