A big data analytics strategy for scalable urban infrastructure condition assessment using semi-supervised multi-transform self-training

This work aims to leverage the recent advances in the field of computer vision and big data computing to develop a scalable framework for image-based monitoring of urban infrastructure and the built environment. Two alternative sources of big visual data, namely web images and Google Street View imagery, were studied in a semi-supervised setting to minimize the costs associated with data collection and expert annotation. The features of interest include infrastructure defects and degradation such as different types of cracks, potholes, patches, faded markings, fallen signs, and sidewalk cracks and trip hazards. In the proposed multi-transform self-training approach, an ensemble of predictions on a set of geometric transformations of each unlabeled street view image was used to automatically pseudo-label images and retrain the model. This concept eliminates the need for human supervision, thus improving the scalability of the approach. Results show that the proposed transforms can significantly improve the performance of the model (more than 20% accuracy improvement) and reduce the domain gap between Google Street View and web images. A sensitivity analysis was also presented to study the factors influencing the method, and an error analysis was performed to explain a number of misclassification cases in the results. The proposed approach can be used to leverage the wealth of information embedded in the massive sources of imagery that are available to researchers, and the resulting models can be used to automatically process image streams from volunteer citizens, social media, as well as private and public vehicle cameras such as city buses and transportation agency vehicles to automate the urban condition monitoring task.