Characterization of food cultivation along roadside transects with Google Street View imagery and deep learning

•Deep CNNs extract high-resolution agricultural data from street-level images.•Multi-class classifier characterizes seven species and three land use types.•Ten-class classifier accuracy is above 99% on 40% of images, and 83% overall.•Prototype specialist detector recognizes presence of individual banana plants.•Specialist single-species detector has an area under the ROC curve of 0.9905. We describe the development of tools to exploit the enormous resource of street-level imagery in Google Street View to characterize food cultivation practices along roadside transects at very high spatial resolution as a potential complement to traditional remote sensing approaches. We report on two software tools for crop identification using a deep convolutional neural network (CNN) applied to Google Street View imagery. The first, a multi-class classifier distinguishes seven regionally common cultivated plant species, as well as uncultivated vegetation, built environment, and water along the roads. The second, a prototype specialist detector, recognizes the presence of a single plant species: in our case, banana. These two classification tools were tested along roadside transects in two areas of Thailand, a country where there is good Google Street View coverage. On the entire test set, the overall accuracy of the multi-class classifier was 83.3%. For several classes, (banana, built, cassava, maize, rice, and sugarcane), the producer's accuracy was over 90%, meaning that the classifier was infrequently making omission errors. This performance on roadside transects is comparable with that of some remote-sensing classifiers, yet ours does not require any additional site-visits for ground-truthing. Moreover, the overall accuracy of the classifier on the 40% of images it is most sure about is excellent: 99.0%. For the prototype specialist detector, the area under the ROC curve was 0.9905, indicating excellent performance in detecting the presence of banana plants. While initially tested over the road network in a small area, this technique could readily be deployed on a regional or even national scale to supplement remote sensing data and yield a fine-grained analysis of food cultivation activities along roadside transects.