Convolutional neural networks for hydrothermal vents substratum classification: An introspective study

The increasing availability of seabed images has created new opportunities and challenges for monitoring and better understanding the spatial distribution of fauna and substrata. To date, however, deep-sea substratum classification relies mostly on visual interpretation, which is costly, time-consuming, and prone to human bias or error. Motivated by the success of convolutional neural networks in learning semantically rich representations directly from images, this work investigates the application of state-of-the-art network architectures, originally employed in the classification of non-seabed images, for the task of hydrothermal vent substrata image classification. In assessing their potential, we conduct a study on the generalization, complementarity and human interpretability aspects of those architectures. Specifically, we independently trained deep learning models with the selected architectures using images obtained from three distinct sites within the Lucky-Strike vent field and assessed the models' performances on-site as well as off-site. To investigate complementarity, we evaluated a classification decision committee (CDC) built as an ensemble of networks in which individual predictions were fused through a majority voting scheme. The experimental results demonstrated the suitability of the deep learning models for deep-sea substratum classification, attaining accuracies reaching up to 80% in terms of F1-score. Finally, by further investigating the classification uncertainty computed from the set of individual predictions of the CDC, we describe a semiautomatic framework for human annotation, which prescribes visual inspection of only the images with high uncertainty. Overall, the results demonstrated that high accuracy values of over 90% F1-score can be obtained with the framework, with a small amount of human intervention. [Display omitted] •We evaluate six state-of-the-art convolutional neural networks for classifying images of hydrothermal vent sea floors.•We analyze uncertainty in a set of deep learning models for semi-automatic operational feasibility.•We conduct a visual interpretability analysis to explain model decision-making in the application.