On Human-Machine Interaction during Online Image Classifier Training

This paper considers a number of issues that arise when a trainable machine vision system learns directly from humans, rather than from a "cleaned" data set, i.e. data items which are perfectly labelled with complete accuracy. This is done within the context of a generic system for the visual surface inspection of manufactured parts. The issues treated are relevant not only to wider computer vision applications, but also to classification more generally. Some of these issues arise from the nature of humans themselves: they will be not only internally inconsistent, but will often not be completely confident about their decisions, especially if they are making decisions rapidly. People will also often differ systematically from each other in the decisions they make. Other issues may arise from the nature of the process, which may require the machine learning to have the capacity for real-time, online adaptation in response to users' input. It may be that the users cannot always provide input to a consistent level of detail. We describe how all of these issues may be tackled within a coherent methodology. Using a range of classifiers trained on real data sets from a CD imprint production process, we will present results which show that properly addressing most of these issues may actually lead to improved performance.