The MSIIA experiment: using speech to enhance human performance on a cognitive task

We performed an exploratory study to examine the effects of speech-enabled input on a cognitive task involving analysis & annotation of objects in aerial reconnaissance videos. We added speech to an information fusion system to allow for hands-free annotation in order to examine the effect on efficiency, quality, task success, & user satisfaction. We hypothesized that speech recognition could be a cognitive-enabling technology by reducing the mental load of instrument manipulation & freeing up resources for the task at hand. Despite the lack of confidence participants had for the accuracy & temporal precision of the speech-enabled input, each reported that speech made it easier & faster to annotate images. When speech input was available, participants chose speech over manual input to make all annotations. Several participants noted that the additional modality was very effective in reducing the necessity to navigate controls & in allowing them to focus more on the task. Quantitative results suggest that people could potentially identify images faster with speech. However, people did not annotate better with speech (precision was lower, & recall was significantly lower). We attribute the lower recall/precision scores to the lack of undo & editing capabilities & insufficient experience by naive users in an unfamiliar domain. This formative study has provided feedback for further development of the system augmented with speech-enabled input, as our results show that the availability of speech may lead to improved performance of expert domain users on more complicated tasks. 7 Tables, 1 Figure, 21 References. Adapted from the source document