Cross-Modal Repair: Gaze and Speech Interaction for List Advancement

Interacting with long lists of instructions or ingredients continues to be a challenge for conversational interaction. In this paper, we conducted a user study to experiment with the use of ‘cued-gaze’ – waiting for the user’s visual attention – to manage the delivery of instructions with a voice agent. In a Wizard-of-Oz setting, 12 participants were instructed to build a simple Lego tower by a conversational agent and were able to advance in the list using either speech interaction, or gaze interaction. The increasing use of speech agents in real-world cause users to encounter failures in interactions, so in this task the agent was designed to fail when providing the list of instruction to explore how the participants proceeded to recover from common failures. This showed that, for this use case, cross-modality repair was more effective than reformulation of speech.