Comparing alternative modalities in the context of multimodal human–robot interaction

With the advancement of interactive technology, alternative input modalities are often used, instead of conventional ones, to create intuitive, efficient, and user-friendly avenues of controlling and collaborating with robots. Researchers have examined the efficacy of natural interaction modalities such as gesture or voice in single and dual-task scenarios. These investigations have aimed to address the potential of the modalities on diverse applications encompassing activities like online shopping, precision agriculture, and mechanical component assembly, which involve tasks like object pointing and selection. This article aims to address the impact on user performance in a practical human–robot interaction application where a fixed-base robot is controlled through the utilization of natural alternative modalities. We explored this by investigating the impact of single-task and dual-task conditions on user performance for object picking and dropping. We undertook two user studies—one focusing on single-task scenarios, employing a fixed-base robot for object picking and dropping and the other encompassing dual-task conditions, utilizing a mobile robot for a driving scenario. We measured task completion times and estimated cognitive workload through the NASA Task Load Index (TLX), which offers a subjective, multidimensional scale measuring the perceived cognitive workload of a user. The studies revealed that the ranking of completion times for the alternative modalities remained consistent across both single-task and dual-task scenarios. However, the ranking based on perceived cognitive load was different. In the single-task study, the gesture-based modality resulted the highest TLX score, contrasting with the dual-task study, where the highest TLX score was associated with the eye gaze-based modality. Likewise, the speech-based modality achieved a lower TLX score compared to eye gaze and gesture in the single-task study, but its TLX score in the dual-task study was between gesture and eye gaze. These outcomes suggest that the efficacy of alternative modalities is contingent not only on user preferences but also on the specific situational context.