Despite Appearances: Comparing Emotion Recognition in Abstract and Humanoid Avatars Using Nonverbal Behavior in Social Virtual Reality

The ability to perceive emotional states is a critical part of social interactions, shaping how people understand and respond to each other. In face-to-face communication, people perceive others’ emotions through observing their appearance and behavior. In virtual reality, how appearance and behavior are rendered must be designed. In this study, we asked whether people conversing in immersive virtual reality (VR) would perceive emotion more accurately depending on whether they and their partner were represented by realistic or abstract avatars. In both cases, participants got similar information about the tracked movement of their partners’ heads and hands, though how this information was expressed varied. We collected participants’ self-reported emotional state ratings of themselves and their ratings of their conversational partners’ emotional states after a conversation in VR. Participants’ ratings of their partners’ emotional states correlated to their partners’ self-reported ratings regardless of which of the avatar conditions they experienced. We then explored how these states were reflected in their nonverbal behavior, using a dyadic measure of nonverbal behavior (proximity between conversational partners) and an individual measure (expansiveness of gesture). We discuss how this relates to measures of social presence and social closeness.