The Effect of Multimodal Emotional Expression on Responses to a Digital Human during a Self-Disclosure Conversation: a Computational Analysis of User Language

Digital humans show promise for use in healthcare as virtual therapists to deliver psychotherapy or companions for social support. For digital humans to be effective and engaging in these roles, it is important they can build close relationships with people. Emotional expressiveness can improve social closeness in human relationships, especially for females. However, it is unknown whether multimodal emotional expression improves relationships with digital humans. Participants were 185 adults aged 18 years or older with English fluency. Participants were block-randomized by gender to complete the Relationship Closeness Induction Task with one of six versions of a digital human. The digital humans varied in modality richness (face, no face) and emotional expression (emotional voice, neutral voice; emotional face, neutral face). Participants’ language was analysed for emotional content using Linguistic Inquiry and Word Count software. A series of three-way ANOVA and ANCOVA were conducted to evaluate the effect of digital human face type, voice type, and participant gender on emotional content in participant language. A digital human with no face was associated with more first-person singular pronoun use than a neutral face and an emotional face digital human. A digital human with no face and a neutral voice received more general negative emotion language than a digital human with no face and an emotional voice. Findings suggest the presence of a face and emotion in the voice may improve emotional responses to digital humans. Results provide evidence for aspects of the theoretical framework of embodied agent-patient communication.