Proof-of-Concept Study: a Mobile Application to Derive Clinical Outcome Measures from Expression and Speech for Mental Health Status Evaluation

This proof-of-concept study aimed to assess the ability of a mobile application and cloud analytics software solution to extract facial expression information from participant selfie videos. This is one component of a solution aimed at extracting possible health outcome measures based on expression, voice acoustics and speech sentiment from video diary data provided by patients. Forty healthy volunteers viewed 21 validated images from the International Affective Picture System database through a mobile app which simultaneously captured video footage of their face using the selfie camera. Images were intended to be associated with the following emotional responses: anger, disgust, sadness, contempt, fear, surprise and happiness. Both valence and arousal scores estimated from the video footage associated with each image were adequate predictors of the IAPS image scores ( p