Annotation and detection of blended emotions in real human-human dialogs recorded in a call center

In the context of call centers, emotion detection is potentially important for customer care. Emotions in natural interaction are often blended. For example, in a Stock Exchange service centre, some customers are angry because they are afraid to lose money. A 100 agent-client dialog corpus has been annotated at the speaker turn level with one label among 5 emotions including fear and anger. In this paper, we report on our experiments of automatic emotion detection using acoustic cues with several classifiers. 73% correct detection was achieved in discriminating between negative and neutral emotions and 60% between anger and fear. An analysis of the confusions led us to question the validity of the initial single valued annotation scheme. It was found that customer emotional states can be a mixture of anger and fear. As a result a new annotation scheme is used allowing the selection of two verbal labels per segment.