Don't Look at Me Like That: Integration of Gaze Direction and Facial Expression

Efficient decoding of facial expressions and gaze direction supports reactions to social environments. Although both cues are processed fast and accurately, when and how these cues are integrated is still debated. We investigated the temporal integration of gaze and emotion cues. Participants responded to letters that were randomly presented on four faces. Two of these faces initially showed direct gaze, two showed averted gaze. Upon target presentation, two faces changed gaze direction (from averted to direct and vice versa). Simultaneously, facial expressions changed from neutral to either an approach- or an avoidance-oriented emotion expression (Experiment 1a: angry/fearful; Experiment 1b: happy/disgusted). Although angry and fearful expressions diminished any effects of gaze direction (Experiment 1a), a direct gaze advantage was found for happy and an averted gaze advantage for disgusted faces (Experiment 1b). This pattern is consistent with hypotheses suggesting a processing benefit when emotion expression and gaze information are congruent in terms of approach- or avoidance-orientation. In Experiment 2, we tracked eye movements and, again, found evidence for an approach-avoidance-congruency advantage for happy and disgusted faces both in performance and gaze behavior. Gaze behavior analyses suggested an integration of gaze and emotion information that was already visible from 300 ms after target onset. Public Significance Statement This study highlights flexible and emotion-specific temporal aspects of gaze and face information integration processes that are involved in social perception and attention. The findings strengthen the approach-avoidance congruency hypothesis with an integration of gaze and face information at early processing stages.