The visual system encodes others’ direction of gaze in a first-person frame of reference

•We examine the spatial reference frame representing others’ gaze direction.•Perceptual adaptation occurs to gaze directions relative to the observer.•No adaptation is observed to gaze directions relative to the head of the stimulus.•Adaptive coding of gaze direction occurs in a first-person coordinate system.•Gaze direction is a perceptual property abstracted from iris and head position. The primate visual system contains specialised neural mechanisms for encoding the direction of others’ gaze. A foundational question that we can ask in this domain concerns the frame of reference in which another person’s gaze is represented. Electrophysiological recordings in macaque monkeys suggest that direction of gaze might be encoded relative to the observer (i.e., in a first person reference frame) or relative to the head or body of the stimulus (i.e., in a second person reference frame). Here we use sensory adaptation to gaze direction as a psychophysical tool to probe how this property of the environment is represented in the human visual system. We find that strong perceptual aftereffects occur following adaptation to images that share a common direction of gaze in the first person reference frame, but vary in their direction of gaze in the second person reference frame. In contrast, sensory aftereffects do not occur following adaptation to images that share a common direction of gaze in the second person reference frame. These findings suggest that the adaptive coding of gaze direction in the human visual system occurs primarily in a first-person coordinate system. These data also provide strong support for perceptual constancy in the context of perceived gaze direction, implying that the direction of others’ attention is represented in the brain as a higher-level perceptual property abstracted from lower-level cues (e.g., iris position and head rotation).