Natural Scene Statistics Account for the Representation of Scene Categories in Human Visual Cortex

During natural vision, humans categorize the scenes they encounter: an office, the beach, and so on. These categories are informed by knowledge of the way that objects co-occur in natural scenes. How does the human brain aggregate information about objects to represent scene categories? To explore this issue, we used statistical learning methods to learn categories that objectively capture the co-occurrence statistics of objects in a large collection of natural scenes. Using the learned categories, we modeled fMRI brain signals evoked in human subjects when viewing images of scenes. We find that evoked activity across much of anterior visual cortex is explained by the learned categories. Furthermore, a decoder based on these scene categories accurately predicts the categories and objects comprising novel scenes from brain activity evoked by those scenes. These results suggest that the human brain represents scene categories that capture the co-occurrence statistics of objects in the world. •Brain activity encodes scene categories that reflect real-world object statistics•These scene categories are represented in many anterior visual regions of interest•Scene categories and individual objects can be decoded from measured brain activity•fMRI signals contain more information about categories than previously appreciated Object context influences human scene categorization, but an objective explanation of this phenomenon has been absent. Stansbury et al. report that much of anterior visual cortex encodes scene categories that capture the co-occurrence statistics of objects in natural scenes.