Distinct effects of different visual cues on sentence comprehension and later recall: The case of speaker gaze versus depicted actions

Language-processing accounts are beginning to accommodate different visual context effects, but they remain underspecified regarding differences between cues, both during sentence comprehension and subsequent recall. We monitored participants' eye movements to mentioned characters while they listened to transitive sentences. We varied whether speaker gaze, a depicted action, neither, or both of these visual cues were available, as well as whether both cues were deictic (Experiment 1) or only speaker gaze (Experiment 2). Speaker gaze affected eye movements during comprehension similarly early to a single deictic action depiction, but significantly earlier than non-deictic action depictions; conversely, depicted actions but not speaker gaze positively affected later recall of sentence content. Thus, cue type and cue-language relations must be accommodated in characterising real-time situated language comprehension and subsequent recall of sentence content. •We examine the effects of two contextual cues on spoken sentence comprehension.•We compare anticipatory fixations to upcoming referents of the sentence.•Listeners rapidly follow the speaker's gaze to upcoming characters.•Depicted actions also allow anticipation, but only after semantic processing.•Initial data suggests that cues differ in how they affect later recall of sentence content.