Understanding production variability in multimodal communication

A comprehensive view of speech communication requires a consideration of dynamic characteristics in language users’ multimodal behaviors. This talk discusses dynamic properties of both verbal and non-verbal communication behaviors, focusing on variability in voice, speech, and co-speech gesture production. I present a series of experiments that leverage quantitative approaches to examine how the multimodal production units of the cognitive system vary in their dynamic control and realization in executing communication goals. The overarching hypothesis is that surface variations in multimodal behaviors are structured to reflect communicative intentions, exhibiting a tight relation between the linguistic system and recruited modalities. Supporting evidence emerges from two types of data: dynamic voice signals drawn from speech corpora of various speakers, speaking styles, and languages, and time-aligned multidimensional signals of speech and co-speech gestures (simultaneously recorded audio, kinematic, and visual signals). Voice variations are acoustically structured by both biologically relevant factors and factors that vary with language-specific phonology. Furthermore, the spatiotemporal patterning of vocal tract actions and cooccurring body movements systematically varies, reflecting the prosodic structure of language. The findings and the utility of the analytical tools employed in these studies have implications for comparative investigations of communicative behaviors in animals. [Work supported by the NSF.]