Audience Response System-Based Evaluation of Intelligibility of Children’s Connected Speech – Validity, Reliability and Listener Differences

•Audience Response System (ARS)-based scores of intelligibility are valid•ARS-based intelligibility scores need to be based on a panel of listeners•Panels of clinicians and untrained listeners provide similar intelligibility scores•ARS-based evaluations are useful primarily in a research context We assessed audience response systems (ARS)-based evaluation of intelligibility, with a view to find a valid and reliable intelligibility measure that is accessible to non-trained participants. In addition, we investigated potential listener differences between pediatric speech and language pathologists (SLPs) and untrained adults. Sixteen one-minute samples of connected speech were compiled, collected from 14 children with a speech sound disorder (SSD) and two children with typical speech. 16 SLPs and 13 untrained adults participated in a series of ARS listening sessions, where they were fitted with headphones and hand controls, and instructed to click a button whenever they did not understand the child speaking. Listeners’ button clicks were registered and, for each speech sample, totaled into an (un)intelligibility index. The proportion of syllables perceived correctly – based on orthographic listener transcripts – was used as a reference score of intelligibility. The ARS-based intelligibility scores correlated strongly with the intelligibility reference score. Reliability was high across listener groups and weaker for single listeners. No significant difference was found between the evaluations of SLPs and untrained adults. ARS-based evaluation offers a valid and reliable measure of intelligibility of particular value in research as a practical tool for collecting input from listeners without experience or knowledge of SSDs. We stress that the ARS design presupposes a listener panel, and that evaluations obtained from individual listeners are predictably inadequate in terms of reliability.