A Preliminary Investigation of the Factors Related to the Design and Scoring of Video‐Based Oral Communication Performance Tasks in Higher Education

Oral communication has been identified as an important skill by higher education institutions and by the workforce community. Despite its importance, minimal research has been conducted around the development of tasks to measure oral communication skills and behaviors. The purpose of this preliminary study is to evaluate the different factors related to the design and scoring of oral communication tasks. For this study, oral communication refers specifically to video‐based oral presentations using a webcam. The study administered 2 informative and 2 persuasive speech tasks of varying length (2 or 4 minutes) to 8 college students. Performance was scored holistically on overall performance, and on content and delivery dimensions by 2 raters. Results show that (a) interrater reliability estimates may be impacted by task type and length, (b) there is no variation in the three scoring criteria, (c) a minimum of 2 tasks is necessary to achieve satisfactory reliability on an oral communication measure, and (d) supporting materials may impact student performance. Results of this preliminary study provide guidance on how to further this foundational research. Future research and next steps are discussed. Report Number: ETS RR–18‐09