Formative Evaluation in Health Education: An Exploratory Study of the Usefulness of the Cognitive Response Method

Purpose: The purpose of this study was to explore the usefulness of a simplified version of cognitive response analysis (CRA) as a formative evaluation tool for health education materials developed for small, unfunded projects. Design/methodology/approach: A group of women (n=15), aged between 25 and 50 years, provided cognitive responses (CRs) that communicated their ongoing thoughts after viewing each slide in a narrated, online slide presentation that focused on nutrient content claims. Participants were assigned in a systematic random fashion to one of three CRA variations. In all three variations, participants simultaneously saw a slide and heard its narration and then provided their CRs while still viewing the slide. In Variation 1, a researcher wrote participants' CRs on a log sheet (n=5). Variation 2 participants' audiotaped their CRs which were later transcribed by a researcher (n=5). Variation 3 participants wrote their CRs on a log sheet. To assess the quality and usefulness of the CRs generated by each variation, the researchers categorized each CR into content-related categories. Findings: The time and effort required by the researcher and participants for all three variations of collecting CRs was similar. However, transcribing the audiotaped CRs presented a greater time burden to the researcher. Analysis of variance revealed that Variation 3 generated significantly fewer CRs than the other two variations. In addition, CRs from those in Variation 3 tended to be shorter and less specific and were less useful in refining the presentation because they provided limited guidance on needed improvements. In contrast, the CRs generated by Variations 1 and 2 were judged to be more useful in identifying improvements that could optimize the value of the presentation. Originality/value: The results suggest that a simplified version of CRA is a valuable, efficient, and low-cost tool for formative evaluation of health education materials. (Contains 1 table.)