Inter-rater reliability of a reflective rubric to assess pharmacy students’ reflective thinking

Many health education programs aspire to foster reflective practices to enhance the reflective thinking of students during their study years and throughout their professional careers. Given the increasing attention paid to incorporating reflective practice activities into health education and evidence attributed to rater variability, it is important for educators to carefully consider the assessment criteria and reliability of assessment tools. A reflective rubric was developed to assess pharmacy students’ levels of reflection from a reflective writing task. This study investigates the inter-rater reliability (IRR) of a rubric as a tool to assess the levels of reflective thinking apparent in students’ written statements. The research involved four raters from different disciplines utilizing a rubric to assess a random sample of the same forty-three reflective statements from a cohort of two hundred and sixty-four students. The IRR was measured using the intra-class correlation coefficient (ICC), using a two-way random effects model (ANOVA) with absolute agreement, to determine reliability of the assessment tool among the raters. Results showed measures between the raters for (i) overall reflective statement scores, and (ii) average scores for stages of reflection with an “almost perfect” agreement, ICC = 0.81 (95% CI 0.61–0.90), (F(42, 126) = 7.83, p < 0.01); and ICC = 0.89 (95% CI 0.83–0.93), (F(42, 840) = 12.49, p < 0.01) respectively. The proposed rubric utilized by four raters showed high agreement with each other's scores, and is a tool for academic assessment of pharmacy students’ reflective thinking processes.