The Reliability of Multisource Feedback in Competency-Based Assessment Programs: The Effects of Multiple Occasions and Assessor Groups

PURPOSEResidency programs around the world use multisource feedback (MSF) to evaluate learners’ performance. Studies of the reliability of MSF show mixed results. This study aimed to identify the reliability of MSF as practiced across occasions with varying numbers of assessors from different profes...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Academic Medicine 2015-08, Vol.90 (8), p.1093-1099
Hauptverfasser: Moonen–van Loon, Joyce M.W, Overeem, Karlijn, Govaerts, Marjan J.B, Verhoeven, Bas H, van der Vleuten, Cees P.M, Driessen, Erik W
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:PURPOSEResidency programs around the world use multisource feedback (MSF) to evaluate learners’ performance. Studies of the reliability of MSF show mixed results. This study aimed to identify the reliability of MSF as practiced across occasions with varying numbers of assessors from different professional groups (physicians and nonphysicians) and the effect on the reliability of the assessment for different competencies when completed by both groups. METHODThe authors collected data from 2008 to 2012 from electronically completed MSF questionnaires. In total, 428 residents completed 586 MSF occasions, and 5,020 assessors provided feedback. The authors used generalizability theory to analyze the reliability of MSF for multiple occasions, different competencies, and varying numbers of assessors and assessor groups across multiple occasions. RESULTSA reliability coefficient of 0.800 can be achieved with two MSF occasions completed by at least 10 assessors per group or with three MSF occasions completed by 5 assessors per group. Nonphysicians’ scores for the “Scholar” and “Health advocate” competencies and physicians’ scores for the “Health advocate” competency had a negative effect on the composite reliability. CONCLUSIONSA feasible number of assessors per MSF occasion can reliably assess residents’ performance. Scores from a single occasion should be interpreted cautiously. However, every occasion can provide valuable feedback for learning. This research confirms that the (unique) characteristics of different assessor groups should be considered when interpreting MSF results. Reliability seems to be influenced by the included assessor groups and competencies. These findings will enhance the utility of MSF during residency training.
ISSN:1040-2446
1938-808X
DOI:10.1097/ACM.0000000000000763