The Effect of Standardization on the Reliability of the Philippine Board of Surgery Oral Examinations

Background Cognizant of potential problems with validity and reliability, the Philippine Board of Surgery (PBS) undertook standardization of its oral examination procedures. This retrospective analytic study was conducted to determine and compare the reliability of the Philippine Board of Surgery oral examinations before and after standardization. Methods The records of oral examinations from June 29, 2003 to March 28, 2010 were reviewed and measures of reliability (intraclass correlation coefficient and kappa agreement) were computed and compared between the time periods before and after standardization and among different case content modules administered. The proportion of passers between time periods and different content areas were also compared. Results All measures of interrater reliability (intraclass correlation coefficient and kappa agreement) increased significantly overall and in all content areas and regardless of examinee outcome. There was also a trend to increased proportion of passers overall and significant improvement in passing rate in 4 out of 6 content areas. Conclusions This study validates the role of standardization of the content and scoring in improving the reliability of the oral examination, which is crucial for high stakes certifying examinations.