Can medical record reviewers reliably identify errors and adverse events in the ED?

Abstract Background Chart review has been the mainstay of medical quality assurance practices since its introduction more than a century ago. The validity of chart review, however, has been vitiated by a lack of methodological rigor. Objectives By measuring the degree of interrater agreement among a 13-member review board of emergency physicians, we sought to validate the reliability of a chart review–based quality assurance process using computerized screening based on explicit case parameters. Methods All patients presenting to an urban, tertiary care academic medical center emergency department (annual volume of 57,000 patients) between November 2012 and November 2013 were screened electronically. Cases were programmatically flagged for review according to explicit criteria: return within 72 hours, procedural evaluation, floor-to-ICU transfer within 24 hours of admission, death within 24 hours of admission, physician complaints, and patient complaints. Each case was reviewed independently by a 13-member emergency department quality assurance committee all of whom were board certified in emergency medicine and trained in the use of the tool. None of the reviewers were involved in the care of the specific patients reviewed by them. Reviewers used a previously validated 8-point Likert scale to rate the (1) coordination of patient care, (2) presence and severity of adverse events, (3) degree of medical error, and (4) quality of medical judgment. Agreement among reviewers was assessed with the intraclass correlation coefficient (ICC) for each parameter. Results Agreement and the degree of significance for each parameter were as follows: coordination of patient care (ICC = 0.67; P < .001), presence and severity of adverse events (ICC = 0.52; P = .001), degree of medical error (ICC = 0.72; P < .001), and quality of medical judgment (ICC = 0.67; P < .001). Conclusion Agreement in the chart review process can be achieved among physician-reviewers. The degree of agreement attainable is comparable to or superior to that of similar studies reported to date. These results highlight the potential for the use of computerized screening, explicit criteria, and training of expert reviewers to improve the reliability and validity of chart review–based quality assurance.