Is risk-adjusted mortality an indicator of quality of care in general surgery?: a comparison of risk adjustment to peer review

Profiling of hospitals using risk-adjusted mortality rates as a measure of quality is becoming increasingly frequent. We sought to determine the validity of this approach by comparing the risk-adjusted predicted mortality to the findings of concurrent peer review and retrospective chart review of deaths that occur on a general surgery service. Consecutive patients admitted to a busy general surgery service from January 2000 to January 2006 were prospectively entered into the Surgical Activity Tracking System. Rigorous, systematic peer review was performed concurrently by service members on all deaths. Adjudication was later validated by an independent senior surgeon. Three methodologies of risk adjustment (University Health Consortium, Physiological and Operative Severity Score for the enUmeration of Mortality, and the Charlson index) were used and compared the "excess mortality" predicted by each to the number of potentially preventable deaths determined by peer review. A total of 9623 patients were admitted and 75 died (0.7%). University Health Consortium and Physiological and Operative Severity Score predicted an excess mortality of 62 and 65 deaths, respectively; Charlson predicted that 73% of the cohort would be dead in 1 year. Concurrent and retrospective peer review found that death was potentially preventable in only 22 and 21 patients, respectively. Peer adjudication and extensive clinical review adds much to the analysis of an adverse outcome, similar to the "black box" in an airplane crash. Although methods of risk adjustment may be helpful in identifying patients for peer review, they should be used for internal process improvement and not published as metrics of hospital or provider performance.