Inter-rater reliability of ACS-NSQIP colorectal procedure coding in Canada

The American College of Surgeons National Surgical Quality Improvement Project (ACS-NSQIP) uses Current Procedural Terminology (CPT) codes for risk-adjusted calculations. This study evaluates the inter-rater reliability of coding colorectal resections across Canada by ACS-NSQIP surgical clinical nurse reviewers (SCNR) and its impact on risk predictions. SCNRs in Canada were asked to code simulated operative reports. Percent agreement and free-marginal kappa correlation were calculated. The ACS-NSQIP risk calculator was utilized to illustrate its impact on risk prediction. Responses from 44 of 150 (29.3 ​%) SCNRs revealed 3 to 6 different codes chosen per case, with agreement ranging from 6.7 ​% to 62.3 ​%. Free-marginal kappa correlation ranged from moderate agreement (0.53) to high disagreement (−0.17). ACS-NSQIP risk calculator predicted large absolute differences in risk for serious complications (0.2 ​%–13.7 ​%) and mortality (0.2 ​%–6.3 ​%). This study demonstrated low inter-rater reliability in coding ACS-NSQIP colorectal procedures in Canada among SCNRs, impacting risk predictions. •This study demonstrated low inter-rater reliability across all ACS-NSQIP colorectal resection procedure coding in Canada among SCNRs.•Coding inconsistencies resulted in significant variation in predicted morbidity and mortality, when using the NSQIP risk prediction calculator.•These results highlight the importance of ongoing efforts to improve coding standardization and education among healthcare professionals.