Interobserver Variability in the International Study Group for Pancreatic Surgery (ISGPS)-Defined Complications After Pancreatoduodenectomy: An International Cross-Sectional Multicenter Study

To determine the interobserver variability for complications of pancreatoduodenectomy as defined by the International Study Group for Pancreatic Surgery (ISGPS) and others. Good interobserver variability for the definitions of surgical complications is of major importance in comparing surgical outcomes between and within centers. However, data on interobserver variability for pancreatoduodenectomy-specific complications are lacking. International cross-sectional multicenter study including 52 raters from 13 high-volume pancreatic centers in 8 countries on 3 continents. Per center, 4 experienced raters scored 30 randomly selected patients after pancreatoduodenectomy. In addition, all raters scored 6 standardized case vignettes. This variability and the "within centers" variability were calculated for 2-fold scoring (no complication/grade A vs grade B/C) and 3-fold scoring (no complication/grade A vs grade B vs grade C) of postoperative pancreatic fistula, postpancreatoduodenectomy hemorrhage, chyle leak, bile leak, and delayed gastric emptying. Interobserver variability is presented with Gwet AC-1 measure for agreement. Overall, 390 patients after pancreatoduodenectomy were included. The overall agreement rate for the standardized cases vignettes for 2-fold scoring was 68% (95% CI: 55%-81%, AC1 score: moderate agreement), and for 3-fold scoring 55% (49%-62%, AC1 score: fair agreement). The mean "within centers" agreement for 2-fold scoring was 84% (80%-87%, AC1 score; substantial agreement). The interobserver variability for the ISGPS-defined complications of pancreatoduodenectomy was too high even though the "within centers" agreement was acceptable. Since these findings will decrease the quality and validity of clinical studies, ISGPS has started efforts aimed at reducing the interobserver variability.