Low Interrater Reliability in Grading of Rectal Bleeding Using National Cancer Institute Common Toxicity Criteria and Radiation Therapy Oncology Group Toxicity Scales: A Survey of Radiation Oncologists

Purpose To measure concordance among genitourinary radiation oncologists in using the National Cancer Institute Common Toxicity Criteria (NCI CTC) and Radiation Therapy Oncology Group (RTOG) grading scales to grade rectal bleeding. Methods and Materials From June 2013 to January 2014, a Web-based survey was sent to 250 American and Canadian academic radiation oncologists who treat prostate cancer. Participants were provided 4 case vignettes in which patients received radiation therapy and developed rectal bleeding and were asked for management plans and to rate the bleeding according to NCI CTC v.4 and RTOG late toxicity grading (scales provided). In 2 cases, participants were also asked whether they would send the patient for colonoscopy. A multilevel, random intercept modeling approach was used to assess sources of variation (case, respondent) in toxicity grading to calculate the intraclass correlation coefficient (ICC). Agreement on a dichotomous grading scale (low grades 1-2 vs high grades 3-4) was also assessed, using the κ statistic for multiple respondents. Results Seventy-two radiation oncologists (28%) completed the survey. Forty-seven (65%) reported having either written or been principal investigator on a study using these scales. Agreement between respondents was moderate (ICC 0.52, 95% confidence interval [CI] 0.47-0.58) when using NCI CTC and fair using the RTOG scale (ICC 0.28, 95% CI 0.20-0.40). Respondents who chose an invasive management were more likely to select a higher toxicity grade ( P