Variability in plus disease diagnosis using single and serial images

The purpose of this study is to assess changes in ROP diagnosis in single and serial retinal images. Cohort study Cases of retinopathy of prematurity recruited from the i-ROP consortium evaluated by seven graders Seven ophthalmologists reviewed both single and three consecutive serial retinal images from 15 ROP cases, and severity was assigned as plus, pre-plus, or none. Imaging data was acquired during routine ROP screening from 2011 to 2015 and a reference standard diagnosis was established for each image. A secondary analysis was performed using the i-ROP deep learning system to assign a vascular severity score (VSS) to each image, ranging from 1-9, with 9 being most severe disease. This score has been previously demonstrated to correlate with the International Classification of ROP. Mean plus disease severity was calculated by averaging 14 labels per image in serial and single images to decrease noise. Grading severity of ROP as defined by plus, pre-plus, or no ROP Assessment of serial retinal images changed the grading severity for >50% of the graders, though there was wide variability. Cohen’s kappa ranged from 0.29 to 1.0, which showed a wide range of agreement from slight to perfect by each grader. Changes in grading of serial retinal images was noted more commonly in cases of pre-plus disease. The mean severity in cases with a diagnosis of plus disease and no disease did not change between single and serial images. The ROP vascular severity score demonstrated good correlation with the range of expert classifications of plus disease, and overall agreement with the mode class (p=0.001). The VSS correlated with mean plus disease severity by expert diagnosis (correlation coefficient 0.89). More aggressive graders tended to be influenced by serial images to increase the severity of their grading. The vascular severity score also demonstrated agreement with disease progression across serial images which progressed to pre-plus and plus disease. Clinicians demonstrated variability in ROP diagnosis when presented with both single and serial images. The use of deep learning as a quantitative assessment of plus disease has the potential to standardize ROP diagnosis and treatment.