PTU-63 Semi-automated annotation tool outperforms medical students and is comparable to clinical experts for polyp detection

IntroductionExpert labelling of each frame in a polyp video is the most robust way for constructing a training set for deep learning, but this is very time-consuming and currently represents a major barrier for widespread implementation of AI in endoscopy. In this study, two alternative approaches are evaluated, an innovative semi-automated labelling tool and trained medical students providing annotations.Methods20 unique polyp white light videos containing 6282 frames (14 adenomas and 6 sessile serrated lesions confirmed by histopathology, mean size 7mm, Olympus) were annotated with bounding boxes by a clinical expert. These annotations are used as the gold standard for comparison. Two cheaper annotation methods were then applied to evaluate their validity and relative performance: (1) a semi-automated labelling technique – this tool only requires 3 manually annotated video frames, from which a representation of the polyp is learned and transferred automatically to all the other frames in the video; (2) independent manual labelling of each video by three medical students – following a training module with polyp images and videos.ResultsThe mean and standard deviation of the frame-level sensitivity, positive predictive value (PPV) and adjudicated PPV (for borderline low-quality frames) over all videos are provided in table 1. The semi-automated method significantly outperforms all three students on sensitivity and annotation time (paired t-test, p-value < 0.05), while also achieving the highest value for PPV, both before and after adjudication.Abstract PTU-63 Table 1 Sensitivity PPV Adjudicated PPV Time (mins) Student 1 74,38 ± 27,30 88,52 ± 30,51 89,92 ± 15,34 264 Student 2 63,08 ± 20,27 94,69 ± 22,30 95,00 ± 07,47 1208 Student 3 66,97 ± 27,37 94,77 ± 22,32 95,00 ± 12,30 234 Semi-automated 94,40 ± 06,22 97,17 ± 05,87 98,97 ± 14,04 25 ConclusionsA semi-automated labelling tool is a faster, more efficient and valid approach for polyp detection. It outperforms three medical students, specifically trained for polyp recognition and is comparable to clinical expert performance.