Distinguishing transformative from incremental clinical evidence: A classifier of clinical research using textual features from abstracts and citing sentences

•It is important to know if a study changes or only supports the current standards of care for specific disease management.•A machine learning approach is proposed to distinguishing transformative from incremental clinical evidence.•Transformative research has typical language patterns in citing sentences. In clinical research and clinical decision-making, it is important to know if a study changes or only supports the current standards of care for specific disease management. We define such a change as transformative and a support as incremental research. It usually requires a huge amount of domain expertise and time for humans to finish such tasks. Faculty Opinions provides us with a well-annotated corpus on whether a research challenges or only confirms established research. In this study, a machine learning approach is proposed to distinguishing transformative from incremental clinical evidence. The texts from both abstract and a 2-year window of citing sentences are collected for a training set of clinical studies recommended and labeled by Faculty Opinions experts. We achieve the best performance with an average AUC of 0.755 (0.705–0.875) using Random Forest as the classifier and citing sentences as the feature. The results showed that transformative research has more typical language patterns in citing sentences than abstract sentences. We provide an efficient tool for identifying those clinical evidence challenging or only confirming established claims for clinicians and researchers.