Assessment of reliability in orthodontic literature

To map the statistical methods applied to assess reliability in orthodontic publications and to identify possible trends over time. Original research articles published in 2009 and 2019 in a subset of orthodontic journals were downloaded. Publication characteristics, including publication year, number of authors, single vs multicenter study, geographic origin of the study, statistician involvement, study category, subject category, types of reliability assessment, and statistical methods applied to assess reliability, were recorded. Descriptive statistics, Chi-square tests, and logistic regression analyses were performed to investigate associations between reliability analysis and study characteristics. A total of 768 original research articles were analyzed. The most prevalent study category was observational (69%) with a statistician involved in 16% of studies. Overall, reliability was assessed in 47% of studies, and the most frequent methods applied to assess reliability were intraclass correlation coefficients or kappa statistics (60.4%). The odds of applying appropriate methods were greater in 2019 than in 2009 (odds ratio [OR]: 2.43; 95% confidence interval [CI]: 1.75, 3.37; P < .001). Involvement of a statistician resulted in greater odds of applying appropriate methods compared to no statistician involvement (OR: 1.88; 95% CI: 1.23, 2.87; P < .01). Over the past decade (2009 vs 2019), reliability assessment became more common in the orthodontic literature, and studies applying correct statistical methods to assess reliability significantly increased. This trend was more apparent in studies that involved a statistician, which may highlight the role of the statistician.