Predicting the progression of mild cognitive impairment using machine learning: A systematic, quantitative and critical review

•We systematically and quantitatively review 234 experiments from 111 articles predicting the future progression to Alzheimer’s disease, reporting their characteristics in terms of algorithm, input features, methodological issues and performance measures.•We show that the best performances were achieved using cognition or fluorodeoxyglucose-positron emission tomography, whereas T1 magnetic resonance imaging lead to relatively lower performance.•We identify methodological issues regarding misuse of the test set in 26.5% of articles.•We show that short-term predictions are likely not to perform better than predicting that subjects stay stable over time.•We propose several guidelines for the development of methods aiming to predict the future, such as the need to pre-register the time-to-prediction and a careful choice of the control group that needs to be followed for the same period of time. [Display omitted] We performed a systematic review of studies focusing on the automatic prediction of the progression of mild cognitive impairment to Alzheimer’s disease (AD) dementia, and a quantitative analysis of the methodological choices impacting performance. This review included 172 articles, from which 234 experiments were extracted. For each of them, we reported the used data set, the feature types, the algorithm type, performance and potential methodological issues. The impact of these characteristics on the performance was evaluated using a multivariate mixed effect linear regressions. We found that using cognitive, fluorodeoxyglucose-positron emission tomography or potentially electroencephalography and magnetoencephalography variables significantly improved predictive performance compared to not including them, whereas including other modalities, in particular T1 magnetic resonance imaging, did not show a significant effect. The good performance of cognitive assessments questions the wide use of imaging for predicting the progression to AD and advocates for exploring further fine domain-specific cognitive assessments. We also identified several methodological issues, including the absence of a test set, or its use for feature selection or parameter tuning in nearly a fourth of the papers. Other issues, found in 15% of the studies, cast doubts on the relevance of the method to clinical practice. We also highlight that short-term predictions are likely not to be better than predicting that subjects stay stable over time. These issues highlight the importance