Development and validity of computerized neuropsychological assessment devices for screening mild cognitive impairment: Ensemble of models with feature space heterogeneity and retrieval practice effect

[Display omitted] •Screening for mild cognitive impairment at the early stage is critical.•Previous approaches do not fully consider the heterogeneity of feature collected by computerized neuropsychological assessment devices.•Three steps of feature engineering（item development, filter, and wrapper）helped us form the optimal feature combination for inclusion in the decision model of Memory Guard.•The accuracy of Memory Guard reached 93.75% and 0.923 of area under curve.•Time features were incorporated into the model to further improve the accuracy of classification.•The development process of Memory Guard can provide a methodological reference for screening tools. This study aimed to develop and validate computerized neuropsychological assessment devices for screening patients with mild cognitive impairment (MCI). We conducted this study in three phases. Phase I involved the development of a conceptual framework of Memory Guard (MG) based on the principles of the cognitive design system (CDS). Phase II involved three steps of feature engineering: item development, filter, and wrapper. Based on the initial items, the number of items in each dimension was determined through analytic hierarchy process. We constructed an initial set with a total of 198 items with three levels of difficulty. Next, we performed feature selection through comprehensive reliability and validity tests, which resulted in the best item bank of 38 test items. The features for modeling were obtained from the best item bank (option scores, reading time scores and total time scores), demographic variables and their MoCA groups. Regarding the heterogeneity of the feature space, we combined the AdaBoost with the Naive Bayes classification algorithm as the decision model of MG. For the screening tool to be used repeatedly, the retrieval practice effect was considered in the design. Phase III involved the validation of measuring instruments. The features incorporated into the modeling process were optimized based on the classification accuracy and area under curve. We also verified the classification effect of the other three classification models with MG. After three steps of feature engineering, a total of 6 dimensions of cognitive areas were included in MG: orientation, memory, attention, calculation, recall, and language & executive function. 38 features were included in the model (17 features of option score, 20 features of time score, and 1 demographic feature). A total of 333 individua