Percentile Analysis for Goodness-of-Fit Comparisons of Models to Data

In cognitive modeling, it is routine to report a goodness-of-fit index (e.g., R2 or RMSE) between a putative model's predictions and an observed dataset. However, there exist no standard index values for what counts as good or bad, and most indices do not take into account the number of data points in an observed dataset. These limitations impair the interpretability of goodness-of-fit indices. We propose a generalized methodology, percentile analysis, which contextualizes goodness-of-fit measures in terms of performance that can be achieved by chance alone. A series of Monte Carlo simulations showed that the indices of randomized models systematically decrease as the number of data points to be fit increases, and that the relationship is nonlinear. We discuss the results of the simulation and how computational cognitive modelers can use them to place commonly used fit indices in context. Published in the Proceedings of the 36th Annual Conference of the Cognitive Science Society. Cognitive Science Society, held in Quebec City, Canada, p737-742, 23-26 July 2014.