Assessing Model Fit by Cross-Validation

When QSAR models are fitted, it is important to validate any fitted modelto check that it is plausible that its predictions will carry over to fresh data not used in the model fitting exercise. There are two standard ways of doing thisusing a separate hold-out test sample and the computationally much more burdensome leave-one-out cross-validation in which the entire pool of available compounds is used both to fit the model and to assess its validity. We show by theoretical argument and empiric study of a large QSAR data set that when the available sample size is smallin the dozens or scores rather than the hundreds, holding a portion of it back for testing is wasteful, and that it is much better to use cross-validation, but ensure that this is done properly.