Feature selection impact analysis for statistical models

The disclosed embodiments provide a system for processing data. During operation, the system obtains a set of feature additions and an evaluation metric for assessing the performance of a statistical model. Next, the system automatically builds treatment versions of the statistical model using a set of baseline features for the statistical model and feature combinations generated using the feature additions. The system then uses a hypothesis test and a fixed set of feature values to compare a baseline value of the evaluation metric for a baseline version of the statistical model that is built using the set of baseline features with additional values of the evaluation metric for the treatment versions. Finally, the system outputs a result of the hypothesis test for use in assessing an impact of the feature combinations on a performance of the statistical model.