GENERATING A CONFIGURATION PORTFOLIO INCLUDING A SET OF MODEL CONFIGURATIONS

This disclosure relates to implementing a configuration portfolio having a compact set of model configurations that are predicted to perform well with respect to a wide variety of input tasks. Systems described herein involve evaluating machine learning models with respect to a set of training tasks to generate a regret matrix based on accuracy of the machine learning models in connection with predicting outputs for the training tasks. The systems described herein can identify a subset of model configurations from a plurality of model configurations based on the subset of model configurations having lower associated metrics of regret with respect to the training tasks. This ensures that each model configuration within the configuration portfolio will perform reasonably well for a given input task and provides a mechanism for selecting an output model configuration using significantly fewer processing resources than conventional model selection systems.