Automated machine learning test system

A computing device selects new test configurations for testing software. Software under test is executed with first test configurations to generate a test result for each test configuration. Each test configuration includes a value for each test parameter where each test parameter is an input to the software under test. A predictive model is trained using each test configuration of the first test configurations in association with the test result generated for each test configuration based on an objective function value. The predictive model is executed with second test configurations to predict the test result for each test configuration of the second test configurations. Test configurations are selected from the second test configurations based on the predicted test results to define third test configurations. The software under test is executed with the defined third test configurations to generate the test result for each test configuration of the third test configurations.