HY-POP: Hyperparameter optimization of machine learning models through parametric programming

Fitting a machine learning model often requires presetting parameter values (hyperparameters) that control how an algorithm learns from the data. Selecting an optimal model that minimizes error and generalizes well to unseen data becomes a problem of tuning or optimizing these hyperparameters. Typical hyperparameter optimization strategies involve discretizing the parameter space and implementing an iterative search procedure to approximate the optimal hyperparameter and model selection through cross-validation. Instead, for machine learning algorithms that are formulated as linear or quadratic programming (LP/QP) models, an exact solution to the hyperparameter optimization problem is obtainable through parametric programming without any approximation. First, the hyperparameter optimization problem is posed more naturally as a bilevel optimization. Second, using parametric programming theory, the bilevel optimization is reformulated into a single level problem. Exact solutions to the hyperparameter optimization problem for LASSO regression and LP L1-norm support vector machine (SVM) are derived and validated on example data.