Automatic neural network hyperparameter optimization for extrapolation: Lessons learned from visible and near-infrared spectroscopy of mango fruit

Configuring a neural network’s architecture and hyperparameters often involves expert intuition and hand-tuning to extrapolate well without overfitting. This paper considers automatic methods for configuring a neural network for the domain of visible and near-infrared (Vis-NIR) spectroscopy. In particular, we study the effect of (a) validation set choice for validating configurations and (b) using ensembles. We consider several validation set choices: a random sample of 33% of non-test data (the technique used in previous work), samples from the latest year (a harvest season), and the first, middle, and latest 33% of samples sorted by time. To test these methods, we do a comprehensive study of a held-out 2018 harvest season of mango fruit given Vis-NIR spectra from 3 prior years. We find that ensembling improves the state-of-the-art model’s variance and accuracy. Furthermore, hyperparameter optimization experiments show that when ensembling is combined with using the latest 33% of samples as the validation set, a neural network configuration is found automatically that performs as well as an expertly-chosen configuration. •Hyperparameter optimization (HPO) can overfit validation set.•Choice of validation (tuning) set affects HPO generalization performance.•Ensemble averaging improves HPO and prediction accuracy of neural networks.