On-farm NIR sensor for milk analysis: Exploitation of bias monitoring and bias correction

Long-term studies have shown a bias drift over time in the prediction performance of near-infrared spectroscopy measurement systems. This bias drift generally requires extra laboratory reference measurements to detect and correct for this bias. Since these reference measurements are expensive and time consuming, there is a need for advanced methodologies for bias drift monitoring and correction without the need for taking extra samples. In this study, we propose and validate a method to monitor the bias drift and two methods to tackle it. The first method requires no extra measurements and uses a modified version of Partial Least Squares Regression to estimate and correct the bias. This method is based on the assumption that the mean concentration of the predicted component remains constant over time. The second method uses regular bulk milk measurements as a reference for bias correction. This method compares the measured concentrations of the bulk milk to the volume-weighted average concentrations of individual milk samples predicted by the sensor. Any difference between the actual and calculated bulk milk composition is then used to perform a bias correction on the predictions by the sensor system. The effectiveness of these methods to improve the component prediction was evaluated on data originating from a custom-built sensor that automatically measures the NIR reflectance and transmittance spectra of raw milk on the farm. We evaluate the practical use case where models for predicting the milk composition are trained upon installation of the sensor at the farm, and later used to predict the composition of subsequent samples over a period of more than 6 months. The effectiveness of the fully unsupervised method was confirmed when the mean concentration of the milk samples remained constant, while the effectiveness reduced when this was not the case. The bulk milk correction method was effective when all relevant samples for the component were measured by the sensor and included in the analyzed bulk milk, but is less effective when samples included in the bulk which are not measured by the sensor system. When the necessary conditions are met, these methods can be used to extend the lifetime of deployed prediction models by significantly reducing the bias on the predicted values.