Strategies for noise reduction and standardization of milk mid-infrared spectra from dairy cattle

The use of Fourier-transform mid-infrared (FTIR) spectroscopy is of interest to the dairy industry worldwide for predicting milk composition and other novel traits that are difficult or expensive to measure directly. Although there are many valuable applications for FTIR spectra, noise from differences in spectral responses between instruments is problematic because it reduces prediction accuracy if ignored. The purpose of this study was to develop strategies to reduce the impact of noise and to compare methods for standardizing FTIR spectra in order to reduce between-instrument variability in multiple-instrument networks. Noise levels in bands of the infrared spectrum caused by the water content of milk were characterized, and a method for identifying and removing outliers was developed. Two standardization methods were assessed and compared: piecewise direct standardization (PDS), which related spectra on a primary instrument to spectra on 5 other (secondary) instruments using identical milk-based reference samples (n = 918) analyzed across the 6 instruments; and retroactive percentile standardization (RPS), whereby percentiles of observed spectra from routine milk test samples (n = 2,044,094) were used to map and exploit primary- and secondary-instrument relationships. Different applications of each method were studied to determine the optimal way to implement each method across time. Industry-standard predictions of milk components from 2,044,094 spectra records were regressed against predictions from spectra before and after standardization using PDS or RPS. The PDS approach resulted in an overall decrease in root mean square error between industry-standard predictions and predictions from spectra from 0.190 to 0.071 g/100 mL for fat, from 0.129 to 0.055 g/100 mL for protein, and from 0.143 to 0.088 g/100 mL for lactose. Reductions in prediction error for RPS were similar but less consistent than those for PDS across time, but similar reductions were achieved when PDS coefficients were updated monthly and separate primary instruments were assigned for the North and South Islands of New Zealand. We demonstrated that the PDS approach is the most consistent method to reduce prediction errors across time. We also showed that the RPS approach is sensitive to shifts in milk composition but can be used to reduce prediction errors, provided that secondary-instrument spectra are standardized to a primary instrument with samples of broadly equivalent milk compo