Enhanced VNIR and MIR proximal sensing of soil organic matter and PLFA-derived soil microbial properties through machine learning ensembles and external parameter orthogonalization

•Machine learning improves VNIR-MIR models for field-condition soils compared to PLSR.•Cubist, SVM and Ensemble-GLM achieved the most accurate soil property predictions.•EPO improves VNIR-MIR model transfer from pre-treated to field-condition soils.•Models for pre-treated (dried/ground) samples still give the most accurate results.•Machine learning coupled with EPO facilitates on-site VNIR and MIR analysis. Portable visible-to-near-infrared (VNIR) and mid-infrared (MIR) spectroscopy coupled with machine learning can provide detailed and inexpensive information on various key soil properties. However, on-site VNIR and MIR proximal sensing applications are hampered by soil moisture and particle size variations, which distort reflectance spectra collected on field-condition soils and impede the integration of established MIR and VNIR soil spectral libraries in predictive models for field measurements. In this study, we explored the capacity of various machine-learning approaches to calibrate VNIR-MIR models for the prediction of soil organic carbon and phospholipid fatty acid (PLFA)-derived microbial soil properties with field-condition spectral data. We further evaluated the potential to integrate soil spectral libraries into VNIR-MIR proximal sensing applications by testing the transfer of VNIR-MIR models calibrated on pre-treated soil samples to field-condition VNIR-MIR scans using the External Parameter Orthogonalization (EPO) approach to minimize soil moisture and particle size effects. We compiled a diverse soil dataset encompassing a wide range of organic matter content, soil texture, and parent material from soils under grassland and arable land use (n = 175). VNIR-MIR models were used to predict soil organic carbon (SOC), bacterial biomass (BAC), fungal biomass (FUN), and different soil quality indicators (C:N, Fungal-to-bacterial ratio, gram-positive-to-gram-negative ratio) for both field-condition and pre-treated soil spectral data. Calibrations were developed with Partial Least Squares Regression (PLSR), Random Forest (RF), Elastic Net (ENET), Cubist, Support Vector Machines (SVM), and an Ensemble-GLM. We further tested the effectiveness of coupling each machine-learning model with the EPO algorithm to transfer models calibrated on pre-treated soils to field-condition scans. Our results show that machine learning methods such as Cubist and SVM readily outperformed the standard PLSR calibration, with average improvements of ΔRMSE ∼15 % for pre-trea