Development of an online prediction system for soil organic matter and soil moisture content based on multi-modal fusion

•A lightweight model based on multimodal feature fusion is proposed.•Deep Pine Plow with Integrated Image and Spectral Acquisition Module Designed.•The prediction of SOM and SM content based on multi-source data was realized.•The effect of soil moisture in the field on image and spectral data was resolved. Realizing accurate determination of in situ soil organic matter (SOM) content and soil moisture (SM) content in the field is of great importance for improving agricultural production efficiency. However, the feature information of a single sensor is limited, and there is still a certain gap in modeling accuracy compared to traditional laboratory methods. The variations of SM in the field also interfere with data collection, limiting the application of single sensor. In order to realize the efficient detection of in-situ SOM and SM content, this study developed an online detection system for SOM and SM content based on the fusion of characteristic wavelengths, visible images and thermal imaging image features. Firstly, based on a vehicle-mounted platform, a visible-thermal imaging camera and a characteristic wavelength integration device were integrated in a deep pine plow to realize the simultaneous acquisition of in-situ soil multi-sensor data. Then, a lightweight multimodal network was constructed to obtain thermal visible light image features and characteristic wavelength features through branch networks, achieving deep fusion of different modal data and predicting SOM and SM content. Finally, output the forecasting results of SOM and SM content, and transmit the predictive information to the cloud platform for storage. It was verified that the Multi-modal proposed in this study worked best in the laboratory environment, with a predicted R2 of 0.91 and RMSE of 2.9 g/kg for SOM, and a predicted R2 of 0.92 and RMSE of 0.77 % for SM. Compared with single image or characteristic spectral data, the fusion of visible image and characteristic wavelength effectively improved the prediction accuracy of SOM. The real-time prediction of SM was realized by fusing thermal imaging data, and the elimination of the moisture effect of visible images and characteristic wavelengths was also realized with the help of deep fusion of Multi-modal network. After field validation, the R2 of the Multi-modal system was 0.84 and the RMSE was 5.0 g/kg for SOM, and the R2 of the SM was 0.88 and the RMSE was 1.03 %. Despite the differences in soil types between the validation field