Defects recognition of pine nuts using hyperspectral imaging and deep learning approaches

[Display omitted] •Defect types of pine nut was identified using hyperspectral imaging at two spectral ranges.•1D CNN models using spectra as inputs obtained promising performances.•3D CNN models using hyperspectral images as input obtained equivalent performances.•1D CNN and 3D CNN models were visualized and interpreted by Grad-CAM++. Pine nuts, as a highly nutritious and medicinally valuable food, are susceptible to various defects during their cultivation, harvesting, and transportation, which can reduce their quality. Therefore, rapid and accurate identification of pine nut defect types is of utmost importance to ensure the overall quality of the pine nuts. In this study, hyperspectral imaging (HSI) systems covering two spectral ranges (400–1000 nm and 900–1700 nm) were employed to capture hyperspectral images of healthy pine nuts and pine nuts with six types of defects. One-dimensional (1D) and three-dimensional (3D) Convolutional Neural Network (CNN) models with multi-head attention mechanisms were constructed using 1D spectra and 3D hyperspectral images, respectively. To validate the effectiveness of the proposed models, Support Vector Classifier (SVC) models were built using 1D spectra and used as a comparison. Overall, the proposed CNN models outperform traditional machine learning methods in two spectral ranges (400–1000 nm and 900–1700 nm). 1D CNN model in the near-infrared spectral range (900–1700 nm) achieved an accuracy of 90.23 % on the training set and 81.32 % on the validation set. Additionally, the Generalized Gradient-Weighted Class Activation Mapping (Grad-CAM++) visualization method was applied to conduct visual analysis on the 1D CNN and 3D CNN models, enabling the identification of important wavelength ranges and pixel regions in the models, thereby enhancing the interpretability of the decision-making process of the models. Overall, the results of this study demonstrated the feasibility of using a combination of hyperspectral imaging and convolutional neural networks for pine nut defects classification, and the visual analysis of the models provided new insights and understanding for pine nut defects identification.