Spectrum-image dual-modality fusion empowered accurate and efficient classification system for traditional Chinese medicine

Traditional Chinese medicine (TCM) influences the Chinese and global medical systems, with its quality essential to its effectiveness. The origin of TCM material impacts the quality of the same TCM materials. However, the existing origin classification methods of the same TCM materials from different places mainly have two disadvantages: slow processing speed and extensive experience. To address these issues, a fast and real-time technology, laser-induced breakdown spectroscopy (LIBS), is introduced into our solution. We propose a TCM classification system that combines one-dimensional LIBS spectra with two-dimensional images. This dual-modality fusion approach represents a significant advancement in multi-view data analysis for TCM classification. As a case study, we focus on wolfberry and construct a new dataset comprising 10,800 pairs of LIBS spectrum and image data to fill the gap. To achieve superior multiple feature fusion, a two-stage fusion network (TFNet) in a coarse-to-fine way is proposed. In the first coarse fusion, the Depth Attention Fusion (DAF) module is applied to extract the key features of stacked spectrum and image. In the second fine fusion, the Line to Area (LTA) module entirely focuses on and highlights the critical spectral line features. Experimental accuracy is over 0.99 with less computation and parameters, indicating the high efficiency and accuracy of the proposed TFNet. Therefore, the classification system achieves exceptional accuracy and efficiency due to its simple sample preparation, real-time data collection and the high-accuracy lightweight network. •A spectrum-image dual-modality fusion empowered TCM classification system.•A two-stage network for multiple feature fusion of multi-view spectrum-image data.•A dual-modality dataset filling the data gap for Chinese wolfberry.