Potato disease detection and prevention using multimodal AI and large language model

•The multimodal AI model for the potato disease detection.•Develop an online platform (PotatoGPT) using large language model.•The multimodal AI model was generalized to other Solanaceae diseases. Potatoes are susceptible to early blight and late blight, which cause significant damage to their yield and quality. This study proposes a multimodal AI model for the detection of early blight and late blight in potatoes, demonstrating exceptional detection performance. Firstly, the MSC-ResViT detection model was proposed from a visual branch perspective, achieving an accuracy of 92.15 % on the test set, surpassing other deep learning networks. Secondly, MSC-TextCNN and CT-CNN were introduced to detect potato diseases based on textual descriptions and image statistical feature (color and texture), with accuracy of 96.86 % and 74.87 %. The multimodal AI model, constructed based on MSC-ResViT, MSC-TextCNN, and CT-CNN, achieved an accuracy of 98.43 % on the test set. Subsequently, the multimodal AI model was generalized to other Solanaceae diseases (tomato and eggplant), achieving satisfactory detection results. Finally, an online potato disease diagnosis and control platform (PotatoGPT) was developed based on the multimodal AI model and the large language model (GPT-4). The multimodal AI model can achieve high-accuracy potato disease detection in complex scenarios, making it highly significant for realizing intelligent disease detection.