Multimodal neural network with clustering-based drop for estimating plant water stress

•A novel methodology using multimodal data for estimating plant water stress is proposed.•Multimodal neural network with dynamic attention is applied to extract appropriate multimodal features.•A new end-to-end deep learning method for multimodal data is proposed.•The proposed method is evaluated using real cultivation data.•The proposed method demonstrates more accurate and stable water stress prediction. Decision-making with low-cost data is an attractive approach in the field of agriculture as it aids to solve the difficulty of inheriting advanced cultivation technologies. To provide expertise in the decision-making process for stress cultivation, precision irrigation based on plant water stress is required to steadily produce high-quality fruits. Single low-cost data namely, single-modal data, is used in the traditional approach. However, for advanced cultivation, multimodal data such as physiological and meteorological data is required. In this study, we propose a multimodal neural network with clustering-based drop (C-Drop) for accurate estimation of plant water stress, as it is an index for irrigation decision-making, using plant image and environmental data. Our proposed method extracts temporal multimodal features from leaf wilting features (physiological data) using environmental features (meteorological data) as an attention mechanism of a multimodal neural network that includes long short-term memory layers. Moreover, the proposed neural network with C-drop realizes a novel end-to-end deep learning architecture in consideration of environmental conditions. On evaluating this method against the existing methods, the proposed method was found to improve the accuracy of plant water stress estimation by 21% for mean absolute error and root-mean-squared error, thereby indicating that this method is precise and stable for the plant water stress estimation. The performance of our proposed method to support precision irrigation will allow new-age farmers to produce high-quality fruits steadily.