Using a CNN-LSTM for basic behaviors detection of a single dairy cow in a complex environment

•A CNN-LSTM network was proposed to recognize the basic behaviors of a single cow.•Accurate recognition of single cow’s basic behavior could be got in complex scenes.•It is helpful for building intelligent behavior monitor system of dairy cows. The basic behaviors of dairy cows (drinking, ruminating, walking, standing and lying) are closely related to their physiological health status. Consequently, intelligent behavior recognition is of significance for the automatic diagnosis and precision farming of dairy cows. Realizing the accurate behaviors classification in complex environments involving low quality surveillance videos, complex illumination and weather changes is a key problem in dairy farming that must be solved. In this study, CNN-LSTM (fusion of convolutional neural network and long short-term memory) an algorithm for recognizing the basic behaviors of a single cow, was proposed. First, the VGG16 trained on ImageNet was used as the network skeleton to extract the feature vector sequence corresponding to each video, so as to avoid the shortcomings of traditional feature engineering which were time-consuming and laborious. Then, these features were input Bi-LSTM (bidirectional long short-term memory) classification model, which could extract semantic information of time series data in two directions, so as to realize accurate recognition of dairy cow’s basic behaviors. To verify the effectiveness of the VGG16 feature extraction network used in this research, 1370 segments of approximately 18 h of videos collected from dairy farm monitoring cameras were tested and compared with those of five different feature extraction networks based on VGG19, ResNet18, ResNet101, MobileNet V2 and DenseNet201. Moreover, the effects of changes in illumination, weather, and wind velocity on behaviors recognition were tested and discussed. The test results indicated that the precision of the proposed algorithm for the recognition of the five behaviors ranged from 0.958 to 0.995, the recall ranged from 0.950 to 0.985, and the specificity ranged from 0.974 to 0.991, while the average precision, recall and specificity were 0.971, 0.965 and 0.983, respectively. The average recognition accuracy of the proposed method was 0.976, which was higher than the methods based on VGG19, ResNet18, ResNet101, MobileNet V2 and DenseNe201 by 0.08 × 10−2, 1.97 × 10−2, 2.19 × 10−2, 2.85 × 10−2 and 2.34 × 10−2, respectively. Furthermore, the influences of illumination, weather and wind spe