This article is excerpted from
Guo Yangyang, Du Shuzeng, Qiao Yongliang, Liang Dong. Research and Application Progress of Deep Learning in Smart Livestock Breeding[J]. Smart Agriculture, 2023, 5(1): 52-65. doi:10.12133/j.smartag.SA202205009
GUO Yangyang, DU Shuzeng, QIAO Yongliang, LIANG Dong. Advances in the Applications of Deep Learning Technology for Livestock Smart Farming[J]. Smart Agriculture, 2023, 5(1): 52-65. doi:10.12133/j.smartag.SA202205009
CNKIRead the full article
Read the full article for free on the official website
Research and application progress of animal behavior recognition based on deep learning
Animal behavior can reflect their physical health and physiological conditions, which is an important basis for livestock management. At present, animal behavior is often monitored by both contact and non-contact methods. Contact monitoring method is to install one or more sensors on animals to complete data collection, and realize animal behavior recognition through data analysis and modeling; Non-contact monitoring methods are usually based on information such as images and videos, extract relevant features and build behavior recognition models. Deep learning is used in both monitoring methods.
1
Contact animal behavior recognition method
Based on electronic contact sensors and deep learning technology, it can overcome the disadvantages of traditional human monitoring methods, liberate labor, and improve the intelligent management of animal husbandry. At present, the commonly used sensors for animal behavior information collection mainly include accelerometers and sound sensors. By installing three-axis accelerometers and radio frequency identification (RFID) devices on animal body parts (Fig. 1) to obtain relevant behavior data, and then using deep learning technology for data analysis, the accuracy and recognition efficiency of animal behavior recognition can be greatly improved.
Fig.1 Wearing position of common cattle behavior detection sensors
（a）颈部 （b）腿部 （c）耳部
Fig. 1 Wear position of sensors for common cattle behavior detection
Wang et al. wore microphones around the necks of sheep and collected feeding sound signals, and compared the performance of deep neural networks (DNNs), CNNs and RNN networks in the recognition of feeding behaviors. The results showed that the accuracy of RNN, CNN and DNN models was 93.17%, 92.53% and 79.43%, respectively. Ying Yewei et al. proposed a prenatal behavior recognition method for ewes based on interval threshold and genetic algorithm optimization support vector machine (GA-SVM) classification model. In this method, wavelet noise reduction and contour extraction preprocessing were carried out on the acceleration data obtained based on the neck ring acquisition node, and the interval threshold classification method and GA-SVM method were used to realize the behavior recognition of ewes. The proposed method achieves an accuracy of 97.88%. Based on the three-axis acceleration data of sheep grazing, Zhang et al. used CNN to classify and identify three kinds of pastoring behaviors of sheep: feeding, chewing and rumination, and the average recognition rate of the method reached 93.8%. Hao Yusheng et al. obtained the motion state data of dairy cows based on Wi-Fi signals, obtained channel state information (CSI) sequence fragments containing cow movements through data processing, and finally constructed a dairy cow behavior recognition model through the LSTM model, with a recognition accuracy of 96.67%. Peng et al. used an inertial measurement unit to obtain cattle behavior data, and used a Long Short Term Memory-Recurrent Neural Network (LSTM-RNN) with long short-term memory to classify eight behaviors of cattle, with a classification accuracy of more than 80%. Hosseininoorbin et al. obtained the neck motion data of beef cattle through a sensor containing a three-axis accelerometer, and combined with deep learning technology to realize the recognition of multiple behaviors of beef cattle, among which the F1 value of class 2 was 94.9% and that of class 9 was 89.3%, indicating that it is feasible and effective to use deep learning technology to analyze sensor data.
Contact methods of obtaining information about animal behavior often require the installation of corresponding sensors on animals, which can lead to stress responses in animals and affect animal health and welfare. In addition, one of the main challenges of sensor-based behavioral modeling is the loss or deviation of data due to sensor failure and physical movement of the animal collar causing the sensor to reposition. In addition, in cattle farms, the communication signal is weak, and whether the data obtained by the sensor can be uploaded to the cloud in real time and completely needs to be tested in practice, and there is also a demand problem after obtaining the sensor data. At present, farmers not only pay attention to animal behavior, but also pay more attention to animal health issues, so the data needs to be further refined to achieve the guidance role of breeders, but this requires a large amount of data accumulation.
2
Non-contact animal behavior recognition method
At present, non-contact animal behavior recognition methods often obtain target images and videos through computer vision systems, and then realize biological visual feature and spatiotemporal feature extraction, target detection and behavior recognition classification through deep learning mode. The following describes the research status of non-contact animal behavior recognition methods from the perspectives of biological visual characteristics and spatiotemporal characteristics.
2.1 Research on behavior recognition based on biological visual features
The behavior recognition method based on biological visual features detects the target area from the sample video frame or image through a deep learning network and obtains the visual features of the area, and then trains and constructs a behavior recognition network model based on this feature. Currently, commonly used models include YOLOv3 and YOLOv4.
Kim et al. used YOLOv3, YOLOv4 and improved YOLOv3 to identify and classify pig feeding and drinking behaviors, and the accuracy was generally greater than 90%, but it was a difficult research point that overlapping or crowding caused by pig herd aggregation would lead to failure of behavior detection. Jiang et al. used the YOLOv4 model to detect the target area of goats, and obtained the location relationship information between the target area and the drinking and feeding areas, as well as the time movement of the centroid, so as to realize the identification and classification of feeding, drinking, activity and inactive behaviors. Wang Shaohua and He Dongjian improved the YOLOv3 model to obtain the visual characteristics of cows' climbing behavior during estrus, and trained the YOLOv3 network to detect cows' estrus behavior with an accuracy rate of 99.15%. Wu et al. proposed a method to classify lame cows and non-lame cows based on the YOLOv3 model and relative step size feature vectors, which used the YOLOv3 model to extract the visual features of key areas (such as legs and heads) of dairy cows and train the detection network model, combined with the relative step size of the front and rear legs of the cows on the basis of detecting the key regions to construct the feature vectors, and input them into the long short-term memory network classification model to judge whether the cows are lame, with a recognition accuracy of 98.57%. Ayadi et al. used the CNN model to extract the visual features of the mouth of dairy cows, and realized the detection and recognition of rumination behavior of dairy cows, with an accuracy of 95%.
However, in the above study, only image features were focused on, not temporal information. Moreover, the feature extraction in video frames or images is susceptible to interference from the external environment (such as light intensity, background color, buildings, etc.), so whether the behavior recognition model based on biological visual features can be applied to farm detection in different scenarios still needs to be further explored.
2.2 Research on behavior recognition based on spatio-temporal features
Animal behavior is a continuous process of movement, which contains time information in addition to spatial information, so extracting the temporal characteristics of animal behavior is of great significance for behavior recognition.
Chen et al. proposed a behavior recognition model combining Xception and LSTM, which extracted the spatial features of the image sequence through Xception, and input them into the LSTM to further extract the spatiotemporal features, and realized the detection of pig drinking behavior through Softmax. Guo et al. and Qiao et al. constructed a typical behavior recognition model of dairy cows based on BiGRU-attention and C3D-ConvLSTM, respectively, and identified the walking, standing, combing, exploration and feeding behaviors of dairy cows based on extracting the temporal characteristics of the video segment of dairy cow behavior, and verified them on the dairy cow dataset at different growth stages. The recognition accuracy of BiGRU-attention and C3D-ConvLSTM is about 82% and 95.5%, respectively. Jiang et al. used a single-stream optical flow convolutional network to detect the lameness behavior of dairy cows, and the accuracy rate was 98.24%. Wu et al. used the CNN-LSTM network model to identify the five behaviors of dairy cows: drinking, ruminating, walking, standing and resting, and first obtained the visual features of the images through the CNN network, and then obtained the spatio-temporal features through the LSTM, and the results were better than other deep learning models based on visual features or spatial features.
The above research results show that most of the studies extract the spatiotemporal features of behavior recognition by combining CNN models with LSTM models to extract spatiotemporal features of data, so as to improve the performance of behavior recognition models. However, there are still misidentification errors in similar behaviors, such as drinking water and playing, in the current study. Therefore, the research on behavior recognition based on spatiotemporal features still needs further research.
3
Quantitative analysis of behavior
At present, there is a lack of quantitative analysis of animal behavior (feeding frequency, exercise duration, rumination duration, etc.), but the analysis of the semantic relationship between animals and their surroundings, the statistical and quantitative analysis of different behaviors such as standing, eating, walking and climbing of domestic animals in the spatio-temporal domain, and the construction of behavior maps, can provide a scientific basis for the judgment of abnormal animal behaviors and precision breeding management. On the basis of detecting animal behavior, further quantitative research on the duration and amplitude of behavior is conducive to improving the management efficiency and efficiency of animal husbandry farms, such as the study of external factors affecting animal feeding, drinking water or rest behavior, which is conducive to the cognition of animal activity rules and improving management decision-making. At present, the impact on animal behavior is mainly qualitatively analyzed by changing feed, light, temperature, bedding and other factors, so as to provide a comfortable place for animals and improve animal welfare. However, most studies have documented changes in animal behaviour primarily through manual observation.
With the development of computer vision technology, this technology is also used to evaluate the correlation between animal behavior and the environment. Guo et al. used the background subtraction method and the ensemble method of inter-frame difference to detect the interaction behaviors between calves and the scene, and realized the recognition of the interaction behaviors between calves and the scene, such as feeding, drinking water and resting behaviors. The experimental results show that the accuracy of the method for recognizing the environmental interaction behaviors of cattle entering the barn, leaving the barn, remaining still (static behaviors such as standing and lying) and turning around are 94.38%, 92.86%, 96.85% and 93.51%, respectively. Costa et al. used image analysis techniques to explore the relationship between pig activity and environmental parameters (ventilation rate, temperature and humidity) in the pig house, aiming to study the influence of the environment on pig growth. Chen et al. proposed an image processing algorithm based on maximum entropy segmentation, HSV color space transformation and template matching based on CNN-LSTM detection of the target region of pigs, which calculated the roundness of the pig's head, the proportion of the feeding area of the head, the cumulative pixels of the head movement, and the distance from the head to the digital label on the pig's back to determine the identity and feeding time of each pig, and the accuracy rate of identifying pig feeding behavior was 95.9%.
As mentioned above, computer vision technology has achieved good results in assessing the correlation between animal behavior and the environment. However, there are still many problems in identifying the behavior of animals interacting with the environment, which are difficult to identify and misidentify. For example, the head of a calf is stationary in the feeding basin before and after eating, or the shadow of the head of a calf is incorrectly identified as a feeding behavior. However, the application of deep learning technology in this aspect is rare, and it still needs to be further explored in combination with actual needs.
4
brief summary
There are two main challenges to the use of deep learning methods for animal behavior recognition. First, the training of deep learning models requires large datasets, and deep learning-based recognition methods often have limitations when generalized to new datasets or other types of animals. In addition, the existing animal behavior recognition lacks the analysis of time dimension, spatial scene information and interaction between animals and the environment, and it is difficult to realize the semantic understanding and analysis of high-level complex behaviors. At present, on the basis of using deep learning technology to detect and identify animal behavior, there is still a lack of research on further mining the duration or frequency of behavior, and further exploration is needed. In addition, the detection and recognition of small behaviors also face some challenges, such as complex scenes, variable lighting, occlusion, contact and overlap between livestock, and deep learning technology combined with attention mechanism module has been proven to solve the problem of local detail recognition to a certain extent, and is used to identify livestock behavior, but it cannot record environmental variables, and still needs to be combined with sensors to build a complete monitoring system. The regulation of environmental parameters is conducive to providing a comfortable place for animals and improving animal health and welfare, so it is of great significance to study the quantitative relationship between environmental parameters and behavior.
Smart agriculture WeChat exchange service group
In order to facilitate the academic exchange of readers, authors and reviewers in the field of agricultural science, promote the development of smart agriculture, and better serve the readers, authors and reviewers, the editorial department has established a WeChat communication service group, where questions related to issues in the professional field and submissions can be consulted. How to join the group: Add the editor's WeChat 331760296, note: name, unit, research direction, the editor pulls you into the group, and the institutional marketing and advertising personnel do not disturb.
Post a call for submissions
Welcome to publish information such as the introduction of the scientific research team, innovative scientific research achievements and related activities on our official account.