Visual recognition of gymnastic exercise sequences. Application to supervision and robot learning by demonstration

This work presents a novel software architecture to autonomously identify and evaluate the gymnastic activity that people are carrying out. It is composed of three different interconnected layers. The first corresponds to a Multilayer Perceptron (MLP) trained from a set of angular magnitudes derived from the information provided by the OpenPose library. This library works frame by frame, so some postures may be incorrectly detected due to eventual occlusions. The MLP layer makes it possible to accurately identify the posture a person is performing. A second layer, based on a Hidden Markov Model (HMM) and the Viterbi algorithm, filters the incorrect spurious postures. Thus, the accuracy of the algorithm is improved, leading to a precise sequence of postures. A third layer identifies the current exercise and evaluates whether the person is doing it at a correct speed. This layer uses an innovative Modified Levenshtein Distance (MLD), which considers not only the number of operations to transform a given sequence, but also the nature of the elements participating in the comparison. The system works in real time with little delay, thus recognizing sequences of arbitrary length and providing continuous feedback on the exercises being performed. An experiment carried out consisted in reproducing the output of the second layer on an autonomous Pepper robot that can be used in environments where physical exercise is performed, such as a residence for the elderly or others. It has reproduced different exercises previously executed by an instructor so that people can copy the robot. The article analyzes the current situation of the automated gymnastic activities recognition, presents the architecture, the different experiments carried out and the results obtained. The integration of the three components (MLP, HMM and MLD) results in a robust system that has allowed us to improve the results of previous works. •Novel architecture: MLP with OpenPose, HMM and Modified Levenshtein Distance.•Detection and evaluation of the user performing gymnastic exercises.•Robot learning of exercise sequences by demonstration.•Use of common 2D cameras, which simplifies the required setup.•Highly accurate detecting 98.05% of complete gymnastic activities.