A gesture recognition algorithm in a robot therapy for ASD children

•A Residual Neural Network (ResNet) was designed to classify small/similar gestures.•The gesture recognition system was within a new robot therapy for autistic children.•The trained ResNet was exploited online for interactive feedback during therapy.•Offline classification reached 95% test accuracy in our training Dataset.•Online classification achieved 79% F1-score during clinical application. Children with Autism Spectrum Disorders (ASDs) exhibit significant impairments in gesture imitation. Newest interventions are based on Human-Robot Interaction (HRI) since children with ASD cope well with stylized, rule-based and predictable systems. These collaborative approaches encompass therapy games based on joint exercises, imitation and interaction between robots and children. This paper’s aim was to implement an algorithm to automatically recognize small and similar gestures within a humanoid-robot therapy called IOGIOCO for ASD children. IOGIOCO is a multi-level HRI therapy meant to teach 19 meaningful gestures in a semantic framework based on a feedback interaction. Gestures were tracked as 3D coordinates of body keypoints captured by a Kinect. A Residual Neural Network was implemented and trained on a segmented Dataset acquired within this study to generate the offline model which was then exploited in a real-time classification using a sliding window. Feedback as sound stimuli from NAO robot was provided based on the automatic evaluation of each performance. Clinical acquisitions were carried out on 4 ASD children within the IOGIOCO therapy. Offline recognition was successful: exploiting Artificial Neural Networks we reached 95% of test accuracy for 19 gestures. A real-time recognition on healthy subjects reached 94% accuracy. Clinical applications were evaluated through the F1 score that achieved 79% value. These outcomes were encouraging considering the wide gesture set and all the challenges the therapy raises. This kind of automatic algorithm was able to decrease the therapist workload and increase the robustness of the therapy and engagement of the child.