Continuous Person Identification and Tracking in Healthcare by Integrating Accelerometer Data and Deep Learning Filled 3D Skeletons

With the technological development in healthcare, environments such as rehabilitation clinics and patients' houses, are increasingly monitored by multi-device systems. To aggregate information, overcome privacy issues and device failures, it is essential to match measurements from different sources and associate them to a particular patient. While cameras are used to detect and track anonymized persons, wearable devices can acquire inertial and health information. The challenge is to correctly pair the tracked persons to their status, such as heart rate, collected by other devices. Recently, many works have been proposed, in several scenarios, to tackle sensor fusion-based tracking using a large variety of information. However, when the budget is limited, the involved sensing devices lack of inertial components, such as gyroscope, and may have low precision. In this work, we propose a novel solution to match unlabeled 3D skeletons, detected by a depth camera, with on-wrist wearable devices equipped only with accelerometer. Additionally, a Deep Learning submodule, named SkeletonRNN, is introduced to overcome camera failures in the 3D skeletons points detection and fill missing joints. A complete dataset containing skeletons and accelerations measurements, of dailies and rehabilitation activities, has been collected, manually annotated, and is available for testing purposes. We trained and tested the SkeletonRNN using data augmentation on our dataset, the final average 3D point prediction error is 11.00cm and the skeleton-device pairing accuracy of the overall system is 76.62% on a total of 231 chucks. Datasets, code and experiments can be found at https://github.com/matteo-bastico/SkeletonRNN .