Self-supervised learning with randomized cross-sensor masked reconstruction for human activity recognition

Self-supervised learning (SSL) has gained prominence in the field of accelerometer-based human activity recognition (HAR) due to its ability to learn from both labeled and unlabeled data. While labeled data acquisition is costly, it is relatively easy to accumulate unlabeled sensor data. However, few works utilize large-scale, unlabeled datasets for pre-training despite its positive impact on downstream HAR performance, shown in recent work. Cross-sensor upstream training has also received limited attention. We introduce a new auxiliary task, randomized cross-sensor masked reconstruction (RCSMR), for SSL. We pre-train a transformer encoder on the large-scale HUNT4 dataset with RCSMR. The resulting model exhibits better performance on two downstream datasets with the same sensor setup as HUNT4 (HARTH and HAR70+), achieving an average F1-score of 74.03%, surpassing two other auxiliary tasks (70.51% to 72.78%) and five supervised baselines (47.51% to 58.84%). Moreover, when applied to three datasets with sensor configurations distinct from HUNT4 (USC-HAD, PAMAP2, MobiAct), RCSMR outperforms nine state-of-the-art SSL methods, with an F1-score of 72.99% compared to F1-scores ranging from 51.46% to 69.88%. We further show that certain activities exhibit improved separability when utilizing latent representations learned through RCSMR, indicating reduced sensor position and orientation bias. Our method is applied in large-scale epidemiological studies, offering valuable insights into the impact of physical activity behavior on public health. [Display omitted] •New self-supervised cross-sensor learning auxiliary task (RCSMR) for HAR.•Large-scale dual-sensor pre-training, single-sensor downstream training.•RCSMR pre-trained Transformer outperforms supervised & SSL methods on 7 HAR datasets.•RCSMR has better activity separability in the latent space than other auxiliary tasks.•RCSMR is effective on limited amounts of labeled data and different model sizes.