Robust multi-dimensional motion features for first-person vision activity recognition

•We design a set of multi-dimensional motion features from first-person video.•We extract virtual inertial data from video only.•We combine motion magnitude, direction and dynamics with virtual inertial data.•The features are independent of the classifier and validated on multiple datasets.•Two new datasets are made available to the research community. We propose robust multi-dimensional motion features for human activity recognition from first-person videos. The proposed features encode information about motion magnitude, direction and variation, and combine them with virtual inertial data generated from the video itself. The use of grid flow representation, per-frame normalization and temporal feature accumulation enhances the robustness of our new representation. Results on multiple datasets demonstrate that the proposed feature representation outperforms existing motion features, and importantly it does so independently of the classifier. Moreover, the proposed multi-dimensional motion features are general enough to make them suitable for vision tasks beyond those related to wearable cameras.