Annotation Generation From IMU-Based Human Whole-Body Motions in Daily Life Behavior

This article describes a stochastic framework for integrating human whole-body motions with natural language. Human whole-body motions in daily life are measured by inertial measurement units (IMU) and subsequently encoded into motion primitives. Sentences are manually attached to the human motion primitives for their descriptions. Two aspects of semantics and syntactics are represented by stochastic modules. One stochastic module trains the linking of motion primitives to words, and the other module represents word order in the sentence structure. These two modules are helpful toward converting human whole-body motions into descriptions, where multiple words are generated from the human motions by the first module, and the second module searches for syntactically consistent sentences consisting of the generated words. The proposed framework is tested on a large dataset of human whole-body motions and their descriptive sentences. The linking of human motions to natural language enables robots to understand observations of human behavior as sentences.