Real-time oriented behavior-driven 3D freehand tracking for direct interaction

Articulated hand tracking systems have been commonly used in virtual reality applications, including selection–move–release systems that use freehand as input device. However, building an effective real-time hand tracker remains a challenge. Motivated by the need for three-dimensional (3D) freehand-based human–computer interface in the selection–move–release systems, the current study aims to develop 3D freehand tracking, in which the high dimensionality of articulated hand models is one of the dominating obstacles for tracking 3D hand in real-time. This study focuses on forming the behavioral model for users in selection–move–release systems to build a natural, direct, and effective human–computer interface. First, the statistical models for users were learned. Second, the behavioral models were derived from the statistical models to form a general behavioral model database. Third, a freehand tracking algorithm for interaction between human and computer was presented based on the behavioral models. Fourth, the proposed approach was tested in several real selection–move–release systems, and the experimental results were provided. Unlike previous studies on this subject, the proposed behavioral model was composed of Correlation among Local Motion Models and cognitive information. Experimental results show that the proposed algorithm can achieve satisfactory results in relation to time cost and accuracy compared with some previous studies. ► Behavioral model is composed of correlation among Local Motion Models and cognitive information. ► We dig out, organize and visualize cognitive information, forming cognitive model for users. ► A new freehand tracking algorithm based on the behavioral models for human–computer interaction.