Intuitive teleoperation with hand-tracking in VR: a study on master–slave system virtualization and 3D workspace visualization

This study proposed an intuitive teleoperation approach utilizing the hand-tracking mode supported by commercialized virtual reality devices. Unlike previous studies that used handheld controllers connected to an HMD (head-mounted display) system, this approach directly synchronizes the virtual hand (user’s hand recognized by the VR device), the 3D model of the telemanipulator, and the practical telemanipulator simultaneously. The entire system was developed by integrating the previously developed master–slave system with Unity, a cross-platform game engine. In this platform, the physical slave system was converted into a virtual 3D model, and numerical optimization techniques were applied to avoid the dependency on conventional URDF (unified robot description format) applications. To verify the feasibility of this approach, several experiments were conducted, including motion consistency between the user and the robot hand, and trajectory-following performance. The rest of this paper presents the approaches for visualization of the workspace in VR. First, the pros and cons of images acquired through RGB cameras were reviewed, and then the point cloud-based 3D reconstruction approach, which was the focus of this study, was introduced. Considered methods included visualizing point clouds with the Mesh Generation Method of Unity and mapping these clouds using BAD SLAM (Bundle Adjusted Direct Simultaneous Localization and Mapping) and Open3D. Furthermore, various camera placements were tested to balance visual clarity and task efficiency, suggesting future improvements in VR visual quality and manipulation control.