On-site human-robot collaboration for lunar exploration based on shared mixed reality

On-site human-robot collaboration is one form of lunar exploration, where astronauts work side-by-side with rovers to accomplish tasks. Mixed reality (MR) is being tried in astronaut-rover teams. While research is primarily focused on MR technology to facilitate natural human-robot interaction, it often neglects the importance of incorporating transparency into decision-making processes. Our proposal is to utilize shared MR and optimized perception-localization techniques for human-robot collaboration in lunar exploration. The establishment of consensus between astronauts and rovers during decision-making processes can be facilitated through shared spatial context and interactive holographic content. This technology combines the strengths of astronauts and rovers for decision-making during rover navigation missions. It avoids blindly relying on rovers or only using manual manipulation by astronauts. In order to improve terrain perception and facilitate visualization for astronaut-rover teams during lunar navigation tasks, we develop a risk-aware lunar terrain parsing method that utilizes multiscale eigenvalue-based features and an optimized Random Forest classifier. Our method outperforms others with an impressive accuracy of 94.2%. Our co-location MR system incorporates a marker & instance-based spatial anchor method, customized specifically for the unique topography of the lunar terrain and optimized for resource conservation. Our experiments on lunar navigation with three terrain conditions and four configurations confirm that that shared MR could improve task performance and reduce workload. The proposed shared MR paradigm and related technologies can provide a reference in future lunar exploration missions.