A cyber-physical prototype system in augmented reality using RGB-D camera for CNC machining simulation

Numerical control (NC) codes verification is an important issue in computer numerical control (CNC) machining simulation because wrong NC codes will lead to the workpiece scrap and collision. The NC code verification methods both in physical space and cyber space (such as 3D computer graphics environment) have been widely investigated in recent years. However, physical verification methods have the problems that the simulation takes time and improper operations may cause danger. On the other hand, cyber verification methods only support some types of machines and cannot reflect the actual conditions of machine tools. This study proposes a cyber-physical prototype system for NC codes verification and CNC machining simulation. Based on the RGB-D camera, the depth-to-stereo model is constructed to obtain the 3D information in images. Without connecting with the CNC controller, the cutting tool and workpiece coordinate system (WCS) movement information in physical space can be got from images captured by the RGB-D camera through a convolutional neural network (CNN). Workpiece size and NC codes are imported into cyber space to render virtual workpiece with augmented reality (AR) technology. So that the operator can directly see the virtual workpiece in the physical machining scene. The virtual workpiece is machined by the cyber-physical system according to cutting tool movement in physical space. This research further confirms the feasibility of using computer vision (CV) methods to build the cyber-physical CNC simulation system based on an RGB-D camera. The potential application of the system is to obtain simulation results from CNC machine tools (especially those that are forbidden to connect the controller) and transfer the machining results to the Internet of Things (IoT).