Deep learning-empowered digital twin for visualized weld joint growth monitoring and penetration control

[Display omitted] •Deep learning empowers digital twin for visualized welding process monitoring and control.•The state-of-the-art accuracy in BSBW prediction is obtained with CNN and composite weld images.•Weld joints are controlled accurately in real time to meet the quality requirement.•Visualized GUI helps users perceiving welding manufacturing intuitively and effectively. This paper presents an innovative digital twin to monitor and control complex manufacturing processes by integrating deep learning which offers strong feature extraction and analysis abilities. Taking welding manufacturing as a case study, a deep learning-empowered digital twin is developed as the visualized digital replica of the physical welding for joint growth monitoring and penetration control. In such a system, the information available directly from sensors including weld pool images, arc images, welding current and arc voltage is collected in pulsed gas tungsten arc welding (GTAW-P). Then, the undirect information charactering the weld joint geometry and determining the welding quality, including the weld joint top-side bead width (TSBW) and back-side bead width (BSBW), is computed/estimated by traditional image processing methods and deep convolutional neural networks (CNNs) respectively. Compared with single image source, weld pool image or arc image, the CNN model performs better when taking the 2-channel composite image combined by both as the input and the state-of-the-art accuracy in BSBW prediction with mean square error (MSE) as 0.047 mm2 is obtained. Then, a decision-making strategy is developed to control the welding penetration to meet the quality requirement and applied successfully in various welding conditions. By modeling the weld joint cross section as an ellipse, the developed digital twin is visualized to offer a graphical user interface (GUI) for users perceiving the weld joint growth intuitively and effectively.