3D reconstruction system and multiobject local tracking algorithm designed for billiards

The use of virtual reality or augmented reality systems in billiards sports are useful tools for pure entertainment or improving the player’s skills. Depending on the purpose of these systems, tracking algorithms based on computer vision must be used. These algorithms are especially useful in systems aiming to reconstruct the trajectories followed by the balls after a strike. However, depending on the billiard modality, the problem of tracking multiple small identical objects, such as balls, is a complex task. In addition, when an amateur or nontop professional player uses low-frame-rate and low-resolution devices, problems such as blurred balls, blurred contours, or fuzzy edges, among others, arise. These effects have a negative impact on ball-tracking accuracy and reconstruction quality. Thus, this work proposes two contributions. The first contribution is a new tracking algorithm called “multiobject local tracking (MOLT)” . This algorithm can track balls with high precision and accuracy even with motion blur caused by low-resolution and low-frame-rate devices. Moreover, the proposed MOLT algorithm is compared with nine tracking methods and four different metrics, outperforming the rest of the methods in the majority of the cases and providing a robust solution. The second contribution is a whole system to track (using the MOLT algorithm) and reconstruct the movements of the balls on a billiard table in a 3D virtual world using computer vision. The proposed system covers all steps from image capture to 3D reconstruction. The 3D reconstruction results have been qualitatively evaluated by different users through a series of questionnaires, obtaining an overall score of 7.6 (out of 10), which indicates that the system is a promising and useful tool for training. Finally, both the MOLT algorithm and the reconstruction system are tested in three billiard modalities: blackball, carom billiards, and snooker.