Revisiting the Effectiveness of 3D Object Recognition Benchmarks

Recently, 3D computer vision has greatly emerged and become essential topic in both research and industry applications. Yet large scale 3D benchmark like ImageNet is not available for many 3D computer vision tasks such as 3D object recognition, 3D body motion recognition, and 3D scene understanding. Existing 3D benchmarks are not enough in the number of classes and quality of data samples, and reported performances on the datasets are nearly saturated. Furthermore, 3D data obtained with existing 3D sensors are noisy and incomplete causing unreliable evaluation results. In this work, we revisit the effectiveness of existing 3D computer vision benchmarks. We propose to refine and re-organize existing benchmarks to provide cheap and easy access but challenging, effective and reliable evaluation schemes. Our task includes data refinement, class category adjusting, and improved evaluation protocols. Biased benchmark subsets and new challenges are suggested. Our experimental evaluations on ModelNet40, a 3D object recognition benchmark, show that our revised benchmark datasets (MN40-CR and MN20-CB) provide improved indicators for performance comparison and reveals new aspects of existing methods. State-of-the-art 3D object classification and data augmentation methods are evaluated on MN40-CR and MN20-CB. Based on our extensive evaluation, we conclude that existing benchmarks that are carefully re-organized are good alternatives of large scale benchmark which is very expensive to build and difficult to guarantee data quality under immature 3D data acquisition environment. We make our new benchmarks and evaluations public.