A Mobile 3D Object Recognition Processor with Deep Learning-based Monocular Depth Estimation

A 3D object recognition system is a heavy task that consumes high sensor power and requires complex 3D data processing. In this article, the proposed processor produces 3D RGB-D data from an RGB image through a deep learning-based monocular depth estimation, and then its RGB-D data is sporadically calibrated with low-resolution depth data from a low-power depth sensor, lowering the sensor power by 27.3×. Then, the proposed processor accelerates various convolution operations in the system by integrating the in-out skipping-based bit-slice-level computing processing elements and flexibly allocating workloads considering data properties. Moreover, the point feature aggregator is designed close to the global memory to support the point feature reuse algorithm's data aggregation. Additionally, the window-based search algorithm and its memory management are presented for efficient point processing in the point processing unit. Consequently, the 210 mW and 34 fps end-to-end 3D object recognition processor is successfully demonstrated.