NeuGPU: An Energy-Efficient Neural Graphics Processing Unit for Instant Modeling and Real-Time Rendering on Mobile Devices

Neural radiance field (NeRF) is an emerging computer graphics task that is used for 3-D modeling and rendering in the metaverse, providing a user-friendly and immersive experience. It successfully replaces conventional 3-D modeling methods such as photogrammetry thanks to its simplicity, photo-realistic rendering quality, and versatility. However, it has limitations to be accelerated on mobile augmented reality (AR)/virtual reality (VR) devices due to its memory-intensive hash encoding and extensive computational load. Previous NeRF application specified integrated circuits (ASICs) only supported high frame-rate rendering while being incapable of immediate 3-D model creation. In this article, we present neural graphics processing unit (NeuGPU) to achieve both instant 3-D modeling and real-time rendering using NeRF with three key features. First, segmented hashing with spatial pruning (SHSP) partitions hash table (HT) into multiple segments and reduces 66% of external memory access (EMA). Spatial management unit (SMU) supports SHSP operation through three types of sub-block management. Second, attention-based hybrid interpolation unit (AHIU) optimizes heterogeneous memory access characteristics of hash-based NeRF, achieving 56.4% power reduction. Third, similarity-sparsity skipping (S 3 ) core supports energy-efficient multi-layer perceptron (MLP) operation by applying coarse-grained skip for both similar and sparse data. As a result, NeuGPU is fabricated in a 28-nm process and occupies a 20.25-mm 2 die area. Evaluated on the synthetic NeRF dataset, NeuGPU finally achieves 8.7 \times faster modeling and 231.4 \times smaller energy per iteration compared with edge GPU even though it has 31 \times smaller external memory bandwidth.