M-NeuS: Volume rendering based surface reconstruction and material estimation

Although significant advances have been made in the field of multi-view 3D reconstruction using implicit neural field-based methods, existing reconstruction methods overlook the estimation of the material information (e.g. the base color, albedo, roughness, and metallic) during the learning process. In this paper, we propose a novel differentiable rendering framework, named as material NueS (M-NeuS), for simultaneously achieving precise surface reconstruction and competitive material estimation. For surface reconstruction, we perform multi-view geometry optimization by proposing an enhanced-low-to-high frequency encoding registration strategy (EFERS) and a second-order interpolated signed distance function (SI-SDF) for precise details and outline reconstruction. For material estimation, inspired by the NeuS, we first propose a volume-rendering-based material estimation strategy (VMES) to estimate the base color, albedo, roughness, and metallic accurately. And then, different from most material estimation methods that need ground-truth geometric priors, we use the geometry information reconstructed in the surface reconstruction stage and the directions of incidence from different viewpoints to model a neural light field, which can extract the lighting information from image observations. Next, the extracted lighting and the estimated base color, albedo, roughness, and metallic are optimized by the physics-based rendering equation. Extensive experiments demonstrate that our M-NeuS can not only reconstruct more precise geometry surface than existing state-of-the-art (SOTA) reconstruction methods but also can estimate competitive material information: the base color, albedo, roughness, and metallic. •We propose an encoding strategy to achieve more accurate surface information reconstruction.•We propose a second-order interpolated signed distance function for precise geometry information optimization.•We propose a material estimation strategy utilizing SDF weights and the physics-based rendering equation.