A comparative review of plausible hole filling strategies in the context of scene depth image completion

•A comparative survey of plausible depth image completion approaches is presented.•A taxonomy is drawn based on information domain and input requirements.•The literature is also categorized based on early principles of inpainting.•The goal is to provide researchers with the means to select the right approach. [Display omitted] Despite significant research focus on 3D scene capture systems, numerous unresolved challenges remain in relation to achieving full coverage scene depth estimation which is the key part of any modern 3D sensing system. This has created an area of research where the goal is to complete the missing 3D information post capture via a secondary depth filling process. In many downstream applications, an incomplete depth scene is of limited value, requiring many special cases for subsequent utilization, and thus techniques are required to “fill the holes” that exist in terms of both missing depth and color scene information. An analogous problem exists within the scope of scene filling post object removal in the same context. Although considerable research has resulted in notable progress in the synthetic expansion or reconstruction of missing color scene information in both statistical (texture synthesis) and structural (image completion) forms, work on the plausible completion of missing scene depth is contrastingly limited. This survey aims to provide a state of the art overview within this growing field of depth synthesis work whilst noting related solutions in the space of traditional texture synthesis and color image completion for hole filling. To these ends, we concentrate on the plausible completion of both underlying depth structure and relief texture to provide both greater understanding and future development in the area. Our analyses are in part supported by illustrative experimental examples of the comparative use of a subset of representative approaches over common depth completion examples.