Apparatus and method for generating new viewpoint image

An apparatus for generating a new viewpoint image includes: a new viewpoint image generation unit that shifts the foreground image region in the input image, based on a depth map to generate a new viewpoint image including a missing pixel region; a representative background pixel selection unit that selects background pixels located a predetermined number of pixels away from edge pixels, as representative background pixels; a texture direction map calculation unit that calculates a texture direction of each of the selected representative background pixels, to calculate a texture direction map indicating texture directions, each corresponding to one of missing pixels, based on locations of the representative background pixels and locations of the missing pixels included in the missing pixel region; and a hole-filling processing unit that fills the missing pixel region with the obtained background pixels in accordance with the texture direction map.