Technique for estimating motion and occlusion

A method and apparatus for estimating motion and occlusion is disclosed. In one aspect, a method of estimating motion and occlusion between three or more frames in a video stream includes identifying a plurality of motion vector candidates for each of a forward direction and a backward direction, generating a plurality of candidate pairs, determining an energy for at least some of the plurality of candidate pairs using a processor by jointly evaluating the forward motion vector and the backward motion vector of at least some of the plurality of candidate pairs based on interframe difference, spatial motion correlation, temporal motion correlation, and spatial occlusion correlation, and estimating motion and occlusion between the three or more frames by selecting a candidate vector from the plurality of candidate vectors based on the determined energies.