Neighbor management module for use in video encoding and methods for use therewith

A motion compensation module can be used in a video encoder for encoding a video input signal. The motion compensation module includes a neighbor management module that generates and stores neighbor data for at least one macroblock of the plurality of macroblocks for retrieval for retrieval by at least one of a motion search module, a motion refinement module, a direct mode module, and an intra-prediction module, when operating on at least one neighboring macroblock of the plurality of macroblocks.