Method and apparatus for reordering candidates of merge with mvd mode in video coding systems

A method and apparatus for video coding using MVP candidates are disclosed. A set of MVP candidates is determined for the current block. Member candidates in the set of MVP candidates are reordered according to template matching costs associated with the member candidates. A diversity condition is checked for a target MVP candidate pair in the reordered set, where the diversity condition comprises a target matching cost difference for the target MVP candidate pair being smaller than a first threshold from a first set of at least one pre-defined value, or a target MV difference for the target MVP candidate being smaller than a second threshold. If the diversity condition is satisfied, one of the target MVP candidate pair is moved to a later position in the reordered set of MVP candidates to form a modified-reordered set of MVP candidates. The current block is encoded or decoded by using motion information comprising the modified-reordered set of MVP candidates.