Method and apparatus of video coding using merge with MVD mode

A method and apparatus for video coding using MMVD mode are disclosed. According to this method, a first expanded merge MV is determined for the current block, where the first expanded merge MV is derived by adding a first selected offset from a first set of offsets to a base MV, and whether the first expanded merge MV is applied to a first reference picture in L0 or a second reference picture in L1 is determined implicitly by the decoder side, or the first expanded merge MV is applied to the first reference picture in the L0 and a second expanded merge MV is applied to the second reference picture in the L1. The current block is encoded or decoded by using motion information comprising the first expanded merge MV. According to another method, separate MVDs are used for reference pictures in different reference lists.