HRI: human reasoning inspired hand pose estimation with shape memory update and contact-guided refinement

Hand pose estimation is a challenging task in hand-object interaction scenarios due to the uncertainty caused by object occlusions. Inspired by human reasoning from a hand-object interaction video sequence, we propose a hand pose estimation model. It uses three cascaded modules to imitate human’s estimation and observation process. The first module predicts an initial pose based on the visible information and the prior hand knowledge. The second module updates the hand shape memory based on the new information coming from the subsequent frames. The bone’s length updating is initiated by the predicted joint’s reliability. The third module refines the coarse pose according to the hand-object contact state represented by the object’s Signed Distance Function field. Our model gets the mean joints estimation error of 21.3 mm, the Procrustes error of 9.9 mm, and the Trans &Scale error of 22.3 mm on HO3Dv2, and Root-Relative error of 12.3 mm on DexYCB which are superior to other state-of-the-art models.