Learning based estimation of hand and finger pose

A method for processing data includes receiving a depth map of a scene containing a human hand, the depth map consisting of a matrix of pixels having respective pixel depth values. The method continues by extracting from the depth map information based on the depth values in a plurality of positions distributed over the human hand and processing the information in order to estimate respective candidate positions of the finger joints. The pose of the human hand is estimated by choosing a combination of the positions of the finger joints, responsively to anatomical constraints of the hand, that gives a hand configuration that is most anatomically probable among the candidate positions.