Selective hand occlusion over virtual projections onto physical surfaces using skeletal tracking

A head mounted device provides an immersive virtual or augmented reality experience for viewing data and enabling collaboration among multiple users. Rendering images in a virtual or augmented reality system may include performing operations for capturing an image of a scene in which a virtual object is to be displayed, recognizing a body part present in the captured image, and adjusting a display of the virtual object based upon the recognized body part. The rendering operations may also include capturing an image with a body mounted camera, capturing spatial data with a body mounted sensor array, recognizing objects within the captured image, determining distances to the recognized objects within the captured image, and displaying the virtual object on a head mounted display.