3D Spatial Mapping in a 3D Coordinate System of an AR Headset Using 2D Images

Technology is described for generating 3D spatial mappings. One operation may be receiving a 2D image of patient anatomy from a medical imaging device. Another operation may be identifying a placement of an imaging device in a 3D coordinate system of an augmented reality (AR) headset when capturing the 2D image. A graphical mark may be received from a user for a point within a body of a person in the 2D image. A 3D coordinate of the point in the 3D coordinate system may be computed using the position and orientation of the imaging device with respect to the AR headset and a depth of the point, in order to generate a 3D spatial mapping.