Eye tracking system with single point calibration

A head mounted display (HMD) comprises an eye tracking system configured to perform a calibration process using an eye tracking system of the HMD that includes determining a pupillary axis and/or determining an angular offset between the pupillary axis and the eye's true line of sight. The eye tracking system obtains an eye model captures images of the user's pupil while the user is looking at a target or other content displayed on the HMD. In some embodiments, the calibration process is based on a single image of the user's eye and is performed only once. For example, the process can be performed the first time the user uses the HMD, which stores the calibration data for the user in a memory for future use.