Using tracking of display device to control image display

Techniques are described for using information from tracking position of a display device to control display of image data. The display device may, for example, be a head-mounted display ("HMD") device used for virtual reality ("VR") and/or augmented reality ("AR") display of images showing part of a simulated environment around a user wearing the HMD device, and the tracking including determining a position of the HMD device in an actual physical environment (e.g., location and/or orientation in 3D space). Operations of the HMD device or other display device and of an associated image rendering system that provides images for display is improved by determining and using information about a latency or other delay between acquiring tracking data for the device and displaying corresponding images on the device, including to initiate a safe mode operation of the device if the determined delay is too large.