Use of Coordinated Local User Devices During a Shared Augmented Reality Session

A shared augmented reality system can support the sharing of video captured by a local user, using a head mounted display (HMD), with a remote user. The remote user may add augmented reality annotations (markings, notes, drawings) to certain objects within the environment captured within the video, where the annotations track the movement of those objects within the shared video. An HMD may not, however, provide a convenient interface for performing certain user-interface intensive tasks, which might be better performed on an additional device such as a mobile phone, tablet, or computer. During a shared augmented reality session, the additional device can be configured to communicate with a HMD such that certain tasks can be performed by the user through the additional device, and other tasks can be performed or experienced through the HMD. The additional device, the HMD and the remote user's device can communicatively coordinate during the session.