Systems and methods for enabling user-controlled extended reality

Systems and methods are described for enabling user-controlled extended reality (XR). The systems and methods may access one or more frames, depicting an environment surrounding a first user, of a video feed captured by a first user's device, and classify each portion of the frames as a physical background of the environment, as depicting the first user, or as depicting an additional entity. First and second masks may be generated for the depictions of the first and second users, respectively, and such masks may respectively define boundaries of the first user's depiction and the second user's depiction at which the XR portion should not be inserted. The frames may be modified by combining portions of the frames belonging to the physical background with the first mask and the second mask, and the video feed comprising the modified frames may be encoded and transmitted to the second user device.