Method and apparatus for displaying panoramas with streaming video

In this invention streaming video data is embedded within a panoramic image for display. The invention teaches a method and system for displaying an image of a scene of interest from a stored environment map of the scene and from video data of the scene. The method includes the steps of: (A) receiving and storing the video data of the scene; (B) selecting an orientation of the scene; (C) retrieving video data according to the selected orientation of the scene; (D) rendering the environment map according to the selected orientation of the scene to generate a first image for the selected orientation; (E) synchronously combining the retrieved video data and the first image to form a composite image for the selected orientation of the scene of interest; and (F) displaying the composite image. In one embodiment the method for displaying the image of the scene of interest further includes a step of continuously receiving video data packets that include image data and sound data. In this embodiment the sound data is continuously played while selected portions of the image data are displayed. In another embodiment, the method for displaying the image of the scene of interest further includes a step of layering the video data and the first image generated by rendering the environment map. In the present invention layering techniques include, for example, drawing order layering, z-buffer-hiding layering, and chroma-key layering.