METHOD FOR BUILDING SCENE REPRESENTATION WITH FEED-FORWARD CORRECTION FOR REAL-TIME VIEW SYNTHESIS

The present invention relates generally to the fields of computer vision and computer graphics to generate a multi-plane image (MPI) structure or a multi-layer image (MLI) structure as a scene representation from an arbitrary set of images, in particular, to methods for building a scene representation and electronic computing devices performing the same methods. The method for building a scene representation comprises receiving a set of images of the scene captured by different cameras and intrinsic and extrinsic parameters of the cameras; obtaining feature tensors for each image of the received set of images by extracting features from the each image of the received set of images by using a feature extractor of a trained artificial intelligence (AI) model; building a plane-sweep volume (PSV) by concatenating the obtained feature tensors in a depth direction of the PSV using the received intrinsic and extrinsic parameters of the cameras; building a multi-plane image (MPI) by aggregating features of the built PSV with a PSV-to-MPI aggregator of the trained AI model; obtaining a MPI having a RGBA texture by processing features of each plane of the built MPI by using a MPI-RGBA converter of the trained AI model; obtaining a set of images corresponding to the received set of images from the MPI having the RGBA texture by using the received intrinsic and extrinsic parameters of the cameras; calculating differences between the respective images of the received set of images and the respective images of the set of images obtained from the MPI having the RGBA texture by using an error function; building a PSV by concatenating the calculated differences in a depth direction of the PSV using the received intrinsic and extrinsic parameters of the cameras; updating the MPI by aggregating features of the MPI and features of the current PSV with a PSV-to-MPI aggregator of the trained AI model. La présente invention concerne d'une manière générale les domaines de la vision artificielle et des graphiques informatiques pour la génération d'une structure d'image multiplan (MPI) ou d'une structure d'image multicouche (MLI) comme représentation de scène à partir d'un ensemble arbitraire d'images, et en particulier, des procédés de construction d'une représentation de scène et des dispositifs informatiques électroniques mettant en œuvre ces mêmes procédés. Le procédé de construction d'une représentation de scène comprend la réception d'un ensemble d'images de la scène capturée