MULTIMODAL METHOD AND APPARATUS FOR SEGMENTATION AND DEPTH ESTIMATION

A multimodal neural network model for combined depth estimation and semantic segmentation of images and a method of training the multimodal neural network model. The multimodal neural network comprising a single encoder, a depth decoder to estimate the depth of the image and a semantic segmentation decoder to determine semantic labels from the image. The method for training the multimodal neural network model comprising receiving a plurality of images at a single encoder, after encoding the images providing them to a depth estimation decoder and a semantic segmentation decoder to estimate the depth of the images and semantic labels to the images. The method further comprising comparing the estimated depth with the actual depth of the images and comparing the calculated semantic labels with the actual labels of the images to determine a depth loss and a semantic segmentation loss, respectively.