Learning depth via leveraging semantics: Self-supervised monocular depth estimation with both implicit and explicit semantic guidance

•Proposed Semantic-aware Spatial Feature Modulation (SSFM) scheme to enforce category-specific depth distributions.•Proposed a semantic-guided ranking loss (SRL) to constrain depth to have consistent borders with the segmentations predictions.•Proposed the robust sampling strategy and prediction uncertainty weighting to alleviate semantic noise for depth estimation.•The proposed method outperforms the state-of-the-art methods by significant margins. Self-supervised monocular depth estimation has shown great success in learning depth using only images for supervision. In this paper, we propose to enhance self-supervised depth estimation with semantics and propose a novel learning scheme, which incorporates both implicit and explicit semantic guidances. Specifically, we propose to relate depth distributions to the semantic category information by proposing a Semantic-aware Spatial Feature Modulation (SSFM) scheme, which implicitly modulates the semantic and depth features in a joint learning framework. The modulation parameters are generated from semantic labels to acquire category-level guidance. Meanwhile, a semantic-guided ranking loss is proposed to explicitly constrain the estimated depth borders using the corresponding segmentation labels. To avoid the impact brought by erroneous segmentation labels, both robust sampling strategy and prediction uncertainty weighting are proposed for the ranking loss. Extensive experimental results show that our method produces high-quality depth maps with semantically consistent depth distributions and accurate depth edges, outperforming the state-of-the-art methods by significant margins.