Understanding image concepts using ISTOP model

This paper focuses on recognizing image concepts by introducing the ISTOP model. The model parses the images from scene to object׳s parts by using a context sensitive grammar. Since there is a gap between the scene and object levels, this grammar proposes the “Visual Term” level to bridge the gap. Visual term is a higher concept level than the object level representing a few co-occurring objects. The grammar used in the model can be embodied in an And–Or graph representation. The hierarchical structure of the graph decomposes an image from the scene level into the visual term, object level and part level by terminal and non-terminal nodes, while the horizontal links in the graph impose the context and constraints between the nodes. In order to learn the grammar constraints and their weights, we propose an algorithm that can perform on weakly annotated datasets. This algorithm searches in the dataset to find visual terms without supervision and then learns the weights of the constraints using a latent SVM. The experimental results on the Pascal VOC dataset show that our model outperforms the state-of-the-art approaches in recognizing image concepts. •In understanding an image there is a significant gap between scene level and object level.•ISTOP model can parse an image form scene level to visual term, object and part level by context sensitive grammar.•The visual term is a new concept which can bridge the gap between scene level and object level.•The context used in the grammar can improve object detection as well as visual term detection.