Multifaceted conceptual image indexing on the world wide web

► User-centered design of an automated multifaceted concept-based indexing framework. ► Automated classification of Web image contextual information into five semantic concept facets. ► Identification of the semantic relationships between the concepts. ► Experimental recall/precision gain over tf–idf and n-gram based indexing techniques. In this paper, we describe a user-centered design of an automated multifaceted concept-based indexing framework which analyzes the semantics of the Web image contextual information and classifies it into five broad semantic concept facets: signal, object, abstract, scene, and relational; and identifies the semantic relationships between the concepts. An important aspect of our indexing model is that it relates to the users’ levels of image descriptions. Also, a major contribution relies on the fact that the classification is performed automatically with the raw image contextual information extracted from any general webpage and is not solely based on image tags like state-of-the-art solutions. Human Language Technology techniques and an external knowledge base are used to analyze the information both syntactically and semantically. Experimental results on a human-annotated Web image collection and corresponding contextual information indicate that our method outperforms empirical frameworks employing tf–idf and location-based tf–idf weighting schemes as well as n-gram indexing in a recall/precision based evaluation framework.