Evaluation of semantic retrieval systems on the semantic web

Purpose – This paper seeks to analyze and evaluate different types of semantic web retrieval systems, with respect to their ability to manage and retrieve semantic documents. Design/methodology/approach – The authors provide a brief overview of knowledge modeling and semantic retrieval systems in order to identify their major problems. They classify a set of characteristics to evaluate the management of semantic documents. For doing the same the authors select 12 retrieval systems classified according to these features. The evaluation methodology followed in this work is the one that has been used in the Desmet project for the evaluation of qualitative characteristics. Findings – A review of the literature has shown deficiencies in the current state of the semantic web to cope with known problems. Additionally, the way semantic retrieval systems are implemented shows discrepancies in their implementation. The authors analyze the presence of a set of functionalities in different types of semantic retrieval systems and find a low degree of implementation of important specifications and in the criteria to evaluate them. The results of this evaluation indicate that, at the moment, the semantic web is characterized by a lack of usability that is derived by the problems related to the management of semantic documents. Originality/value – This proposal shows a simple way to compare requirements of semantic retrieval systems based in DESMET methodology qualitatively. The functionalities chosen to test the methodology are based on the problems as well as relevant criteria discussed in the literature. This work provides functionalities to design semantic retrieval systems in different scenarios.