Evaluating information retrieval using document popularity: An implementation on MapReduce

Over the last few years, one major research direction of information retrieval includes user behaviour prediction. For that reason many models have been proposed aiming at the accurate evaluation of information retrieval systems and the best prediction of web search users׳ behaviour. In this paper we propose a new evaluation metric for information retrieval systems which employs two relevance factors; a relevance judgement grade and a popularity grade that represents users׳ vote for a document. We show that this new metric performs better than other evaluation metrics when expressing user behaviour. Moreover, in order to test the performance of our metric on different and scalable ranking algorithms, we develop a pairwise text similarity algorithm using cosine similarity, implemented on the MapReduce model, and then perform experiments on rankings generated by the algorithm.