User Effect in Evaluating Personalized Information Retrieval Systems

Evaluation of personalized information retrieval (IR) systems is challenged by the user effect, which is manifested in terms of users’ inconsistency in relevance judgment, ranking and relevance criteria usage. Two empirical studies on evaluating a personalized search engine were performed. Two types of relative measures computed with different mathematical formulae were compared. The ranking similarity and the randomness of relevance criteria usage were estimated. Results show some undesirable personalization effects. Implications for the future development and research of adaptive IR systems are discussed.