Evaluation of an Information System in an Information Seeking Process

This paper presents a holistic evaluation of an operational information system that employs the Boolean search technique. An equal focus is laid on both the system (system perspective) and its users (user perspective) in the actual environment where the system and its users are functioning (contextuality). In addition to these research objectives, the study has a methodological objective to test an evaluation approach developed by Borlund [1] in a real life setting. Our evaluation methodology involves triangulation (pre-search questionnaires; search log; post-interviewing) as well as novel interactive performance measures, such as the Ranked Half-Life measure and the Satisfaction and Novelty perception by users supplementing the traditional Precision. The study confirms the finding of earlier research and reveals the discrepancy between the evaluation results according to the system and the user perspectives. More specifically, the system performed better when evaluated from the user perspective than from the system perspective.