The consistency of human judgments of relevance

A comparison of the ability of humans to consistently judge the relevance of documents to their general interests from bases of citations, s, keywords, and total text was made under controlled experimental conditions. The results showed that 1) humans are able to make such judgments consistently, and 2) the consistency of the judgment is independent of the particular base from which it is made. Apparent inconsistency arising from judgments made on the basis of s remains unexplained. This experiment, as well as others concerned with human evaluations of text material, leave unexplored the basic problem of providing a metric scale on which such evaluations can be measured.