Preference judgements for relevance

The claimed subject matter provides a system that trains or evaluates ranking techniques by employing or obtaining relative preference judgments. The system can include mechanisms that retrieve a set of documents from a storage device, combine the set of documents with a query or judgment task received via an interface to form a comparative selection panel, and present the comparative selection panel for evaluation by an assessor. The system further requests the assessor to make a selection as to which document included in the set of documents and presented in the comparative selection panel most satisfies the query or judgment task, and thereafter produces a comparative assessment of the set of documents based on the selections elicited from the assessor and associated with the set of documents.