Automated selection of search ranker

A method for automated selection of a search result ranker comprising: providing a set of queries; for each of said queries, receiving, from a search engine, a plurality of relevancy score sets, wherein each relevancy score set is associated with search results found in a corpus of electronic documents using each of a plurality of computerized search result rankers; calculating a difficulty score for each of said queries relative to all other queries in the set, based on said plurality of relevancy score sets associated with said query; calculating a quality score for each of said search result rankers based on said plurality of relevancy score sets associated with said search result ranker, wherein each of said plurality of relevancy score sets is weighed according to the difficulty score of its associated query; and selecting one of said search rankers based on said quality score.