Grading the grain: Consistent evaluation of research proposals

This describes the selection procedure for research proposals which was introduced by the Technology Foundation of the Netherlands in 1981. The procedure involves obtaining peer reviews, then the responses of the principal investigator are added to the review document. Twenty such proposal documents are then judged by a non-peer jury. We focus on the consistency of the judgement of the jury. A statistical analysis is given of how the jury verdicts are used, and the reliability of the method. As an experiment, some proposals were sent to two different juries. The outcome confirmed the theory and showed a positive correlation between the two juries. Although the first part of the paper is quite statistical, the section starting on page 9 (‘Main advantages’) should interest all readers.