Simple steps for improving multiple-reader studies in radiology

Multiple-reader study designs have become popular in the radiology literature. We reviewed the major papers published in the American Journal of Roentgenology in the first 4 months of each of the years 1990 and 1995. The review was restricted to prospective studies of image interpretation. In the 1990 literature, we noted eight multiple-reader and 18 single-reader studies; in contrast, in the 1995 literature, we found 29 multiple-reader and eight single-reader studies. This trend reflects an increased awareness of the importance of multiple-reader studies. We examined the Results sections of the 29 multiple-reader studies from 1995 to assess the authors' motives for incorporating such a design. In 16 studies (55%), readers independently interpreted all images. However, the authors usually reported only the average interpretation of the readers; in only seven of the 29 studies (24%) did the authors describe differences among readers' interpretations. In 13 studies, interpretations were performed exclusively through "consensus reading." The method(s) used to achieve a consensus often were not explained. Only two of the 29 studies had more than three readers. In contrast, all of these studies included multiple patients. The average patient sample size was 45. Furthermore, differences observed among patients were routinely reported and/or depicted.