Significance testing for small microarray experiments

Which significance test is carried out when the number of repeats is small in microarray experiments can dramatically influence the results. When in two sample comparisons both conditions have fewer than, say, five repeats traditional test statistics require extreme results, before a gene is considered statistically significant differentially expressed after a multiple comparisons correction. In the literature many approaches to circumvent this problem have been proposed. Some of these proposals use (empirical) Bayes arguments to moderate the variance estimates for individual genes. Other proposals try to stabilize these variance estimate by combining groups of genes or similar experiments. In this paper we compare several of these approaches, both on data sets where both experimental conditions are the same, and thus few statistically significant differentially expressed genes should be identified, and on experiments where both conditions do differ. This allows us to identify which approaches are most powerful without identifying many false positives. We conclude that after balancing the numbers of false positives and true positives an empirical Bayes approach and an approach which combines experiments perform best. Standard t‐tests are inferior and offer almost no power when the sample size is small. Copyright © 2005 John Wiley & Sons, Ltd.