Permutation methods

Permutation tests are a paradox of old and new. Permutation tests pre‐date most traditional parametric statistics, but only recently have become part of the mainstream discussion regarding statistical testing. Permutation tests follow a permutation or ‘conditional on errors’ model whereby a test statistic is computed on the observed data, then (1) the data are permuted over all possible arrangements of the data—an exact permutation test; (2) the data are used to calculate the exact moments of the permutation distribution—a moment approximation permutation test; or (3) the data are permuted over a subset of all possible arrangements of the data—a resampling approximation permutation test. The earliest permutation tests date from the 1920s, but it was not until the advent of modern day computing that permutation tests became a practical alternative to parametric statistical tests. In recent years, permutation analogs of existing statistical tests have been developed. These permutation tests provide noteworthy advantages over their parametric counterparts for small samples and populations, or when distributional assumptions cannot be met. Unique permutation tests have also been developed that allow for the use of Euclidean distance rather than the squared Euclidean distance that is typically employed in parametric tests. This overview provides a chronology of the development of permutation tests accompanied by a discussion of the advances in computing that made permutation tests feasible. Attention is paid to the important differences between ‘population models’ and ‘permutation models’, and between tests based on Euclidean and squared Euclidean distances. WIREs Comp Stat 2011 3 527–542 DOI: 10.1002/wics.177 This article is categorized under: Statistical and Graphical Methods of Data Analysis > Bootstrap and Resampling Statistical and Graphical Methods of Data Analysis > Nonparametric Methods Statistical and Graphical Methods of Data Analysis > Robust Methods