Is self-fertilization an evolutionary dead end? Revisiting an old hypothesis with genetic theories and a macroevolutionary approach

G. Ledyard Stebbins suggested that self-fertilization (selfing) may be an evolutionary dead end because it may result in the loss of genetic diversity and consequently preclude adaptation to changing environments. While the basic premise of selfing as a dead end is widely accepted, there have been few rigorous evaluations of the hypothesis. We examine the foundations of the dead-end hypothesis by considering theoretical advances in the study of mating-system evolution. We discuss theories predicting the irreversibility of self-fertilization and the extinction of selfing lineages through the loss of adaptive potential and genetic degradation. In the second portion of the review, focusing on the irreversibility of selfing, we summarize the contribution of phylogenetic studies of mating-system evolution to determine if evolutionary history supports this well-established hypothesis. Most studies are in accord with the hypothesis; no single study unequivocally demonstrates the transition from highly selfing to outcrossing lineages. Finally, we discuss the problems encountered when phylogenetic studies rely on reconstruction of ancestral mating systems. To avoid some of these problems, we applied likelihood ratio tests of irreversibility of mating-system evolution to several data sets and found that current data sets are probably too small for this test.