Measuring population differentiation using GST or D? A simulation study with microsatellite DNA markers under a finite island model and nonequilibrium conditions

The genetic differentiation of populations is a key parameter in population genetic investigations. Wright’s FST (and its relatives such as GST) has been a standard measure of differentiation. However, the deficiencies of these indexes have been increasingly realized in recent years, leading to some new measures being proposed, such as Jost’s D (Molecular Ecology, 2008; 17, 4015). The existence of these new metrics has stimulated considerable debate and induced some confusion on which statistics should be used for estimating population differentiation. Here, we report a simulation study with neutral microsatellite DNA loci under a finite island model to compare the performance of GST and D, particularly under nonequilibrium conditions. Our results suggest that there exist fundamental differences between the two statistics, and neither GST nor D operates satisfactorily in all situations for quantifying differentiation. D is very sensitive to mutation models but GST noticeably less so, which limits D’s utility in population parameter estimation and comparisons across genetic markers. Also, the initial heterozygosity of the starting populations has some important effects on both the individual behaviours of GST and D and their relative behaviours in early differentiation, and this effect is much greater for D than GST. In the early stages of differentiation, when initial heterozygosity is relatively low (