Expected and empirical coverages of different methods for generating noncentral t confidence intervals for a standardized mean difference

Different methods have been suggested for calculating “exact” confidence intervals for a standardized mean difference using the noncentral t distributions. Two methods are provided in Hedges and Olkin ( 1985 , “H”) and Steiger and Fouladi ( 1997 , “S”). Either method can be used with a biased estimator, d , or an unbiased estimator, g , of the population standardized mean difference (methods abbreviated H d , H g , S d , and S g ). Coverages of each method were calculated from theory and estimated from simulations. Average coverages of 95% confidence intervals across a wide range of effect sizes and across sample sizes from 5 to 89 per group were always between 85 and 98% for all methods, and all were between 94 and 96% with sample sizes greater than 40 per group. The best interval estimation was the S d method, which always produced confidence intervals close to 95% at all effect sizes and sample sizes. The next best was the H g method, which produced consistent coverages across all effect sizes, although coverage was reduced to 93–94% at sample sizes in the range 5–15. The H d method was worse with small sample sizes, yielding coverages as low as 86% at n = 5. The S g method produced widely different coverages as a function of effect size when the sample size was small (93–97%). Researchers using small sample sizes are advised to use either the Steiger & Fouladi method with d or the Hedges & Olkin method with g as an interval estimation method.