An Empirical Assessment of Realized A-Risk when Testing Hypotheses

The effect on the realized a-risk of a pilot sample variance to estimate o2 in the sample size formula for testing a population mean is examined. Two types of populations-normal and positively skewed, two values for nominal a-risk-.05 and .10, and two pilot sample sizes-10 and 20, were considered. For the underlying normal population, the distribution of realized a-risk did not seem to follow a readily discernable probability model. Nonetheless, the mean of the distribution appeared to be extremely close to the value of the specified a-risk, independent of the pilot sample size, indicating that a^ is an unbiased estimator of a in repeated sampling. On the other hand, the variability in the distribution was not nearly as tight as might be expected. This clearly suggests that using the pilot sample variance to estimate o2 on a one-time basis rather than repeated sampling may produce an extreme value for the realized a-risk which is much larger than specified. For the positively skewed finite population, the distribution of 6 greatly exaggerated the skewness. The mean was much smaller than the specified a-risk, while the variance was larger relative to the normal case and seemingly unaffected by an increase in the pilot sample size. At least one research implication manifests itself from this study. In controlling a hypothesis testing experiment with a designed sample size, investigators should recognize that the realized a-risk may be quite different than the nominal value.