On scoping a test that addresses the wrong objective

The Department of Defense test and evaluation community uses power as a key metric for sizing test designs. Power depends on many elements of the design, including the selection of response variables, factors and levels, model formulation, and sample size. The experimental objectives are expressed as hypothesis tests, and power reflects the risk associated with correctly assessing those objectives. Statistical literature refers to a different, yet equally important, type of error that is committed by giving the right answer to the wrong question. If a test design is adequately scoped to address an irrelevant objective, one could say that a Type III error occurs. In this paper, we focus on a specific Type III error that on some occasions test planners commit to reduce test size and resources. We provide a case study example that shows how reparameterizing a factor space from fewer factors with more levels per factor to a space that has more factors with fewer levels per factor fundamentally changes the hypothesis tests, and hence may no longer be aligned with the original objectives of the experiment. Despite the perceived increase in power and decrease in test resources that comes from this reparameterization, we conclude, it is not a prudent way to gain test efficiency. Through the case study example, we highlight the information that is lost in this decision and its implications on test objectives.