Designing Probability Samples to Study Treatment Effect Heterogeneity

This chapter explains a new approach that survey samplers can use when designing probability samples for survey experiments where there is a possibility of treatment heterogeneity. It begins by explaining why probability samples are preferred to nonprobability samples for estimating two quantities (or estimands): population average treatment effects and treatment effects within subgroups. The chapter furthermore explains why typical probability sampling methods that optimize statistical power for the average effect in a population do not necessarily optimize statistical power for the subgroup effects of interest – especially when one‐s interest is in estimating effects within a rare subgroup. Next, it explains why even large, well‐constructed, highly representative probability samples with randomized treatments can produce confounded analyses of differences across subgroups. The chapter illustrates the proposed approach using an empirical case study of a survey‐administered behavioral science intervention: The US National Study of Learning Mindsets.