Efficacy vs Effectiveness Trial Results of an Indicated "Model" Substance Abuse Program: Implications for Public Health

The US Department of Education requires schools to choose substance abuse and violence prevention programs that meet standards of effectiveness. The Substance Abuse and Mental Health Services Agency certifies "model" programs that meet this standard. We compared findings from a large, multisite effectiveness trial of 1 model program to its efficacy trial findings, upon which the certification was based. 1370 high-risk youths were randomized to experimental or control groups across 9 high schools in 2 large urban school districts. We used intent-to-treat and on-treatment approaches to examine baseline equivalence, attrition, and group differences in outcomes at the end of the program and at a 6-month follow-up. Positive efficacy trial findings were not replicated in the effectiveness trial. All main effects were either null or worse for the experimental than for the control group. These findings suggest that small efficacy trials conducted by developers provide insufficient evidence of effectiveness. Federal agencies and public health scientists must work together to raise the standards of evidence and ensure that data from new trials are incorporated into ongoing assessments of program effects.