A critical reflection on the use of improvement science approaches in public health

Objective: ‘Improvement science’ is used to describe specific quality improvement methods (including tests of change and statistical process control). The approach is spreading from clinical settings to population-wide interventions and is being extended from supporting the adoption of proven interventions to making generalisable claims about new interventions. The objective of this narrative review is to evaluate the strengths and risks of current improvement science practice, particularly in relation to how they might be used in population health. Methods: A purposive sampling of published studies to identify how improvement science methods are being used and for what purpose. The setting was Scotland and studies that focused on health and wellbeing outcomes. Results: We have identified a range of improvement science approaches which provide practitioners with accessible tools to assess small-scale changes in policy and practice. The strengths of such approaches are that they facilitate consistent implementation of interventions already known to be effective and motivate and empower staff to make local improvements. However, we also identified a number of potential risks. In particular, their use to assess the effectiveness of new interventions often seems to pay insufficient attention to random variation, measurement bias, confounding and ethical issues. Conclusions: The use of current improvement science methods to generate evidence of effectiveness for population-wide interventions is problematic and risks unjustified claims of effectiveness, inefficient resource use and harm to those not offered alternative effective interventions. Newer methodological approaches offer alternatives and should be more widely considered.