An assessment of program evaluation methods and quality in Australian prevention agencies

Issue addressed: This study aimed to examine evaluation methods and quality in Australian health promotion agencies and the factors associated with this. The evidence base for prevention strategies is limited, with the evidence generated through program evaluation by health promotion and disease prevention agencies lacking rigour. Despite the need to improve the quality of evaluation, there is limited evidence of what influences evaluation quality in the prevention field. Methods: Data were collected using the Evaluation Practice Analysis Survey and an audit and appraisal of evaluation reports. Descriptive analysis was used to examine evaluation characteristics and multivariable regression was used to explore the association between evaluation and organisational attributes and evaluation quality. Results: In total, 392 evaluation reports were reviewed from 78 government and non-government agencies. Process evaluation was conducted most frequently, followed by impact evaluation. Overall evaluation quality was low (median 24.5%). In multivariable regression analysis, only two factors were associated with evaluation quality: health promotion budget (ratio of geometric means 1.53 [95% CI 1.02-2.29]); and, conducting statewide or national prevention programs (1.38 [95% CI 1.05-1.82]). Conclusions: The findings show that the potential to improve evaluation quality is greatest in smaller organisations that deliver health promotion at a local or regional scale. So what? By improving the rigour of existing evaluation, there is opportunity to build the evidence base for prevention strategies, which highlights the importance of embedding the enablers of program learning and evidence generation within health promotion and prevention organisations.