Exogenous factors matter when interpreting the results of an impact evaluation: a case study of rainfall and child health programme intervention in Rwanda

Objective Public health interventions are often implemented at large scale, and their evaluation seems to be difficult because they are usually multiple and their pathways to effect are complex and subject to modification by contextual factors. We assessed whether controlling for rainfall‐related variables altered estimates of the efficacy of a health programme in rural Rwanda and have a quantifiable effect on an intervention evaluation outcomes. Methods We conducted a retrospective quasi‐experimental study using previously collected cross‐sectional data from the 2005 and 2010 Rwanda Demographic and Health Surveys (DHS), 2010 DHS oversampled data, monthly rainfall data collected from meteorological stations over the same period, and modelled output of long‐term rainfall averages, soil moisture, and rain water run‐off. Difference‐in‐difference models were used. Results Rainfall factors confounded the PIH intervention impact evaluation. When we adjusted our estimates of programme effect by controlling for a variety of rainfall variables, several effectiveness estimates changed by 10% or more. The analyses that did not adjust for rainfall‐related variables underestimated the intervention effect on the prevalence of ARI by 14.3%, fever by 52.4% and stunting by 10.2%. Conversely, the unadjusted analysis overestimated the intervention's effect on diarrhoea by 56.5% and wasting by 80%. Conclusion Rainfall‐related patterns have a quantifiable effect on programme evaluation results and highlighted the importance and complexity of controlling for contextual factors in quasi‐experimental design evaluations. Objectif Les interventions de santé publique sont souvent implémentées à grande échelle et leur évaluation semble être difficile car elles sont généralement multifactorielles et leurs voies vers l'effet sont complexes et sujettes à des modifications par des facteurs contextuels. Nous avons évalué si les ajustements selon les variables liées à la pluviosité pouvaient modifier les estimations de l'efficacité d'un programme de santé en zone rurale au Rwanda et avoir un effet quantifiable sur les résultats d’évaluation d'une intervention. Méthodes Nous avons mené une étude quasi‐expérimentale rétrospective utilisant des données transversales collectées précédemment à partir d'Enquêtes Démographiques et de Santé (EDS) au Rwanda en 2005 et 2010, des données suréchantillonnées de l’EDS de 2010, des données mensuelles de pluviosité recueillies dans les stations météorolog