Investigation of bias due to selective inclusion of study effect estimates in meta‐analyses of nutrition research

We aimed to explore, in a sample of systematic reviews (SRs) with meta‐analyses of the association between food/diet and health‐related outcomes, whether systematic reviewers selectively included study effect estimates in meta‐analyses when multiple effect estimates were available. We randomly selected SRs of food/diet and health‐related outcomes published between January 2018 and June 2019. We selected the first presented meta‐analysis in each review (index meta‐analysis), and extracted from study reports all study effect estimates that were eligible for inclusion in the meta‐analysis. We calculated the Potential Bias Index (PBI) to quantify and test for evidence of selective inclusion. The PBI ranges from 0 to 1; values above or below 0.5 suggest selective inclusion of effect estimates more or less favourable to the intervention, respectively. We also compared the index meta‐analytic estimate to the median of a randomly constructed distribution of meta‐analytic estimates (i.e., the estimate expected when there is no selective inclusion). Thirty‐nine SRs with 312 studies were included. The estimated PBI was 0.49 (95% CI 0.42–0.55), suggesting that the selection of study effect estimates from those reported was consistent with a process of random selection. In addition, the index meta‐analytic effect estimates were similar, on average, to what we would expect to see in meta‐analyses generated when there was no selective inclusion. Despite this, we recommend that systematic reviewers report the methods used to select effect estimates to include in meta‐analyses, which can help readers understand the risk of selective inclusion bias in the SRs.