Reevaluation of statistically significant meta‐analyses in advanced cancer patients using the Hartung–Knapp method and prediction intervals—A methodological study

Using the Hartung–Knapp method and 95% prediction intervals (PIs) in random‐effects meta‐analyses is recommended by experts but rarely applied. Therefore, we aimed to reevaluate statistically significant meta‐analyses using the Hartung–Knapp method and 95% PIs. In this methodological study, three databases were searched from January 2010 to July 2019. We included systematic reviews reporting a statistically significant meta‐analysis of at least four randomized controlled trials in advanced cancer patients using either a fixed‐effect or random‐effects model. We investigated the impact of switching from fixed‐effect to random‐effects meta‐analysis and of using the recommended Hartung–Knapp method in random‐effects meta‐analyses. Furthermore, we calculated 95% PIs for all included meta‐analyses. We identified 6234 hits, of which 261 statistically significant meta‐analyses were included. Our recalculations of these 261 meta‐analyses produced statistically significant results in 132 of 138 fixed‐effect and 114 of 123 random‐effects meta‐analyses. When switching to a random‐effects model, 19 of 132 fixed‐effect meta‐analyses (14.4%) were no longer statistically significant. Using the Hartung–Knapp method in random‐effects meta‐analyses resulted in 34 of 114 nonsignificant meta‐analyses (29.8%). In the full sample (N = 261), the null effect was included by the 95% PI in 195 (74.7%) and the opposite effect (e.g., hazard ratio 0.5, opposite effect 2) in 98 meta‐analyses (37.5%). Using the Hartung–Knapp method and PIs substantially influenced the interpretation of many published, statistically significant meta‐analyses. We strongly encourage researchers to check if using the Hartung–Knapp method and reporting 95% PIs is appropriate in random‐effects meta‐analyses.