Mega mistakes in meta-analyses: devil in the detail

Science is a work in progress, with each new study attempting to add to, or improve upon, those that have come before. In this way we have moved initially from a characterization of the species richness-productivity relationship (SRPR) as being ubiquitously unimodel (Rosenzweig and Abramsky 1993, Huston and deAngelis 1994) to a transitional view of the relationship as one in which unimodel relationships were seen not to be dominant but to instead depend on the geographic scale of study (Mittelbach et al. 2001). More recently there has been a re-characterization in which the dominant form of the relationship has been found to be positive at both fine and course grains and at all but very local geographic scales (Gillman and Wright 2006). However, Robert Whittaker's (Whittaker 2010) analysis gives the impression that on this issue we have recently descended into chaos. He first suggests that a set of prescribed criteria should be followed, but then concludes that we should entirely abandon meta-analyses in favor of narrative review or more directed primary data collection. We have some sympathy with Whittaker's arguments but we take issue with the analysis he has undertaken and the conclusions he makes. We revisit his analysis and in so doing conclude that he has overstated the problem and that the way forward is not to abandon meta-analyses, but to ensure that greater caution is exercised when undertaking, reviewing and citing them. The habitual problem with any meta-analysis is, we believe, not primarily with the statistical analysis, but with the widespread indiscriminate use of studies that are fed into the analysis, that are entirely inappropriate to the question being asked. Statistical meta-analysis can resolve some of these issues. However, in some cases the statistical analysis that has been performed, rather than overcoming problems, adds to the deception with a veneer of respectability. Unfortunately, poorly derived meta-analyses continue to be cited without question.