Adequacy of sample size for estimating a value from field observational data

In 2011, the U.S. EPA Office of Research and Development released a field-based method for deriving aquatic life benchmarks for conductivity. Since its release, it has been verified, validated, and corroborated by the authors, reviewers, and independent researchers. However, the method and published results have been recently challenged as being artifacts of small sample sizes, prompting this re-evaluation. This paper supplements prior causal analyses by weighing evidence that specifically addresses the hypothesis that the benchmark is a statistical artifact. Four types of evidence are presented: (1) Permutation analyses show that the data sets are able to reliably estimate the extirpation of 5% of genera. (2) Analyses show that 25 occurrences of a genus are sufficient to estimate extirpation. (3) Coherent ecological explanations show that the claimed influence of sample size is actually a result of community ecology. (4) A review of relevant independent studies supports the benchmark. The permutation test is a useful test of the adequacy of field data sets. Furthermore, this weight-of-evidence approach and the individual types of evidence can be a model for analysis of other field-based benchmark values. •The derivation of benchmark values from field observations is validated.•Observed effects were compared to effects after permutation of exposures.•Effects relative to sample size are consistent with ecology not sampling error.•Independent studies with different methods and data corroborated the estimates.•Evidence shows that effects are real and sample sizes are not unduly influential.