Compared to What?: Establishing Environmental Baselines for Tornado Warning Skill

In any discussion of forecast evaluation, it is tempting to fall back on statements reflecting unverified assumptions: “this tornado warning had lower skill because the underlying meteorology reflected a complicated or atypical scenario,” or “that forecast performed worse than we would have expected given the straightforward setup.” These statements of what is and is not a reasonable expectation for warning skill are particularly relevant as the meteorological community’s focus has begun to emphasize non-classic storm environments (e.g., tornadoes spawned by quasi-linear convective systems). In this paper, we build a proof-of-concept methodology to quantify the effect of the near-storm environment on tornado warning skill, and we then test these methods on a 15-yr dataset composed of tens of thousands of tornado events and warnings over the contiguous United States. Our findings include that significant tornadoes rated (E)F2+ have a higher probability of detection (POD) than expected based on their near-storm environments, that nocturnal tornadoes have both worse POD and false alarm ratio (FAR) than even their marginal near-storm environments would suggest, and that tornadoes occurring during the summer months also show worse POD and FAR than their environment-based expectation. Quantifying these shifts in performance in an environmental skill score framework allows us to target the situations in which the greatest improvements may be possible, in terms of forecaster training and/or conceptual models. This work also highlights the essential question that should always be asked in the context of forecast verification: what, exactly, is the baseline standard to which we are comparing forecast performance?