Object-Based Evaluation of a Storm-Scale Ensemble during the 2009 NOAA Hazardous Weather Testbed Spring Experiment

Object-based verification of deterministic forecasts from a convection-allowing ensemble for the 2009 NOAA Hazardous Weather Testbed Spring Experiment is conducted. The average of object attributes is compared between forecasts and observations and between forecasts from subensembles with different model dynamics. Forecast accuracy for the full ensemble and the subensembles with different model dynamics is also evaluated using two object-based measures: the object-based threat score (OTS) and the median of maximum interest (MMI). Forecast objects aggregated from the full ensemble are generally more numerous, have a smaller average area, more circular average aspect ratio, and more eastward average centroid location than observed objects after the 1-h lead time. At the 1-h lead time, forecast objects are less numerous than observed objects. Members using the Advanced Research Weather Research and Forecasting Model (ARW) have fewer objects, more linear average aspect ratio, and smaller average area than members using the Nonhydrostatic Mesoscale Model (NMM). The OTS aggregated from the full ensemble is more consistent with the diurnal cycles of the traditional equitable threat score (ETS) than the MMI because the OTS places more weight on large objects, while the MMI weights all objects equally. The group of ARW members has higher OTS than the group of NMM members except at the 1-h lead time when the group of NMM members has more accurate maintenance and evolution of initially present precipitation systems provided by radar data assimilation. The differences between the ARW and NMM accuracy are more pronounced with the OTS than the MMI and the ETS.