A general framework for forecast verification

A general framework for forecast verification based on the joint distribution of forecasts and observations is described. For additional elaboration of the framework, two factorizations of the joint distribution are investigated: 1) the calibration-refinement factorization, which involves the conditional distributions of observations, given forecasts, and the marginal distribution of forecasts; and 2) the likelihood-base rate factorization, which involves the conditional distributions of forecasts, given observations, and the marginal distribution of observations. The names given to the factorizations reflect the fact that they relate to different attributes of the forecasts and/or observations. Several examples are used to illustrate the interpretation of these factorizations in the context of verification and to describe the relationship between the respective factorizations. Some insight into the potential utility of the framework is provided by demonstrating that basic elements and summary measures of the joint, conditional, and marginal distributions play key roles in current verification methods. The need for additional investigation of the implications of this framework for verification theory and practice is emphasized, and some possible directions for further research in this area are identified.