A Two‐Stage Framework for Bias and Reliability Tests of Ensemble Hydroclimatic Forecasts

The popular probability integral transform (PIT) uniform plot presents informative empirical illustrations of five types of ensemble forecasts, that is, reliable, under‐confident, over‐confident, negatively biased and positively biased. This paper has built a novel two‐stage framework upon the PIT uniform plot to quantitatively examine the forecast attributes of bias and reliability. The first stage utilizes the test statistic on bias to examine whether the mean of PIT values is equal to the theoretical mean of standard uniform distribution. Then, the second stage uses the test statistic on reliability to examine whether the mean squared deviation from the theoretical mean is equal to the theoretical variance of standard uniform distribution. Therefore, by using the two‐tailed bootstrap hypothesis testing, the first stage identifies unbiased ensemble forecasts, negatively biased forecasts and positively biased forecasts; the second stage focuses on unbiased ensemble forecasts to furthermore identify reliable forecasts, under‐confident forecasts and over‐confident forecasts. Numerical experiments are devised for the National Centers for Environmental Prediction's Climate Forecast System version 2 ensemble forecasts of global precipitation. The results highlight the existence of various shapes of the PIT uniform plots. Due to extreme values of observed precipitation, the PIT uniform plots in some cases can substantially deviate from the 1:1 line even though the mean and variance of ensemble forecasts are respectively in accordance with the mean and variance of observations. Nevertheless, the two‐stage framework along with the two test statistics serves as a robust tool for the verification of ensemble hydroclimatic forecasts. Plain Language Summary Ensemble hydroclimatic forecasts provide useful information for water resources planning and management. The probability integral transform (PIT) uniform plot presents informative visualizations of five types of ensemble forecasts, that is, reliable, under‐confident, over‐confident, negatively biased and positively biased. While the use of PIT uniform plot is for individual grid cells, this paper proposes a novel two‐stage framework along with two test statistics for the examinations of forecast bias and reliability at the global scale. Specifically, the test of bias in the first stage identifies unbiased, negatively biased and positively biased ensemble forecasts; and then, the test of reliability in the second s