Reliable prediction intervals with regression neural networks

This paper proposes an extension to conventional regression neural networks (NNs) for replacing the point predictions they produce with prediction intervals that satisfy a required level of confidence. Our approach follows a novel machine learning framework, called Conformal Prediction (CP), for assigning reliable confidence measures to predictions without assuming anything more than that the data are independent and identically distributed (i.i.d.). We evaluate the proposed method on four benchmark datasets and on the problem of predicting Total Electron Content (TEC), which is an important parameter in trans-ionospheric links; for the latter we use a dataset of more than 60000 TEC measurements collected over a period of 11 years. Our experimental results show that the prediction intervals produced by our method are both well calibrated and tight enough to be useful in practice. ► A regression neural network approach for producing reliable prediction intervals. ► Validity is guaranteed without assuming anything more than i.i.d. data. ► Prediction interval sizes vary according to the difficulty of each example. ► Experiments demonstrate that the resulting prediction intervals are well calibrated. ► Moreover, the obtained prediction intervals are tight enough to be useful in practice.