Improving Wind Forecasts in the Lower Stratosphere by Distilling an Analog Ensemble Into a Deep Neural Network

We discuss improving forecasts of winds in the lower stratosphere using machine learning to postprocess the output of the European Centre for Medium‐Range Weather Forecasts (ECMWF) Integrated Forecast System. We postprocess global three‐dimensional predictions and demonstrate distilling the analog ensemble (AnEn) method into a deep neural network, which reduces postprocessing latency to near zero maintaining increased forecast skill. This approach reduces the error with respect to ECMWF high‐resolution deterministic prediction between 2–15% for wind speed and 15–25% for direction and is on par with ECMWF ensemble (ENS) forecast skill to hour 60. Verifying with Loon data from stratospheric balloons, AnEn has 20% lower error than ENS for wind speed and 15% for wind direction, despite significantly lower real‐time computational cost to ENS. Similar performance patterns are reported for probabilistic predictions, with larger improvements of AnEn with respect to ENS. We also demonstrate that AnEn generates a calibrated probabilistic forecast. Plain Language Summary We demonstrate improvements in predicting winds in the stratosphere using machine learning. Our approach uses predictions and analyses from the European Centre for Medium‐Range Weather Forecasts (ECMWF). By comparing how previous forecasts differed from what the winds ultimately were over many data points, we are able to modify the current forecast in a way that improves prediction of the winds observed by Loon high‐altitude balloons in the stratosphere. A common barrier to using approaches like this to generate global predictions is processing a large amount of information quickly enough to be useful. We demonstrate that by using machine learning, we are able to perform many of the slow calculations ahead of time and that these forecast improvements can be deployed in real applications. Key Points An analog ensemble generates accurate predictions of lower‐stratosphere winds and reliably quantifies the prediction uncertainty A cloud‐based distributed computing implementation builds global three‐dimensional predictions in tens of minutes Distilling the analog ensemble into a deep neural network allows scaling historical forecasts without slowing postprocessing speed