An Optimal Linear Transformation for Data Assimilation

Linear transformations are widely used in data assimilation for covariance modeling, for reducing dimensionality (such as averaging dense observations to form “superobs”), and for managing sampling error in ensemble data assimilation. Here we describe a linear transformation that is optimal in the sense that, in the transformed space, the state variables and observations have uncorrelated errors, and a diagonal gain matrix in the update step. We conjecture, and provide numerical evidence, that the transformation is the best possible to precede covariance localization in an ensemble Kalman filter. A central feature of this transformation in the update step are scalars, which we term canonical observation operators (COOs), that relate pairs of transformed observations and state variables and rank‐order those pairs by their influence in the update. We show for an idealized problem that sample‐based estimates of the COOs, in conjunction with covariance localization for the sample covariance, can approximate well the true values, but a practical implementation of the transformation for high‐dimensional applications remains a subject for future research. The COOs also completely describe important properties of the update step, such as observation‐state mutual information, signal‐to‐noise and degrees of freedom for signal, and so give new insights, including relations among reduced‐rank approximations to variational schemes, particle‐filter weight degeneracy, and the local ensemble transform Kalman filter. Plain Language Summary Good estimates of the state of the Earth system rely on combining the latest observations with recent forecasts through a technique called data assimilation. To some degree, there is freedom to choose which variables are used in data assimilation and, before performing assimilation, to combine and transform the observations. Careful choices of variables and transformations of the observations not only have practical benefits, but also, by greatly simplifying the mathematical formulation of data assimilation, yield insights into how new observations influence Earth‐system state estimates and common measures of the information provided by observations. Key Points A linear transformation for observations and state variables uniquely diagonalizes the Kalman filter update equation A set of positive real numbers rank order the importance of transformed observations and state variables in the update In the new variables, optimal covariance loca