Least squares in calibration: Dealing with uncertainty in x

The least-squares (LS) analysis of data with error in x and y is generally thought to yield best results when carried out by minimizing the "total variance" (TV), defined as the sum of the properly weighted squared residuals in x and y . Alternative "effective variance" (EV) methods project the uncertainty in x into an effective contribution to that in y , and though easier to employ are considered to be less reliable. In the case of a linear response function with both σ x and σ y constant, the EV solutions are identically those from ordinary LS; and Monte Carlo (MC) simulations reveal that they can actually yield smaller root-mean-square errors than the TV method. Furthermore, the biases can be predicted from theory based on inverse regression - x upon y when x is error-free and y is uncertain - which yields a bias factor proportional to the ratio σ 2 x / σ 2 xm of the random-error variance in x to the model variance. The MC simulations confirm that the biases are essentially independent of the error in y , hence correctable. With such bias corrections, the better performance of the EV method in estimating the parameters translates into better performance in estimating the unknown ( x 0 ) from measurements ( y 0 ) of its response. The predictability of the EV parameter biases extends also to heteroscedastic y data as long as σ x remains constant, but the estimation of x 0 is not as good in this case. When both x and y are heteroscedastic, there is no known way to predict the biases. However, the MC simulations suggest that for proportional error in x , a geometric x -structure leads to small bias and comparable performance for the EV and TV methods. While special methods are usually needed to fit data with uncertainty in x and y , OLS is sometimes better!