CONVERGENCE RATES OF LEAST SQUARES REGRESSION ESTIMATORS WITH HEAVY-TAILED ERRORS

We study the performance of the least squares estimator (LSE) in a general nonparametric regression model, when the errors are independent of the covariates but may only have a pth moment (p ≥ 1). In such a heavy-tailed regression setting, we show that if the model satisfies a standard “entropy condition” with exponent α ∈ (0, 2), then the L₂ loss of the LSE converges at a rate O P ( n − 1 2 + α ∨ n − 1 2 + 1 2 p ) . Such a rate cannot be improved under the entropy condition alone. This rate quantifies both some positive and negative aspects of the LSE in a heavy-tailed regression setting. On the positive side, as long as the errors have p ≥ 1 + 2/α moments, the L₂ loss of the LSE converges at the same rate as if the errors are Gaussian. On the negative side, if p < 1 + 2/α, there are (many) hard models at any entropy level α for which the L₂ loss of the LSE converges at a strictly slower rate than other robust estimators. The validity of the above rate relies crucially on the independence of the covariates and the errors. In fact, the L₂ loss of the LSE can converge arbitrarily slowly when the independence fails. The key technical ingredient is a new multiplier inequality that gives sharp bounds for the “multiplier empirical process” associated with the LSE. We further give an application to the sparse linear regression model with heavy-tailed covariates and errors to demonstrate the scope of this new inequality.