U-Statistics for Importance-Weighted Variational Inference

We propose the use of U-statistics to reduce variance for gradient estimation in importance-weighted variational inference. The key observation is that, given a base gradient estimator that requires \(m > 1\) samples and a total of \(n > m\) samples to be used for estimation, lower variance is achieved by averaging the base estimator on overlapping batches of size \(m\) than disjoint batches, as currently done. We use classical U-statistic theory to analyze the variance reduction, and propose novel approximations with theoretical guarantees to ensure computational efficiency. We find empirically that U-statistic variance reduction can lead to modest to significant improvements in inference performance on a range of models, with little computational cost.