Robust empirical calibration of p-values using observational data

Gruber and Tchetgen recently published a simulation that is intended to demonstrate the theoretical scenario in which empirical calibration may not be recommended. This article highlights and challenges the premise of some of the concerns raised by Gruber and Tchetgen and demonstrate how some empirical findings support empirical calibration as a robust approach to Type I error control. The authors believe their simulations are not realistic: the simulated estimates of negative controls showed severe bias with an odds ratio of 3, and perhaps more important, this bias was also simulated to be unusually homogeneous across the sample of negative controls.