Bootstrap test procedure for variance components in nonlinear mixed effects models in the presence of nuisance parameters and a singular Fisher information matrix

We examine the problem of variance component testing in general mixed effects models using the likelihood ratio test. We account for the presence of nuisance parameters, ie, the fact that some untested variances might also be equal to zero. Two main issues arise in this context, leading to a nonregular setting. First, under the null hypothesis, the true parameter value lies on the boundary of the parameter space. Moreover, due to the presence of nuisance parameters, the exact locations of these boundary points are not known, which prevents the use of classical asymptotic theory of maximum likelihood estimation. Then, in the specific context of nonlinear mixed effects models, the Fisher information matrix is singular at the true parameter value. We address these two points by proposing a shrunk parametric bootstrap procedure, which is straightforward to apply even for nonlinear models. We show that the procedure is consistent, solving both the boundary and the singularity issues, and we provide a verifiable criterion for the applicability of our theoretical results. We show through a simulation study that, compared to the asymptotic approach, our procedure has a better small sample performance and is more robust to the presence of nuisance parameters. A real data application on bird growth rates is also provided.