Exotic quantum phase transitions of strongly interacting topological insulators

Using determinant quantum Monte Carlo simulations, we demonstrate that an extended Hubbard model on a bilayer honeycomb lattice has two novel quantum phase transitions. The first is a quantum phase transition between the weakly interacting gapless Dirac fermion phase and a strongly interacting fully gapped and symmetric trivial phase, which cannot be described by the standard Gross-Neveu model. The second is a quantum critical point between a quantum spin Hall insulator with spin S super(z) conservation and the previously mentioned strongly interacting fully gapped phase. At the latter quantum critical point the single-particle excitations remain gapped, while spin and charge gaps both close. We argue that the first quantum phase transition is related to the Z sub(16) classification of the topological superconductor super(3)He-B phase with interactions, while the second quantum phase transition is a topological phase transition described by a bosonic O(4) nonlinear sigma model field theory with a [Theta] term.