Anomalous behavior of the electronic structure of (Bi\(_{1-x}\)In\(_x\))\(_2\)Se\(_3\) across the quantum-phase transition from topological to trivial insulator

Using spin- and angle-resolved spectroscopy and relativistic many-body calculations, we investigate the evolution of the electronic structure of (Bi\(_{1-x}\)In\(_x\))\(_2\)Se\(_3\) bulk single crystals around the critical point of the trivial to topological insulator quantum-phase transition. By increasing \(x\), we observe how a surface gap opens at the Dirac point of the initially gapless topological surface state of Bi\(_2\)Se\(_3\), leading to the existence of massive fermions. The surface gap monotonically increases for a wide range of \(x\) values across the topological and trivial sides of the quantum-phase transition. By means of photon-energy dependent measurements, we demonstrate that the gapped surface state survives the inversion of the bulk bands which occurs at a critical point near \(x=0.055\). The surface state exhibits a non-zero in-plane spin polarization which decays exponentially with increasing \(x\), and that persists on both the topological and trivial insulator phases. Its out-of-plane spin polarization remains zero demonstrating the absence of a hedgehog spin texture expected from broken time-reversal symmetry. Our calculations reveal qualitative agreement with the experimental results all across the quantum-phase transition upon the systematic variation of the spin-orbit coupling strength. A non-time reversal symmetry breaking mechanism of bulk-mediated scattering processes that increase with decreasing spin-orbit coupling strength is proposed as explanation.