Quantum Transitions of Nematic Phases in a Spin-\(1\) Bilinear-Biquadratic Model and Their Implications for FeSe

Since its discovery, iron-based superconductivity has been known to develop near an antiferromagnetic order, but this paradigm fails in the iron chalcogenide FeSe, whose single-layer version holds the record for the highest superconducting transition temperature in the iron-based superconductors. The striking puzzle that FeSe displays nematic order (spontaneously broken lattice rotational symmetry) while being non-magnetic, has led to several competing proposals for its origin in terms of either the \(3d\)-electron's orbital degrees of freedom or spin physics in the form of frustrated magnetism. Here we argue that the phase diagram of FeSe under pressure could be qualitatively described by a quantum spin model with highly frustrated interactions. We implement both the site-factorized wave-function analysis and the large-scale density matrix renormalization group (DMRG) in cylinders to study the spin-\(1\) bilinear-biquadratic model on the square lattice, and identify quantum transitions from the well-known \((\pi,0)\) antiferromagnetic state to an exotic \((\pi,0)\) antiferroquadrupolar order, either directly or through a \((\pi/2,\pi)\) antiferromagnetic state. These many phases, while distinct, are all nematic. We also discuss our theoretical ground-state phase diagram for the understanding of the experimental low-temperature phase diagram obtained by the NMR [P. S. Wang {\it et al.}, Phys. Rev. Lett. 117, 237001 (2016)] and X-ray scattering [K. Kothapalli {\it et al.}, Nature Communications 7, 12728 (2016)] measurements in pressurized FeSe. Our results suggest that superconductivity in a wide range of iron-based materials has a common origin in the antiferromagnetic correlations of strongly correlated electrons.