Antiferroquadrupolar order and rotational symmetry breaking in a generalized bilinear-biquadratic model on a square lattice

The magnetic and nematic properties of the iron chalcogenides have recently been the subject of intense interest. Motivated by the proposed antiferroquadrupolar and Ising-nematic orders for the bulk FeSe, we study the phase diagram of an \(S=1\) generalized bilinear-biquadratic model with multi-neighbor interactions. We find a large parameter regime for a (\(\pi\),0) antiferroquadrupolar phase, showing how quantum fluctuations stabilize it by lifting an infinite degeneracy of certain semiclassical states. Evidence for this C\(_4\)-symmetry-breaking quadrupolar phase is also provided by an unbiased density matrix renormalization group analysis. We discuss the implications of our results for FeSe and related iron-based superconductors.