Antiferroquadrupolar Order and Rotational Symmetry Breaking in a Generalized Bilinear-Biquadratic Model on a Square Lattice

The magnetic and nematic properties of the iron chalcogenides have recently been the subject of intense interest. Motivated by the proposed antiferroquadrupolar and Ising-nematic orders for the bulk FeSe, we study the phase diagram of an S=1 generalized bilinear-biquadratic model with multineighbor interactions. We find a large parameter regime for a (π, 0) antiferroquadrupolar phase, showing how quantum fluctuations stabilize it by lifting an infinite degeneracy of certain semiclassical states. Evidence for this C_{4}-symmetry-breaking quadrupolar phase is also provided by an unbiased density matrix renormalization group analysis. We discuss the implications of our results for FeSe and related iron-based superconductors.