Nematic spin liquid phase in a frustrated spin-1 system on the square lattice

Frustration in quantum spin systems promote a variety of novel quantum phases. An important example is the frustrated spin-1 model on the square lattice with the nearest-neighbor bilinear (J1) and biquadratic (K1) interactions. We provide strong evidence for a nematic spin liquid phase in a range of K1/J1 near the SU(3)-symmetric point (J1=K1), based on the linear flavor-wave theory and extensive density matrix renormalization group calculation. This phase displays no spin dipolar or quadrupolar order, preserves translational symmetry but spontaneously breaks C4 lattice rotational symmetry, and possesses fluctuations peaked at the wave vector (π,2π/3). The spin excitation gap drops rapidly with system size and appears to be gapless, and the nematic order is attributed to the dominant (π,2π/3) fluctuations. Our results provide a novel mechanism for electronic nematic order and, more generally, open up a new avenue to explore frustration-induced exotic ground states.