Stability of a spin-triplet nematic state near to a quantum critical point

We analyze a model of itinerant electrons interacting through a quadrupole density-density repulsion in three dimensions. At the mean-field level, the interaction drives a continuous Pomeranchuk instability towards d-wave, spin-triplet nematic order, which simultaneously breaks the SU(2) spin-rotation and spatial-rotation symmetries. This order is characterized by spin-antisymmetric, elliptical deformations of the Fermi surfaces of up and down spins. We show that the effects of quantum fluctuations are similar to those in metallic ferromagnets, rendering the nematic transition first order at low temperatures. Using the fermionic quantum order-by-disorder approach to self-consistently calculate fluctuations around possible modulated states, we show that the first-order transition is preempted by the formation of a helical spin-triplet d-density wave. Such a state is closely related to d-wave bond density wave order in square-lattice systems. Moreover, we show that it may coexist with a modulated, p-wave superconducting state.