Nonmagnetic Ground States and a Possible Quadrupolar Phase in 4d and 5d Lacunar Spinel Selenides Ga M4Se8 (M = Nb , Ta)

Structural and magnetic properties of cubic spinel selenides Ga M4 Se8 (M = Nb , Ta), which are candidates for the molecular Jeff = 3/2 Mott insulators, are investigated. The effective magnetic moments are reduced compared to the spin only value, indicating the presence of sizable spin-orbit coupling. GaNb4 Se8 and GaTa4 Se8 exhibit phase transitions into the nonmagnetic ground states with orthorhombic and tetragonal structures, respectively, which are robust against magnetic field up to at least 60 T. A cubic-cubic phase transition is observed in GaNb4Se8 preceding the magnetic transition, suggesting the existence of a quadrupolar-ordered phase theoretically predicted in the Jeff = 3/2 Mott insulator.