Quantitative investigation of the short-range magnetic correlations in the candidate quantum spin liquid NaYbO2

Here, we present a neutron diffraction study of NaYbO2, a candidate quantum spin-liquid (QSL) compound hosting a geometrically frustrated triangular lattice of magnetic Yb3+ ions. We observe diffuse magnetic scattering that persists to at least 20 K, demonstrating the presence of short-range magnetic correlations in this system up to a relatively high-energy scale. Using a reverse Monte Carlo and magnetic pair distribution function analysis, we confirm the predominant antiferromagnetic nature of these correlations and show that the diffuse scattering data can be well described by noninteracting layers of Heisenberg or XY spins on the triangular lattice. We rule out Ising spins and short-range-ordered stripe or 120° phases as candidate ground states of NaYbO2. These results are consistent with a possible QSL ground state in NaYbO2 and showcase the benefit of a combined reciprocal- and real-space analysis of materials with short-range magnetic correlations.