Dimensionality crossover to a two-dimensional vestigial nematic state from a three-dimensional antiferromagnet in a honeycomb van der Waals magnet

The effects of fluctuations and disorder, which are substantially enhanced in reduced dimensionalities, can play a crucial role in producing non-trivial phases of matter such as vestigial orders characterized by a composite order parameter. However, fluctuation-driven magnetic phases in low dimensions have remained relatively unexplored. Here we demonstrate a phase transition from the zigzag antiferromagnetic order in the three-dimensional bulk to a Z 3 vestigial Potts nematicity in two-dimensional few-layer samples of van der Waals magnet NiPS 3 . Our spin relaxometry and optical spectroscopy measurements reveal that the spin fluctuations are enhanced over the gigahertz to terahertz range as the layer number of NiPS 3 reduces. Monte Carlo simulations corroborate the experimental finding of threefold rotational symmetry breaking but show that the translational symmetry is restored in thin layers of NiPS 3 . Therefore, our results show that strong quantum fluctuations can stabilize an unconventional magnetic phase after destroying a more conventional one. Magnetic phases that are stabilized by quantum fluctuations in low dimensions are rare. A thickness-dependent crossover from three-dimensional antiferromagnetism to a two-dimensional vestigial nematic state that is driven by fluctuations has now been observed.