Monte Carlo study of Potts versus Kosterlitz-Thouless transitions in a 2D generalized XY model

The generalized XY model has seen renewed interest recently. This model contains both the spin-spin exchange interaction and q-fold nematic interaction. Due to competition between these two kinds of interaction, the model exhibits a rich phase diagram with various phases and unusual phase transitions. Here, we perform an extensive Monte Carlo simulation for this model, focusing on the case of q=3, which corresponds to a three-fold nematic interaction. We use the correlation length to locate the phase transition lines, including the Kosterlitz-Thouless (KT) line, the 1/3 KT line, and the Potts line, which meet at the tricritical point. Further, the magnetic correlation length gives more insights into the tricritical region. Finally, it signifies an intermediate region starting from the tricritical point, where the transition line is neither of the same physics as the Pott transition below nor the KT transition far above the tricritical point. •We study the phase transition of a generalized 2D XY model with a three-fold nematic-like interaction.•We reconstruct the phase diagram in this model based purely on the magnetic and nematic correlation lengths.•The numerical evidence shows an intermediate region between the three-state Potts and the KT phase transition lines.