Phase competition in frustrated anisotropic antiferromagnet in strong magnetic field

•Continuous Ising-type phase transition between conical and fan phase is described.•Competing commensurate ordering leads to complicated scenarios of phase transitions.•Analytical expressions for critical fields are derived.•Theory successfully describes phase transitions in multiferroic MnWO4. We discuss theoretically a frustrated Heisenberg antiferromagnet in magnetic field close to the saturation one. It is demonstrated that a small biaxial anisotropy and/or the magnetic dipolar interaction produce a delicate balance between phases with a commensurate canted, incommensurate helical (conical), and fan spin orderings. As a result, different sequences of phase transitions are realized depending on values of these small anisotropic interactions. We derive analytical expressions for critical fields and ground-state energies of the phases which are in a quantitative agreement with our and previous Monte-Carlo simulations.