Competition of Dzyaloshinskii-Moriya and higher-order exchange interactions in Rh/Fe atomic bilayers on Ir(111)

Using spin-polarized scanning tunneling microscopy and density functional theory we demonstrate the occurrence of a novel type of noncollinear spin structure in Rh/Fe atomic bilayers on Ir(111). We find that higher-order exchange interactions depend sensitively on the stacking sequence. For fcc-Rh/Fe/Ir(111) frustrated exchange interactions are dominant and lead to the formation of a spin spiral ground state with a period of about 1.5 nm. For hcp-Rh/Fe/Ir(111) higher-order exchange interactions favor a double-row wise antiferromagnetic or "uudd" state. However, the Dzyaloshinskii- Moriya interaction at the Fe/Ir interface leads to a small angle of about 4{\deg} between adjacent magnetic moments resulting in a canted "uudd" ground state.