Modulation instability induced by cross-phase modulation in negative index materials with higher-order nonlinearity

Based on the dispersive Drude model in negative index materials (NIMs), nonlinear coupled Schodinger equations are derived for two copropagating optical waves with cubic-quintic nonlinearity and modulation instabilities induced by cross-phase modulation (XMI) are studied by using standard linear stability analysis and the Drude electromagnetic model. It is shown that the quintic nonlinearity strengthens the XMI with broader XMI spectra and higher peak gain. It is found that the XMI gain is obviously larger in the region with anomalous group velocity dispersion (GVD) than that in the region with normal GVD. It is also shown that the serious XMI occurs when two optical waves propagate simultaneously in the positive refractive index region or in different refractive index regions. The work provides a theoretical basis for the extinction or utilization of XMI in the propagation of high power and high speed signals in NIMs. •Study modulation instabilities in cross-phase modulation (XMI) in metamaterials.•Based on the dispersive Drude model in negative index materials (NIM).•Derive nonlinear coupled Schodinger equations with cubic-quintic nonlinearity.•Show quintic nonlinearity strengthens XMI with broader spectra and higher gain.•Provide a theoretical basis for the extinction or utilization of XMI.