The effect of Lévy index coefficient on modulational instability and rogue wave excitation in nonlocal media with cubic–quintic nonlinearities

This paper explores the modulational instability (MI) of a plane wave and its behavior in the nonlinear Schrödinger equation (NLSE) with a fractional diffraction term quantified by its Lévy index coefficient and nonlocal cubic–quintic nonlinearities. First, we analyze the stability of the plane wave solution and examine how nonlocal nonlinearities and the Lévy index coefficient affect the MI gain. We observe that the stability in the fractional NLSE exhibits new features that differ from those in the standard NLSE. Specifically, when dealing with competing cubic and quintic nonlinearities, the interaction between nonlocality and the Lévy index coefficient α can eliminate MI for low values of α, unlike the classical NLSE with α=2, where we find the plane wave to be unstable. Besides the linear stability analysis, numerical simulations are performed to understand further the plane wave dynamics from its nonlinear stage in this model. The results reveal the generation of periodic chains of localized peaks. Guided by analytical predictions and using the plane wave solution subject to Gaussian perturbation, we numerically investigate the possibility of exciting rogue waves in the parameter spaces where MI exists. We find that the different combinations of signs of the cubic and quintic nonlinearities (focusing and defocusing) and the fractional diffraction term significantly impact the formation of rogue waves. These results may pave the way for the theoretical and experimental study of nonlinear phenomena in physical models with fractional derivatives and nonlocal nonlinearities. •Modulational instability is studied in the nonlinear Schrödinger equation with a Lévy index and nonlocal cubic–quintic nonlinearities.•Under competing cubic–quintic nonlinearities, the Lévy index tends to suppress wave instability.•Periodic chains of localized peaks are generated along with rogue waves in modulational instability parameter spaces.•Different combinations of cubic–quintic nonlinearities and the Lévy term significantly influence rogue wave formation.