Effects of nonstandard neutrino self-interactions and magnetic moment on collective Majorana neutrino oscillations

We derive the effective Hamiltonian describing collective oscillations of Majorana neutrinos with a transition magnetic moment, allowing for the presence of scalar and pseudoscalar nonstandard neutrino self-interactions (NSSIs). Using this Hamiltonian, we analyze new flavor instability channels of collective oscillations in a core-collapse supernova environment that open up in the presence of a small but nonzero neutrino magnetic moment. It turns out that, contrary to certain claims in the literature, within the minimally extended Standard Model (i.e., without NSSIs), no new instabilities arise within the linear order, nor do they produce any observable signatures in the neutrino flavor-energy spectra, at least for magnetic moments up to 10(-15)mu B and quite realistic fields of the order of 10(12) Gauss. On the other hand, in the presence of NSSIs, new fast and slow instabilities mixing neutrinos and antineutrinos appear, which show up in the spectra even for tiny magnetic moments of the order of 10(-24)mu B, leading to considerable distortions of the spectra and nonstandard spectral splits. We study sensitivity of collective oscillations to these, NSSI-induced instabilities in detail and discuss the observability of the NSSI couplings triggering them.