System and method for hybrid minimum mean squared error matrix-pencil separation weights for blind source separation

A technique for blind source separation ("BSS") of statistically independent signals with low signal-to-noise plus interference ratios under a narrowband assumption utilizing cumulants in conjunction with spectral estimation of the signal subspace to perform the blind separation is disclosed. The BSS technique utilizes a higher-order statistical method, specifically fourth-order cumulants, with the generalized eigen analysis of a matrix-pencil to blindly separate a linear mixture of unknown, statistically independent, stationary narrowband signals at a low signal-to-noise plus interference ratio having the capability to separate signals in spatially and/or temporally correlated Gaussian noise. The disclosed BSS technique separates low-SNR co-channel sources for observations using an arbitrary un-calibrated sensor array. The disclosed BSS technique forms a separation matrix with hybrid matrix-pencil adaptive array weights that minimize the mean squared errors due to both interference emitters and Gaussian noise. The hybrid weights maximize the signal-to interference-plus noise ratio.