Deep unfolding for cooperative rate splitting multiple access in hybrid satellite terrestrial networks

Rate splitting multiple access (RSMA) has shown great potentials for the next generation communication systems. In this work, we consider a two-user system in hybrid satellite terrestrial network (HSTN) where one of them is heavily shadowed and the other uses cooperative RSMA to improve the transmission quality. The non-convex weighted sum rate (WSR) problem formulated based on this model is usually optimized by computational burdened weighted minimum mean square error (WMMSE) algorithm. We propose to apply deep unfolding to solve the optimization problem, which maps WMMSE iterations into a layer-wise network and could achieve better performance within limited iterations. We also incorporate momentum accelerated projection gradient descent (PGD) algorithm to circumvent the complicated operations in WMMSE that are not amenable for unfolding and mapping. The momentum and step size in deep unfolding network are selected as trainable parameters for training. As shown in the simulation results, deep unfolding scheme has WSR and convergence speed advantages over original WMMSE algorithm.