A Reinforcement Learning Based Routing in Cognitive Radio Networks for Primary Users with Multi-stage Periodicity

Designing an efficient routing protocol for cognitive radio networks is critical due to the dynamic behavior of the primary users. Based on empirical studies, the primary users activity on the licensed channels has periodicity comprised of several stages, and that the model of primary users activity may change during different stages. This paper has identified two main challenges facing designers: how to transmit packets via a stable route, and how to ensure imposing of minimal interference on the primary users. To address these, we propose a routing protocol which is based on a generalized version of Q-learning and which exploits the said model of primary users behavior. We divide the infinite time horizon into cycles (corresponding to periods), then break each cycle into several sub-cycles (corresponding to stages), making an assumption that the statistical model parameters of the primary users activity will not change during a sub-cycle. The extensive simulations confirm that our proposed routing approach outperforms existing schemes in terms of throughput and minimal interference. The paper also verifies that imposing of significant interference on the primary users and degradation of QoS of secondary users stem from lack of attention to the multi-stage periodic behavior of primary users.