Towards optimal load balancing in multi-zone Kubernetes clusters via reinforcement learning

With the advent of container technology, companies have been developing microservice-based applications, converting the old monolithic software into a group of loosely coupled containers, with the aim of offering greater flexibility and improving operational efficiency. When users access microservices, their initial point of contact is typically a load balancer. This component is responsible for distributing incoming traffic or requests between multiple instances of microservices. Traditional load balancing approaches mainly rely on round-robin, or weighted roundrobin algorithms which are inadequate to maintain the overall performance and scalability of microservice-based applications. Microservices are often deployed in dynamic environments needing a more adaptive and efficient load balancing strategy to optimize resources and reduce the overall latency for end users. This paper presents a dynamic load balancer for Kubernetes (K8s) clusters based on Reinforcement Learning (RL). It aims to minimize the overall latency while promoting fair distribution of requests. To achieve this goal, the load balancer considers both current network delays and processing loads in the cluster. The evaluation shows that our solution is effective even in environments where both the network traffic and the processing loads in the cluster change dynamically over time. In addition, this study highlights the flexibility of DeepSets neural networks in solving the load balancing challenge in diverse setups without retraining. The results show that the DeepSets algorithms can solve the microservice load balancing problem even in scenarios up to 30 times larger than the trained setup.