Federated Large Language Model: Solutions, Challenges and Future Directions

Large language models (LLMs) have become increasingly popular due to their exceptional performance in various artificial intelligence applications. However, their development often suffers from the scarcity of high-quality data and the extensive requirements for computing resources. These obstacles are even more severe for enterprises in vertical industries, which have limited computer resources but urgently require large-scale models for specific activities. To address these issues, LLMs call for the integration of federated learning (FL), which enables the collaborative learning of a powerful LLM using private data and computing resources from multiple entities. In this article, we present a systematic introduction to the federated large language model (Fed-LLM), a distributed learning of LLM in the FL manner. We first introduce the learning paradigm of Fed-LLM, which is called federated parameter-efficient fine-tuning (Fed-PEFT). Fed-PEFT empowers the collaborative fine-tuning of pre-trained LLMs by only involving a small subset of parameters in local LLMs. Specifically, we detail the workflow of Fed-PEFT, and summarize the state-of-the-art solutions in this area. Additionally, we discuss the challenges faced in Fed-LLMs, including efficiency, privacy, and security. Finally, we introduce future directions to facilitate the research of Fed-LLMs and guide coming explorations in this nascent field.