cLegal-QA: a Chinese legal question answering with natural language generation methods

Legal question answering (Legal QA) aims to provide accurate and timely answers to legal questions, significantly reducing the workload of legal professionals. This approach improves the efficiency of the judiciary and ensures prompt, professional legal assistance to the public. Currently, a major challenge is the absence of a large-scale dataset tailored for Chinese generative legal question answering. To address this, our study developed a comprehensive automatic question answering dataset for Chinese civil law, named cLegal-QA, which comprises 14,000 high-frequency questions from Chinese legal communities. This dataset spans various legal disputes and includes questions, disputes, scenarios, multiple lawyer responses, and gold-standard answers from human annotators. Additionally, we employed a generative QA model specifically designed for the cLegal-QA dataset. The results indicate that fully-supervised models, notably UniLM, T5, and BART, substantially outperform zero-shot models on this dataset, with ChatYuan being the most effective among the zero-shot models. Our analysis also reveals that answers labeled with 60–80% accuracy yield the highest efficiency. Furthermore, we evaluated the real-world performance of these models with expert validation and applied transfer learning to new civil disputes. While the QA models demonstrate commendable performance on the dataset, there is still potential for further improvement.