Combining N-gram Statistical Model with Pre-trained Model to Correct Chinese Sentence Error

There have been a fund of studies on Chinese Grammatical Error Correction (CGEC) since it was proposed by NLPCC 2018 shared task 2. In previous studies, most researchers regarded this task as a Neural Machine Translation (NMT) task, which treated erroneous sentences as source-language and correct sentences as target-language. But this method relies on large-scale parallel corpus. In recent years, Bidirectional Encoder Representations from Transformers (BERT) and its variants have made an exciting breakthrough on various NLP tasks and inspire NLP practitioners to explore the utilization of pre-trained model. However, BERT performs better on Natural Language Understanding (NLU) benchmarks (e.g., SQuAD v1.1), the applications on generative tasks are inadequate. In NLP-TEA CGED Shared Task 2020, many methods based on BERT Pre-trained model have emerged. Unlike CGED tasks, whose purpose is to detect error position and error types in a sentence, are usually regarded as sequence labelling or binary classification problem. CGEC is a sequence generation task. In this study, we leverage n-gram statistical language model as a spelling checker and BERTbased pre-trained model as the encoder in sequence-to-sequence (seq2seq) structure to solve CGEC problem. Our baseline is Transformer. The experimental results demonstrate that our method outperforms the other three participating teams but also some latest methods, and we analyze how different checkpoints affect our results.