Text deduplication method and device, electronic equipment and computer readable storage medium

The embodiment of the invention provides a text deduplication method and a device, electronic equipment and a computer readable storage medium, and relates to the technical field of text deduplication. According to the m of the invention, the method comprises the steps: firstly, carrying out the sorting of all obtained text data; secondly, performing splitting processing on each piece of text dataobtained after sorting processing is conducted, and obtaining a text set corresponding to each piece of text data; then, similarity comparison is conducted on the adjacent text sets, and duplicate removal processing is conducted; through the method, the text deduplication efficiency can be improved. 本申请实施例提供的文本去重方法和装置、电子设备及计算机可读存储介质，涉及文本去重技术领域。在本申请实施例中，首先，对获取的各文本数据进行排序处理。其次，针对进行排序处理后的每个文本数据分别进行拆分处理，得到与每个文本数据分别对应的文本集合。然后，对相邻的文本集合进行相似度比较并进行去重处理。通过上述方法，可以提高文本去重的效率。